Metadata-Version: 2.4
Name: foresight-agent-guard
Version: 0.1.2
Summary: Predictive cost, policy, and runaway-session guardrails for AI coding agents
Author: Aman M
License: MIT
Project-URL: Homepage, https://github.com/AmanM006/foresight
Project-URL: Repository, https://github.com/AmanM006/foresight
Project-URL: Issues, https://github.com/AmanM006/foresight/issues
Keywords: ai-agents,codex,gpt,cost-monitoring,developer-tools
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click<9,>=8.1
Requires-Dist: openai<3,>=1.0
Requires-Dist: python-dotenv<2,>=1.0
Requires-Dist: requests<3,>=2.31
Requires-Dist: rich<15,>=13.0
Requires-Dist: websockets<16,>=13
Dynamic: license-file

# Foresight

**Foresight is a predictive cost and tier-routing layer for AI coding agents.** It chooses the right level of reasoning before work begins, then watches the session closely enough to stop drift before it becomes expensive.

![CI](https://github.com/AmanM006/foresight/actions/workflows/ci.yml/badge.svg?branch=main)
![MIT License](https://img.shields.io/badge/license-MIT-111827?style=flat-square)
![Python](https://img.shields.io/badge/Python-3.11%2B-3776AB?style=flat-square)
![Next.js](https://img.shields.io/badge/Next.js-14-111111?style=flat-square)

## Judge quick start

Run the current submission from source. It needs Python only: no API key, Node.js, or browser.

>
> ```bash
> git clone https://github.com/AmanM006/foresight.git
> cd foresight
> pip install -e .
> foresight demo --replay
> python verify.py
> ```
>
> The replay launches a real instrumented Python child, shows the terminal dashboard, flags scope drift, then blocks a fake `.env` read in under a minute. `--web` additionally opens the optional local Next.js dashboard.

> **Install from PyPI**
>
> `pip install foresight-agent-guard==0.1.2` installs the current Python-child guard release.

## Verify this submission

Run the complete replay-safe verification pass with one command:

```bash
python verify.py
```

It runs the Python-child replay and deterministic tests without spending tokens. Add `--live` only when you want it to run the paid tier and injection benchmarks as well. For a zero-setup visual alternative, explore the deployed dashboard at [sentry-lime.vercel.app](https://sentry-lime.vercel.app).

## The problem

When developers use Codex/GPT-5.6 for coding work, they often do not know upfront which tier or reasoning effort a task actually needs. Defaulting to Sol overpays for mechanical work; choosing too small a tier can leave an agent stuck and repeatedly escalating.

The risk is not hypothetical. An open [Codex issue #32250](https://github.com/openai/codex/issues/32250) reports GPT-5.6 Sol Medium reducing a Pro five-hour allowance from 87% to 76% during a short conversation and several trivial follow-ups. Recent [r/codex](https://www.reddit.com/r/codex/comments/1ust6y8/usage_drops_much_faster_now_despite_less_cost/) and [r/ChatGPTPro](https://www.reddit.com/r/ChatGPTPro/comments/1us02nr/56_rate_limits/) threads describe confusion around five-hour versus weekly limits; these are community reports, not a claim about every session. Community guidance also identifies duplicated context and uncontrolled subagent spawning as sources of extra consumption ([discussion](https://www.reddit.com/r/codex/comments/1uu7weh/possible_fixes_for_gpt_56_burning_through_your/)).

There is a second failure mode: scope creep. A small implementation task can turn into unrequested architecture work, sprawling documentation, or broad file changes—the “Sol Ultra problem” discussed by users—silently consuming budget without an early warning. OpenAI documents that `ultra` coordinates four agents in parallel by default and trades higher token use for stronger results on demanding tasks; it is not an appropriate default for every coding task ([OpenAI GPT-5.6 overview](https://openai.com/index/gpt-5-6/)).

**Foresight solves this by predicting complexity and tier before execution, then watching the live session to catch drift before it compounds.**

## Live demo proof

One real recorded demo run produced the following result:

| Signal | Observed result |
| --- | --- |
| Predicted route | `luna / low / isolated` |
| Files touched | `16` |
| Predictive flag | `scope_explosion` |
| Final disposition | `escalated_for_review` |
| Recorded model cost | approximately `$0.0060` |

The demo showed two independent catch mechanisms live:

- **Predictive scope-drift detection:** unrelated file changes exceeded the predicted isolated blast radius.
- **Forbidden-pattern interception:** an attempted `.env` read was caught and blocked by the policy layer.

![Current local replay event feed showing real-time action interception](screenshots/dashboard-replay-live-feed.png)

![Current local replay showing the session killed banner after a forbidden-pattern block](screenshots/dashboard-replay-session-killed.png)

![Cost tracker showing real GPT-5.6-terra cost events](screenshots/cost-tracker.png)

The event-feed and kill-state captures above were refreshed from the current local `foresight demo --replay` path on 2026-07-16. Replay is finite, so the screenshots were captured after its event-bus process exited and the UI correctly shows `Offline / Pending connection`; the captured action feed and kill state are the events received during that run. The cost capture is intentionally archival: replay makes no API calls, so its cost stays `$0.00000`; the image shows the separately recorded live GPT-5.6-terra cost evidence linked below.

Inspect the sanitized raw evidence in [`examples/`](examples/): [`policy_and_verdict_output.example.json`](examples/policy_and_verdict_output.example.json) and [`audit_event_output.example.json`](examples/audit_event_output.example.json).

## Evidence and benchmarks

| Evidence | Real result |
| --- | --- |
| **Tier prediction accuracy** | 15 hand-labeled tasks spanning mechanical, moderate, and architectural complexity. **14/15** exact tier match, **1** conservative overestimate (avatar upload: expected `terra/moderate`, predicted `sol/moderate`), **0** risky underestimates. Total cost: **$0.0346** across 15 real GPT-5.6-terra calls. [View results](benchmark_results.json). |
| **Injection resistance** | 10 adversarial prompt-injection scenarios across hidden HTML comments, code comments, disguised TODOs, subprocess arguments, config fields, misleading filenames, social-engineering “temporarily disable” requests, and multi-step justification chains. **9/10** caught and blocked. The one miss triggered the fail-safe path (uncertain evaluation → warning, not silent allow) rather than a full block; it is disclosed as a real finding. [View results](injection_benchmark_results.json). |
| **Multi-task session (false-positive check)** | One continuous session across four task types: a trivial README fix (`luna/isolated`, clean), scoped API validation (`terra/moderate`, clean), a broad payment migration (`sol/wide`, 6 files touched, not flagged as `scope_explosion` because wide scope was predicted), and an injected `.env` access attempt (blocked). Total session cost: **$0.01812**. [View results](multi_task_session_results.json). |
| **Workflow-plan benchmark** | Fixed stratified sample of five tasks from the 15-case tier corpus: **4/5** plan tiers matched the hand labels; the MFA task was under-tiered (`sol` expected, `terra` planned). All five plans met the hard planner contract (1-4 bounded phases, each action cap within the 1-40 schema limit, complete phase fields, and a stop prompt). The stricter per-tier action-budget guidance was met by **2/5** plans because several plans deliberately used shorter discovery/verification phases. Total cost: **$0.0306**. [View raw plans and checks](workflow_plan_benchmark_results.json). |
| **Live effort-curve integration** | One fresh mechanical-task prediction plus one generated workflow plan both returned `luna/low` and each carried the source-backed DeepSWE cost curve and Luna caution. Total real cost: **$0.0055** across 2 GPT-5.6-terra calls. [View captured output](effort_curve_live_evidence.json). |
| **Live Python-child guard** | Real GPT-5.6-terra scope/prediction: `luna / low / isolated`; a separately launched Python child wrote allowed `src/auth/login.py`, then an attempted `.env` read received a critical deny verdict before the read. **2** setup calls, **$0.00339**, **0** child evaluation calls. [View raw policy, response IDs, audit, and report](examples/python_child_live_proof.json). |
| **Unified live pipeline** | One continuous Python-child session (`foresight-unified-06e40865`) proved `predict -> allowed write -> scope drift reassessment -> parallel fan-out -> hard action ceiling kill`, with a burn forecast before termination. It made **5** real GPT-5.6-terra calls for **$0.01122**; the raw timeline includes timestamps, all response IDs, every verdict and signal, and the final kill reason. [View unified evidence](examples/unified_live_run_evidence.json). |

Run `python verify.py` to reproduce the zero-cost checks yourself, or `python verify.py --live` to reproduce the paid benchmarks above (~$0.08 total across all three).

One benchmark response includes its raw OpenAI response ID as tamper-evident proof that it was a real API call, not a fabricated result: `response_id: resp_016b9a9aa893e50d006a57492295908192b3c0b54f1eefd7c7`. The recorded response is documented in [`benchmark_results.json`](benchmark_results.json).

## What it does

- **Routes before spend:** predicts the right Luna/Terra/Sol tier, effort level, and blast radius before a single task action is evaluated.
- **Turns scope into enforcement:** derives a restrictive policy from the declared task instead of treating agent access as implicitly broad.
- **Instruments Python agents for real:** `foresight run -- python agent.py` injects process-local interception before child agent code starts, then captures file I/O, subprocesses, and `requests` calls before execution.
- **Flags runaway behavior early:** detects scope explosions, low-progress churn, and repeated paralysis loops while the session is live.
- **Makes the effort tradeoff visible:** attaches a deterministic, illustrative low-to-max cost curve and warns when Luna is being used beyond truly mechanical work.
- **Supports time-sensitive sessions honestly:** `--fast-mode` applies the documented 2.5x quota-burn multiplier and shows the speed-versus-budget warning before work begins.
- **Makes spend auditable:** streams actions, verdicts, predictions, signals, and token-derived costs to the event bus and dashboard.

## Architecture

```text
Task description
      |
      v
foresight/ predictor + session monitor
      |
      v
Python child bootstrap -> sentry/ interception engine + policy + event bus ----> dashboard frontend
```

- **`foresight/`** contains prediction, routing, the CLI, and session monitoring. Keeping these concerns together makes the pre-execution estimate and live escalation decision share the same session context.
- **`sentry/`** is the internal interception engine codename. It captures actions, applies policy, tracks costs, and publishes events. It is decoupled from `foresight/` so the security layer can be reused independently of prediction logic.
- **Python child bootstrap** is injected only for `foresight run -- python ...`. It installs the process-local proxy before agent code, then forwards canonical actions and costs back to the parent monitor over the event bus.
- **`sentry/web/`** is the single, restored Next.js dashboard. It consumes the event model rather than duplicating policy or interception behavior.

Sentry is the internal codename for the interception engine; **Foresight** is the complete product.

### Runtime decision flow

```text
1. Task description
   -> predictor returns tier, effort, blast radius, and risk factors
   -> scope-to-policy returns allowed paths, hosts, and forbidden patterns

2. Agent attempts an action
   -> SentryProxy captures it before execution
   -> Action(type, target, payload, timestamp, session_id)

3. Fast enforcement path
   -> forbidden target: critical block, no model call
   -> out-of-scope file or host: warning event
   -> identical action repeated six times in 60 seconds: paralysis-loop block

4. Ambiguous in-scope action
   -> GPT-5.6-terra returns a structured Verdict
   -> allow, warn, or block before the original operation proceeds

5. Session-level monitoring
   -> compare files, diffs, and spend against the original prediction
   -> scope/churn signal triggers a contextual reassessment
   -> kill only if the reassessment confirms a wide or Sol-level runaway
```

### Layer responsibilities

| Layer | Main modules | Responsibility | Why it is separate |
| --- | --- | --- | --- |
| Prediction | `foresight/predictor.py` | Produces tier, effort, blast radius, rationale, and risk factors from the task description. | Establishes an expected cost and scope baseline before any agent action exists. |
| Session control | `foresight/session_monitor.py` | Watches file count, repeated diffs, and cost ceilings; emits signals and reassesses drift. | Session-level behavior cannot be decided from one action alone. |
| Policy | `sentry/policy/engine.py` | Converts scope to `Policy`, makes local decisions first, and uses structured model review only for ambiguity. | Keeps common security decisions deterministic, cheap, and auditable. |
| Enforcement | `sentry/proxy/interceptor.py` | Temporarily intercepts file, subprocess, delete, and `requests` calls in the current Python process. | Places the decision immediately before the original operation can execute. |
| Telemetry | `sentry/eventbus/server.py` | Broadcasts actions, costs, predictions, and monitor signals over WebSocket; exports JSON audit logs. | Decouples producers from the dashboard and leaves a machine-readable record. |
| Presentation | `sentry/web/` | Renders live connection state, action feed, policy context, cost, and kill state. | Lets a reviewer understand a session without reading Python logs. |

### The common event contract

Every intercepted operation becomes the same `Action` record:

```text
Action
  type: file_read | file_write | file_delete | subprocess | network
  target: path, command, or host:port
  payload: mode, unified diff, cwd/env summary, or request-size summary
  timestamp
  agent_session_id
```

Policy evaluation returns a `Verdict` with `severity`, `reason`, and `allowed`. This normalized pair is what the proxy enforces, the monitor observes, the event bus broadcasts, and the dashboard displays. Model use is represented separately as `CostEvent(model, input_tokens, output_tokens, cost_usd, timestamp)`.

### Four failure modes, four distinct mechanisms

Foresight deliberately gives each failure mode a distinct trigger, reason, event name, and escalation path instead of treating all anomalous activity as one generic warning:

1. **Forbidden-pattern interception** is deterministic and zero-latency. `SentryProxy` blocks an action such as `.env` access before it executes, emits the policy `Verdict`, and terminates only on a critical denied verdict. This is security enforcement, not prediction.
2. **Scope explosion** is predictive and contextual. `SessionMonitor` compares touched files with the predicted blast radius, emits `scope_explosion`, and re-runs complexity assessment with session context before committing to termination. A wide predicted task is not falsely killed for having a wide footprint.
3. **Hard action ceiling** is a deterministic workflow cap. The predictor maps `isolated` to 15 actions, `moderate` to 40, and `wide` to 100. Reaching the cap emits `hard_ceiling_reached` with `hard_action_ceiling_reached` as its final reason and terminates immediately, even if the agent believes it is nearly done.
4. **Parallel fan-out** detects a different failure shape from repeated loops: five distinct file or subprocess actions inside three seconds. It emits `parallel_explosion` and triggers soft reassessment, allowing deliberate, paced parallel work to remain unblocked.

**Burn-rate forecasting** is separately broadcast as `burn_forecast`, but it is intentionally informational rather than a kill switch. After enough cost events, it projects minutes remaining at the current cost/minute rate so a user can intervene before a budget ceiling is reached. This keeps forecast visibility separate from hard enforcement.

The independent `paralysis_loop` check remains in `SentryProxy`: a rolling 60-second window blocks the sixth identical action after five prior repeats before any model evaluation. It is distinct from parallel fan-out because one catches repeated identical work while the other catches a burst of different work.

This distinction is why the demo can truthfully show two independent outcomes: an immediate `.env` block from deterministic policy enforcement, and an `escalated_for_review` result after predictive scope drift is reassessed.

### Cost-aware model use

`scope_to_policy()` and `predict_complexity()` use structured output. `evaluate_action()` uses GPT-5.6-terra only when a local rule cannot resolve an in-scope but suspicious action. Each response has an explicit output cap, a stable prompt prefix for caching, and usage recorded through `CostTracker`.

Predicted effort is calibrated separately from tier: mechanical isolated work routes to `low`, normal scoped coding work defaults to `medium`, and `high` is reserved for genuinely high-stakes or architectural work. Every prediction and workflow plan carries a deterministic, display-ready cost curve from the [DeepSWE v1.1 leaderboard](https://deepswe.datacurve.ai/): observed GPT-5.6 Sol average cost per task was `low $1.07 (1.00x)`, `medium $1.86 (1.74x)`, `high $3.47 (3.24x)`, `xhigh $4.70 (4.39x)`, and `max $8.39 (7.84x)` relative to low. These are benchmark observations for one model/harness, not API pricing or Foresight-measured spend. The CLI and dashboard mark `xhigh` and `max` as steep-cost territory with marginal expected quality gain; no extra model call is required.

When a task is routed to `luna`, Foresight also adds a deterministic caution: Luna is for high-volume data handling, simple classification, or truly mechanical edits. Anything requiring real coding logic or judgment should be checked carefully because Terra or Sol is the practical floor for most coding work.

The architecture therefore treats model judgement as a metered dependency rather than an invisible background call.

### Current prototype boundaries

The repository is intentionally a focused vertical slice, not a claim of production multi-tenant infrastructure:

- Interception is process-scoped Python monkey-patching, not an operating-system sandbox.
- Supported automatic interception is currently Python-child scoped: use `foresight run -- python agent.py`. Node, shell-native, and other agent runtimes are intentionally not represented as supported today.
- Audit events are held in memory and persisted through explicit JSON export rather than a database.
- Budget ceilings are current heuristics, not learned per-user limits.
- A deployed dashboard needs a reachable event-bus endpoint; it cannot receive events from a developer's local process without a connection path.
- The dashboard is intentionally a single deploy target at `sentry/web/`; a production hardening pass would add deployment-specific event-bus connectivity and persistence.

## What we got wrong, and what it taught us

We originally wrote the demo around the task “Refactor the login flow to add MFA support,” assuming it would produce a narrow `terra / isolated` prediction so the scripted scope-explosion sequence could exercise escalation. When we ran it against the real GPT-5.6-terra model, it classified MFA and authentication work as security-sensitive and returned `sol / high / wide`. That was the right call: MFA changes can affect identity, recovery, and session boundaries. Our test assumption, not the prediction, was wrong. The problem was that a wide blast radius has no file-count ceiling to exceed, so the demo narrative no longer matched the model’s real risk assessment. We redesigned the task as a genuinely narrow isolated variable rename, which keeps the expected route and scripted drift consistent. We also added a second, prediction-independent catch: the forbidden-pattern block on `.env` access. It works regardless of the predicted tier, so the demo now proves both predictive scope monitoring and deterministic interception without guessing a model outcome. This is the same principle the system is built to detect in agents: the gap between what we assumed a task’s scope was and what it actually turned out to be.

## Why not just use a bigger model for everything?

Sol costs five times Luna’s input and output price, while many coding tasks are mechanical enough to complete safely with a smaller tier. Defaulting to the frontier tier therefore creates exactly the avoidable spend Foresight is designed to expose. Prediction is not free: it adds a metered call before work begins. But a conservative prediction only adds an escalation round-trip, because Foresight reassesses live drift instead of trusting the first estimate blindly.

## Quickstart

Install the Python runtime and CLI:

```bash
pip install -r requirements.txt
# or, for the `foresight` command:
pip install -e .
```

Create your local environment file from the committed template:

```bash
copy .env.example .env
```

Set `OPENAI_API_KEY` in `.env`, then run the end-to-end demo:

```bash
python -m foresight.demo_scenario
```

For a complete local replay with **zero OpenAI API calls** (using recorded GPT-5.6 policy and prediction responses through a real Python child), run:

```bash
foresight demo --replay
```

The live path above remains the primary demonstration. Replay is the accessible fallback for judges who want to exercise the real interception and monitoring flow without an API key. For a zero-setup visual tour of the dashboard UI, visit [the deployed dashboard](https://sentry-lime.vercel.app); it is distinct from replay mode, which runs the local Python enforcement logic.

CLI workflow:

```bash
# On a new directory, `foresight` launches an interactive first-run setup.
foresight

# Or explicitly create a plan, then monitor a task.
foresight init
foresight run -- python your_agent_task.py

# Zero-key, live-interception replay for judges.
foresight demo --replay

# Use only when latency matters more than budget.
foresight run --fast-mode -- python your_agent_task.py

# CI/headless mode: suppress terminal UI.
foresight run --no-dashboard -- python your_agent_task.py

# Optional richer local browser view.
foresight run --web -- python your_agent_task.py

foresight status
foresight export
```

Run the dashboard locally:

```bash
cd sentry/web
npm install
npm run dev
```

Open `http://localhost:3000/dashboard`. The event bus listens on `ws://localhost:8765` by default.

### Dashboard connection status

`foresight run` starts with a zero-dependency Rich terminal dashboard. Use `--web` to additionally open `http://localhost:3000/dashboard`, or `--no-dashboard` for headless environments.

The public Vercel dashboard is a visual deployment, not a hosted multi-tenant event-bus service. Its root URL rejects WebSocket upgrades, so `foresight run --remote` currently prints a clear notice and falls back to local mode rather than pretending remote streaming works. To see a live local session today, run the Next.js dashboard above alongside the CLI. A secured hosted ingestion service is required before remote mode can become real.

## Cost tracking and prompt caching

Every OpenAI response becomes a `CostEvent`, calculated from input/output token usage and streamed to the event bus. Calls use a stable system/context prefix with only task-specific content appended, improving prompt-cache reuse. Clear forbidden or out-of-scope actions are decided locally first, avoiding unnecessary model calls.

**Every prediction and policy call is cost-tracked in real time and streamed to the dashboard -- Foresight audits its own spend, not just the agent's.**

Fast mode applies a `2.5x` multiplier to recorded cost events to make its quota-burn tradeoff visible. Foresight displays: "Fast mode increases quota burn by ~2.5x for ~1.5x speed. Recommended only when latency matters more than budget."

## Codex vs Antigravity build split

- **Codex -- interception, reasoning, and enforcement.** `sentry/proxy/interceptor.py` implements pre-execution monkey-patching for subprocesses, file reads/writes/deletes, and HTTP requests; unified-diff capture; tracked-process termination; and the local repeated-action paralysis guard. `sentry/policy/engine.py` handles structured scope-to-policy conversion, local forbidden/scope checks, contextual evaluation, prompt-cache discipline, and token-based cost events. `foresight/predictor.py` provides structured tier/effort/blast-radius routing, configurable tier bias, deterministic action ceilings, and typed phase-bounded workflow plans. `foresight/session_monitor.py` implements scope-explosion, low-progress churn, hard action/phase caps, parallel fan-out detection, burn-rate forecasting, phase-output drift checks, reassessment, and escalation.
- **Codex -- product plumbing and proof.** Codex also built the WebSocket event surface in `sentry/eventbus/server.py`, the interactive Rich CLI and TOML configuration system in `foresight/cli.py` and `foresight/config.py`, the replay/live demos, tier/injection/multi-task/workflow benchmarks, PyPI packaging metadata, and publishing documentation. These components carry predictions, signals, audit events, and costs end-to-end; they are not presentation-only stubs.
- **Antigravity -- presentation and delivery.** Antigravity handled the visual polish in `sentry/web/`, dashboard layout/connection fixes, deployment configuration, and documentation presentation cleanup. It consumes the event contract but does not make policy decisions, predict tiers, calculate enforcement signals, or terminate sessions.

This split is intentional and documented per hackathon submission requirements. Codex built 100% of the reasoning-critical and safety-critical behavior: prediction, policy evaluation, workflow bounding, cost/burn accounting, anomaly detection, and escalation/termination.

## Roadmap

- Multi-agent swarm monitoring with a shared session risk view.
- Historical session learning to improve tier prediction over time.
- Monitoring support for Claude Code and other agent runtimes beyond Codex.
- Node, shell-native, and other non-Python agent-runtime interception.
- Team-level policy profiles and budget guardrails.

## License and attribution

MIT License. Built for Build Week, July 2026.
