Metadata-Version: 2.4
Name: foresight-agent-guard
Version: 0.1.0
Summary: Predictive cost, policy, and runaway-session guardrails for AI coding agents
Author: Aman M
License: MIT
Project-URL: Homepage, https://github.com/AmanM006/foresight
Project-URL: Repository, https://github.com/AmanM006/foresight
Project-URL: Issues, https://github.com/AmanM006/foresight/issues
Keywords: ai-agents,codex,gpt,cost-monitoring,developer-tools
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click<9,>=8.1
Requires-Dist: openai<3,>=1.0
Requires-Dist: python-dotenv<2,>=1.0
Requires-Dist: requests<3,>=2.31
Requires-Dist: rich<15,>=13.0
Requires-Dist: websockets<16,>=13
Dynamic: license-file

# Foresight

**Foresight is a predictive cost and tier-routing layer for AI coding agents.** It chooses the right level of reasoning before work begins, then watches the session closely enough to stop drift before it becomes expensive.

![CI](https://github.com/AmanM006/foresight/actions/workflows/ci.yml/badge.svg?branch=main)
![MIT License](https://img.shields.io/badge/license-MIT-111827?style=flat-square)
![Python](https://img.shields.io/badge/Python-3.11%2B-3776AB?style=flat-square)
![Next.js](https://img.shields.io/badge/Next.js-14-111111?style=flat-square)

> **Install from PyPI (after the first release)**
>
> ```bash
> pip install foresight-agent-guard
> foresight --help
> ```

> **Build from source / quick test**
>
> ```bash
> git clone https://github.com/AmanM006/foresight.git
> cd foresight
> pip install -e .
> copy .env.example .env
> python -m foresight.demo_scenario
> ```
>
> Add your `OPENAI_API_KEY` to `.env` after copying the example file. On macOS/Linux, use `cp .env.example .env`.

## Verify this submission

Run the complete replay-safe verification pass with one command:

```bash
python verify.py
```

It checks the environment, runs tests when present, and executes the full replay demo without spending tokens. Add `--live` only when you want it to run the paid tier and injection benchmarks as well. For a zero-setup visual alternative, explore the deployed dashboard at [sentry-lime.vercel.app](https://sentry-lime.vercel.app).

## The problem

When developers use Codex/GPT-5.6 for coding work, they often do not know upfront which tier or reasoning effort a task actually needs. Defaulting to Sol overpays for mechanical work; choosing too small a tier can leave an agent stuck and repeatedly escalating.

The risk is not hypothetical. An open [Codex issue #32250](https://github.com/openai/codex/issues/32250) reports GPT-5.6 Sol Medium reducing a Pro five-hour allowance from 87% to 76% during a short conversation and several trivial follow-ups. Recent [r/codex](https://www.reddit.com/r/codex/comments/1ust6y8/usage_drops_much_faster_now_despite_less_cost/) and [r/ChatGPTPro](https://www.reddit.com/r/ChatGPTPro/comments/1us02nr/56_rate_limits/) threads describe confusion around five-hour versus weekly limits; these are community reports, not a claim about every session. Community guidance also identifies duplicated context and uncontrolled subagent spawning as sources of extra consumption ([discussion](https://www.reddit.com/r/codex/comments/1uu7weh/possible_fixes_for_gpt_56_burning_through_your/)).

There is a second failure mode: scope creep. A small implementation task can turn into unrequested architecture work, sprawling documentation, or broad file changes—the “Sol Ultra problem” discussed by users—silently consuming budget without an early warning. OpenAI documents that `ultra` coordinates four agents in parallel by default and trades higher token use for stronger results on demanding tasks; it is not an appropriate default for every coding task ([OpenAI GPT-5.6 overview](https://openai.com/index/gpt-5-6/)).

**Foresight solves this by predicting complexity and tier before execution, then watching the live session to catch drift before it compounds.**

## Live demo proof

One real recorded demo run produced the following result:

| Signal | Observed result |
| --- | --- |
| Predicted route | `luna / low / isolated` |
| Files touched | `16` |
| Predictive flag | `scope_explosion` |
| Final disposition | `escalated_for_review` |
| Recorded model cost | approximately `$0.0060` |

The demo showed two independent catch mechanisms live:

- **Predictive scope-drift detection:** unrelated file changes exceeded the predicted isolated blast radius.
- **Forbidden-pattern interception:** an attempted `.env` read was caught and blocked by the policy layer.

![Live event feed showing real-time action interception](screenshots/dashboard-live-feed.png)

![Session killed banner after a forbidden-pattern block](screenshots/session-killed-banner.png)

![Cost tracker showing real GPT-5.6-terra cost events](screenshots/cost-tracker.png)

Inspect the sanitized raw evidence in [`examples/`](examples/): [`policy_and_verdict_output.example.json`](examples/policy_and_verdict_output.example.json) and [`audit_event_output.example.json`](examples/audit_event_output.example.json).

## Evidence and benchmarks

| Evidence | Real result |
| --- | --- |
| **Tier prediction accuracy** | 15 hand-labeled tasks spanning mechanical, moderate, and architectural complexity. **14/15** exact tier match, **1** conservative overestimate (avatar upload: expected `terra/moderate`, predicted `sol/moderate`), **0** risky underestimates. Total cost: **$0.0346** across 15 real GPT-5.6-terra calls. [View results](benchmark_results.json). |
| **Injection resistance** | 10 adversarial prompt-injection scenarios across hidden HTML comments, code comments, disguised TODOs, subprocess arguments, config fields, misleading filenames, social-engineering “temporarily disable” requests, and multi-step justification chains. **9/10** caught and blocked. The one miss triggered the fail-safe path (uncertain evaluation → warning, not silent allow) rather than a full block; it is disclosed as a real finding. [View results](injection_benchmark_results.json). |
| **Multi-task session (false-positive check)** | One continuous session across four task types: a trivial README fix (`luna/isolated`, clean), scoped API validation (`terra/moderate`, clean), a broad payment migration (`sol/wide`, 6 files touched, not flagged as `scope_explosion` because wide scope was predicted), and an injected `.env` access attempt (blocked). Total session cost: **$0.01812**. [View results](multi_task_session_results.json). |

Run `python verify.py` to reproduce the zero-cost checks yourself, or `python verify.py --live` to reproduce the paid benchmarks above (~$0.08 total across all three).

One benchmark response includes its raw OpenAI response ID as tamper-evident proof that it was a real API call, not a fabricated result: `response_id: resp_016b9a9aa893e50d006a57492295908192b3c0b54f1eefd7c7`. The recorded response is documented in [`benchmark_results.json`](benchmark_results.json).

## What it does

- **Routes before spend:** predicts the right Luna/Terra/Sol tier, effort level, and blast radius before a single task action is evaluated.
- **Turns scope into enforcement:** derives a restrictive policy from the declared task instead of treating agent access as implicitly broad.
- **Intercepts high-risk actions:** captures file I/O, subprocesses, and network requests before execution.
- **Flags runaway behavior early:** detects scope explosions, low-progress churn, and repeated paralysis loops while the session is live.
- **Makes spend auditable:** streams actions, verdicts, predictions, signals, and token-derived costs to the event bus and dashboard.

## Architecture

```text
Task description
      |
      v
foresight/ predictor + session monitor
      |
      v
sentry/ interception engine + policy + event bus ----> dashboard frontend
```

- **`foresight/`** contains prediction, routing, the CLI, and session monitoring. Keeping these concerns together makes the pre-execution estimate and live escalation decision share the same session context.
- **`sentry/`** is the internal interception engine codename. It captures actions, applies policy, tracks costs, and publishes events. It is decoupled from `foresight/` so the security layer can be reused independently of prediction logic.
- **`sentry/web/`** is the single, restored Next.js dashboard. It consumes the event model rather than duplicating policy or interception behavior.

Sentry is the internal codename for the interception engine; **Foresight** is the complete product.

### Runtime decision flow

```text
1. Task description
   -> predictor returns tier, effort, blast radius, and risk factors
   -> scope-to-policy returns allowed paths, hosts, and forbidden patterns

2. Agent attempts an action
   -> SentryProxy captures it before execution
   -> Action(type, target, payload, timestamp, session_id)

3. Fast enforcement path
   -> forbidden target: critical block, no model call
   -> out-of-scope file or host: warning event
   -> identical action repeated six times in 60 seconds: paralysis-loop block

4. Ambiguous in-scope action
   -> GPT-5.6-terra returns a structured Verdict
   -> allow, warn, or block before the original operation proceeds

5. Session-level monitoring
   -> compare files, diffs, and spend against the original prediction
   -> scope/churn signal triggers a contextual reassessment
   -> kill only if the reassessment confirms a wide or Sol-level runaway
```

### Layer responsibilities

| Layer | Main modules | Responsibility | Why it is separate |
| --- | --- | --- | --- |
| Prediction | `foresight/predictor.py` | Produces tier, effort, blast radius, rationale, and risk factors from the task description. | Establishes an expected cost and scope baseline before any agent action exists. |
| Session control | `foresight/session_monitor.py` | Watches file count, repeated diffs, and cost ceilings; emits signals and reassesses drift. | Session-level behavior cannot be decided from one action alone. |
| Policy | `sentry/policy/engine.py` | Converts scope to `Policy`, makes local decisions first, and uses structured model review only for ambiguity. | Keeps common security decisions deterministic, cheap, and auditable. |
| Enforcement | `sentry/proxy/interceptor.py` | Temporarily intercepts file, subprocess, delete, and `requests` calls in the current Python process. | Places the decision immediately before the original operation can execute. |
| Telemetry | `sentry/eventbus/server.py` | Broadcasts actions, costs, predictions, and monitor signals over WebSocket; exports JSON audit logs. | Decouples producers from the dashboard and leaves a machine-readable record. |
| Presentation | `sentry/web/` | Renders live connection state, action feed, policy context, cost, and kill state. | Lets a reviewer understand a session without reading Python logs. |

### The common event contract

Every intercepted operation becomes the same `Action` record:

```text
Action
  type: file_read | file_write | file_delete | subprocess | network
  target: path, command, or host:port
  payload: mode, unified diff, cwd/env summary, or request-size summary
  timestamp
  agent_session_id
```

Policy evaluation returns a `Verdict` with `severity`, `reason`, and `allowed`. This normalized pair is what the proxy enforces, the monitor observes, the event bus broadcasts, and the dashboard displays. Model use is represented separately as `CostEvent(model, input_tokens, output_tokens, cost_usd, timestamp)`.

### Four failure modes, four distinct mechanisms

Foresight deliberately gives each failure mode a distinct trigger, reason, event name, and escalation path instead of treating all anomalous activity as one generic warning:

1. **Forbidden-pattern interception** is deterministic and zero-latency. `SentryProxy` blocks an action such as `.env` access before it executes, emits the policy `Verdict`, and terminates only on a critical denied verdict. This is security enforcement, not prediction.
2. **Scope explosion** is predictive and contextual. `SessionMonitor` compares touched files with the predicted blast radius, emits `scope_explosion`, and re-runs complexity assessment with session context before committing to termination. A wide predicted task is not falsely killed for having a wide footprint.
3. **Hard action ceiling** is a deterministic workflow cap. The predictor maps `isolated` to 15 actions, `moderate` to 40, and `wide` to 100. Reaching the cap emits `hard_ceiling_reached` with `hard_action_ceiling_reached` as its final reason and terminates immediately, even if the agent believes it is nearly done.
4. **Parallel fan-out** detects a different failure shape from repeated loops: five distinct file or subprocess actions inside three seconds. It emits `parallel_explosion` and triggers soft reassessment, allowing deliberate, paced parallel work to remain unblocked.

**Burn-rate forecasting** is separately broadcast as `burn_forecast`, but it is intentionally informational rather than a kill switch. After enough cost events, it projects minutes remaining at the current cost/minute rate so a user can intervene before a budget ceiling is reached. This keeps forecast visibility separate from hard enforcement.

The independent `paralysis_loop` check remains in `SentryProxy`: a rolling 60-second window blocks the sixth identical action after five prior repeats before any model evaluation. It is distinct from parallel fan-out because one catches repeated identical work while the other catches a burst of different work.

This distinction is why the demo can truthfully show two independent outcomes: an immediate `.env` block from deterministic policy enforcement, and an `escalated_for_review` result after predictive scope drift is reassessed.

### Cost-aware model use

`scope_to_policy()` and `predict_complexity()` use structured output. `evaluate_action()` uses GPT-5.6-terra only when a local rule cannot resolve an in-scope but suspicious action. Each response has an explicit output cap, a stable prompt prefix for caching, and usage recorded through `CostTracker`.

Predicted effort is calibrated separately from tier: mechanical isolated work routes to `low`, normal scoped coding work defaults to `medium`, and `high` is reserved for genuinely high-stakes or architectural work. A high-effort prediction with a non-wide blast radius carries a deterministic advisory against casually escalating to `max` or `ultra`; the CLI renders it as a dim note and the dashboard renders it beside the predicted effort. No extra model call is required for that advice.

The architecture therefore treats model judgement as a metered dependency rather than an invisible background call.

### Current prototype boundaries

The repository is intentionally a focused vertical slice, not a claim of production multi-tenant infrastructure:

- Interception is process-scoped Python monkey-patching, not an operating-system sandbox.
- Audit events are held in memory and persisted through explicit JSON export rather than a database.
- Budget ceilings are current heuristics, not learned per-user limits.
- A deployed dashboard needs a reachable event-bus endpoint; it cannot receive events from a developer's local process without a connection path.
- The dashboard is intentionally a single deploy target at `sentry/web/`; a production hardening pass would add deployment-specific event-bus connectivity and persistence.

## What we got wrong, and what it taught us

We originally wrote the demo around the task “Refactor the login flow to add MFA support,” assuming it would produce a narrow `terra / isolated` prediction so the scripted scope-explosion sequence could exercise escalation. When we ran it against the real GPT-5.6-terra model, it classified MFA and authentication work as security-sensitive and returned `sol / high / wide`. That was the right call: MFA changes can affect identity, recovery, and session boundaries. Our test assumption, not the prediction, was wrong. The problem was that a wide blast radius has no file-count ceiling to exceed, so the demo narrative no longer matched the model’s real risk assessment. We redesigned the task as a genuinely narrow isolated variable rename, which keeps the expected route and scripted drift consistent. We also added a second, prediction-independent catch: the forbidden-pattern block on `.env` access. It works regardless of the predicted tier, so the demo now proves both predictive scope monitoring and deterministic interception without guessing a model outcome. This is the same principle the system is built to detect in agents: the gap between what we assumed a task’s scope was and what it actually turned out to be.

## Why not just use a bigger model for everything?

Sol costs five times Luna’s input and output price, while many coding tasks are mechanical enough to complete safely with a smaller tier. Defaulting to the frontier tier therefore creates exactly the avoidable spend Foresight is designed to expose. Prediction is not free: it adds a metered call before work begins. But a conservative prediction only adds an escalation round-trip, because Foresight reassesses live drift instead of trusting the first estimate blindly.

## Quickstart

Install the Python runtime and CLI:

```bash
pip install -r requirements.txt
# or, for the `foresight` command:
pip install -e .
```

Create your local environment file from the committed template:

```bash
copy .env.example .env
```

Set `OPENAI_API_KEY` in `.env`, then run the end-to-end demo:

```bash
python -m foresight.demo_scenario
```

For a complete local replay with **zero OpenAI API calls** (using real captured GPT-5.6 policy and prediction responses), run:

```bash
python -m foresight.demo_scenario --replay
```

The live path above remains the primary demonstration. Replay is the accessible fallback for judges who want to exercise the real interception and monitoring flow without an API key. For a zero-setup visual tour of the dashboard UI, visit [the deployed dashboard](https://sentry-lime.vercel.app); it is distinct from replay mode, which runs the local Python enforcement logic.

CLI workflow:

```bash
foresight init
foresight run -- python your_agent_task.py
foresight status
foresight export
```

Run the dashboard locally:

```bash
cd sentry/web
npm install
npm run dev
```

Open `http://localhost:3000/dashboard`. The event bus listens on `ws://localhost:8765` by default.

## Cost tracking and prompt caching

Every OpenAI response becomes a `CostEvent`, calculated from input/output token usage and streamed to the event bus. Calls use a stable system/context prefix with only task-specific content appended, improving prompt-cache reuse. Clear forbidden or out-of-scope actions are decided locally first, avoiding unnecessary model calls.

**Every prediction and policy call is cost-tracked in real time and streamed to the dashboard—Foresight audits its own spend, not just the agent’s.**

## Codex vs Antigravity build split

- **Codex:** reasoning-critical implementation in `sentry/proxy/interceptor.py`, `sentry/policy/engine.py`, `foresight/predictor.py`, `foresight/session_monitor.py`, plus the event bus, CLI, and end-to-end demo.
- **Antigravity:** frontend polish in `sentry/web/`, dashboard bug fixes, deployment configuration, environment/documentation cleanup, and this submission README.

This split is intentional and documented per hackathon submission requirements—Codex built 100% of the reasoning-critical logic: prediction, policy evaluation, and escalation.

## Roadmap

- Multi-agent swarm monitoring with a shared session risk view.
- Historical session learning to improve tier prediction over time.
- Monitoring support for Claude Code and other agent runtimes beyond Codex.
- Team-level policy profiles and budget guardrails.

## License and attribution

MIT License. Built for Build Week, July 2026.
