Metadata-Version: 2.4
Name: reflight
Version: 0.1.0
Summary: Flight recorder for AI agents: record every run, replay it deterministically, turn failures into regression tests.
Project-URL: Homepage, https://github.com/pauti04/reflight
Project-URL: Demo, https://pauti04.github.io/reflight-demo/
Project-URL: Documentation, https://github.com/pauti04/reflight/tree/main/docs
Project-URL: Issues, https://github.com/pauti04/reflight/issues
Author: Parth Auti
License-Expression: Apache-2.0
Keywords: ai-agents,evals,llm,observability,regression-testing,replay,testing
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Debuggers
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.12
Requires-Dist: anthropic>=0.40
Requires-Dist: fastapi>=0.115
Requires-Dist: pyyaml>=6
Requires-Dist: uvicorn>=0.30
Provides-Extra: otel
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.27; extra == 'otel'
Requires-Dist: opentelemetry-sdk>=1.27; extra == 'otel'
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.2; extra == 'postgres'
Description-Content-Type: text/markdown

# ⏺ Reflight

**Flight recorder for AI agents: record every run, replay it deterministically,
turn failures into regression tests.**

Reflight is *reliability infrastructure*, not a detector or an eval: it makes
agent behavior reproducible, testable, and governable. Recordings use an
[open, documented format](docs/format.md) that anything can consume —
detectors, evals, and observability tools run *on top* of it.

[**Live demo**](https://pauti04.github.io/reflight-demo/) · real recorded runs, replayable in your browser, no install.

**New: a real one.** Fifteen live runs of a gpt-4o-mini scheduling agent
all passed every tool-level check — and all booked the meeting on a Sunday.
The recordings, the judge whose catch rate swung from 5/5 to 1/5 on
identical failures, and the 15-line assertion that caught every one:
[the case study](docs/case-study.md).

*A real recording, replaying: a support agent sends the refund amount as a string, retries the same broken call, and gets caught —*

![The flight recorder replaying a recorded failure: the agent retries a malformed refund call until the classifier labels it](https://raw.githubusercontent.com/pauti04/reflight/main/docs/assets/fdr-panel.gif)

![A recorded agent run: the loop caught, labeled, and frozen on the timeline](https://raw.githubusercontent.com/pauti04/reflight/main/docs/assets/hero-run.png)

*Two runs of the same task, diffed — the passing run sent `query`, the failing run sent `q`. First divergence highlighted:*

![Run diff: first divergence highlighted at the exact event](https://raw.githubusercontent.com/pauti04/reflight/main/docs/assets/hero-diff.png)

Agents are programs whose most important steps are non-deterministic and
external. When one fails, the failure evaporates — re-running gives you a
*different* run. Reflight makes agent failures **reproducible**, and builds
the whole reliability loop on top:

> agent fails → the recorded run is already a reproducible test case →
> `reflight promote <run_id>` adds it to your suite → CI replays it forever,
> so that failure can never silently come back.

## What you get

| | |
|---|---|
| 🎥 **Record** | Every LLM call, tool call, token and dollar — 3 added lines, sync or asyncio (`record_async`) |
| ⏪ **Replay** | Re-run any recording byte-identically: offline, ~7ms, $0.00 |
| 🔍 **Debug** | Timeline UI with event inspector; `--step` CLI debugger; run-diff with first-divergence highlighting |
| 🏷 **Classify** | Rule-based failure labels (loop, wrong_tool_args, cascade, crash, runaway) + LLM judge with ensemble voting (`--votes 3`) |
| 🛂 **Flight check** | `flight_check=True` flags network I/O that bypassed the session — the run is marked `unrecorded_io` instead of silently un-replayable |
| 🔱 **Fork** | Replay to step N, go live after — test a fix mid-run |
| ✅ **Promote** | One command: recorded failure → editable YAML regression test |
| 📊 **Harness** | N-run consistency scoring, baselines, CI gate that blocks reliability regressions |
| ⛔ **Govern** | Hard cost/token budgets, loop circuit breaker, tool-call cache, cost dashboard with anomaly flags |
| 📡 **Export** | `reflight otel <run_id>` ships any run to your OTLP collector as GenAI-convention spans — works *with* Langfuse/Datadog/Jaeger, not against them |

## Quickstart

```bash
git clone <repo> && cd reflight
uv sync                      # installs the SDK + CLI (Python 3.12+)

# record two demo runs (scripted model — no API key needed)
uv run python examples/research_agent/main.py record \
    "What is the population of Tokyo, and what is that number divided by 2?" \
    --offline --run-id demo-research
uv run python examples/research_agent/main.py record \
    "What is 12 divided by 0? Use the calculator." --offline --run-id demo-failure

# replay the failure — network off, $0.00, byte-identical
uv run python examples/research_agent/main.py replay demo-failure --step

# query them
uv run reflight import runs
uv run reflight runs
uv run reflight show demo-failure
```

### The timeline UI

```bash
uv run reflight serve            # API on :8724
cd ui && npm install && npm run dev   # UI on :3000
```

Runs list → click a run → color-coded timeline → event inspector. Findings
banner on failed runs; pick two runs to diff; `/costs` for the money view.

To build the zero-backend static demo site (what the hosted demo runs):

```bash
uv run reflight export-static        # db → ui/public/demo/*.json
cd ui && STATIC_EXPORT=1 NEXT_PUBLIC_STATIC_DEMO=1 npm run build   # → ui/out/
```

### Instrument your own agent — 3 lines

```python
import reflight

session = reflight.record("runs/my-run", task=task, db_path="runs/reflight.db")  # 1
client = session.wrap(anthropic.Anthropic())                                          # 2
my_tool = session.tool(my_tool)                        # 3 — or @session.tool

# ... your agent code runs unchanged ...
session.end(final_text=answer)
```

OpenAI-compatible clients: `client = session.wrap_openai(OpenAI())`.
MCP tool calls: `mcp = session.wrap_mcp(mcp_client_session)` — recorded and
replayed on the same timeline as everything else (async).

Recordings contain no API keys by construction (arguments are recorded, not
HTTP headers). For secrets that flow through *tool data*, pass
`redact=reflight.redact_patterns(r"sk-\w+")` — masked before disk, hash
fields preserved so the recording stays replayable.

**LangGraph / LangChain** agents instrument without code changes:

```python
from reflight.adapters.langchain import instrument

model, tools = instrument(session, ChatOpenAI(model="gpt-4o-mini"), tools)
agent = create_react_agent(model, tools)   # unchanged LangGraph code
```

Validated against the real thing: [examples/langgraph_live.py](examples/langgraph_live.py)
records a live LangGraph run and replays it byte-identically offline.
(Sync paths; coroutine-only tools rejected loudly.)

Replay it later — same agent code, session swapped:

```python
session = reflight.replay("runs/my-run")     # no network, no key, no cost
client = session.wrap()
```

### Every failure becomes a regression test

This is how you build a **golden dataset from real failures, automatically** —
the thing every 2026 eval-methodology guide says reliable agent teams need,
assembled one `promote` at a time instead of hand-curated.

```bash
uv run reflight promote my-failed-run       # → agent_tests/my-failed-run.yaml
```

Edit the assertions to state what SHOULD happen — then they're just pytest
tests. Point pytest at your agent once:

```ini
# pytest.ini
[pytest]
reflight_agent = my_pkg.agent:run_agent            # agent(session, task)
reflight_tools_factory = my_pkg.agent:make_tools   # optional
reflight_client_factory = my_pkg.agent:make_client # optional: enables live re-verify
```

and every `agent_tests/*.yaml` collects and runs in your normal `pytest`
invocation. Replay-first economics: passing tests cost $0.00; replay failures
are re-verified live; code changes trigger a live re-run. Programmatic
alternative: `reflight.testing.run_suite`. See the full loop in
[examples/flaky_agent/regression_demo.py](examples/flaky_agent/regression_demo.py)
and the CI gate in [examples/flaky_agent/ci_gate.py](examples/flaky_agent/ci_gate.py).

### The governor

```python
from reflight import Governor

session = reflight.record(..., governor=Governor(
    max_cost_usd=0.50,       # hard kill at the cap — reason recorded in the run
    loop_breaker=3,          # N identical consecutive tool calls allowed
    cache_tool_calls=True,   # serve repeats from cache (still recorded)
))
```

## Demos (all offline, no API key)

```bash
uv run python examples/quickstart/agent.py record && uv run python examples/quickstart/agent.py replay
uv run python examples/flaky_agent/fleet.py 10          # classifier labels a flaky fleet
uv run python examples/flaky_agent/fix_demo.py          # fork a failed run mid-flight
uv run python examples/flaky_agent/regression_demo.py   # fail → promote → fix → pass
uv run python examples/flaky_agent/governor_demo.py     # runaway killed at $0.50
uv run python examples/flaky_agent/ci_gate.py           # CI reliability gate (add --degrade)
```

## Where it sits in your stack

The question everyone asks: "how is this different from what I already use?"

| You already use… | It does | Reflight adds |
|---|---|---|
| **LangSmith / Langfuse / Braintrust** | Hosted observability: traces, dashboards, datasets | **Deterministic replay** — their traces describe a run; a Reflight recording can *re-execute* it. Local-first, no SaaS. Composes with them via `reflight otel`. |
| **pytest-vcr / vcrpy** | Records HTTP for API tests | The same idea *lifted to the agent layer*: tool calls, parallel execution, streaming, divergence detection, failure classification — plus `promote`, which VCR never had. |
| **Eval harnesses** (capability benchmarks) | "Can the model do X?" | "Does *my agent* still do X, every time, this week?" — consistency over capability, wired into CI as a merge gate. |
| **Detectors / guardrails** (hallucination checkers, semantic judges) | Judge content in the moment | The substrate they should run on: a detector consuming [recordings](docs/format.md) gets reproducible inputs and can write findings back. Reflight's own judge is one small example. |
| **Docker cagent** | VCR cassettes for agents built in *its* runtime | Reflight instruments **your existing Python agent** — any loop, any framework — and adds everything downstream of the cassette: classification, promote→pytest, fingerprinting, fork, governor. |
| **Laminar** | Hosted replay-from-a-step in their UI | The same debugging move, local-first: `reflight.fork(run, at_seq=N)` — plus the recording is a file you own, not a SaaS row. |
| **MCP recorders** (mcp-recorder, Agent VCR) | Record/replay one MCP server's wire protocol | `session.wrap_mcp(...)` records MCP tool calls *inside the whole agent recording* — one timeline for LLM calls, local tools, and MCP together. |

Short version: everything else observes or evaluates. Reflight makes runs
**reproducible** — and everything downstream of reproducibility (regression
tests, CI gates, recurrence tracking) is what the others can't offer.

## How replay works (and its honest limits)

Recording captures every request/response pair in an append-only
`events.jsonl`. Replay re-executes **your agent code** with all external I/O
served from the recording, verifying at each step that the code is making the
*same* requests it made before — a changed prompt or tool raises
`ReplayDivergence` instead of lying. Replay is deterministic for the recorded
path; it is **not** time travel for arbitrary changes — that's what fork mode
and live re-verification are for. Streaming agents are supported (the
`messages.stream()` helper pattern replays chunk-identically), and so are
**parallel tool calls** — replay matches by `tool_use_id`, so any completion
order replays. Agent code that consults the clock, PRNG, or `uuid.uuid4()`
between calls is covered too: wrap the loop in `with session.pin():` and
those draws are recorded and served back on replay, so timestamped requests
and generated ids replay exactly. The full honest map of what replay can and
can't see — including what the pin does *not* cover — is
[docs/limits.md](docs/limits.md); smaller open items are tracked in
[NOTES.md](NOTES.md).

Verified against a real API: [examples/live_api_check.py](examples/live_api_check.py)
records two dependent live calls and replays them byte-identically with the
network blocked. Judge accuracy vs seeded ground truth: 12/12
([examples/flaky_agent/judge_accuracy.py](examples/flaky_agent/judge_accuracy.py)).

## Layout

```
sdk/reflight/    the library: recorder, replayer, fork, classify, judge,
                   testing (promote/runner), executor, reliability, governor,
                   store (SQLite), server (FastAPI), cli
ui/                Next.js timeline UI
examples/          research agent, quickstart, flaky fleet + demos
tests/             the whole story as pytest (100+ tests)
docs/              quickstart, concepts, blog drafts
```

## Development

```bash
uv sync && uv run pytest       # tests
uv run ruff check .            # lint
```

Plans live in [PROJECT_PLAN.md](PROJECT_PLAN.md), [GAMEPLAN.md](GAMEPLAN.md),
[SPRINTS.md](SPRINTS.md). Apache-2.0.
