Metadata-Version: 2.5
Name: llm-run-recorder
Version: 0.3.0
Summary: Flight recorder for LLM agents: trace every call, replay runs deterministically, diff runs to find where behavior diverged.
Project-URL: Homepage, https://github.com/Shravya29M/agentrewind
Author: Shravya Munugala
License: MIT
License-File: LICENSE
Keywords: agents,debugging,llm,observability,replay,tracing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Debuggers
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: fastapi>=0.110; extra == 'dev'
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: psycopg[binary]>=3.1; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Requires-Dist: uvicorn>=0.29; extra == 'dev'
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.1; extra == 'postgres'
Provides-Extra: server
Requires-Dist: fastapi>=0.110; extra == 'server'
Requires-Dist: uvicorn>=0.29; extra == 'server'
Description-Content-Type: text/markdown

# AgentRewind

[![CI](https://github.com/Shravya29M/agentrewind/actions/workflows/ci.yml/badge.svg)](https://github.com/Shravya29M/agentrewind/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/llm-run-recorder)](https://pypi.org/project/llm-run-recorder/)
![Python](https://img.shields.io/pypi/pyversions/llm-run-recorder)
![coverage](https://img.shields.io/badge/coverage-%E2%89%A597%25-brightgreen)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)

**Flight recorder for LLM agents.** Trace every LLM and tool call an agent makes, replay a
run deterministically (no API calls, no cost, no nondeterminism), and diff two runs to find
exactly where their behavior diverged.

![demo: recording two agent runs and diffing them](https://raw.githubusercontent.com/Shravya29M/agentrewind/main/docs/demo.gif)

*"It worked yesterday — why is it different today?"* is the defining debugging problem of
agent development. Tracing tools show you what happened; AgentRewind also lets you **re-execute**
what happened and **compare** runs structurally.

## Install

```bash
pip install llm-run-recorder              # core (stdlib-only)
pip install 'llm-run-recorder[server]'    # + web trace viewer
pip install 'llm-run-recorder[postgres]'  # + PostgreSQL storage backend (psycopg 3)
```

## Trace

```python
import agentrewind as al

@al.traced(kind="tool")
def search(query: str) -> str:
    ...

with al.trace("research-agent"):
    plan = call_llm(...)          # record with al.record_llm_call(...) or a Recorder
    evidence = search(plan.query)
```

Traces (span tree, inputs/outputs, latency, token counts, errors) are stored in a local
SQLite db at `~/.agentrewind/traces.db` — no account, no server required.

### PostgreSQL backend (optional)

SQLite keeps traces on one machine. PostgreSQL gives a team or a fleet one **shared trace
store**: agents, eval workers and CI jobs on different machines all record into the same
database. Any teammate can then `list`, `show`, `diff` or open the viewer on any run, wherever
it was recorded, and query every run with ordinary SQL (for example, error rates or token
usage by model across all of last week's runs). Choose it for sharing and querying, not
speed. On one machine the SQLite default wrote faster in our benchmark.

```bash
pip install 'llm-run-recorder[postgres]'
export AGENTREWIND_DB_URL=postgresql://user:password@host:5432/dbname
```

Everything else stays the same: recording, replay, diff, the CLI and the web viewer. The
schema is created on first use. You can also pass the URL explicitly with
`agentrewind --db postgresql://… list`, `al.configure(db_path="postgresql://…")` or
`agentrewind.open_store("postgresql://…")`. To use a schema other than `public`, append
`?options=-csearch_path%3Dmyschema`. Both backends store the same schema and return the same
results. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md#storage) for the concurrency model
and [docs/EVALUATION.md](docs/EVALUATION.md#concurrent-writers-sqlite-vs-postgresql) for the
concurrent-writer benchmark.

Upgrading from 0.2.x: an existing `traces.db` is migrated in place the first time 0.3 opens
it. The migration is one-way, so 0.2.x cannot write to the file afterwards.

### Sensitive-data controls

For production traffic, opt into recursive persistence-time redaction. It removes common
credential fields (`api_key`, `authorization`, `password`, `secret`, and `token`) and common
token formats from trace metadata, span payloads, and the local replay cache:

```python
import agentrewind as al

al.configure(redaction=al.RedactionPolicy())
```

Redaction intentionally also applies to cached responses, so use it where protecting local
data matters more than replaying the original secret verbatim.

## Replay

Wrap your provider call in a `Recorder`. Responses are cached by a canonical request
fingerprint; in replay mode the whole agent run re-executes deterministically and offline:

```python
from agentrewind.replay import Recorder

llm = Recorder(call_openai, mode="record")   # live run, responses cached
llm = Recorder(call_openai, mode="replay")   # deterministic re-run, zero API calls
llm = Recorder(call_openai, mode="auto")     # replay on hit, record on miss
```

Or instrument an SDK client in place — existing code keeps calling it exactly as before:

```python
import agentrewind as al

client = al.instrument(OpenAI(), mode="auto")       # or Anthropic()
client.chat.completions.create(model="gpt-4o", messages=[...])  # traced + replayable
```

(Prefer explicit wrappers? `agentrewind.providers.OpenAIChat` / `AnthropicMessages` do the
same without monkey-patching.)

Streaming is captured too: `Recorder.call_stream` / `acall_stream` pass chunks through
live while recording them, and replay re-streams the identical chunks offline.

Requests containing volatile fields (timestamps, request ids) can be normalized before
fingerprinting with `Recorder(..., canonicalize=strip_volatile)` so they still replay.

Async agents are supported throughout: `@al.traced` works on `async def` functions
(context propagates through awaits and `asyncio.gather`), and `Recorder.acall` is the
awaitable variant for async provider clients.

## Diff

```
$ agentrewind diff 3f2a91 8c17d0
2 divergence(s); first divergence is where the runs split:

1. [input] /llm:mock-4o — inputs differ
     left : {"messages":[{"content":"decide next step",...
     right: {"messages":[{"content":"decide the next step",...
2. [output] /tool:search — outputs differ
```

The diff walks both span trees in parallel and reports structure, input, and output
divergences in execution order — entry #1 is where the runs first split.

## CLI & viewer

```bash
agentrewind list                 # recent runs
agentrewind show <trace-id>      # span tree with latencies
agentrewind diff <run1> <run2>   # structural diff (exit code 2 if runs differ)
agentrewind serve                # web viewer at http://127.0.0.1:4317
agentrewind export <trace-id> -o baseline.json  # portable artifact for a bug report or CI fixture
agentrewind import baseline.json
```

The web viewer shows the span waterfall per run, and lets you select any two runs to see
a side-by-side divergence view (`/api/diff/{a}/{b}` for programmatic access).

## Try the demo (offline, no API key)

```bash
python examples/research_agent.py   # records two runs with a seeded regression
agentrewind diff <run1> <run2>        # pinpoints the prompt change that caused it
```

## Development

```bash
pip install -e '.[dev]'
pytest                # SQLite half of the suite; PostgreSQL tests skip without a database
ruff check .
```

Every test is parametrized over both storage backends. To run the full suite locally, start
the bundled PostgreSQL first:

```bash
docker compose up -d
export AGENTREWIND_DB_URL=postgresql://agentrewind:agentrewind@localhost:55432/agentrewind
pytest --cov-fail-under=97
```

**290 tests (145 per backend), 99.56% branch coverage** with PostgreSQL available. CI runs the full suite
against a PostgreSQL 17 service container on Python 3.10, 3.12 and 3.13, and enforces the 97%
coverage floor. The floor is CI-only because a local run without PostgreSQL leaves the
backend's module uncovered.

The suites worth knowing about:

- `test_diff_alignment.py` pins the greedy span-alignment walk in `diff.py`. When two runs take
  different shapes, that walk decides whether an extra step is reported against the left or the
  right run, which is what makes the first divergence point at the real cause. Covers
  insertions, deletions, positional swaps, trailing spans and same-name-different-kind.
- `test_replay.py` and `test_streaming.py` check that a replayed run is byte-for-byte identical
  to the recording and makes no provider calls.
- `test_concurrency.py` and `test_async.py` cover concurrent recording into one store.
- `test_redaction.py` covers credential scrubbing before anything reaches the store.
- `test_storage_backends.py` is the backend contract: identical ordering, lookup and
  round-trip behaviour on both backends; migrating a database written by the published 0.2.2
  release (`tests/fixtures/traces_v0_2_2.db`); and separate writer processes saving
  distinct traces and contending for the same one.

Dependency updates come through Renovate (`renovate.json`); GitHub Actions bumps are grouped
and automerged once CI is green, majors always get a human review.

## Reproducible evaluation

The offline benchmark verifies that replay is byte-for-byte faithful and makes zero provider
calls; it also reports local recording and replay throughput:

```bash
python benchmarks/replay_benchmark.py --calls 1000
```

A second benchmark measures 1, 4, 8, 16 and 32 concurrent writer processes on each storage
backend (traces/sec and p95 write latency):

```bash
python benchmarks/storage_benchmark.py --output results.json
```

See [the evaluation protocol](docs/EVALUATION.md) for a repeatable benchmark and CI regression
gate workflow, and [the architecture overview](docs/ARCHITECTURE.md) for implementation details.

Note for macOS: if this repo lives in an iCloud-synced folder (e.g. `~/Documents`), create
your virtualenv *outside* it (e.g. `~/.venvs/agentrewind`) — the file provider marks files in
dot-directories as hidden, and Python ≥3.13 skips hidden `.pth` files, which breaks
editable installs.

MIT licensed.
