Metadata-Version: 2.5
Name: crashlab
Version: 0.0.3
Summary: Crash-test your AI agents: record, replay, diagnose and verify agent failures.
Project-URL: Homepage, https://github.com/Mpasha17/crashlab
Project-URL: Repository, https://github.com/Mpasha17/crashlab
Project-URL: Issues, https://github.com/Mpasha17/crashlab/issues
Project-URL: Changelog, https://github.com/Mpasha17/crashlab/blob/main/CHANGELOG.md
Author: Mahaboob Pasha
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agent-evaluation,agents,ai,ai-agents,anthropic,debugging,litellm,llm,observability,openai,pytest,replay,testing
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Debuggers
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: pydantic>=2
Requires-Dist: rich>=13
Requires-Dist: tomli>=1.1; python_version < '3.11'
Requires-Dist: typer>=0.12
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.30; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: pyyaml>=6; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: litellm
Requires-Dist: litellm>=1.40; extra == 'litellm'
Requires-Dist: mcp>=1.0; extra == 'litellm'
Provides-Extra: openai
Requires-Dist: openai>=1.40; extra == 'openai'
Provides-Extra: yaml
Requires-Dist: pyyaml>=6; extra == 'yaml'
Description-Content-Type: text/markdown

# crashlab

[![CI](https://github.com/Mpasha17/crashlab/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/Mpasha17/crashlab/actions/workflows/ci.yml)
[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://github.com/Mpasha17/crashlab/blob/main/LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://github.com/Mpasha17/crashlab/blob/main/pyproject.toml)

**Crash-test your AI agents.** Your agent failed in production. crashlab replays the exact
run offline, tells you which step broke and why, and checks whether a fix works before you
ship it.

crashlab is an open-source, framework-agnostic Python library that turns agent failures into fixes you can trust. It works today:

- **Record** every LLM and tool call from your existing agent loop (OpenAI, Anthropic, LiteLLM), with no rewrite.
- **Replay** any failed run exactly, offline, and fork it at any step.
- **Diagnose** the failing step and its root cause, backed by verbatim evidence from the trace.
- **Verify** a proposed fix by rerunning the failure and your past good runs.
- **Export** the failure as a pytest regression test.
- **Test RAG apps** with deterministic oracle helpers in `crashlab.rag` ([docs/rag-testing.md](https://github.com/Mpasha17/crashlab/blob/main/docs/rag-testing.md)).

![crashlab catching the smolagents empty-:memory: bug (terminal demo, no keys)](https://raw.githubusercontent.com/Mpasha17/crashlab/main/docs/demo/smolagents-text-to-sql.gif)

```bash
pip install "crashlab[openai]"
```

Then try it [offline with no API key](#try-it-with-no-api-key-and-no-model) or see the
[Quickstart](#quickstart).

> **Status: pre-alpha (0.0.3).** Today crashlab **records** runs, lets you inspect them,
> **replays** them strictly offline, grades them with **`@crashlab.check`** oracles,
> **forks** them at any step with typed **patches** and **diffs** two runs. It also
> **diagnoses** failed runs, **proposes fixes** and **verifies** them with repeated live
> forks, then **exports** a pytest regression test and a shareable HTML report (see
> [ARCHITECTURE.md](https://github.com/Mpasha17/crashlab/blob/main/ARCHITECTURE.md)). On PyPI: `pip install crashlab`.

## Reproduced a real upstream bug

We pointed crashlab at well-known open-source agent examples, with oracles from their
documented behaviour — no injected faults.

**Flagship: [huggingface/smolagents](https://github.com/huggingface/smolagents)
`examples/text_to_sql.py`.** This is a known, still-unmerged bug with an open fix PR
([#2475](https://github.com/huggingface/smolagents/pull/2475), opened 8 Jul 2026 by xiangyuhao723); crashlab reproduced and
diagnosed it independently. The example builds `sqlite:///:memory:`, then
`LocalPythonExecutor` runs every code block in a fresh `ThreadPoolExecutor`.
SQLAlchemy's default `SingletonThreadPool` for `:memory:` opens a new, **empty**
database in that thread, so the agent hits `no such table: receipts` and invents a
failure. Direct calls on the main thread work, which hides the bug. crashlab adds a
no-LLM repro through smolagents' own `LocalPythonExecutor`: upstream → error,
`StaticPool` + `check_same_thread=False` → `('Woodrow Wilson',)`.

Case studies: [docs/case-studies/](https://github.com/Mpasha17/crashlab/blob/main/docs/case-studies/README.md). Flagship write-up: [smolagents text_to_sql](https://github.com/Mpasha17/crashlab/blob/main/docs/case-studies/smolagents-text-to-sql.md).
Terminal demo (no keys): [docs/demo/](https://github.com/Mpasha17/crashlab/blob/main/docs/demo/README.md) (GIF above). We did **not** file an
upstream issue: it would duplicate PR #2475.

Also tried: OpenAI Agents SDK `examples/customer_service` (Triage Agent answers
follow-up FAQ questions itself on a smaller open model; two scoped prompt patches were
correctly **REJECTED** by `crashlab verify` for regressing past FAQ runs — the rejection
is the point). CrewAI: recording gap on `with_raw_response.create` fixed; 3/3 live
scenarios pass. Integration notes: [examples/integrations/](https://github.com/Mpasha17/crashlab/tree/main/examples/integrations).

## Results so far

Live runs of the bundled demo refund agent with injected faults, on free tiers. Each
fix was checked by `crashlab verify`: patched forks vs unpatched controls, plus past good
runs as a regression check. Small n; the labels are deliberately conservative.

| fault in the demo agent | model | verdict |
|---|---|---|
| `policy_drop`: the 30-day refund-window rule is missing from the system prompt | Groq gpt-oss-120b | **VERIFIED**: patched 5/5 vs control 0/5, 2/2 past good runs still pass, $0.002 |
| `policy_drop` (same restore-policy patch) | Groq gpt-oss-20b, Gemini flash-lite | **VERIFIED**: 3/3 vs control 0/3 on each |
| `misleading_desc`: the tool description says to refund `total * 0.9` | Groq gpt-oss-20b, Gemini flash-lite | **VERIFIED**: 3/3 vs control 0/3, regression run passes (gpt-oss-120b: INCONCLUSIVE, rate-limited) |
| `html_502`: `get_order` returns an HTML 502 page and the agent retries in a loop | Groq gpt-oss-120b | **LIKELY** only: 3/3 vs control 1/3 (intervals overlap) |

On a real, unmodified example (OpenAI Agents SDK `customer_service`), both prompt
patches for a genuine triage failure were **REJECTED** because they broke past good FAQ
runs. That is the verifier doing its job.

Evidence: [live scoreboard](https://github.com/Mpasha17/crashlab/blob/main/docs/case-studies/worker-c-live-scoreboard.md),
[case studies](https://github.com/Mpasha17/crashlab/blob/main/docs/case-studies/README.md), [Verify a fix](#verify-a-fix).

## What works today

- **`crashlab.wrap(client)`** records LLM calls, sync and async:
  - OpenAI `OpenAI` / `AsyncOpenAI` and any OpenAI-compatible endpoint (Ollama, vLLM,
    LiteLLM proxy ...): `chat.completions.create` and `responses.create`.
  - Anthropic `Anthropic` / `AsyncAnthropic`: `messages.create` and `messages.stream(...)`.
- **LiteLLM:** `crashlab.integrations.litellm.patch()` (or `crashlab.wrap(litellm)`) records
  `litellm.completion` / `litellm.acompletion`; `unpatch()` restores them.
- **Streaming:** with `stream=True` (and Anthropic's `messages.stream()`), chunks reach your
  code unchanged; crashlab assembles and records the final response when the stream ends,
  errors, or is closed early (marked incomplete).
- **One common shape** for every provider: system, messages as typed parts (text,
  tool calls, tool results, images/audio/files as placeholders, reasoning), tools, params,
  plus finish reason and token usage. The same conversation on OpenAI and Anthropic
  normalizes to the same messages, and each request gets a stable sha256 `request_hash`.
- **`@crashlab.tool(side_effects=...)`** records tool arguments, results and exceptions;
  **`@crashlab.agent(name=...)`** / **`with crashlab.record(name, input=...)`** mark runs
  (input, output, status, usage totals, entry point, git sha). A generator / async
  generator agent (streaming UIs) is one run over the whole iteration; its yielded items
  are the output, and replay re-chunks recorded streams as they arrived.
- **Threads:** `crashlab.bind_context(fn)`, `crashlab.ContextThreadPoolExecutor` and
  `crashlab.run_in_thread(fn, ...)` keep recording tool/LLM calls made in worker threads
  (asyncio tasks and `asyncio.to_thread` work out of the box). Calls in a child process
  (multiprocessing, `os.fork`) are not part of the parent's run; an agent called there
  records a run of its own.
- **Local store** `.crashlab/` (or `$CRASHLAB_DIR`): a SQLite index plus one JSONL cassette
  per run. API keys, auth headers (`Authorization`, `x-api-key` ...) and token-like strings
  are redacted before anything is written; add your own rules with
  `crashlab.redaction.add_pattern(...)`, `add_key(...)`, `add_hook(fn)`.
- **Strict offline replay:** `crashlab.replay(run)` / `crashlab replay <run_id|last>`
  re-runs the recorded entry point with every LLM and tool call served from the cassette:
  no API key, no network (outbound connections are blocked), and tools never run, so
  `side_effects=True` tools cannot fire. Calls are matched by step order plus the normalized
  request hash; anything else raises `DivergenceError` with the step, the differing fields
  and a diff. Works for OpenAI chat / Responses, Anthropic (`messages.stream()` included) and
  LiteLLM, plain and streamed, sync and async. The replay is stored as a new run with
  `replay_of` pointing at the source.
- **Outcome oracles:** `@crashlab.check("agent-name")` functions get the finished run and
  pass (return `True` / `None`) or fail (`False`, a message string, or a failed `assert`).
  They run when a run ends and set `outcome` to `pass` / `fail`;
  `crashlab check <run_id>` re-runs them on a stored run.
- **Fork at step N with a patch:** `crashlab.fork(run, at_step=N, patch=...)` /
  `crashlab fork <run> --at N ...` re-runs the agent with steps before N served strictly from
  the cassette and step N onwards live (or against a client you pass), with the patch
  applied to every LLM request from N on. Tool calls identical to the parent's are served
  from the cassette; `side_effects=True` tools never run live (an identical call gets the
  recorded result, anything else is blocked and recorded as an attempt). The fork is a new
  run linked by `forked_from`, `fork_step` and the patch.
- **Typed patches** (pydantic, JSON/YAML-serializable, composable with `+`, each renders a
  human-readable diff): `SystemPromptPatch` (replace / append / prepend / insert, replace
  or delete a line / substitute; optional `when` scopes to one agent's prompt), `MessagePatch`, `InjectMessagePatch`, `ToolResultPatch`
  (counterfactual tool result or error), `ToolDescriptionPatch`, `ToolSchemaPatch`,
  `ModelParamPatch` (model and params). Work for OpenAI chat / Responses, Anthropic and
  LiteLLM.
- **`crashlab diff A B`** / `crashlab.diff(a, b)`: how the runs relate (fork / replay),
  outcome and check changes, tokens / steps / time side by side, a step-by-step comparison
  (same, different request, different response, only in A / B) and the first divergence
  with a field diff. `B` defaults to `A`'s parent.
- **Runtime checks** (built in, on by default): cheap deterministic detectors that run
  while the agent is recorded and on stored runs (`crashlab check`). They emit typed
  findings (kind, severity `info` / `warn` / `error` / `critical`, steps, a message and
  verbatim evidence quotes):
  `step_repetition` (same tool + arguments again and again, near-identical LLM requests),
  `tool_error_streak`, `bad_tool_args` (validated against the tool's JSON schema),
  `suspicious_tool_result` (HTML / error pages, empty or truncated results, a result whose
  shape drifted from earlier results of the same tool), `budget` (steps, LLM / tool calls,
  tokens, USD, seconds), `length_cutoff` and `no_llm_calls` (tools ran but no model call
  was recorded: the framework's client is not wrapped), `failed_side_effect_claim` (every
  call of a `side_effects=True` tool failed, yet the final answer says it went through),
  `tool_result_instructions` (a tool result talks to the model, e.g. "note to the
  assistant: … do not mention this": possible prompt injection; an error when a
  side-effect tool ran after it), `unhandled_tool_error` (a tool raised and the
  exception ended the run, so the model never saw it: a prompt fix cannot help).
  Findings are stored with the run and shown by
  `show`, `diff` and `ls --flagged`. They describe *how* a run behaved; `@crashlab.check`
  oracles still decide whether it passed. Opt-in **circuit breaker**: a critical finding
  stops the agent (`CircuitBreakerError`), and strict replay stops at the same step.
- **Cost estimates** per step and per run in USD, from a small built-in price table
  (OpenAI, Anthropic, Groq, Gemini; dated prices) plus LiteLLM's cost map when LiteLLM is
  installed. Local servers (Ollama, localhost) cost $0. Unknown models show no cost; prices
  are never guessed. Your own prices: `crashlab.pricing.set_price(...)` or
  `[tool.crashlab.prices]`.
- **Diagnose:** `crashlab diagnose RUN` / `crashlab.diagnose(run, model=...)` names the
  failing step, a failure label (`TOOL_ERROR_MISHANDLED`, `BAD_TOOL_ARGUMENTS`,
  `WRONG_TOOL_SELECTION`, `STEP_REPETITION`, `TASK_SPEC_VIOLATION`, `EXTERNAL_FAILURE`, or
  `NO_FAILURE` / `UNKNOWN`), a one-sentence root cause and verbatim evidence quotes, from
  one LLM call over a compacted trace with the checks and findings as hints. Every quote
  is checked against the step it cites; an answer without valid evidence becomes
  `UNKNOWN`. Any model works (Groq, Gemini, Ollama, OpenAI, Anthropic, LiteLLM);
  `--heuristic` needs none. Stored with the run, shown by `crashlab show`, cached.
- **Fix proposals:** `crashlab fix RUN` / `crashlab.propose_fixes(run)` makes a second,
  separate LLM call that turns the diagnosis into 1-3 ranked typed patches (system prompt
  line, tool description or schema, model params, a message; tool-result counterfactuals
  are marked test-only), each with a rationale and expected effect. A proposal is kept
  only if it parses, its target exists in the run (dry-run diff), it is small, and it does
  not hard-code this run's data (order ids, amounts ...). Proposals that hand the decision
  to a person ("let me know", "ask the customer to confirm") are dropped: verification
  forks run unattended. If the agent has past passing runs whose system prompt or tool
  descriptions differ, a **restore** proposal puts the missing or changed text back where
  it was (no LLM, so it is proposed even with no fix model or when the model call fails;
  only text most good runs agree on) and is ranked first, then in-place
  edits, tool edits and appended rules. Kept fixes are stored with the
  run and saved as patch files for `crashlab fork --patch`; `--try` forks once with the
  top fix and shows the checks.
- **Secondary causes:** when a run has more than one independent failure (e.g. an HTML
  error page followed by an over-limit order), the diagnosis keeps a primary cause plus
  capped, deduplicated secondary causes; they show in `--json`, `show`, HTML reports, fix
  prompts and exported tests. Vague failures with no clear cause stay `UNKNOWN`.
- **RAG testing:** `crashlab.rag` has small deterministic oracle helpers for wrong
  retrieval (`retrieved_ids`), missing context (`abstains_without_context`), stale
  documents (`no_stale_docs`), citation mismatch (`citations_were_retrieved`) and
  numeric / claim grounding (`answer_grounded`, `claims_supported`). You wrap your own
  retriever as a `@crashlab.tool`; there are no vector-DB integrations, and the grounding
  checks are heuristics (number and word overlap plus negation), not semantic equivalence. See
  [docs/rag-testing.md](https://github.com/Mpasha17/crashlab/blob/main/docs/rag-testing.md).
- **`crashlab report RUN`:** one screen with the outcome, checks, findings, diagnosis,
  fixes with diffs and the next commands to run (`crashlab diagnose --fix` prints it too).
- **Share runs:** `crashlab export RUN -o run.jsonl` / `crashlab import run.jsonl`
  (redacted again on the way out).
- **CLI:** `crashlab ls`, `crashlab show <run_id|last>`, `crashlab replay`, `crashlab check`,
  `crashlab diagnose`, `crashlab fix`, `crashlab report`, `crashlab fork`, `crashlab diff`, `crashlab export`, `crashlab import`; in Python,
  `crashlab.load("last")`.

**Recording format and compatibility.** Runs are stored as full JSONL cassettes by default.
crashlab can read a compact cassette format (much smaller files on long chats: about
0.43 MB vs 54 MB in a 1,500-message test), but *writing* it is experimental and opt-in
(`CRASHLAB_EXPERIMENTAL_COMPACT=1`); short chats can write more slowly than the full
format, and `export-test` / `export` always write the full format. Older crashlab
versions may not read newer runs; see
[docs/compatibility.md](https://github.com/Mpasha17/crashlab/blob/main/docs/compatibility.md).

**Start agents under `if __name__ == "__main__":`.** `crashlab check`, `replay` and
`export-test` import your entry-point module. If an agent runs at import time, crashlab
now blocks the stray recording (including threads that agent starts) and prints a clear
error instead of recursing, but subprocesses and custom queues are not covered.

Not yet: a live HTML viewer server. Streamed calls replay as a synthetic stream rebuilt from the recorded
final response, cut at the recorded chunk boundaries (OpenAI-style chat, Anthropic and
Responses streams), so per-chunk consumers see the same pieces.
Also not intercepted yet: OpenAI `with_streaming_response.create` (raw HTTP body; crashlab
warns once when it is used in a run), Anthropic `messages.batches` / beta `tool_runner`.
Structured outputs — Anthropic `messages.parse()` (also `client.beta.messages`), OpenAI
`chat.completions.parse()` / `responses.parse()` (also `client.beta.chat`) — the `.stream()`
helpers and `with_raw_response.create` (CrewAI path) **are** recorded. OpenAI
`embeddings.create` records as an `embeddings` step (its tokens count; no cost estimate);
Anthropic `messages.count_tokens` as a `custom` step (so replay stays offline).
OpenAI streams only report token usage if you pass `stream_options={"include_usage": True}`.

## Quickstart

```bash
pip install "crashlab[openai]"
# or crashlab[anthropic], crashlab[litellm] (also installs mcp, which litellm 1.106+
# imports for tool calls)
```

**Pick the model that diagnoses and fixes runs** (any OpenAI-compatible endpoint, not
only OpenAI). Set it once, then `crashlab diagnose` / `fix` / `fix --verify` need no flags:

```bash
# Gemini (free tier) via its OpenAI-compatible endpoint
export CRASHLAB_DIAGNOSER_MODEL=gemini-flash-lite-latest
export CRASHLAB_DIAGNOSER_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai/
export CRASHLAB_DIAGNOSER_API_KEY="$GEMINI_API_KEY"
# Groq: openai/gpt-oss-120b + https://api.groq.com/openai/v1 + $GROQ_API_KEY
# Ollama: qwen2.5:7b + http://localhost:11434/v1 (no key)
```

Or per command: `--model M --base-url URL --api-key-env NAME_OF_THE_ENV_VAR` (the flag takes
the variable's *name*, never the key). This model is separate from your agent's: forks
and `verify` re-run your agent with its own client and keys.

**Using a framework?** Wrap the client it already uses. One-liners below are from the
live notes in
[examples/integrations/](https://github.com/Mpasha17/crashlab/tree/main/examples/integrations)
(do not wrap `ChatOpenAI` itself — that raises `TypeError`):

- **LangChain / LangGraph:** `oc = crashlab.wrap(OpenAI(...)); ChatOpenAI(..., client=oc.chat.completions, root_client=oc)` — [langchain](https://github.com/Mpasha17/crashlab/blob/main/examples/integrations/langchain.md) · [langgraph](https://github.com/Mpasha17/crashlab/blob/main/examples/integrations/langgraph.md)
- **PydanticAI:** `OpenAIProvider(openai_client=crashlab.wrap(AsyncOpenAI(...)))` — [pydantic-ai](https://github.com/Mpasha17/crashlab/blob/main/examples/integrations/pydantic-ai.md)
- **OpenAI Agents SDK:** wrap `AsyncOpenAI`, `set_default_openai_client(...)`, pass `OpenAIChatCompletionsModel(..., openai_client=client)` — [openai-agents](https://github.com/Mpasha17/crashlab/blob/main/examples/integrations/openai-agents.md)
- **smolagents:** `m.client = crashlab.wrap(m.client)` — [smolagents](https://github.com/Mpasha17/crashlab/blob/main/examples/integrations/smolagents.md)
- **CrewAI:** `llm._client = crashlab.wrap(llm._client)` — [crewai](https://github.com/Mpasha17/crashlab/blob/main/examples/integrations/crewai.md)
- **LlamaIndex:** `OpenAILike(..., async_openai_client=crashlab.wrap(AsyncOpenAI(...)))` — [llamaindex](https://github.com/Mpasha17/crashlab/blob/main/examples/integrations/llamaindex.md)
- **Agno:** `OpenAILike(..., client=crashlab.wrap(OpenAI(...)))` — [agno](https://github.com/Mpasha17/crashlab/blob/main/examples/integrations/agno.md)
- **Haystack:** `gen.client = crashlab.wrap(OpenAI(...))` — [haystack](https://github.com/Mpasha17/crashlab/blob/main/examples/integrations/haystack.md)
- **AutoGen:** `mc._client = crashlab.wrap(mc._client)` — [autogen](https://github.com/Mpasha17/crashlab/blob/main/examples/integrations/autogen.md)
- **AG2:** subclass the config's `create()` and wrap `client._client` — [ag2](https://github.com/Mpasha17/crashlab/blob/main/examples/integrations/ag2.md)
- **DSPy:** `crashlab.integrations.litellm.patch()` + `dspy.LM(..., engine="litellm", cache=False)` — [dspy](https://github.com/Mpasha17/crashlab/blob/main/examples/integrations/dspy.md)
- **Google ADK:** `crashlab.integrations.litellm.patch()` + `LiteLlm(...)` (native Gemini not recorded yet) — [google-adk](https://github.com/Mpasha17/crashlab/blob/main/examples/integrations/google-adk.md)

```python
import crashlab
from openai import OpenAI

client = crashlab.wrap(OpenAI())          # or OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
# client = crashlab.wrap(anthropic.Anthropic())

@crashlab.tool()
def get_order(order_id: str) -> dict: ...

@crashlab.tool(side_effects=True)
def issue_refund(order_id: str, amount: float) -> dict: ...

@crashlab.agent(name="refund-bot")
def run_agent(user_msg: str) -> str:
    ...                                    # your existing tool-calling loop, unchanged

@crashlab.check("refund-bot")              # outcome oracle: pass / fail
def refund_not_above_total(run) -> None:
    for step in run.tool_calls("issue_refund", ok_only=True):
        assert step.args["amount"] <= 49.0, f"refunded {step.args['amount']}"

run_agent("Refund my order A17, it arrived broken")

run = crashlab.load("last")                # Run with .steps (chat / execute_tool), .outcome
replayed = crashlab.replay(run)            # offline; raises DivergenceError if it differs

# what if the prompt had said so? fork at the first LLM call with a patch, run live
branch = crashlab.fork(run, at_step=0, patch=crashlab.SystemPromptPatch(
    op="append", text="Never refund more than the order total."))
print(crashlab.diff(run, branch).summary())
```

```bash
crashlab ls            # runs: id, name, status, outcome, steps, tokens, cost, findings
crashlab ls --flagged  # only runs with runtime-check findings (--min-severity error)
crashlab show last     # checks, findings + step table: idx, op, name, args / result, error
crashlab replay last   # offline re-run from the cassette; exit 1 + diff on divergence
crashlab check last    # re-run the oracles and the runtime checks (--save to store them)
crashlab check --all --findings-only --max-cost 0.05 --fail-on error   # e.g. in CI
crashlab check --all --save   # after upgrading crashlab: apply new runtime checks to stored runs
crashlab fork last --at 0 --system-append "Never refund more than the order total."
crashlab fork last --at 2 --patch fix.yaml --dry-run   # show the patch diff, run nothing
crashlab fork last --at 0 --patch fix-1   # a fix proposed by `crashlab fix` (by id)
crashlab diff last     # the fork vs its parent: checks, usage, steps, first divergence
```

Runtime checks are configured per run or in `pyproject.toml`, and `record(checks=...)`
wins over pyproject:

```python
@crashlab.agent(name="refund-bot", checks={"max_steps": 20, "max_cost_usd": 0.10,
                                           "circuit_breaker": True})
```

```toml
[tool.crashlab.checks]
circuit_breaker = true          # stop the agent on a critical finding (default: off)
max_steps = 20                  # budget: also max_llm_calls, max_tool_calls, max_tokens,
max_cost_usd = 0.10             #   max_seconds; warn at 80% (budget.warn_at)
[tool.crashlab.checks.step_repetition]
threshold = 3                   # warn at 3 identical tool calls, critical at 5
```

### Try it with no API key and no model

The repo ships five real recorded runs of the demo agent (secrets and local paths scrubbed):
a clean refund, a policy violation the oracles fail, a 502 retry loop the runtime checks
flag, a run the circuit breaker stopped, and a small local model passing a bad argument.

```bash
git clone https://github.com/Mpasha17/crashlab && cd crashlab
python -m venv .venv && . .venv/bin/activate && pip install -e ".[openai]"
export CRASHLAB_DIR=/tmp/crashlab-demo
for f in examples/refund_bot/demo/*.jsonl; do crashlab import "$f"; done
crashlab ls                        # 5 runs, with cost and findings
crashlab show 01M57GM9Q5           # Groq gpt-oss-120b retrying an HTML 502 page 3 times
crashlab check --all               # oracles + runtime checks, offline (exit 1: C03 fails)
crashlab replay 01M57GM7H9         # strict offline replay of the policy violation
crashlab diagnose 01M57GM7H9 --heuristic   # failing step + label from checks/findings
crashlab fork 01M57GM7H9 --at 0 --dry-run \
  --system-append "Never refund an order delivered more than 30 days ago."
```

### Diagnose a run

```bash
# any OpenAI-compatible endpoint; the key is read from the env var you name
crashlab diagnose 01M57GM7H9 --model openai/gpt-oss-120b \
  --base-url https://api.groq.com/openai/v1 --api-key-env GROQ_API_KEY
crashlab diagnose last --model gemini/gemini-flash-latest      # via LiteLLM
crashlab diagnose last --model qwen2.5:7b --base-url http://localhost:11434/v1  # Ollama
crashlab diagnose last --show-prompt   # the compacted trace the diagnoser reads, no call
```

```python
d = crashlab.diagnose(run, model="openai/gpt-oss-120b",
                      base_url="https://api.groq.com/openai/v1", api_key=key)
d.label, d.failing_step, d.root_cause, d.evidence   # also run.diagnosis, `crashlab show`
```

Or set `CRASHLAB_DIAGNOSER_MODEL` / `CRASHLAB_DIAGNOSER_BASE_URL` /
`CRASHLAB_DIAGNOSER_API_KEY`. The diagnoser's own calls are recorded in
`.crashlab/diagnoser/`, apart from your runs.

On 31 hand-labelled live runs of the demo agent (`bench/diagnose_eval.py`, details in
[bench/README.md](https://github.com/Mpasha17/crashlab/blob/main/bench/README.md)), with prompt v3 Groq `openai/gpt-oss-120b` gets the label
right on 90% and Gemini `gemini-flash-lite-latest` on 87% (heuristic baseline: 81%, tuned on
the same set); exact culprit step on 71% / 76% of the failing runs, with no false alarms
on clean runs. These LLM numbers were measured on that 31-case set with prompt v3.
The `--heuristic` diagnoser (no LLM) was later re-measured on an expanded 35-case set:
correct label on 35/35, exact failing step on 92%, no false alarms; on 21 held-out stored
real-agent runs it got the expected category on 21/21, and on 13 adversarial runs it landed
on an acceptable cause on 13/13. These are small sets used during development, not a
general accuracy claim.

### Propose a fix and try it

```bash
crashlab fix 01M57GM7H9 --model openai/gpt-oss-120b \
  --base-url https://api.groq.com/openai/v1 --api-key-env GROQ_API_KEY
crashlab report 01M57GM7H9          # diagnosis, fix-1 with its diff, next commands
# --try forks once with fix-1, live: the agent uses its own client, e.g. on Groq
OPENAI_BASE_URL=https://api.groq.com/openai/v1 OPENAI_API_KEY=$GROQ_API_KEY \
  crashlab fix 01M57GM7H9 --model openai/gpt-oss-120b \
  --base-url https://api.groq.com/openai/v1 --api-key-env GROQ_API_KEY --try
```

If verify REJECTS fix-1, run `crashlab fix` again: once a fix was verified, the fix model
is told which ones were rejected and proposes something new (fix-2 ...). `fix --verify`
does this on its own: after a REJECTED verdict it tries the next fix (`--attempts 2`, the
default). In live tests on seven agent stacks the first proposal was REJECTED on four and a
later one was VERIFIED.

### Verify a fix

```bash
# 5 forks with fix-1 + 5 unpatched controls + the patch rerun on past good runs
OPENAI_BASE_URL=https://api.groq.com/openai/v1 OPENAI_API_KEY=$GROQ_API_KEY \
  crashlab verify 01M57GM7H9 --fix fix-1 -k 5 --regress 01M57GM0VB,01M57GM5RH --max-cost 0.50
# or all in one: diagnose (if needed) -> fix -> verify
crashlab fix 01M57GM7H9 --model ... --verify -k 5
```

**Side-effect tools in forks.** A `side_effects=True` tool never runs for real outside
recording. A fork serves it the recorded result of an *identical* call; a call the
recording never made (order 5 instead of 10, "widgets" for "widget") is blocked, and that
trial can't be graded. If your fix is meant to change such a call, register a fake.
crashlab calls the fake instead, and the trial counts (steps show `◇ fake`):

```python
@crashlab.fake_tool("place_order")             # same signature as the real tool
def fake_place_order(product: str, qty: int) -> dict:
    return {"order_id": "FAKE-1", "product": product, "qty": qty}
# or: @crashlab.tool(side_effects=True, fake=fake_place_order)
```

Or pass `--allow-live-side-effects` when the tools point at a sandbox. When trials were
blocked, the verify panel prints a one-line hint naming both.

Verdicts: **VERIFIED** (patched Wilson 95% lower bound above the control's upper bound,
or controls all fail and patched all pass; no regression), **LIKELY** (better, but not
significant or incomplete), **INCONCLUSIVE** (too few usable trials), **REJECTED** (a
past good run now fails, or no better than control). Provider errors (429/5xx) are
counted as `errored`, never as fails. So are trials where the agent makes a
`side_effects=True` call the source run never made (crashlab blocks it, so the outcome is
unknown); if the fix is *supposed* to change such a call (e.g. order 5 instead of 10), pass
`--allow-live-side-effects` with sandboxed tools to grade those trials. `--no-control`, `--regress none|tag:X|ID,ID`,
`--json`, `--pace SECONDS` (free-tier rate limits). Child runs are tagged `verify:<id>`; `crashlab report` lists fix attempts.
On Groq gpt-oss-120b: policy_drop fix-1 went 5/5 vs control 0/5, 2/2 good runs still
pass (VERIFIED, $0.002).

**What VERIFIED means.** It describes the trial outcomes observed with the patch that was
actually applied; it is not proof that a particular wording caused the fix. Each result
names that patch (kind, operation and a short digest) in the CLI, `--json` and HTML
report, so check it: in one of our own tests, verifying "fix-1" by rank ran the
top-ranked *restore* patch rather than the weaker "Be polite" proposal we meant to test.
Selected explicitly, the no-op and "Be polite" patches were REJECTED (0/3) and the real
policy fix VERIFIED (3/3).

### Export a regression test and an HTML report

```bash
crashlab export-test 01M5GTK1FR --fix auto -o tests/crashlab   # test + cassettes
pytest --crashlab            # (pip install pytest) exported tests only: offline, no keys, no network
crashlab report 01M5GTK1FR --html report.html   # one self-contained file to share
```

Offline recipe: the exported test needs only `crashlab`, your agent's code and its
framework installed, with no keys and no network. To prove it in CI or locally:

```bash
crashlab export-test last --fix auto -o tests/crashlab
env -u OPENAI_API_KEY -u GEMINI_API_KEY pytest --crashlab tests/crashlab
unshare -rn pytest --crashlab tests/crashlab    # Linux: no network namespace at all
```

A replay re-runs your code, so if your agent reads its model name or streaming flag from
the environment, list those variables and crashlab records their values and sets them
again on replay, whatever the shell has: `@crashlab.agent(name="bot", env=["MODEL",
"STREAM"])`. Names that look like secrets are refused, and replays never need keys.

The exported test replays the failing run and asserts its `@check` results, and (with
`--fix`) replays a passing patched run from `crashlab verify` with the fix re-applied,
asserting it still passes. The pytest plugin (auto-loaded) adds `--crashlab`, the
`@pytest.mark.crashlab` marker and the `crashlab_replay(path)` / `crashlab_store`
fixtures. Store and home paths are replaced in exported cassettes. See
[examples/refund_bot/tests](https://github.com/Mpasha17/crashlab/tree/main/examples/refund_bot/tests), which CI runs.
The HTML report has the outcome, checks, findings, diagnosis with quotes, fixes with
diffs, verification stats and the step timeline; inline CSS, no scripts, no external
assets.

### Free, local, with Ollama

Running the demo agent, or a fork for real (without `--dry-run`), needs a model. A local
one is free:

```bash
ollama pull qwen2.5:3b             # or llama3.2:3b / qwen2.5:7b: any tool-calling model
OPENAI_BASE_URL=http://localhost:11434/v1 OPENAI_API_KEY=ollama MODEL=qwen2.5:3b \
  python -m examples.refund_bot "Refund my order A17, it arrived broken"
FAULT_HTML_502=1 OPENAI_BASE_URL=http://localhost:11434/v1 OPENAI_API_KEY=ollama MODEL=qwen2.5:3b \
  python -m examples.refund_bot "Refund my order A17"     # get_order returns an HTML 502 page
# or through LiteLLM (pip install -e ".[litellm]"): ollama_chat/ uses Ollama's native tools
python -m examples.refund_bot --provider litellm --model ollama_chat/qwen2.5:3b "Refund A17"
crashlab ls && crashlab show last  # cost: $0
crashlab replay last               # no model server needed

# the same agent on Anthropic
PROVIDER=anthropic ANTHROPIC_API_KEY=sk-ant-... MODEL=claude-haiku-4-5 \
  python -m examples.refund_bot "Refund my order A17, it arrived broken"
```

See [examples/refund_bot](https://github.com/Mpasha17/crashlab/blob/main/examples/refund_bot/README.md) for details.

See [CHANGELOG.md](https://github.com/Mpasha17/crashlab/blob/main/CHANGELOG.md) for what landed in each release.

## Benchmark (seed, small n)

`crashlab bench` injects 4 faults into 2 demo agents (3 episodes per cell, fixes verified
with k=3) and scores the diagnoser on a held-out agent (`sql-analyst`) that nothing was
tuned on. On Gemini flash-lite, the LLM diagnoser labelled **6/6 triggered faults
correctly (Wilson 95% CI 61–100%)**; the heuristic baseline got 2/6 (10–70%). No fix was
VERIFIED on the held-out agent (0/3, 1/3 LIKELY): its faults were in the tool and the
database, which a prompt patch can't repair, so the rejections are correct. On Groq
gpt-oss-120b the LLM diagnoser labelled only **3/11 (10–57%)** on the held-out agent.
Samples are small and the intervals are wide; details, all CIs and caveats in
[bench/README.md](https://github.com/Mpasha17/crashlab/blob/main/bench/README.md#seed-results).

## Development

```bash
python -m venv .venv && . .venv/bin/activate
pip install -e ".[dev,openai,anthropic,litellm]"
ruff check . && ruff format --check . && pytest
```

The tests use fake clients and the real SDKs over mocked HTTP transports (LiteLLM via
`mock_response`), so they need no network access and no API key. Tests for SDKs that are
not installed are skipped.

Security notes: [SECURITY.md](https://github.com/Mpasha17/crashlab/blob/main/SECURITY.md).

## License

Apache-2.0
