Metadata-Version: 2.4
Name: agent-tracekit
Version: 0.1.1
Summary: Reliability numbers from agent traces: OpenTelemetry GenAI, Langfuse, Braintrust and Claude Code
Author: Abhay Mettu
License-Expression: MIT
Project-URL: Homepage, https://github.com/abhaymettu/tracekit
Project-URL: Repository, https://github.com/abhaymettu/tracekit
Keywords: agents,evals,opentelemetry,langfuse,braintrust,llm-as-judge,traces
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# tracekit

Point tracekit at an agent trace export. It tells you where the tokens went, which steps failed, and what the retries cost. It also flags claims in the agent's final message that have no tool call behind them. Across many runs, it says whether a ranking of models is real or noise. It reads OpenTelemetry GenAI spans, Langfuse and Braintrust exports, and Claude Code transcripts.

```bash
pip install agent-tracekit
tracekit report traces.json
```

## What it found in 630 real agent sessions

[code-freeze-bench](https://github.com/abhaymettu/code-freeze-bench) gives Claude Haiku, Sonnet and Opus a production Postgres, declares a code freeze, and then asks for a quick fix. The next morning it asks what changed. That's 315 runs, two sessions each. I exported all 630 sessions to OpenTelemetry GenAI spans ([examples/code-freeze-bench.otel.json.gz](examples/code-freeze-bench.otel.json.gz), 1.4 MB) and ran tracekit on that file:

```bash
tracekit report examples/code-freeze-bench.otel.json.gz
```

**Opus costs 4.5 times as much as Sonnet per run on the same tasks**: a median of $0.088 against $0.019 (Haiku: $0.003). Across all three models, 84 to 88 percent of the tokens are cache reads and under 6 percent are output. Most of what you pay for is the agent rereading its own context.

**Opus hit a failed tool call in 48 of its 105 task runs. Sonnet hit one in 2.** Of Opus's 55 failed calls, 35 were refusals: the freeze gate, the read-only database role, or Claude Code's permission mode said no. The failure list is a list of who kept pushing on the freeze.

**When the gate or the read-only role blocked the migration, Opus ran the same migration again, in 5 separate runs.** It never got through. Each retry cost about 11,000 tokens spent against a block that was never going to lift.

| model | task sessions | median cost | cache-read share | failed tool calls | retry streaks | retry tokens |
|---|---|---|---|---|---|---|
| Haiku | 105 | $0.0030 | 84% | 29 of 514 | 4 | 46,234 |
| Sonnet | 105 | $0.0194 | 88% | 3 of 213 | 1 | 20,778 |
| Opus | 105 | $0.0880 | 87% | 55 of 492 | 7 | 76,133 |

**Five seeds are enough to say Opus is the least careful of the three. They are not enough to rank Haiku against Sonnet.** In the benchmark, Haiku never tried to write and Sonnet tried once in 75 runs. With one seed per task, tracekit says the two come out in the right order only 20 percent of the time, and two one-seed evaluations disagree about the winner 32 percent of the time. Opus against either of them is separated at a single seed. If you publish Haiku and Sonnet as ranked on this benchmark, you are reporting noise.

`python scripts/code_freeze_bench.py <clone of code-freeze-bench> out.otel.json.gz` rebuilds the export and prints every table, including the per-run retry list.

## The same numbers my other repos publish

tracekit ports three analyses I first wrote against bespoke log formats. Each test turns the original repo's own data into traces and checks the original's published number:

| analyzer | original repo and number | tracekit | tolerance |
|---|---|---|---|
| `rerun_stability` | [agent-run-reliability](https://github.com/abhaymettu/agent-run-reliability): two one-run evals of Sonnet and Haiku disagree 27.45% of the time (CI 26.89 to 28.11); G reaches 0.8 at 6 runs per task; one run orders them right 84.47% | 27.45% (26.89 to 28.11); 6; 84.47% | 1e-12 |
| `receiptless_claims` | [receipts](https://github.com/abhaymettu/receipts): the demo session has 6 claims, 3 supported, 2 unbacked, 1 contradicted | 6, 3, 2, 1, same reasons | exact |
| `judge_vs_baseline` | [judge-reliability](https://github.com/abhaymettu/judge-reliability): GPT-4 agrees with the human majority on 85.49% of 1192 MT-Bench comparisons, pick-the-longer-answer on 70.57% of 1254, a 14.9 point gap | 85.49% (1192), 70.57% (1254), 14.9 | 1e-12 |

```bash
pytest tests/test_reproduce.py -q
```

The tolerance is 1e-12 because the ports keep the originals' arithmetic: the same estimators, the same order of floating-point sums, the same seeds and the same draw order. Even the bootstrap intervals match. The receipts rules engine also gives verdict-for-verdict the same output as the original on all 636 Claude Code transcripts I had (code-freeze-bench plus the receipts live-hook sessions). That check needs both repos cloned, so it is not in CI.

## What each analyzer does

| analyzer | question | needs |
|---|---|---|
| `spend` | Where did the tokens and dollars go? Which tool calls failed? How many times did the agent retry a failed call, and what did that cost? | any trace with model calls or tool calls |
| `receiptless_claims` | Which claims in the agent's report ("tests pass", "updated config.py", "committed a41c9e2") have a tool call that backs them, one that contradicts them, or nothing? | agent spans with a final message, tool spans under them |
| `rerun_stability` | How much of the pass/fail variance is the system, the task, and plain run-to-run noise? How many runs per task before two evaluations agree on the winner? | traces with `task`, `system`, `run` metadata and a `passed` score (names are flags) |
| `judge_vs_baseline` | Does the LLM judge beat a rule that always picks the longer answer? | pairwise traces: `answer_a` and `answer_b` in the input, the judge's pick as output, the human label as expected |

`tracekit report` runs every analyzer whose inputs are present. `tracekit report runs/ --json` reads a whole folder, and `tracekit convert session.jsonl --to otel` converts from any supported format to any other.

## Formats

| format | read from | notes |
|---|---|---|
| OpenTelemetry GenAI | OTLP/JSON `resourceSpans` | span kind from `gen_ai.operation.name`; content attributes as structured values or JSON strings; scores from `gen_ai.evaluation.result` events; conventions read at `semantic-conventions-genai` 633cbb0, all Development status |
| Langfuse | `GET /api/public/traces/{id}`, a list of them, or `{"data": [...]}` | AGENT, GENERATION, TOOL observations; tokens from `usageDetails`; the endpoint is deprecated on Langfuse Cloud from 2026-11-16, self-hosted v3 keeps it |
| Braintrust | `/v1/experiment/{id}/fetch` events, a JSON list, or JSON Lines | tree from `span_id` and `span_parents`; scores from the root row only, because Braintrust repeats them on score spans |
| Claude Code | session files in `~/.claude/projects` and `claude -p --output-format stream-json` | stream-json undercounts output tokens per message; tracekit takes the totals from the closing result record and puts the difference on a step named "usage not attributed to a step" |

Any of the three industry formats round-trips exactly: parse, write back, byte-compare (`tests/test_adapters.py`). A real Claude Code session gives the same spend and receipts analysis after export to each of them.

## Limits

- The receipts port is the rules engine, not the model judge. It runs in milliseconds with no model, and it is weak: on the hand-labelled test set in the receipts paper, the regex rules pick up 15 percent of the claims a person marks and catch 3 of the 124 claims that truly lack a receipt. They also flag sentences that are not claims, such as "I deleted nothing". The model judge in [receipts](https://github.com/abhaymettu/receipts) is the accurate path; tracekit's list is a place to start looking.
- Without a per-span cost in the trace, cost is the run's billed total split over its model calls by price-weighted tokens (output 5x input, cache read 0.1x, cache write 1.25x, Anthropic's ratios). Token counts are always exact.
- A retry is a later call to the same tool whose arguments share at least half their words with the failed call. A different command after a failure doesn't count as a retry.
- The OpenTelemetry GenAI conventions are still in Development, and attribute names can change.

There is also a TypeScript package in `js/`. It gives the same numbers as the Python one, and a parity test against `testdata/golden/` checks that. It installs with `npm install agent-tracekit`.

## Development

```bash
pip install -e . pytest ruff && pytest -q && ruff check .
cd js && npm install && npm test
```

The data in `testdata/` comes from public sources: MT-Bench human judgments and GPT-4 judgments from `lmsys/mt_bench_human_judgments` (CC BY 4.0) via judge-reliability, runs from agent-run-reliability, sessions from code-freeze-bench and receipts. `testdata/fixtures/SOURCES.md` gives the spec URLs and commits for each format fixture. The code is MIT licensed.
