The Benchmark
Every compression method shrinks tokens. The only number that matters is how much it saves without changing what the agent decides — and what it truly costs once prompt caching is priced in. This is a head-to-head on exactly that axis, where every technique runs through the same decision-equivalence gate and the same cache-aware cost model. The winner is computed, not assumed.
The standings
Ranked by cache-aware dollar savings on the varied corpus (see methodology below). Certified means the method passed a statistical non-inferiority test and preserved 100% of decisions; anything less is disqualified, however much it saved. Competitor rows are produced by the real installed packages via the reproducible --external seam — not our reimplementations.
| Technique | What it is | Tokens saved | $ saved | Decision equiv. | Verdict | Fidelity |
|---|---|---|---|---|---|---|
| distil-causal | cache-aware + causal pruning | 80.5% | 81.5% | 100% | ✔ certified · leader | lossy* |
| truncate-tail | sliding-window / truncation | 78.7% | 79.6% | 14% | ✘ fails gate | lossy |
| distil-stream | lossless + cross-turn dedup (fully reversible) | 61.0% | 61.7% | 100% | ✔ certified | reversible |
| distil-lossless | cache-aware lossless + structured fold + template mining | 57.4% | 58.1% | 100% | ✔ certified | byte-exact |
| summarize | abstractive / rolling summary | 56.5% | 57.2% | 39% | ✘ fails gate | lossy |
| Headroom (headroom-ai, real) | structural crusher (SmartCrusher) | 36.1% | 36.5% | 84% | ✘ fails gate | lossy |
| extractive-prune | extractive importance (LLMLingua family) | 18.2% | 18.4% | 77% | ✘ fails gate | lossy |
| RTK (rtk-ai, real) | command-output wrapper — no raw-text mode | n/a | n/a | n/a | — not comparable | — |
| minify-all | lossless minification | 0.1% | 0.1% | 100% | ✔ certified | byte-exact |
*distil-causal drops context that ablation proves never changed a decision — not byte-reversible, but certified decision-equivalent. The three Distil operating points are all certified at 100%: distil-lossless (byte-exact: structured fold + template mining), distil-stream (adds cross-turn dedup of recurring tool output the cache can't reach — fully recoverable), and distil-causal (adds causal pruning). Headroom and RTK rows are the actual installed packages run through the --external seam: Headroom's SmartCrusher cuts 36% but its lossy transforms flip ~16% of decisions → disqualified; RTK only compresses the output of specific wrapped commands (git/cargo/…) and exposes no text API, so it isn't comparable on the context-compression axis (a layer difference, not a defect). Numbers: claude-opus-4-8 pricing, deterministic runner. Reproduce below.
The corpus
To avoid any one tool's home-turf advantage, the comparison runs on 64 reproducible trajectories across 8 families deliberately spanning both regimes: structured/repetitive data (JSON record arrays, SQL rows, metrics, logs) where structural compaction pays off, and diagnostic/prose content (Kubernetes incidents, stack traces, RAG chunks, support transcripts) that conservative crushers protect and lossy methods mangle. Decisions are buried inside large tool outputs — as on real agents — so naive head/tail truncation drops them (14% equivalence). A large reference doc is re-read every turn (the recurring tool output prompt caching can't reach), which Distil's cross-turn dedup collapses and others re-bill in full.
Why this is the honest comparison
No method is special-cased
Every technique — including Distil's own — is scored by the identical decision-equivalence + non-inferiority gate and the identical cache-aware cost model. The harness will happily rank a competitor above Distil if it earns it. It doesn't, because no other family combines lossless, cache-stability, causal pruning, and certification — but the door is open.
Best-form, not strawmen
The baselines are faithful reference implementations of the real technique families — sliding-window truncation, extractive importance pruning (the LLMLingua / Selective-Context lineage), abstractive summarization, naive minification — each in its best reasonable form. They genuinely remove tokens. They just can't prove they kept the decision.
Two ways to "win" that don't count
A method can post a big raw token cut yet flip decisions — Headroom's SmartCrusher cuts 36% but changes ~16% of decisions; summarize cuts 56% but keeps only 39%; truncation cuts 79% but keeps 14%. All disqualified. Or a method can shave tokens yet bust the prompt cache and cost more in real dollars. The benchmark prices both, so neither illusion survives.
Plug in a real tool
The Headroom and RTK rows aren't our reimplementations — they're the actual installed packages run through the --external seam (pip install headroom-ai, then one command). Register any other compressor the same way. "Distil leads on certified savings" is reproducible — and falsifiable. That's the point.
Reproduce it
# the bundled 7-domain standings, offline, zero API key distil benchmark # the 24-trajectory varied corpus above (from a repo clone) python benchmarks/gen_corpus.py distil benchmark --corpus benchmarks/corpus_xl --html standings.html # verify against a REAL external compressor (list[str] -> list[str] over block texts) distil benchmark --external mypkg.compressor:compress:MyTool # grade with the live model instead of the deterministic runner distil benchmark --runner anthropic --tokenizer anthropic
benchmarks/gen_corpus.py. ~80% is near the certified ceiling for this corpus: beyond it you start dropping decisions, and no honest tool can exceed that without lying — which is exactly what every disqualified row tried to do. For task-accuracy on a public benchmark (τ-bench, SWE-bench, GSM8K), ingest its traces into a corpus and run with --runner anthropic: the same comparison, graded by the live model. The harness is the deliverable; the corpus is swappable.