The Benchmark
Every compression method shrinks tokens. The only number that matters is how much it saves without changing what the agent decides — and what it truly costs once prompt caching is priced in. This is a head-to-head on exactly that axis, where every technique runs through the same decision-equivalence gate and the same cache-aware cost model. The winner is computed, not assumed.
The standings
Ranked by cache-aware dollar savings on the varied corpus (see methodology below). Certified means the method passed a statistical non-inferiority test and preserved 100% of decisions; anything less is disqualified, however much it saved.
| Technique | Family | Tokens saved | $ saved | Decision equiv. | Verdict | Fidelity |
|---|---|---|---|---|---|---|
| distil-causal | cache-aware + causal pruning | 75.2% | 76.6% | 100% | ✔ certified · leader | lossy* |
| truncate-tail | sliding-window / truncation | 70.7% | 72.0% | 12% | ✘ fails gate | lossy |
| distil-lossless | cache-aware lossless + structured fold | 53.3% | 54.3% | 100% | ✔ certified | byte-exact |
| summarize | abstractive / rolling summary | 47.2% | 48.1% | 38% | ✘ fails gate | lossy |
| extractive-prune | extractive importance (LLMLingua family) | 14.0% | 14.3% | 75% | ✘ fails gate | lossy |
| minify-all | lossless minification | 0.1% | 0.1% | 100% | ✔ certified | byte-exact |
| baseline | control (no compression) | 0.0% | 0.0% | 100% | ✔ certified | byte-exact |
*distil-causal drops context that ablation proves never changed a decision — not byte-reversible, but certified decision-equivalent. distil-lossless is the byte-exact operating point — its 53% comes from reversible structured compaction (columnar-folding the JSON/tabular records agents traffic in) plus cache-aware digest, with the original always one expand() away. Numbers: claude-opus-4-8 pricing, deterministic runner. Reproduce below.
The corpus
To avoid any one tool's home-turf advantage, the comparison runs on 24 reproducible trajectories across 8 families deliberately spanning both regimes: structured/repetitive data (JSON record arrays, SQL rows, metrics, logs) where structural compaction pays off, and diagnostic/prose content (Kubernetes incidents, stack traces, RAG chunks, support transcripts) that conservative crushers protect and lossy methods mangle. Decisions are buried inside large tool outputs — as on real agents — so naive head/tail truncation drops them (12% equivalence), exactly as it would in production.
Why this is the honest comparison
No method is special-cased
Every technique — including Distil's own — is scored by the identical decision-equivalence + non-inferiority gate and the identical cache-aware cost model. The harness will happily rank a competitor above Distil if it earns it. It doesn't, because no other family combines lossless, cache-stability, causal pruning, and certification — but the door is open.
Best-form, not strawmen
The baselines are faithful reference implementations of the real technique families — sliding-window truncation, extractive importance pruning (the LLMLingua / Selective-Context lineage), abstractive summarization, naive minification — each in its best reasonable form. They genuinely remove tokens. They just can't prove they kept the decision.
Two ways to "win" that don't count
A method can post a big raw token cut yet flip decisions (truncate-tail: 71% saved, only 12% of decisions preserved). Or it can shave tokens yet bust the prompt cache and cost more in real dollars. The benchmark prices both, so neither illusion survives.
Plug in a real tool
Don't trust our reference baselines? Register any installed compressor through the external seam and it's measured on the identical axes. The claim "Distil leads on certified savings" is reproducible — and falsifiable. That's the point.
Reproduce it
# the bundled 7-domain standings, offline, zero API key distil benchmark # the 24-trajectory varied corpus above (from a repo clone) python benchmarks/gen_corpus.py distil benchmark --corpus benchmarks/corpus_xl --html standings.html # verify against a REAL external compressor (list[str] -> list[str] over block texts) distil benchmark --external mypkg.compressor:compress:MyTool # grade with the live model instead of the deterministic runner distil benchmark --runner anthropic --tokenizer anthropic
benchmarks/gen_corpus.py. 75% is near the certified ceiling for this corpus: beyond it you start dropping decisions, and no honest tool can exceed that without lying. For task-accuracy on a public benchmark (τ-bench, SWE-bench, GSM8K), ingest its traces into a corpus and run with --runner anthropic: the same comparison, graded by the live model. The harness is the deliverable; the corpus is swappable.