The Benchmark
Every compression method shrinks tokens. The only number that matters is how much it saves without changing what the agent decides — and what it truly costs once prompt caching is priced in. This is a head-to-head on exactly that axis, where every technique runs through the same decision-equivalence gate and the same cache-aware cost model. The winner is computed, not assumed.
The standings
Ranked by cache-aware dollar savings. Certified means the method passed a statistical non-inferiority test and preserved 100% of decisions; anything less is disqualified, however much it saved.
| Technique | Family | Tokens saved | $ saved | Decision equiv. | Verdict | Fidelity |
|---|---|---|---|---|---|---|
| distil-causal | cache-aware + causal pruning | 33.8% | 37.4% | 100% | ✔ certified · leader | lossy* |
| truncate-tail | sliding-window / truncation | 26.9% | 33.0% | 25% | ✘ fails gate | lossy |
| summarize | abstractive / rolling summary | 19.7% | 24.2% | 0% | ✘ fails gate | lossy |
| distil-lossless | cache-aware lossless | 8.4% | 10.3% | 100% | ✔ certified | byte-exact |
| extractive-prune | extractive importance (LLMLingua family) | 7.4% | 9.0% | 68% | ✘ fails gate | lossy |
| minify-all | lossless minification | 0.0% | 0.0% | 100% | ✔ certified | byte-exact |
| baseline | control (no compression) | 0.0% | 0.0% | 100% | ✔ certified | byte-exact |
*distil-causal drops context that ablation proves never changed a decision — so it isn't byte-reversible, but it is certified decision-equivalent. distil-lossless is the byte-exact operating point. Numbers: claude-opus-4-8 pricing, deterministic runner, bundled corpus — reproduce them yourself below.
Why this is the honest comparison
No method is special-cased
Every technique — including Distil's own — is scored by the identical decision-equivalence + non-inferiority gate and the identical cache-aware cost model. The harness will happily rank a competitor above Distil if it earns it. It doesn't, because no other family combines lossless, cache-stability, causal pruning, and certification — but the door is open.
Best-form, not strawmen
The baselines are faithful reference implementations of the real technique families — sliding-window truncation, extractive importance pruning (the LLMLingua / Selective-Context lineage), abstractive summarization, naive minification — each in its best reasonable form. They genuinely remove tokens. They just can't prove they kept the decision.
Two ways to "win" that don't count
A method can post a big raw token cut yet flip decisions (truncate-tail: 27% saved, only 25% of decisions preserved). Or it can shave tokens yet bust the prompt cache and cost more in real dollars. The benchmark prices both, so neither illusion survives.
Plug in a real tool
Don't trust our reference baselines? Register any installed compressor through the external seam and it's measured on the identical axes. The claim "Distil leads on certified savings" is reproducible — and falsifiable. That's the point.
Reproduce it
# the full standings, offline, zero API key distil benchmark # render the comparison as a shareable page distil benchmark --html standings.html # verify against a REAL external compressor (list[str] -> list[str] over block texts) distil benchmark --external mypkg.compressor:compress:MyTool # grade with the live model instead of the deterministic runner distil benchmark --runner anthropic --tokenizer anthropic
--runner anthropic: the same comparison, graded by the live model. The harness is the deliverable; the corpus is swappable.