Distil vs everything else
Benchmark measures savings and decision-equivalence head-to-head; this page covers the other axis — what each tool actually guarantees, and what happens to real task success on a real harness. Competitors here are real, shipping tools, and the honest gap is narrower on savings than on proof.
Capability matrix
Marked from what each project documents or ships, not from marketing. Distil's own claims are the ones audited throughout this site — see Concepts and Deploy & Security for the code behind each ✔.
| Capability | Distil | Headroom | rtk | LLMLingua-2 | Provider caching |
|---|---|---|---|---|---|
| Per-step statistical certificate | ✔ DERC (Learn-Then-Test / CRC) | ✘ none published | ✘ none published | ✘ raw score deltas, not an equivalence test | — not a compressor |
| Trajectory-level (task-outcome) certificate | ✔ distil certify-trajectories |
✘ | ✘ | ✘ | — not a compressor |
| Reversible (bytes recoverable on demand) | ✔ Tier-1 digest + handle, distil_expand |
✔ documents a local store + headroom_retrieve; the compression itself is lossy in-context (the configuration benchmarked in E8) |
✘ lossy filtering (strips boilerplate; keeps raw logs on failure) | ✘ lossy | ✔ trivially — nothing is removed |
| Live shadow decision-equivalence on real traffic | ✔ --shadow / shadow-stats |
not published | not published | not published | — no decision risk to measure |
| SSE streaming pass-through (TTFT preserved) | ✔ chunk-by-chunk relay | n/a — library, not an HTTP relay | n/a — CLI output proxy, not an LLM API relay | n/a — library, not an HTTP relay | n/a — no proxy layer in the path |
| Savings ledger + status line | ✔ local ledger, leaderboard/statusline |
not published | not published | not published | — provider billing dashboard covers this natively |
| Multi-tenant gateway (per-tenant accounting) | ✔ distil gateway |
not published | not published | not published | — orthogonal, provider-side |
git, ls, psql, aws…) fits shell-heavy agents, which is why it's adopted there. It doesn't compress arbitrary agent context, so it isn't a head-to-head contender (see the note on Benchmark). LLMLingua-2 posts genuinely high raw compression ratios and is the most-cited method in this space for a reason. Provider prompt caching is zero-risk by construction — it doesn't touch what the model reads, only what you pay to re-send it — and composes cleanly with everything above, including Distil, whose cache-aware compression exists to keep that discount intact rather than compete with it.
Real task success — SWE-bench Verified, official harness
Capability tables can't settle whether any of this holds up on a real, long-horizon coding agent. E8 runs six conditions of the identical ReAct agent (only the compressor differs) end-to-end on the full 500-instance SWE-bench Verified set, scored by the official swebench harness (hidden tests, per-instance Docker) — not a proxy metric.
| Condition | Task success | Tied with full context? | Reversible + certified? |
|---|---|---|---|
| Distil (gated + surprise digest, v1.7) | 42.0% | ✔ +2.8pp over full (paired CI −0.6..+6.2pp) | ✔ |
| Distil (relevance-gated, E8) | 36.8% | ✔ | ✔ |
| Headroom (lossy) | 32.6% | ✘ −6.6 pp | ✘ |
| LLMLingua-2 (lossy) | 2.4% | ✘ −36.8 pp | ✘ |
| No compression (full) | 39.2% | — | — |
500-instance long-horizon ReAct agent, full SWE-bench Verified, official swebench harness (hidden tests, per-instance Docker), paired McNemar across all six conditions. Distil's gate is the only condition statistically non-inferior to full context (−2.4 pp, 95% CI [−5.7, +0.9], McNemar p=0.19) and beats Headroom with significance (+4.2 pp, p=0.035). rtk was not run in this harness — it does not compress arbitrary agent context (see above), so it has no comparable condition. Full methodology, CIs, and the recovery-round-trip ablation: Research → E8.
claude-haiku-4-5) — not a claim that Distil wins every workload (see E7 for the negative result that motivated the trajectory-level certificate). It's a claim that when someone measured end-to-end outcomes instead of raw compression ratios, the guarantee-carrying compressor led on the number that pays the bills.