Distil vs everything else

Benchmark measures savings and decision-equivalence head-to-head; this page covers the other axis — what each tool actually guarantees, and what happens to real task success on a real harness. Competitors here are real, shipping tools, and the honest gap is narrower on savings than on proof.

Capability matrix

Marked from what each project documents or ships, not from marketing. Distil's own claims are the ones audited throughout this site — see Concepts and Deploy & Security for the code behind each ✔.

Capability	Distil	Headroom	rtk	LLMLingua-2	Provider caching
Per-step statistical certificate	✔ DERC (Learn-Then-Test / CRC)	✘ none published	✘ none published	✘ raw score deltas, not an equivalence test	— not a compressor
Trajectory-level (task-outcome) certificate	✔ `distil certify-trajectories`	✘	✘	✘	— not a compressor
Reversible (bytes recoverable on demand)	✔ Tier-1 digest + handle, `distil_expand`	✔ documents a local store + `headroom_retrieve`; the compression itself is lossy in-context (the configuration benchmarked in E8)	✘ lossy filtering (strips boilerplate; keeps raw logs on failure)	✘ lossy	✔ trivially — nothing is removed
Live shadow decision-equivalence on real traffic	✔ `--shadow` / `shadow-stats`	not published	not published	not published	— no decision risk to measure
SSE streaming pass-through (TTFT preserved)	✔ chunk-by-chunk relay	n/a — library, not an HTTP relay	n/a — CLI output proxy, not an LLM API relay	n/a — library, not an HTTP relay	n/a — no proxy layer in the path
Savings ledger + status line	✔ local ledger, `leaderboard`/`statusline`	not published	not published	not published	— provider billing dashboard covers this natively
Multi-tenant gateway (per-tenant accounting)	✔ `distil gateway`	not published	not published	not published	— orthogonal, provider-side
Cache contract (compression may never rewrite already-cached bytes)	✔ suffix-only and cache-monotonic, enforced as an invariant — the contract	✔ genuinely strong prefix replay (`overlay_cached_prefix`, `prefix_tracker.py:456-463`) — real engineering	n/a — not an LLM API relay	✘ no prefix-stability mechanism — the compressed prompt changes as the conversation grows	✔ it is the cache
Adversarial validation (COMA-class, in CI)	✔ `distil validate --adversarial`, seven COMA-class cases; no keep budget shared between blocks, asserted as an equality — threat model	not published	not published	not published	— nothing is removed, so there is nothing to bait
Degradation curve across every rung (not one operating point)	✔ `distil bench --curve`, offline and free — the curve	not published	not published	not published	— no dial to sweep
Exact-quote guarantee (an `Edit` still matches the file that was read)	✔ provenance read from the shell command, not just the tool name; quote survival is a CI invariant	✘ not a property the tool has	n/a — strips known boilerplate, does not fold file reads	✘ lossy on all content alike	✔ trivially — nothing is removed
Paired live estimator that can report harm	✔ A/A′/B replay, unclipped difference with a bootstrap CI, one reporting floor on every surface	✘ no shadow or dual-send path in the codebase	— no decision risk to measure	✘ raw score deltas, not an equivalence test	— no decision risk to measure
Live alarm that acts (stops lossy compression on a proven budget breach)	✔ anytime-valid e-process over the paired verdicts; a breach holds every proxy at lossless-only until released (on by default, rc soak pending) — exit summary	✘ no shadow or dual-send path to feed one	— no decision risk to measure	not published	— no decision risk to measure
Audit log a third party can check one entry at a time	✔ hash-chained per-request receipts, sealed segments with Merkle-root checkpoints, inclusion proofs — `distil receipts`	not published	not published	not published	— provider billing records
Per-command shell output profiles (`git`, `pytest`, …)	✘ measured and deliberately not built: on real traffic the generic digest already takes most of it — ADR 0015	not published	✔ its core design	n/a	n/a

What each does well. Headroom is a real, adopted compressor with a useful whole-conversation optimizer and a ModernBERT relevance scorer — and on the live head-to-head, the cheaper compute path (Distil's path has no model either, but Headroom's scorer is lighter than a full digest cycle in some configs). rtk solves a different problem well: stripping known boilerplate from wrapped CLI commands (git, ls, psql, aws…) fits shell-heavy agents, which is why it's adopted there. It doesn't compress arbitrary agent context, so it isn't a head-to-head contender (see the note on Benchmark). We measured whether distil should grow rtk-style per-command profiles on top of its digest, on real transcripts with the provider's billed usage, and it should not: most shell volume is grep, scripts and sed, not the test runners and git that profiles target (ADR 0015). LLMLingua-2 posts genuinely high raw compression ratios and is the most-cited method in this space for a reason. Provider prompt caching is zero-risk by construction — it doesn't touch what the model reads, only what you pay to re-send it — and composes cleanly with everything above, including Distil, whose cache-aware compression exists to keep that discount intact rather than compete with it.

Field note — read the numbers honestly (2026). Headroom's headline is 60–95% (JSON) / 15–20% (coding), but its own published production telemetry shows a fleet-median compression of ~4.8% — the big ratios are best-case content, not what a typical agent session gets. Its accuracy claims (GSM8K, BFCL) are cited in the README but not reproduced on its own benchmarks page. And correctness is verified offline only (benchmark regression + Rust↔Python byte-parity) — there is no per-request check that the compressed request yields the same decision at runtime. Distil's wedge is exactly there: a live decision-equivalence gate on every request, savings calibrated to your billed usage, and a pure-Python install (Headroom's Rust core is mandatory — a missing native extension hard-exits). Where Headroom is genuinely strong: a real shipped model (Kompress-v2, ModernBERT/ONNX), prefix-freeze cache economics, its SmartCrusher JSON folder, and its multi-language CodeCompressor — all four Distil matches (learned salience; cache-aware compression; a reversible nested-JSON fold with constant-column collapse, ~62% on enum-heavy tool output; a zero-dependency code skeleton across JS/TS/Go/Rust/Java/C, ~44% on a TS file) without giving up the runtime proof its crusher can't offer — and Distil's code skeleton needs no native grammar, where a tree-sitter core would be one more mandatory native extension.

Headroom v0.37.0 — verified, as of v0.37.0, 2026-09-04. Read directly against the public headroom-ai 0.37.0 source (Apache-2.0); each line cites the file:line in that release so it can be checked against the tag.

No per-request behavioural check. There's no shadow or dual-send path anywhere in the codebase. accuracy_guard="strict" is echoed back on /healthz and /stats (server.py:3341,4686) but nothing branches on its value — setting it changes nothing.
Reversibility is a TTL cache, not a certificate. The retrieval store (CompressionStore, SQLite-backed) expires entries after 1800 seconds with a 1000-entry FIFO cap, and performs no integrity or round-trip check on what it hands back.
Lossy compression runs with no retrieval tool on four paths: OpenAI chat streaming (openai.py:1205), the Responses API under ChatGPT auth, Gemini streaming (gemini.py:682), and Bedrock, which has no retrieval path at all. Content removed on those paths is not recoverable.
Savings are uncalibrated. The reported number falls back to chars/3.5 (prefix_tracker.py:1195-1252) rather than a count reconciled against the provider's billed usage.*.
The shipped default bypasses compression. Default mode cache (server.py:5535) is a full bypass of compression on Bedrock and freeze-only (no compression) on OpenAI.
The one blocking CI eval doesn't check what it claims to. compression_only.py:66's docstring claims a byte-exact round-trip; the assertion at :112-114 is a case-insensitive substring scan.
Genuinely strong: prompt-cache preservation. overlay_cached_prefix (prefix_tracker.py:456-463) replays previously-forwarded bytes with conversation-lineage scoping — real engineering, unlike the no-op CacheAligner stage next to it. Breadth is real too: 20 wrap targets and 6 framework integrations shipped.

Fresh head-to-head, 2026-09-04

Re-run against headroom-ai 0.37.0 ([ml,code] extras), llmlingua 0.2.2, distil 1.51.1 (main), Python 3.12. Raw output and a versions note are committed at benchmarks/results/2026-09-04/.

`distil benchmark` corpus (9 domains, warm)	tokens	$	decision-equiv	gate
distil-causal	52.9%	58.7%	100%	✔ PASS
Headroom	1.7%	2.0%	81%	✘ FAIL

Warm run (Headroom's Kompress model preloaded before timing). A cold run instead shows a misleading Headroom 0.0% tokens / 100% PASS: its 274MB Kompress model loads in a background thread and fails open while loading, so the first requests pass through completely uncompressed and trivially "pass" the decision-equivalence check. Preload before comparing, or the cold number understates what it does and overstates its equivalence rate.

codebench (read→edit→reread, 20 sessions)	tokens	$ (cache-aware)
Headroom 0.37.0	35.6%	+4.9%
distil digest (PAYG)	0.0%^*	0.0%^*
distil cache-delta	34.9%	43.8%
distil verbatim	18.5%	3.7%

*By design, not a regression: this workload is read → edit → re-read, and once the agent has quoted a Read/Grep/Glob result byte-exact, that provenance is exempted from digest for the rest of the session — the exact-quote guarantee an Edit depends on. Headroom's own number here is a real improvement over the June measurement, kept on benchmark.html as a dated historical block: it moved from a net cost increase to a net decrease on a newer, non-ML code path, 2.4× faster per turn. benchmark.html's own headline table was refreshed to these same 2026-09-04 numbers; see docs/claims.json (s-benchmark-headline).

Real task success — SWE-bench Verified, official harness

Capability tables can't settle whether any of this holds up on a real, long-horizon coding agent. E8 runs six conditions of the identical ReAct agent (only the compressor differs) end-to-end on the full 500-instance SWE-bench Verified set, scored by the official swebench harness (hidden tests, per-instance Docker) — not a proxy metric.

Condition	Task success	Tied with full context?	Reversible + certified?
Distil (gated + surprise digest, v1.7)	42.0%	✔ tied with full (paired CI −0.6..+6.2pp)	✔
Distil (relevance-gated, E8)	36.8%	✔	✔
Headroom (lossy)	32.6%	✘ −6.6 pp	✘
LLMLingua-2 (lossy)	2.4%	✘ −36.8 pp	✘
No compression (full)	39.2%	—	—

500-instance long-horizon ReAct agent, full SWE-bench Verified, official swebench harness (hidden tests, per-instance Docker), paired McNemar across all six conditions. Distil's gate is the only condition statistically non-inferior to full context (−2.4 pp, 95% CI [−5.7, +0.9], McNemar p=0.19) and beats Headroom with significance (+4.2 pp, p=0.035). rtk was not run in this harness — it does not compress arbitrary agent context (see above), so it has no comparable condition. Full methodology, CIs, and the recovery-round-trip ablation: Research → E8.

Honest scope. This table is task success on one harness (SWE-bench Verified), one agent design, one base model (claude-haiku-4-5) — not a claim that Distil wins every workload (see E7 for the negative result that motivated the trajectory-level certificate). It's a claim that when someone measured end-to-end outcomes instead of raw compression ratios, the guarantee-carrying compressor led on the number that pays the bills.