Every other compressor trusts.
Distil proves.
- Drops tokens and hopes the agent still decides the same
- “Looks fine” on a demo — no proof on your traffic
- You discover the regression in production
- Certifies decision-equivalence before it ever serves
- A shadow gate re-checks live decisions at temperature 0
- Fails closed — if quality would drop, it doesn't compress
How it works — in plain English#
No jargon. Here's the whole idea in three pictures.
Don't re-send what's already remembered
Every turn, your agent re-sends the entire conversation — and you pay for all of it, again. The model can cheaply "remember" the parts that haven't changed, so Distil keeps those parts perfectly stable and only touches the new bits. Like not re-reading someone the whole book to add one sentence.
Drop only what doesn't matter
Most of that context never changes what the agent decides to do next. Distil finds those dead-weight parts with a simple test — remove it, replay, did the decision change? If no, it was safe to cut. Keep the clues, drop the filler.
Prove it — don't just trust it
Here's the part nobody else does: Distil measures that your agent still makes the same choice on the shrunk context. If it can't prove that, it sends everything, untouched. A warranty on the compression, not a promise.
And nothing is truly thrown away — trimmed detail is filed nearby, and the agent can pull back the exact original the moment it needs it. That's why Distil can compress hard and stay safe. The gentle, longer walkthrough →
Under the hood — a cost-optimized cache hierarchy with a contract#
Most compressors optimize bytes. In an agent loop the money is somewhere else.
Two highest-leverage techniques — and why they win
Cache-aware compression
You re-send the growing context every turn. With prompt caching a cache read is ~10× cheaper than fresh input — so the dominant cost is cache misses, not context size. Distil keeps the prefix byte-stable and compresses only the volatile tail. Naive recompression sends fewer tokens yet costs more, because it rewrites the cached prefix every turn.
Causal / counterfactual pruning
The eval engine isn't a ruler — it's a discovery engine. Remove a context block, replay, did any decision change? Blocks that never change a decision are provably free to drop: speculative retrievals, stale history. The measurement produces the compression policy.
The proof — a quality contract, not an estimate#
We didn't just benchmark ourselves. We ran the real LLMLingua-2 and Headroom packages through the same gate, graded live by the real model.
Proxy caveat: this is next-action equivalence on a (partly synthetically decision-determined) corpus — not end-to-end task success. On real SWE-bench Verified (E7) aggressive compression drops pass@1 52%→16%; the proxy certificate does not transfer. E8 (500-instance long-horizon ReAct agent, 6 conditions, full 500-instance SWE-bench Verified) shows relevance-gated reversible compression is non-inferior to full context (−2.4 pp, 95% CI [−5.7, +0.9]; McNemar p=0.19), beats Headroom by +4.2 pp (p=0.035), and beats LLMLingua-2 by 34 points (36.8% vs 2.4% pass@1, p<0.001) at nearly identical context-reduction ratios. The newest digest (E14) keeps this on par with full context — 42.0% vs 39.2% (paired CI −0.6..+6.2pp; non-inferiority certified, superiority not yet significant). Distil is the only certified and reversible compressor in the comparison; it does not claim cheapest.
Underneath it: a strategy ships only if a pre-registered non-inferiority test certifies it. Lossless passes; quality-degrading compression is rejected.
$ distil savings --pricing claude-opus-4-8 strategy $ / run vs baseline cache hits ------------------------------------------------------------------------ baseline (no cache, no compress) 0.01524 0.0% 0 cache only 0.01115 26.8% 1,028 naive compress + cache 0.01691 -11.0% 0 ← busts cache distil (cache-aware lossless) 0.01019 33.1% 1,028 $ distil certify --strategy distil decision-equivalence match rate: 100.0% VERDICT: PASS (certified non-inferior) $ distil certify --strategy aggressive decision-equivalence match rate: 0.0% VERDICT: FAIL (would degrade quality — blocked)
Measured across 7 domains — the same gate, not one example#
A strategy isn't trustworthy because it works once. distil bench certifies
every domain — ops, coding, support, research, data, devops, finance — or it doesn't ship.
Risk-graded tiers#
Apply the provably-safe ones everywhere; gate the rest on evidence.
Provably lossless
Reconstructable transforms — JSON minify, reversible run-length encoding. Always on.
Reversible digest
Decision-aware digest + a handle; the full original stays local and re-expands on demand.
Lossy, but gated
Pruning & summarization allowed only at ratios the non-inferiority gate certifies.
Capabilities — every layer of the cost stack#
| Capability | What it does | Loss profile |
|---|---|---|
| Priced cache-aware engine | Models the multi-turn loop and proves naive recompression busts the cache | — |
| Schema canonicalization | Recursively key-sorts JSON/tool payloads so the prefix is byte-stable | lossless |
| Volatile-field extraction | Lifts dates/UUIDs/JWTs out of the cached prefix so it stops churning | lossless · reversible |
| Reversible digest + handles | Originals stay local and re-expand on demand | reversible |
| Reject-if-bigger invariant | Never emits a block larger than its original | safety |
| Adversarial validation gate | distil validate — drives the real compressor against hostile inputs and asserts reversibility, recency-exactness, fail-open, and content-free telemetry hold | CI gate |
| Causal / counterfactual pruning | Ablation discovers context that never changes a decision | certified |
| TOST non-inferiority gate (DERC) | Per-step: a strategy ships only if it passes the quality contract | the moat |
| Trajectory-level certificate | Task-level: distil certify-trajectories bounds end-to-end degradation on matched full/compressed runs | the differentiator |
| Savings ledger + leaderboard | Local-first, privacy-preserving cumulative savings | opt-in |
| Content-type keep policy | Pins each kind's load-bearing lines — a log's pass/fail verdict, a traceback's frames, a diff's hunk headers | 1.15 |
| Query-aware salience | Sees the agent's intent in the same request and pins the line it's asking about — lexically (a grep hit, a SHA), and now semantically: a learned model keeps the line that answers the query with no shared word ("retry limit" → max_attempts = 5) | 1.16 |
| Columnar fold | Re-encodes JSON record arrays into a compact self-describing table — real token savings even on a flat-rate plan | lossless |
| Session dissect | distil dissect — per-session deep-dive with an anomaly list that catches silent failures automatically | 1.15 |
Works with every SDK — one proxy, no code change#
Point any base_url-honoring client at the proxy — Python, TypeScript, any language —
and get cache-aware lossless compression. See integrations →
Install — pick a format#
The stdlib-only core makes the packaging clean: zero runtime deps, corpus bundled.
uvx — zero install, recommended. Runs straight from PyPI, nothing to clean up.
uvx --from distil-llm distil bench
Run straight from PyPI — nothing installed. Needs uv.
pipx install distil-llm distil certify
Permanent, isolated venv — PEP 668-safe. brew install pipx if needed.
brew install dshakes/tap/distil distil bench
Isolated keg — doesn't touch your system Python.
docker build -t distil . docker run distil bench
Container image — reproducible, zero host deps.
make pyz python dist/distil.pyz bench
Single file — copy distil.pyz anywhere with Python 3.9+.
Already installed? Do this next → distil onboard detects your agent + billing, wires the savings status line, and gives you a guided tour tailored to your setup — then distil default makes it the default for every session (distil offboard reverses that).
pip install is blocked by PEP 668 on modern macOS/Linux, and that's fine). distil onboard wires your agent and the savings status line; distil default makes it the default and distil offboard fully reverses it — nothing is touched that isn't put back. Node/other stacks just point their SDK baseURL at distil proxy. Verify anytime: distil bench → expect GATE: PASS.
from distil.adapters.anthropic import wrap client = wrap(anthropic.Anthropic()) # compresses + keeps the cache warm
What we won't pretend#
--tokenizer anthropic and
--runner anthropic make it billing-grade against the real model.
Compress the context.
Keep the decisions.
One proxy in front of any agent. A signed decision-equivalence certificate on the way out — or it doesn't compress. No API key to try it.