Distil & Proof

Proving it didn't hurt

Every technique in Module 2 can quietly degrade quality, and the failure is often invisible — the output still looks reasonable. So the load-bearing question isn't "how much did you save?" It's "how do you know the model still makes the same decision?" Three things to internalize:

• Decision-equivalence, not cosine similarity. Embedding similarity is invariant to exactly the small changes — a flipped negation, a changed digit, a swapped entity — that flip a discrete answer. Cosine ≈ 0.99 can still cross a decision boundary. The only sound test is whether the model's downstream output is the same. See Concepts → decision-equivalence.
• Non-inferiority, not a difference test. A standard "are they equal?" test can only fail to reject — absence of evidence is not evidence of absence, so it can never prove "as good as." The correct frame is TOST / non-inferiority (Schuirmann, 1987, imported from clinical trials): pre-declare a margin δ, and prove the compressed variant is not worse by more than δ. That yields a positive guarantee.
• This is rare. The headline compression papers report raw score deltas, not equivalence tests. Carrying a statistical non-inferiority certificate is the upgrade — and it's what Distil's quality contract does on every certified run.
• Step-level proof isn't the whole proof. Distil's own end-to-end test (E7) found that a statistically valid per-step certificate can pass while multi-turn task success collapses under aggressive lossy compression. So Distil certifies a second, coarser invariant too — distil certify-trajectories bounds how many solvable tasks compression may cost you, on matched full/compressed runs. See Concepts → the trajectory-level certificate.

How Distil applies it — in tiers

Distil's design choice is to apply the safe techniques aggressively, the risky ones only under proof, and to refuse anything it can't certify. That's the tier ladder. Each rung saves more and risks more — and a rung is only allowed to ship if it clears the gate above it.

The pieces that make the ladder work — each documented in depth on Techniques (which also carries the full tier reference table):

• Cache-aware compression. Distil knows repeated tokens cost 0.1× and fresh tokens cost 1.0×, so it compresses to preserve the cacheable prefix and spends its byte budget where it matters. Naïve compression that rewrites the prefix can cost you more by busting the cache.
• Causal pruning. Rather than guessing importance from surprisal, Distil ablates context and checks whether the agent's decision changes — removing only what's provably inert.
• The keep-model codec (heuristic → logistic → transformer), trained on those causal labels, never-regressing because the gate has the final say.
• Input and output. Distil compresses the context going in and shapes the tool output coming back before it re-enters history — attacking both sides of the quadratic.
• Genuine measurement. Run it as a drop-in proxy and every real request's actual token reduction is accumulated into a local ledger — so distil leaderboard shows what you genuinely saved, not a benchmark.

# See the certified frontier and what's safe to take on your own corpus
distil eval
distil bench            # lossless + causal savings, with the non-inferiority verdict
distil proxy --port 8788 # drop-in; records genuine savings as real traffic flows
distil leaderboard      # your real cumulative savings (local, private)

A practical playbook

Independent of any tool, this is the order of operations that saves the most for the least risk:

Where the field is heading

• "Context engineering" replaced "prompt engineering." The discipline is now managing the entire token state against a finite attention budget. Anthropic's framing of context rot (quality degrading non-uniformly as the window fills) makes "fewer, higher-signal tokens" a quality strategy, not just a cost one.
• Agentic memory matured. Tiered/temporal memory systems (MemGPT/Letta, mem0, Zep) report >90% token savings versus stuffing full context, with comparable accuracy — compression reframed as memory.
• Prompt caching became table stakes across every major provider (≈50–90% discounts on cached input) — which is precisely why cache-aware compression beats naïve compression.
• Self-evolving context. Newer work (e.g. ACE — Agentic Context Engineering) improves agents by evolving the context rather than the weights, fighting "context collapse" from over-aggressive summarization — the same lesson as Distil's never-regress gate, at the agent level.
• Sub-quadratic models (Mamba and linear-attention) attack cost at the architecture layer — genuinely sub-quadratic, distinct from IO-aware exact attention like FlashAttention (still O(n²) compute). Orthogonal to, and composable with, token compression.

Further reading

Primary sources behind this course — verified, for the curious:

• Sennrich et al., Neural Machine Translation of Rare Words with Subword Units (BPE) — arXiv:1508.07909
• Jiang et al., LLMLingua (2310.05736) · LongLLMLingua (2310.06839) · Pan et al., LLMLingua-2 (2403.12968)
• Li et al., Selective Context (2310.06201) · Xu et al., RECOMP (2310.04408)
• Zhang et al., H2O (2306.14048) · Li et al., SnapKV (2404.14469) · Xiao et al., StreamingLLM (2309.17453)
• Mu et al., Gisting (2304.08467) · Chevalier et al., AutoCompressors (2305.14788) · Ge et al., ICAE (2307.06945)
• Hinton et al., Distilling the Knowledge in a Neural Network (1503.02531) · Chen et al., FrugalGPT (2305.05176) · Ong et al., RouteLLM (2406.18665)
• Packer et al., MemGPT (2310.08560) · Chhikara et al., mem0 (2504.19413) · Rasmussen et al., Zep (2501.13956)
• Dao et al., FlashAttention (2205.14135) · Gu & Dao, Mamba (2312.00752) · Zhang et al., ACE (2510.04618)
• Schuirmann, Two One-Sided Tests, J. Pharmacokinet. Biopharm. 15(6):657–680 (1987) · Anthropic, Effective context engineering for AI agents (2025)