Distil & Proof

Proving it didn't hurt

Every technique in Module 2 can degrade quality invisibly — the output still looks reasonable. So the load-bearing question isn't "how much did you save?" It's "how do you know the model still makes the same decision?" Three things to internalize:

• Decision-equivalence, not cosine similarity. Embedding similarity is invariant to exactly the small changes — a flipped negation, a changed digit, a swapped entity — that flip a discrete answer. Cosine ≈ 0.99 can still cross a decision boundary. The only sound test is whether the model's downstream output is the same. See Concepts → decision-equivalence.
• Non-inferiority, not a difference test. A standard "are they equal?" test can only fail to reject — absence of evidence is not evidence of absence, so it can never prove "as good as." The correct frame is TOST / non-inferiority (Schuirmann, 1987, imported from clinical trials): pre-declare a margin δ, and prove the compressed variant is not worse by more than δ. That yields a positive guarantee.
• This is rare. The headline compression papers report raw score deltas, not equivalence tests. Carrying a statistical non-inferiority certificate is the upgrade — and it's what Distil's quality contract does on every certified run.
• Step-level proof isn't the whole proof. Distil's own end-to-end test (E7) found that a statistically valid per-step certificate can pass while multi-turn task success collapses under aggressive lossy compression. So Distil certifies a second, coarser invariant too — distil certify-trajectories bounds how many solvable tasks compression may cost you, on matched full/compressed runs. See Concepts → the trajectory-level certificate.

How Distil applies it — in tiers

Distil's design choice is to apply the safe techniques aggressively, the risky ones only under proof, and to refuse anything it can't certify. That's the tier ladder: each rung saves more and risks more, and only ships if it clears the gate above it.

The pieces that make the ladder work — each documented in depth on Techniques (which also carries the full tier reference table):

• Cache-aware compression. Distil knows repeated tokens cost 0.1× and fresh tokens cost 1.0×, so it compresses to preserve the cacheable prefix and spends its byte budget where it matters. Naïve compression that rewrites the prefix can cost you more by busting the cache.
• Causal pruning. Rather than guessing importance from surprisal, Distil ablates context and checks whether the agent's decision changes — removing only what's provably inert.
• The keep-model codec (heuristic → logistic → transformer), trained on those causal labels, never-regressing because the gate has the final say.
• Input and output. Distil compresses the context going in and shapes the tool output coming back before it re-enters history — attacking both sides of the quadratic.
• Genuine measurement. Run it as a drop-in proxy and every real request's actual token reduction is accumulated into a local ledger — so distil leaderboard shows what you genuinely saved, not a benchmark.

# See the certified frontier and what's safe to take on your own corpus
distil eval
distil bench            # lossless + causal savings, with the non-inferiority verdict
distil proxy --port 8788 # drop-in; records genuine savings as real traffic flows
distil leaderboard      # your real cumulative savings (local, private)

A practical playbook

Independent of any tool, this is the order of operations that saves the most for the least risk:

Where the field is heading

• "Context engineering" replaced "prompt engineering." The discipline is now managing the entire token state against a finite attention budget. Anthropic's framing of context rot (quality degrading non-uniformly as the window fills) makes "fewer, higher-signal tokens" a quality strategy, not just a cost one.
• Agentic memory matured. Tiered/temporal memory systems (MemGPT/Letta, mem0, Zep) report >90% token savings versus stuffing full context, with comparable accuracy — compression reframed as memory.
• Prompt caching became table stakes across every major provider (≈50–90% discounts on cached input) — which is precisely why cache-aware compression beats naïve compression.
• Self-evolving context. Newer work (e.g. ACE — Agentic Context Engineering) improves agents by evolving the context rather than the weights, fighting "context collapse" from over-aggressive summarization — the same lesson as Distil's never-regress gate, at the agent level.
• Sub-quadratic models (Mamba and linear-attention) attack cost at the architecture layer — genuinely sub-quadratic, distinct from IO-aware exact attention like FlashAttention (still O(n²) compute). Orthogonal to, and composable with, token compression.

Further reading

Primary sources behind this course — verified, for the curious:

• Sennrich et al., Neural Machine Translation of Rare Words with Subword Units (BPE) — arXiv:1508.07909
• Jiang et al., LLMLingua (2310.05736) · LongLLMLingua (2310.06839) · Pan et al., LLMLingua-2 (2403.12968)
• Li et al., Selective Context (2310.06201) · Xu et al., RECOMP (2310.04408)
• Zhang et al., H2O (2306.14048) · Li et al., SnapKV (2404.14469) · Xiao et al., StreamingLLM (2309.17453)
• Mu et al., Gisting (2304.08467) · Chevalier et al., AutoCompressors (2305.14788) · Ge et al., ICAE (2307.06945)
• Hinton et al., Distilling the Knowledge in a Neural Network (1503.02531) · Chen et al., FrugalGPT (2305.05176) · Ong et al., RouteLLM (2406.18665)
• Packer et al., MemGPT (2310.08560) · Chhikara et al., mem0 (2504.19413) · Rasmussen et al., Zep (2501.13956)
• Dao et al., FlashAttention (2205.14135) · Gu & Dao, Mamba (2312.00752) · Zhang et al., ACE (2510.04618)
• Schuirmann, Two One-Sided Tests, J. Pharmacokinet. Biopharm. 15(6):657–680 (1987) · Anthropic, Effective context engineering for AI agents (2025)