Token Economics · Module 3 of 3 ~5 min read

# Distil & Proof

In this module

- Why "it still looks fine" is not proof — and what is.
- Non-inferiority testing, borrowed from clinical trials.
- How Distil applies every technique in certified, risk-graded tiers.
- A practical playbook, and where the field is heading.

## Proving it didn't hurt

Every technique in Module 2 can quietly degrade quality, and the failure is often invisible — the output still looks reasonable. So the load-bearing question isn't "how much did you save?" It's "how do you know the model still makes the same decision?" Three things to internalize:

- Decision-equivalence, not cosine similarity. Embedding similarity is invariant to exactly the small changes — a flipped negation, a changed digit, a swapped entity — that flip a discrete answer. Cosine ≈ 0.99 can still cross a decision boundary. The only sound test is whether the model's downstream output is the same. See Concepts → decision-equivalence.
- Non-inferiority, not a difference test. A standard "are they equal?" test can only fail to reject — absence of evidence is not evidence of absence, so it can never prove "as good as." The correct frame is TOST / non-inferiority (Schuirmann, 1987, imported from clinical trials): pre-declare a margin δ, and prove the compressed variant is not worse by more than δ. That yields a positive guarantee.
- This is rare. The headline compression papers report raw score deltas, not equivalence tests. Carrying a statistical non-inferiority certificate is the upgrade — and it's what Distil's quality contract does on every certified run.

## How Distil applies it — in tiers

Distil's design choice is to apply the safe techniques aggressively, the risky ones only under proof, and to refuse anything it can't certify. That's the tier ladder. Each rung saves more and risks more — and a rung is only allowed to ship if it clears the gate above it.

The pieces that make the ladder work — each documented in depth on Techniques (which also carries the full tier reference table):

- Cache-aware compression. Distil knows repeated tokens cost 0.1× and fresh tokens cost 1.0×, so it compresses to preserve the cacheable prefix and spends its byte budget where it matters. Naïve compression that rewrites the prefix can cost you more by busting the cache.
- Causal pruning. Rather than guessing importance from surprisal, Distil ablates context and checks whether the agent's decision changes — removing only what's provably inert.
- The keep-model codec (heuristic → logistic → transformer), trained on those causal labels, never-regressing because the gate has the final say.
- Input and output. Distil compresses the context going in and shapes the tool output coming back before it re-enters history — attacking both sides of the quadratic.
- Genuine measurement. Run it as a drop-in proxy and every real request's actual token reduction is accumulated into a local ledger — so `distil leaderboard` shows what you genuinely saved, not a benchmark.

    # See the certified frontier and what's safe to take on your own corpus
    distil eval
    distil bench            # lossless + causal savings, with the non-inferiority verdict
    distil proxy --port 8788 # drop-in; records genuine savings as real traffic flows
    distil leaderboard      # your real cumulative savings (local, private)

## A practical playbook

Independent of any tool, this is the order of operations that saves the most for the least risk:

### Do first — free wins

- Stabilize your prefix. Put the system prompt and tool schemas first and keep them byte-identical; move anything volatile (timestamps, IDs) after the last cache breakpoint. This alone can 10× the discount on repeated tokens.
- Turn on prompt caching and verify hits via the cache-token counts.
- Minify structured payloads losslessly before they enter context.
- Prune dead tool definitions — every unused schema is paid on every turn.

### Then — measured wins

- Digest verbose tool output reversibly; let the agent expand on demand instead of carrying full dumps forever.
- Compact or summarize stale history — but treat summaries as lossy and keep recent turns verbatim.
- Prune causally, not by surprisal — remove what the decision doesn't depend on.
- Certify every aggressive step with a non-inferiority test; if it can't pass, don't ship it.
- Measure genuine savings on real traffic, not a synthetic corpus.

## Where the field is heading

- "Context engineering" replaced "prompt engineering." The discipline is now managing the entire token state against a finite attention budget. Anthropic's framing of context rot (quality degrading non-uniformly as the window fills) makes "fewer, higher-signal tokens" a quality strategy, not just a cost one.
- Agentic memory matured. Tiered/temporal memory systems (MemGPT/Letta, mem0, Zep) report >90% token savings versus stuffing full context, with comparable accuracy — compression reframed as memory.
- Prompt caching became table stakes across every major provider (≈50–90% discounts on cached input) — which is precisely why cache-aware compression beats naïve compression.
- Self-evolving context. Newer work (e.g. ACE — Agentic Context Engineering) improves agents by evolving the context rather than the weights, fighting "context collapse" from over-aggressive summarization — the same lesson as Distil's never-regress gate, at the agent level.
- Sub-quadratic models (Mamba and linear-attention) attack cost at the architecture layer — genuinely sub-quadratic, distinct from IO-aware exact attention like FlashAttention (still O(n²) compute). Orthogonal to, and composable with, token compression.

You've finished the course. The throughline of every advance in this field is one move: find the tokens the outcome doesn't depend on, and stop paying for them — without changing the outcome. Distil makes that last clause a proof, applied in tiers, measured on your real traffic. Next: Install & Quickstart, then go deeper in Concepts and Techniques.

## Further reading

Primary sources behind this course — verified, for the curious:

- Sennrich et al., Neural Machine Translation of Rare Words with Subword Units (BPE) — arXiv:1508.07909
- Jiang et al., LLMLingua (2310.05736) · LongLLMLingua (2310.06839) · Pan et al., LLMLingua-2 (2403.12968)
- Li et al., Selective Context (2310.06201) · Xu et al., RECOMP (2310.04408)
- Zhang et al., H2O (2306.14048) · Li et al., SnapKV (2404.14469) · Xiao et al., StreamingLLM (2309.17453)
- Mu et al., Gisting (2304.08467) · Chevalier et al., AutoCompressors (2305.14788) · Ge et al., ICAE (2307.06945)
- Hinton et al., Distilling the Knowledge in a Neural Network (1503.02531) · Chen et al., FrugalGPT (2305.05176) · Ong et al., RouteLLM (2406.18665)
- Packer et al., MemGPT (2310.08560) · Chhikara et al., mem0 (2504.19413) · Rasmussen et al., Zep (2501.13956)
- Dao et al., FlashAttention (2205.14135) · Gu & Dao, Mamba (2312.00752) · Zhang et al., ACE (2510.04618)
- Schuirmann, Two One-Sided Tests, J. Pharmacokinet. Biopharm. 15(6):657–680 (1987) · Anthropic, Effective context engineering for AI agents (2025)

← Previous Module 2 · Cost & Compression Finish →Install & Quickstart