Token Economics

Before you reach for a compression tool, understand the thing it compresses. This is the whole story — what a token is, where an agent quietly burns thousands of them, why multi-turn loops get expensive faster than you'd guess, every family of compression that has been invented to fight back, and exactly where Distil fits — applied in tiers, certified to change nothing the model decides.

1. What a token actually is

A token is not a character and not a word. It is a subword unit — the atomic chunk a model reads and predicts. Modern models build their vocabulary with Byte-Pair Encoding (BPE): start from individual bytes, then greedily merge the most frequent adjacent pair over and over until you hit a target vocabulary size. Frequent words collapse into one token; rare words fall back to a handful of subword pieces. There is no <unknown> — every possible string is representable, in any language.

Three consequences fall straight out of how BPE works, and they drive everything below:

• Structured text is expensive. Prose lands near ~4 characters per token; JSON, logs and code are roughly 2× denser because punctuation, indentation and snake_case/camelCase boundaries break merges into many short tokens. Agents traffic almost entirely in structured text — tool calls, file contents, stack traces — so they live in the expensive regime.
• Tokenizers are not interchangeable. Every model family segments differently. You cannot estimate one model's tokens with another's tokenizer — and even within a family it can shift: Anthropic's newer tokenizer (introduced with Opus 4.7) produces roughly 30% more tokens than older models for the same text, and billing follows the new count. Always count with the target model's tokenizer.
• Tokens are the unit of both the bill and the budget. The context window is denominated in tokens; price is per-token; and output costs several times more than input (Opus 4.8: $5/M in, $25/M out — a 5× ratio). Input dominates agent bills anyway, for the reason in §3.

2. Where the tokens go — anatomy of one agent request

Here is the part most cost surprises come from. A single agent turn is not "the user's question." The API is stateless, so every turn re-sends the entire working set. A typical request, before the model has read a single word of your actual task, already carries:

The biggest, most-overlooked sources of waste, with the hardest public numbers we have:

• Tool definitions, repeated every turn. Anthropic has reported tool definitions consuming 134K tokens before optimization, and a typical five-MCP-server setup spending ~55K tokens before the conversation even starts. This is paid on every single turn.
• Tool-result bloat. Logs, JSON dumps, file contents and search results get appended to history and replayed on every subsequent turn, long after they're stale. Keeping intermediate results out of context cut one Anthropic research workload from 43,588 → 27,297 tokens (−37%).
• History re-send. The whole transcript goes back up the wire each turn — the single largest growing cost, and the reason for §3.
• Redundant retrieval. RAG often injects overlapping, low-signal chunks; context is a finite attention budget with diminishing returns, so more tokens can mean worse answers, not just costlier ones.

3. The multi-turn tax: why cost is quadratic

Single-turn cost is linear and boring. Multi-turn is where bills explode — and the shape is quadratic, not linear. Because each turn re-bills all the tokens before it, the cumulative cost of an n-turn loop is the sum 1 + 2 + … + n = n(n+1)/2 = O(n²). A four-turn conversation costs 4 + 3 + 2 + 1 = 10× the first turn's tokens — not 4×.

Two more mechanics make this concrete:

• Prefill vs decode. Reading the prompt (prefill) is compute-bound and parallel; generating output (decode) is memory-bandwidth-bound and sequential — every output token re-reads the whole model and KV cache from memory. That's why output is priced 3–10× higher than input, and why a cache hit (which skips prefill compute) can discount input so aggressively.
• Prompt caching rewrites the coefficient, not the shape. If the prefix (system prompt + tools + frozen early history) is byte-stable, you write it once and read it every later turn. Anthropic's documented model: a 5-minute cache write costs 1.25× base input, a read costs 0.10× — a 10× gap. The curve is still quadratic; caching just shrinks the dominant term's coefficient by roughly 10×. Any byte change anywhere in the prefix invalidates everything after it — a datetime.now() near the top, or editing a tool definition, silently throws the cache away.

Try it: the cost of a loop

Real arithmetic — no marketing. This computes the n(n+1)/2 input cost from your inputs, using live catalog pricing and the documented cache multipliers (0.1× read / 1.25× write). The "with Distil" line applies a flat lossless reduction to fresh tokens; the default 26.8% is Distil's measured corpus aggregate (see Research). Assumptions are shown so you can check the math.

Assumptions: input-token cost only (output excluded for clarity); prefix is cacheable and byte-stable; per-turn input = prefix + accumulated history; cached line writes the prefix once at 1.25× then reads it at 0.10× while the growing tail stays fresh; Distil line additionally removes 26.8% of fresh tokens losslessly. Illustrative — your numbers depend on your traffic.

4. Compression, the whole family tree

"Compression" is an umbrella over a dozen genuinely different ideas, each trading a different amount of quality risk for savings. The single most common mistake in this space is conflating them — input-token compression (fewer tokens enter the model), KV-cache compression (the prompt is read in full; the retained attention memory shrinks), and cost reduction (same tokens, cheaper compute) are three different axes. Here is the honest map.

5. Why training a small model helps

The most effective compressors stopped using fixed heuristics and started learning. The root idea is knowledge distillation (Hinton et al., 2015): train a small "student" to imitate a large "teacher," using the teacher's soft probabilities — which carry far richer signal than a one-hot label.

Applied to compression, the move is to reframe it as per-token keep/drop classification. LLMLingua-2 is the clean example: take a small bidirectional encoder (XLM-RoBERTa, ~355M params), and train it on labels distilled from GPT-4 — prompt GPT-4 to compress text by word-removal only, then mark each original token "keep" or "drop." A trained classifier beats a perplexity heuristic for three concrete reasons:

• It sees both directions. Perplexity is computed left-to-right by a causal model; a bidirectional encoder uses full left-and-right context to judge importance.
• It learns task/domain importance and outputs calibrated keep probabilities, instead of a fixed, task-blind proxy.
• It's faster — one encoder pass replaces an autoregressive scoring loop (reported 3–6× faster), and the win is largest out-of-domain.

Distil takes this learning seriously but conservatively: it ships a built-in heuristic keep-model, a zero-dependency logistic keep-model, and an optional ONNX transformer keep-model you can train on your own traces — and critically, it trains on causal labels (what actually changed the agent's decision under ablation), then lets a model graduate only if it passes the non-inferiority gate. A keep-model can make Distil better; it can never make it worse.

6. Proving it didn't hurt

Every technique above can quietly degrade quality, and the failure is often invisible — the output still looks reasonable. So the load-bearing question isn't "how much did you save?" It's "how do you know the model still makes the same decision?" Three things to internalize:

• Decision-equivalence, not cosine similarity. Embedding similarity is invariant to exactly the small changes — a flipped negation, a changed digit, a swapped entity — that flip a discrete answer. Cosine ≈ 0.99 can still cross a decision boundary. The only sound test is whether the model's downstream output is the same. See Concepts → decision-equivalence.
• Non-inferiority, not a difference test. A standard "are they equal?" test can only fail to reject — absence of evidence is not evidence of absence, so it can never prove "as good as." The correct frame is TOST / non-inferiority (Schuirmann, 1987, imported from clinical trials): pre-declare a margin δ, and prove the compressed variant is not worse by more than δ. That yields a positive guarantee.
• This is rare. The headline compression papers report raw score deltas, not equivalence tests. Carrying a statistical non-inferiority certificate is the upgrade — and it's what Distil's quality contract does on every certified run.

7. How Distil applies all of this — in tiers

Distil's design choice is to apply the safe techniques aggressively, the risky ones only under proof, and to refuse anything it can't certify. That's the tier ladder. Each rung saves more and risks more — and a rung is only allowed to ship if it clears the gate above it.

The pieces that make the ladder work — each documented in depth on Techniques:

• Cache-aware compression. Distil knows repeated tokens cost 0.1× and fresh tokens cost 1.0×, so it compresses to preserve the cacheable prefix and spends its byte budget where it actually matters. Naïve compression that rewrites the prefix can cost you more by busting the cache.
• Causal pruning. Rather than guessing importance from surprisal, Distil ablates context and checks whether the agent's decision changes — removing only what's provably inert.
• The keep-model codec (heuristic → logistic → transformer), trained on those causal labels, never-regressing because the gate has the final say.
• Input and output. Distil compresses the context going in and shapes the tool output coming back before it re-enters history — attacking both sides of the quadratic.
• Genuine measurement. Run it as a drop-in proxy and every real request's actual token reduction is accumulated into a local ledger — so distil leaderboard shows what you genuinely saved, not a benchmark.

# See the certified frontier and what's safe to take on your own corpus
distil eval
distil bench            # lossless + causal savings, with the non-inferiority verdict
distil proxy --port 8788 # drop-in; records genuine savings as real traffic flows
distil leaderboard      # your real cumulative savings (local, private)

8. A practical playbook

Independent of any tool, this is the order of operations that saves the most for the least risk:

9. Where the field is heading

The frontier as of 2025–2026, in one breath:

• "Context engineering" replaced "prompt engineering." The discipline is now managing the entire token state — system prompt, tools, history, retrieval — against a finite attention budget. Anthropic's framing of context rot (quality degrading non-uniformly as the window fills) makes "fewer, higher-signal tokens" a quality strategy, not just a cost one.
• Agentic memory matured. Tiered/temporal memory systems (MemGPT/Letta, mem0, Zep) report >90% token savings versus stuffing full context, with comparable accuracy — compression reframed as memory.
• Prompt caching became table stakes across every major provider (≈50–90% discounts on cached input) — which is precisely why cache-aware compression beats naïve compression.
• Self-evolving context. Newer work (e.g. ACE — Agentic Context Engineering) improves agents by evolving the context rather than the weights, fighting "context collapse" from over-aggressive summarization — the same lesson as Distil's never-regress gate, at the agent level.
• Sub-quadratic models (Mamba and linear-attention) attack the cost at the architecture layer — genuinely sub-quadratic, distinct from IO-aware exact attention like FlashAttention (still O(n²) compute). Orthogonal to, and composable with, token compression.

Further reading

Primary sources behind this page — verified, for the curious:

• Sennrich et al., Neural Machine Translation of Rare Words with Subword Units (BPE) — arXiv:1508.07909
• Jiang et al., LLMLingua (2310.05736) · LongLLMLingua (2310.06839) · Pan et al., LLMLingua-2 (2403.12968)
• Li et al., Selective Context (2310.06201) · Xu et al., RECOMP (2310.04408)
• Zhang et al., H2O (2306.14048) · Li et al., SnapKV (2404.14469) · Xiao et al., StreamingLLM (2309.17453)
• Mu et al., Gisting (2304.08467) · Chevalier et al., AutoCompressors (2305.14788) · Ge et al., ICAE (2307.06945)
• Hinton et al., Distilling the Knowledge in a Neural Network (1503.02531) · Chen et al., FrugalGPT (2305.05176) · Ong et al., RouteLLM (2406.18665)
• Packer et al., MemGPT (2310.08560) · Chhikara et al., mem0 (2504.19413) · Rasmussen et al., Zep (2501.13956)
• Dao et al., FlashAttention (2205.14135) · Gu & Dao, Mamba (2312.00752) · Zhang et al., ACE (2510.04618)
• Schuirmann, Two One-Sided Tests, J. Pharmacokinet. Biopharm. 15(6):657–680 (1987) · Anthropic, Effective context engineering for AI agents (2025)