Token Economics · Module 2 of 3 ~7 min read

Cost & Compression

In this module

Why multi-turn cost grows with the square of the conversation.
How prompt caching rewrites the math — and what silently breaks it.
The full family tree of compression, mapped by risk vs. savings.
Why a trained compressor beats a heuristic — and how it fails.

The multi-turn tax: why cost is quadratic

Single-turn cost is linear and boring. Multi-turn is where bills explode — and the shape is quadratic, not linear. Because each turn re-bills all the tokens before it, the cumulative cost of an n-turn loop is the sum 1 + 2 + … + n = n(n+1)/2 = O(n²). A four-turn conversation costs 4 + 3 + 2 + 1 = 10× the first turn's tokens — not 4×.

Prefill vs decode. Reading the prompt (prefill) is compute-bound and parallel; generating output (decode) is memory-bandwidth-bound and sequential — every output token re-reads the whole model and KV cache from memory. That's why output is priced 3–10× higher than input, and why a cache hit (which skips prefill compute) can discount input so aggressively.
Prompt caching rewrites the coefficient, not the shape. If the prefix (system prompt + tools + frozen early history) is byte-stable, you write it once and read it every later turn. Anthropic's documented model: a 5-minute cache write costs 1.25× base input, a read costs 0.10× — a 10× gap. The curve stays quadratic; caching just shrinks the dominant term's coefficient by ~10×. Any byte change anywhere in the prefix invalidates everything after it — a datetime.now() near the top, or editing a tool definition, silently throws the cache away.

This is the foundation of Distil's approach — see Concepts → cache misses dominate cost for the measured numbers behind it.

Try it: the cost of a loop

Real arithmetic — no marketing. This computes the n(n+1)/2 input cost from your inputs, using live catalog pricing and the documented cache multipliers (0.1× read / 1.25× write). The "with Distil" line applies a flat lossless reduction to fresh tokens; the default 26.8% is Distil's measured corpus aggregate (see Research). Assumptions are shown so you can check the math.

Turns 12 Fixed prefix (system + tools), tokens 40,000 New tokens added per turn 3,000 Model

Naïve — no caching, no compression

With prompt caching (prefix read @ 0.1×)

With caching + Distil lossless (−26.8%)

Assumptions: input-token cost only (output excluded for clarity); prefix is cacheable and byte-stable; per-turn input = prefix + accumulated history; cached line writes the prefix once at 1.25× then reads it at 0.10× while the growing tail stays fresh; Distil line additionally removes 26.8% of fresh tokens losslessly. Illustrative — your numbers depend on your traffic.

Compression, the whole family tree

"Compression" is an umbrella over a dozen genuinely different ideas, each trading a different amount of quality risk for savings. The most common mistake in this space is conflating them — input-token compression (fewer tokens enter the model), KV-cache compression (the prompt is read in full; the retained attention memory shrinks), and cost reduction (same tokens, cheaper compute) are three different axes. Here is the honest map.

Lossless · mechanical

Make the same bytes cost fewer tokens

Whitespace and JSON normalization, key shortening, de-duplication, run-length collapse, compact serialization. Risk: essentially zero — information is preserved exactly. The ceiling is modest, but it is free savings you should always take first.

Extractive prompt compression

Drop the least-informative tokens

Score each token's informativeness (self-information / perplexity) and prune the bottom. Systems like Selective Context and the LLMLingua family reach 2–20× on prose. Risk: real — a low-surprisal but load-bearing token (a negation, a constraint) can be dropped, and output is often human-unreadable.

Abstractive · summarization

Rewrite the context shorter

Summarize retrieved docs or older turns into a compact form (RECOMP; rolling-summary agent memory; OS-style tiered memory like MemGPT). High ratios. Risk: lossy — specific facts, numbers and entities can be distorted or dropped, and errors compound across turns.

Retrieval as compression

Keep knowledge outside the window

RAG and semantic caching hold the corpus in an index and pull only the top-k per query — lossy compression of a whole knowledge base into what fits. Risk: retriever-bounded — a missed chunk becomes a hallucination; a too-loose semantic-cache hit returns a stale answer.

KV-cache compression

Shrink the attention memory, not the input

H2O, Scissorhands, SnapKV, StreamingLLM keep only "heavy-hitter" or recent keys/values in GPU memory. Different axis: the model still reads every input token, so this cuts memory/latency, not your input bill. Eviction is permanent — a wrongly-dropped key can't return.

Soft-prompt / gist

Compress context into learned vectors

Gisting, AutoCompressors and ICAE fold a long prompt into a few learned "summary" vectors (up to ~26× reported). Risk: model-specific & trained — the artifact only works for the model it was trained on, and reconstruction is lossy.

And two cost-reduction cousins that aren't token compression at all: speculative decoding (a small draft model proposes tokens the big model verifies in parallel — provably identical output, 2–3× faster) and model cascades / routing (FrugalGPT, RouteLLM — try a cheap model first, escalate only when needed). They lower cost without removing tokens, and compose cleanly with everything above.

Why training a small model helps

The most effective compressors stopped using fixed heuristics and started learning. The root idea is knowledge distillation (Hinton et al., 2015): train a small "student" to imitate a large "teacher," using the teacher's soft probabilities — which carry far richer signal than a one-hot label.

Applied to compression, the move is to reframe it as per-token keep/drop classification. LLMLingua-2 is the clean example: take a small bidirectional encoder (XLM-RoBERTa, ~355M params), and train it on labels distilled from GPT-4 — prompt GPT-4 to compress text by word-removal only, then mark each original token "keep" or "drop." A trained classifier beats a perplexity heuristic for three concrete reasons:

It sees both directions. Perplexity is computed left-to-right by a causal model; a bidirectional encoder uses full left-and-right context to judge importance.
It learns task/domain importance and outputs calibrated keep probabilities, instead of a fixed, task-blind proxy.
It's faster — one encoder pass replaces an autoregressive scoring loop (reported 3–6× faster), and the win is largest out-of-domain.

The risk that comes with learning. A trained compressor internalizes its training distribution and fails quietly. The documented failure modes are real: out-of-distribution regression; over-compression dropping exactly the load-bearing tokens (high-information tokens are often the most important, yet get pruned at the same rate); and task-dependent fragility (classification and code degrade far harder than summarization). Past a point, aggressive compression flips from cost-saving to cost-increasing as garbled outputs trigger retries. This is why a learned keep-model must be gated, never trusted blindly — the subject of Module 3.

Key takeaways

Multi-turn cost is O(n²) because every turn re-bills all prior tokens — a 4-turn loop is 10×, not 4×.
Prompt caching is the biggest lever (read @ 0.10× vs fresh @ 1.0×), but any byte change in the prefix throws the cache away.
Compression is a family of techniques on three different axes; pick by your risk tolerance, lossless-first.
A trained keep/drop model beats heuristics — but fails quietly, so it must be gated by proof.