Token Economics · Module 2 of 3 ~7 min read

# Cost & Compression

In this module

- Why multi-turn cost grows with the square of the conversation.
- How prompt caching rewrites the math — and what silently breaks it.
- The full family tree of compression, mapped by risk vs. savings.
- Why a trained compressor beats a heuristic — and how it fails.

Module 1 showed where the tokens pile up inside a single request. Now we follow them across a whole conversation — where the real money is.

## The multi-turn tax: why cost is quadratic

Single-turn cost is linear and boring. Multi-turn is where bills explode — and the shape is quadratic, not linear. Because each turn re-bills all the tokens before it, the cumulative cost of an n-turn loop is the sum 1 + 2 + … + n = n(n+1)/2 = O(n²). A four-turn conversation costs 4 + 3 + 2 + 1 = 10× the first turn's tokens — not 4×.

- Prefill vs decode. Reading the prompt (prefill) is compute-bound and parallel; generating output (decode) is memory-bandwidth-bound and sequential — every output token re-reads the whole model and KV cache from memory. That's why output is priced 3–10× higher than input, and why a cache hit (which skips prefill compute) can discount input so aggressively.
- Prompt caching rewrites the coefficient, not the shape. If the prefix (system prompt + tools + frozen early history) is byte-stable, you write it once and read it every later turn. Anthropic's documented model: a 5-minute cache write costs 1.25× base input, a read costs 0.10× — a 10× gap. The curve stays quadratic; caching just shrinks the dominant term's coefficient by ~10×. Any byte change anywhere in the prefix invalidates everything after it — a `datetime.now()` near the top, or editing a tool definition, silently throws the cache away.

This is the foundation of Distil's approach — see Concepts → cache misses dominate cost for the measured numbers behind it.

### Try it: the cost of a loop

Real arithmetic — no marketing. This computes the n(n+1)/2 input cost from your inputs, using live catalog pricing and the documented cache multipliers (0.1× read / 1.25× write). The "with Distil" line applies a flat lossless reduction to fresh tokens; the default 26.8% is Distil's measured corpus aggregate (see Research). Assumptions are shown so you can check the math.

Turns 12 Fixed prefix (system + tools), tokens 40,000 New tokens added per turn 3,000 Model
$0
Naïve — no caching, no compression
$0
With prompt caching (prefix read @ 0.1×)
$0
With caching + Distil lossless (−26.8%)

Assumptions: input-token cost only (output excluded for clarity); prefix is cacheable and byte-stable; per-turn input = prefix + accumulated history; cached line writes the prefix once at 1.25× then reads it at 0.10× while the growing tail stays fresh; Distil line additionally removes 26.8% of fresh tokens losslessly. Illustrative — your numbers depend on your traffic.

## Compression, the whole family tree

"Compression" is an umbrella over a dozen genuinely different ideas, each trading a different amount of quality risk for savings. The most common mistake in this space is conflating them — input-token compression (fewer tokens enter the model), KV-cache compression (the prompt is read in full; the retained attention memory shrinks), and cost reduction (same tokens, cheaper compute) are three different axes. Here is the honest map.

Lossless · mechanical

### Make the same bytes cost fewer tokens

Whitespace and JSON normalization, key shortening, de-duplication, run-length collapse, compact serialization. Risk: essentially zero — information is preserved exactly. The ceiling is modest, but it is free savings you should always take first.

Extractive prompt compression

### Drop the least-informative tokens

Score each token's informativeness (self-information / perplexity) and prune the bottom. Systems like Selective Context and the LLMLingua family reach 2–20× on prose. Risk: real — a low-surprisal but load-bearing token (a negation, a constraint) can be dropped, and output is often human-unreadable.

Abstractive · summarization

### Rewrite the context shorter

Summarize retrieved docs or older turns into a compact form (RECOMP; rolling-summary agent memory; OS-style tiered memory like MemGPT). High ratios. Risk: lossy — specific facts, numbers and entities can be distorted or dropped, and errors compound across turns.

Retrieval as compression

### Keep knowledge outside the window

RAG and semantic caching hold the corpus in an index and pull only the top-k per query — lossy compression of a whole knowledge base into what fits. Risk: retriever-bounded — a missed chunk becomes a hallucination; a too-loose semantic-cache hit returns a stale answer.

KV-cache compression

### Shrink the attention memory, not the input

H2O, Scissorhands, SnapKV, StreamingLLM keep only "heavy-hitter" or recent keys/values in GPU memory. Different axis: the model still reads every input token, so this cuts memory/latency, not your input bill. Eviction is permanent — a wrongly-dropped key can't return.

Soft-prompt / gist

### Compress context into learned vectors

Gisting, AutoCompressors and ICAE fold a long prompt into a few learned "summary" vectors (up to ~26× reported). Risk: model-specific & trained — the artifact only works for the model it was trained on, and reconstruction is lossy.

And two cost-reduction cousins that aren't token compression at all: speculative decoding (a small draft model proposes tokens the big model verifies in parallel — provably identical output, 2–3× faster) and model cascades / routing (FrugalGPT, RouteLLM — try a cheap model first, escalate only when needed). They lower cost without removing tokens, and compose cleanly with everything above.

## Why training a small model helps

The most effective compressors stopped using fixed heuristics and started learning. The root idea is knowledge distillation (Hinton et al., 2015): train a small "student" to imitate a large "teacher," using the teacher's soft probabilities — which carry far richer signal than a one-hot label.

Applied to compression, the move is to reframe it as per-token keep/drop classification. LLMLingua-2 is the clean example: take a small bidirectional encoder (XLM-RoBERTa, ~355M params), and train it on labels distilled from GPT-4 — prompt GPT-4 to compress text by word-removal only, then mark each original token "keep" or "drop." A trained classifier beats a perplexity heuristic for three concrete reasons:

- It sees both directions. Perplexity is computed left-to-right by a causal model; a bidirectional encoder uses full left-and-right context to judge importance.
- It learns task/domain importance and outputs calibrated keep probabilities, instead of a fixed, task-blind proxy.
- It's faster — one encoder pass replaces an autoregressive scoring loop (reported 3–6× faster), and the win is largest out-of-domain.

The risk that comes with learning. A trained compressor internalizes its training distribution and fails quietly. The documented failure modes are real: out-of-distribution regression; over-compression dropping exactly the load-bearing tokens (high-information tokens are often the most important, yet get pruned at the same rate); and task-dependent fragility (classification and code degrade far harder than summarization). Past a point, aggressive compression flips from cost-saving to cost-increasing as garbled outputs trigger retries. This is why a learned keep-model must be gated, never trusted blindly — the subject of Module 3.
Key takeaways

- Multi-turn cost is O(n²) because every turn re-bills all prior tokens — a 4-turn loop is 10×, not 4×.
- Prompt caching is the biggest lever (read @ 0.10× vs fresh @ 1.0×), but any byte change in the prefix throws the cache away.
- Compression is a family of techniques on three different axes; pick by your risk tolerance, lossless-first.
- A trained keep/drop model beats heuristics — but fails quietly, so it must be gated by proof.

← Previous Module 1 · Fundamentals Next →Module 3 · Distil & Proof