Compression Techniques

Each technique targets a specific slice of agent cost. Two carry the bulk of the win: cache-aware lossless compression keeps the prefix byte-stable so every cached token stays cached, and causal pruning drops what ablation proves never changed a decision. The rest stack cleanly on top — and one of them, recoverable compression, is something no lossy compressor can structurally replicate.

① Cache-aware compression

Module: distil/compress/cache_aware.py · Loss profile: lossless

The dominant cost lever in a multi-turn agent loop is not token count — it is cache hit rate. A cache read costs roughly 0.1× fresh input; a cache miss costs full price. Distil models the full multi-turn loop, tracking the longest contiguous prefix that is identical between consecutive turns. Any compressor that rewrites that prefix — even to save tokens — destroys the prefix match and loses the 10× discount.

Cache stabilization

Two lossless pre-processing steps run before any compression, keeping the prefix byte-stable turn over turn:

• Schema canonicalization — recursively sorts JSON object keys in tool payloads and API responses. Two semantically identical JSON objects with different key orders produce byte-identical text after canonicalization and hit the prefix cache.
• Volatile-field extraction — fields that change every turn (timestamps, UUIDs, JWT tokens, request IDs) are lifted out of the stable prefix into the volatile tail. The stable portion stops churning; the volatile tail is compressed separately.

The simulation

cache_aware.simulate() models four strategies across a full trajectory and reports per-turn cost breakdown — cache reads, cache writes, fresh tokens. Run distil savings to see the numbers on any trajectory.

$ distil savings --pricing claude-opus-4-8
strategy                               $ / run   vs baseline  cache hits
------------------------------------------------------------------------
baseline (no cache, no compress)       0.01524          0.0%           0
cache only                             0.01115         26.8%       1,028
naive compress + cache                 0.01691        -11.0%           0  ← busts cache
distil (cache-aware lossless)          0.01019         33.1%       1,028

② Salience protection (frontier shifter)

Module: distil/compress/cache_aware.py · Loss profile: lossless

Lossless compression is safe by construction, but it is also blind — it will happily drop the one line that carries the agent's decision if that line happens to be compressible. Salience protection is a model-free guard that runs before any content reduction and marks lines that must survive regardless of savings pressure.

Three signals combine, all computed from the text alone (no external model call, no learned weights):

• Pattern matching — decision markers (DECISION:), error keywords, structured-data signals (key:value, pipe tables, digit-dense lines).
• Entropy scoring — high-entropy lines carry information-dense content (IDs, hashes, addresses) and are protected regardless of keyword matches.
• Cross-reference detection — a line referenced elsewhere in the context (an ID that reappears, a filename cited later) is treated as load-bearing even if it looks inert in isolation.

Protection is line-aware: the guard operates on individual lines, not blocks, so it can preserve a single critical row inside an otherwise-compressible table. The cost is a small savings reduction — roughly 2.4 percentage points on the benchmark corpus — in exchange for a structural guarantee.

③ Causal / counterfactual pruning

Module: distil/replay/ablation.py · Loss profile: certified

The eval engine in Distil is a discovery engine. The algorithm:

1. For each context block in the trajectory, remove it and replay the trajectory.
2. Record whether any turn's decision changed.
3. A block that never changed a decision across all its occurrences is causally inert — provably safe to drop.

This is the sense in which measurement produces the compression policy. Speculative retrievals never cited, stale history that no longer affects any tool call, diagnostic context superseded by later turns — these all show up as zero-impact in ablation and can be pruned with a causal guarantee, not a heuristic guess.

$ distil prune
causal ablation over 'sre-disk-incident' — what is free to drop?

block                  occ   tokens  verdict
doc-0                    4      312  PRUNE (causally inert)
doc-1                    4      303  PRUNE (causally inert)
obs-0                    4      234  keep (changed a decision)
obs-1                    4      198  keep (changed a decision)
system                   4      218  keep (changed a decision)

tokens provably free to drop: 615 across 2 block(s).

Risk-graded tiers

Every technique is assigned to a tier based on its loss profile. Distil applies them in order — lossless ones unconditionally, certified-lossy ones only when the gate passes.

Keep-model codec (pluggable)

Modules: distil/codec/learned.py · distil/codec/transformer.py · Loss profile: pluggable

For per-line decisions about which content in a tool result is worth keeping, Distil exposes a pluggable KeepModel seam. Three implementations ship, from the heuristic default through to a retrain-on-your-traces transformer.

The three tiers at a glance

Heuristic default — SalienceKeepModel

distil/codec/keep_model.py · A deterministic rule-set that fires on error keywords, structured data markers (DECISION:, key:value lines, pipe tables), digit density, and debug-noise keywords. Always available; no training step, no external dependencies.

Logistic keep-model — built-in, zero-dep

distil/codec/learned.py · A logistic-regression classifier trained entirely in pure Python (stdlib only), implementing the same KeepModel protocol. Key design decisions grounded in the source:

• Feature set: nine features defined in codec/features.py — bias, decision-marker indicator, error keywords, structured-data signal, digit density, length, debug keywords, uppercase ratio, blank-line flag. All normalized to [0, 1].
• Training: full-batch gradient descent on binary cross-entropy with L2 regularization (l2=1e-4, 400 epochs). Labels are derived from SalienceKeepModel: a line is labelled keep (1.0) if the heuristic scores it ≥ 0.6.
• Train/test split: deterministic, SHA-256 hash of the raw line — reproducible across machines with no random seed dependency.
• Held-out performance: 96.4% accuracy, 0.98 F1 on the bundled corpus test split. Note that the labels are distilled from the salience heuristic, so this measures agreement with that label source, not an independent ground truth.
• Weights: persisted to distil/codec/weights.json; load with LogisticKeepModel.load(). The weights ship inside the installed wheel — no training step required.

Transformer keep-model — ONNX inference adapter

distil/codec/transformer.py · An ONNX-backed token-classification transformer implementing the same KeepModel protocol. Heavy deps (onnxruntime, transformers) are lazy-imported so the stdlib core runs untouched. This class is the inference adapter only — it does not ship a pretrained checkpoint.

• Inference: tokenizes each line, runs the ONNX session, applies softmax over the label axis, and aggregates per-token keep-probabilities via mean of top-3 — more robust than plain mean on mixed-content lines.
• Never-drop floor: any line containing DECISION: is forced to score 1.0 regardless of model output, matching the SalienceKeepModel guarantee.
• Demo checkpoint: a checkpoint trained on the bundled corpus (96.3% acc / 0.98 F1) is attached to the v0.1.0 release. This is a demo — for production quality, retrain on your own traces.
• Loading: TransformerKeepModel.from_pretrained(onnx_path, tokenizer_dir). Requires pip install 'distil-llm[onnx]' for inference, or 'distil-llm[train]' to train.

Training the transformer on your traces

distil/codec/train_transformer.py · Fine-tunes a HuggingFace token-classification model (default: google/bert_uncased_L-2_H-128_A-2, ~4 MB) using the same label pipeline as the logistic model, then exports to ONNX. Requires the train extras.

copy$ pip install 'distil-llm[train]'

# Train on the bundled corpus (replace with --corpus ./mycorpus for real traces)
$ distil train-transformer --out ./my-keep-model
Epoch 1/3  train_loss=0.4821
Epoch 2/3  train_loss=0.2134
Epoch 3/3  train_loss=0.1589
Exported ONNX to ./my-keep-model/model.onnx
Accuracy=0.9630  F1=0.9812
Load with: TransformerKeepModel.from_pretrained('./my-keep-model/model.onnx', './my-keep-model')

# Use the checkpoint via the Python API
from distil.codec.transformer import TransformerKeepModel
model = TransformerKeepModel.from_pretrained(
    "./my-keep-model/model.onnx",
    "./my-keep-model",
)
score = model.score("ERROR: disk at 94% capacity", "tool_output")
# → 0.97  (keep)

Gist caching

Module: distil/gist.py · Loss profile: lossless

Tool schemas are verbose and change rarely within a session. Gist caching detects when the same tool schema has been sent before (by content hash) and replaces subsequent occurrences with a compact reference handle. The model sees the full schema once; subsequent turns pay only for the reference token. Implemented as a content-addressed local cache — no network calls, no external state.

Delta / append-only context

Module: distil/delta.py · Loss profile: lossless

Context history is append-only — a block present in turn N must carry identical bytes in turn N+1 (enforced by the byte-fidelity gate). The delta engine exploits this: only the new blocks added since the last turn are transmitted at full cost. Stable history is already in the cache and costs only cache-read price. The delta layer ensures no block is re-sent in a way that would corrupt the prefix hash.

BM25 partial retrieval

Module: distil/retrieval.py

When a context block is digested (Tier 1) and represented by a handle, BM25 partial retrieval lets the agent query the most relevant chunks of the original without re-expanding the entire block. This keeps the context window focused while preserving access to the full original content — useful for large tool outputs, retrieved documents, or log dumps where only a fraction is decision-relevant.

Reversible structured compaction

Module: distil/compress/structured.py · in the Tier-1 path · Loss profile: lossless

The content agents actually traffic in — JSON record arrays, SQL rows, telemetry — is re-encoded into a compact columnar form: repeated keys stated once, values listed as rows. A homogeneous array of N records pays its keys and punctuation N times in JSON; the folded form pays them once — typically a 40–70% reduction with no loss of meaning. Unlike lossy structural crushing, nothing is discarded: the byte-exact original is kept in the restore table and is one expand() away.

Template mining (Drain / LogPai family)

Module: distil/compress/structured.py · template_fold() · Loss profile: lossless

Logs and telemetry are runs of near-identical lines that differ only by a timestamp, ID, or number. Where run-length encoding only collapses byte-identical lines, template mining masks the varying tokens, groups lines by their skeleton, and collapses a run into one template + a compact variable table — the production log-parsing technique (Drain / LogPai), applied to agent context. Every variable is retained; the original is fully recoverable. On log-heavy content this lifts lossless savings several points on its own.

Cross-turn dedup (streaming)

Module: distil/compress/dedup.py · StreamingDedup · Loss profile: lossless

Agents re-read the same artifact — a file, a log, a design doc — every turn. Prompt caching only discounts the contiguous prefix, so a large block that recurs in the volatile tail is re-billed in full each time it reappears. The streaming compressor remembers what it has already sent and replaces a recurring inert block with a compact reference, recoverable on demand. It only references blocks that carry no decision marker. This is the lever that turns a multi-turn loop's quadratic re-read cost into near-linear, on exactly the content the prefix cache cannot reach. See the Benchmark for the measured contribution.

Cache-delta context coding (coding-agent hot path)

Module: distil/cachedelta.py · distil proxy --session-delta · opt-in · Loss profile: lossless

Multi-turn coding agents (Claude Code, Codex, Gemini CLI) loop on read → edit → re-read. The re-read file is a near-duplicate of what the agent already has in context — one hunk changed. Exact-duplicate dedup (the state of the art elsewhere, e.g. Headroom's dedup_identical_items) misses it entirely and re-sends the whole file. Cache-delta coding handles both cases, confined to the volatile suffix so the stable, already-cached prefix is never touched:

• Exact re-send → back-reference (a compact pointer to the earlier occurrence already in cache).
• Near-duplicate → reference + unified diff (only the changed hunk is transmitted).

Both transforms preserve cache-monotonicity: the stable prefix is never mutated, so every prompt-cache hit that existed before the turn still exists after it. The prior version of the file remains present earlier in the cached conversation, so prior version + diff carries the same information the agent needs to take its next action — decision-equivalent by construction. The full file is recoverable byte-exact via distil_expand.

AST-structural delta (deepest layer of cache-delta)

Module: distil/astdelta.py · stdlib ast · model-free · Loss profile: lossless

The deepest layer of cache-delta coding. For Python source, the cross-version diff operates on parsed structure rather than raw text: each top-level definition is fingerprinted with ast.dump (attributes off), which is invariant to whitespace, comments, and import order. Two consequences a textual diff cannot offer:

• A reformat-only re-read — whitespace changes, comment edits, import reordering — is recognised as "no definition changed" and referenced whole. Textual diff would produce a large spurious patch.
• Only definitions whose AST actually changed are sent in full. Everything unchanged is a back-reference, regardless of how many lines moved around it.

Non-Python files and unparseable source (mid-edit, syntax errors) fall back automatically to the standard unified diff — the technique never fails a request, it just does less when the AST is unavailable.

Certified-fallback & the equivalence dial

Module: distil/compress/adaptive.py · certified_fallback · distil frontier

The aggressive digest is recoverable but not byte-identical to the model's eye, so on genuinely ambiguous decisions it can occasionally tip a different (still reasonable) choice. Certified-fallback removes that risk by construction: for each turn it keeps the most aggressive transform only if a runner confirms the decision is unchanged — else it falls back to byte-exact Tier-0, else to no compression. The result is 100% decision-equivalent by construction under whichever runner you pass — the free structural runner, or the live model itself (--runner anthropic).

On top of that sits the equivalence dial, target_equivalence. At 100% you get certified-safe compression. Relax it — say 0.95 — and Distil grants a bounded divergence budget of floor((1−target) × turns) turns that keep the deepest transform even when it changes the decision, spending the budget on the highest-saving turns first. You trade a measured, explicit amount of equivalence for deeper savings — never a hidden one. Trace the full curve with distil frontier; see Benchmark → equivalence dial.

The learning flywheel (self-improving)

Module: distil/learn.py · distil learn

This is what makes recoverable compression compound. Every distil_expand call is ground truth that a digested block was load-bearing. Distil tallies these by a coarse, content-free signature (content class × size bucket, e.g. json:l — never the content itself) and learns which signatures your agents keep expanding. Those it stops digesting, keeping them byte-exact. Because the policy only ever makes Distil more conservative, it is never-regressing by construction — it can reduce savings on an expand-prone signature, never reduce decision-equivalence — so it needs no gate to be safe. The stats live locally (atomic writes, content-free); inspect the learned policy with distil learn. The more your agents run, the better the fit to your workload.