Compression Techniques

Every technique in Distil targets a specific slice of agent cost. Two carry the bulk of the win; the rest stack cleanly on top.

① Cache-aware compression

Module: distil/compress/cache_aware.py · Loss profile: lossless

The dominant cost lever in a multi-turn agent loop is not token count — it is cache hit rate. A cache read costs ~0.1× fresh input; a cache miss costs the full price. Distil models the full multi-turn loop, tracking the longest contiguous prefix that is identical between consecutive turns. Any compressor that rewrites the prefix (even to save tokens) destroys this prefix match and loses the 10× discount.

Cache stabilization

To keep the prefix byte-stable turn over turn, Distil applies two lossless pre-processing steps before any compression runs:

• Schema canonicalization — recursively sorts JSON object keys in tool payloads and API responses. Two semantically identical JSON objects with different key orders produce byte-identical text after canonicalization, which means they hash identically and hit the prefix cache.
• Volatile-field extraction — fields whose values change every turn (timestamps, UUIDs, JWT tokens, request IDs) are lifted out of the stable prefix into the volatile tail. The stable portion stops churning; the volatile tail is compressed separately.

The simulation

cache_aware.simulate() models four strategies across a full trajectory and reports per-turn cost breakdown (cache reads, cache writes, fresh tokens). Run distil savings to see the numbers on any trajectory.

$ distil savings --pricing claude-opus-4-8
strategy                               $ / run   vs baseline  cache hits
------------------------------------------------------------------------
baseline (no cache, no compress)       0.01524          0.0%           0
cache only                             0.01115         26.8%       1,028
naive compress + cache                 0.01691        -11.0%           0  ← busts cache
distil (cache-aware lossless)          0.01019         33.1%       1,028

④ Causal / counterfactual pruning

Module: distil/replay/ablation.py · Loss profile: certified

The eval engine in Distil is not a ruler bolted on the side of compression — it is a discovery engine. The algorithm:

1. For each context block present in the trajectory, remove it and replay the trajectory.
2. Record whether any turn's decision changed.
3. A block that never changed a decision in any of its occurrences is causally inert — provably free to drop.

This is the sense in which the measurement produces the compression policy. Speculative retrievals that were never cited, stale history that no longer affects any tool call, diagnostic context that was superseded — these all show up as zero-impact in ablation and can be pruned with a causal guarantee, not a heuristic guess.

$ distil prune
causal ablation over 'sre-disk-incident' — what is free to drop?

block                  occ   tokens  verdict
doc-0                    4      312  PRUNE (causally inert)
doc-1                    4      303  PRUNE (causally inert)
obs-0                    4      234  keep (changed a decision)
obs-1                    4      198  keep (changed a decision)
system                   4      218  keep (changed a decision)

tokens provably free to drop: 615 across 2 block(s).

Risk-graded tiers

Every technique is assigned to a tier based on its loss profile. Distil applies them in order — the safest ones unconditionally, the certified ones only when the gate passes.

Keep-model codec (pluggable)

Modules: distil/codec/learned.py · distil/codec/transformer.py · Loss profile: pluggable

For per-line decisions about which content in a tool result is worth keeping, Distil exposes a pluggable KeepModel seam. Three implementations ship, from the heuristic default through to a retrain-on-your-traces transformer.

The three tiers at a glance

Heuristic default — SalienceKeepModel

distil/codec/keep_model.py · The baseline. A deterministic rule-set that fires on error keywords, structured data markers (DECISION:, key:value lines, pipe tables), digit density, and debug-noise keywords. Always available, no training step, no external dependencies.

Logistic keep-model — built-in, zero-dep

distil/codec/learned.py · A logistic-regression classifier trained entirely in pure Python (stdlib only), implementing the same KeepModel protocol. Key design decisions grounded in the source:

• Feature set: nine features defined in codec/features.py — bias, decision-marker indicator, error keywords, structured-data signal, digit density, length, debug keywords, uppercase ratio, blank-line flag. All normalized to [0, 1].
• Training: full-batch gradient descent on binary cross-entropy with L2 regularization (l2=1e-4, 400 epochs). Labels are derived from SalienceKeepModel: a line is labelled keep (1.0) if the heuristic scores it ≥ 0.6.
• Train/test split: deterministic, SHA-256 hash of the raw line — reproducible across machines with no random seed dependency.
• Held-out performance: 96.4% accuracy, 0.98 F1 on the bundled corpus test split.
• Weights: persisted to distil/codec/weights.json; load with LogisticKeepModel.load(). The weights ship inside the installed wheel — no training step required.

Transformer keep-model — ONNX inference adapter

distil/codec/transformer.py · An ONNX-backed token-classification transformer that implements the same KeepModel protocol. Heavy deps (onnxruntime, transformers) are lazy-imported so the stdlib core of distil runs untouched. The class is the inference adapter only — it does not ship a pretrained checkpoint.

• Inference: tokenizes each line, runs the ONNX session, applies softmax over the label axis, and aggregates per-token keep-probabilities via mean of top-3 — more robust than plain mean on mixed-content lines.
• Never-drop floor: any line containing DECISION: is forced to score 1.0 regardless of model output, matching the SalienceKeepModel guarantee.
• Demo checkpoint: a checkpoint trained on the bundled corpus (96.3% acc / 0.98 F1) is attached to the v0.1.0 release. This is a demo — for production quality, retrain on your own traces.
• Loading: TransformerKeepModel.from_pretrained(onnx_path, tokenizer_dir). Requires pip install 'distil-llm[onnx]' for inference only, or 'distil-llm[train]' to train.

Training the transformer on your traces

distil/codec/train_transformer.py · Fine-tunes a HuggingFace token-classification model (default: google/bert_uncased_L-2_H-128_A-2, ~4 MB) using the same label pipeline as the logistic model, then exports to ONNX. Requires the train extras.

copy$ pip install 'distil-llm[train]'

# Train on the bundled corpus (replace with --corpus ./mycorpus for real traces)
$ distil train-transformer --out ./my-keep-model
Epoch 1/3  train_loss=0.4821
Epoch 2/3  train_loss=0.2134
Epoch 3/3  train_loss=0.1589
Exported ONNX to ./my-keep-model/model.onnx
Accuracy=0.9630  F1=0.9812
Load with: TransformerKeepModel.from_pretrained('./my-keep-model/model.onnx', './my-keep-model')

# Use the checkpoint via the Python API
from distil.codec.transformer import TransformerKeepModel
model = TransformerKeepModel.from_pretrained(
    "./my-keep-model/model.onnx",
    "./my-keep-model",
)
score = model.score("ERROR: disk at 94% capacity", "tool_output")
# → 0.97  (keep)

Gist caching

Module: distil/gist.py · Loss profile: lossless

Tool schemas are verbose and essentially never change within a session. Gist caching detects when the same tool schema has been sent before (by content hash) and replaces subsequent occurrences with a compact reference handle. The model only needs to see the full schema once; subsequent turns pay only for the reference token.

This is implemented as a content-addressed local cache — no network calls, no external state. The handle is embedded inline in the message; a session-aware proxy or adapter layer resolves it back to the full schema if needed.

Delta / append-only context

Module: distil/delta.py · Loss profile: lossless

Context history is append-only — a block present in turn N must carry identical bytes in turn N+1 (this is enforced by the byte-fidelity gate). The delta engine exploits this: only the new blocks added since the last turn need to be transmitted at full cost. Stable history is already in the cache and costs only cache-read price. The delta layer ensures no block is re-sent in a way that would corrupt the prefix hash.

BM25 partial retrieval

Module: distil/retrieval.py

When a context block is digested (Tier 1) and represented by a handle, BM25 partial retrieval lets the agent query for the most relevant chunks of the original without re-expanding the entire block. This keeps the context window focused while preserving access to the full original content on demand — useful for large tool outputs, retrieved documents, or log dumps where only a fraction is decision-relevant.

Reversible structured compaction (SOTA)

Module: distil/compress/structured.py · in the Tier-1 path

The token-dense content agents actually traffic in — JSON record arrays, SQL rows, telemetry — is re-encoded into a compact columnar form: the repeated keys are stated once, the values listed as rows. A homogeneous [{"id":1,"name":"a","ok":true}, …] of N records pays its keys and punctuation N times in JSON; the folded form pays them once — typically a 40–70% reduction with no loss of meaning. Unlike a lossy structural crusher, nothing is discarded: the byte-exact original is kept in the restore table and is one expand() away. This is the reversible counterpart to the lossy "crushers" other tools use — same savings on clean structured data, but it never changes a value.

Template mining (Drain / LogPai family)

Module: distil/compress/structured.py · template_fold()

Logs and telemetry are runs of near-identical lines that differ only by a timestamp, id, or number. Where run-length encoding only collapses byte-identical lines, template mining masks the varying tokens, groups lines by their skeleton, and collapses a run into one template + a compact variable table — the production log-parsing technique (Drain / LogPai), applied to agent context. Every variable is retained (information-preserving) and the original is recoverable. On log/metric-heavy content this lifts the byte-exact lossless savings several points on its own.

Cross-turn dedup (streaming)

Module: distil/compress/dedup.py · StreamingDedup

Agents re-read the same artifact — a file, a log, a design doc — every turn. Prompt caching only discounts the contiguous prefix, so a large block that recurs in the volatile tail is re-billed in full each time it reappears. The streaming compressor remembers what it has already sent and replaces a recurring inert block with a compact reference, recoverable on demand. It only references blocks that carry no decision marker, so the decision signal is never perturbed. This is the lever that turns a multi-turn loop's quadratic re-read into near-linear, on exactly the content the cache can't reach. See the Benchmark for the measured contribution.