Architecture Overview

Distil is a cost-optimized cache hierarchy with a statistical quality contract. Six components, one invariant: the agent makes the same decisions on compressed context as on the original — provably, not heuristically.

Interactive — select any block (click, or Tab then Enter) to open the concept or technique behind it.

The request path, in order

What one Anthropic Messages request passes through on distil proxy or distil wrap, left to right in the diagram. The OpenAI and Gemini adapters run the same compression and replay; the drift guard and the cold point are noted where they differ.

1. Drift guard (distil/drift.py). If shadow mode's paired verdicts have pushed the anytime-valid drift alarm over the one risk budget (distil.conformal.BUDGET_ALPHA), the request is served lossless-only: Tier-0, no digest, no output shaping. The response carries x-distil-mode: lossless-only and x-distil-drift-guard: held. distil_expand stays injected, so stubs already in the history remain recoverable. The hold is one per machine, survives restarts, is written to the receipt chain, and is released with distil reset --drift-guard. Checking it costs one attribute read per request. A guard that fails serves the request as configured. The multi-tenant gateway runs no shadow and is exempt by design (ADR 0016). New since 1.53.0rc1 and still to pass an rc soak.
2. Cold-point planner (distil/coldpoint.py). When a turn arrives after the provider's cache entry has certainly expired, the whole prefix is re-billed anyway, so older tool output is replaced by distil_expand-recoverable stubs there and those stubs are forwarded on every later turn. Anthropic only, only where the recoverable digest runs; --no-cold-point opts out. Not yet measured live: the rc soak and a live A/B are its gate. Contract: clause (h); decision record: ADR 0014.
3. Compression. Salience pins the load-bearing lines, Tier-0 runs lossless transforms, Tier-1 digests large tool output behind a content handle, and the re-read delta references lines the conversation already holds. Exact-quote provenance exempts anything an Edit may quote back. A dropped run no longer than its << +N lines, handle=… >> marker is shown inline rather than pointed at; that gain is small, and zero on the bundled corpus.
4. Prefix replay (distil/prefixreplay.py). For the longest canonically-equal leading prefix, the bytes forwarded last turn are forwarded again. Each request records why replay stopped (replay_stop, and x-distil-replay-stop on the response), so an in-window cache write can be attributed to the client or to distil. See below.
5. Upstream. Streams are relayed chunk by chunk. Provider-signed, opaque content is forwarded untouched: Anthropic thinking and server-side compaction blocks, OpenAI reasoning encrypted_content and compaction items (clause (g)). So is the request tools array (why).
6. After the response. One content-free receipt is appended to the hash chain; when the active file reaches the segment size it is sealed with a Merkle-root checkpoint (distil receipts). A shadow-sampled request is replayed off the request path, and its paired verdict is folded into the drift e-process the guard reads.

Observability — savings you can see

Distil surfaces savings at four widening scopes — each one verifiable, none requiring that any prompt or response content leave your machine.

• Per request — response headers (x-distil-tokens-saved, x-distil-cache-prefix-msgs, …).
• Per session — distil shadow-stats (live decision-change rate), the distil gateway dashboard, and the Proof Ledger (distil/proof_ledger.py) — a compact end-of-session printout on distil wrap exit showing calibrated tokens/cost, shadow verdict with honest suppression labels, and restorability. Its budget and risk lines quote one bound against one budget (distil.conformal.BUDGET_ALPHA), so it never prints intact beside a bound over budget, and when the drift alarm has tripped it says the proxy is holding compression at lossless-only. Opt out with DISTIL_NO_LEDGER=1.
• Across sessions, forward-looking — distil discover ranks what is still costing you over your recent sessions (including MCP servers whose definitions rode along on 20 or more requests and were never called), each with its derivation and the command to act on it. Module: distil/discover.py; see the CLI reference.
• Audit trail — a content-free, hash-chained receipt per request, sealed into segments with Merkle-root checkpoints so an auditor can check one receipt from an inclusion proof and a pinned checkpoint. Module: distil/receipts.py; see distil receipts.
• Per machine, lifetime — distil leaderboard rolls up the local savings ledger (~/.distil/savings.jsonl); --html exports a page.
• Community, verifiable — distil federated-leaderboard aggregates HMAC-signed, content-free, tamper-evident savings across instances; only certified submissions count, and sharing is opt-in (no telemetry by default).
• OTel traces (opt-in) — pip install 'distil-llm[otel]' emits GenAI semantic-convention spans plus distil.tokens.original, distil.tokens.compressed, distil.compression.ratio, distil.shadow.sampled, and distil.session.id for per-session trace correlation, plus OTel counters (distil.requests, distil.tokens.baseline/.sent/.saved) recorded at the same point. Module: distil/otel.py.
• Prometheus scrape — the gateway serves text exposition at /distil/metrics (stdlib-only, no client library), behind the same admin gate as /distil/stats because the series are tenant-labelled. Module: distil/metrics.py; see Metrics & Observability.
• Dry run, before you trust it — distil simulate runs the real pipeline with no model in the path and reports what would be compressed, what would be left alone, and which rule protected it. Module: distil/simulate.py; see the CLI reference.
• Vision — repeated byte-identical images are replaced by recoverable references; the first occurrence and every distinct image are untouched, and nothing is ever re-encoded or downscaled. Gated on a certificate and certified at 100% decision-equivalence against a live vision model. Module: distil/compress/vision.py; see Techniques § Vision.

Cache-aware prefix stability

Module: compress/cache_aware.py · compress/stabilize.py

The dominant cost in a multi-turn agent loop is not context size — it is cache miss rate. A cache read costs ~0.1× fresh input price; a naive compressor that rewrites the context each turn destroys the prefix match and loses that discount on every turn. The result: naive recompression sends fewer tokens yet costs more than not compressing at all (measured: −11% vs. the baseline on an SRE trajectory).

Distil models this explicitly. Two lossless pre-processing steps keep the prefix byte-stable before any compression runs:

• Schema canonicalization — recursively sorts JSON object keys so semantically identical payloads hash identically and keep hitting the cache regardless of key-insertion order.
• Volatile-field extraction — timestamps, UUIDs, JWT tokens, and request IDs are lifted out of the stable prefix into the volatile tail. The stable portion stops churning; the tail is compressed separately.

Both of those keep Distil's output stable. The other half of the problem is the client, which rewrites its own history every turn without changing a token the model reads — the cache_control breakpoint advances to the newest block, an SDK stamps positional index fields, a string becomes a single text block. Distil forwards what it receives, so each of those reached the provider as changed bytes and missed a prefix it was already holding. Measured offline, that cost the entire prefix, on every provider, on every turn.

Forwarded-bytes prefix replay (prefixreplay.py, on by default) closes it. Distil remembers the previous turn's forwarded bytes per conversation lineage and, for the longest canonically-equal leading prefix, forwards the bytes it forwarded last turn rather than the bytes it would produce now — re-placing the client's current breakpoint markers at the client's current positions, and stopping at the first real divergence. The comparison ignores cache_control, index, the interchangeable spellings of a single text block, and JSON key order, and nothing else. Replay restores bytes, never decisions: a message is replayed only when what Distil would send this turn is itself canonically equal to what it sent last turn, so the exact-quote guarantee can never be overwritten by an older stub. See the cache contract, clause (f).

The tier model

Every compression operation is assigned to a tier based on its loss profile. Higher tiers require stronger evidence before they are permitted.

The measure → discover → certify loop

Step 1 — Measure

The cache-aware cost engine (compress/cache_aware.py) simulates the full multi-turn loop with four variants: baseline (no cache, no compress), cache-only, naive compress + cache, and Distil. It computes per-turn cost breakdowns — cache reads, cache writes, fresh tokens — and reports the total dollar cost of each strategy. Run distil savings to see the numbers on any trajectory.

Step 2 — Discover

The causal ablation engine (replay/ablation.py) is not a ruler bolted to compression — it is a discovery engine. For each context block, it removes the block, replays the trajectory, and records whether any decision changed. The output is a per-block verdict: causally inert (safe to prune) or decision-driving (must be kept). This step produces the compression policy; it does not assume one.

$ distil prune
causal ablation over 'sre-disk-incident' — what is free to drop?

block                  occ   tokens  verdict
doc-0                    4      312  PRUNE (causally inert)
doc-1                    4      303  PRUNE (causally inert)
obs-0                    4      234  keep (changed a decision)
obs-1                    4      198  keep (changed a decision)
system                   4      218  keep (changed a decision)

tokens provably free to drop: 615 across 2 block(s).

Step 3 — Certify

The conformal DERC certificate (certify/stats.py via certify/gate.py and conformal.py) computes the decision-change rate across all turns and certifies it. The implementation uses Learn-Then-Test with Hoeffding–Bentkus p-values — distribution-free, valid at finite sample size, zero scipy/numpy. The guarantee: P(R(λ̂) ≤ α) ≥ 1 − δ.

A certified strategy has its policy registered in the compress registry. A failing strategy is rejected and produces a non-zero exit code from distil certify and distil bench.

$ distil certify --strategy distil
decision-equivalence match rate: 100.0%
TOST non-inferiority (margin=0.02, alpha=0.05): mean diff=+0.000, p=0.0000

VERDICT: PASS  (certified non-inferior)

$ distil certify --strategy aggressive
decision-equivalence match rate: 0.0%
TOST non-inferiority (margin=0.02, alpha=0.05): mean diff=−1.000, p=1.0000

VERDICT: FAIL  (NOT certified — would degrade quality)

The DERC certificate

Module: distil/conformal.py

DERC — the Decision-Equivalence Risk Certificate — is the conformal certificate that gates every lossy strategy. It answers one question: is this compression level's decision-change rate provably bounded at α with confidence 1 − δ?

Salience protection

Module: compress/salience.py · codec/keep_model.py

The salience layer is a model-free safety net that runs before the DERC gate, not instead of it. It protects content the gate might not have seen enough of to certify. The default is SalienceKeepModel — a deterministic rule-set that forces a keep score of 1.0 on decision markers (DECISION:) and command verdict lines (1955 passed, BUILD SUCCESSFUL, exit codes), and fires on error keywords, structured data, and high digit-density lines. Error-noise repeats are deduped by line shape — first occurrences kept as signal, the rest folded behind the handle, recoverable. The dedup budget routes on the log’s own outcome: a green run keeps one sample per shape (that noise didn’t fail anything), a red or unknown run keeps the cautious two. No model in the path, no training step required.

Because it only ever widens the set of blocks kept, salience protection is strictly conservative relative to the DERC certificate — it shifts the safe frontier further out, never closer in. The logistic keep-model (codec/learned.py) improves on the rules via a zero-dependency, pure-Python logistic classifier that ships in the wheel. The content-type layer (compress/keep_policy.py) sharpens the base rules per block: each block is classified as log, traceback, diff, or generic, then kind-specific patterns pin the load-bearing lines on top — verdict lines for log blocks, stack frames for tracebacks, file and hunk headers for diffs.

Query-aware salience

Module: compress/intent.py · Phase 1: deterministic, model-free

distil is a proxy, so at compress time it holds the agent’s intent and the tool output in the same request. Lines matching a discriminating intent term — a salient token from the preceding tool_use arguments or the latest user turn — are additively pinned, so the one line the agent is looking for (a grep hit, a config value, a specific SHA) survives even in arbitrary output where no fixed rule knows which line matters. A term matching most lines is dropped by a selectivity guard, preserving compression when intent is too broad. The layer is strictly additive: it only ever widens the keep set, leaving reversibility and the decision-equivalence certificate untouched.

No post-hoc filter — a shell filter, a content router, a vector finder — has this pairing: none sees the live query alongside the output at compression time. Phase 2 is a learned relevance scorer via the KeepModel protocol; the deterministic phase-1 is the floor it improves on.

Byte-fidelity invariants

Module: distil/fidelity.py · distil/harness.py · Gates: distil verify (corpus) + distil validate (adversarial)

Two structural invariants are enforced across the entire corpus by the verify gate. A second gate, distil validate, re-asserts them (plus reject-if-bigger, recency-exactness, fail-open, and content-free telemetry) against hostile inputs the corpus never contains — huge/unicode/nested/malformed tool output, content that mimics distil's own << handle >> stubs, secret-looking strings — because that is where real-traffic bugs hide. Both run in CI on every push.

A third gate closes the synthetic-vs-live gap: a nightly job (live-cert.yml) re-certifies every corpus trajectory against a real model via distil certify --runner anthropic, budget-capped with --max-live-calls 60 per trajectory so an unattended run cannot spend silently. Per-commit gates = synthetic oracle; nightly gate = live model; both are labeled as such.

Reversibility

Every original block is recoverable — either because the compressed text is byte-identical to the original, or because the original lives in the local restore table (keyed by block ID for Tier-0, by content handle for Tier-1). Recovery is confirmed via SHA-256 equality: auditable and machine-checkable.

Append-only history

A block ID that appears in two consecutive turns must carry identical bytes. Mutating a previously-seen block ID is a violation. This invariant is what makes the prefix cache hit rate reliable — the cache cannot be invalidated by history rewriting.

Numeric precision

JSON canonicalization must not lose numeric precision. numeric_precision_preserved() confirms that the original and transformed strings parse to the same JSON value.

$ distil verify
byte-fidelity gate — reversibility + append-only across the corpus

  sre-disk-incident        reversible + append-only: ok
  coding-bugfix            reversible + append-only: ok
  support-refund           reversible + append-only: ok
  research-synthesis       reversible + append-only: ok
  data-analysis-sql        reversible + append-only: ok
  devops-rollback          reversible + append-only: ok
  finance-reconcile        reversible + append-only: ok

FIDELITY: PASS — Tier-0/1 byte-reversible and history append-only across the corpus.

Sub-millisecond inline proxy

Module: distil/proxy.py · distil/native.py

Distil compresses at ~0.026 ms/turn — roughly 1,000× faster than model-based compressors (LLMLingua-2: ~1,480 ms/turn; Headroom: ~26 ms/turn). The reason: there is no model in the path. Tier-0/1 transforms are pure string operations; the causal ablation cache is populated offline. The hot path runs on stdlib-only Python with an optional Rust extension (distil/native.py, rust/distil-core) for JSON minification and run-length encoding when the wheel is built.

This is what makes inline proxy deployment viable. LLMLingua-2 loads a transformer; Headroom loads ModernBERT. Both add hundreds of milliseconds per turn. Distil adds less than one.

Auth-mode gating

Module: distil/policy.py

Aggressive compression applied to a subscription or OAuth session can alter conversations in ways that violate provider terms — injected retrieval tools the user never authorized, rewritten history that changes the conversation record. Auth-mode gating is a safety boundary, not an optimization.

The policy is a tightening boundary — a project's config can never loosen it. A PolicyError is raised if a non-permitted strategy is requested under the active auth mode. This maps onto the proxy's --lossless-only flag for subscription-context deployments.

Holdout A/B validation

Module: distil/certify/holdout.py · Command: distil holdout

The holdout module is an additional validation layer on top of the DERC gate. It:

1. Deterministically partitions the corpus into a control group (default: 20%, held out and not counted toward the headline) and a treatment group, using a hash-based split for reproducibility.
2. Measures savings in both groups separately.
3. Bootstrap-resamples the treatment group to compute a 95% confidence interval on the savings estimate.

The reported confidence interval is honest: it reflects the uncertainty given the finite corpus size. If control and treatment savings diverge significantly, something is wrong with the partition or the measurement — not with Distil's estimate.

Module map