compression with a quality contract

The cache contract

Prompt caching is the largest single lever on an agent's bill, and it is all-or-nothing: rewrite one byte at or before the provider's cache boundary and the entry for the whole prefix is discarded. This page states, precisely, what Distil guarantees about the bytes it forwards across turns — and what it does not. Every clause is enforced by tests/test_cache_contract.py, against all three provider shapes.

This is the contract. Prompt Caching is the operational guide — how to mark a prefix, what distil cache reports, and how to read a miss. Start there if you are debugging a bill; start here if you are asking what Distil promises.

Why this is written down

Distil has been on the wrong side of this once. A recency carve-out that counted back from the end of the message list slides forward as a conversation grows, so every block was protected while fresh and rewritten one turn later — after the client had already committed it to a cached prefix. Measured against the real API, that produced zero cache reads on every turn and 2× the cost of compressing nothing at all: the entire prefix was re-written at 1.25× instead of re-read at 0.1×.

The fix was to anchor the carve-out to the client's own cache_control breakpoint rather than to the end of the list. The reason it needs a contract and not just a fix is that the failure is silent. Every request still succeeds. The savings report still shows tokens removed. Nothing raises and nothing 5xxs. The only symptom is the invoice.

Where the boundary is

The contract is stated relative to the index the provider has cached through, and that differs by provider:

ProviderCache boundaryRecency carve-out
Anthropicthe last cache_control marker the client placedonly after that marker
OpenAIthe last index — implicit prefix caching commits everything sentnone
Geminithe last index — implicit prefix caching commits everything sentnone

The empty carve-out on OpenAI and Gemini is a decision, not an oversight. Those providers cache what they are sent the moment they are sent, so a window counted back from the end would digest, one turn later, content already committed. _recent_chat_verbatim_indices and its siblings return the empty set for exactly that reason, and say so in their docstrings.

The binding form of the contract uses a high-water mark rather than the current turn's boundary: once the provider has cached through index i, rewriting i on any later turn invalidates the entry, even if this turn's marker sits earlier.

What is guaranteed

(a) Prefix stability

For every message the client re-sends byte-identical to the previous turn, Distil forwards it byte-identical, at every index at or before the cache boundary.

(b) Compression touches only the volatile suffix

Anything that does change lies strictly after the boundary. Two mechanisms enforce this and both are load-bearing. The recency carve-out anchors to the breakpoint. And query-aware salience is scoped to blocks after it — intent terms come from the newest user turn and change every turn by design, so letting them choose which lines survive inside an already-cached block would rewrite that block on every new question. That is the same cache bust from a second direction, and it is invisible to any single-request test.

(c) Digest determinism

A digest emitted for a block at turn N is byte-identical when the same block is forwarded at turn N+1. Handles are content-addressed — sha256(text)[:8], with no per-request nonce, counter, or timestamp. This was checked in the code rather than assumed, and it is asserted on its own, because a random handle would rewrite every digest stub on every turn and bust the cache by itself while every other clause still passed.

(e) Prefix determinism — the other way to satisfy (a)

Clause (b) is how most of Distil stays inside the contract, but it is not the only way and it is not always available. Under the client shape that actually bills — Claude Code pins its newest turn, so the entire history is cached — there is no volatile suffix, and a transform gated on one would simply never run.

The re-read delta takes the other route. Its plan is a pure function of the message prefix: what a block encodes to depends only on the blocks before it, never on where the boundary currently sits or on how long the conversation has grown. So the block's bytes never change at all, which is strictly stronger than (a) asks for — there is nothing to invalidate because nothing is ever rewritten. Cache-delta coding relies on the same construction. The recency carve-out still applies on top — the freshest tool output is never elided — and because that carve-out is itself anchored to the breakpoint, the one rendering change it causes lands strictly after the boundary, exactly as clause (d) describes.

A gate on the boundary would have been worse here, not safer: it flips a block from stub to verbatim exactly as the boundary advances past it, which is the failure clause (a) exists to catch. The decision record is ADR 0010.

(f) A canonically-equal prefix is forwarded as previously sent

Real agentic clients rewrite their own history on every turn without changing a token the model reads: the cache_control breakpoint advances to the newest block, an SDK stamps positional index fields, a string becomes a single text block or back again. Clause (d) below used to end the story there — the input is not the same input, so no promise — and that cost the whole prefix. Measured offline before the fix, a client doing nothing but renumbering an index field forwarded 0% of its prefix byte-identical, on every provider, on every turn.

So Distil remembers the previous turn's forwarded bytes per conversation lineage, and for the longest canonically-equal leading prefix it forwards the bytes it forwarded last turn rather than the bytes it would produce now. The canonical comparison ignores cache_control, index, the interchangeable spellings of a single text block, and JSON key order. It ignores nothing else; a tool payload — the arguments going out, the result coming back — is compared exactly as it arrived, because a key called index one level down inside a tool result is data, not SDK bookkeeping. The client's current breakpoint markers are re-placed at the client's current block positions, because the marker delimits the cached span and is not part of it.

The guard that makes this safe is narrower than it looks. Replay restores bytes, never decisions: a message is replayed only when what Distil would send this turn is itself canonically equal to what it sent last turn. Distil's compressor is not a pure function of one message — the exact-quote guarantee keeps a tool result verbatim because of an edit that arrives later in the list — so overlaying an older stub there would break the agent's next edit to buy a cache hit. That trade is not made. distil validate carries the invariant.

On by default in all three servers — the threaded proxy, the async proxy, and the multi-tenant gateway, which scopes the lineage per tenant. --no-prefix-replay forwards exactly what the compressor produced. Any exception falls open to the same. The decision record is ADR 0011.

(g) Opaque, provider-signed content is never touched

Anthropic. Server-side compaction (beta compact-2026-01-12 / compact-2026-09-04) returns a compaction content block whose signature the provider re-validates on the next request: alter it, move it or re-encode it, even losslessly, and that request fails with compaction_signature_invalid. Distil never touches it, on any path: compress_messages in every mode, the SDK wrap() adapter, the proxy (the context_management field and the anthropic-beta header pass through), and the streaming splice, including when the tool results around it are digested. The same guard already protected thinking and redacted_thinking: an unknown block type falls through untouched. tests/test_compaction_passthrough.py pins it. The block's tokens are censused (compaction_billed, and signed_block_billed for any future block carrying a signature), so a cost distil cannot reduce is not also hidden from the savings percentage.

OpenAI Responses. Some content a client must return byte-identical not because of the cache, but because the provider validates it: the provider re-derives state from those exact bytes on the next turn, and altering them — even a lossless re-encode — can make the provider reject the request or silently lose state it never got back. The OpenAI Responses API (/v1/responses) carries two such item types in its input array:

Dispatch in compress_responses_input is by item["type"] — never by duck-typing an output/encrypted_content key — so neither item type is ever mistaken for a digestible function_call_output, and both survive every mode (verbatim, digest, recency, the distil_expand re-query, the buffered-stream re-emit) as the same object, not a copy. The generalisation is on the presence of encrypted_content rather than an allowlist of type names, so a future opaque item type is safe by construction. A function_call_output immediately before or after an opaque item still digests normally — an opaque item must never accidentally halt or exempt what surrounds it.

The request-side fields that anchor server state — context_management and previous_response_id — reach upstream unaltered; the proxy spreads the request body and replaces only the input key. POST /v1/responses/compact is a distinct endpoint from /v1/responses (a path, not a query-string variant of it) — it fails the compressible-path match entirely and is forwarded through the plain byte-for-byte relay, never parsed as a compressible body.

These items still cost tokens the provider bills, so they are censused (reasoning_billed / compaction_billed / signed_item_billed — heuristic counts of base64 ciphertext, an approximation of the provider's real billed reasoning tokens, not an exact figure) rather than silently absent from the eligibility breakdown — the same principle behind the Anthropic adapter's thinking/compaction-block census. A near-zero savings percentage on a request carrying thousands of these tokens is "billed, uncompressible, and now visible," not "the compressor gave up."

This is deliberately asymmetric with the Anthropic side. Anthropic's compressible-zone baseline (proxy._count_messages) counts thinking/redacted_thinking tokens even though it never rewrites them, so a signature-pinned block still shows up in the before/after diff. The OpenAI Responses baseline (count_responses_tokens) does not extend that same inclusion to reasoning/compaction items — it counts only content distil is actually allowed to touch, the same design choice that already excludes assistant text and function_call items. The consequence: on a reasoning- or compaction-heavy Responses session, the eligibility census total can legitimately exceed x-distil-compressible-tokens. That is the intended signal — a real, billed cost outside what the baseline claims to cover — not a bug in either number.

(h) Old bytes change only when the cache is already gone

Every clause above protects a prefix the provider still holds. Anthropic keeps a cache entry for a fixed TTL after the last request that touched it (five minutes by default, one hour when the client marks ttl: "1h"). A turn that arrives after that pays to re-write the whole prefix whatever Distil forwards. That turn is the one moment old bytes can change for free.

So on a turn where expiry is certain from Distil's own observation, it replaces older tool output with a recoverable stub: <<distil evicted older tool output (N lines); distil_expand handle=H recovers it>>. It then forwards that stub on every later turn, so the smaller prefix is what gets cached. "Certain" is strict. The lineage has to be known to this process (first-seen or restarted means nothing happens). Nothing of it can be in flight, since a stream, an expand re-query or a shadow replay refreshes the entry after the request that started it. The monotonic gap since Distil last finished forwarding it has to exceed the longest TTL the lineage ever asked for, plus a margin. And the conversation has to be unambiguously one conversation. Two histories under one lineage key (parallel subagents, a fork, a rewind) turn the feature off for that lineage for good.

Eviction never touches the freshest tool turns, exact-quote results (the file reads an Edit quotes back, and distil_expand results), text an Edit already quotes, learned-keep content, or anything the model has expanded. The stub is a pure function of the block's content, and the evicted set only grows. It is persisted (content-free: hashed lineage keys and the provider's random tool-call ids), so a restart or an upgrade within the same wrap session re-applies the same stubs rather than un-evicting a warm prefix. A fresh distil wrap is a new session and starts with none. Every block goes into the existing restore store, so distil_expand recovers it. The feature runs only where the recoverable digest already runs, and lossless-only is untouched. Opt out with --no-cold-point or DISTIL_COLD_POINT=0. Anthropic only for now, because OpenAI's automatic cache gives no TTL Distil could call certain. Not yet measured live: the savings from this clause are unproven until an rc soak and a live A/B on distil cache read/write totals have run. The decision record, including those two gates, is ADR 0014.

(i) The quote guard never rewrites the prefix for nothing

The exact-quote guard is the one place a decision may legitimately change for a client item already sent — the reason (f) replays bytes and not decisions. It checks that every Edit's old_string still occurs in the payload about to be forwarded, and on a miss re-compresses with the exemption widened. Until this release the widened pass was forwarded whether or not it found the quote. A quote no read ever carried byte-exact — text the agent Write-ed itself, or a multi-line quote against Claude Code's line-numbered Read output — is lost under both passes, and forwarding the widened one turned every re-read stub and superseded read already in the cached prefix back into verbatim text. That rewrote the prefix inside the cache window for zero rescued quotes, then kept the class off for the rest of the session.

The widened pass is now forwarded only when the quotes it loses are a strict subset of the ones the narrow pass loses: it must rescue at least one and lose none the narrow pass kept. Replayed over 12 local Claude Code transcripts (4,658 requests, rebuilt append-only, so any change to the forwarded prefix is Distil's by construction), the widened pass ran on 2,625 requests and rescued a quote on none. Before the change that replay found one in-window break, 176,944 bytes of prefix rewritten. After it, none. benchmarks/in_window_prefix_breaks.py reproduces both, offline. tests/test_cache_contract.py drives the case through the threaded proxy and asserts the forwarded prefix hashes are identical turn over turn.

Tool definitions are forwarded untouched

Distil never rewrites the request tools array. Tool definitions come first in the cached prefix, so on real traffic they bill almost entirely at the cache-read rate. Only a small fraction of a real array can be removed without changing what the model can call, such as a $schema URI or a title that repeats the property name. A rewrite would save little, and one non-deterministic byte would turn every cache read into a cache write. The measurement and the condition for reopening the question are in ADR 0012. To cut tool overhead, remove unused connectors; distil discover ranks them by cost, and names MCP servers whose definitions were sent on 20 or more requests and never called.

Leaving the array alone also covers Claude Code's own tool search. Claude Code defers MCP tool definitions and loads one only when the model asks for it, but turns that off whenever ANTHROPIC_BASE_URL names a non-first-party host. distil wrap -- claude therefore sets ENABLE_TOOL_SEARCH=true for the child (an exported ENABLE_TOOL_SEARCH=false still wins), and distil default --always-on adds the same key to Claude Code's settings only if it is absent. Distil forwards defer_loading, tool_reference blocks and the beta header byte-identical. Verified live (2026-09-25): a wrapped Claude Code session that called an MCP tool recorded deferred tools on every request, cache reads intact, no failures. It is confirmed when a session that uses an MCP tool records tools_deferred > 0 in its request log with no request failures. Why Distil does not hide unused tools itself: ADR 0013.

(d) What is not guaranteed

How it is enforced

tests/test_cache_contract.py replays growing six-turn synthetic sessions through the same public entry points the proxy calls — compress_messages, compress_chat_completions, compress_responses_input, and compress_generate_request — and fails on any same-input byte drift at or before the boundary. Three Anthropic marker placements are covered: pinned at the head, moved forward each turn, and absent entirely.

Every transform that touches cross-block state is replayed the same way. The re-read delta gets its own assertion under the moving-marker (fully-cached) shape in tests/test_reread_delta.py, plus a test that fails if it stops firing — a silently-dead transform must not pass as a silently-safe one.

Clause (f) is driven the same way, with a control arm: each of the three rewrite shapes is replayed against every provider both with replay on and with it off, and the test fails unless the rewrite demonstrably shortened the byte-stable prefix without replay and demonstrably did not with it. A stability assertion whose control never destabilises proves nothing. benchmarks/prefix_replay_stability.py reports the same measurement as a table, offline and with no API calls.

Measured at adoption, the contract holds on all three providers. That includes the realistic Claude Code shape, where the client pins its newest turn so the entire history is cached: there is no uncached tail in that configuration and nothing moves at all. No bug was found. The test codifies a property that was true and undefended.

Clause (g) is driven by tests/test_openai_opaque_passthrough.py: identity preservation for reasoning/compaction items across verbatim and digest mode, a still-compresses control on a neighbouring function_call_output, a duck-typing-immunity check, the /v1/responses/compact routing exclusion, and the census assertions that failed before the fix landed (see the changelog entry) and pass now.

Checking it against the provider

The clauses above are what Distil intends. The provider's own accounting is what actually happened, and distil dissect now reports it per session:

cache-read share: 87.4% of billed input was served from the
                  provider's prompt cache (at ~0.1x)

Beside it, what Distil did to hold that share:

prefix replay: 812 messages forwarded as previously sent,
               96 compressed fresh; 143 client rewrites repaired

Repaired is the number that moved money — messages the client rewrote non-semantically and Distil forwarded as previously sent, so the prefix survived a rewrite it would otherwise have lost. Hits with zero repairs is the healthy steady state, not a dead feature, which is why the two are never added together. The line reads not recorded for sessions proxied before 1.52 or run with --no-prefix-replay.

The number is cache_read_input_tokens over total billed input, taken from the fields the ledger already recorded. A near-zero share on a long session means something rewrote the prefix, and it is the only signal that says so — the requests all still succeed either way.

When the records predate those fields the line reads not captured and the JSON field is null, never 0.0. “We did not measure this” and “the cache never hit” are opposite diagnoses and must not share a rendering. This matters for a specific shape: older sessions carry only the aggregate usage_cache_tokens and neither split field, so summing reads over them gives zero against a nonzero input total — a confident 0.0% for a session nobody measured. At least one record must carry a split field before the number is reported at all.

Each request's ledger record also says why replay stopped, in replay_stop: held (every earlier item went out as before), client (the client changed or dropped an item it had already sent), distil (same client item, a different compression decision), marker (a breakpoint with nowhere to sit), cold (no state for this conversation: first turn, a restart, or eviction), or untracked (a history over the state cap). An in-window cache write on a client turn is not Distil's to fix. One on a distil turn is a bug against this page.

Why that field exists: on the maintainer's own ledger (16,093 requests, $2,793.40 billed), in-window breaks of every cause cost $35.83, or 1.28% of billed. That counts cache writes on a request the provider had seen the whole of less than five minutes earlier. $10.42 of it followed a change to the system prompt or the tool list, which is the client's. $0.16 came on turns where Distil re-sent every earlier item unchanged. The remaining $25.26 came on turns where replay diverged before this field was recorded, so the ledger cannot say whose rewrite it was. The transcript replay under (i) bounds Distil's part: one break in 4,658 requests before the fix and none after it.

The decision records are ADR 0008 (the contract) and ADR 0011 (forwarded-bytes prefix replay). For the mechanics of what gets compressed, see Techniques; for the adversarial side of the same surface, see the Threat Model.

Distil · compression with a quality contract · Apache-2.0 · github.com/dshakes/distil