The cache contract

Prompt caching is the largest single lever on an agent's bill, and it is all-or-nothing: rewrite one byte at or before the provider's cache boundary and the entry for the whole prefix is discarded. This page states, precisely, what Distil guarantees about the bytes it forwards across turns — and what it does not. Every clause is enforced by tests/test_cache_contract.py, against all three provider shapes.

This is the contract. Prompt Caching is the operational guide — how to mark a prefix, what distil cache reports, and how to read a miss. Start there if you are debugging a bill; start here if you are asking what Distil promises.

Why this is written down

Distil has been on the wrong side of this once. A recency carve-out that counted back from the end of the message list slides forward as a conversation grows, so every block was protected while fresh and rewritten one turn later — after the client had already committed it to a cached prefix. Measured against the real API, that produced zero cache reads on every turn and 2× the cost of compressing nothing at all: the entire prefix was re-written at 1.25× instead of re-read at 0.1×.

The fix was to anchor the carve-out to the client's own cache_control breakpoint rather than to the end of the list. The reason it needs a contract and not just a fix is that the failure is silent. Every request still succeeds. The savings report still shows tokens removed. Nothing raises and nothing 5xxs. The only symptom is the invoice.

Where the boundary is

The contract is stated relative to the index the provider has cached through, and that differs by provider:

The empty carve-out on OpenAI and Gemini is a decision, not an oversight. Those providers cache what they are sent the moment they are sent, so a window counted back from the end would digest, one turn later, content already committed. _recent_chat_verbatim_indices and its siblings return the empty set for exactly that reason, and say so in their docstrings.

The binding form of the contract uses a high-water mark rather than the current turn's boundary: once the provider has cached through index i, rewriting i on any later turn invalidates the entry, even if this turn's marker sits earlier.

What is guaranteed

(a) Prefix stability

For every message the client re-sends byte-identical to the previous turn, Distil forwards it byte-identical, at every index at or before the cache boundary.

(b) Compression touches only the volatile suffix

Anything that does change lies strictly after the boundary. Two mechanisms enforce this and both are load-bearing. The recency carve-out anchors to the breakpoint. And query-aware salience is scoped to blocks after it — intent terms come from the newest user turn and change every turn by design, so letting them choose which lines survive inside an already-cached block would rewrite that block on every new question. That is the same cache bust from a second direction, and it is invisible to any single-request test.

(c) Digest determinism

A digest emitted for a block at turn N is byte-identical when the same block is forwarded at turn N+1. Handles are content-addressed — sha256(text)[:8], with no per-request nonce, counter, or timestamp. This was checked in the code rather than assumed, and it is asserted on its own, because a random handle would rewrite every digest stub on every turn and bust the cache by itself while every other clause still passed.

What is not guaranteed

• A client that rewrites its own history. If the client edits, re-orders, re-serialises, or moves its own cache_control marker, the input is not the same input and the contract does not apply. Distil cannot make an unstable client stable.
• Provider TTL. Cache entries expire on the provider's schedule. Byte-stability is necessary for a hit, not sufficient.
• A client that sends no cache marker at all. With nothing marked there is no Anthropic prefix to protect, so the plain last-k recency window applies and a block does go verbatim on one turn and digested on the next. Nothing is being invalidated, so this is intended. It is bounded rather than unbounded, and the test asserts the bound in both directions: the churn must occur strictly after the boundary, and it must still occur at all — a silently-dead carve-out fails too.

How it is enforced

tests/test_cache_contract.py replays growing six-turn synthetic sessions through the same public entry points the proxy calls — compress_messages, compress_chat_completions, compress_responses_input, and compress_generate_request — and fails on any same-input byte drift at or before the boundary. Three Anthropic marker placements are covered: pinned at the head, moved forward each turn, and absent entirely.

Measured at adoption, the contract holds on all three providers. That includes the realistic Claude Code shape, where the client pins its newest turn so the entire history is cached: there is no uncached tail in that configuration and nothing moves at all. No bug was found. The test codifies a property that was true and undefended.

Checking it against the provider

The clauses above are what Distil intends. The provider's own accounting is what actually happened, and distil dissect now reports it per session:

cache-read share: 87.4% of billed input was served from the
                  provider's prompt cache (at ~0.1x)

The number is cache_read_input_tokens over total billed input, taken from the fields the ledger already recorded. A near-zero share on a long session means something rewrote the prefix, and it is the only signal that says so — the requests all still succeed either way.

When the records predate those fields the line reads not captured and the JSON field is null, never 0.0. “We did not measure this” and “the cache never hit” are opposite diagnoses and must not share a rendering. This matters for a specific shape: older sessions carry only the aggregate usage_cache_tokens and neither split field, so summing reads over them gives zero against a nonzero input total — a confident 0.0% for a session nobody measured. At least one record must carry a split field before the number is reported at all.

The decision record is ADR 0008. For the mechanics of what gets compressed, see Techniques; for the adversarial side of the same surface, see the Threat Model.