Prompt caching

Provider prompt caching is the largest single lever on an agent's bill, and it is all-or-nothing: the stable prefix must be byte-identical between turns, or the whole cached span is re-billed at full price. One changed character at the front costs the entire prefix.

Compression cannot break the prefix

This is a property of the design, not a promise. strategies.distil compresses the volatile tail — the newest user turn and the freshest tool output — and leaves every stable block byte-for-byte untouched. If it ever rewrote a stable block, distil verify would fail on reversibility and distil bench on decision-equivalence, so the guarantee is enforced by the same gates as everything else.

Two further things distil does with the cache:

• Marks the boundary. adapters.anthropic.place_cache_control places the breakpoint at the end of the stable prefix. A stable prefix nobody marked is a discount nobody claims.
• Sends re-reads as deltas. --session-delta keeps a per-session map of what has already been sent, so a file re-read after an edit goes as a diff rather than a fresh copy. Different lever, composes with the cache: the cache makes repeats cheap, the delta stops them being sent.

What distil cache reports

Deliberately two different kinds of number. Reads and writes come from the provider's own usage — ground truth about money. Drift is our diagnosis of why, from a content-free hash of the stable blocks we sent. A diagnosis with no measurement behind it is a guess, so the report never prints one without the other.

Three live requests, the third with a session id prepended to the system prompt:

requests        3
cache reads     31,614 tokens  (billed at a discount)
cache writes    15,819 tokens  (billed at a surcharge)
uncached        40 tokens
hit ratio       66.6% of cacheable tokens were reads
prefix drift    1 of 2 turns changed the stable prefix (50%)
                Each one re-bills the whole prefix. distil never rewrites a
                stable block, so the cause is upstream: a timestamp or session
                id in the system prompt, or a tool list whose order varies.

The two halves are derived independently and agree: the turn our hash flagged is the turn the provider re-billed 15,819 tokens to re-create. Turns 1 and 2 read the cache even though messages grew between them — which is the property that matters. A conversation is supposed to grow, and a diagnostic that called that drift would fire on every healthy turn and be switched off within a day.

Two readings that are easy to misinterpret

• “not reported by the provider” means the response carried no cache usage at all — usually the prefix is under the model's minimum, or the client never sent a breakpoint. It does not mean a miss.
• “no comparable pairs” means fewer than two requests carried a prefix hash; rows written before 1.41 do not. That is not the same as no drift, and it says so rather than printing a reassuring zero.

With no proxied requests at all, distil cache exits non-zero. A cache report over zero rows can print a perfectly plausible “0 drifts” and read as a clean bill of health for a session nobody measured.

The one cache feature we deliberately do not ship

A response cache — hash the request, serve a stored response, skip inference entirely. distil has no equivalent, and that is a decision rather than a gap:

• Agent contexts grow monotonically. Every turn appends. Two requests being byte-identical is rare outside retries and replays, so the hit rate is lowest exactly where the bill is highest.
• The correctness surface is sharp. The key must include the system prompt, tool set, sampling config and output shape, or two requests with identical messages collide and the second caller is served the first's answer. That is a wrong-answer bug, not a slow one.
• It is the wrong layer. A compression proxy narrows what a request costs. Deciding a request need not run at all is the application's call, with the application's knowledge of staleness — and it can be built above distil without distil owning the risk.

Worth noting: the implementations we compared ship this off by default too.

Cold prefixes

Byte-faithful forwarding only pays while the cache is warm. Once a session idles past the provider TTL (Anthropic 5 minutes, extended on each hit), resending the prefix verbatim buys nothing — which makes that the safe moment to rewrite it, since there is no warm cache left to destroy. distil's serving strategy is conservative here and does not recompact cold prefixes today; distil cache is what would tell you whether it is worth it for your traffic.

Checking your own setup

distil cache                  # hit ratio + prefix drift over recent sessions
distil cache --session <sid>  # one session
distil cache --json           # machine-readable, for a CI check
distil doctor                 # what the proxy is actually doing
distil stats                  # savings, last-7-days rate beside lifetime
distil proxy --session-delta  # cross-turn dedupe + cross-version deltas

distil cache reads the per-session request ledger, which only a wrap session writes. Run your agent under distil wrap -- <cmd> (or distil default --always-on), then re-run it.

If distil stats shows near-zero recent compression, the cause is usually mode rather than cache: a subscription session defaults to lossless-only, which is Tier-0 only. distil default --mode expand turns the recoverable digest on permanently.