What an outside benchmark found

In August 2026 an engineer ran distil against bare Claude Code across 75 agent sessions and measured the result from the API’s own usage fields rather than from any tool’s dashboard. One distil configuration finished 6 of 15 coding tasks where bare Claude Code finished 13. This page is what they found, why it happened, and what changed in 1.49.0.

We publish it for the same reason we publish the bugs our own evaluation harness had: a compressor whose public record contains no failures is not a compressor that never failed, it is one that never looked. The measurement that matters is the one taken by someone with no stake in the answer.

The result

Cache writes are the number that decides cost on a cached workload, and they are taken from cache_creation in the provider’s response — not from distil’s own accounting, which is the thing under test.

The failure that mattered

Seven --expand runs ended with an empty diff: the agent ran for 18, 22, even 51 turns, wrote nothing to disk, and reported success. On the long refactor it did this all three times — and the repository’s 1,371 existing tests still passed, because nothing had been modified. A benchmark that graded on “did the suite stay green” would have scored those runs as wins.

No other arm produced a single empty diff in 60 runs.

Cause 1 — the agent never ran its own tool call

Claude Code emits parallel tool calls, so one assistant message routinely carries both an Edit and a distil_expand. distil’s recovery loop assumed distil_expand was the only tool call in the turn. It answered just that block and re-queried — which left the client’s Edit as an unanswered tool_use upstream, and let the continuation’s stop_reason: end_turn replace the turn’s real tool_use.

Claude Code executes tools only when stop_reason == "tool_use". So it received the Edit, ran nothing, printed the continuation’s “all done”, and finished. Fixed: a turn carrying a client tool call is now terminal — relayed verbatim, stop_reason intact, no re-query.

Cause 2 — a failed recovery truncated the stream

When the re-query failed, the connection closed with no message_delta and no message_stop. A terminator-less SSE message is truncated, not finished, so the SDK retries it invisibly: wall-clock burned, no progress, and no error the user can see. That is the benchmark’s 808-second, zero-write signature. Fixed: every exit emits a terminator.

Cause 3 — a file read could be digested before it was edited

Recency is positional, so a file read three turns ago was eligible for digesting. But the agent must still reproduce that text character-for-character in an Edit(old_string=…). Once the read is a digest, no exact match exists. Fixed: the exemption is now keyed on provenance — results answering Read/Grep/Glob and their MCP equivalents stay byte-exact at any age. Logs and test output still compress normally.

Compression that saved nothing still cost cache

On a subscription distil stays lossless-only by design and correctly reported 0.0% savings — while still writing 1.56× the baseline’s cache-creation tokens, and 2.52× on a short session. Doing nothing to the content is not the same as doing nothing to the cache.

Two causes, both fixed. The distil_expand tool was injected only once a recoverable handle existed, so the tools array — which Anthropic caches ahead of the system prompt and all history — changed shape on the turn compression first fired. And an unmodified request body was re-serialized rather than forwarded byte-for-byte; json.dumps is not a byte-faithful round-trip, so the prefix moved even when nothing had been compressed.

Worse, distil’s own drift report sent users hunting in the wrong place: it blamed “a tool list whose order varies” upstream, while distil’s tool list was the one varying. It now names distil’s own causes first.

The mode that won had nothing to show for it

Hook mode was the strongest distil arm measured — 12/15 tasks, 0.94× baseline cache writes, the highest cache-hit rate of any configuration — and the only one that wrote no ledger at all. Its effect was visible solely in the provider’s billing, which is an awkward position for a product whose claim is that it proves its numbers instead of asking to be trusted.

It now writes a content-free receipt per compressed result:

distil hook --install     # documented Claude Code extension point
distil hook --stats       # what it actually saved

What this does not establish

The benchmark’s own caveat is worth repeating: an empty diff proves the agent stopped early, not that compression caused it. The runs used --shadow 0.0 to conserve tokens, so distil’s decision-equivalence sampler — the tool built for exactly this question — captured nothing. We have since reproduced the mechanism locally, without API spend, and pinned it with regression tests that fail without their fixes. But the association came first and the proof came second, which is the right order to admit to.

The lesson we are keeping: a green test suite does not prove the work was done. Acceptance has to check that the required change exists, not merely that nothing broke.

Related: the bugs our own harness had · negative results · why removing tokens can cost more · what “savings” means on a flat-rate plan