Evaluation

Every compressor asks you to trust it. This page is what distil measures instead — including the numbers that make distil look worse, because a suite that only ever produces good news is a suite nobody should believe.

Why compression ratio is not a result

Token reduction and task success are not linearly related, and the relationship can cliff-edge. One study found 4× compression collapsing SWE-bench resolution from 86% to 7% — not a decline, a cliff.

We hit it ourselves. E7 ran the full SWE-bench Verified agent loop under distil's aggressive tier and measured real resolution: 52% → 16%. The reversible tier held (56% vs 52% full-context). Per-step decision-equivalence had already passed. End-to-end task success still cratered.

We publish E7 as evidence, not confession. It is why the shipped default is the conservative tier rather than the one that wins a compression-ratio leaderboard.

The rule that follows: every compression claim reports token savings and a task-success delta together, at the setting we actually ship. A ratio with no success number beside it is an invitation to find the cliff in production.

Five layers, five different questions

All five are offline, need no API key, and run in seconds. That is deliberate: a gate you skip because it costs money is not a gate. All five run per-commit in CI.

The case recall cannot see

Recall asks "is the fact still there". Here is a trajectory where the answer is yes and the agent is still broken:

turn 2   Write(file_path="net/scratch_bench.py")
turn 4   rm net/scratch_bench.py

Compress away turn 4. Every path token is still present — string recall reads 100% — while the agent now believes a file exists that does not, and will plan around it. Nothing in the transcript says otherwise.

distil fidelity folds tool calls into a file-state ledger and grades the final state, splitting two outcomes that presence metrics conflate:

These are never averaged into one accuracy figure, because a compressor that drops a whole file history is safer than one that preserves half of it. On the case above: string recall 100%, state fidelity 0%.

Factory.ai measured every method they tested at 2.19–2.45 / 5.0 on this axis across 36,611 production engineering messages. Presence-based metrics are structurally unable to detect it — the string is present.

Three more things recall misses

Overclaim. "approximately 4200 ms" → "4200 ms". Byte-identical value, and every recall metric scores it perfect — but the agent has been handed a precision the source never asserted. The hedge was the information. Hedges are grouped into classes so reshaping (approximately → about) is not penalised; only the disappearance of hedging is. Direction is asymmetric: overclaim is gated, underclaim only reported.

Continuation. Whether the agent still knows what is left to do. Dropping a completed item is cheap — work gets redone. Dropping a pending one is silent: the work is skipped and success is reported anyway.

Propagation. Whether a loss at turn k shows up as a behaviour change at turn k+n, as a lag-lift profile. Two limits are printed in the tool's own output: it is association rather than causation, and periodic workloads alias at multiples of their period.

The numbers, on the shipped tier

$ distil fidelity
artifact-state fidelity   100.0%  (7/7 artifacts intact)
  stale (wrong state)       0.0%  (0)
  lost  (absent)            0.0%  (0)

hedge fidelity             94.7%  (162/171 claims kept their hedging)
  overclaimed               5.7%  (9)  <- value kept, uncertainty dropped

pending-work recall       100.0%  (2/2 remaining items still visible)

error propagation  (36 turns, base decision-change rate 0.0%)
  verdict: no decisions changed at all — nothing to propagate, and nothing tested

OUTPUT surface — past answers digested on re-entry (the other half of the bill)
  (6 blocks changed by digestion, 4 facts removed from them — the graded evidence)
  artifact state         100.0%  (7/7)   stale 0
  hedge fidelity         100.0%  (6/6)   overclaimed 0
  pending-work recall    100.0%  (2/2)   dropped 0

silent failures, BOTH surfaces: 9  (input 9, output 0)

Those 9 overclaims are real, and we ship the number rather than tune it away. Tier-1 digests spans behind restore handles, and a digested hedged sentence can keep its value while losing its qualifier. Unlike a lost fact this is not recoverable in practice: a missing fact prompts the agent to expand, a missing hedge gives it no reason to look.

So the CI gate is --max-silent 15, not 0. Gating at zero would assert a property the compressor does not have. The band fails a regression that doubles it — and fails an improvement too, forcing the number to be re-read rather than drifting.

Public benchmarks — falsifiable by a stranger

Every other number on this page is graded against distil's own corpus. Rigorous, and unfalsifiable by someone who doesn't trust us. distil suite closes that: twelve public benchmarks whose answer keys were written by other people.

It costs nothing to run. Grading is deterministic recall against the answer key — no model in the loop, no API key, no spend. Suites that grade with an LLM judge cost real money per tier, which makes them something you run before a launch rather than before a merge. This one is wired into make gate and the CI gate job.

The payload column is the important one. A GSM8K case is a one-line word problem: there is nothing in it to compress, so an unchanged score proves the compressor left it alone. That is a control. Only rich-payload rows — tool schemas, retrieved passages, long-form narrative, code — can demonstrate compression quality. A run that grades only controls exits 1.

bfcl leads tier 1 deliberately. Berkeley Function Calling compresses the tool schema and checks that every name the gold call is built from — the function and each argument it passes — survives compression: the failure an agent proxy is most likely to cause and least likely to notice, because no QA benchmark ever asks the model to act. The gold call itself is not graded as an answer — it never appears in the schema text, and checking whether a model still emits it would need a model in the loop, which would stop this suite running in the CI gate. What is measured is stated exactly.

uv run distil suite --tier 1              # tool-calling + retrieval
uv run distil suite --tier 1 --tier 2     # add the harder payloads
uv run distil suite --only bfcl -n 50     # one benchmark
uv run distil suite --json --max-lost 0   # gate

A benchmark that cannot be fetched is reported as a FAILED row and exits 1 — never dropped. A suite that silently skips what it could not load reports a clean sheet for a run that measured less than it claimed.

Every run is a record, not a number

--json emits an envelope, validated against a published JSON Schema on every test run:

{
  "schema":  "distil.eval/1",
  "subject": { "compressor": "serving", "module": "distil.compress.strategies" },
  "dataset": { "trajectories": 9, "fingerprint": "sha256:051b836358932883" },
  "grader":  { "kind": "deterministic",
               "detail": "synthetic DECISION: oracle — NOT a model" },
  "metrics": { ... },
  "gates":   [ { "name": "max_silent", "threshold": 15, "observed": 9,
                 "passed": true, "rationale": "..." } ],
  "passed":  true
}

• fingerprint — order-independent content hash of exactly what was graded. Adding a trajectory changes every number; without this that reads as a compressor regression.
• grader — a synthetic oracle is never reported as a model. That conflation is what makes a result look stronger than it is.
• gates — threshold, observed value, outcome and rationale. A bound with no reason attached is one somebody "fixes" later without knowing what it protected. An empty gate list is explicitly not a pass: nothing was checked.

On your own traffic

The offline gates are graded on our corpus against our oracle — rigorous, but not checkable by you. Two things close that:

# graded against a PUBLIC benchmark's answer key, not ours
distil retention --dataset hotpotqa

# A/B on YOUR traffic, with an A/A control
distil wrap --shadow claude
distil shadow-stats --record

Shadow mode replays the same compressed request to measure the grader's own disagreement with itself (A/A), then subtracts that floor from the A/B divergence. Without it you would read sampling noise as compression harm. --record emits the same envelope as the offline gates, so a live result is as attributable as a corpus one — with a traffic-window descriptor in place of a content hash, because live traffic has none and must never grow one.

Everything here is content-free. The meters store counts; no prompt, path or tool output leaves your machine.

What the probes found in our own work

An eval suite's real test is whether it catches its authors. These are recorded in full in the paper:

• The corpus was certifying nothing. Across eight trajectories: 4 file operations and 0 stated obligations. All three state probes reported 100% against almost no evidence — the same shape as the HTML transform reporting 0% savings before any trajectory carried HTML. A coverage test now fails if the corpus stops carrying enough to grade.
• The harness graded no-ops at 100%. The compressors take a list of blocks; the runner passed strings, every call raised, and a blanket exception handler substituted the original — so each input was compared against itself. The handler is gone: a compressor that cannot run now fails the gate.
• The overclaim metric produced 24 false findings before 9 real ones. Three false-positive classes, each found by inspecting instances rather than trusting the aggregate.
• The gate graded a surface users never receive. It compressed every block with a bare tier; the serving path leaves the stable prefix untouched and compresses only the volatile tail. Found by cross-audit.

Each of those is a failure the probes exist to detect, occurring in the probes. We publish them because the alternative — a suite that has never been wrong — is the less trustworthy artifact.

What the exit summary proves

Every distil wrap session ends with a proof ledger, and the same four verdicts appear in distil stats and distil dissect — one function behind all three, so two surfaces reading one ledger cannot disagree about it. Each line is allowed to come back negative. That is the only reason any of them is worth printing.

  distil proof ledger — session 47m        # the four lines, with YOUR traffic's numbers
    budget   decision-change budget: intact (e-value <E>, n=<N>) — bound within the <B>% budget
    risk     decision-change risk ≤ <P>% (95% conformal bound, n=<N>) — within the <B>% budget
    output   the model's replies were <T> tokens shorter per request under compression
             (95% CI [<lo>, <hi>], n=<N>, shadow-measured on all traffic; not --shape-output)
    receipts <N> receipts, chain intact — <k> re-checked since this machine's last pass; `distil receipts` re-hashes all

No sample values are shown here on purpose: these are the only numbers on this site that are not published measurements, because they are yours. Each line can also read BREACHED at sample k … compression held at lossless-only, unproven … the bound is above the budget, ABOVE the budget, chain BROKEN at receipt k, or no measurable effect on reply length.

• budget — is the certified risk budget still holding? The certificate is a one-shot statement about a calibration corpus; live traffic drifts away from it. This line runs a betting e-process for the null "decision-change ≤ 5%": capital that crosses 20 means the budget is breached at 95% confidence. Method: under the null the capital is a non-negative supermartingale starting at 1, so by Ville's inequality the chance it ever crosses 1/δ is at most δ — which is what makes it safe to check after every single request, with no penalty for peeking. A null simulation of 2,000 runs at exactly the budget is a build gate; it alarms on 99 of those 2,000 runs (4.95%, against a δ of 5%), and it catches a true rate ten points over budget on 500 of 500 runs, within a median of 172 requests. It acts, not just reports: there is one e-process per machine, in ~/.distil/drift.json. Each proxy folds the paired verdict it just produced into it under a file lock, so a restart, a hot-swap or a second wrap continues the same capital instead of taking another look at the same evidence. The report surfaces only read it. The request after a breach is served lossless-only: Tier-0, no digest, no output shaping. The trip is written to the receipt chain. It holds across restarts for every session, because a new distil wrap must not resume what the last one proved harmful. A running proxy notices another process's trip, or a release, within 30s. It shows in the status line as ⚠ drift hold. It clears with distil reset --drift-guard after distil calibrate, and that release leaves savings and shadow stats alone. A state file that cannot be read is treated as held, and copied aside before anything replaces it, because it may have recorded a breach. DISTIL_NO_DRIFT_GUARD=1 opts out of the hold; the line then says compression was not held. The multi-tenant gateway runs no shadow and is deliberately exempt (ADR 0016). The hold is new since 1.53.0rc1 and, like every default-on change, still has to clear an rc soak. "intact" is earned: the e-process not having tripped means no breach is proven, which is weaker than "within budget" — so the line prints intact only when the risk bound beside it is inside the budget, and unproven otherwise.
• risk — how bad could the live rate be? An upper bound on the decision-change rate, computed from the same paired losses the budget line bets on, and compared against the same budget. Every budget on this page is one constant, distil.conformal.BUDGET_ALPHA, read by the certificate, this line, the budget line and the proxy's hold alike. Method: a betting (Waudby-Smith–Ramdas) confidence bound, which assumes no distribution and holds at finite n. It is deliberately wider than the bootstrap interval printed next to it: the bootstrap is an estimate of where the rate is, this is a statement about where it is not. A 1,000-run coverage simulation is a build gate.
• output — what did compression do to reply length? A shorter prompt that buys a longer answer can cost more than it saved, because output is priced several times input. This is measured on shadow's paired replays, on traffic that asked for nothing; it is not --shape-output, which asks the model for shorter replies. Method: paired bootstrap over per-request output-token differences. The direction word is printed only when the 95% interval excludes zero — otherwise the line says there is no measurable effect.
• receipts — is the audit trail intact? Each request writes a content-free receipt, hash-chained so a receipt cannot be edited, reordered or removed without breaking every receipt after it. Method: recompute every hash and every link; anyone can run it with nothing but the file (distil receipts --verify). A break names the receipt it fails at. The chain is sealed into segments, each with a Merkle-root checkpoint, so an auditor can check one receipt from an inclusion proof and a pinned checkpoint without the rest of the log (distil receipts --prove / --check-proof). A line that is not a receipt at all — a torn write after a crash — does not break the chain, but the line then reads VERIFIED WITH GAPS and counts what was skipped.

Below the reporting floor, every line withholds its number and prints how far along it is instead — not enough samples yet (12/50). A verdict computed over evidence too thin to support it is worse than no verdict, because it teaches the reader to ignore the line.

The numbers above are the shape of the output, not a claim. The maintainer's own sample — 398 A/B replays against 399 A/A controls, published content-free — puts compression at 2.5 pp of decision agreement, 95% CI [0.5, 4.5] pp, inside the 5% budget, with replies 68.3 tokens shorter per request, 95% CI [-147.2, -2.9]. Your traffic is not that traffic, which is the entire reason the lines are computed on yours.

Run it yourself

$ uvx --from distil-llm distil bench      # ~10s, no API key
$ uvx --from distil-llm distil fidelity   # the state probes
$ make gate                              # everything CI runs

Full methodology in EVALUATION.md; step-by-step instructions in RUNNING-EVALS.md.