Frequently Asked Questions

Honest answers to the questions that matter most — including the things Distil won't paper over.

"Is 100% decision-equivalence real, or is it marketing?"

It is real, in a specific and honest sense. 100% decision-equivalence means every turn in the benchmark trajectories produced the same decision (tool call name + arguments, or final answer) when using the Distil strategy versus the uncompressed baseline.

The default corpus gate uses a deterministic offline runner. The live head-to-head benchmark also confirmed this: graded by claude-opus-4-8 (majority-of-3) on a realistic 120-turn corpus, Distil certified 83.2% token savings at a 0% live decision-change rate (2026-07-05, distil 1.10.1 vs llmlingua 0.2.2 and headroom-ai 0.27.0) — with the non-inferiority bound (≤5% at 95% confidence) met with margin (hb_p = 0.0058). See the live head-to-head for the full methodology and caveats.

Three things the number depends on:

• Decision-equivalence, not byte-equivalence. The compressed context may differ byte-for-byte from the original. What is certified is that the agent's decisions match, not that two texts are identical strings.
• Decision-determined corpus. Certification only makes sense on traffic where the model reproduces its own decision on identical context — confirmed by the byte-exact = 0% precondition check. On ambiguous traffic the model varies by itself, and the number is undefined (that's the model's noise, not the compressor's fault).
• Marginal guarantee, not per-prompt promise. The non-inferiority bound holds over the calibration distribution. Generalization to new domains is a reasonable assumption, not a certificate — run distil ingest + distil bench --runner anthropic on your own traffic to certify your domain.

"How did you run the live benchmark, and how do I reproduce it?"

The live head-to-head used actual installed packages — llmlingua 0.2.2, headroom-ai 0.27.0, and Distil 0.23.0 — run against the same 120-turn, 5-domain corpus (4.5–6.5 KB/turn) where decisions are the agent's real next {action, target}. Each compressed context was graded by claude-opus-4-8 majority-of-3, independently of the compression step. Latency is wall-clock per turn on the same hardware. The Distil version is the one the run was executed on and is deliberately not bumped without re-running: these numbers have not been re-measured on a current release, so read them as a dated result rather than a claim about today's build.

Results in brief: Distil certified 83.2% savings / 0% decision-change / 0.026 ms per turn. LLMLingua-2 achieved 53.1% savings but flipped 1-in-8 decisions (fails the ≤5% gate). Headroom 0.27.0 passed the gate at 39.7% savings but runs a ModernBERT scorer at ~26 ms per turn. Those are 2026-07-05, distil 1.10.1 vs llmlingua 0.2.2 and headroom-ai 0.27.0. Against headroom-ai 0.37.0, re-run 2026-09-04: distil-causal takes 52.9% tokens / 58.7% dollars at 100% decision-equivalence (PASS) vs Headroom 1.7% / 2.0% / 81% (FAIL) on the warm corpus gate, and on the read→edit→re-read codebench workload Headroom takes 35.6% tokens for +4.9% dollars.

To reproduce:

# Run with pinned versions and the live model
python benchmarks/derc_live_compare.py

# Or run the same gate on your own corpus
export ANTHROPIC_API_KEY=sk-ant-…
distil benchmark --corpus benchmarks/corpus_xl --runner anthropic --tokenizer anthropic

Those three numbers are from 2026-06 and are superseded. A fresh head-to-head was run on 2026-09-04 against headroom-ai 0.37.0 [ml,code], llmlingua 0.2.2 and distil 1.51.1 on Python 3.12, with raw output committed at benchmarks/results/2026-09-04/: on the 9-domain corpus gate (warm, with Headroom's Kompress model preloaded) distil-causal reaches 52.9% tokens / 58.7% $ / 100% decision-equivalent / PASS against Headroom's 1.7% / 2.0% / 81% / FAIL. See the fresh head-to-head for the coding-agent workload too, where the ordering reverses for a reason worth reading.

Full methodology, corpus generation script, and caveats are in BENCHMARKS.md. For a step-by-step walkthrough of every headline number — all three CI gates, real-model grading via prove.py, the invariant scorecard, and the E7 negative result — see Reproduce the Numbers.

"How is this different from Headroom?"

Headroom is a real, adopted compressor and the closest thing to a peer. The difference is not the compression ratio — it is what each project is willing to be held to. Read directly against the public headroom-ai 0.37.0 source on 2026-09-04, with every line cited to a file:line in that release (the full audit):

• No per-request behavioural check. There is no shadow or dual-send path anywhere in the codebase. accuracy_guard="strict" is echoed back on /healthz and /stats, but nothing branches on its value.
• Recovery is a TTL cache, not a guarantee. The store expires entries after 1800 seconds with a 1000-entry FIFO cap and does no integrity or round-trip check on what it returns. Distil's is content-addressed, verified by a gate, and the agent can call it mid-task.
• Four lossy paths ship with no retrieval at all — OpenAI chat streaming, Responses under ChatGPT auth, Gemini streaming, and Bedrock. Distil's gateway ships Tier-0 only rather than emit a stub it cannot restore.
• Savings are uncalibrated, falling back to chars/3.5; distil reconciles its count against the provider's billed usage.
• Where it is genuinely strong, and we say so: its prompt-cache prefix replay is real engineering, and its breadth — 20 wrap targets, 6 framework integrations — is real.

On measured output, 2026-09-04: distil 52.9%/58.7%/100%/PASS vs Headroom 1.7%/2.0%/81%/FAIL on the corpus gate — and on a read→edit→re-read coding workload Headroom reaches 35.6% tokens where distil's digest is 0.0% by design, because a file the agent may have to quote back byte-exact is exempt from digestion. Both numbers are on one page, not just the flattering one.

"Isn't provider-native context compaction the same thing?"

No, and we measured it rather than argued it. distil certify-provider ran pre-registered, replicated experiments (n=40 per run) against both providers' own context management. Anthropic's default context-editing policy (keep=3) changed the agent's next action in 95–100% of cases against a 2.5% A/A noise floor — keeping the three most recent tool uses did not lower the change rate, it turned stalling into acting on missing facts. OpenAI's compaction changed 12.5–20%. Summarisation is roughly 7× safer than deletion, and neither certifies at α=0.1. The study, and how to re-run it on your own config →

If you run both, distil leaves the provider's output alone: Anthropic's signed compaction block and OpenAI's reasoning/compaction items are forwarded byte-identical and pinned by contract tests (cache contract, clause (g)).

The structural difference: provider compaction is server-side, on by default, and reports nothing about what it removed. Distil's folds are reversible, declared in-context, and the agent can pull the original back. The two are not alternatives — you can run distil precisely to measure what the provider's compaction is costing you.

"Does distil touch my tool definitions or MCP connectors?"

No. The request tools array is forwarded untouched. Tool definitions sit at the head of the cached prefix and bill almost entirely at the cache-read rate, so a lossless compactor would save little and one unstable byte would turn every cache read into a write (ADR 0012).

What distil does instead is stop switching off Claude Code's own fix. Claude Code defers MCP tool definitions and loads one only when the model searches for it, but disables that whenever ANTHROPIC_BASE_URL points somewhere non-first-party — which every distil wrap -- claude does. So wrap now sets ENABLE_TOOL_SEARCH=true for the child (your own ENABLE_TOOL_SEARCH=false still wins), distil default --always-on adds the key to Claude Code's settings only if it is absent, and distil forwards defer_loading and tool_reference blocks byte-identical. Verified live (2026-09-25): a wrapped Claude Code session that called an MCP tool recorded deferred tools on every request, cache reads intact, no failures. distil discover names MCP servers whose definitions were sent on 20 or more requests and never called. Why distil does not hide unused tools behind a meta-tool itself: ADR 0013.

"My status line says drift hold — what happened?"

Shadow mode's paired replays proved, with the anytime-valid e-process, that live decision change on your traffic crossed the risk budget (≤5% at 95% confidence). That alarm used to print and do nothing. Now every proxy on the machine serves the next request lossless-only — Tier-0, no digest, no output shaping — and keeps doing so across restarts, because a new session must not resume what the last one proved harmful. distil_expand stays available so stubs already in your history still resolve. Recalibrate on your own traffic with distil calibrate, then release with distil reset --drift-guard; running proxies resume within 30s and your savings history is untouched. DISTIL_NO_DRIFT_GUARD=1 keeps compressing through a breach (the alarm still prints). The multi-tenant gateway is exempt (ADR 0016). Details: what the exit summary proves.

"Is the per-commit CI gate the same as the live benchmark?"

No — they are complementary, and the README labels them as such. The per-commit gates (distil bench / distil verify / distil validate) grade decision-equivalence against an offline deterministic oracle over the committed corpus — fast, free, runs on every push, but synthetic. A nightly job (.github/workflows/live-cert.yml) re-certifies the same trajectories against a real model (distil certify --runner anthropic --model claude-haiku-4-5-20251001), with a hard --max-live-calls 60 ceiling per trajectory so an unattended run can never spend silently (the job fails at the ceiling rather than continuing to bill). The empirical head-to-head results in the README were graded by real models; the per-commit badge alone does not claim that.

"Is the compression lossless?"

Depends on the tier — the answer is precise:

The byte-fidelity gate (distil verify) confirms that every Tier-0 and Tier-1 operation across the corpus is byte-reversible and that frozen history never mutates between turns. The adversarial gate (distil validate) goes further — it drives the compressor against hostile inputs (huge/unicode/nested/malformed tool output, content that mimics distil's own << handle >> stubs, secret-looking strings) and asserts reversibility, reject-if-bigger, recency-exactness, fail-open, and content-free telemetry on every one. Both run in CI on every push.

"How accurate is the tokenizer?"

The default tokenizer is an offline heuristic — it estimates tokens by a character-based approximation without calling any API. This means:

• Compression ratios are robust. The numerator and denominator of the ratio use the same heuristic, so the ratio is a good estimate regardless of the absolute count.
• Absolute counts self-calibrate. distil is a proxy, so it sees the provider's real usage.* on every response. It learns the systematic correction between the heuristic and the real BPE — content-free, no network — and calibrates the absolute token figures to your model + content mix automatically. The leaderboard prints "calibrated to your billed usage (N requests, ±X%)" once enough traffic has flowed; below that threshold the factor is identity, so an uncalibrated ledger reports exactly the raw heuristic (no skew).
• tiktoken undercounts Claude. If you use OpenAI's tiktoken to count tokens for Claude pricing, you will systematically undercount — Claude's vocabulary differs. The self-calibration above corrects for this from real usage; --tokenizer anthropic gives per-string exactness.

Three offline choices, all zero-dependency except the last: --tokenizer heuristic (flat, the default), --tokenizer subword (length-aware — a closer BPE approximation, better as a pre-calibration base), and --tokenizer anthropic for per-string billing-grade counts on demand (e.g. offline benchmarking with no traffic to calibrate from):

pip install 'distil-llm[live]'
export ANTHROPIC_API_KEY=sk-ant-…
distil savings --tokenizer anthropic
distil bench   --tokenizer anthropic

This calls the Anthropic count_tokens API — accurate billing-grade counts, but requires network access and consumes a small amount of API quota.

"What is and isn't done?"

"Does it work with models other than Claude?"

The core techniques — cache-aware compression, causal pruning, TOST certification, byte-fidelity invariants — are model-agnostic and apply to any LLM with prompt caching. The corpus and adapter are built around the Anthropic Messages API format, but:

• The proxy intercepts /v1/chat/completions and /v1/responses in addition to /v1/messages, making it compatible with OpenAI-format APIs.
• A first-class OpenAI adapter (distil/adapters/openai.py) handles both Chat Completions and the Responses API. The same Tier-0/1 machinery and recency carve-out apply; the only difference is the shape-walking for OpenAI's role:"tool" content (bare string or list-of-parts) and the Responses API's input array. A duplicate image_url/input_image is elided to a reversible reference, same certificate gate as the Anthropic adapter (ADR 0003). Documented seams: expand-tool injection and output shaping for the Responses shape are not yet wired.
• Token counts and dollar figures use Anthropic pricing by default; other models can be added to the pricing catalog.
• The TOST gate and causal ablation are purely statistical and format-independent.

Google Gemini

The Gemini adapter (distil/adapters/gemini.py) intercepts generateContent and streamGenerateContent requests when the proxy runs with --upstream https://generativelanguage.googleapis.com. What works today:

• Request compression: text parts from non-model turns — Tier-0 lossless. functionResponse payloads — Tier-1 reversible digest. A duplicate inlineData/fileData image is elided to a reversible reference, same certificate gate as the Anthropic adapter (ADR 0003); first occurrence, functionCall, model-authored text, and systemInstruction pass through untouched.
• Recency carve-out: the most recent functionResponse turns are kept byte-exact (same as the Anthropic/OpenAI adapters), so the agent always sees its freshest tool output uncompressed.
• Query-aware intent: the agent's intent is extracted from the most recent functionCall arguments and used to pin relevant lines in large tool outputs — the same additive mechanism as the other adapters.
• Output shaping: pass shape="gemini" in the wrap/proxy flags to enable generation-side verbosity shaping tuned for Gemini's output structure.
• Savings reporting: x-distil-tokens-saved response header + savings ledger, same as all other adapters.
• Shadow-mode decision-equivalence: reads the chosen action from candidates[0].content.parts[].functionCall.
• --verbatim / --lossless-only modes: work identically to the Anthropic adapter.

Documented seams (not yet wired): expand-tool injection (--expand) for the Gemini tool shape, and cachedContent prefix caching. Output shaping via --shape-output is shipped and wired (uses shape="gemini" internally).

"Which compression mode should I use?"

The short answer: you usually don't need to choose. Run distil onboard once — it detects whether you're on a pay-per-use API key or a flat monthly subscription and writes the right setting into your shell. Every session after that just works. Pass a flag on the command line any time you want to override for a specific session.

Here's what each mode actually means in plain terms:

• digest (the default) — Distil shortens long things (big files, command output, past steps) into compact summaries, and can pull back the full original the moment the AI needs it. Originals are kept on your machine and restored automatically. You save the most, and nothing is truly gone. Most people should just use this.
• expand (--expand) — Same shortening as digest, but Distil also gives the AI a "show me the full version" button it can press on its own. Best when the AI runs for a long time on its own — long coding sessions, autonomous tasks. Picked automatically if you pay per use (API key).
• lossless-only (--lossless-only / --safe) — The cautious setting: Distil only trims things it can rebuild perfectly (like extra blank space), and never summarizes. You save less, but there is zero chance of losing any detail. Picked automatically on a flat monthly subscription.
• verbatim (--verbatim) — The lightest touch: just tidies formatting, changes nothing else. Almost no savings. Use it when you want to see or audit exactly what's being sent.

For the technical breakdown:

Subscription users should not force --expand — it crosses the lossless safety boundary. Coding re-reads? Add --session-delta regardless of mode.

"What does de / decision-equivalence mean, and when should I run --shadow 1.0?"

de is the decision-equivalence rate — the fraction of shadow-checked requests where the agent made the same next action (tool call) with and without compression. It appears in the status line as de n/50 while collecting samples, then as a ✓/⚠/✗ percentage once enough data has accrued.

Gate mechanics: a verdict requires 50 A/B samples (compressed-vs-original) plus 30 A/A samples (same request replayed against itself — the sampling-noise baseline). The A/A arm measures how often the model disagrees with itself on identical input, and the verdict is the paired difference against it. Replays are not always at temperature 0 — on Claude Code traffic they usually run hot — which is exactly why the A/A arm is there. Shadow sampling is on by default at 2% of requests — at that rate, the gate takes days to fill on typical traffic.

Live result — 2026-09-15, build 1.53.0rc1, paired estimator (signature v5). The sample now clears the reporting floor: 398 A/B and 399 A/A samples, both modes mixed (digest 206, lossless-only 192), paired equivalence 97.5% [95.5, 99.5] from a paired difference of −0.025 [−0.045, −0.005]. Under 99%, which is why the status line prints ⚠de 97.5% (398) rather than a ✓. Replays run hot — 399 of 399, temperature is not pinned — so the paired difference is the statistic to read, not the 53.0% raw agreement; the A/A arm carries the same nondeterminism and is what the difference subtracts out. Artifact: benchmarks/results/shadow-live-2026-09-15.json. Two earlier readings stay on the record: the 1.13.0 number (100% over 116 sampled requests, A/A 31/31) is withdrawn — the 1.51.1 changelog found that every replay carrying a prior thinking block failed with a signature error, biasing the sample — and the 1.51.1 reading (44 A/B, 11 A/A, raw agreement 81.8%, below the floor, unpaired estimator) is kept at benchmarks/results/shadow-live-2026-09-04.json.

--shadow 1.0 (high-fidelity validation) samples every request — every turn re-issues 2 extra upstream calls (~3× tokens). This fills the 50 A/B + 30 A/A gate in minutes rather than days, proving decision-equivalence quickly. It is a validation setting, not for daily use — drop back to the default 2% once you've confirmed equivalence:

distil wrap --shadow 1.0 -- claude   # validate; accrues gate in minutes
distil shadow-stats                  # check the de rate
# once equivalence is confirmed, remove --shadow 1.0 and use the 2% default

"Where can I see community savings?"

By design, savings are local-first and privacy-preserving — Distil never sends your prompts or responses anywhere. Three views:

• distil leaderboard — your own cumulative savings. The proxy records real per-request token savings from live traffic into the ledger (the live-proxy source), so this reflects actual usage, not benchmark projections.
• distil federated-leaderboard --dir ./submissions --keys keys.json --html out.html — a verifiable multi-instance leaderboard built from HMAC-signed, content-free aggregates (numbers + certification verdict only). Savings you can verify, not just claim.
• distil leaderboard --html your-savings.html — render your real ledger as a self-contained page (genuine numbers from your usage, including the live-proxy source — no dummy data).

There is no hosted public dashboard harvesting your data. Opt-in federation means you control submission and verification (ed25519 public-key signing is the documented upgrade for share-the-key-free verification).

"What is the license?"

Apache-2.0. See LICENSE in the repository.

You can use it commercially, modify it, distribute it, and sublicense it. The only requirements are attribution and preservation of the license notice. No patent traps beyond the standard Apache-2.0 patent grant.

"distil wrap ran but I see no savings — what went wrong?"

When distil wrap ends a session with zero proxied requests and the agent is one of the recognized presets (distil wrap --list), it prints a warning. If a Claude Code settings file pins the base URL over the wrap's environment, it names that file instead:

  warning: no requests flowed through distil this session — claude may have
  stopped honoring ANTHROPIC_BASE_URL (agent update?). Fix: distil doctor

This is the upstream-contract tripwire: distil's interception relies on the agent honoring the injected env var (ANTHROPIC_BASE_URL etc.), which is not a documented stable API. An agent update can silently stop honoring it. Common causes:

• Agent updated and now pins its own endpoint regardless of the env var. Restart the wrap and watch for the env var line at startup (→ ANTHROPIC_BASE_URL=http://127.0.0.1:…).
• OAuth / subscription login overrides the base URL. Claude Code on a Pro/Max subscription occasionally ignores the env var for that session. Restart the wrap — the next session usually picks it up.
• Wrong agent name. The preset only fires when argv[0] matches a name in distil wrap --list. If you aliased the command, pass --env-var (and --upstream) explicitly.
• Codex. Its preset is unverified: codex-rs reads its base URL from the openai_base_url config key, and no mapping from OPENAI_BASE_URL was found. A zero-traffic warning on codex is most likely this.

Run distil doctor for a wiring self-test, or DISTIL_DEBUG=1 distil wrap -- <agent> to see the full proxy log.

"Does distil break streaming?"

No. SSE (Server-Sent Events) responses are relayed chunk-by-chunk as the upstream produces them — the proxy, the async proxy, and the gateway all route through the same relay (distil/streamrelay.py), so Distil preserves time-to-first-token like a direct connection. Nothing is buffered start-to-finish before it reaches the client.

Content-free accounting is teed off that same stream rather than requiring a separate buffered pass: the complete body is assembled in memory as it streams by, so token-saved headers and shadow-mode decision comparison run exactly as they would on a non-streaming response. A slow or unreachable upstream is bounded by DISTIL_UPSTREAM_TIMEOUT (default 600s) and maps to a 504 — see Deploy & Security → Streaming and upstream timeouts.

"Can I contribute a new compression strategy?"

Yes — PRs are welcome. The rule is simple: a new strategy must pass make gate (non-inferior on every corpus domain, byte-reversible on Tier-0/1 operations). No green gate, no merge. See CONTRIBUTING.md for details.