Frequently Asked Questions
Honest answers to the questions that matter most — including the things Distil won't paper over.
"Is 100% decision-equivalence real, or is it marketing?"
It is real, in a specific and honest sense. 100% decision-equivalence means every turn in the benchmark trajectories produced the same decision (tool call name + arguments, or final answer) when using the Distil strategy versus the uncompressed baseline.
The default corpus gate uses a deterministic offline runner. The live head-to-head benchmark also confirmed this: graded by claude-opus-4-8 (majority-of-3) on a realistic 120-turn corpus, Distil certified 83.2% token savings at a 0% live decision-change rate — with the non-inferiority bound (≤5% at 95% confidence) met with margin (hb_p = 0.0058). See the live head-to-head for the full methodology and caveats.
Three things the number depends on:
- Decision-equivalence, not byte-equivalence. The compressed context may differ byte-for-byte from the original. What is certified is that the agent's decisions match, not that two texts are identical strings.
- Decision-determined corpus. Certification only makes sense on traffic where the model reproduces its own decision on identical context — confirmed by the
byte-exact = 0%precondition check. On ambiguous traffic the model varies by itself, and the number is undefined (that's the model's noise, not the compressor's fault). - Marginal guarantee, not per-prompt promise. The non-inferiority bound holds over the calibration distribution. Generalization to new domains is a reasonable assumption, not a certificate — run
distil ingest+distil bench --runner anthropicon your own traffic to certify your domain.
"How did you run the live benchmark, and how do I reproduce it?"
The live head-to-head used actual installed packages — llmlingua 0.2.2, headroom-ai 0.27.0, and Distil 0.23.0 — run against the same 120-turn, 5-domain corpus (4.5–6.5 KB/turn) where decisions are the agent's real next {action, target}. Each compressed context was graded by claude-opus-4-8 majority-of-3, independently of the compression step. Latency is wall-clock per turn on the same hardware.
Results in brief: Distil certified 83.2% savings / 0% decision-change / 0.026 ms per turn. LLMLingua-2 achieved 53.1% savings but flipped 1-in-5 decisions (fails the ≤5% gate). Headroom passed the gate at 35.3% savings but runs a ModernBERT scorer at ~26 ms per turn.
To reproduce:
# Run with pinned versions and the live model python benchmarks/derc_live_compare.py # Or run the same gate on your own corpus export ANTHROPIC_API_KEY=sk-ant-… distil benchmark --corpus benchmarks/corpus_xl --runner anthropic --tokenizer anthropic
Full methodology, corpus generation script, and caveats are in BENCHMARKS.md.
"Is the compression lossless?"
Depends on the tier — the answer is precise:
| Tier | Lossless? | How |
|---|---|---|
| Tier 0 — JSON minify, RLE | Provably lossless | Reconstructable by inversion — no side state needed. The original can always be recovered from the transform output alone. |
| Tier 1 — Digest + handles | Reversible | The original is stored in a local RestoreStore keyed by an 8-hex SHA-256 handle embedded in the compressed text. Recovery requires the RestoreStore, which is kept locally and never sent to the model. |
| Certified — Pruning | Not lossless — but causally certified | Pruned blocks are causally inert: ablation confirmed they never changed any decision. The bytes are dropped but agent behaviour is preserved within the non-inferiority margin. |
The byte-fidelity gate (distil verify) confirms that every Tier-0 and Tier-1 operation across the corpus is byte-reversible and that frozen history never mutates between turns.
"How accurate is the tokenizer?"
The default tokenizer is an offline heuristic — it estimates tokens by a character-based approximation without calling any API. This means:
- Compression ratios are robust. The numerator and denominator of the ratio use the same heuristic, so the ratio is a good estimate regardless of the absolute count.
- Absolute dollar figures are approximate. The heuristic is not calibrated to Claude's BPE tokenizer. Dollar amounts are directional, not billing-grade.
- tiktoken undercounts Claude. If you use OpenAI's tiktoken to count tokens for Claude pricing, you will systematically undercount — Claude's vocabulary differs. Use
--tokenizer anthropicfor accurate counts.
To get billing-grade token counts:
pip install 'distil-llm[live]'
export ANTHROPIC_API_KEY=sk-ant-…
distil savings --tokenizer anthropic
distil bench --tokenizer anthropic
This calls the Anthropic count_tokens API — accurate billing-grade counts, but requires network access and consumes a small amount of API quota.
"What is and isn't done?"
| Item | Status |
|---|---|
| Real tokenizer + live runner (billing-grade) | Done — --tokenizer anthropic / --runner anthropic |
| Multi-domain trajectory corpus + CI gate | Done — distil bench |
| Runtime adapter (no-code-change compression) | Done — wrap(client) |
| Auth-mode gating | Done — distil/policy.py |
| Holdout A/B with confidence intervals | Done — distil holdout |
| Byte-fidelity invariants | Done — distil verify |
| BM25 partial retrieval · delta context · keep-model codec · gist caching | Done — all wired, real implementations |
| Provider proxy + managed gateway | Done — distil proxy / distil gateway (live per-tenant dashboard) |
| Google Gemini adapter (request compression + decision-equivalence) | Done — distil/adapters/gemini.py; proxy with --upstream https://generativelanguage.googleapis.com. Gaps: --expand, --shape-output, context caching not yet wired. |
| Output compression (input and output) | Done — gated shaping + lossless re-entry digest, distil output-savings |
| Learned keep-model (logistic + transformer) | Done — logistic ships built-in (96.4%/0.98 on held-out lines, labels from the heuristic); transformer adapter + training pipeline + demo checkpoint |
| Certified compression frontier (proof pack) | Done — distil eval |
| Trajectory-level task-outcome certificate (bounds end-to-end degradation, not just per-step decisions) | Done — distil certify-trajectories; see Concepts |
| Self-distilling keep-model (causal labels, never-regressing) | Done — distil online |
| Outcome-guided compression policy (protects content classes that break tasks when digested) | Done — distil/compress/guideline.py, fed by matched certify-trajectories outcomes |
| Verifiable federated telemetry | Done — distil federated-leaderboard |
| Async high-concurrency proxy · Rust hot-path core | Done — distil proxy --async · rust/distil-core (python parity fallback) |
| Real-trace ingestion · performance benchmark | Done — distil ingest · distil perf |
| Live head-to-head vs. real competitor packages | Done — python benchmarks/derc_live_compare.py; graded by claude-opus-4-8, majority-of-3. See Benchmark. |
| Production keep-model checkpoint | Your turn — a demo checkpoint ships on the release; production = retrain on your traces (distil train-transformer) |
| Proven at scale / adoption | Needs real traffic — every tool to prove it ships (distil ingest + distil eval --runner anthropic); run it on your traces and publish |
"Does it work with models other than Claude?"
The core techniques — cache-aware compression, causal pruning, TOST certification, byte-fidelity invariants — are model-agnostic and apply to any LLM with prompt caching. The corpus and adapter are built around the Anthropic Messages API format, but:
- The proxy intercepts
/v1/chat/completionsand/v1/responsesin addition to/v1/messages, making it compatible with OpenAI-format APIs. - Token counts and dollar figures use Anthropic pricing by default; other models can be added to the pricing catalog.
- The TOST gate and causal ablation are purely statistical and format-independent.
Google Gemini
Phase 2 ships a dedicated Gemini adapter (distil/adapters/gemini.py). Run the proxy with --upstream https://generativelanguage.googleapis.com and it automatically intercepts generateContent and streamGenerateContent requests. What works today:
- Request compression:
textparts from non-model turns — Tier-0 lossless (minify + duplicate-run collapse).functionResponsepayloads — Tier-1 reversible digest (original kept in the localRestoreStore, never sent, zero tokens).functionCall,inlineData,fileData, model-authored text, andsystemInstructionpass through untouched. - Savings reporting:
x-distil-tokens-savedresponse header + savings ledger, same as all other adapters. - Shadow-mode decision-equivalence: reads the chosen action from
candidates[0].content.parts[].functionCall. --verbatimmode: Tier-0-only transforms — the Tier-1 digest is skipped entirely so the model sees message content essentially verbatim. No digest stubs. Applies to all adapters including Gemini. Use for interactive sessions or where recovery (distil_expand) is unavailable. Lower savings.--lossless-onlymode: Policy mode — restricts the proxy to lossless strategies (no lossy output-shaping, no tool injection). The reversible Tier-1 digest still runs. Use for subscription or OAuth-gated sessions.
Not yet wired for Gemini (messages-format-only today): expand-tool injection (--expand), output verbosity shaping (--shape-output), and Gemini context caching.
"Where can I see community savings?"
By design, savings are local-first and privacy-preserving — Distil never sends your prompts or responses anywhere. Three views:
distil leaderboard— your own cumulative savings. The proxy records real per-request token savings from live traffic into the ledger (thelive-proxysource), so this reflects actual usage, not benchmark projections.distil federated-leaderboard --dir ./submissions --keys keys.json --html out.html— a verifiable multi-instance leaderboard built from HMAC-signed, content-free aggregates (numbers + certification verdict only). Savings you can verify, not just claim.distil leaderboard --html your-savings.html— render your real ledger as a self-contained page (genuine numbers from your usage, including thelive-proxysource — no dummy data).
There is no hosted public dashboard harvesting your data. Opt-in federation means you control submission and verification (ed25519 public-key signing is the documented upgrade for share-the-key-free verification).
"What is the license?"
Apache-2.0. See LICENSE in the repository.
You can use it commercially, modify it, distribute it, and sublicense it. The only requirements are attribution and preservation of the license notice. No patent traps beyond the standard Apache-2.0 patent grant.
"Does distil break streaming?"
No. SSE responses are relayed chunk-by-chunk as the upstream produces them — the proxy, the async proxy, and the gateway all route through the same relay (distil/streamrelay.py), so time-to-first-token is preserved through Distil the same as a direct connection. Nothing is buffered start-to-finish before it reaches the client.
Content-free accounting is teed off that same stream rather than requiring a separate buffered pass: the complete body is assembled in memory as it streams by, so token-saved headers and shadow-mode decision comparison run exactly as they would on a non-streaming response. A slow or unreachable upstream is bounded by DISTIL_UPSTREAM_TIMEOUT (default 600s) and maps to a 504 — see Deploy & Security → Streaming and upstream timeouts.
"Can I contribute a new compression strategy?"
Yes — PRs are welcome. The rule is simple: a new strategy must pass make gate (non-inferior on every corpus domain, byte-reversible on Tier-0/1 operations). No green gate, no merge. See CONTRIBUTING.md for details.