Frequently Asked Questions

Honest answers to the questions that matter most — including the things Distil won't paper over.

"Is 100% decision-equivalence real, or is it marketing?"

It is real, in a specific and honest sense. 100% decision-equivalence means every turn in the benchmark trajectories produced the same decision (tool call name + arguments, or final answer) when using the Distil strategy versus the uncompressed baseline.

The default corpus gate uses a deterministic offline runner. The live head-to-head benchmark also confirmed this: graded by claude-opus-4-8 (majority-of-3) on a realistic 120-turn corpus, Distil certified 83.2% token savings at a 0% live decision-change rate — with the non-inferiority bound (≤5% at 95% confidence) met with margin (hb_p = 0.0058). See the live head-to-head for the full methodology and caveats.

Three things the number depends on:

Decision-equivalence, not byte-equivalence. The compressed context may differ byte-for-byte from the original. What is certified is that the agent's decisions match, not that two texts are identical strings.
Decision-determined corpus. Certification only makes sense on traffic where the model reproduces its own decision on identical context — confirmed by the byte-exact = 0% precondition check. On ambiguous traffic the model varies by itself, and the number is undefined (that's the model's noise, not the compressor's fault).
Marginal guarantee, not per-prompt promise. The non-inferiority bound holds over the calibration distribution. Generalization to new domains is a reasonable assumption, not a certificate — run distil ingest + distil bench --runner anthropic on your own traffic to certify your domain.

What makes this different from other "100% accuracy" claims: accuracy and compression are measured on the same trajectories in the same run. No separate "quality benchmark at low compression" and "compression benchmark at high compression." You see both numbers for the same corpus, or you see nothing.

"How did you run the live benchmark, and how do I reproduce it?"

The live head-to-head used actual installed packages — llmlingua 0.2.2, headroom-ai 0.27.0, and Distil 0.23.0 — run against the same 120-turn, 5-domain corpus (4.5–6.5 KB/turn) where decisions are the agent's real next {action, target}. Each compressed context was graded by claude-opus-4-8 majority-of-3, independently of the compression step. Latency is wall-clock per turn on the same hardware.

Results in brief: Distil certified 83.2% savings / 0% decision-change / 0.026 ms per turn. LLMLingua-2 achieved 53.1% savings but flipped 1-in-5 decisions (fails the ≤5% gate). Headroom passed the gate at 35.3% savings but runs a ModernBERT scorer at ~26 ms per turn.

To reproduce:

# Run with pinned versions and the live model
python benchmarks/derc_live_compare.py

# Or run the same gate on your own corpus
export ANTHROPIC_API_KEY=sk-ant-…
distil benchmark --corpus benchmarks/corpus_xl --runner anthropic --tokenizer anthropic

Full methodology, corpus generation script, and caveats are in BENCHMARKS.md.

"Is the compression lossless?"

Depends on the tier — the answer is precise:

Tier	Lossless?	How
Tier 0 — JSON minify, RLE	Provably lossless	Reconstructable by inversion — no side state needed. The original can always be recovered from the transform output alone.
Tier 1 — Digest + handles	Reversible	The original is stored in a local `RestoreStore` keyed by an 8-hex SHA-256 handle embedded in the compressed text. Recovery requires the RestoreStore, which is kept locally and never sent to the model.
Certified — Pruning	Not lossless — but causally certified	Pruned blocks are causally inert: ablation confirmed they never changed any decision. The bytes are dropped but agent behaviour is preserved within the non-inferiority margin.

The byte-fidelity gate (distil verify) confirms that every Tier-0 and Tier-1 operation across the corpus is byte-reversible and that frozen history never mutates between turns.

"How accurate is the tokenizer?"

The default tokenizer is an offline heuristic — it estimates tokens by a character-based approximation without calling any API. This means:

Compression ratios are robust. The numerator and denominator of the ratio use the same heuristic, so the ratio is a good estimate regardless of the absolute count.
Absolute dollar figures are approximate. The heuristic is not calibrated to Claude's BPE tokenizer. Dollar amounts are directional, not billing-grade.
tiktoken undercounts Claude. If you use OpenAI's tiktoken to count tokens for Claude pricing, you will systematically undercount — Claude's vocabulary differs. Use --tokenizer anthropic for accurate counts.

To get billing-grade token counts:

pip install 'distil-llm[live]'
export ANTHROPIC_API_KEY=sk-ant-…
distil savings --tokenizer anthropic
distil bench   --tokenizer anthropic

This calls the Anthropic count_tokens API — accurate billing-grade counts, but requires network access and consumes a small amount of API quota.

"What is and isn't done?"

Item	Status
Real tokenizer + live runner (billing-grade)	Done — `--tokenizer anthropic` / `--runner anthropic`
Multi-domain trajectory corpus + CI gate	Done — `distil bench`
Runtime adapter (no-code-change compression)	Done — `wrap(client)`
Auth-mode gating	Done — `distil/policy.py`
Holdout A/B with confidence intervals	Done — `distil holdout`
Byte-fidelity invariants	Done — `distil verify`
BM25 partial retrieval · delta context · keep-model codec · gist caching	Done — all wired, real implementations
Provider proxy + managed gateway	Done — `distil proxy` / `distil gateway` (live per-tenant dashboard)
Google Gemini adapter (request compression + decision-equivalence)	Done — `distil/adapters/gemini.py`; proxy with `--upstream https://generativelanguage.googleapis.com`. Gaps: `--expand`, `--shape-output`, context caching not yet wired.
Output compression (input and output)	Done — gated shaping + lossless re-entry digest, `distil output-savings`
Learned keep-model (logistic + transformer)	Done — logistic ships built-in (96.4%/0.98 on held-out lines, labels from the heuristic); transformer adapter + training pipeline + demo checkpoint
Certified compression frontier (proof pack)	Done — `distil eval`
Self-distilling keep-model (causal labels, never-regressing)	Done — `distil online`
Verifiable federated telemetry	Done — `distil federated-leaderboard`
Async high-concurrency proxy · Rust hot-path core	Done — `distil proxy --async` · `rust/distil-core` (python parity fallback)
Real-trace ingestion · performance benchmark	Done — `distil ingest` · `distil perf`
Live head-to-head vs. real competitor packages	Done — `python benchmarks/derc_live_compare.py`; graded by `claude-opus-4-8`, majority-of-3. See Benchmark.
Production keep-model checkpoint	Your turn — a demo checkpoint ships on the release; production = retrain on your traces (`distil train-transformer`)
Proven at scale / adoption	Needs real traffic — every tool to prove it ships (`distil ingest` + `distil eval --runner anthropic`); run it on your traces and publish

"Does it work with models other than Claude?"

The core techniques — cache-aware compression, causal pruning, TOST certification, byte-fidelity invariants — are model-agnostic and apply to any LLM with prompt caching. The corpus and adapter are built around the Anthropic Messages API format, but:

The proxy intercepts /v1/chat/completions and /v1/responses in addition to /v1/messages, making it compatible with OpenAI-format APIs.
Token counts and dollar figures use Anthropic pricing by default; other models can be added to the pricing catalog.
The TOST gate and causal ablation are purely statistical and format-independent.

Google Gemini

Phase 2 ships a dedicated Gemini adapter (distil/adapters/gemini.py). Run the proxy with --upstream https://generativelanguage.googleapis.com and it automatically intercepts generateContent and streamGenerateContent requests. What works today:

Request compression: text parts from non-model turns — Tier-0 lossless (minify + duplicate-run collapse). functionResponse payloads — Tier-1 reversible digest (original kept in the local RestoreStore, never sent, zero tokens). functionCall, inlineData, fileData, model-authored text, and systemInstruction pass through untouched.
Savings reporting: x-distil-tokens-saved response header + savings ledger, same as all other adapters.
Shadow-mode decision-equivalence: reads the chosen action from candidates[0].content.parts[].functionCall.
--verbatim mode: Tier-0-only transforms — the Tier-1 digest is skipped entirely so the model sees message content essentially verbatim. No digest stubs. Applies to all adapters including Gemini. Use for interactive sessions or where recovery (distil_expand) is unavailable. Lower savings.
--lossless-only mode: Policy mode — restricts the proxy to lossless strategies (no lossy output-shaping, no tool injection). The reversible Tier-1 digest still runs. Use for subscription or OAuth-gated sessions.

Not yet wired for Gemini (messages-format-only today): expand-tool injection (--expand), output verbosity shaping (--shape-output), and Gemini context caching.

"Where can I see community savings?"

By design, savings are local-first and privacy-preserving — Distil never sends your prompts or responses anywhere. Three views:

distil leaderboard — your own cumulative savings. The proxy records real per-request token savings from live traffic into the ledger (the live-proxy source), so this reflects actual usage, not benchmark projections.
distil federated-leaderboard --dir ./submissions --keys keys.json --html out.html — a verifiable multi-instance leaderboard built from HMAC-signed, content-free aggregates (numbers + certification verdict only). Savings you can verify, not just claim.
distil leaderboard --html your-savings.html — render your real ledger as a self-contained page (genuine numbers from your usage, including the live-proxy source — no dummy data).

There is no hosted public dashboard harvesting your data. Opt-in federation means you control submission and verification (ed25519 public-key signing is the documented upgrade for share-the-key-free verification).

"What is the license?"

Apache-2.0. See LICENSE in the repository.

You can use it commercially, modify it, distribute it, and sublicense it. The only requirements are attribution and preservation of the license notice. No patent traps beyond the standard Apache-2.0 patent grant.

"Can I contribute a new compression strategy?"

Yes — PRs are welcome. The rule is simple: a new strategy must pass make gate (non-inferior on every corpus domain, byte-reversible on Tier-0/1 operations). No green gate, no merge. See CONTRIBUTING.md for details.

The gate is the philosophy in one sentence. If a strategy can't pass non-inferiority on the bundled corpus with the default tokenizer and runner, it is not safe to ship — regardless of how impressive the savings look.