Frequently Asked Questions
Honest answers to the questions that matter most — including the things Distil won't pretend.
"Is 100% decision-equivalence real, or is it marketing?"
It is real, in a specific and honest sense. 100% decision-equivalence means 100% of turns in the default corpus trajectories produced the same decision (tool call name + arguments, or final answer) when using the distil strategy versus the uncompressed baseline, as measured by the deterministic offline runner.
Three caveats the numbers depend on:
- Decision-equivalence, not byte-equivalence. The compressed context may differ byte-for-byte from the original — what is certified is that the agent's decisions match, not that the two texts are identical strings.
- Deterministic offline runner. The default runner is a keyed stand-in, not the live Claude model. It produces reproducible, ground-truth decisions based on decision markers embedded in the trajectories. Running with
--runner anthropicagainst the live model is implemented but marked UNVERIFIED until you exercise it with a key (real model outputs have non-determinism at temperature > 0). - Bundled corpus. The 7 trajectories are real, captured-style agent workflows — but they are a finite corpus. The gate certifies the strategies work on these domains; generalization to new domains is a reasonable assumption, not a guarantee.
"Is the compression lossless?"
Depends on the tier — and the answer is precise:
| Tier | Lossless? | How |
|---|---|---|
| Tier 0 — JSON minify, RLE | Provably lossless | Reconstructable by inversion — no side state needed. The original can always be recovered from the transform output alone. |
| Tier 1 — Digest + handles | Reversible | The original is stored in a local RestoreStore keyed by an 8-hex SHA-256 handle embedded in the compressed text. Recovery requires the RestoreStore, which is kept locally and never sent to the model. |
| Certified — Pruning | Not lossless — but causally certified | Pruned blocks are causally inert: ablation confirmed they never changed any decision. The bytes are dropped but the agent behaviour is preserved within the non-inferiority margin. |
The byte-fidelity gate (distil verify) confirms that every Tier-0 and Tier-1 operation across the corpus is byte-reversible and that frozen history never mutates between turns.
"How accurate is the tokenizer?"
The default tokenizer is an offline heuristic — it estimates tokens by a character-based approximation without calling any API. This means:
- Compression ratios are robust. The numerator and denominator of the ratio use the same heuristic, so the ratio is a good estimate regardless of the absolute token count.
- Absolute dollar figures are approximate. The heuristic is not calibrated to Claude's actual tokenizer, which uses BPE. Dollar amounts should be treated as directional, not billing-grade.
- tiktoken undercounts Claude. If you use OpenAI's tiktoken to count tokens for Claude pricing, you will systematically undercount — Claude's vocabulary differs. Use
--tokenizer anthropicfor accurate counts.
To get billing-grade token counts:
pip install 'distil-llm[live]'
export ANTHROPIC_API_KEY=sk-ant-…
distil savings --tokenizer anthropic
distil bench --tokenizer anthropic
This calls the Anthropic count_tokens API — accurate billing-grade counts, but requires network access and consumes a small amount of API quota.
"What is not done yet?"
From the roadmap — implemented items that are real and wired, and items that are planned seams:
| Item | Status |
|---|---|
| Real tokenizer + live runner (billing-grade) | Done — --tokenizer anthropic / --runner anthropic |
| Multi-domain trajectory corpus + CI gate | Done — distil bench |
| Runtime adapter (no-code-change compression) | Done — wrap(client) |
| Auth-mode gating | Done — distil/policy.py |
| Holdout A/B with confidence intervals | Done — distil holdout |
| Byte-fidelity invariants | Done — distil verify |
| BM25 partial retrieval · delta context · keep-model codec · gist caching | Done — all wired, real implementations |
| Provider proxy + managed gateway | Done — distil proxy / distil gateway (live per-tenant dashboard) |
| Output compression (input and output) | Done — gated shaping + lossless re-entry digest, distil output-savings |
| Learned keep-model (logistic + transformer) | Done — logistic ships built-in (96.4%/0.98 on held-out lines, labels from the heuristic); transformer adapter + training pipeline + demo checkpoint |
| Certified compression frontier (proof pack) | Done — distil eval |
| Self-distilling keep-model (causal labels, never-regressing) | Done — distil online |
| Verifiable federated telemetry | Done — distil federated-leaderboard |
| Async high-concurrency proxy · Rust hot-path core | Done — distil proxy --async · rust/distil-core (python parity fallback) |
| Real-trace ingestion · performance benchmark | Done — distil ingest · distil perf |
| Production keep-model checkpoint | Your turn — a demo checkpoint ships on the release; production = retrain on your traces (distil train-transformer) |
--runner anthropic verified against live model | Needs a key — implemented but UNVERIFIED without one; the deterministic runner covers all offline measurements |
| Proven at scale / adoption | Needs real traffic — every tool to prove it ships (distil ingest + distil eval --runner anthropic); run it on your traces and publish |
"Does it work with models other than Claude?"
The core techniques — cache-aware compression, causal pruning, TOST certification, byte-fidelity invariants — are model-agnostic and apply to any LLM with prompt caching. The corpus trajectories and the adapter are built around the Anthropic Messages API format, but:
- The proxy intercepts
/v1/chat/completionsand/v1/responsesin addition to/v1/messages, making it compatible with OpenAI-format APIs. - The token count and dollar figures use Anthropic pricing by default; other models can be added to the pricing catalog.
- The TOST gate and causal ablation are purely statistical and format-independent.
"Where can I see community savings?"
By design, savings are local-first and privacy-preserving — Distil never sends your prompts or responses anywhere. There are three views:
distil leaderboard— your own genuine cumulative savings.distil proxyrecords real per-request token savings from live traffic into the ledger (thelive-proxysource), so this reflects actual usage, not a benchmark.distil federated-leaderboard --dir ./submissions --keys keys.json --html out.html— a verifiable, multi-instance leaderboard built from HMAC-signed, content-free aggregates (numbers + the certification verdict only). Savings you can verify, not just claim.distil leaderboard --html your-savings.html— render your real ledger as a self-contained page (genuine numbers from your usage, including thelive-proxysource from real traffic — no dummy data).
There is no hosted public dashboard harvesting your data — opt-in federation means you control submission and verification (ed25519 public-key signing is the documented upgrade for share-the-key-free verification).
"What is the license?"
Apache-2.0. See LICENSE in the repository.
The short version: you can use it commercially, modify it, distribute it, and sublicense it. The only requirements are attribution and preservation of the license notice. There are no patent traps beyond the standard Apache-2.0 patent grant.
"Can I contribute a new compression strategy?"
Yes — PRs are welcome. The rule is simple: a new strategy must pass make gate (non-inferior on every corpus domain, byte-reversible on Tier-0/1 operations). No green gate, no merge. See CONTRIBUTING.md for details.