Frequently Asked Questions
Honest answers to the questions that matter most — including the things Distil won't pretend.
"Is 100% decision-equivalence real, or is it marketing?"
It is real, in a specific and honest sense. 100% decision-equivalence means 100% of turns in the default corpus trajectories produced the same decision (tool call name + arguments, or final answer) when using the distil strategy versus the uncompressed baseline, as measured by the deterministic offline runner.
Three caveats the numbers depend on:
- Decision-equivalence, not byte-equivalence. The compressed context may differ byte-for-byte from the original — what is certified is that the agent's decisions match, not that the two texts are identical strings.
- Deterministic offline runner. The default runner is a keyed stand-in, not the live Claude model. It produces reproducible, ground-truth decisions based on decision markers embedded in the trajectories. Running with
--runner anthropicagainst the live model is implemented but marked UNVERIFIED until you exercise it with a key (real model outputs have non-determinism at temperature > 0). - Bundled corpus. The 7 trajectories are real, captured-style agent workflows — but they are a finite corpus. The gate certifies the strategies work on these domains; generalization to new domains is a reasonable assumption, not a guarantee.
"Is the compression lossless?"
Depends on the tier — and the answer is precise:
| Tier | Lossless? | How |
|---|---|---|
| Tier 0 — JSON minify, RLE | Provably lossless | Reconstructable by inversion — no side state needed. The original can always be recovered from the transform output alone. |
| Tier 1 — Digest + handles | Reversible | The original is stored in a local RestoreStore keyed by an 8-hex SHA-256 handle embedded in the compressed text. Recovery requires the RestoreStore, which is kept locally and never sent to the model. |
| Certified — Pruning | Not lossless — but causally certified | Pruned blocks are causally inert: ablation confirmed they never changed any decision. The bytes are dropped but the agent behaviour is preserved within the non-inferiority margin. |
The byte-fidelity gate (distil verify) confirms that every Tier-0 and Tier-1 operation across the corpus is byte-reversible and that frozen history never mutates between turns.
"How accurate is the tokenizer?"
The default tokenizer is an offline heuristic — it estimates tokens by a character-based approximation without calling any API. This means:
- Compression ratios are robust. The numerator and denominator of the ratio use the same heuristic, so the ratio is a good estimate regardless of the absolute token count.
- Absolute dollar figures are approximate. The heuristic is not calibrated to Claude's actual tokenizer, which uses BPE. Dollar amounts should be treated as directional, not billing-grade.
- tiktoken undercounts Claude. If you use OpenAI's tiktoken to count tokens for Claude pricing, you will systematically undercount — Claude's vocabulary differs. Use
--tokenizer anthropicfor accurate counts.
To get billing-grade token counts:
pip install 'distil-llm[live]'
export ANTHROPIC_API_KEY=sk-ant-…
distil savings --tokenizer anthropic
distil bench --tokenizer anthropic
This calls the Anthropic count_tokens API — accurate billing-grade counts, but requires network access and consumes a small amount of API quota.
"What is not done yet?"
From the roadmap — implemented items that are real and wired, and items that are planned seams:
| Item | Status |
|---|---|
| Real tokenizer + live runner (billing-grade) | Done — --tokenizer anthropic / --runner anthropic |
| Multi-domain trajectory corpus + CI gate | Done — distil bench |
| Runtime adapter (no-code-change compression) | Done — wrap(client) |
| Auth-mode gating | Done — distil/policy.py |
| Holdout A/B with confidence intervals | Done — distil holdout |
| Byte-fidelity invariants | Done — distil verify |
| BM25 partial retrieval · delta context · keep-model codec · gist caching | Done — all wired, real implementations |
| Learned per-content-type keep-model weights | In progress — the logistic classifier is implemented; neural/soft-prompt variant is a planned seam |
| Provider proxy CLI subcommand | In progress — module works (python -m distil.proxy), CLI flag wrapper being added |
--runner anthropic verified against live model | Implemented — UNVERIFIED without a key; the deterministic runner covers all corpus measurements |
"Does it work with models other than Claude?"
The core techniques — cache-aware compression, causal pruning, TOST certification, byte-fidelity invariants — are model-agnostic and apply to any LLM with prompt caching. The corpus trajectories and the adapter are built around the Anthropic Messages API format, but:
- The proxy intercepts
/v1/chat/completionsand/v1/responsesin addition to/v1/messages, making it compatible with OpenAI-format APIs. - The token count and dollar figures use Anthropic pricing by default; other models can be added to the pricing catalog.
- The TOST gate and causal ablation are purely statistical and format-independent.
"What is the license?"
Apache-2.0. See LICENSE in the repository.
The short version: you can use it commercially, modify it, distribute it, and sublicense it. The only requirements are attribution and preservation of the license notice. There are no patent traps beyond the standard Apache-2.0 patent grant.
"Can I contribute a new compression strategy?"
Yes — PRs are welcome. The rule is simple: a new strategy must pass make gate (non-inferior on every corpus domain, byte-reversible on Tier-0/1 operations). No green gate, no merge. See CONTRIBUTING.md for details.