Output & I/O

Most compressors only touch the input side; Distil compresses both sides — input context in, generation verbosity out — plus a lossless re-entry digest that keeps long outputs from re-billing as full history. This page covers all three, plus real-trace ingestion and the performance benchmark.

Input and output, together

Input compression has always been Distil's core: cache-aware pruning certified to change 0% of live agent decisions at 83.2% token savings (2026-07-05, distil 1.10.1 vs llmlingua 0.2.2 and headroom-ai 0.27.0). Output compression closes the other half of the invoice:

Output compression

Module: distil/output.py

Two mechanisms work together, with different fidelity guarantees:

Generation-side shaping (--shape-output)

Before forwarding the request, the proxy injects a verbosity-control directive instructing the model to be concise — routed the way each provider accepts it. The Anthropic Messages API rejects a role:"system" entry inside messages (400), so the directive is appended to the top-level system field there instead; the OpenAI chat shape takes an appended role:"system" message, and Gemini takes a systemInstruction part.

This changes the model’s own words, so unlike input compression it cannot be justified by construction — it has to be justified by evidence. That is what auto is. It is the default on a metered session, and it decides once, at session start, from the same live referee every other surface reports.

The rule

Shaping is on only when all three of these hold:

1. the paired decision-equivalence verdict is at or above the one reporting floor the whole product shares — 50 A/B and 30 A/A shadow samples;
2. its harm bound (the lower end of the paired difference interval) is inside the pre-registered certification budget of 2 percentage points — the same margin distil certify tests against, read from the same constant; and
3. the shadow output-token delta excludes zero on the saving side, over at least as many samples as the verdict floor.

Miss any one and shaping is off, with the reason printed at startup, written into the session manifest, and shown by distil dissect. There is no silent state: “off” always comes with why.

All three are read from rows measured with shaping off. Once shaping is on, the shadow replay of the served request carries the directive, so “replies got shorter” on those rows is shaping measuring itself — a gate that counted them would keep itself on by construction. Every shadow row now records the levers that were active when it was measured ("levers": {"compression": …, "shape": …}), and auto turns shaping on only from shape: off rows. Rows written before the tag existed are excluded too: their shaping state is unknown, not off.

Shaped rows can still turn it off. They are the only rows that measure the directive's own effect on decisions, so once they clear the reporting floor their harm bound has to sit inside the same 2pp budget — if it does not, auto resolves off. And all of this is read over the last 7 days only, so an “on” decision never rests on traffic that has since changed.

Condition 3 deserves a caveat, because it is not a measurement of shaping. It measures what input compression already does to reply length on your traffic, which is the only live output-side evidence that exists before the session runs. Read it as an enabling signal — replies on this traffic move shorter when the prompt changes — not as a forecast of what shaping itself will save. A shaping A/B needs a live model and is what distil output-savings measures after the fact.

Two things auto never does. It never picks aggressive: an automatic decision gets the conservative directive, and the strongest lossy setting stays something you ask for. And it never runs on a subscription or OAuth session — that is a policy boundary (policy.may_compress_lossy), not a preference, so an explicit --shape-output aggressive there is suppressed and says so.

Explicit levels override the evidence, on a metered session:

• light — trims preamble, restatement and closing recaps. What auto turns on.
• aggressive — strong brevity constraint; omits elaboration and hedging.
• off — never shape, whatever the evidence says.

# Default: the referee decides, and says why
$ distil proxy --port 8788
distil proxy listening on http://127.0.0.1:8788
  → upstream: https://api.anthropic.com

# Forced on, metered session
$ distil proxy --port 8788 --shape-output aggressive
  → output shaping: aggressive (explicitly requested)

# Forced on, subscription session — refused, with the reason
$ distil proxy --lossless-only --shape-output aggressive
  ⚠ --shape-output aggressive SUPPRESSED: subscription/OAuth session — distil never alters a flat-rate prompt

Reply length is reported on its own line — the output verdict in the proof ledger, distil stats and distil dissect — never folded into the input-token saving: a shorter reply has no per-request counterfactual, so the only honest figure is the shadow estimate with its interval.

Lossless re-entry digest

When a long model response is about to re-enter history on the next turn, Distil compresses it using the same Tier-1 digest mechanism used for input blocks: the full output is stored in the local RestoreStore under an 8-hex handle, and only the handle travels as history. The model never re-reads the full prior output.

This is fully lossless and transparent to the application. The digest expands back to the original on any path that reads history — certify, replay, or export.

Measuring output savings

$ distil output-savings --input shaped_pairs.jsonl
output-token A/B — n=8 turns

  baseline mean output tokens:   487
  shaped   mean output tokens:   134

  reduction: 72.5%  [95% CI: 67.5%–77.1%]  (illustrative)
  answer preserved: 100%  (semantic match gate, n=8)

Real-trace ingestion

Module: distil/ingest.py · Command: distil ingest

Distil ships a bundled synthetic corpus, but production traffic differs. distil ingest converts recorded API request logs into a Distil corpus so you can measure savings on real traffic.

Supported formats

Pass --provider anthropic (default) or --provider openai to select the input format. One input file produces one trajectory — there is no session grouping.

• anthropic (default) — JSONL where each record is an Anthropic messages.create request body.
• openai — JSONL where each record is an OpenAI chat.completions.create request body.

$ distil ingest --input prod.jsonl --out ./mycorpus
ingesting prod.jsonl → ./mycorpus

corpus written to ./mycorpus/

Benchmarking real traces

Real-trace corpora have no DECISION labels, so the offline non-inferiority gate cannot run. Use --savings-only to measure input-token savings; certify quality with a live runner:

# Savings gate on real traces (no DECISION labels required)
$ distil bench --corpus ./mycorpus --savings-only
corpus gate — 42 trajectories | savings-only (no DECISION labels)

  aggregate savings: 28.3%  [95% CI: 25.1%–31.6%]
  note: certify with --runner anthropic for a live quality gate.

# Live quality gate on a single trajectory from your corpus
$ distil certify --trajectory ./mycorpus/session_abc123.json --runner anthropic
certifying strategy 'distil' on 'session_abc123' (runner=anthropic)
  ...
VERDICT: PASS  (certified non-inferior)

Performance benchmark

Module: distil/perf.py · Command: distil perf

Distil is designed to add negligible latency to the critical path. The performance benchmark measures raw compression throughput and per-request overhead of the in-process adapter.

$ distil perf
distil performance benchmark

  compressor throughput:   ~27,000 distil-compressions/sec
  in-process adapter p50:  0.006 ms
  in-process adapter p95:  0.011 ms
  in-process adapter p99:  0.018 ms

  (measured on Apple M-series, CPython 3.12, bundled sre-disk-incident trajectory)

At 0.006 ms p50, the in-process adapter adds under 1% to a typical LLM round-trip (which runs in seconds). The live head-to-head benchmark confirms 0.026 ms per turn — ~1,000× faster than LLMLingua-2's ~1,480 ms and ~1,000× faster than Headroom's ~26 ms. The proxy path adds a small network hop; the compression cost is the same either way.

Quick reference