Output & I/O

Distil now compresses both sides of the bill: input tokens going in, and output tokens coming back. This page covers output compression, real-trace ingestion, and the performance benchmark.

Both sides of the bill

Classic context compression only addresses the input side — the tokens you send to the model. That leaves half the invoice untouched. Distil's output compression closes the loop:

Output compression

Module: distil/output.py

Output compression has two mechanisms that work together:

Generation-side shaping (--shape-output)

Before the request is forwarded to the model, the proxy injects a role:"system" directive that instructs the model to be concise. Two modes:

• light — conservative; trims unnecessary preamble and padding. Typical reduction: 20–35%.
• aggressive — strong brevity constraint; omits elaboration and hedging. Measured reduction: 72.5% (95% CI 67.5–77.1%), answer preserved 100% in A/B.

# Start the proxy with aggressive output shaping
$ python -m distil.proxy --port 8788 --shape-output aggressive
distil proxy listening on http://127.0.0.1:8788
  → upstream: https://api.anthropic.com
  → output shaping: aggressive (PAYG-only; will verify key tier on first request)

Lossless re-entry digest

When the model's output is long and is about to be re-injected as history in the next turn, Distil compresses it using the same Tier-1 digest mechanism used for input blocks: the full output is stored in the local RestoreStore under an 8-hex handle, and only the handle is forwarded as history. The model never sees the full prior output again — it sees the digest. On demand, the original expands from the handle.

This is fully lossless and transparent to the application. The output digest is re-expanded on any path that reads history (e.g. certify, replay, or export).

Measuring output savings

$ distil output-savings --mode aggressive
output-token A/B — mode=aggressive | runner=anthropic | n=8 turns

  baseline mean output tokens:   487
  shaped   mean output tokens:   134

  reduction: 72.5%  [95% CI: 67.5%–77.1%]
  answer preserved: 100%  (semantic match gate, n=8)

note: realized savings require --runner anthropic (live model).
      offline --runner deterministic gives a lower-bound proxy only.

Real-trace ingestion

Module: distil/ingest.py · Command: distil ingest

Distil ships with a bundled synthetic corpus, but your production workload may differ significantly. distil ingest converts your own recorded API request logs into a Distil corpus so you can measure savings on real traffic.

Supported formats

• Anthropic — JSONL where each record is an Anthropic messages.create request body.
• OpenAI-compatible — JSONL where each record is an OpenAI chat.completions.create request body.

The format is auto-detected from the first record; pass --format anthropic or --format openai to override.

Grouping into trajectories

Ingest groups records into trajectories by session_id (or x-session-id header if present in metadata). Records without a session ID are each treated as a single-turn trajectory.

$ distil ingest --input prod.jsonl --out ./mycorpus
ingesting prod.jsonl → ./mycorpus

  format detected: anthropic
  records read:    1,204
  trajectories:       42  (grouped by session_id)
  skipped:             3  (malformed or missing messages field)

corpus written to ./mycorpus/

Benchmarking real traces

Real-trace corpora have no DECISION labels, so the offline non-inferiority gate cannot run. Use --savings-only to get input-token savings; certify quality with a live runner:

# Savings-only gate on real traces (no DECISION labels needed)
$ distil bench --corpus ./mycorpus --savings-only
corpus gate — 42 trajectories | savings-only (no DECISION labels)

  aggregate savings: 28.3%  [95% CI: 25.1%–31.6%]
  note: certify with --runner anthropic for a live quality gate.

# Live quality gate on a single trajectory from your corpus
$ distil certify --trajectory ./mycorpus/session_abc123.json --runner anthropic
certifying strategy 'distil' on 'session_abc123' (runner=anthropic)
  ...
VERDICT: PASS  (certified non-inferior)

Performance benchmark

Module: distil/perf.py · Command: distil perf

Distil is designed to add negligible latency to the critical path. The performance benchmark measures the raw compression throughput and the per-request overhead of the in-process adapter.

$ distil perf
distil performance benchmark

  compressor throughput:   ~27,000 distil-compressions/sec
  in-process adapter p50:  0.006 ms
  in-process adapter p95:  0.011 ms
  in-process adapter p99:  0.018 ms

  (measured on Apple M-series, CPython 3.12, bundled sre-disk-incident trajectory)

At 0.006 ms p50, the in-process adapter adds less than 1% to a typical LLM round-trip (which is measured in seconds). The proxy path adds a small additional network hop but the compression itself is the same cost.

Quick reference