# Research & Frontier

The proof artifacts no competitor publishes: a savings-vs-quality curve where every point carries a certification verdict, a distribution-free risk certificate on the decision-change rate, a never-regressing self-distilling loop, and savings telemetry you can verify yourself.

## Certified compression frontier

Module: `distil/eval.py` · Command: `distil eval [--corpus --runner --out]`

Most compression tools publish a headline savings number. Distil publishes the full curve: a sweep of compression aggressiveness levels where every point is tagged with (token savings, decision-equivalence, certified/not). This locates the cliff past which lossy compression starts dropping agent decisions — and proves that distil sits safely inside the certified region.

The proof artifact. The `FrontierReport` produced by `distil eval` is the artifact no competitor publishes: a curve where every point carries a TOST non-inferiority verdict. The certified ceiling is the highest savings level the gate accepts; beyond it, lossy strategies start changing agent decisions and are rejected. Distil operates below the cliff.

### How the sweep works

`distil/eval.py` defines `frontier()`, which measures two reference operating points (`tier-0 lossless` and `distil (cache-aware)`) and then sweeps a range of truncation limits (`[4000, 2000, 1200, 700, 450, 300, 220, 160, 120, 90]` chars) across the corpus. For each point it measures:

- Token savings — average reduction fraction over the corpus.
- Decision equivalence — average match rate between compressed and uncompressed decisions.
- Certified — both TOST non-inferiority passes and equivalence equals 100%. A single trajectory failure uncertifies the point.

Results are sorted by savings ascending and printed as the frontier table. Run offline with the deterministic runner (structural decision-equivalence). For real task-accuracy on a benchmark (tau-bench, SWE-bench, GSM8K) ingest its traces and run with `--runner anthropic`.

    $ distil eval
    certified compression frontier  (runner=deterministic)

    level                   savings   equiv  certified  curve
    --------------------------------------------------------------------------
    distil (cache-aware)       8.4%    100%    ✔ PASS   ██
    truncate@1200              7.2%     79%       ✘ —    ██
    truncate@700              20.0%     36%       ✘ —    ████
    truncate@300              41.3%      0%       ✘ —    █████████
    --------------------------------------------------------------------------

    distil: 8.4% savings @ 100% decision-equivalence — certified.
    certified ceiling: 8.4% savings (beyond this, lossy compression drops decisions and the gate rejects it).

The raw curve can be written to JSONL with `--out <dir>` for CI integration or archiving.

Offline mode

### Deterministic runner

Uses structural decision-equivalence (ground-truth `DECISION:` markers). Runs fully offline, no API key required. Results are reproducible bit-for-bit across machines.

Live mode

### `--runner anthropic`

Grade the curve with a live model. Use this when evaluating against real benchmarks (tau-bench / SWE-bench / GSM8K) — ingest their traces first with `distil ingest`.

### Flags

Flag | Default | Description
--- | --- | ---
`--corpus` | bundled corpus | Custom corpus dir (e.g. from `distil ingest` on benchmark traces)
`--runner` | `deterministic` | `deterministic` (offline) or `anthropic` (live model, requires API key)
`--out` | off | Write the raw curve as timestamped JSONL to this directory

## Decision-Equivalence Risk Certificate (DERC)

Module: `distil/conformal.py` · Command: `distil conformal [--alpha --delta --method --corpus --runner]`

The frontier above tells you where the cliff is. The DERC turns that into an operational guarantee: you pick a risk budget — a maximum tolerable decision-change rate α — and Distil certifies the most aggressive compression level whose risk is provably below it, distribution-free and finite-sample, calibrated on your own traffic. This is conformal risk control, not a heuristic threshold.

The guarantee. For the certified level λ̂, Learn-Then-Test gives `P( R(λ̂) ≤ α ) ≥ 1 − δ` — with no distributional assumptions and valid at finite n. The loss on each calibration turn is `1` iff the agent's decision flips versus the uncompressed context, graded by the same runner as the gate. `R` is the decision-change rate.

### How it certifies

Distil calibrates a ladder of compression levels (byte-exact → lossless → a truncation sweep), measures the decision-change rate at each, and selects the most aggressive controlled level via one of two procedures:

- Learn-Then-Test (LTT) — Angelopoulos, Bates, Candès, Jordan & Lei, Ann. Appl. Stat. 2025 (arXiv:2110.01052). Risk control as multiple hypothesis testing; Hoeffding–Bentkus p-values with fixed-sequence testing yield `P(R≤α)≥1−δ`. The default.
- Conformal Risk Control (CRC) — Angelopoulos, Bates, Fisch, Lei & Schuster, ICLR 2024 (arXiv:2208.02814). For a monotone 0/1 loss, controls the expected rate `E[L]≤α`, tight to O(1/n). Use `--method crc`.

    $ distil conformal --corpus ./mycorpus --alpha 0.05 --delta 0.05
    Decision-Equivalence Risk Certificate (DERC)

      method      : LTT  (Learn-Then-Test)
      risk target : α = 0.05  (max allowed decision-change rate)
      confidence  : 95%  (1 − δ)
      calibration : n = 320 turns,  runner = deterministic
    ----------------------------------------------------------------
      ✔ CERTIFIED  'lossless'  →  57.4% token savings

      the decision-change rate vs. uncompressed context is ≤ 5.0%
      with 95% confidence (Learn-Then-Test, n=320 calibration turns).

### Why this is novel

Conformal prediction is established theory, but applying it to context compression with the loss defined as agent decision-equivalence is, to our reading of the literature, open white space. The nearest neighbour (arXiv:2511.17908, ECIR 2026) applies conformal guarantees to RAG retrieval recall — a different task (which documents to fetch), not how far you can crush the context an agent already holds while it keeps acting identically.

The one honest caveat (load-bearing). Conformal guarantees require exchangeability: your calibration traffic must resemble your live traffic. Under distribution shift (new agent, prompt change, workload drift) the bound can silently weaken — recalibrate on a rolling window. The guarantee is marginal over the calibration distribution (an average rate, not a per-prompt promise). And it is honestly conservative: on a small corpus it will refuse to certify a tight α rather than over-claim — more calibration turns certify the same α (double-validated: 320 turns certify `lossless` at α=2%, 640 turns at α=1%).

### Flags

Flag | Default | Description
--- | --- | ---
`--alpha` | `0.05` | Risk target: max tolerable decision-change rate
`--delta` | `0.05` | LTT failure probability; confidence is 1−δ (ignored by CRC)
`--method` | `ltt` | `ltt` (high-probability bound) or `crc` (expected-rate bound)
`--corpus` | bundled corpus | Calibration corpus (use your own traffic via `distil ingest`)
`--runner` | `deterministic` | `deterministic` (offline) or `anthropic` (live model)

## Salience protection — keep the needle, crush the haystack

Module: `distil/compress/salience.py` · Composes with: the certificate (it shifts the frontier)

The certificate tells you how far you can compress safely. Salience protection makes "far" go further: before a lossy level crushes a block, it guarantees the decision-bearing lines survive — so aggressive compression stops dropping load-bearing identifiers and directives. It is model-free (a few hundred microseconds, no transformer in the path), and unlike syntactic entity protectors it blends three signals and works at line granularity so the decision unit (verb + target) stays intact.

Signal 1 — pattern

### Identifier shapes

UUIDs, git SHAs, `PREFIX-NNN` IDs, emails, IPs, semver — the obvious load-bearing tokens.

Signal 2 — entropy

### Novel IDs no regex anticipates

High-information mixed-alnum tokens caught by Shannon entropy + character diversity, so an unfamiliar key format is still protected.

Signal 3 — reference

### Cross-block anchors

A token that recurs across blocks (a tool name in the schema and in an output) is something the agent navigates by — load-bearing, not noise.

Composition

### A frontier shifter, not a heuristic

Protection lowers the decision-change rate, so the conformal certificate can certify a more aggressive level. The guarantee still comes from the gate; this just moves where the gate says “safe.”

Measured. On the realistic corpus, wrapping blind `truncate@200` with `protect(…)` took its load-bearing-target survival from 0/120 → 120/120, at a cost of only 2.4 points of savings (73.9% → 71.5%) — and ~0.4 ms per turn, no model. The needle survives; the haystack still compresses. Distil's `distil conformal` ladder includes `protect+truncate` levels, so the certificate can pick a protected-aggressive operating point automatically.

## Shadow-mode — live decision-equivalence on real traffic

Module: `distil/shadow.py` · Command: `distil proxy --shadow RATE` · `distil shadow-stats`

The certificate proves decision-equivalence offline, on a calibration corpus. Shadow mode closes the loop online. Cost is free to measure live (token deltas); equivalence is not — it needs the counterfactual, the decision the agent would have made on the uncompressed context. So shadow mode samples a fraction of live requests and runs each one both ways, comparing the agent's chosen next action.

How it stays production-safe. The second (uncompressed) call runs in a background thread — the client gets the compressed response immediately, never blocked. Only `RATE` of requests are sampled, so the overhead is `RATE` (e.g. 5%), not 2×. And it is content-free: the ledger stores a decision signature and an `equivalent` bool — never prompt or response text.

The "decision" is the agent's next action — the first `tool_use` (Anthropic), `tool_call` (OpenAI), or `functionCall` (Gemini). Two responses are equivalent iff that action matches, exactly the `{action, target}` fingerprint the certificate uses. It is streaming-aware: the decision is reconstructed from SSE, so shadow-mode works on real agent sessions (Claude Code, Codex, Gemini CLI) that stream — not just non-streaming SDK calls. The result is a rolling, live decision-change rate on your traffic:

    $ distil proxy --shadow 0.05 --upstream https://api.anthropic.com
      → shadow-mode live decision-equivalence: sampling 5%

    $ distil shadow-stats
      shadowed requests : 412
      decision changes  : 2
      decision-change rate (rolling): 0.49%
      decision-equivalence          : 99.51%

This complements periodic re-certification (`distil ingest` → `distil conformal`) under the exchangeability caveat: shadow mode is the continuous monitor, the certificate is the guarantee. Together they make decision-equivalence observable in production, not just in eval.

## Cache-delta context coding — cross-version delta, decision-equivalent

Module: `distil/cachedelta.py` · Command: `distil proxy --session-delta` · For: coding agents (Claude Code / Codex / Gemini CLI)

The coding-agent hot path is read → edit → re-read. The re-read file is not byte-identical to the first read — one hunk changed — so exact-duplicate dedup (the state of the art elsewhere, e.g. Headroom's `dedup_identical_items`) misses it and re-sends the whole file as fresh tokens. Cache-delta coding instead sends only the diff against the previously-delivered version, referencing the rest.

Why it is decision-equivalent (the motto). The prior version is still present earlier in the (cached) conversation, so prior-version + diff carries exactly the information the agent needs for its next action — the decision is unchanged. It is reversible (the full current version is kept locally; `distil_expand` recovers it byte-exact) and measurable (shadow mode records the live decision-change rate), so the equivalence is proven, not asserted.
Win 1 — exact re-send

### Back-reference

Content delivered earlier this session is replaced by a compact handle reference. Table stakes — competitors do this too.

Win 2 — near-duplicate

### Cross-version delta

A re-read-after-edit is replaced by a reference plus a unified diff. The leap exact-only dedup can't make — and it's the most common coding pattern.

Cache-monotonic

### Never bust the prefix

Only the volatile suffix is touched; the stable, already-cached prefix is left byte-identical, so prompt-cache hits survive. Optimising the real bill (`cached·0.1 + writes·1.25 + new·1.0`), not raw tokens.

Measured (honest)

### The digest is the lever; cache-delta is the verbatim lever

On the 256-turn coding benchmark, the Tier-1 digest already wins (~91% cache-aware, reversible), so cache-delta adds little on top of it. Its real niche is verbatim/interactive mode (digest disallowed): there it turns a 0% floor into 43.8% cache-aware savings, reversible. See the coding-agent benchmark.

AST-structural delta (`distil/astdelta.py`, stdlib `ast`, model-free) is the deepest layer: for Python it diffs files by parsed structure, fingerprinting each top-level definition with `ast.dump` — invariant to whitespace, comments, and import order. A reformat-only re-read is recognised as no definition changed and referenced; only definitions whose AST actually changed are sent in full. Textual diff explodes on reformatting; the structural delta isolates exactly what changed. Non-Python or mid-edit (unparseable) source falls back to the textual delta — it never fails a request, it just does less.

## Self-distilling keep-model

Module: `distil/online.py` · Command: `distil online [--corpus --promote-to]`

Most learned compressors train a keep-model on a generic judge applied to synthetic data. Distil trains on causal labels — certified-safe drops discovered by the ablation engine running on your own real traffic. The label source is the moat: a model that learns what actually changed a decision, not what a judge guessed.

### The never-regressing loop

The loop has four steps, each grounded in the source:

1. Collect causal labels (`collect_causal_labels()`) — run counterfactual ablation (`discover()`) on every trajectory. Blocks that never changed a decision are PRUNABLE (label=0). Blocks that changed at least one decision are KEPT (label=1). Only VOLATILE blocks are scanned; the stable cacheable prefix is out of scope. Conflict resolution: keep wins (label=1 is never overridden to 0).
2. Retrain (`retrain()`) — featurize the labeled lines (9 features, same as the built-in logistic model) and train logistic regression with full-batch gradient descent, L2 regularization, and a deterministic SHA-256 train/test split. No random seed dependency.
3. Certify promotion (`certify_promotion()`) — wrap the new weights as a compression strategy, run the TOST non-inferiority gate on every single corpus trajectory. A single regression anywhere blocks the promotion.
4. Persist iff certified (`online_round()`) — write the new weights to `--promote-to` only if step 3 passed. A cycle that would degrade quality is silently discarded.

Never-regressing by construction. Because step 3 requires non-inferior TOST on every trajectory before weights are touched, the loop can only improve the keep-model. A candidate that regresses on any trajectory is discarded without modifying production weights.

### Illustrative output — your numbers depend on your corpus

Example from one `distil online` round on a bundled corpus; the flywheel is real and gated, but exact figures vary by traffic.

286
Causal labels collected
0.873
Accuracy (held-out)
0.932
F1 (held-out)
PASS
Certified (gate)

    $ distil online --corpus ./mycorpus --promote-to ./weights/keep_model.json
    self-distilling round — keep-model learns from causal labels, gated by non-inferiority

      n_labels: 286
      accuracy: 0.873
      f1: 0.932
      certified: True
      promoted: True

If the gate fails (a regression anywhere in the corpus), promotion is blocked and the output will show `certified: False` and `promoted: False` with a message that the candidate failed the non-inferiority gate.

### Flags

Flag | Default | Description
--- | --- | ---
`--corpus` | bundled corpus | Corpus directory of traffic to learn from. Use `distil ingest` to convert real logs first.
`--promote-to` | off (dry run) | If certified, persist the new weights as a `LogisticKeepModel`-compatible JSON file at this path.

## Verifiable federated telemetry

Module: `distil/telemetry.py` · Command: `distil federated-leaderboard --dir <dir> [--keys <keys.json> --html <out.html>]`

Each opt-in instance contributes a signed, content-free savings aggregate with its certification verdict attached. The leaderboard aggregates only verified submissions — every number on the board is tamper-evident. No prompt text, no response content, no tool outputs ever leave the machine: only (instance_id, tokens_saved, dollars_saved, runs, certified, timestamp).

### How signing works

Signing uses HMAC-SHA256 (symmetric: both sides share a per-instance key). The canonical form of the aggregate is `json.dumps(fields, sort_keys=True, separators=(',', ':'))` — deterministic across Python versions and dict orderings. Verification uses `hmac.compare_digest` for constant-time comparison.

ed25519 is the documented upgrade. The module docstring in `distil/telemetry.py` documents the natural upgrade: if you want a leaderboard anyone can verify without sharing keys, swap HMAC-SHA256 for ed25519. The instance keeps the signing key, publishes the verify key, and the aggregator uses the public key only. The swap is a drop-in change at the `sign`/`verify` boundary — nothing else in the module changes.

### Leaderboard rules

- Submissions whose `instance_id` has no entry in the keys map are rejected.
- Submissions whose signature does not verify are rejected.
- For each instance, the latest (highest timestamp) verified submission wins.
- Only certified submissions (where `certified == True`) contribute to headline totals. Uncertified instances are shown but their numbers do not roll into the aggregate.

    $ distil federated-leaderboard \
        --dir ./submissions \
        --keys ./instance-keys.json \
        --html ./leaderboard.html
    verifiable savings — 3 verified instance(s), 0 rejected

      totals (certified only): {'tokens_saved': 142816, 'dollars_saved': 0.071408, 'runs': 47, 'instances': 3}

The `--html` flag writes a self-contained dark HTML page (the same style as the gateway dashboard) showing per-instance verified savings. The page is served inline — no CDN, no external assets.

Content-free

### Zero prompt leakage

The `SavingsAggregate` dataclass contains only numeric fields. No text from your prompts, tool results, or model responses is ever included in a submission.

Tamper-evident

### Savings you can verify

Every number on the leaderboard was produced by a verified submission. Tampered or unsigned entries are counted in `rejected` and excluded from totals.

### Flags

Flag | Default | Description
--- | --- | ---
`--dir` | required | Directory containing `submissions.jsonl` (one signed aggregate per line)
`--keys` | off | Path to a JSON file mapping `instance_id` → HMAC key. Submissions with no matching key are rejected.
`--html` | off | Write a self-contained leaderboard HTML page to this path