Research & Frontier

The proof artifacts no competitor publishes: a savings-vs-quality curve where every point carries a certification verdict, a distribution-free risk certificate on the decision-change rate, a never-regressing self-distilling loop, and savings telemetry you can verify yourself.

E7: SWE-bench Verified end-to-end task-success

Harness: benchmarks/swe_bench_e2e/ (official swebench 4.1.0) · Numbers: docs/paper/results/swe_bench_verified_e2e.json · Source: PR #8

E1–E5 measure decision-equivalence on edit-localization — a proxy. E7 is the first non-proxy test: does the certified operating point survive real execution? A real agent (aider + claude-sonnet-4-6, temp 0, diff edit format) is run end-to-end on SWE-bench Verified (n=50, seed 1729) and scored by the official swebench harness, across three conditions sharing the identical agent: (A) full context, (B) distil at its certified trunc@500 operating point, and (C) LLMLingua-2.

distil's certified point loses to full context by 36 points (paired McNemar p<0.001) and does not beat LLMLingua-2. At trunc@500 it strips 86% of agentic context (vs LLMLingua-2's 48%), so its lower score is partly aggression, not purely method.

Certified compression frontier

Module: distil/eval.py · Command: distil eval [--corpus --runner --out]

Most compression tools publish a headline savings number. Distil publishes the full curve: a sweep of compression aggressiveness levels where every point is tagged with (token savings, decision-equivalence, certified/not). This locates the cliff past which lossy compression starts dropping agent decisions — and proves that distil sits safely inside the certified region.

How the sweep works

distil/eval.py defines frontier(), which measures two reference operating points (tier-0 lossless and distil (cache-aware)) and then sweeps a range of truncation limits ([4000, 2000, 1200, 700, 450, 300, 220, 160, 120, 90] chars) across the corpus. For each point it measures:

• Token savings — average reduction fraction over the corpus.
• Decision equivalence — average match rate between compressed and uncompressed decisions.
• Certified — both TOST non-inferiority passes and equivalence equals 100%. A single trajectory failure uncertifies the point.

Results are sorted by savings ascending and printed as the frontier table. Run offline with the deterministic runner (structural decision-equivalence). For real task-accuracy on a benchmark (tau-bench, SWE-bench, GSM8K) ingest its traces and run with --runner anthropic.

copy$ distil eval
certified compression frontier  (runner=deterministic)

level                   savings   equiv  certified  curve
--------------------------------------------------------------------------
distil (cache-aware)       8.4%    100%    ✔ PASS   ██
truncate@1200              7.2%     79%       ✘ —    ██
truncate@700              20.0%     36%       ✘ —    ████
truncate@300              41.3%      0%       ✘ —    █████████
--------------------------------------------------------------------------

distil: 8.4% savings @ 100% decision-equivalence — certified.
certified ceiling: 8.4% savings (beyond this, lossy compression drops decisions and the gate rejects it).

The raw curve can be written to JSONL with --out <dir> for CI integration or archiving.

Flags

Decision-Equivalence Risk Certificate (DERC)

Module: distil/conformal.py · Command: distil conformal [--alpha --delta --method --corpus --runner]

The frontier above tells you where the cliff is. The DERC turns that into an operational guarantee: you pick a risk budget — a maximum tolerable decision-change rate α — and Distil certifies the most aggressive compression level whose risk is provably below it, distribution-free and finite-sample, calibrated on your own traffic. This is conformal risk control, not a heuristic threshold.

How it certifies

Distil calibrates a ladder of compression levels (byte-exact → lossless → a truncation sweep), measures the decision-change rate at each, and selects the most aggressive controlled level via one of two procedures:

• Learn-Then-Test (LTT) — Angelopoulos, Bates, Candès, Jordan & Lei, Ann. Appl. Stat. 2025 (arXiv:2110.01052). Risk control as multiple hypothesis testing; Hoeffding–Bentkus p-values with fixed-sequence testing yield P(R≤α)≥1−δ. The default.
• Conformal Risk Control (CRC) — Angelopoulos, Bates, Fisch, Lei & Schuster, ICLR 2024 (arXiv:2208.02814). For a monotone 0/1 loss, controls the expected rate E[L]≤α, tight to O(1/n). Use --method crc.

copy$ distil conformal --corpus ./mycorpus --alpha 0.05 --delta 0.05
Decision-Equivalence Risk Certificate (DERC)

  method      : LTT  (Learn-Then-Test)
  risk target : α = 0.05  (max allowed decision-change rate)
  confidence  : 95%  (1 − δ)
  calibration : n = 320 turns,  runner = deterministic
----------------------------------------------------------------
  ✔ CERTIFIED  'lossless'  →  57.4% token savings

  the decision-change rate vs. uncompressed context is ≤ 5.0%
  with 95% confidence (Learn-Then-Test, n=320 calibration turns).

Why this is novel

Conformal prediction is established theory, but applying it to context compression with the loss defined as agent decision-equivalence is, to our reading of the literature, open white space. The nearest neighbour (arXiv:2511.17908, ECIR 2026) applies conformal guarantees to RAG retrieval recall — a different task (which documents to fetch), not how far you can crush the context an agent already holds while it keeps acting identically.

Flags

Salience protection — keep the needle, crush the haystack

Module: distil/compress/salience.py · Composes with: the certificate (it shifts the frontier)

The certificate tells you how far you can compress safely. Salience protection makes "far" go further: before a lossy level crushes a block, it guarantees the decision-bearing lines survive — so aggressive compression stops dropping load-bearing identifiers and directives. It is model-free (a few hundred microseconds, no transformer in the path), and unlike syntactic entity protectors it blends three signals and works at line granularity so the decision unit (verb + target) stays intact.

Shadow-mode — live decision-equivalence on real traffic

Module: distil/shadow.py · Command: distil proxy --shadow RATE · distil shadow-stats

The certificate proves decision-equivalence offline, on a calibration corpus. Shadow mode closes the loop online. Cost is free to measure live (token deltas); equivalence is not — it needs the counterfactual, the decision the agent would have made on the uncompressed context. So shadow mode samples a fraction of live requests and runs each one both ways, comparing the agent's chosen next action.

The "decision" is the agent's next action — the first tool_use (Anthropic), tool_call (OpenAI), or functionCall (Gemini). Two responses are equivalent iff that action matches, exactly the {action, target} fingerprint the certificate uses. It is streaming-aware: the decision is reconstructed from SSE, so shadow-mode works on real agent sessions (Claude Code, Codex, Gemini CLI) that stream — not just non-streaming SDK calls. The result is a rolling, live decision-change rate on your traffic:

copy$ distil proxy --shadow 0.05 --upstream https://api.anthropic.com
  → shadow-mode live decision-equivalence: sampling 5%

$ distil shadow-stats
  shadowed requests : 412
  decision changes  : 2
  decision-change rate (rolling): 0.49%
  decision-equivalence          : 99.51%

This complements periodic re-certification (distil ingest → distil conformal) under the exchangeability caveat: shadow mode is the continuous monitor, the certificate is the guarantee. Together they make decision-equivalence observable in production, not just in eval.

Cache-delta context coding — cross-version delta, decision-equivalent

Module: distil/cachedelta.py · Command: distil proxy --session-delta · For: coding agents (Claude Code / Codex / Gemini CLI)

The coding-agent hot path is read → edit → re-read. The re-read file is not byte-identical to the first read — one hunk changed — so exact-duplicate dedup (the state of the art elsewhere, e.g. Headroom's dedup_identical_items) misses it and re-sends the whole file as fresh tokens. Cache-delta coding instead sends only the diff against the previously-delivered version, referencing the rest.

AST-structural delta (distil/astdelta.py, stdlib ast, model-free) is the deepest layer: for Python it diffs files by parsed structure, fingerprinting each top-level definition with ast.dump — invariant to whitespace, comments, and import order. A reformat-only re-read is recognised as no definition changed and referenced; only definitions whose AST actually changed are sent in full. Textual diff explodes on reformatting; the structural delta isolates exactly what changed. Non-Python or mid-edit (unparseable) source falls back to the textual delta — it never fails a request, it just does less.

Self-distilling keep-model

Module: distil/online.py · Command: distil online [--corpus --promote-to]

Most learned compressors train a keep-model on a generic judge applied to synthetic data. Distil trains on causal labels — certified-safe drops discovered by the ablation engine running on your own real traffic. The label source is the moat: a model that learns what actually changed a decision, not what a judge guessed.

The never-regressing loop

The loop has four steps, each grounded in the source:

1. Collect causal labels (collect_causal_labels()) — run counterfactual ablation (discover()) on every trajectory. Blocks that never changed a decision are PRUNABLE (label=0). Blocks that changed at least one decision are KEPT (label=1). Only VOLATILE blocks are scanned; the stable cacheable prefix is out of scope. Conflict resolution: keep wins (label=1 is never overridden to 0).
2. Retrain (retrain()) — featurize the labeled lines (9 features, same as the built-in logistic model) and train logistic regression with full-batch gradient descent, L2 regularization, and a deterministic SHA-256 train/test split. No random seed dependency.
3. Certify promotion (certify_promotion()) — wrap the new weights as a compression strategy, run the TOST non-inferiority gate on every single corpus trajectory. A single regression anywhere blocks the promotion.
4. Persist iff certified (online_round()) — write the new weights to --promote-to only if step 3 passed. A cycle that would degrade quality is silently discarded.

Illustrative output — your numbers depend on your corpus

Example from one distil online round on a bundled corpus; the flywheel is real and gated, but exact figures vary by traffic.

copy$ distil online --corpus ./mycorpus --promote-to ./weights/keep_model.json
self-distilling round — keep-model learns from causal labels, gated by non-inferiority

  n_labels: 286
  accuracy: 0.873
  f1: 0.932
  certified: True
  promoted: True

If the gate fails (a regression anywhere in the corpus), promotion is blocked and the output will show certified: False and promoted: False with a message that the candidate failed the non-inferiority gate.

Flags

Verifiable federated telemetry

Module: distil/telemetry.py · Command: distil federated-leaderboard --dir <dir> [--keys <keys.json> --html <out.html>]

Each opt-in instance contributes a signed, content-free savings aggregate with its certification verdict attached. The leaderboard aggregates only verified submissions — every number on the board is tamper-evident. No prompt text, no response content, no tool outputs ever leave the machine: only (instance_id, tokens_saved, dollars_saved, runs, certified, timestamp).

How signing works

Signing uses HMAC-SHA256 (symmetric: both sides share a per-instance key). The canonical form of the aggregate is json.dumps(fields, sort_keys=True, separators=(',', ':')) — deterministic across Python versions and dict orderings. Verification uses hmac.compare_digest for constant-time comparison.

Leaderboard rules

• Submissions whose instance_id has no entry in the keys map are rejected.
• Submissions whose signature does not verify are rejected.
• For each instance, the latest (highest timestamp) verified submission wins.
• Only certified submissions (where certified == True) contribute to headline totals. Uncertified instances are shown but their numbers do not roll into the aggregate.

copy$ distil federated-leaderboard \
    --dir ./submissions \
    --keys ./instance-keys.json \
    --html ./leaderboard.html
verifiable savings — 3 verified instance(s), 0 rejected

  totals (certified only): {'tokens_saved': 142816, 'dollars_saved': 0.071408, 'runs': 47, 'instances': 3}

The --html flag writes a self-contained dark HTML page (the same style as the gateway dashboard) showing per-instance verified savings. The page is served inline — no CDN, no external assets.

Flags