Research & Frontier

The proof artifacts no competitor publishes: a savings-vs-quality curve where every point carries a certification verdict, a distribution-free risk certificate on the decision-change rate, a never-regressing self-distilling loop, and savings telemetry you can verify yourself.

E7: SWE-bench Verified end-to-end task-success

Harness: benchmarks/swe_bench_e2e/ (official swebench 4.1.0) · Numbers: docs/paper/results/swe_bench_verified_e2e.json · Source: PR #8

E1–E5 measure decision-equivalence on edit-localization — a proxy. E7 is the first non-proxy test: does the certified operating point survive real execution? A real agent (aider + claude-sonnet-4-6, temp 0, diff edit format) is run end-to-end on SWE-bench Verified (n=50, seed 1729) and scored by the official swebench harness, across three conditions sharing the identical agent: (A) full context, (B) distil at its certified trunc@500 operating point, and (C) LLMLingua-2.

distil's certified point loses to full context by 36 points (paired McNemar p<0.001) and does not beat LLMLingua-2. At trunc@500 it strips 86% of agentic context (vs LLMLingua-2's 48%), so its lower score is partly aggression, not purely method.

E8: long-horizon agent task-success

Harness: benchmarks/swe_bench_e2e/ (official swebench) · Numbers: docs/paper/results/swe_e2e_longhorizon/reports/ (per-condition resolved-instance reports)

E7 flagged relevance-gated reversible compression as an untested path on real long-horizon execution. E8 runs that test across six conditions, including a Headroom competitor and LLMLingua-2. A custom multi-turn ReAct coding agent (read / search / edit_file / run_tests tools, up to 30 turns, claude-haiku-4-5, temp 0) is run end-to-end on the full 500-instance SWE-bench Verified set, scored by the official swebench harness (hidden tests, per-instance Docker). Runs average ~27 turns — read-file and tool outputs accumulate into large peripheral context behind a small active working set, which is exactly the regime the relevance gate was designed for. Six conditions share the identical agent; only the compressor differs.

All six conditions score the same 500 instances (paired McNemar). Gate vs full context: difference = −2.4 pp, 95% CI [−5.7, +0.9], McNemar p=0.19 — non-inferior to full at a 6 pp margin (borderline at a strict 5 pp margin; p=0.19 means no significant difference was detected, not that the two are equivalent). Gate vs Headroom: +4.2 pp, McNemar p=0.035 — distil's relevance-gated tier beats Headroom with statistical significance. Lossy truncation (B) vs full context: p<0.001. Gate vs LLMLingua-2: 174 instances the gate solved that LLMLingua-2 did not, 2 the reverse — McNemar p<0.001. Recovery round-trips tell the mechanism: ungated (D) issues 9.6 distil_expand calls per instance (skeleton digested the whole history, must keep recovering the working set); gated (E) issues only 0.58 per instance (digests only aged-out periphery, keeping the working set full). An honest ablation: applying the skeleton digest to the gated tier (E) — which uses head-truncation in its passive path — regressed pass@1 from 36.8% to 5.6%, matching condition B. The navigable digest makes the agent over-trust the summary and stop re-reading; head-truncation is the correct digest for the passive tier.

New techniques in E8: skeleton digest and sticky expansion

Two new techniques were introduced and ablated in E8:

• Content-aware skeleton digest. For the ACTIVE-recovery tier (ungated, condition D), large files are digested to a navigable skeleton: every import, class, and def signature is retained (so the agent can locate and re-read anything it needs), tracebacks keep their tail lines, and bodies are elided. The transform is deterministic, stdlib-only (no model, no network call — auditable and secure), and byte-exact reversible via a content handle. Measured effect: lifted ungated pass@1 from 28.8% to 32.4% by cutting re-expansion thrash (~9.6 → fewer expand calls).
• Sticky expansion. Once the agent recovers a block via distil_expand, the block stays full across all subsequent turns (handles are deterministic, so re-compression is suppressed for that block). This eliminates re-expansion thrash on repeatedly-accessed files without widening the context window for content the agent has moved past.
• Digest mode per tier (honest ablation, detailed above). The skeleton digest helped the ungated (D) tier but collapsed the gated (E) tier to 5.6% — a navigable summary makes the agent stop re-reading, which the gate's head-truncation passive path depends on it doing. Skeleton digest suits active-recovery; head-truncation suits passive relevance-gating.

Certificate scope on long-horizon trajectories

The skeleton digest is 100% byte-exact reversible — its decision-equivalence rate with recovery is 0% (96% raw, like all lossy methods; safety comes from reversibility). The E9 trajectory-composition bound extends the per-turn certificate to multi-turn trajectories: across ~27-turn runs, the naive composition bound becomes vacuous (only ~1.8 turns are outcome-determining), so the per-turn certificate's integrity on full trajectories is tighter than a naive product would suggest, though the formal bound for arbitrary trajectories remains an open problem. The honest position: reversibility is the safety guarantee for the active-recovery tier; non-inferiority to full context is the empirical result for the relevance-gated tier.

E10: trajectory-level decision-equivalence certificate

E2 gave a per-turn guarantee. E7 showed that guarantee does not naively transfer to task success under aggressive compression. E8 showed relevance-gated compression is empirically non-inferior to full context. E10 closes the loop with a real statistical guarantee at the trajectory (task) level — lifting the per-turn certificate all the way to whole-run outcomes.

E10 is the first trajectory-level, distribution-free decision-equivalence certificate for agent context compression (to our knowledge). It runs the same Learn-Then-Test / Hoeffding–Bentkus engine as E2 (distil.conformal.certified_risk_bound), but inverted to a (1−δ) upper confidence bound on per-trajectory 0/1 loss, over the full 500-instance SWE-bench Verified set used in E8. Two loss functions are measured:

• Divergence — trajectory outcome differs from full context (1 if different, 0 if same).
• Harm — full context resolved the task but the compressor did not (1 if compression cost a solvable task, 0 otherwise). This is the stricter, user-relevant loss.

Results — relevance-gated (passive) tier, 500 instances, δ=0.05

Out-of-sample proof (the bound actually holds)

Like E2, E10 is proven out-of-sample, not merely asserted. Over 1000 random calibration/test splits of the 500 instances: the bound β is certified on the calibration half, then the disjoint test half’s realized rate is measured. Coverage lands at 95.4% (divergence) and 96.7% (harm), both at or above the 95% target. The bound holds on held-out data.

Honest scope: ungated reversible tier

The ungated reversible tier (skeleton digest, condition D in E8) also certifies under E10: divergence ≤23.2%, out-of-sample coverage 93.9%. Coverage is marginally below the 95% target — a borderline result, reported here without softening. The relevance gate is the tier with a clean certificate.

Honest scope: exchangeability

The guarantee holds for traffic exchangeable with the calibration distribution (SWE-bench Verified, this agent + model combination). It is not a universal bound. Changing the agent, the model, or the task distribution requires re-certification. Reproducible via benchmarks/trajectory_certificate.py; numbers trace to docs/paper/results/swe_e2e_longhorizon/trajectory_certificate.json.

E11: cross-model generality (five models, three vendors)

Reproducible: benchmarks/long_horizon/run.py --backend openai (DeepSeek-V3, OpenAI) · results in docs/paper/results/swe_e2e_longhorizon_deepseek/, docs/paper/results/swe_e2e_longhorizon_sonnet/, docs/paper/results/swe_e2e_longhorizon_gpt4omini/, docs/paper/results/swe_e2e_longhorizon_gpt41/

E7–E10 use claude-haiku-4-5. To test whether the gate's non-inferiority generalizes, the long-horizon harness was re-run on four more models spanning three vendors: DeepSeek-V3 (deepseek-chat, n=200), Claude Sonnet 4.6 (n=50), gpt-4o-mini (OpenAI, n=50), and gpt-4.1 (OpenAI, n=50). Full-context strength spans a wide range — gpt-4o-mini 12.0%, gpt-4.1 26.0%, Haiku 39.2%, Sonnet 54.0%, DeepSeek-V3 60.0% — letting us separate capability from compression aggressiveness.

gate@6 (aggressive setting): held on Haiku (−2.4 pp), Sonnet (−2.0 pp), and gpt-4o-mini (+0.0 pp, realized 58%); broke on DeepSeek (−31 pp, realized 60%); gpt-4.1 gate@6 partial — OpenAI account credit exhausted mid-run (32/50 instances scored), not reported.

Honest scope: 3 of 5 runs are n=50 (wide CIs, directional not powered). gpt-4.1 full 26% is modest — the ReAct harness is tuned for Claude/DeepSeek (harness-fit caveat, not a distil result). The certificate itself (E2/E10) is model-agnostic by construction.

Operationalized: auto-calibration of the operating point

A workload-dependent operating point is a deployment hazard only if it is hand-tuned — point distil at a new model or task distribution and it could silently ship a lossy setting. distil removes the hazard with the operating-point analogue of the certificate: just as conformal risk control selects the most aggressive compression level whose decision-change rate is provably controlled, distil calibrate (distil/calibrate.py) selects the most aggressive working-set size whose task-success loss is non-inferior to full context (same paired McNemar test), over a small calibration run. If no candidate certifies non-inferior, it fails safe to full context — absence of evidence degrades to no compression, never to silent loss. On the E11 data the procedure recovers the manual choice automatically (selects gate@12, rejects gate@6 on DeepSeek-V3). The relevance gate itself is now a shippable library primitive (distil/gate.py), not benchmark-only. Production status: docs/GA_READINESS.md.

distil calibrate \
  --baseline scores/full.json \
  --candidate gate@6=scores/gate6.json:6 \
  --candidate gate@12=scores/gate12.json:12 \
  --margin 0.05
# → ✔ SELECTED 'gate@12' → DISTIL_E7_GATE_RECENT=12  (gate@6 ✘ too aggressive)

E12: the cost frontier under the motto

Five techniques are shipped or in-progress; status is reported honestly for each.

Honest caveat on #1 (cache-monotone gate). On content that is already fully cacheable, caching alone can be cheaper than any compression — compressing rewrites cached bytes as fresh. The cache-monotone gate’s win is over a cache-hostile gate, not over no-compression; the gate’s primary payoff remains accuracy (E8/E11).

Honest caveat on #3 (empirical-Bernstein). For binary decision-change losses, Bentkus is already near-optimal. Empirical-Bernstein applies to the graded losses introduced by the graded gate (#2), which live in the low-variance regime where EB tightens.

Full details and production status: docs/GA_READINESS.md.

E13: continuous assurance under drift

The certificate is valid under exchangeability, so its standing operational risk is silent drift — a new model or workload pushes the true decision-change rate above the budget α the operating point was certified at. Three shipped pieces close it:

• Anytime-valid drift monitor (distil/drift.py:DriftMonitor) — a betting e-process for H0: risk ≤ α (hedged capital, Waudby-Smith & Ramdas 2023) whose capital is a supermartingale under H0; by Ville’s inequality the false-alarm probability is ≤ δ however often the stream is inspected. So you can check live decision-change after every turn with no multiplicity penalty; crossing 1/δ signals drift beyond budget → recalibrate or fall back to full context. Validated for bounded false alarms under continuous peeking + high detection power (tests/test_drift.py).
• Anytime-valid, variance-adaptive certificate (distil/conformal.py:betting_upper_bound) — the same betting bound certifies graded losses simultaneously at every t. Honest tradeoff: for one-shot binary losses Bentkus is already near-optimal (betting is comparable there); betting’s edge is continuous monitoring and graded-loss adaptivity.
• Cross-family grader ensemble (distil/ensemble.py:EnsembleGrader) — grade with multiple model families; the default “any-change” aggregation is conservative (can only raise measured risk), so the certificate stays valid even if one grader family is unfaithful. Aggregation logic shipped + tested; multi-family validation needs a live multi-API run.

Certified compression frontier

Module: distil/eval.py · Command: distil eval [--corpus --runner --out]

Most compression tools publish a headline savings number. Distil publishes the full curve: a sweep of compression aggressiveness levels where every point is tagged with (token savings, decision-equivalence, certified/not). This locates the cliff past which lossy compression starts dropping agent decisions — and proves that distil sits safely inside the certified region.

How the sweep works

distil/eval.py defines frontier(), which measures two reference operating points (tier-0 lossless and distil (cache-aware)) and then sweeps a range of truncation limits ([4000, 2000, 1200, 700, 450, 300, 220, 160, 120, 90] chars) across the corpus. For each point it measures:

• Token savings — average reduction fraction over the corpus.
• Decision equivalence — average match rate between compressed and uncompressed decisions.
• Certified — both TOST non-inferiority passes and equivalence equals 100%. A single trajectory failure uncertifies the point.

Results are sorted by savings ascending and printed as the frontier table. Run offline with the deterministic runner (structural decision-equivalence). For real task-accuracy on a benchmark (tau-bench, SWE-bench, GSM8K) ingest its traces and run with --runner anthropic.

copy$ distil eval
certified compression frontier  (runner=deterministic)

level                   savings   equiv  certified  curve
--------------------------------------------------------------------------
distil (cache-aware)       8.4%    100%    ✔ PASS   ██
truncate@1200              7.2%     79%       ✘ —    ██
truncate@700              20.0%     36%       ✘ —    ████
truncate@300              41.3%      0%       ✘ —    █████████
--------------------------------------------------------------------------

distil: 8.4% savings @ 100% decision-equivalence — certified.
certified ceiling: 8.4% savings (beyond this, lossy compression drops decisions and the gate rejects it).

The raw curve can be written to JSONL with --out <dir> for CI integration or archiving.

Flags

Decision-Equivalence Risk Certificate (DERC)

Module: distil/conformal.py · Command: distil conformal [--alpha --delta --method --corpus --runner]

The frontier above tells you where the cliff is. The DERC turns that into an operational guarantee: you pick a risk budget — a maximum tolerable decision-change rate α — and Distil certifies the most aggressive compression level whose risk is provably below it, distribution-free and finite-sample, calibrated on your own traffic. This is conformal risk control, not a heuristic threshold.

How it certifies

Distil calibrates a ladder of compression levels (byte-exact → lossless → a truncation sweep), measures the decision-change rate at each, and selects the most aggressive controlled level via one of two procedures:

• Learn-Then-Test (LTT) — Angelopoulos, Bates, Candès, Jordan & Lei, Ann. Appl. Stat. 2025 (arXiv:2110.01052). Risk control as multiple hypothesis testing; Hoeffding–Bentkus p-values with fixed-sequence testing yield P(R≤α)≥1−δ. The default.
• Conformal Risk Control (CRC) — Angelopoulos, Bates, Fisch, Lei & Schuster, ICLR 2024 (arXiv:2208.02814). For a monotone 0/1 loss, controls the expected rate E[L]≤α, tight to O(1/n). Use --method crc.

copy$ distil conformal --corpus ./mycorpus --alpha 0.05 --delta 0.05
Decision-Equivalence Risk Certificate (DERC)

  method      : LTT  (Learn-Then-Test)
  risk target : α = 0.05  (max allowed decision-change rate)
  confidence  : 95%  (1 − δ)
  calibration : n = 320 turns,  runner = deterministic
----------------------------------------------------------------
  ✔ CERTIFIED  'lossless'  →  57.4% token savings

  the decision-change rate vs. uncompressed context is ≤ 5.0%
  with 95% confidence (Learn-Then-Test, n=320 calibration turns).

Why this is novel

Conformal prediction is established theory, but applying it to context compression with the loss defined as agent decision-equivalence is, to our reading of the literature, open white space. The nearest neighbour (arXiv:2511.17908, ECIR 2026) applies conformal guarantees to RAG retrieval recall — a different task (which documents to fetch), not how far you can crush the context an agent already holds while it keeps acting identically.

Flags

Salience protection — keep the needle, crush the haystack

Module: distil/compress/salience.py · Composes with: the certificate (it shifts the frontier)

The certificate tells you how far you can compress safely. Salience protection makes "far" go further: before a lossy level crushes a block, it guarantees the decision-bearing lines survive — so aggressive compression stops dropping load-bearing identifiers and directives. It is model-free (a few hundred microseconds, no transformer in the path), and unlike syntactic entity protectors it blends three signals and works at line granularity so the decision unit (verb + target) stays intact.

Shadow-mode — live decision-equivalence on real traffic

Module: distil/shadow.py · Command: distil proxy --shadow RATE · distil shadow-stats

The certificate proves decision-equivalence offline, on a calibration corpus. Shadow mode closes the loop online. Cost is free to measure live (token deltas); equivalence is not — it needs the counterfactual, the decision the agent would have made on the uncompressed context. So shadow mode samples a fraction of live requests and runs each one both ways, comparing the agent's chosen next action.

The "decision" is the agent's next action — the first tool_use (Anthropic), tool_call (OpenAI), or functionCall (Gemini). Two responses are equivalent iff that action matches, exactly the {action, target} fingerprint the certificate uses. It is streaming-aware: the decision is reconstructed from SSE, so shadow-mode works on real agent sessions (Claude Code, Codex, Gemini CLI) that stream — not just non-streaming SDK calls. The result is a rolling, live decision-change rate on your traffic:

copy$ distil proxy --shadow 0.05 --upstream https://api.anthropic.com
  → shadow-mode live decision-equivalence: sampling 5%

$ distil shadow-stats
  shadowed requests : 412
  decision changes  : 2
  decision-change rate (rolling): 0.49%
  decision-equivalence          : 99.51%

This complements periodic re-certification (distil ingest → distil conformal) under the exchangeability caveat: shadow mode is the continuous monitor, the certificate is the guarantee. Together they make decision-equivalence observable in production, not just in eval.

Cache-delta context coding — cross-version delta, decision-equivalent

Module: distil/cachedelta.py · Command: distil proxy --session-delta · For: coding agents (Claude Code / Codex / Gemini CLI)

The coding-agent hot path is read → edit → re-read. The re-read file is not byte-identical to the first read — one hunk changed — so exact-duplicate dedup (the state of the art elsewhere, e.g. Headroom's dedup_identical_items) misses it and re-sends the whole file as fresh tokens. Cache-delta coding instead sends only the diff against the previously-delivered version, referencing the rest.

AST-structural delta (distil/astdelta.py, stdlib ast, model-free) is the deepest layer: for Python it diffs files by parsed structure, fingerprinting each top-level definition with ast.dump — invariant to whitespace, comments, and import order. A reformat-only re-read is recognised as no definition changed and referenced; only definitions whose AST actually changed are sent in full. Textual diff explodes on reformatting; the structural delta isolates exactly what changed. Non-Python or mid-edit (unparseable) source falls back to the textual delta — it never fails a request, it just does less.

Self-distilling keep-model

Module: distil/online.py · Command: distil online [--corpus --promote-to]

Most learned compressors train a keep-model on a generic judge applied to synthetic data. Distil trains on causal labels — certified-safe drops discovered by the ablation engine running on your own real traffic. The label source is the moat: a model that learns what actually changed a decision, not what a judge guessed.

The never-regressing loop

The loop has four steps, each grounded in the source:

1. Collect causal labels (collect_causal_labels()) — run counterfactual ablation (discover()) on every trajectory. Blocks that never changed a decision are PRUNABLE (label=0). Blocks that changed at least one decision are KEPT (label=1). Only VOLATILE blocks are scanned; the stable cacheable prefix is out of scope. Conflict resolution: keep wins (label=1 is never overridden to 0).
2. Retrain (retrain()) — featurize the labeled lines (9 features, same as the built-in logistic model) and train logistic regression with full-batch gradient descent, L2 regularization, and a deterministic SHA-256 train/test split. No random seed dependency.
3. Certify promotion (certify_promotion()) — wrap the new weights as a compression strategy, run the TOST non-inferiority gate on every single corpus trajectory. A single regression anywhere blocks the promotion.
4. Persist iff certified (online_round()) — write the new weights to --promote-to only if step 3 passed. A cycle that would degrade quality is silently discarded.

Illustrative output — your numbers depend on your corpus

Example from one distil online round on a bundled corpus; the flywheel is real and gated, but exact figures vary by traffic.

copy$ distil online --corpus ./mycorpus --promote-to ./weights/keep_model.json
self-distilling round — keep-model learns from causal labels, gated by non-inferiority

  n_labels: 286
  accuracy: 0.873
  f1: 0.932
  certified: True
  promoted: True

If the gate fails (a regression anywhere in the corpus), promotion is blocked and the output will show certified: False and promoted: False with a message that the candidate failed the non-inferiority gate.

Flags

Verifiable federated telemetry

Module: distil/telemetry.py · Command: distil federated-leaderboard --dir <dir> [--keys <keys.json> --html <out.html>]

Each opt-in instance contributes a signed, content-free savings aggregate with its certification verdict attached. The leaderboard aggregates only verified submissions — every number on the board is tamper-evident. No prompt text, no response content, no tool outputs ever leave the machine: only (instance_id, tokens_saved, dollars_saved, runs, certified, timestamp).

How signing works

Signing uses HMAC-SHA256 (symmetric: both sides share a per-instance key). The canonical form of the aggregate is json.dumps(fields, sort_keys=True, separators=(',', ':')) — deterministic across Python versions and dict orderings. Verification uses hmac.compare_digest for constant-time comparison.

Leaderboard rules

• Submissions whose instance_id has no entry in the keys map are rejected.
• Submissions whose signature does not verify are rejected.
• For each instance, the latest (highest timestamp) verified submission wins.
• Only certified submissions (where certified == True) contribute to headline totals. Uncertified instances are shown but their numbers do not roll into the aggregate.

copy$ distil federated-leaderboard \
    --dir ./submissions \
    --keys ./instance-keys.json \
    --html ./leaderboard.html
verifiable savings — 3 verified instance(s), 0 rejected

  totals (certified only): {'tokens_saved': 142816, 'dollars_saved': 0.071408, 'runs': 47, 'instances': 3}

The --html flag writes a self-contained dark HTML page (the same style as the gateway dashboard) showing per-instance verified savings. The page is served inline — no CDN, no external assets.

Flags