Research & Frontier

The proof artifacts no competitor publishes: a savings-vs-quality curve where every point carries a certification verdict, a distribution-free risk certificate on the decision-change rate, a never-regressing self-distilling loop, and savings telemetry you can verify yourself.

E14: provider-native context manipulation (the instrument turned outward)

Harness: distil certify-provider · Numbers: benchmarks/results/provider-compaction/ · Full study: Provider Compaction

E1–E13 point the decision-equivalence contract at distil’s own compressor. E14 points the same instrument at the two features that now rewrite the context window by default, server-side: Anthropic context editing and OpenAI server-side compaction. Neither vendor publishes what its feature does to an agent’s decisions — a structural conflict of interest, since no vendor ships a tool that measures its own degradation. Same protocol as everywhere else: pre-registered (protocol.json hashed before the first API call), majority-of-3 decisions per arm, an A/A arm for the self-agreement floor, SIG v3 signatures, live calls hard-capped, Hoeffding–Bentkus upper bound.

The finding is the mechanism, not the rate. Anthropic’s default retention (keep=3) did not reduce the decision-change rate versus deleting everything — 92.5% → 95–100%. It changed the failure mode. Under keep=0 the agent lost the load-bearing facts and stopped acting: 37 of 40 flips were tool→text. Under the default, stalls nearly vanish and 29–31 of 40 flips become tool→wrong tool — the three retained results are the routine ones (inventory, shipping, promotions), enough to keep the agent confidently acting while the records its policy depends on are gone. It refunds where it should escalate fraud. A stalled agent is a visible incident; a wrong one is a silent write.

Why retention counted in recency cannot help. The load-bearing facts in an agent trajectory are gathered early, then buried under routine calls. A policy that keeps the most recent N tool uses cannot protect a decision that depends on relevance. That ordering is the normal shape of agentic work, not an adversarial construction — and it is why the default is the dangerous setting rather than the safe one.

The contrast. Summarization preserved decisions roughly 5–7× better than deletion, at both the aggressive and default configs. Neither feature certifies decision-safe at α=0.1, and per the house rule (see E7) both numbers are published either way. The pre-registered protocol was executed twice against Anthropic and both runs are published — selecting the kinder or crueler one after the fact is the practice this experiment exists to refuse.

Honest scope. Synthetic decision-bearing corpora, deliberately ordered so that keep=3 retains only routine rounds: this measures what the policy’s mechanism does when load-bearing facts age out, not how often that arises in your traffic. The trigger is lowered to 3k (from the 100k default) so 40 cases fit a budget; the retention rule is the real default. Decision-equivalence is the measured invariant, not task success — E7 showed those can diverge. Point-in-time, 2026-07-26, on the models named. Reproduce, don’t extrapolate — the commands are published.

E7: SWE-bench Verified end-to-end task-success

Harness: benchmarks/swe_bench_e2e/ (official swebench 4.1.0) · Numbers: docs/paper/results/swe_bench_verified_e2e.json · Source: PR #8

E1–E5 measure decision-equivalence on edit-localization — a proxy. E7 is the first non-proxy test: does the certified operating point survive real execution? A real agent (aider + claude-sonnet-4-6, temp 0, diff edit format) is run end-to-end on SWE-bench Verified (n=50, seed 1729) and scored by the official swebench harness, across three conditions sharing the identical agent: (A) full context, (B) distil at its certified trunc@500 operating point, and (C) LLMLingua-2.

distil's certified point loses to full context by 36 points (paired McNemar p<0.001) and does not beat LLMLingua-2. At trunc@500 it strips 86% of agentic context (vs LLMLingua-2's 48%), so its lower score is partly aggression, not purely method.

E8: long-horizon agent task-success

Harness: benchmarks/swe_bench_e2e/ (official swebench) · Numbers: docs/paper/results/swe_e2e_longhorizon/reports/ (per-condition resolved-instance reports)

E7 flagged relevance-gated reversible compression as an untested path on real long-horizon execution. E8 runs that test across six conditions, including a Headroom competitor and LLMLingua-2. A custom multi-turn ReAct coding agent (read / search / edit_file / run_tests tools, up to 30 turns, claude-haiku-4-5, temp 0) is run end-to-end on the full 500-instance SWE-bench Verified set, scored by the official swebench harness (hidden tests, per-instance Docker). Runs average ~27 turns — read-file and tool outputs accumulate into large peripheral context behind a small active working set, which is exactly the regime the relevance gate was designed for. Six conditions share the identical agent; only the compressor differs.

All six conditions score the same 500 instances (paired McNemar). Gate vs full context: difference = −2.4 pp, 95% CI [−5.7, +0.9], McNemar p=0.19 — non-inferior to full at a 6 pp margin (borderline at a strict 5 pp margin; p=0.19 means no significant difference was detected, not that the two are equivalent). Gate vs Headroom: +4.2 pp, McNemar p=0.035 — distil's relevance-gated tier beats Headroom with statistical significance. Lossy truncation (B) vs full context: p<0.001. Gate vs LLMLingua-2: 174 instances the gate solved that LLMLingua-2 did not, 2 the reverse — McNemar p<0.001. Recovery round-trips tell the mechanism: ungated (D) issues 9.6 distil_expand calls per instance (skeleton digested the whole history, must keep recovering the working set); gated (E) issues only 0.58 per instance (digests only aged-out periphery, keeping the working set full). An honest ablation: applying the skeleton digest to the gated tier (E) — which uses head-truncation in its passive path — regressed pass@1 from 36.8% to 5.6%, matching condition B. The navigable digest makes the agent over-trust the summary and stop re-reading; head-truncation is the correct digest for the passive tier.

New techniques in E8: skeleton digest and sticky expansion

Two new techniques were introduced and ablated in E8:

• Content-aware skeleton digest. For the ACTIVE-recovery tier (ungated, condition D), large files are digested to a navigable skeleton: every import, class, and def signature is retained (so the agent can locate and re-read anything it needs), tracebacks keep their tail lines, and bodies are elided. The transform is deterministic, stdlib-only (no model, no network call — auditable and secure), and byte-exact reversible via a content handle. Measured effect: lifted ungated pass@1 from 28.8% to 32.4% by cutting re-expansion thrash (~9.6 → fewer expand calls).
• Sticky expansion. Once the agent recovers a block via distil_expand, the block stays full across all subsequent turns (handles are deterministic, so re-compression is suppressed for that block). This eliminates re-expansion thrash on repeatedly-accessed files without widening the context window for content the agent has moved past.
• Digest mode per tier (honest ablation, detailed above). The skeleton digest helped the ungated (D) tier but collapsed the gated (E) tier to 5.6% — a navigable summary makes the agent stop re-reading, which the gate's head-truncation passive path depends on it doing. Skeleton digest suits active-recovery; head-truncation suits passive relevance-gating.

Certificate scope on long-horizon trajectories

The skeleton digest is 100% byte-exact reversible — its decision-equivalence rate with recovery is 0% (96% raw, like all lossy methods; safety comes from reversibility). The E9 trajectory-composition bound extends the per-turn certificate to multi-turn trajectories: across ~27-turn runs, the naive composition bound becomes vacuous (only ~1.8 turns are outcome-determining), so the per-turn certificate's integrity on full trajectories is tighter than a naive product would suggest, though the formal bound for arbitrary trajectories remains an open problem. The honest position: reversibility is the safety guarantee for the active-recovery tier; non-inferiority to full context is the empirical result for the relevance-gated tier.

E10: trajectory-level decision-equivalence certificate

E2 gave a per-turn guarantee. E7 showed that guarantee does not naively transfer to task success under aggressive compression. E8 showed relevance-gated compression is empirically non-inferior to full context. E10 closes the loop with a real statistical guarantee at the trajectory (task) level — lifting the per-turn certificate all the way to whole-run outcomes.

E10 is the first trajectory-level, distribution-free decision-equivalence certificate for agent context compression (to our knowledge). It runs the same Learn-Then-Test / Hoeffding–Bentkus engine as E2 (distil.conformal.certified_risk_bound), but inverted to a (1−δ) upper confidence bound on per-trajectory 0/1 loss, over the full 500-instance SWE-bench Verified set used in E8. Two loss functions are measured:

• Divergence — trajectory outcome differs from full context (1 if different, 0 if same).
• Harm — full context resolved the task but the compressor did not (1 if compression cost a solvable task, 0 otherwise). This is the stricter, user-relevant loss.

Results — relevance-gated (passive) tier, 500 instances, δ=0.05

Out-of-sample proof (the bound actually holds)

Like E2, E10 is proven out-of-sample, not merely asserted. Over 1000 random calibration/test splits of the 500 instances: the bound β is certified on the calibration half, then the disjoint test half’s realized rate is measured. Coverage lands at 95.4% (divergence) and 96.7% (harm), both at or above the 95% target. The bound holds on held-out data.

Honest scope: ungated reversible tier

The ungated reversible tier (skeleton digest, condition D in E8) also certifies under E10: divergence ≤23.2%, out-of-sample coverage 93.9%. Coverage is marginally below the 95% target — a borderline result, reported here without softening. The relevance gate is the tier with a clean certificate.

Honest scope: exchangeability

The guarantee holds for traffic exchangeable with the calibration distribution (SWE-bench Verified, this agent + model combination). It is not a universal bound. Changing the agent, the model, or the task distribution requires re-certification. Reproducible via benchmarks/trajectory_certificate.py; numbers trace to docs/paper/results/swe_e2e_longhorizon/trajectory_certificate.json.

E11: cross-model generality (five models, three vendors)

Reproducible: benchmarks/long_horizon/run.py --backend openai (DeepSeek-V3, OpenAI) · results in docs/paper/results/swe_e2e_longhorizon_deepseek/, docs/paper/results/swe_e2e_longhorizon_sonnet/, docs/paper/results/swe_e2e_longhorizon_gpt4omini/, docs/paper/results/swe_e2e_longhorizon_gpt41/

E7–E10 use claude-haiku-4-5. To test whether the gate's non-inferiority generalizes, the long-horizon harness was re-run on four more models spanning three vendors: DeepSeek-V3 (deepseek-chat, n=200), Claude Sonnet 4.6 (n=50), gpt-4o-mini (OpenAI, n=50), and gpt-4.1 (OpenAI, n=50). Full-context strength spans a wide range — gpt-4o-mini 12.0%, gpt-4.1 26.0%, Haiku 39.2%, Sonnet 54.0%, DeepSeek-V3 60.0% — letting us separate capability from compression aggressiveness.

gate@6 (aggressive setting): held on Haiku (−2.4 pp), Sonnet (−2.0 pp), and gpt-4o-mini (+0.0 pp, realized 58%); broke on DeepSeek (−31 pp, realized 60%); gpt-4.1 gate@6 partial — OpenAI account credit exhausted mid-run (32/50 instances scored), not reported.

Honest scope: 3 of 5 runs are n=50 (wide CIs, directional not powered). gpt-4.1 full 26% is modest — the ReAct harness is tuned for Claude/DeepSeek (harness-fit caveat, not a distil result). The certificate itself (E2/E10) is model-agnostic by construction.

Operationalized: auto-calibration of the operating point

A workload-dependent operating point is a deployment hazard only if it is hand-tuned — point distil at a new model or task distribution and it could silently ship a lossy setting. distil removes the hazard with the operating-point analogue of the certificate: just as conformal risk control selects the most aggressive compression level whose decision-change rate is provably controlled, distil calibrate (distil/calibrate.py) selects the most aggressive working-set size whose task-success loss is non-inferior to full context (same paired McNemar test), over a small calibration run. If no candidate certifies non-inferior, it fails safe to full context — absence of evidence degrades to no compression, never to silent loss. On the E11 data the procedure recovers the manual choice automatically (selects gate@12, rejects gate@6 on DeepSeek-V3). The relevance gate itself is now a shippable library primitive (distil/gate.py), not benchmark-only. Production status: docs/GA_READINESS.md.

distil calibrate \
  --baseline scores/full.json \
  --candidate gate@6=scores/gate6.json:6 \
  --candidate gate@12=scores/gate12.json:12 \
  --margin 0.05
# → ✔ SELECTED 'gate@12' → DISTIL_E7_GATE_RECENT=12  (gate@6 ✘ too aggressive)

E12: the cost frontier under the motto

Five techniques are shipped or in-progress; status is reported honestly for each.

Honest caveat on #1 (cache-monotone gate). On content that is already fully cacheable, caching alone can be cheaper than any compression — compressing rewrites cached bytes as fresh. The cache-monotone gate’s win is over a cache-hostile gate, not over no-compression; the gate’s primary payoff remains accuracy (E8/E11).

Honest caveat on #3 (empirical-Bernstein). For binary decision-change losses, Bentkus is already near-optimal. Empirical-Bernstein applies to the graded losses introduced by the graded gate (#2), which live in the low-variance regime where EB tightens.

Full details and production status: docs/GA_READINESS.md.

E13: continuous assurance under drift

The certificate is valid under exchangeability, so its standing operational risk is silent drift — a new model or workload pushes the true decision-change rate above the budget α the operating point was certified at. Three shipped pieces close it:

• Anytime-valid drift monitor (distil/drift.py:DriftMonitor) — a betting e-process for H0: risk ≤ α (hedged capital, Waudby-Smith & Ramdas 2023) whose capital is a supermartingale under H0; by Ville’s inequality the false-alarm probability is ≤ δ however often the stream is inspected. So you can check live decision-change after every turn with no multiplicity penalty; crossing 1/δ signals drift beyond budget → recalibrate or fall back to full context. Validated for bounded false alarms under continuous peeking + high detection power (tests/test_drift.py).
• Anytime-valid, variance-adaptive certificate (distil/conformal.py:betting_upper_bound) — the same betting bound certifies graded losses simultaneously at every t. Honest tradeoff: for one-shot binary losses Bentkus is already near-optimal (betting is comparable there); betting’s edge is continuous monitoring and graded-loss adaptivity.
• Cross-family grader ensemble (distil/ensemble.py:EnsembleGrader) — grade with multiple model families; the default “any-change” aggregation is conservative (can only raise measured risk), so the certificate stays valid even if one grader family is unfaithful. Aggregation logic shipped + tested; multi-family validation needs a live multi-API run.

Certified compression frontier

Module: distil/eval.py · Command: distil eval [--corpus --runner --out]

Most compression tools publish a headline savings number. Distil publishes the full curve: a sweep of compression aggressiveness levels where every point is tagged with (token savings, decision-equivalence, certified/not). This locates the cliff past which lossy compression starts dropping agent decisions — and proves that distil sits safely inside the certified region.

How the sweep works

distil/eval.py defines frontier(), which measures two reference operating points (tier-0 lossless and distil (cache-aware)) and then sweeps a range of truncation limits ([4000, 2000, 1200, 700, 450, 300, 220, 160, 120, 90] chars) across the corpus. For each point it measures:

• Token savings — average reduction fraction over the corpus.
• Decision equivalence — average match rate between compressed and uncompressed decisions.
• Certified — both TOST non-inferiority passes and equivalence equals 100%. A single trajectory failure uncertifies the point.

Results are sorted by savings ascending and printed as the frontier table. Run offline with the deterministic runner (structural decision-equivalence). For real task-accuracy on a benchmark (tau-bench, SWE-bench, GSM8K) ingest its traces and run with --runner anthropic.

$ distil eval
certified compression frontier  (runner=deterministic)

level                   savings   equiv  certified  curve
--------------------------------------------------------------------------
distil (cache-aware)       8.4%    100%    ✔ PASS   ██
truncate@1200              7.2%     79%       ✘ —    ██
truncate@700              20.0%     36%       ✘ —    ████
truncate@300              41.3%      0%       ✘ —    █████████
--------------------------------------------------------------------------

distil: 8.4% savings @ 100% decision-equivalence — certified.
certified ceiling: 8.4% savings (beyond this, lossy compression drops decisions and the gate rejects it).

The raw curve can be written to JSONL with --out <dir> for CI integration or archiving.

Flags

Decision-Equivalence Risk Certificate (DERC)

Module: distil/conformal.py · Command: distil conformal [--alpha --delta --method --corpus --runner]

The frontier above tells you where the cliff is. The DERC turns that into an operational guarantee: you pick a risk budget — a maximum tolerable decision-change rate α — and Distil certifies the most aggressive compression level whose risk is provably below it, distribution-free and finite-sample, calibrated on your own traffic. This is conformal risk control, not a heuristic threshold.

How it certifies

Distil calibrates a ladder of compression levels (byte-exact → lossless → a truncation sweep), measures the decision-change rate at each, and selects the most aggressive controlled level via one of two procedures:

• Learn-Then-Test (LTT) — Angelopoulos, Bates, Candès, Jordan & Lei, Ann. Appl. Stat. 2025 (arXiv:2110.01052). Risk control as multiple hypothesis testing; Hoeffding–Bentkus p-values with fixed-sequence testing yield P(R≤α)≥1−δ. The default.
• Conformal Risk Control (CRC) — Angelopoulos, Bates, Fisch, Lei & Schuster, ICLR 2024 (arXiv:2208.02814). For a monotone 0/1 loss, controls the expected rate E[L]≤α, tight to O(1/n). Use --method crc.

$ distil conformal --corpus ./mycorpus --alpha 0.05 --delta 0.05
Decision-Equivalence Risk Certificate (DERC)

  method      : LTT  (Learn-Then-Test)
  risk target : α = 0.05  (max allowed decision-change rate)
  confidence  : 95%  (1 − δ)
  calibration : n = 320 turns,  runner = deterministic
----------------------------------------------------------------
  ✔ CERTIFIED  'lossless'  →  57.4% token savings

  the decision-change rate vs. uncompressed context is ≤ 5.0%
  with 95% confidence (Learn-Then-Test, n=320 calibration turns).

Why this is novel

Conformal prediction is established theory, but applying it to context compression with the loss defined as agent decision-equivalence is, to our reading of the literature, open white space. The nearest neighbour (arXiv:2511.17908, ECIR 2026) applies conformal guarantees to RAG retrieval recall — a different task (which documents to fetch), not how far you can crush the context an agent already holds while it keeps acting identically.

Flags

Salience protection — keep the needle, crush the haystack

Module: distil/compress/salience.py · Composes with: the certificate (it shifts the frontier)

The certificate tells you how far you can compress safely. Salience protection makes "far" go further: before a lossy level crushes a block, it guarantees the decision-bearing lines survive — so aggressive compression stops dropping load-bearing identifiers and directives. It is model-free (a few hundred microseconds, no transformer in the path), and unlike syntactic entity protectors it blends three signals and works at line granularity so the decision unit (verb + target) stays intact.

Shadow-mode — live decision-equivalence on real traffic

Module: distil/shadow.py · Command: distil proxy --shadow RATE · distil shadow-stats

The certificate proves decision-equivalence offline, on a calibration corpus. Shadow mode closes the loop online. Cost is free to measure live (token deltas); equivalence is not — it needs the counterfactual, the decision the agent would have made on the uncompressed context. So shadow mode samples a fraction of live requests and runs each one both ways, comparing the agent's chosen next action.

The "decision" is the agent's next action — the first tool_use (Anthropic), tool_call (OpenAI), or functionCall (Gemini). Two responses are equivalent iff that action matches, exactly the {action, target} fingerprint the certificate uses. It is streaming-aware: the decision is reconstructed from SSE, so shadow-mode works on real agent sessions (Claude Code, Codex, Gemini CLI) that stream — not just non-streaming SDK calls. The result is a rolling, live decision-change rate on your traffic:

$ distil proxy --shadow 0.05 --upstream https://api.anthropic.com
  → shadow-mode live decision-equivalence: sampling 5%

$ distil shadow-stats
  shadowed requests : 412
  decision changes  : 2
  decision-change rate (rolling): 0.49%
  decision-equivalence          : 99.51%

This complements periodic re-certification (distil ingest → distil conformal) under the exchangeability caveat: shadow mode is the continuous monitor, the certificate is the guarantee. Together they make decision-equivalence observable in production, not just in eval.

Cache-delta context coding — cross-version delta, decision-equivalent

Module: distil/cachedelta.py · Command: distil proxy --session-delta · For: coding agents (Claude Code / Codex / Gemini CLI)

The coding-agent hot path is read → edit → re-read. The re-read file is not byte-identical to the first read — one hunk changed — so exact-duplicate dedup (the state of the art elsewhere, e.g. Headroom's dedup_identical_items) misses it and re-sends the whole file as fresh tokens. Cache-delta coding instead sends only the diff against the previously-delivered version, referencing the rest.

AST-structural delta (distil/astdelta.py, stdlib ast, model-free) is the deepest layer: for Python it diffs files by parsed structure, fingerprinting each top-level definition with ast.dump — invariant to whitespace, comments, and import order. A reformat-only re-read is recognised as no definition changed and referenced; only definitions whose AST actually changed are sent in full. Textual diff explodes on reformatting; the structural delta isolates exactly what changed. Non-Python or mid-edit (unparseable) source falls back to the textual delta — it never fails a request, it just does less.

Self-distilling keep-model

Module: distil/online.py · Command: distil online [--corpus --promote-to]

Most learned compressors train a keep-model on a generic judge applied to synthetic data. Distil trains on causal labels — certified-safe drops discovered by the ablation engine running on your own real traffic. The label source is the moat: a model that learns what actually changed a decision, not what a judge guessed.

The never-regressing loop

The loop has four steps, each grounded in the source:

1. Collect causal labels (collect_causal_labels()) — run counterfactual ablation (discover()) on every trajectory. Blocks that never changed a decision are PRUNABLE (label=0). Blocks that changed at least one decision are KEPT (label=1). Only VOLATILE blocks are scanned; the stable cacheable prefix is out of scope. Conflict resolution: keep wins (label=1 is never overridden to 0).
2. Retrain (retrain()) — featurize the labeled lines (9 features, same as the built-in logistic model) and train logistic regression with full-batch gradient descent, L2 regularization, and a deterministic SHA-256 train/test split. No random seed dependency.
3. Certify promotion (certify_promotion()) — wrap the new weights as a compression strategy, run the TOST non-inferiority gate on every single corpus trajectory. A single regression anywhere blocks the promotion.
4. Persist iff certified (online_round()) — write the new weights to --promote-to only if step 3 passed. A cycle that would degrade quality is silently discarded.

Illustrative output — your numbers depend on your corpus

Example from one distil online round on a bundled corpus; the flywheel is real and gated, but exact figures vary by traffic.

$ distil online --corpus ./mycorpus --promote-to ./weights/keep_model.json
self-distilling round — keep-model learns from causal labels, gated by non-inferiority

  n_labels: 286
  accuracy: 0.873
  f1: 0.932
  certified: True
  promoted: True

If the gate fails (a regression anywhere in the corpus), promotion is blocked and the output will show certified: False and promoted: False with a message that the candidate failed the non-inferiority gate.

Flags

Verifiable federated telemetry

Module: distil/telemetry.py · Command: distil federated-leaderboard --dir <dir> [--keys <keys.json> --html <out.html>]

Each opt-in instance contributes a signed, content-free savings aggregate with its certification verdict attached. The leaderboard aggregates only verified submissions — every number on the board is tamper-evident. No prompt text, no response content, no tool outputs ever leave the machine: only (instance_id, tokens_saved, dollars_saved, runs, certified, timestamp).

How signing works

Signing uses HMAC-SHA256 (symmetric: both sides share a per-instance key). The canonical form of the aggregate is json.dumps(fields, sort_keys=True, separators=(',', ':')) — deterministic across Python versions and dict orderings. Verification uses hmac.compare_digest for constant-time comparison.

Leaderboard rules

• Submissions whose instance_id has no entry in the keys map are rejected.
• Submissions whose signature does not verify are rejected.
• For each instance, the latest (highest timestamp) verified submission wins.
• Only certified submissions (where certified == True) contribute to headline totals. Uncertified instances are shown but their numbers do not roll into the aggregate.

$ distil federated-leaderboard \
    --dir ./submissions \
    --keys ./instance-keys.json \
    --html ./leaderboard.html
verifiable savings — 3 verified instance(s), 0 rejected

  totals (certified only): {'tokens_saved': 142816, 'dollars_saved': 0.071408, 'runs': 47, 'instances': 3}

The --html flag writes a self-contained dark HTML page (the same style as the gateway dashboard) showing per-instance verified savings. The page is served inline — no CDN, no external assets.

Flags