Provider Compaction — certifying the provider's own context manipulation

Anthropic context editing silently clears old tool results before the model sees them. OpenAI server-side compaction silently summarizes them into an opaque item. Every customer of these features is exposed to them; neither vendor publishes what they do to an agent's decisions. That is a structural conflict of interest, not a roadmap gap — no vendor will ship a tool that measures its own degradation. A third party on the wire can. distil certify-provider is that measurement.

Method — the same contract distil applies to itself. Each case is a multi-turn tool transcript replayed three ways against the same model: baseline (manipulation off), edited (manipulation on), and a second independent baseline — the A/A arm that measures the model's self-agreement on identical input, without which any change rate is uninterpretable sampling noise. Decisions are majority-of-3 per arm and compared with the same normalized signature the shadow gate uses (SIG v3). Firing is ground-truthed per request — Anthropic's applied_edits response field, OpenAI's compaction output item — and cases where the manipulation never fired are excluded from the sample rather than diluting it. The protocol (n, α, δ, votes, trigger, model) is written to disk before the first API call; live calls are hard-capped in code; the decision-change rate carries a distribution-free Hoeffding–Bentkus upper bound. Pre-registration matters because deciding n after seeing results is exactly the failure mode this project criticises in others.

Experiment 1 — the worst-case regime: manipulation fires on decision-bearing content

Corpus: 40 seeded synthetic support transcripts (~1.7k tokens, two tool rounds) built so every decision-bearing fact lives in a tool result buried in log noise, with no directive telling the model what to do · Config: maximally aggressive — Anthropic clear_tool_uses with keep=0, trigger 500 tokens; OpenAI compact_threshold=1000 (the API minimum) · Subjects: claude-opus-4-8, gpt-5.2 · Spend: 360 hard-capped calls per provider · Numbers: benchmarks/results/provider-compaction/

Provider manipulation	Fired	A/B change	A/A floor	Adjusted	95% upper bound	Certified at α=0.1
Anthropic context editing (clear, `keep=0`)	40/40	92.5%	7.5%	91.9%	98.6%	No
OpenAI server-side compaction	40/40	12.5%	0.0%	12.5%	27.8%	No

The shape of the failures says more than the rates. Anthropic's clearing produced 37 tool→text flips out of 40: with the load-bearing tool results deleted (mean 2,355 input tokens cleared per case, reported by the API itself), the agent does not pick a wrong action — it stops acting altogether and falls back to prose. Zero tool→different-tool flips. OpenAI's compaction preserved the decision on 35/40, its five flips concentrated in the one deliberately ambiguous scenario (cancel-after-shipment), and gpt-5.2 was perfectly self-consistent (A/A floor 0.0%).

The headline contrast. In the same regime, on the same corpus, summarization preserved agent decisions roughly 7× better than deletion (12.5% vs 92.5% decision change). And the dominant failure mode of deletion is not corruption but stalling: the agent loses the facts it needed and stops acting. Neither feature certifies decision-safe at α=0.1 — and per the house rule (see E7 on the Research page), the numbers are published either way.

Experiment 2 — the default clearing policy (`keep=3`)

Experiment 1 could be dismissed as a strawman: keep=0 deletes everything, so of course decisions move. The question that matters to a customer is what the shipped default does. Anthropic's default clearing policy keeps the 3 most recent tool uses — so this experiment holds keep=3 fixed and asks whether that retention is protection or decoration.

Corpus: 40 seeded transcripts of 7 tool rounds (~4.3k tokens): two decision-bearing rounds at the head (the order and billing records), then four routine ones — inventory, shipping scans, comms log, promotions — carrying no decision-relevant fact · Config: keep=3, the default policy, at a 3k-token trigger · Subjects: claude-opus-4-8, gpt-5.2 · Spend: 360 hard-capped calls per run

Provider manipulation	Fired	A/B change	A/A floor	Adjusted	95% upper bound	Certified at α=0.1
Anthropic context editing (clear, `keep=3`) — run 1	40/40	95.0%	2.5%	94.9%	99.5%	No
Anthropic context editing (clear, `keep=3`) — run 2	40/40	100%	2.5%	100%	100%	No
OpenAI server-side compaction (`compact_threshold=3000`)	40/40	20.0%	2.5%	17.9%	36.7%	No

The pre-registered protocol was executed twice against Anthropic. Both runs are published — selecting the kinder or the crueler one after the fact is precisely the practice this page exists to refuse. They agree on the finding and disagree on the last two cases.

The default is not a weaker version of Experiment 1 — it is a different failure. Under keep=0 the agent lost everything and stopped acting: 37 of 40 flips were tool→text. Under the default keep=3 the stalls nearly vanish and 29–31 of 40 flips become tool→wrong tool. The three surviving tool results are the routine ones — inventory, shipping, promotions — and they are enough to keep the agent confidently in acting mode while the records the policy depends on are gone. It refunds where it should escalate fraud, cancels where it should offer credit. A stalled agent is a visible incident; a wrong one is a silent write.

What retention buys. Keeping the 3 most recent tool uses did not reduce the decision-change rate at all (92.5% → 95.0/100%). It converted a loud failure into a quiet one. Retention counted in recency cannot protect a decision that depends on relevance — and in a long agent trajectory the load-bearing facts are gathered early, then buried under routine calls. That ordering is the normal shape of agentic work, not an adversarial construction.

The contrast holds at the default. Summarization changed 20.0% of decisions where deletion changed 95–100% — roughly a 5× gap, consistent with the ~7× seen in Experiment 1. OpenAI's compaction left 32 of 40 decisions untouched. Neither certifies decision-safe at α=0.1, and both numbers are published either way.

Honest scope

Synthetic corpora, and Experiment 2's is built to be cleared. Experiment 1 bounds the regime where the manipulation fires on decision-bearing content. Experiment 2's corpus is deliberately ordered so that keep=3 retains only routine rounds — it measures what the default policy does when the load-bearing facts are older than the three most recent tool uses, not how often that ordering arises in your traffic. It is a claim about the policy's mechanism, not a prevalence estimate.
The trigger is lowered, the policy is not. Experiment 2 keeps Anthropic's real default keep=3 but fires it at 3k tokens rather than the default 100k, because a 4.3k-token episode cannot cross 100k. What generalises is the retention rule; the threshold is a lever to make it fire at a budget we can afford 40 cases of.
Decision-equivalence, not task success. The measured invariant is the agent's next action. distil's own E7 showed per-step equivalence can pass while end-to-end success collapses; the same caveat applies here in both directions.
The true default-threshold number on long real traffic (100k+-token transcripts crossing the default triggers naturally) remains open — it needs real long-horizon dumps and a proportionally larger budget. The harness runs it unchanged when the transcripts exist.
Point-in-time. Both features are beta surfaces measured on 2026-07-26 against the models named above; vendors can change behavior without notice. Reproduce, don't extrapolate.

Reproduce it

# corpus (seeded, byte-reproducible)
python benchmarks/fixtures/make_provider_episodes.py

# dry-run prints the exact call budget before anything is spent
distil certify-provider benchmarks/fixtures/provider_compaction_episodes.json --dry-run

# experiment 1 (360 hard-capped live calls per provider)
distil certify-provider benchmarks/fixtures/provider_compaction_episodes.json \
  --n 40 --votes 3 --alpha 0.1 --delta 0.05 --trigger-tokens 500 --keep 0 \
  --max-live-calls 360 --out results/anthropic
distil certify-provider benchmarks/fixtures/provider_compaction_episodes.json \
  --provider openai --trigger-tokens 1000 --n 40 --votes 3 \
  --max-live-calls 360 --out results/openai

# experiment 2 — the default keep=3 policy, on the 7-round corpus
distil certify-provider benchmarks/fixtures/provider_compaction_episodes_long.json \
  --n 40 --votes 3 --trigger-tokens 3000 --keep 3 \
  --max-live-calls 360 --out results/anthropic-default-policy
distil certify-provider benchmarks/fixtures/provider_compaction_episodes_long.json \
  --provider openai --n 40 --votes 3 --trigger-tokens 3000 \
  --max-live-calls 360 --out results/openai-threshold-3000

Each output directory carries protocol.json (written before the first call; its hash is embedded in the report) and report.json with per-case signatures. Runs are resumable: every finished case is appended to cases.jsonl stamped with the protocol hash, so an interrupted run replays what it already bought instead of re-buying it — and a changed parameter orphans the ledger rather than blending two experiments. Re-invoke the same command to continue. The same test design extends to model-version migration, routing, quantization, and RAG changes — anything that manipulates what the model sees between you and your agent.

Provider Compaction — certifying the provider's own context manipulation

Experiment 1 — the worst-case regime: manipulation fires on decision-bearing content

Experiment 2 — the default clearing policy (keep=3)

Honest scope

Reproduce it

Experiment 2 — the default clearing policy (`keep=3`)