Cache-aware compaction and a prompt-cache keep-alive

Sorcar, 2026-09-20. Model in production: claude-fable-5-1 (82 % of spend). Sources: projects/cost-levers-followup-2026-09-20/FINDINGS.md sections 2 and 4, the Anthropic prompt-caching, context-editing and preserved-thinking documentation, and the 72 h replay in results/compaction_sim_gated_2026-09-20.json.

1. The problem

Every step of an agent re-sends the whole conversation. With prompt caching that costs 0.025× the input price on Fable 5.1 ($0.25 per million tokens) as long as the prefix is unchanged and the cache is warm. Two things break the prefix:

Compaction: stubbing old tool outputs edits messages in the middle of the conversation, so the next request re-writes everything after the first edit at $12.50 per million tokens. In the 72 h window this was 56 steps and $93.92 on Fable 5.1.
TTL expiry: the cache lives five minutes, counted from the start of the request that last touched it. A build, test suite or sub-agent fan-out that runs longer lets it expire. 40 steps, $87.58.

2. What changed

Compaction gate (`src/kiss/core/context_compaction.py`)

The compaction is planned first (plan_compaction) and applied only when should_compact agrees:

drop_tokens = dropped_chars(plan) // CHARS_PER_TOKEN          # 4 chars per token
compact if drop_tokens >= 0.25 * context_tokens               # MIN_DROP_FRACTION
       and context_tokens < 0.8 * handoff_limit                # HANDOFF_PROXIMITY_FRACTION
                                                              # handoff_limit = 0.7 * window (350k on 500k)

Defaults moved from keep 20 / min 2,000 chars / re-trigger +50k to keep 6 / min 500 / +100k. A skipped compaction is re-planned on every following step, because more results age out of the kept window. context_tokens is the input size the last completed request reported, which is what the agent knows before sending the next one.

Keep-alive (`src/kiss/core/prompt_cache_keepalive.py`, `AnthropicModel.keep_prompt_cache_warm`)

A tool call qualifies when its timeout/timeout_seconds argument is at least 300 s or it is a fan-out (run_parallel, run_commands_parallel, run_agent). While it runs, a daemon thread sends a ping every 240 s, the first one 240 s after the model request that preceded the tool call. The ping is the same request the step sent (tools, system, thinking, messages, byte for byte) plus one user message answering each in-flight tool_use with a placeholder tool_result, with max_tokens lowered to 256 (not part of the cached prefix). Its reply is discarded; its cost is added to the agent's budget on the agent thread after the tool returns, also when the tool ends by an interrupt or budget error. At most 10 pings per call; none when caching is disabled for the model; retries off and a 30 s timeout so the agent never waits long for a ping in flight.

Timeline of one long tool call. The TTL clock starts at the model request, not when the tool starts, which is why the first ping is scheduled from the request timestamp.

3. Measurement

3.1 Live 72 h KPI (`compare_kpis.py --hours 72`, 07:30 UTC)

Unchanged from the 06:24 UTC run except for the window shift, because the running daemon still executes the pre-change code. These are therefore the before numbers for this lever (claude-fable-5-1, filtered to post-lever trees, 3,728 steps, $1,193):

step type	steps	cost	share	$/step	avg context
normal step	3,561	$626.86	52.5 %	0.176	129,686
fan-out step (children folded in)	14	$369.25	30.9 %	26.37	217,255
cache miss: compaction	56	$93.92	7.9 %	1.68	132,112
cache miss: ≥ 5 min gap	40	$87.58	7.3 %	2.19	173,934
first step of a session	56	$14.46	1.2 %	0.26	24,832

The scheduled cost-levers-72h-recheck cron job (2026-09-23 06:30 UTC) re-runs this table after the change has been running; the two cache-miss rows are what should move.

3.2 Replay of the last 72 h through the new code

213 Claude sessions of at least 20 steps were rebuilt from the event log and replayed through the real compaction functions (compaction_simulation.py). Input cost is priced at Fable 5.1 cache rates: the first step and every step after a compaction write the whole context, every other step reads it. Per-step growth is left out because it is the same in every policy.

policy	compactions	input bill	vs production	steps ≥ 100k	sessions reaching 350k
A: no compaction	0	$542	0.62	52.1 %	12
B: production 09-19 (100k/50k, keep 20, min 2000)	217	$874	1.00	45.3 %	6
H: new defaults, no gate	164	$706	0.81	42.4 %	6
I: new defaults + gate (implemented)	67	$575	0.66	46.5 %	11
J: drop gate only, no hand-off skip	67	$575	0.66	46.5 %	11

The gate removes 70 % of the compactions and a third of the compaction-related input bill, but on this model the cheapest input bill is still "never compact".

3.3 What the replay says

The gate cuts the compaction-related input bill by 34 % against the 09-19 production policy (about $299 per 72 h at this volume) and the number of compactions from 217 to 67.
At a 0.025× cache-read price a compaction that drops a quarter of the context needs about 196 further steps to pay back, so "no compaction" is still $32 cheaper than the gated policy on input alone. What compaction buys on Fable 5.1 is avoiding hand-offs, and the 25 % gate gives most of that back: 11 sessions reach the 350k hand-off limit against 6 with the old policy and 12 with none.
The hand-off proximity skip changed no decision (I equals J): a 25 % drop is never available that late in a session, so the drop gate already covers it.

4. Risk found while researching

Anthropic's preserved-thinking prefix check (Claude Fable 5.1 and later) treats a shortened earlier tool_result as invalidating every later thinking block. Accounts created on or after 2026-08-31 get a 400 by default, and "later models will enforce the prefix check for all accounts". The client-side stubbing that compaction does today therefore works on this account but is a latent failure. Anthropic's replacement is server-side tool-result clearing (clear_tool_uses_20250919, beta header context-management-2025-06-27) with clear_at_least as the cache-worth gate. Not changed in this task.

5. Verification

19 end-to-end tests in src/kiss/tests/core/test_context_compaction.py and test_prompt_cache_keepalive.py; the keep-alive tests run a real KISSAgent with the native Anthropic adapter against a scripted local Messages server (SSE for steps, JSON for pings) and check the ping's tools/system/thinking/messages against the step request, the placeholder tool_result, that the placeholder never enters the conversation, the budget arithmetic, the 10-ping cap, the failed-ping stop, the no-cache no-op and the exceptional-exit accounting.
492 impacted core and Sorcar tests pass; uv run check --full passes (ruff, mypy).
Read-only review by gpt-5.6-sol found six demonstrated issues (no-op when caching is disabled, lost ping accounting on exceptional exit, unbounded join on an in-flight ping, replay gate one step ahead of the agent, a ratio computed before its baseline, an int() floor); all fixed.

6. Follow-ups

Re-run compare_kpis.py --hours 72 after the deployment has been live for a few days (the cron job does this on 2026-09-23) and compare the two cache-miss rows.
Make the gate price-aware: break-even steps = (write − read) / read × context / drop, against the steps left before hand-off, using the model's cache prices in MODEL_INFO. On 0.1× models this compacts about as now; on 0.025× models it compacts only to avoid a hand-off.
Move the Anthropic path to server-side tool-result clearing (section 4).