SN pipeline model selection + reviewer-resilience (2026-07)

Research doc — informs model-selection-and-global-refine. All engineering landed in imas-codex; SHAs are cited inline. Benchmark numbers are from the 2026-07-19/20 all-seat run.

§1 — Outcome

The documentation-simplification campaign stalled when its reviewer models returned empty/rate-limited responses under load. The root cause was upstream OpenRouter provider capacity, not a prompt bug: the blind-pair reviewers qwen3.7-max and minimax-m3 are served by thin-capacity providers that HTTP-429 under the review pool's 64-way concurrency. Resolving it produced three things:

A resilient, better-discriminating reviewer blind pair — claude-sonnet-5 + grok-4.5 — chosen on a new labelled discrimination benchmark plus concurrency vetting.
Pipeline-resilience fixes so a provider outage degrades gracefully and is observable, not a silent multi-hour stall: a global AIMD rate governor with bounded recovery, a quorum-integrity guard, and an LLM-activity heartbeat.
An all-seat model benchmark exercising the real production prompts, with a reasoning-effort sweep — reusable as both a model discriminator and a debugging tool.

§2 — The incident and its root cause

Docs-campaign rotation 1 (stratified 200 names) was halted by the orchestrator ~6 min in when 2 of the 3 RD-quorum reviewer models entered an upstream empty-response outage and 67% of reviews degraded to single-reviewer — violating the "review quorum unchanged" churn-mitigation invariant.

Reproduction settled the cause. Single high-effort review calls to qwen3.7-max and minimax-m3 succeed (they are still listed and reasoning-capable). Under 16-way concurrency they fail:

Model (16-way concurrency, effort=high)	Result
`qwen3.7-max`	13/16 OK, 3 empty ("returned empty/no reviews")
`minimax-m3`	mostly HTTP 429 "temporarily rate-limited upstream"
`grok-4.5`, `gemini-3.5-flash`, `gemini-3.1-pro`, `gpt-5.6-terra`, `deepseek-v4-pro`, `claude-sonnet-5`	16/16 OK

So the discriminating axis for a blind-pair seat is provider concurrency capacity first, then quality. (Muse Spark 1.1 is excluded — 403 US-geo-restricted from this environment.)

§3 — Pipeline resilience (imas-codex)

Fix	What it does	SHA
Global AIMD rate governor	Process-global ceiling on concurrent LLM calls; multiplicative pullback on 429, bounded time-based recovery (~9 s from floor, not the tens-of-minutes completion-gated crawl), `min_ceiling` 8. No-op under healthy load.	`8b59d8a3`, `e26c387b`
Quorum-integrity guard	A name intended for a ≥2-model quorum can no longer be accepted on a single review when a secondary is throttled — it defers (releases to `drafted`) with a loud per-run counter. Legitimate single-model/derived-parent reviews unaffected.	`39ea6a9f`
LLM-activity heartbeat	At the shared call chokepoint: logs in-flight/completed/spend every 15 s and WARNs on a >120 s stall — turns a black-box run into an observable one.	`e26c387b`
Bench parallelization + fail-fast	The per-item review / refine / docs-gen / classifier / compose bench calls were serial; a single hung provider call (×5-retry×120 s) ground a role for ~46 min. Now bounded-concurrent with `max_retries=2` — a hang fails fast; flushed progress heartbeat for liveness.	`74e0725b`, `7a8d606b`, `d1e9375f`, `46c27da4`

§4 — Benchmark methodology

All seat benches run through sn bench --role <seat> and exercise the production prompts and graph context (so a bad result is a real production signal and the harness doubles as a debugging tool). Model slates are pyproject-driven ([sn-benchmark].candidate-models; each seat's incumbent read live from its own [sn-*] config), never hardcoded (7087430d). Enums — including kind — derive from the ISN Kind authority and are injected into the prompts (940ec73f).

Reviewer discrimination (new): labelled corpus of accepted (good) items vs deterministic seeded-defect twins (bad) — banned-prose / vacuous / unit-contradiction for docs; name mismatch / bare-base / non-canonical for names. Metric: AUC + calibration (good-pass, bad-recall). (1bfeb2b1, names-corpus fix 9c008c6d.)
Concurrency resilience (new): N parallel production review calls, tally success vs 429/empty.
Reasoning-effort sweep (new): one row per (model, effort) so overthinking is measured, not assumed.
Existing seats: refine-from-critique (held-out judge), review-breaker independence, docs-generation rubric + banned-prose, domain-classifier gold set, and the compose (name-generation) bake-off.

Effort is a real, seat-and-model-dependent lever

The clearest cross-cutting finding: reasoning effort is not "more is better." For judgment seats several models overthink at high effort; refine benefits from it. Measure per seat.

Review-docs discrimination AUC, low→high effort. Open dot = low, filled = high; red drop = the model overthinks at high effort (grok, gemini-pro, deepseek, gemini-flash), green rise = improves (sonnet-5, terra). The pair runs best at low effort where both chosen reviewers are near their peak and cheaper.

§5 — Results by seat

Reviewer blind pair (the driving decision)

Discrimination AUC (labelled good/bad), best effort per model, all concurrency-resilient unless noted:

Model	docs AUC	names AUC	notes
claude-sonnet-5 (Anthropic)	0.910	0.906	best docs discriminator; improves with effort
grok-4.5 (xAI)	0.861	0.920	strong both axes, well-calibrated, cheap ($2/$6)
deepseek-v4-pro	0.860	0.985	best names, but ~8× slower provider → poor for a concurrency-heavy pool
gemini-3.1-pro	0.854	0.889	overthinks at high
gpt-5.6-terra	0.854	0.924	over-rejects docs (good-pass 0.08) — correlates with the OpenAI breaker
gemini-3.5-flash	0.837	0.882	cheapest

Decision — blind pair = claude-sonnet-5 + grok-4.5 (imas-codex b5b5eccb), both axes of both [sn-review.names.profiles.default] and [sn-review.docs]; breakers held (luna names / gpt-5.5 docs). Vendor-diverse (Anthropic + xAI), both hold 16/16 under load. grok kept deliberately — its figures beat the non-grok alternative on both axes and it is cheaper, so it is not a marginal call.

Other seats — all HOLD their incumbent

Seat	Decision	Evidence
compose (name-gen)	HOLD `deepseek-v4-flash` (local, free)	100% grammar-valid, $0, desc 0.942 on the isolated real-vLLM run; paid models (grok 0.968, kimi-k3 0.966) only marginally better — not worth it at compose volume. Claude family fails compose outright (0% grammar-valid).
refine	HOLD `gpt-5.5`	defect-resolution 0.865 @high, valid-rate 0.93; sonnet-5 fails the seat (provider hangs); no candidate edge. Refine benefits from high effort.
breaker-names	HOLD `gpt-5.6-luna`	flip-quality 0.80 @low = best
breaker-docs	HOLD `gpt-5.5`	weak signal (small disagreement counts); no better option
docs generation	HOLD `gpt-5.6-luna`	rubric 0.847, zero banned prose, cheaper than terra's +0.008
classifier	`gpt-5.5`	Seated on gpt-5.5 (0.833) as the robust OpenAI-tier pick over terra (0.818). gemini-3.5-flash scores higher (0.876) and cheaper — recorded as the data-optimal alternative; OpenAI robustness chosen by the lead.

Attribution fix — `[sn-classifier]`

The physics-domain classifier (whose labels SN names inherit) had been borrowing the generic [language] model, so its config and model choice were not attributable to the SN project. It now has a dedicated [sn-classifier] seat (5580b707). Convention recorded in the imas-codex AGENTS.md: every model the SN pipeline drives lives under an sn-* seat, never a generic section.

§6 — Test infrastructure (made green)

Two pre-existing blockers that broke the imas-codex SN test suite were fixed (not weakened): a GraphClient() built at pytest-collection time that hung on an SSH tunnel → bounded on a daemon thread (be11a3fc); and five content/logic failures fixed at root cause (public-name sn-abbreviation rename, an atomic run-id clear, the kind enum verified against the live ISN Kind enum, an ISN-inconsistent compose-prompt example replaced, and a refined golden). Full suite: exit 0.

§7 — Follow-ups

Re-run docs-campaign rotation 1 on the new resilient blind pair (the original driver in model-selection-and-global-refine §5).
Optional: a names-axis reasoning-effort study before lowering the shared [sn-review].reasoning-effort from high (the docs data favours low for the pair; names not yet studied at that granularity).
Optional: reconsider the classifier as gemini-3.5-flash if the accuracy/cost gap (0.876 & ~5× cheaper vs gpt-5.5 0.833) outweighs the OpenAI-robustness preference.