§5 — Documentation campaign: 25-name stratified pilot (landed 2026-07-17)
Evergreen: model-selection-and-global-refine §5. The pilot is the first live use of the §4 campaign engine. Scaled batches are gated on lead approval of the manifest and the measured cost below.
Quick status
| Item | Status | Evidence |
|---|---|---|
| Revalidation sweep (pre-campaign) | ✓ shipped | 436 re-stamped; derived quarantines 62 → 10 (4 unit defects + 6 genuine audit findings); accepted-stage quarantines 125 |
Stratified pilot selection (--campaign-pilot N) | ✓ shipped | imas-codex 0882e417 — domain round-robin, defect-class rotation, deterministic seed; 7 tests |
| Full-campaign manifest (dry-run) | ✓ generated | 2,288 names → 23 batches of 100; per-predicate audit 2,260 / typical-values 1,257 / estimator 779 / padding 676 / quarantined 125 |
| 25-name pilot (live) | ✓ converged | 1 batch: 24/25 docs accepted (96% ≥ 90% gate), 0 banned-prose reintroductions, 0 name drift |
| Campaign cost telemetry fix | ✓ shipped | imas-codex 763a5a14 — drain returns pool-session spend; ceiling now accounts real dollars (pilot had logged $0.00) |
| Banned-prose vocab relocation | ✓ shipped | imas-codex a469501d — neutral prose_policy module; production no longer imports from the benchmark module |
| Scaled batches (23 × 100) | ⏸ awaiting lead approval | decision inputs below |
Pilot composition
Deterministic stratified sample (seed 0) drawn from the 2,288-name selection: all 18 physics domains represented (2 names from the larger domains, 1 from the smaller), defect classes rotated within each domain. Predicate mix: audit 21, typical-values 13, estimator-recipe 9, procedural-padding 5, quarantined 3. The same seed reproduces the identical member set in the live run — the manifest the lead reviews is exactly what runs.
Pilot outcome
| Gate axis | Threshold | Measured | Verdict |
|---|---|---|---|
| Docs acceptance | ≥ 90% | 24/25 = 96% | PASS |
| Banned prose reintroduced | 0 | 0 (grep-audit on all 25 refreshed docs) | PASS |
| Name-identity drift | 0 | 0 (all 25 still accepted under the same id) | PASS |
| Provenance | full | 25/25 campaign_docs_refine change events; 23/25 DocsRevision snapshots (2 members had no prior documentation — nothing to snapshot, benign) | PASS |
The single unaccepted name, minimum_magnetic_field, sits at docs_stage='drafted' — the $10 pilot budget cap stopped the pools before its final review cycle. Any subsequent docs drain (including the first scaled batch) completes it; no defect.
Refinement quality (samples the lead can spot-check)
Three before/after pairs from the pilot (full text in the graph; prior text preserved in DocsRevisions):
alfven_time— recipe prose ("Using the magnetic-axis major radius … the Alfvén time is defined by") replaced by a direct definition with every symbol defined; typical-value padding gone.electron_temperature_at_plasma_boundary— boundary definition kept, coordinate frame now explicit, procedural padding removed.oxygen_source_rate— equivalent-electron convention stated normatively; estimator framing dropped.
Scale-up decision inputs (for the lead)
| Input | Value |
|---|---|
| Selection | 2,288 accepted names (post-sweep), 23 batches of 100 |
| Measured unit cost | $0.42/name as run, but 53% of spend was the refine claim-race livelock (see corrected figure) — healthy-path cost ≈ $0.20/name (seats: docs-gen luna, refine gpt-5.5, review quorum unchanged) |
| Full-campaign projection | as-run rate ≈ $850–960; post-livelock-fix ≈ $400–500 for the full spec, ≈ $300–380 for the docs-fixable spec — the fix gates the launch |
| Suggested caps if approved | per-batch --campaign-batch-cost-cap 50, campaign --campaign-cost-ceiling 1000, gate 0.90 unchanged |
| Cheaper alternatives | (a) narrow the spec to prose only (1,384 names ≈ $520–580), deferring pure-audit fixes; (b) run domain-by-domain tranches with review between tranches; (c) accept as-is — quality evidence at 96%/0/0 is strong |
| Command (full, as piloted) | imas-codex sn run --source dd --campaign all --include-accepted --campaign-batch-cost-cap 50 --campaign-cost-ceiling 1000 |
Commits
| Repo | SHA | What |
|---|---|---|
| imas-codex | 0882e417 | stratified campaign pilot selection (physics-domain round-robin, defect-class rotation, manifest per-domain + pilot marker) |
| imas-codex | 763a5a14 | campaign batches account real pool-session spend; drift note moved after embed preflight; stale clear-gate test patches removed |
| imas-codex | a469501d | banned-prose policy vocabulary moved to neutral prose_policy module |