# CI Reliability Plan — 2026-05-05

**Status:** Working plan. Locked 2026-05-05 after Ben surfaced 6 red CI runs across PRs #158 / #162 / #164 / `axiom-ext-openmc` in his inbox over a single afternoon.

**Owner:** Ben (R/A) · **Drafter:** Claude
**Companion memory:** `feedback_default_green_ci_culture.md`

---

## Five Whys — root-cause analysis

**Surface event:** 6 red CI runs hit Ben's inbox over a single afternoon (2026-05-05).

**Why 1.** Why did CI fail on every push today?
→ Because `ci.yml` and adjacent workflow files reference action versions that don't resolve at runtime: `actions/upload-artifact@v7`, `actions/cache@v5`, `actions/setup-python@v6`, `actions/download-artifact@v8`, `actions/github-script@v8`. The latest stable majors are v4 / v4 / v5 / v4 / v7 respectively. Jobs fail at runner-setup time before any step runs, so no signal reaches the developer beyond a generic red ❌.

**Why 2.** Why does the workflow reference non-existent action versions?
→ Because somewhere in the recent past, the workflow YAML was edited and the action pins were bumped — by a process or a session — without verification that the versions exist. "Newer must be better" is a plausible heuristic for library deps because pip would error visibly on a non-existent version; for GitHub Actions, the resolver is silent at PR time. The bad pin lands without complaint.

**Why 3.** Why didn't anyone catch the bad pin before merge?
→ Because workflow YAML changes don't get the same review discipline as test or source edits. The diff is small, the syntax looks identical to working YAML, and the human reviewer treats action versions like configuration strings rather than runtime-resolved code. The PR's own CI run could have caught it — but if the PR introducing the bad pin merged on a runner cached against an earlier (working) version, the failure surfaces *only* on subsequent runs that re-resolve.

**Why 4.** Why is workflow YAML treated as configuration rather than code?
→ Because there's no test surface for it in the codebase. You can't `pytest .github/workflows/`. Validation is "push, wait for CI, see what happens." There's no `tests/workflows/test_action_pins_resolve.py` that fetches each `uses: actions/X@vN` and checks it exists. Tools like `act` (run workflows locally) exist but aren't part of the standard pre-push surface. As a result, workflow edits are systematically diagnosed *after* they affect real PRs, never before.

**Why 5.** Why is there no test surface for the workflow itself?
→ Because the deeper assumption baked into the development model is *"CI is the test surface; nothing tests CI."* CI exists to test the rest of the codebase; the rest of the codebase doesn't test CI. This is a circular trust — CI is trusted to surface failures, but no upstream gate ensures CI itself is functional. When CI breaks, the only feedback channel is the developer's email inbox 30 minutes later.

### Root cause (the structural answer)

**CI pipeline configuration is treated as fire-and-forget infrastructure rather than as code that fails like any other code.** This shows up in three concrete places:

1. **No test surface for workflow YAML.** Action pins, env vars, runner images, secrets references — all unchecked until a real run.
2. **No drift detection on CI health.** TIDY surveils branch drift but ignores CI conclusion patterns. A workflow that's been broken for hours generates emails, not alerts that compound into a single signal.
3. **No "default-green" cultural commitment.** Without it, individual CI failures feel like special events rather than a stream worth aggregating. Without aggregation, you can't see "all these PRs are failing the same way" — and so you re-fix the same problem N times instead of fixing it once at the source.

### The fix is simultaneously easy and hard

**Easy:** pin action versions to known-good (Phase A1 — five lines edited; ships in this PR).

**Hard:** establish that workflow changes ARE code, ARE testable, AND that "default-green" is the cultural norm. That requires:
- A `test_workflow_action_versions` test that fetches and validates every `uses:` line at PR time
- TIDY's `ci-health` surface (Phase C) so failures aggregate into actionable signal
- Pre-push reactivation (Phase B) so workflow changes get caught before they affect anyone else
- Branch protection (Phase D) requiring CI green before merge, so a broken workflow can't propagate to main
- Memory: `feedback_default_green_ci_culture` to make the cultural commitment durable

The Five Whys point at all five — none of them alone is the answer.

---

## Diagnosis — what actually went wrong today

| Symptom | Real cause | Class |
|---|---|---|
| Three PRs (#158/#162/#164) failing at runner-setup time (`steps: []` in API output) | Workflow references non-existent action versions: `actions/upload-artifact@v7`, `actions/cache@v5`, `actions/setup-python@v6`. Latest stable: v4 / v4 / v5. | **Infrastructure** — workflow YAML is broken; no code change can land until fixed |
| `axiom-ext-openmc` initial CI runs failed | Same action version issue (copied from axiom-os ci.yml as a template) | **Infrastructure** — inherited pin error |
| PR #158 had real Unit Tests (3.11) failure earlier | `test_axiom_cli_dispatches_tasks_verb` — landed test asserting wiring that hadn't been added | **Real bug** — caught by test, missed in pre-push because `--no-verify` was in use |
| Quiz_id flake on 3.13 (resolved earlier today via PR #147) | `secrets.token_urlsafe(9)` produces dash-leading IDs ~1.6% of generations | **Real flake, root-caused** — class of bug fixed, not just retried-around |
| Memory perf-flaky test (`test_repeated_writes_to_same_edge_stay_bounded`) | Wall-clock-dependent assertion, environment-sensitive | **Known flake** — not yet root-caused |

The compound effect: every push today triggered CI; CI was broken at the workflow level *regardless of what was pushed*; emails accumulated. TIDY had no visibility because TIDY's drift dashboard surveils branch-level state (susp commits, age, behind-count), not CI health.

---

## Principles (per `feedback_default_green_ci_culture.md`)

1. A green CI run is the expected state. Failure is the rare anomaly worth investigating, not background noise to retry through.
2. Pre-push validation is the primary gate. CI is verification of what the developer already saw work, not discovery of what nobody checked.
3. `--no-verify` is exceptional, with a documented reason in the commit message.
4. Failures must surface a root cause; "retry till green" is a future-incident in waiting.
5. CI pipeline health (workflow YAML, action pins, runner config) is itself a build artifact and gets the same review discipline as test code.

---

## Plan — five phases

### Phase A — Stop-the-bleed (today)

| Item | Action |
|---|---|
| **A1.** Pin action versions to known-good | `actions/checkout@v4`, `actions/setup-python@v5`, `actions/cache@v4`, `actions/upload-artifact@v4` across `ci.yml`, `publish.yml`, `memory-benchmarks.yml` in axiom-os, and `ci.yml`/`publish.yml` in axiom-ext-openmc |
| **A2.** Re-run failed CI on PRs #158/#162/#164 | After A1 lands |
| **A3.** Fix any genuine code-level issues that surface once infrastructure passes | One by one |

This is a small, focused PR — likely 4–6 file edits. Lands first, unblocks everything else.

### Phase B — Pre-push reactivation (today/tomorrow)

R2-widening + the `test_tidy_discover` fix landed earlier today; the chicken-and-egg that justified `--no-verify` is over. Pre-push hooks are now safe to run by default.

| Item | Action |
|---|---|
| **B1.** Update CLAUDE.md (axiom-os) | Default behavior: run pre-push hook. Document the *exceptional* cases requiring `--no-verify` (documented chicken-and-egg only) |
| **B2.** Add lightweight pre-commit hook | Just lint (ruff, ~30s) — runs on every commit, not just push. Catches 80% of today's failures (most failures were lint-or-manifest level). |
| **B3.** Tier the pre-push surface | Fast tier (lint + manifest tests + changed-file tests) always; slow tier (full unit sweep) opt-in via `git push --verify-full` or env var |
| **B4.** Document failure-investigation procedure | Open the log, classify (real / flake / infra / env), then fix |

### Phase C — TIDY `ci-health` surface

This is M-O's scope expansion — drift dashboard is the first hygiene surface, CI health is the second.

| Item | Action |
|---|---|
| **C1.** New module: `axiom.extensions.builtins.hygiene.ci_health` | Read GitHub Actions API; aggregate per-branch CI history (last N runs, conclusion distribution, flake patterns) |
| **C2.** New CLI: `axi tidy ci-health` | Per-repo + cross-workspace surface; dashboard-style output similar to `axi tidy drift` |
| **C3.** Detection categories | Workflow-level vs test-level vs lint-level vs flake; chronic-red flag at 3+ consecutive failures |
| **C4.** RACI integration | Per ADR-043, M-O surfaces findings via the RACI ledger; pre-approval skips for routine flake re-runs |
| **C5.** Daily digest mode | If user opted into `SCHEDULED` autonomy state (ADR-043 D1), batch CI-health findings into a daily summary |

### Phase D — CI quality gates (this week)

| Item | Action |
|---|---|
| **D1.** Branch protection | GitHub branch protection on main: CI green required before merge |
| **D2.** Required checks list | Lint + Unit Tests (3.11/3.12/3.13) + Install-Path Tests + AEOS-manifest test as required |
| **D3.** Auto-bisect on main regression | When a green-on-PR merge turns main red, bisect the last N commits to surface guilty SHA |

### Phase E — Build-quality metrics (post-Prague)

| Item | Action |
|---|---|
| **E1.** Rolling green-rate on main | Target 95%+; surface in TIDY |
| **E2.** Mean-time-to-green-after-failure | Target < 1 hour; surface in TIDY |
| **E3.** Per-test flake rate | Target < 0.1%; auto-quarantine over a threshold with a tracking issue |
| **E4.** Workflow-change review burden | Workflow YAML edits get an extra reviewer/approval; preserved as a CODEOWNERS line |

---

## What changes about my (Claude's) behavior

Per `feedback_default_green_ci_culture.md`:

1. **Default to running pre-push.** Today I `--no-verify`'d every push because of the R2-pollution concern. That concern is gone after the test_tidy_discover fix. Going forward: `git push` (not `git push --no-verify`). Document the exceptional case if I do skip.
2. **Before saying "CI failed, will retry":** open the log, classify the failure (real / flake / infra / env), and respond accordingly. Today I should have noticed the `steps: []` pattern after run #1 and immediately diagnosed the workflow as broken.
3. **Workflow YAML changes are code review.** Lint pins, action versions, env vars all get the same scrutiny as test/source edits.
4. **Failures aren't normal.** If I see 3 reds in a row on different PRs, that's a systemic issue requiring diagnosis, not 3 separate retries.

---

## Sequencing

- **Right now:** A1 — fix the action version pins (small PR; gets all the other PRs unstuck).
- **Tonight/tomorrow:** B1–B4 — reactivate pre-push, document exceptional cases.
- **This week (post-Wed-class):** C1–C5 — TIDY CI-health surface.
- **This week:** D1–D3 — branch protection.
- **Post-Prague:** E1–E4 — metrics + auto-quarantine.

---

## What this plan does NOT solve

- **Local-vs-CI environment differences** beyond the action-version issue. The pre-push hook claims CI-equivalence; today's failures showed gaps. A separate audit of local-vs-CI divergence is a follow-up after Phase B.
- **Cross-repo CI policy.** axiom-os, axiom-ext-openmc, NeutronOS, dendra each have their own CI. This plan starts with axiom-os; other repos inherit the pattern as they regenerate workflows.
- **The cultural shift.** "Green is normal" requires consistent reinforcement, not just tooling. The memory file is the durable artifact; the tooling is the supporting infrastructure.
