# CI Hygiene Plan — Keeping Axiom Green

**Status:** Captured (2026-04-28).
**Driver:** User-driven question: *"diagnose the history of Axiom CI failures and let's come up with a plan to make sure these become the exception, not the norm."*
**Related:** `.github/workflows/ci.yml`, `lessons-rascal-slm-dependency-2026-04-28.md`.

---

## 1. The diagnosis

### 1.1 Numbers

- **Last green CI on `main`:** 2026-04-15 14:01 UTC.
- **Runs since then:** 81 (over ~13 days as of 2026-04-28).
- **Successful runs in that window:** 0.
- **Failure breakdown:** 66 failure / 15 cancelled / 0 success.

### 1.2 Failure causes — in chronological strata

**Apr 15 14:37 → Apr 24 (~9 days, ~25 runs) — real code failures:**
1. `test_cli.py::test_add_ssh_creates_entry` + `test_add_ssh_json` (federation extension) — first failures the day after green.
2. Then `test_standard.py` collection errors — pytest's `prepend` import mode collided on the duplicate filename across 24 extensions (fixed in this commit window: `0b98e61`).
3. Then EC term test corpus drift — keyword list narrowed (good) but test corpus wasn't updated (fixed in `0b98e61`).
4. Plus accumulated lint regressions (`ruff` errors fixed in `0b98e61`).

Each push compounded the problem because **nobody was watching CI vs. shipping locally**. Every red run added another diff that needed disentangling later.

**Apr 25 → Apr 28 (~3 days, ~56 runs) — billing block:**
GitHub Actions billing exhausted; jobs literally don't start (`The job was not started because recent account payments have failed or your spending limit needs to be increased`). The earlier real failures became invisible because CI couldn't even attempt them.

### 1.3 Today's HEAD locally

- `pytest -n auto` → 5770 passed, 0 failures, 185 skipped.
- `ruff check src/ --select E,F,W --ignore E501` → All checks passed.
- `python -m build` → wheel built clean.

The real-failure backlog is closed. Once billing is unblocked, CI should go green on next push.

### 1.4 The pattern under the failures

Three structural conditions allowed this to compound:

**Condition A — pre-push gate doesn't match CI exactly.**
The pre-push hook runs `pytest -n auto` but local Python is 3.14; CI runs the matrix 3.11/3.12/3.13. Some failures (especially those involving stdlib differences) are version-specific. Pre-push won't catch them.

**Condition B — no visibility into "is `main` green?" before pushing more.**
Operators push locally-clean changes onto a CI that's been red for days. Each new push compounds the diagnostic burden. The CI failure-issue auto-creator (`notify-failure` in ci.yml) creates an issue when a run fails — but during the billing block, *that job also doesn't run*, so issues don't pile up either. Net: CI is invisibly red.

**Condition C — failure issues don't track resolution.**
Even when the auto-issue creator does work, there's no mechanism that says "this issue must be closed before the next merge." Issues become noise.

## 2. The plan

Five layers, ordered from "no infrastructure" to "would benefit from automation."

### 2.1 Layer 0 — Discipline (free, immediately enforceable)

**Rule 1: never push onto red main.** Before every push, check `gh run list --branch main --limit 1 --json conclusion`. If the last main run was a failure, **fix or revert is the next commit**, not a new feature.

**Rule 2: a failed CI run is a P0.** Drop everything; next commit fixes or reverts. This is the discipline that closes the compounding loop.

**Rule 3: commits are signed off as "I ran the test suite locally."** Every commit body contains an explicit confirmation (one line): `Local: pytest -n auto = 5770 passed, ruff clean`. Today's commits have this in some form; codify it.

These three rules are free. They're 80% of the fix.

### 2.2 Layer 1 — Pre-push hook hardening

Today's pre-push hook runs `pytest -n auto` (verified during the v0.11.0 push: `✓ pre-push: all tests passed (3722)`). Three additions:

1. **Run `ruff check src/ --select E,F,W --ignore E501` before pytest.** Lint is a hard CI gate; catching it in pre-push removes a class of failures.
2. **Check `gh run list --branch main --limit 1 --json conclusion`.** Warn if last main run was a failure and require `--no-verify` to bypass. Implements Rule 1 mechanically.
3. **Run `axi doctor`** (once shipped — Item 3). Catches dependency-gap issues that pytest can't see (missing ollama models, empty RAG, etc.).

Implementation: extend `.git/hooks/pre-push` (or wherever the hook lives — find via `ls -la .git/hooks/`). Keep the existing pytest invocation; add the three checks ahead of it.

### 2.3 Layer 2 — `axi ci status` command

A new axi subcommand that pulls recent CI runs + summarizes:

```
$ axi ci status
Axiom CI — main branch
─────────────────────────────────────────────
Last green:    2026-04-28 22:14:01 UTC (3h ago)
Last run:      2026-04-28 22:14:01 UTC ✓ success
Recent (last 10):
  ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Status:        green — safe to push
```

Or when red:

```
$ axi ci status
Axiom CI — main branch
─────────────────────────────────────────────
Last green:    2026-04-15 14:01:16 UTC (13 days ago)
Last run:      2026-04-28 19:06:09 UTC ✗ failure
Recent (last 10):
  ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
Status:        RED — fix or revert before pushing
Last failure:  Lint job — billing block
              View: gh run view 25072280239
```

Reads via `gh run list` (already authenticated locally). Use it before any push; pre-push hook calls it (Layer 1.2).

Cost: ~half-day. Lands in 0.11.x or 0.12.

### 2.4 Layer 3 — CI workflow improvements

Five changes to `.github/workflows/ci.yml`:

1. **Add a `pre-flight-check` job that runs first + cheap.** It runs `ruff check` + a fast smoke subset of pytest. If pre-flight fails, the rest of the matrix is cancelled — saves minutes when there's a trivial failure. Currently, all jobs run in parallel; a lint failure doesn't stop the test matrix.

2. **Failure-issue auto-creator: dedupe by issue body, not just SHA.** The current `notify-failure` job dedupes on commit SHA — so multiple workflow runs of the same SHA don't pile up. But if the *same issue type* (e.g., billing block) recurs across many SHAs, each gets its own issue. Add a labeling system: issues tagged `ci-failure: billing` get auto-closed when a green run lands.

3. **Failure-issue body includes "what to do next."** Today's body is generic ("Investigate the failed jobs"). Better: include the failed-jobs list, the most recent test failure if any, and a one-line `gh run view` command. Ops trying to triage spend zero time orienting.

4. **Add a daily scheduled CI run.** A nightly cron triggers CI on `main` even when nobody pushes. Catches drift caused by external dependencies (PyPI yanks, GitHub action updates, etc.) before someone hits them at push time.

5. **Add a `ci-status` badge to README.** Public visibility — hard to ignore a red badge.

### 2.5 Layer 4 — Failure-class reporting (post-Prague)

Once billing is consistently green, monthly reporting of failure classes:

- "Real test failures": N — root-caused to which test, fixed in which commit.
- "Lint failures": N.
- "Flaky tests": N — list which.
- "Infrastructure failures" (billing, runner outages): N.

Lands in a `docs/working/ci-trend.md` doc parallel to `memory-benchmarks-trend.md`. Quarterly review pattern matches the maturity-benchmarks discipline.

## 3. Specific actions for the next push window

When billing is unblocked + we push v0.11.0:

| Action | Owner | When |
|---|---|---|
| Verify CI runs at least one job to completion | User | After billing fix, before tag |
| Push v0.11.0 to main; watch CI to green | Ben + me | Tag-blocking |
| Post-push: implement Layer 0 discipline rules in `CONTRIBUTING.md` (or CLAUDE.md) | Me | Same window, separate commit |
| Implement Layer 1 pre-push additions | Me | 0.11.x next-week window |
| Implement Layer 2 `axi ci status` | Me | 0.11.x or 0.12 |
| Implement Layer 3 ci.yml improvements | Me | 0.12 |
| Set up Layer 4 trend reporting | Me + automation | Post-Prague |

## 4. The bottom line

**13 days of red CI is a discipline failure, not a tooling failure.** The tooling did its job — `notify-failure` runs, `gh run list` reports honestly. What was missing was the human discipline that says "red CI on main is a P0 to fix or revert, not background noise to push past."

Three things make CI green sustainably:
1. **Rule 1: never push onto red.** Free; immediate.
2. **Pre-push hook hardening** (lint + ci-status check). Cheap; one-time.
3. **`axi doctor`** as the local-equivalent-to-CI gate. Catches dependency-gap issues no test can see.

Layers 3 + 4 are nice-to-haves that automate the discipline once it's a habit.
