# Lessons — Rascal SLM Dependency Gap (2026-04-28)

**Status:** Captured (2026-04-28). Improvements 1–3 in flight (this commit window).
**Driver:** User-driven head-to-head evaluation: NeutronOS Rascal (Qwen + RAG) vs Claude Code (Opus 4.7). User asked: *"how did this happen, and how can we make sure any future Axiom node configuring an LLM won't repeat the mistake?"*
**Pattern this captures:** L1 — Operational pattern observed once; promote to L2 fact ("implicit dependencies are bugs") if the pattern recurs in a second incident.
**Related:** `spec-memory-maturity.md`, `working/memory-competitive-analysis.md`, ADR-035, `prd-agents.md`.

---

## 1. The incident

**What the user observed:** asking Rascal's NeutronOS RAG endpoint a routine reactor-physics question (e.g., about ANSI/ANS-15.1 reactivity insertion limits) produced one of:
- An EC-block that refused to even attempt the answer.
- An invented answer with hallucinated specific numbers not present in the corpus.

**The chain that produced it:**

1. Operator (or installer) configured `routing.ollama_model = "llama3.2:1b"` in Rascal's settings.
2. The model file (~1.3 GB) was never pulled into Ollama on Rascal — `ollama pull llama3.2:1b` was an implicit prerequisite, not a declared one.
3. First request to `/api/generate?model=llama3.2:1b` returned a model-not-found error.
4. `OllamaClassifier.classify()` swallowed the exception (`except Exception: return None`) — silent.
5. Router treated `None` as "Stage-2 unavailable, fall through to fallback."
6. Strict-default sensitivity made fallback = EC.
7. Operator saw over-blocking. Root cause invisible without reading audit log internals.

**At no step did the system raise its hand.** Every component handled its local failure correctly in isolation; nobody owned "this node isn't fully provisioned."

## 2. Root cause — three structural gaps

### 2.1 Gap A — Models are implicit dependencies, not declared ones

Axiom treats LLM *endpoints* as first-class connections (manifest declarations, `axi connect <preset>` framework, probing). But the *models* those endpoints serve are an opaque side-quest. We say "set `routing.ollama_model = llama3.2:1b`" without any contract that the model exists or any helper that ensures it.

Compare: connection declarations have `requires_vpn`, `probe_path`, etc. Model declarations have nothing equivalent.

### 2.2 Gap B — Health checked lazily, not at boot

The first `OllamaClassifier.classify()` call discovers the model is missing — but only as a return value of `None`. `axi config --status` showed Ollama *server* up but didn't probe the configured *model*. Operator's status view said "OK" while Stage-2 was effectively disabled.

### 2.3 Gap C — Silent fallback masks dependency failure

`OllamaClassifier.classify()` catches *all* exceptions and returns `None`. Router treats `None` indistinguishably from "Ollama returned uncertain in balanced mode." A missing-model error and a benign uncertain-on-purpose look identical in the audit log. Worst kind of silent failure: locally correct, globally invisible.

## 3. What changes — six improvements, ranked

| # | Improvement | Closes | Cost | Lands |
|---|---|---|---|---|
| 1 | Typed `ClassifierFailure` modes + audit events | C | small | 0.11.x (this window) |
| 2 | Routing classifier health in `axi config --status` | B | small | 0.11.x (this window) |
| 3 | `axi doctor` proactive prerequisite checklist | A, B, C | medium | 0.11.x (this window) |
| 4 | `[[extension.requires.models]]` manifest schema | A | medium | 0.12 (post-Prague) |
| 5 | `axi connect` preset declares required models | A | medium | 0.12 (post-Prague) |
| 6 | Opt-in auto-pull on first use | A | larger | 0.13 or later |

Items 1–3 close the *operator-visible* gap and ship in this 0.11.x window. Items 4–5 close the *architectural* gap (manifest schema bump → AEOS update post-Prague).

## 4. The general principle

The Rascal SLM oversight is not an isolated bug — it's a **category**. Three improvements 1–3 close this specific instance; the architectural lesson is bigger:

> **Anything an extension *needs* to function is a declared dependency, not an implicit assumption.** Connections, models, corpora, env vars, files — all get the same treatment: declared in manifest, validated by `axi doctor`, ensured by install/connect commands, audited when missing.

This is the discipline that catches the next 10 silent-dependency failures before they ship.

### Manifest schema sketch (post-Prague)

```toml
[extension]
name = "..."

[[extension.requires.models]]
provider = "ollama"
name = "llama3.2:1b"
purpose = "stage-2 routing classifier"
size_gb = 1.3
ensure = "auto"   # auto | warn | manual

[[extension.requires.corpus]]
name = "regulatory-baseline"
min_chunks = 100
purpose = "stage-3 RAG grounding"
ensure = "warn"

[[extension.requires.env]]
name = "ANTHROPIC_API_KEY"
purpose = "public-tier LLM auth"
ensure = "warn"
```

`axi ext install <ext>` reads these and either auto-fulfills (`ensure="auto"`), warns (`ensure="warn"`), or notes (`ensure="manual"`).

`axi doctor` walks installed extensions + checks each declared dependency.

`axi connect <preset>` declares preset-level required models (extends Item 8 / connect.py).

## 5. Promotion to L2 fact

Per `spec-memory-maturity.md` §2: an L1 pattern promotes to L2 fact when:
- Frequency threshold met (this incident + N more), OR
- Cross-scope evidence (multiple cohorts hit the same issue), OR
- Human curator approval (Ben can promote on judgment).

**Recommend:** promote to L2 fact ("implicit dependencies are bugs") on Ben's judgment. The remediation pattern (declare → validate → ensure → audit) becomes the rule any new extension authoring is held to. Codify in `spec-aeos-0.1.md` §4 amendment when Item 4 ships.

## 6. Future incident triage

When the next "node configured but broken" report comes in:

1. Check whether the failure mode is explicitly classified by the typed `ClassifierFailure` enum (or its successors). If it's still hiding behind `None` or a generic Exception swallow, that's a new instance of Gap C.
2. Run `axi doctor` on the affected node. If the failure isn't surfaced, that's a missing check and `default_checks()` needs an addition.
3. Check whether the missing dependency was declared. If yes, `axi doctor` should have caught it. If no, the extension manifest needs an `[[extension.requires.*]]` block (or equivalent post-Prague).

Each future incident either confirms the pattern (graduate to L4 procedure) or surfaces a new gap (extend the framework). Every iteration tightens the discipline.

---

## Appendix — the full feedback list this lessons doc was distilled from

User's verbatim 2026-04-28 evaluation feedback included items spanning 🔴 high, 🟠 high, 🟡 medium, 🔵 low. Captured fully in commit history (`1ea4746`, `90d425a`, `5447583`, `a88b610`, `bd2904d` for the axiom side; `25fedf0` for NeutronOS side). The SLM dependency gap was one of seven; the broader CI hygiene pattern is captured in `working/ci-hygiene-plan-2026-04-28.md`.
