# `.claude/` — Claude Code configuration for sumo-qa

Project-shared automations for contributors using Claude Code in this repo. Personal per-user state (`settings.local.json`, `scheduled_tasks.lock`, user-installed skills) stays in this directory but is excluded from git — see `.gitignore` for the precise rules.

## What ships here

| Path | Type | Triggers | Purpose |
|---|---|---|---|
| `settings.json` | Config | Loaded at session start | Wires the three hooks below |
| `hooks/block-generated-paths.py` | PreToolUse hook | Before any `Edit`, `Write`, `MultiEdit`, `NotebookEdit` | Denies edits to generated / cached / captured paths (see "Protected paths" below) |
| `hooks/validate-on-content-edit.py` | PostToolUse hook | After `Edit`, `Write`, `MultiEdit` under `knowledge/` or `standards/` | Runs `sumo-qa-validate` and surfaces schema failures in-loop |
| `hooks/route-qa-runners.py` | PostToolUse hook | After a `Bash` `mutmut run` or `promptfoo eval` | Detects surviving mutants / eval FAILs and reminds Claude to route through the matching agent below (see "Route QA runners" below). Advisory — never blocks the Bash result |
| `agents/mutation-survivor-triage.md` | Subagent | Manually invoked by Claude when classifying mutmut survivors | Classifies survivors as equivalent / tautology-killable / genuine-gap / infrastructure-noise; produces a per-module report. Read-only. |
| `agents/eval-failure-diagnoser.md` | Subagent | Manually invoked by Claude after a failing `npm run eval` | Locates the failing promptfoo assertion and recommends a SKILL.md strengthening. Never proposes loosening the rubric. Read-only. |
| `skills/scaffold-sumo-qa-skill/SKILL.md` | User-invocable skill | `/scaffold-sumo-qa-skill` | Scaffolds a new `sumo-qa-<name>` skill: SKILL.md skeleton, matching promptfoo eval YAML stub, and a new `## <approach-tag>` block appended to `knowledge/approaches.md`. The routing line for `skills/sumo-qa-deciding-approach/SKILL.md` is printed as a manual step the contributor adds by hand — the deciding-approach routing table is context-sensitive and resists mechanical patching. Enforces the `sumo-qa-` prefix. |
| `skills/regen-eval-baseline/SKILL.md` | User-invocable skill | `/regen-eval-baseline <skill-name> [label]` | Runs one skill's promptfoo eval, snapshots the JSON to `docs/qa/runs/eval-baselines/` (gitignored), diffs against the prior snapshot. |

## Protected paths (PreToolUse deny)

`hooks/block-generated-paths.py` blocks Claude from editing anything under:

- `tests/fixtures/**` — byte-for-byte CLI captures; see `.pre-commit-config.yaml:46`
- `.claude/hooks/fixtures/**` — byte-for-byte `mutmut` / `promptfoo` captures used by the route-qa-runners matcher tests; hand-editing them would mask matcher bugs (regenerate via a real tool run instead)
- `mutants/**` — mutmut working copy, regenerated by `uv run mutmut run`
- `dist/**`, `build/**` — hatchling build output
- `node_modules/**`, `.venv/**` — dependency installs
- `.hypothesis/**`, `.pytest_cache/**`, `.ruff_cache/**`, `.mypy_cache/**` — tool caches
- `**/__pycache__/**`, `.coverage`, `.coverage.*` — Python caches and coverage data
- `.idea/**` — IDE state

If you legitimately need to override a deny (e.g. you're regenerating a fixture by hand for a one-off debug session), edit the producer instead. The pipeline-first principle applies: if the output is wrong, the generator is wrong.

## Validate-on-edit (PostToolUse)

When Claude edits anything under `knowledge/` or `standards/`, `hooks/validate-on-content-edit.py` runs `sumo-qa-validate` automatically. On a clean repo it's silent; on a schema break it exits non-zero and Claude sees the failure in the next turn. This mirrors the `.pre-commit-config.yaml:122` hook but fires during the edit loop instead of waiting for `git commit`.

## Route QA runners (PostToolUse)

After a `Bash` command, `hooks/route-qa-runners.py` inspects the result and, when it sees the output of an actual QA runner, injects a reminder (via `additionalContext`) to route through the matching subagent:

- A `mutmut run` that left survivors → reminder to use `mutation-survivor-triage`.
- A `promptfoo eval` / `npm run eval` / `npm run eval:all` that produced a FAIL → reminder to use `eval-failure-diagnoser`.

It is advisory only — it never blocks the Bash result and exits 0 on every path, including internal errors. The detection is built against **real** runner output, captured byte-for-byte under `hooks/fixtures/`: `mutmut run` reports survivors with emoji counters (`🙁` survived, `⏰` timeout, `🤔` suspicious), *never* the word "survived", and exits 0 even with survivors — so the hook parses those counters, not the exit code. `promptfoo eval` exits non-zero (100) on failure and prints `[FAIL]` in its table; either signal counts. The command shape is gated first, so reading a log (`cat mutmut.log`, `grep survived`) or a non-run subcommand (`promptfoo generate`, `npm run eval:view`) never triggers a route.

## Personal vs project-shared

The `.gitignore` rule under `.claude/*` works as a denylist with explicit re-includes:

- **Shared** (committed to the repo): `settings.json`, `hooks/`, `agents/`, and the two named skills above.
- **Personal** (stays on your machine): `settings.local.json`, `scheduled_tasks.lock`, any other skill folder you drop under `.claude/skills/`.

If you want to share a new skill or subagent with the team, add an explicit `!.claude/skills/<your-skill>/` line to `.gitignore` and commit. The blanket `.claude/skills/*` keeps everything else local by default.

## Quick start

After cloning, no setup is required for the hooks — they activate automatically when you open this repo in Claude Code. To verify:

```bash
# In a fresh Claude Code session for this repo:
ls .claude/hooks/                       # the .py hook files should be executable
# Ask Claude to edit a fixture file — it should be denied with a reason.
```

To use the subagents:

> "Use the `mutation-survivor-triage` agent on the latest mutmut run."

> "Use the `eval-failure-diagnoser` agent on the last `npm run eval` output."

To use the skills:

```
/scaffold-sumo-qa-skill triaging-flaky-tests
/regen-eval-baseline implementing-with-tdd postcut
```

## Adding new automations

- **New hook** — drop a script in `hooks/`, wire it in `settings.json` under the matching event. Use the existing three as templates. PreToolUse hooks return JSON to deny; PostToolUse hooks either exit non-zero with stderr to surface a failure (`validate-on-content-edit.py`) or emit `hookSpecificOutput.additionalContext` JSON to inject a non-blocking reminder while still exiting 0 (`route-qa-runners.py`). If a hook matches the output of an external CLI, build its matcher against a **real** captured run under `hooks/fixtures/`, not an invented sample — runner output rarely looks like you'd guess (mutmut emits emoji, not the word "survived").
- **New subagent** — add a markdown file under `agents/` with YAML frontmatter (`name`, `description`, `tools`). Subagents are read by Claude when it decides to dispatch one — write the description as the trigger condition, then the body as the system prompt.
- **New skill** — use `/scaffold-sumo-qa-skill` if it's a sumo-qa sub-skill (it'll handle the catalogue registration). For repo-local helper skills not tied to the sumo-qa estate, create `skills/<name>/SKILL.md` directly and add a `!.claude/skills/<name>/` line to `.gitignore`.

## Source tree vs installed package vs MCP runtime

Three distinct content sources matter when working on sumo-qa, and conflating them is the most common source of confusion:

| Source | Path | Authority for |
|---|---|---|
| **Source tree** | `./knowledge/`, `./standards/`, `./skills/` | What WILL ship once committed and released. May contain in-flight entries not yet served by the running MCP. `sumo-qa-validate` and the `validate-on-content-edit.py` hook check this surface. |
| **Installed package** | `site-packages/sumo_qa/_data/` | What the MCP server's loaders read at process start. Updates only when `pip install -e .` (or a fresh `pip install`) runs. |
| **MCP server runtime** | Live `sumo-qa` process serving JSON-RPC | What Claude actually sees. Reflects the installed package, modulo any process-level caching. |

When the subagents or skills under `.claude/` need to verify that a sumo-qa skill / approach / classification / technique exists, **prefer the MCP tool surface** (`sumo_qa_load_approaches`, `sumo_qa_load_classifications`, `sumo_qa_load_principles`, `sumo_qa_load_techniques`, etc.) over reading the source-tree markdown directly. The MCP tools are the authoritative shipped view; the source tree may be ahead of release.

If you author a new approach tag or skill, `sumo-qa-validate` will pass against the source immediately — but `sumo_qa_load_approaches` will keep returning the prior catalogue until the package is reinstalled and the MCP host restarted. That gap is normal mid-development; just don't surprise yourself with it.

## Related

- `hooks/` (repo root, not `.claude/hooks/`) — MCP plugin hooks shipped with the `sumo-qa` package itself; see `hooks/hooks.json`. Distinct from the Claude Code hooks documented above.
- `.pre-commit-config.yaml` — git pre-commit hooks that overlap with some of the in-editor protection here (ruff, shellcheck, sumo-qa-validate, audits). The Claude Code hooks fire earlier in the loop; pre-commit catches anything that slipped past.
- `AGENTS.md` — AI-agent-facing setup instructions for end users installing the sumo-qa MCP server. Not about contributors hacking on this repo.
