# CLAUDE.md

Guidance for Claude Code (claude.ai/code) when working in this repository.

> This file is the always-loaded guide — it documents *current behavior, config keys, and landmines*. Forensic history (why old behavior was wrong, which commit/version introduced a fix, which test pins it) lives in git log and `memory/`, not here.

## Build & Test

```bash
uv sync --extra dev          # install all deps incl. dev tools (CI uses this, NOT plain `uv sync`)
uv run pytest tests/ -q      # full test suite
uv run pytest tests/path::ClassName::test_name -v   # single test
uv run pytest tests/ -q --cov=claw_forge --cov-report=term-missing   # coverage (CI gate: 90% branch)
uv run ruff check claw_forge/ tests/         # lint (CI uses this exact command; --fix to auto-fix)
uv run mypy claw_forge/ --ignore-missing-imports
```

## Release Process

1. Push to `main`. 2. Create a GitHub Release with a tag (e.g. `v0.1.0a64`). 3. The `Publish to PyPI` workflow triggers **on release publication** (not on CI passing) — re-runs tests, builds the UI, patches the version from the tag, publishes the wheel.

**`pyproject.toml` version is a static `0.0.0.dev0` placeholder — never bump it manually.** The publish workflow overwrites it from the release tag at build time.

## Architecture Overview

claw-forge is a **multi-provider autonomous coding agent harness**: reads an app spec, decomposes it into a feature DAG, orchestrates parallel AI agents to implement features / fix bugs / run tests — all tracked via a local REST+WebSocket state service.

### Module Map

```
CLI (Typer)
  cli.py                — Typer app assembly + residue commands (version, init) + back-compat shim
  _console.py           — shared Rich Console singleton
  config.py             — config load, env-var expansion, default scaffolding
  state/cli.py          — subapp: status|state|pause|resume|input|add|reset (reset → /tasks/requeue)
  state/client.py       — state-service HTTP client + session resolution
  spec/cli.py           — subapp: plan|validate-spec|import (+ _write_plan_to_db)
  orchestrator/run_cli.py       — `run` command + dispatcher loop + signal handlers
  orchestrator/task_handler.py  — per-task coroutine (RunContext-driven) + agent-prep
  orchestrator/checkpointing.py — periodic auto-checkpoint task
  pool/cli.py · bugfix/cli.py · git/merge_cli.py    — pool-status · fix · merge
  git/cli.py            — subapp: worktrees list|prune
  git/stash_cli.py      — subapp: stash list|drop
  ui_cli.py             — subapp: ui|dev (Kanban launcher)
  export_cli.py         — subapp: export csv|sql|json|training
  training/cli.py       — subapp: traces status|prune
  boundaries/cli.py     — subapp: boundaries audit|apply|status
  agent/sandbox.py      — OS filesystem jail (macOS sandbox-exec / Linux bwrap); owns _make_sdk_sandbox_settings
  agent/container.py    — opt-in per-agent container isolation (agent.isolation: container)
  git/leak_watch.py     — mid-run leak detector (snapshot+auto-stash around each agent)
  git/cleanup.py        — smart-mode startup cleanup (preserve/salvage/remove per task state)
  git/conflict_advisor.py — opt-in LLM advisor drafting CONFLICT_PROPOSAL.md on salvage conflict

State Service (FastAPI + SQLite via aiosqlite)
  state/service.py      — REST + WebSocket /ws + SSE /events (god-object decomposition in progress)
  state/metrics.py      — Prometheus /metrics rendering (pure, unit-testable)
  state/auth.py         — opt-in bearer-token auth (pure check + client header helpers)
  state/models.py       — Session, Task, Event ORM models
  state/scheduler.py    — topological DAG scheduler

Orchestrator + Agent + Pool
  orchestrator/dispatcher.py — asyncio.TaskGroup parallel dispatch
  agent/runner.py            — wraps claude-agent-sdk query()
  pool/manager.py            — multi-provider rotation + circuit breaker

Git Workspace
  git/slug.py      — centralized slug gen (make_slug, make_branch_name)
  git/branching.py — feature branch create/switch/delete + worktree lifecycle
  git/commits.py   — checkpoint commits with structured trailers
  git/merge.py     — squash-merge to main; sync_worktree_with_target (pre-dispatch catch-up)

Spec & Export
  spec/parser.py   — XML/plain-text spec → ProjectSpec (honours <feature index, depends_on, shape>)
  exporter.py      — read-only CSV/SQL/JSON export from state.db

Boundaries Harness  (walker · signals · scorer · audit · classifier · report · refactor · apply)
```

### How a `claw-forge run` Works

1. State service (`localhost:8420`) starts as a subprocess; manages all task state in SQLite.
2. Dispatcher pulls "ready" tasks from the scheduler (deps all completed-and-merged).
3. Each task → matching **plugin** (coding, testing, reviewer, bugfix).
4. Plugin calls `run_agent()` (wraps `claude_agent_sdk.query()`, yields messages).
5. Result PATCHed to the state service; WebSocket broadcasts to the UI.
6. On failure: exponential-backoff retry (up to `retry_attempts`); dependents become "blocked".

### Provider Pool (`pool/`)

`ProviderPoolManager` wraps 8 providers (Anthropic, Bedrock, Azure, Vertex, OpenAI-compat, Anthropic-compat, Ollama, OAuth). Each request routes through `Router` (5 strategies: PRIORITY, ROUND_ROBIN, WEIGHTED_RANDOM, LEAST_COST, LEAST_LATENCY), respects per-provider `CircuitBreaker` (CLOSED→OPEN after N failures→HALF_OPEN after timeout), tracks cost/tokens/latency, and falls through the full chain before raising `ProviderPoolExhausted`.

**`anthropic_oauth` never reads the keychain** (`pool/providers/registry.py`): each OAuth provider must supply its own `oauth_token` (or `api_key`) explicitly — claw-forge does **not** read `~/.claude/.credentials.json` (auto-reading collapsed a multi-account pool onto one identity). A provider with neither raises `ValueError` at build time and is skipped. The keychain readers in `oauth.py` remain as an opt-in manual helper, unwired from pool construction.

**Fail-fast on fatal auth** (`agent/rate_limit.py:is_fatal_auth_error` + `provider_exhaustion_message`): a *"no session"* auth error (`"Not logged in"` / `"Please run /login"` / `setup-token`) is distinct from a *rejected* credential (`invalid_api_key`/`401`/`authentication_failed` — those stay **rotatable** so a multi-account pool can skip a bad account). When provider rotation **exhausts** the whole pool on a fatal-auth error, `task_handler` emits the accurate actionable message (not the misleading "transient errors; wait for recovery_timeout") and tags the result `fatal_auth: True`; the dispatcher then `pause()`s (drain mode) so remaining tasks stay **pending** for a re-run instead of cascading the whole DAG into `failed`. Empty/misconfigured `oauth_token` is the usual trigger (agent falls back to the unauthenticated `claude` CLI).

### Local Config Overlay (`claw-forge.local.yaml`)

Per-machine overrides live in a gitignored `claw-forge.local.yaml` deep-merged on top of committed `claw-forge.yaml`. `config.py:_load_config` loads base → merges overlay (`_deep_merge`) → expands `${ENV}` (merge before expansion, so overlay may use env refs). `run --no-local` loads base only (reproducible/CI). **`claw-forge init` scaffolds an all-commented overlay** (`config.py:_scaffold_local_overlay`, gitignored at creation) so overrides have a home that survives `run` — editing the *tracked* base instead gets stashed/reverted by the startup uncommitted-state handler. An all-comment overlay parses to `None` → the merge no-ops (`isinstance(local_raw, dict) and local_raw`).

- **Merge** (`_deep_merge` + `_merge_providers`): nested dicts recurse (overlay wins); `providers` merges by name in both formats — name-keyed mapping and list-of-dicts (unmatched providers appended last = lowest priority). Inputs never mutated.
- **Persist target**: the Kanban UI's provider enable/disable writes the **overlay**, never the base — committed config stays untouched and a per-machine toggle never becomes a tracked diff. `_persist_provider_enabled` gitignores the overlay at creation. The base stays tracked (only `${ENV}` refs, no raw secrets).

### Agent Runner (`agent/runner.py`)

`run_agent()` adds on top of the raw SDK:
- **Auto-injected MCP**: SDK MCP (in-process, zero cold-start) for features; LSP detected from file types (`.py`→pyright, `.ts`→typescript-language-server).
- **Auto-injected skills**: role-specific + task-keyword matched from `skills/`.
- **Thinking strategy** varies by `agent_type` (initializer=deep, coding=adaptive, testing=medium).
- **Security hooks**: `CanUseTool` callbacks block dangerous bash; filesystem writes restricted to `project_dir`.

`collect_structured_result()` for structured JSON output (review verdicts, feature summaries).

### Worktree-Leakage Defenses

Six independent layers prevent the leak class — agent writing to the *parent project's* working tree instead of its worktree under `.claw-forge/worktrees/<slug>/`. Each layer is an independent backstop:

1. **File-tool path sandbox** (`agent/permissions.py:make_can_use_tool`): Read/Write/Edit/Glob/Grep sandboxed to `project_dir`. Bash absolute paths outside the project are denied — **except read-only discovery** (`READONLY_DISCOVERY_COMMANDS`: ls/stat/file/which/realpath/…, plus `find` with no write predicate) against system roots (`/usr`, `/lib`, `/tmp`, …) and `$HOME` toolchain dirs (`~/.cargo`, `~/.rustup`, `~/.local`, …) — agents must locate compilers/toolchains (lowband Rust: `ls /usr/bin/gcc*`, `ls ~/.cargo/bin` were denied, blocking builds). `/root`, foreign homes, `~/.ssh`-style credential dirs and content dumpers (cat/head/tail) stay denied. Layers 1–2 are enforced via the SDK's `can_use_tool` callback under `permission_mode="default"` on the dispatch path; both are inert under `bypassPermissions`, which the dispatcher must never use.
2. **Exempt-command jail check** (`forbidden_jail_root=…`): even for exempt commands (`git`, `python`, `npm`…), denies path args inside the parent project but outside the worktree (catches `git -C`, `python /abs/script.py`, `--git-dir=…`). Wired to `project_path` whenever `git_enabled`.
3. **Agent-prompt boundary directive**: appends a `## Workspace boundary (CRITICAL)` paragraph; the parent project's absolute path is *not* named (information-hiding — the agent only sees its cwd as "the project root").
4. **OS filesystem jail** (`agent/sandbox.py`, macOS sandbox-exec / Linux bwrap): kernel denies `file-write*` outside the worktree → `EACCES`/`EROFS`. Falls back to behavioural-only stack where unsupported (Windows/BSD), with a logged warning.
   - **Use canonical paths** (`Path.resolve()`): macOS compares against the kernel's resolved view (`/tmp`→`/private/tmp`); Linux bwrap follows symlinks at mount time. Linux layer-orders `--dev-bind / /` → `--ro-bind <project>` → `--bind <worktree>` (later mounts override) to mirror macOS.
   - **Git common-dir carve-out**: linked worktrees share object store/refs/metadata at `<project>/.git/`; that subtree must be writable for `git commit` (resolved via `_resolve_git_common_dir`), else `index.lock: Operation not permitted`.
   - **Subagent inheritance**: Task-tool subagents share the parent claude process; sandbox-exec's process-tree inheritance carries deny rules to every descendant.
5. **Container isolation** (`agent/container.py`, opt-in `agent.isolation: container`): worktree mounted at `/workspace`, parent project absent from the namespace (escapes return `ENOENT`). ~1–3s per-task startup. Auth via env-var forwarding (`-e ANTHROPIC_AUTH_TOKEN`); no `~/.claude/` bind-mount. `init --container` materialises `.claw-forge/agent.Dockerfile`; when `agent.container.image` is empty, `resolve_auto_container_image` hashes the Dockerfile (sha256[:12]) → `claw-forge-agent:<hash>`, building only on cache miss.
6. **Supply-chain hardening** (defaults-on): closes the host-side blast radius Layers 1–4 leave open (a malicious npm `postinstall` runs as the npm child, not a Bash command our hook sees).
   - **`agent.npm_ignore_scripts: true`** injects `npm_config_ignore_scripts=true` + `YARN_ENABLE_SCRIPTS=false` into `sdk_env` — disables install-time hooks for npm/pnpm/yarn (`npm run <name>` still works). Container mode forwards via `-e`.
   - **`agent.sandbox.home_protection: true`** extends the sandbox with deny-write rules against credential stores (`~/.ssh`, `~/.aws`, `~/.gnupg`, `~/.config/gh`, `~/.docker`, `~/.kube`), shell rc files, package caches (`~/.npm`, `~/.cache`, `~/.yarn`), and persistence dirs (`~/Library/LaunchAgents`, `/etc/cron.d`, `/etc/systemd/system`). Reads still allowed. Linux uses `--ro-bind-try` (missing optional paths no-op). No-op in container mode.
   - **`agent.container.network`** (container-only, default unset): `write_container_wrapper` emits `--network <value>` — `"none"` cuts egress, a named network attaches a pre-firewalled allowlist, unset keeps Docker's bridge. **Non-container sandbox egress remains unconstrained** (a compromised package can still phone home over HTTPS).

**Per-plugin sandbox fence** (`agent.sandbox.fence_to_touches_files`, opt-in, default `false`): narrows a `shape="plugin"` task's writable set from the whole worktree to its `touches_files` dirs (via `derive_plugin_writable_roots`) + `<worktree>/.claw-forge/` + the git common dir; out-of-footprint writes then fail with `EACCES`/`EROFS`. `shape="core"`/unannotated tasks and un-narrowable footprints (leading-glob like `**/*.py` → `None`) fall back to whole-worktree. Both wrapper writers `mkdir -p` each fenced subpath first (`_ensure_writable_subpaths_exist`) — Linux bwrap refuses to `--bind` a missing source path, which would kill every agent at launch in a greenfield worktree.

**Cross-layer rule — nested sandbox prevention (CRITICAL):** when Layer 4 (sandbox-exec) or Layer 5 (container) is active, the SDK's `SandboxSettings.enabled` MUST be `False`. macOS forbids nested `sandbox_apply()`; nesting → sandbox-exec exits 71 → Bash tool fails `Exit code 71`. Enforced in `cli.py:_make_sdk_sandbox_settings(isolation_active=…)`. **Before adding a third sandbox layer (gVisor, Firecracker, seccomp/landlock) or re-enabling SDK sandbox anywhere: verify the nesting matrix.** See `memory/project_nested_sandbox_landmine.md`.

**Snapshot-rollback backstop** (`git/leak_watch.py`): enforces `project_path` stays HEAD-clean for the run's lifetime.
1. **Boot baseline** at dispatcher startup — `capture_boot_baseline` stashes residual dirt as `claw-forge-boot-<ts>` (no-op when smart-cleanup already cleaned).
2. **Per-task post-rollback** (dispatcher `finally`, before `apply_on_completion`, under `git_ops.exclusive_section()`): snapshot status, stash dirt as `claw-forge-leak-<task_id>-<ts>`, verify clean.
3. **Pre-task assertion**: warns (diagnostic only) if `project_path` is unexpectedly dirty.

`claw-forge stash list` shows all kinds (`auto`/`manual`/`leak`/`boot`); recover via `git stash apply <ref>`. Best-effort throughout — a git failure logs at `error` and continues (leak prevention must not become a liveness gate).

### Plugin System (`plugins/`)

Discovered via Python entry points (`[project.entry-points."claw_forge.plugins"]`). Each implements `AgentPlugin`: `name`, `description`, `get_system_prompt(context)`, `execute(context) -> PluginResult`. Built-ins: `initializer`, `coding`, `testing`, `reviewer`, `bugfix`.

### MCP Servers (`mcp/`)

- **`sdk_server.py`** (`create_feature_mcp_server`) — in-process SDK MCP, default (`use_sdk_mcp=True`).
- **`feature_mcp.py`** — subprocess FastMCP fallback. Tools: `claim_feature`, `update_feature`, `list_features`, `get_feature`.

### State Service Auto-Start (`cli.py:_ensure_state_service`)

`run` and `ui` both auto-start the service:
- **Project-aware**: passes `--config` (project's `claw-forge.yaml`) so the subprocess finds the right config even when CWD differs.
- **Project verification**: confirms `/info` `project_path` matches the intended project.
- **Port fallback**: configured port occupied → tries port+1 … port+4.
- **Version check**: restarts if running version ≠ CLI version.
- **Opt-in bearer auth** (`state/auth.py`): unauthenticated by default. `CLAW_FORGE_STATE_TOKEN` (env, primary) or `state.auth_token` (config) makes the service require `Authorization: Bearer <token>` on every REST request and WS handshake (`?token=` query param accepted since browsers can't set WS headers). The secret in an env var means server + spawned clients agree with zero plumbing. Clients thread `auth_headers()` (returns `{}` when unset → no-op when off). The UI proxy holds the token server-side; keep it localhost-bound.
- **Bind host** (`state/cli.py:resolve_bind_host`): `--host` → `state.host` → `0.0.0.0` default. **Default is `0.0.0.0`** (non-breaking; shared-box workflow relies on it — see `memory/reference_remote_eyeworld_box.md`); opt into local-only via `state.host: 127.0.0.1`. `exposure_warning` prints at startup when resolved host is non-loopback and no auth token is set.

### Session Resolution (`cli.py:_resolve_latest_session`)

All auto-detection (ui prod/dev, dev command) is **project-filtered** (`WHERE project_path = ?`) and centralized in this one function.

### State Service API (default `localhost:8420`)

- `POST/GET /sessions`, `GET /sessions/{id}`
- `POST /sessions/{id}/tasks` (optional `touches_files: list[str]` for file-locks), `PATCH …/tasks/{id}` (status, cost, tokens, merged_to_target_branch)
- `POST …/tasks/{id}/human-input`
- `POST …/tasks/stop-all` · `…/resume-all` — pause/resume in-flight
- `POST …/tasks/requeue` — batch reset failed/blocked → pending. Body `{statuses=["failed","blocked"], error_pattern?}` (SQL `LIKE` on `error_message`, e.g. `"%rate_limit%"`). Backs the Kanban "Reset All" button.
- `POST …/file-claims` (200 / 409 with conflicts) · `DELETE …/file-claims/{task_id}` · `GET …/file-claims`
- `POST /tasks/{id}/agent-log` — record streaming log entry; persists to events table + WS broadcast.
- `GET …/agent-log?since=<id>&limit=<N>` — hydration for the Kanban activity log (default 500, max 5000). UI calls `since=0` on join, then last-seen id on reconnect.
- `WebSocket /ws` — feature updates, cost events, pool health
- `GET /pool/status`
- `GET /metrics` — Prometheus text (`render_prometheus_metrics`): `claw_forge_tasks{status}`, `_cost_usd_total`, `_tokens_total{type}`, `_ws_clients`, per-provider `_provider_circuit_closed`/`_cost_usd`/`_latency_ms`. Best-effort on the pool sub-snapshot (degrades rather than 500s).
- `POST /shutdown`

### UI (`ui/`)

React + Vite + TypeScript Kanban. Built with `npm --prefix ui run build` → copied to `claw_forge/ui_dist/` before wheel packaging; dev-served via `claw-forge ui`. Connects to the WS for real-time updates.

**`ui` prod init order** (ordering matters — `state_port` resolves first): `_ensure_state_service()` → `_resolve_latest_session()` → patch `index.html` with runtime JS config → build reverse-proxy `httpx.AsyncClient` → construct Starlette app (proxy routes + static serving).

### spec/parser.py

Parses `app_spec.{txt,xml}` (greenfield or brownfield) → `ProjectSpec` (features + dependency edges). The `initializer` plugin seeds the task DAG.

**Terminal e2e-verification task** (`spec.append_e2e_task: true` default): closes the "units pass but the assembled app doesn't run" gap. `plan` appends a synthetic **"End-to-End Verification"** feature depending on *every* real feature (true terminal node, dispatches last), `shape="core"`, `touches_files=["tests/e2e/**"]`, run by the **`coding`** plugin (not `testing`, which is a read-only runner). `spec_declares_e2e` makes it idempotent and **suppresses** the auto-append when the spec already designs e2e work. The acceptance gate then enforces these tests at completion. **Preferred path**: `/create-spec` *authors* a `<category name="End-to-End Verification">` (one `shape="core" touches_files="tests/e2e/**"` feature per primary journey + a final `<phase>`), visible/editable in the spec; the synthetic single-task fallback only fires for specs that don't design e2e.

**XML schema — two `<category>` forms** (coexist):
- Legacy bullets: `- User can register` lines.
- `<feature>` elements with optional `index` + `depends_on` (1-based feature numbers, resolved to 0-based positional indices):
  ```xml
  <feature index="18" depends_on="14"><description>…</description></feature>
  ```

**Architectural-shape attributes on `<feature>`:**
- **`shape="plugin"|"core"`** — vertical (own directory) vs cross-cutting. Plugin auto-derives `touches_files`; core MUST declare `touches_files` (else parse-time `ValueError`).
- **`plugin="<name>"`** — directory ownership; `shape="plugin" plugin="auth"` defaults `touches_files=["src/plugins/auth/**"]`.
- **`touches_files="a,b,…"`** — explicit globs; required for `core`, optional override for `plugin`.

Both `shape` and `plugin` persist as `Task` columns and are exposed via the API. Specs without shape attributes parse identically (backward compatible). Parse-time `ValueError` for: unknown `shape=` values, `shape="core"` without `touches_files=` (these bypass Layer 4 because the dispatcher acts on `Task.shape`).

**Brownfield manifest generator** (`generate_brownfield_manifest`): the greenfield post-run hook in `run_cli.py` is the sole producer of `brownfield_manifest.json` (the README's `claw-forge analyze` was never implemented — see `memory/project_analyze_command_unimplemented.md`). Honesty contract on `test_baseline`: `"greenfield build: N feature(s) completed — run your test suite to record a real baseline"` (provenance prefix so consumers don't misread feature-count as test posture). `_sanitize_raw_stack` strips XML tags + caps at 200 chars. Shape `{stack, test_baseline, conventions}`.

**Dependency inference** (`_assign_dependencies`): assigns each feature to an `<implementation_steps>` phase by case-insensitive, punctuation-stripped word-overlap (`phase_keyword_tokens` splits on `[^a-z0-9]+`) between category name and phase title. Features in phase N get inferred `depends_on` = all phase N-1 features. **Features matching no phase get no inferred deps** (cycle-safe — declare `depends_on=` explicitly for foundation features, or rename a phase title to include a category keyword). A defense-in-depth filter drops any inferred edge that would close a cycle with explicit edges; explicit edges are never dropped (scheduler `validate_no_cycles()` catches user-authored cycles).

### spec/validator.py — Layer 4 (Shape)

`validate-spec` catches shape-awareness gaps. **Strict-shape is on by default** (`--soft-shape` keeps promotable WARNINGs 3/5/9 at WARNING during migration; `--strict-shape` is a deprecated no-op).

| Gap | Severity | Check |
|---|---|---|
| 1 | ERROR always | `shape="plugin"` missing both `plugin=` and `touches_files=` (disables file-locking) |
| 3 | ERROR / soft-WARN | `shape="core"` `touches_files` overlaps a plugin dir (conservative fnmatch) |
| 5 | ERROR / soft-WARN | brownfield-only — `plugin="X"` references a non-existent `src/plugins/` dir |
| 6 | WARNING | heuristic — unannotated bullets look vertical (≥3 bullets, 0.6 vertical-ratio) |
| 7 | WARNING | `shape="core"` `depends_on` chain depth ≥4 (every link serializes) |
| 8 | ERROR / `--allow-soft-migration-shape` | migration/DB-schema wording but not `shape="core"` (alembic revision tree is shared; recommended glob `migrations/versions/**`) |
| 9 | ERROR / soft-WARN | no explicit `depends_on=` and category shares no keyword with any phase title (`/fix-spec` auto-adds prev-in-category dep) |
| 10 | ERROR always | brownfield-only — resolved `touches_files` match zero real files (file-locking silently disabled). **Exempt**: `tests/e2e/`, `tests/integration/` globs (created by the terminal e2e task) |
| 11 | WARNING | `shape="plugin"` `touches_files` reach outside its own dir into a shared registration seam (where parallel plugin agents collide; what the fence forbids) |
| 12 | WARNING | brownfield-only — ≥5-feature spec where no feature declares a dependency manifest (`pyproject.toml`/`package.json`/…); recommends a front-loaded foundation `shape="core"` task |

**Brownfield detection** (`detect_brownfield`, used by Gaps 5/10): layout-agnostic markers — `.git`, `pyproject.toml`, `package.json`, `go.mod`, `Cargo.toml`, `Gemfile`, `build.gradle`, `pom.xml`, `composer.json`, legacy `src/plugins/`. Filesystem Gaps 5/10 *also* require `spec.is_brownfield` (a greenfield spec's `plugin=` dirs are aspirational — running them against the surrounding dir produced spurious ERRORs). Both reference specs (`app_spec.template.xml`, repo-root `app_spec.example.xml`) must validate with zero errors AND zero warnings (regression anchor).

`--project-root` overrides root detection; otherwise walks up from the spec file for `.claw-forge.yaml` then `.git` (filesystem checks skipped with a notice if neither found).

### Export (`exporter.py`)

`claw-forge export` reads `.claw-forge/state.db` directly via `sqlite3` (no service dep, safe mid-session) → CSV (flat/split), SQL dump, or JSON. `--scope session|all`, `--csv-mode flat|split`, `--session UUID`.

### Boundaries Harness (`boundaries/`)

`claw-forge boundaries audit|apply|status` identifies and refactors plugin-extension hotspots.
- **Audit (read-only)**: walker enumerates via `git ls-files`; scorer rates dispatch density / import centrality / churn / function centrality → weighted composite ranks hotspots; classifier labels each `registry`/`split`/`extract_collaborators`/`route_table`; emits `boundaries_report.md`.
- **Apply (mutates)**: per hotspot, spawns a coding subagent on its own feature branch under `.claw-forge/worktrees/`, runs the test command, squash-merges on green / reverts on red. Reuses `git/branching.py` + `git/merge.py`. Runs **serially** (refactors share files / may depend on each other). `apply --hotspot <path>` one-at-a-time; `apply --auto` batch.

Refactoring a hotspot into a registry/split/route-table pattern lets future additions land as `<feature shape="plugin">` cleanly. See `docs/commands.md` → boundaries → "Brownfield workflow".

### Git Branch Naming (`git/slug.py`)

`feat/{category}-{description-slug}` (e.g. `feat/auth-jwt-authentication`). Leading action verbs stripped (read as noun phrases). `make_slug()`/`make_branch_name()` are the single source. Squash-merge commit messages include completed steps + task-ID + session trailers.

### Git Worktree Lifecycle (`git/branching.py`)

Each concurrent agent gets an isolated worktree under `.claw-forge/worktrees/` (shared `.git` object store; only working-tree files separate). Cleanup layers:
- **Post-merge** (`squash_merge`): `remove_worktree()` + `delete_branch()`.
- **Pre-create guard** (`create_worktree`): force-removes an existing target path first.
- **Failure preservation**: worktree preserved if the branch has checkpoint commits (retry resumes).
- **Orphan scan** (`scan_orphaned_branches`, `merge_strategy: manual`): lists orphaned branches with copy-pasteable git commands.
- **Manual cleanup** (`claw-forge worktrees list|prune`): `prune` salvage-merges then removes; `--discard` force-removes. **Safe mid-session** — reads `state.db` for `status='running'` tasks and skips their slugs (`--force` opts out, only after stopping the dispatcher).
- **Smart-mode startup cleanup** (`git/cleanup.py`, default `git.cleanup_orphan_worktrees: smart`): walks each worktree at startup, dispatches preserve/salvage/remove per the task's DB status. Matrix: `pending`+commits → preserve; `failed`/`completed`+commits → salvage; no matching task+commits → salvage; empty → remove. Salvage conflicts preserve + report. No-op when `merge_strategy: manual`. Owns cleanup (replaces the unconditional sweep).
- **LLM conflict advisor** (`git/conflict_advisor.py`, default `git.llm_conflict_proposals: true`): on smart-salvage conflict, drafts `CONFLICT_PROPOSAL.md` in the worktree. Advisory only (never lands on main); one agent call per conflict. ⚠️ Drafts proposals *before* the dispatcher loop — can manifest as "0 tasks in process" after restart; disable with `false` (see `memory/project_llm_advisor_startup_block.md`).
- **Post-merge target push** (`git.push_after_merge`, opt-in / `--push-after-merge`, default `false`): pushes only the *target* branch after each non-no-op squash. Replaces deprecated per-turn `agent.auto_push` (now warns + ignored). Push failures log a warning, don't fail the task.
- **Startup uncommitted-state handling** (`git.handle_uncommitted_at_startup`, default `smart`): agent worktrees branch from main's HEAD, so anything not in HEAD is invisible + a future squash collision. `smart` (autonomous): archives untracked → `.claw-forge/orphans/<ts>/<path>` + `PROVENANCE.json`, stashes dirty-tracked as `claw-forge-auto-<ts>` (both reversible, never aborts). `abort`: fail-fast on any uncommitted state. `ignore`: legacy no-op. Hard-gate: no worktrees created until main is clean.
- **Non-repo init + unborn-repo bootstrap** (`git/repo.py`, run at startup before the uncommitted handler). Step 1 — `ensure_git_repo` (gated on "not a work tree", `rev-parse --is-inside-work-tree` exits 128): `git init`, then `git symbolic-ref HEAD refs/heads/<target>` (NOT `git init -b`) so the unborn HEAD targets `git.target_branch` (a bare init honours machine `init.defaultBranch`, often `master`, breaking every squash). Established/unborn repos untouched. Step 2 — `bootstrap_initial_commit` (gated on "no HEAD", `rev-parse --verify HEAD` fails): writes `.gitignore` via `ensure_gitignore` first (so `.claw-forge/`, build dirs, `node_modules` aren't vacuumed in), then `git add -A` + one `--allow-empty --no-verify` commit so worktrees can anchor. Established repos never auto-committed.

### Pre-Dispatch Worktree Sync (`task_handler` + `git/merge.py:sync_worktree_with_target`)

Worktree is synced to `target_branch` **before** agent execution (not after). `_create_and_sync_worktree()` creates/resumes then runs `git merge --no-verify --no-edit <target>` *inside* the worktree. Outcomes:
- **No-op** (`{synced, no_op, merged_count:0}`) — at target HEAD; agent runs.
- **Clean merge** (`{synced, merged_count:N}`) — agent runs on synced worktree.
- **Conflict / dirty** (`{synced:False, conflicts|dirty_worktree}`) — agent does NOT run; task → `failed` with `resume_conflict:` error, claims released, worktree preserved for manual resolution.

This prevents a resumed branch only seeing target's newer commits at squash time (after the agent already wasted its turn). `squash_merge`'s catch-up rebase is the second-line defence for target moving mid-execution. `branch_overlap_files()` is the read-only diagnostic counterpart.

### Periodic Auto-Checkpoint (`task_handler`)

Background `asyncio.Task` periodically commits dirty worktree files. Config `git.auto_checkpoint_interval_seconds` (default 300, `0` disables). Commits use `Phase: auto-save`. **All checkpoint commits are best-effort** — `commit_checkpoint()` catches `git commit` failures (e.g. pre-commit hooks), logs at `warning` with `exc_info`, returns `None`. Started after worktree creation, cancelled in `finally`.

### Emergency Commit on Signals (`cli.py` + `commits.py`)

SIGTERM/SIGINT handlers do a best-effort `git add -A && commit` on all active worktrees before exit. `emergency_commit()` is synchronous, catches all exceptions (safe from signal handlers). `_active_worktrees` registry tracks paths; original handlers restored after the dispatcher loop.

### Resume Context on Retry (`task_handler`)

On retry, the agent gets a resume preamble: prior commit subjects (`branch_commit_subjects()`), the prior `error_message`, `HANDOFF.md` contents if present, and instructions not to redo completed work.

### Task-Level Acceptance Gating (`orchestrator/acceptance.py`, opt-in `agent.acceptance_gate.enabled: false`)

Makes "done" mean *verified*. When enabled, after the agent reports success but **before** the completion PATCH + squash, runs the test command in the task's **worktree**; non-zero exit flips `success→False` → task `failed` with output tail as `error_message` → feeds the retry preamble (bounded by `retry_attempts`). Running in the worktree (= target's merged features + this task's work) validates "passes + didn't regress" without the greenfield trap. Command = `agent.acceptance_gate.command` or `detect_test_command(cwd)` (pytest/npm/cargo/go/make); **no command → gate skips**. **Exit 126/127 → gate skips** (command not found / not executable = gate-*infrastructure* failure, not a test failure — e.g. lowband: worktree-detected `cargo test` but cargo absent from the dispatcher's bare PATH → every task false-failed with exit 127 while the agents' own cargo runs passed; the fix warns and defers rather than discarding verified work). Pure `evaluate_acceptance_gate` + async `apply_acceptance_gate`. **`detect_test_command` is extra-aware for uv/pyproject** (`_uv_pytest_command`): `uv run pytest` installs only default-synced deps (main `dependencies` + PEP 735 `[dependency-groups]`), NOT `[project.optional-dependencies]` extras — so when pytest lives only in an extra (e.g. `dev = ["pytest"]`, common), it emits `uv run --extra <name> pytest` (prefers `dev`/`test`). Without this the gate false-fails every task with `Failed to spawn: pytest` (observed on atlas).

**Pre-gate worktree sync** (`git/merge.py:sync_worktree_before_gate`, git-enabled tasks with a worktree): the worktree is only synced to target at *dispatch* time, but a long-running task finishes long after — target may be many features ahead. Running the whole-suite gate against that stale snapshot false-fails on tests for **concurrently-merged** features (and discards good work — the observed Serviq symptom: a 2h task's own tests passed 74/74 but the gate failed on app-level OpenAPI tests that pass on current main). So immediately before the gate, `sync_worktree_before_gate` commits the agent's work (checkpoint, squashed at merge) then merges target in → the gate evaluates the *real assembled app* (`current-target + this-feature`). `shape="plugin"` footprints are disjoint by construction so the catch-up is clean; a `shape="core"` task can conflict with concurrently-merged work. **When the catch-up can't reach target** (conflict/dirty/error), `gate_worktree_ready(sync_result)` returns False and the gate is **skipped** (not run on the stale worktree — that false-failed the atlas `core/workflow` task on a whole-app e2e test that passes on current main). Verification then defers to the squash-merge catch-up + merge-recovery, which reconcile with target properly. `None` sync result (git disabled / no worktree) → gate runs as before.

### SQLite Crash Safety

WAL mode + multi-layer defense: `synchronous=FULL` (fsync every WAL page); lifespan teardown `wal_checkpoint(PASSIVE)`; atexit `wal_checkpoint(TRUNCATE)` via plain `sqlite3` (no event loop); SafeJSON `TypeDecorator` columns catch `JSONDecodeError` → fallback; tiered startup recovery (drop WAL/SHM → `sqlite3 .recover` → actionable error). `dev.sh` and `claw-forge ui` POST `/shutdown` before SIGTERM.

### Training-Trace Capture (`training/`)

**On by default**: `init` scaffolds `training_traces.enabled: true` + `acknowledged_terms: true`, so every SDK message from `run_agent()` is teed to the `events` table as `event_type="agent_message"`. Set both `false` to disable. Caller-side wrapper (runner stays a pure SDK wrapper); best-effort (a service hiccup never aborts a task). Anthropic Usage Policies apply.

`claw-forge export training` reads `state.db` directly → OpenAI tool-use JSONL (default; ShareGPT/ChatML/raw alts), filters by outcome (`success`/`recovered`/both, `--include-failed` for DPO negatives), redacts secrets + usernames, single file or train/val/test split. `Task.merge_commit_sha` lets `--include-diff` attach the ground-truth diff. Lives in `export_cli.py` (not `cli.py`) so tests import without the full SDK chain.

`claw-forge traces status|prune` is the retention surface; auto-prune at startup via `training_traces.retention.keep_days`. Downstream training (Qwen2.5-Coder + LoRA via Unsloth) lives outside the harness (`docs/training/unsloth-recipe.md`) — personal/internal distillation only; Usage Policies prohibit training competing models.

## Key Conventions

- **Async throughout**: pure `asyncio`, no threads. `asyncio.TaskGroup` for structured concurrency.
- **No DB migrations**: tables created fresh on startup (`CREATE TABLE IF NOT EXISTS`).
- **`uv run`** always — never activate the venv manually.
- **Coverage gate**: CI `fail_under = 90` (branch). New source files without tests fail CI.
- **`pyproject.toml` version** is a static placeholder — never bump (the publish workflow sets it from the tag).
- **`.claw-forge/state.log`** is a gitignored runtime file. Don't commit it.
- **Default state port `8420`** (`--port` / `state.port`).
- **Merge-gating** (`merged_to_target_branch`): a dependent is unblocked only when its parent is `completed AND merged_to_target_branch=True`. Dispatcher PATCHes `False` when starting on a feature branch (`merge_strategy: auto`), `True` after a successful squash. Squash failure → autonomous merge-recovery re-pends up to `git.merge_retry_attempts` before falling back to "completed but not merged". Target defaults to `main` (`git.target_branch`).
- **Autonomous merge-recovery** (`git.merge_retry_attempts: 3` default, 0 disables): squash failure → `_set_merged_after_merge` re-pends with `status='pending'` + `merge_retry_count++` + `error_message='merge_retry_<N>: …'`. Outer-loop sidecar `_pending_merge_retries` re-picks under `prefer_resumable`; the agent re-runs in the preserved worktree with a Merge Conflict Recovery preamble. Symmetric on the pre-dispatch sync-conflict path (re-sync with `leave_conflicts=True`, markers stay in worktree). Cap-hit falls through to preserve-and-warn.
- **Shared-file conflict resolution** (`git/shared_resolve.py`, `git.shared_file_resolution.enabled: true` default): ecosystem-aware *mechanical* resolution of the dominant conflict class — dependency manifests/lockfiles/env templates parallel agents edit as undeclared side-effects. On a catch-up conflict (`sync_worktree_with_target` or `squash_merge`), `auto_resolve` classifies paths against `DEFAULT_RULES` (Node/Python/Rust/Go/Ruby/PHP/Java/.NET/Elixir/Dart/Swift/Haskell) and applies: `union` (additive lists), `union_validate` (manifests — line-union + strict dup-key-rejecting parse, so "both bumped the same dep" bails), `regenerate` (lockfiles — `checkout --theirs` + run lock command, `take_target` fallback), `take_target` (`*.lock` catch-all), `discard` (`git rm` regenerable caches `*.vite-cache/*`/`*.turbo/*`; the only strategy resolving modify/delete). **Two invariants**: (1) *whole-set gate* — a single conflicted file with no rule (real source) disables auto-resolution for the whole merge → falls through to the agent; (2) *best-effort* — any internal error returns `resolved=False`. Wired ONLY on abort-fallback paths (not `leave_conflicts=True`), in front of the merge-recovery loop. Proactive complement: validator Gap 12.
- **Catch-up merge orphan-sideline** (`merge.py:_merge_catchup_with_orphan_sideline`): a distinct failure shared-file resolution does NOT cover — agent left **untracked** scaffolding (Rust workspace root `Cargo.toml`, fresh crate dir) at a path target now tracks → catch-up `git merge` aborts before starting (`untracked working tree files would be overwritten`, no 3-way merge, so `auto_resolve` never engages). Routes the catch-up leg through a handler that sidelines the untracked files into the worktree's `.claw-forge/orphans/<ts>/` and retries once (target's copy wins); any other failure re-raises. **Only** on the `squash_merge` catch-up leg, NOT `sync_worktree_with_target` (its dirty-worktree guard refuses on any untracked file first).
- **File-claim locks** (`touches_files`): before starting an agent the dispatcher POSTs a claim; if any file is held by a running task, the task defers to the next cycle. Auto-release on `completed`/`failed`/`paused`. No `touches_files` → no locking (backward compatible).
- **Footprint-aware slot filling** (`dispatcher.py:select_disjoint_footprint_batch`): a pre-dispatch filter on the wave — greedily select only tasks whose `touches_files` are mutually disjoint (seeded with running tasks' footprints), leaving colliders `pending` for the next re-dispatch. Without it, N features all declaring the same `src/plugins/<plugin>/**` glob would all dispatch, collide on the claim, and the losers would **defer while holding a slot** (the `_run_task` deferred-retry loop reserves the slot) — starving disjoint work in other dirs down to ~1 effective agent even at `--concurrency 4`. Exact-string overlap mirrors the claim's `UNIQUE(session_id, file_path)` semantics. Empty footprint never collides (legacy tasks unaffected); file-claims remain the authoritative guard, this just stops slot-starvation. Note the ceiling is *distinct ready footprints*, not `max_concurrency` — features sharing a whole-plugin glob still serialize (finer `touches_files` in the spec lifts it).
- **Dispatch-time dependency gate** (`Scheduler.are_dependencies_satisfied`): the static wave plan (`get_execution_order`) is an *ordering optimizer* assuming every earlier-wave task completes — but a task can **defer** (file-claim conflict → `deferred` sentinel before marking running, stays `pending`). Before creating each coroutine, the dispatcher re-validates `depends_on` against **live** scheduler state; a dependent whose dep isn't `completed AND merged` is held back (left `pending`) and re-picked later. Advisory plan + authoritative guard.
- **Re-dispatch selection is DB-status-authoritative** (`run_cli.py:select_redispatch_pending`): between waves, re-fetch tasks and re-run anything `status="pending"` in the DB — does NOT gate on in-memory `total_completed`/`total_failed` (those are summary-only). A dict-returning failure lands in `result.completed`, and a `requeue` reset has no channel to in-memory sets — so the old in-memory gate stranded reset-failed tasks as permanently-pending. Terminal tasks are excluded by the `status` check, so status-only selection never re-runs finished work.
- **Resume preference** (`git.prefer_resumable: true` default): within a priority tier, prefers pending tasks whose branch already has committed work (priority still dominates). Staleness gate `git.resume_stale_threshold: 50` skips the preference when target moved >N commits ahead. Independent of it, every dispatched worktree runs through `sync_worktree_with_target` first; conflicts fail the task with `resume_conflict:` rather than running on stale state.
- **Shape-aware scheduling** (`<feature shape>` → `TaskNode.shape`): `shape="plugin"` dispatches up to `max_concurrency` — but footprints are disjoint only *across* plugins; several features in the **same** plugin share its `**` glob and serialize (the footprint-aware slot filter above stops them from starving other plugins' slots). `shape="core"` is **boundary-aware** (not blanket single-flight): a core candidate runs in parallel *unless it can't be proven disjoint* — its `touches_files` overlap a running task's, OR it declares no `touches_files` (footprintless core ↔ running core stays serialized). Migration tasks share `migrations/versions/**` so they overlap and serialize for free. File-claim locks remain the authoritative guard; the `_core_blocked` filter (in `get_ready_tasks`, gated on `running`) just stops over-serialization. Unannotated tasks (legacy) fall through to concurrency-cap + file-claim behaviour.
