# claw-forge SOTA Gap Analysis — Chief Architect Synthesis

> Generated 2026-07-04 by a 14-agent audit (8 subsystem code auditors + 5 SOTA-landscape researchers + 1 verifying synthesizer). 101 raw gaps deduplicated and re-verified against the repo before inclusion.

*(All load-bearing audit claims below were spot-verified against the repo: `runner.py:242` unconditional `can_use_tool` + string prompt, `task_handler.py:1379/1663` `bypassPermissions`, `task_handler.py:1575` stale `!= "env"` guard vs `_NON_POOL_PROVIDER_LABELS:347`, `dispatcher.py:465-466` unconditional `mark_completed`, `run_cli.py:1416-1420` `/project/paused` poll, zero `cost_usd` PATCHes in `task_handler.py`, `registry.py:143` ollama alias, `ui_cli.py:324` hardcoded `0.0.0.0`, `_migrate_schema` (service.py:384) never called and no `merge_commit_sha` ALTER, no `analyze` command in `cli.py`, `slug.py` ignoring `task_id`, `config.py:361` `acknowledged_terms: true`.)*

---

## 1. Verdict

claw-forge's orchestration core — spec-derived DAG, footprint-aware parallel dispatch, worktree isolation with six defense layers, and merge-gated dependencies — is genuinely competitive with the best open-source harnesses (Bernstein, Claude Code agent teams) and ahead of most on scheduling sophistication; its deterministic-scheduler design is exactly what the field converged on. But the audit reveals a wide gap between documented architecture and executing code: the behavioral security layers never fire in production (`bypassPermissions`), `run_agent()` crashes on every non-dispatcher consumer, cost/token tracking records nothing on the real dispatch path, and failed tasks are marked scheduler-completed — several headline features are shelfware. Versus commercial SOTA (Devin, Copilot, Jules), the two biggest product gaps are the absence of any human review surface between "agent done" and "landed on main" (PR-native delivery is now the universal contract) and verification that stops at "suite exit code 0" while the field has moved to F2P/P2P splits, critics, and artifact-backed evidence.

---

## 2. Top 10 Improvements (ranked by impact ÷ effort)

### 1. Fix `run_agent()` — it unconditionally crashes (critical, small)
**Why:** `runner.py:242` always sets `can_use_tool=make_can_use_tool(...)` then calls `query()` with a string prompt; claude-agent-sdk 0.1.46 raises `ValueError` for that combination before any work happens. This breaks `claw-forge fix`, `boundaries apply`, the boundaries classifier, and every plugin routed through `run_agent`. The same path also exports the wrong OAuth env var (`CLAUDE_OAUTH_TOKEN`, which the CLI never reads — should be `ANTHROPIC_AUTH_TOKEN` per `task_handler.py:1141`'s correct handling).
**Sketch:** In `claw_forge/agent/runner.py`, wrap string prompts in a one-item AsyncIterable (or route through `AgentSession` as the dispatcher does); fix the env var by reusing `_resolve_sdk_credentials` so the two paths can't diverge again. Add one integration test that drives `run_agent` to transport validation (the ValueError fires before connect — no API call needed).
**Effort:** small.

### 2. Fatal-auth drain is defeated by run_cli's pause-poll (high, small)
**Why:** The dispatcher's `pause()` on pool exhaustion is process-local; `run_cli.py:1416` polls `GET /project/paused`, which reads a DB flag only stop-all sets, gets `paused=False`, prints "Resumed", and re-dispatches — repeating until the whole DAG is `failed`. This defeats the exact cascade-prevention the fatal-auth feature (CLAUDE.md) was built for, and it's the failure mode observed on the Serviq box.
**Sketch:** Have `Dispatcher` expose *why* it paused (e.g. `self.pause_reason = "fatal_auth"`); in `run_cli.py`'s paused branch, on fatal-auth exit the wave loop with the actionable exhaustion message, leaving tasks pending. Add a run_cli-level regression test (the existing `tests/test_dispatcher.py` drain test doesn't cover the loop that un-drains it).
**Effort:** small.

### 3. Failed tasks are scheduler-"completed" — dependents dispatch against missing parent work (high, medium)
**Why:** `dispatcher.py:465-466` records any dict-returning result as completed and calls `mark_completed(task_id)` without inspecting `task_result["success"]`; the in-memory node keeps `merged_to_target_branch=True`, so `are_dependencies_satisfied` passes and dependents dispatch in the same run against a target lacking the parent's work. The documented "dependents become blocked on failure" invariant does not hold; `select_redispatch_pending` has the mirror hole (absent failed dep = satisfied).
**Sketch:** In the wave-result processing, branch on `task_result.get("success")` → `mark_failed` + `result.failed` for unsuccessful dicts (keep deferred/fatal_auth special cases first); update the in-memory node's merged flag from the merge outcome; in `run_cli.py`'s re-dispatch selection, load failed/blocked tasks as scheduler nodes so dependents are held back, and PATCH them `blocked` so the UI reflects reality.
**Effort:** medium. This is the single most important *correctness* fix — merge-gating is the product's core invariant.

### 4. Restore the behavioral security layers (high, medium — fix `ALWAYS_BLOCK` first)
**Why:** `task_handler.py:1379` sets `permission_mode="bypassPermissions"`, under which the SDK never invokes `can_use_tool` — so Layers 1–2 of the documented six-layer defense (file-tool path sandbox, Bash path checks, exempt-command jail check, dangerous-command blocklist) are dead code on the production path. OS sandbox (Layer 4) still blocks writes, but dangerous non-write commands (`sudo`, credential exfil via `curl --upload-file`) are unblocked, and the Windows "behavioural-only fallback" doesn't actually exist. Prerequisite: `permissions.py:35-37` substring-matches `"dd"`/`"su"` — flipping the mode today would deny `git add` and `mkdir subdir` on every task.
**Sketch:** Tokenize `ALWAYS_BLOCK` matching in `claw_forge/agent/permissions.py` (match blocked entries as argv[0] of each sub-command via the existing shlex split; keep substring only for multi-token patterns), fix the tests pinning the broken behavior, then switch the dispatcher to `permission_mode="default"` (the callback already allows everything it doesn't deny). Add a transport-level smoke test asserting the can_use_tool round trip so this can't silently regress.
**Effort:** medium.

### 5. Wire cost/tokens/circuit feedback into the SDK dispatch path + session budget cap (high, medium)
**Why:** `UsageTracker.record_request` and `CircuitBreaker.record_success` are only called in `manager.execute` — the fallback path. The real path (`task_handler.py` ResultMessage handling) extracts only `.result` text: Prometheus cost metrics are always 0, LEAST_LATENCY sorts by `inf`, RPM gating never engages, and circuits tripped by `mark_provider_rate_limited` oscillate OPEN→HALF_OPEN forever. Commercial SOTA treats hard budget caps as table stakes (Devin ACU caps); claw-forge has no runaway-spend kill-switch for an overnight autonomous run.
**Sketch:** In the ResultMessage branch of `task_handler.py`, extract `total_cost_usd`/usage, PATCH to the task (API fields exist), and call `pool.tracker.record_request` + circuit `record_success` for the serving provider. Then add `pool.max_cost_usd` checked between waves in `dispatcher.py` → `pause()` with an actionable message. One wiring point fixes metrics, routing, circuit recovery, and unlocks budget governance. Fix the base_url/bedrock misrouting in `_resolve_sdk_credentials` in the same pass (seed `ANTHROPIC_BASE_URL` for compat providers; skip provider types the SDK path can't honour).
**Effort:** medium.

### 6. Acceptance gate v2: F2P/P2P split, flaky rerun, anti-gaming diff lint (high, medium)
**Why:** The gate today is "whole suite, exit code" — vacuously passable by an agent that writes no tests, and false-failable by one flaky test after a 2-hour run (the exact cost class the pre-gate sync fixed). SOTA acceptance semantics everywhere (SWE-bench harnesses, Devin's eval) is FAIL_TO_PASS (task added tests that fail on target HEAD, pass in worktree) + PASS_TO_PASS (baseline still green). Flake handling: rerun failed tests once in isolation, pass-on-rerun = flaky annotation not task failure (Google/Atlassian/Trunk pattern). Anti-gaming: execution gates are gameable by editing tests.
**Sketch:** In `orchestrator/acceptance.py`, capture the baseline result set at `sync_worktree_before_gate` time (P2P for free); require the task's diff to include tests failing on target HEAD (F2P — checkable in the worktree via `git stash`/target checkout). On failure, parse failed node IDs and rerun (`pytest --lf`); all-pass-on-rerun → gate passes with `flaky:` annotation persisted. Add a cheap diff linter in `git/merge.py:squash_merge` flagging added `skip`/`xfail`, deleted test files, net-negative assertions. Bonus held-out property: the existing `fence_to_touches_files` sandbox mechanism can make `tests/e2e/**` read-only for coding tasks — SWE-bench Pro's held-out set, using Layer 4 machinery that already ships.
**Effort:** medium (staged; F2P/P2P and flaky rerun are independent increments).

### 7. Per-task ceilings: wall-clock timeout, max_turns, per-task budget (medium, medium)
**Why:** Nothing bounds a task: the `async for msg` loop in `task_handler.py:1484` has no timeout, no `max_turns`, no `max_budget_usd` (the SDK supports all; `agent/tools.py`'s MAX_TURNS table is only consulted by the dead runner path). A wedged CLI subprocess holds a semaphore slot forever, and the heartbeat reports "alive". Copilot's 59-minute cap and Devin's ACU limits are the market norm; claw-forge's periodic checkpoint even guarantees killed work survives.
**Sketch:** Add `agent.task_timeout_seconds` (generous default, 0=off) enforced via `asyncio.timeout` around the AgentSession loop in `task_handler.py`, failing with a distinctive error the resume preamble picks up; wire `max_turns=get_max_turns(agent_type)` and a config `agent.max_budget_usd` into the ClaudeAgentOptions at `task_handler.py:1375`. Optionally an idle-timeout (no SDK message for N min) to catch wedges without capping long tasks.
**Effort:** medium.

### 8. SDK-native session resume on retry (SOTA, small-medium)
**Why:** Retries rebuild context from a lossy text preamble while the SDK offers full-context `resume=<session_id>` (files read, reasoning intact) — and claw-forge's preserved-worktree retries satisfy the stable-cwd requirement exactly. Also unlocks resume-with-raised-limit on `error_max_turns`/`error_max_budget_usd` instead of cold restart, which pairs with improvement #7.
**Sketch:** Capture `ResultMessage.session_id` in `task_handler.py`, persist as a `Task` column (folds into #9's reconciler), pass `resume=` on the retry/merge-recovery/gate-failure paths with the existing preamble as fallback when the transcript is missing (cross-host, cleaned worktree).
**Effort:** small-medium.

### 9. Replace the three-site ad-hoc migration system with one generic reconciler (high for upgrades, medium)
**Why:** Verified: `_migrate_schema` (service.py:384) is dead code, and `Task.merge_commit_sha` has no ALTER anywhere — a pre-existing `state.db` upgrading to current code fails every `select(Task)` with "no such column". The pattern guarantees recurrence with every new model column. (The PG-specific startup crashes are the same root cause: SQLite-dialect introspection unconditionally applied.)
**Sketch:** In `state/service.py`, one reconciler: for each table in `Base.metadata`, diff `PRAGMA table_info` against model columns, emit `ALTER TABLE ADD COLUMN` for missing ones (gated on `self._is_sqlite`, with `information_schema` for PG or an honest "PG requires fresh schema" error). Delete `_migrate_schema` and the two inline DDL lists. Regression test: boot the current service against a v0.1-era schema fixture.
**Effort:** medium.

### 10. Default-deny egress for the non-container sandbox (SOTA security, medium-large)
**Why:** CLAUDE.md itself admits "non-container sandbox egress remains unconstrained." The 2026 local-harness SOTA (Claude Code's own sandboxing, Codex CLI, Anthropic's open-sourced `sandbox-runtime`) pairs the filesystem jail claw-forge already has with a domain-allowlist proxy outside the jail — and it's the single structural defense that breaks both the prompt-injection "lethal trifecta" (home_protection is deny-*write* only; `~/.ssh` stays readable with open egress) and post-Shai-Hulud credential exfiltration.
**Sketch:** Extend `agent/sandbox.py`: Linux bwrap `--unshare-net` + socat bridge to a localhost proxy; macOS Seatbelt deny non-localhost with the proxy carve-out (or vendor `anthropic-experimental/sandbox-runtime`'s proxy machinery). Config `agent.sandbox.egress: {mode: allowlist, domains: [...]}` seeded from pool provider hosts + package registries, mirroring the existing `agent.container.network` surface. This is policy *inside* Layer 4, not a new layer — the nested-sandbox landmine doesn't apply. Cheaper interim trifecta-breaker: deny-*read* rules on credential stores behind a flag.
**Effort:** medium-large (staged: deny-read is small; the proxy is the larger half).

---

## 3. Quick Wins (<1 day each)

1. **Fix the stale rotation guard** — change `task_handler.py:1575` from `sdk_provider_name != "env"` to `not in _NON_POOL_PROVIDER_LABELS` (the module's own docstring at :396-399 mandates this). Kills the phantom "rotating…" log and the misleading "All 1 configured provider(s) tripped" message replacing the real error under env/keychain credentials. One line + one regression test.
2. **Slug uniqueness** — `git/slug.py`/`branching.py` accept `task_id` but never use it; two same-category tasks with colliding truncated descriptions share one branch *and one worktree* (two agents in one tree, or force-remove under a live agent). Append `task_id[:6]` in `make_branch_name`. Also fixes `cleanup.py:_build_slug_to_task_map` silent overwrites.
3. **Delete the `"ollama": "openai_compat"` alias** (`pool/providers/registry.py:143`) — it makes the dedicated `OllamaProvider` (localhost default, model fallbacks, `ollama pull` guidance) unreachable and breaks the minimal config in its own docstring. Add a pin test.
4. **`claw-forge ui --host` defaulting to `127.0.0.1`** — `ui_cli.py:324` hardcodes `0.0.0.0` with no option, and the proxy injects the bearer token server-side, handing any LAN peer authenticated access to `/commands/execute`. CLAUDE.md already says "keep it localhost-bound"; the code just offers no way. (The *state service* default stays as-is — the shared-box workflow depends on it — but the token-injecting UI proxy is strictly more dangerous.)
5. **Acceptance-gate process-tree kill** — `acceptance.py` `proc.kill()` kills only the `/bin/sh` wrapper; a hung pytest is orphaned holding the worktree's files. `start_new_session=True` + `os.killpg` + reap. One-line semantics fix matching the docstring's existing claim.

*(Honorable mentions, same size class: excise the fabricated `claw-forge analyze` docs from `docs/commands.md`/README + add a docs↔Typer drift test; scaffold `acknowledged_terms: false` in `config.py:361` so the training-trace consent gate actually gates; `rerere.enabled=true` in `git/repo.py`; add `npm --prefix ui run test` to CI.)*

---

## 4. Strategic Bets

### A. PR-native delivery mode (`merge_strategy: pr`) + CI-fix loop
Every major commercial agent (Copilot, Codex, Devin, Jules, Cursor) delivers as a reviewable PR with review comments as the iteration channel; Devin publicizes *merge rate* as the KPI. claw-forge squash-merges to main with zero human checkpoint — the single biggest product gap vs. the market. The build is incremental on existing machinery: push the feature branch + `gh pr create --draft` instead of squashing (checkpoint commits are already the progress trail), then a poll loop that feeds PR review comments and red CI logs into the existing retry-preamble path (`task_handler`'s resume machinery is exactly the right shape). This also gives the enterprise/audit story (branch protections, review gates) essentially for free, and surfaces merged-rate/gate-pass-rate/cost-per-merged-feature as the metrics the market compares on.

### B. Self-measuring eval harness: internal replay benchmark + honest public numbers
The current benchmark posture is a liability: the SWE-bench 64% claim has no in-repo runner, `terminal-bench.md` says "Design — awaiting implementation" while the harness exists, and — most damning — every benchmark bypasses the actual product (direct SDK runner, no DAG/worktrees/gate). Research is unanimous that harness choices swing outcomes 10–20 points and that you must "measure real success on your own repositories." Build: (1) a nightly pipeline-level smoke — small fixture spec through the *real* dispatcher, asserting tasks complete, merge, and the assembled app's tests pass; (2) a replay benchmark from claw-forge's own completed-task corpus (spec-fragment → `merge_commit_sha` diff pairs are ground truth already in the DB) so harness/prompt changes get A/B-measured instead of shipped on vibes; (3) either check in a reproducible SWE-bench runner or retract the number. This bet also feeds a future critic model — the training-trace capture already collects the trajectories.

### C. Continuous scheduling + unified agent-capability path
Two large refactors that compound: (1) Replace the wave barrier in `dispatcher.py` with a ready-queue loop — dispatch any task the moment `are_dependencies_satisfied` + footprint filter + file claim all pass (all three guards already exist; the code's own comments and `busy_slot_seconds` instrumentation anticipate this). With 30min–2h tasks, one straggler currently idles `max_concurrency-1` slots; this is the largest throughput lever in the system. (2) Extract a single `build_agent_options(ctx, task)` factory shared by `task_handler.py` and `runner.py` — the audit found *five* independent divergences of the same "dispatcher bypasses the plugin layer" class (permissions, OAuth env, turn caps, MCP, thinking config), and roughly half of `agent/` is shelfware presented as active architecture. Once unified, the SOTA context-management adoptions (inline test-runner subagents for context isolation, `get_context_usage()`-driven resets, a per-project agent-maintained memory dir) land on one path instead of two.

---

## 5. Explicitly NOT Recommended

- **Migrating orchestration to LangGraph/AutoGen.** The research is clear: no leading coding harness is built on a generic graph framework; purpose-built git/worktree/test-gate primitives win in this domain. claw-forge's deterministic scheduler is the validated pattern (Bernstein independently converged on it).
- **LLM-in-the-scheduling-loop / manager-agent hierarchies (Gas Town-style Mayor/Deacon roles).** Gas Town's own documented weaknesses — cost, "watchers watching watchers," reliability contingent on agent cooperativeness — are the cautionary tale. Keep the coordinator deterministic; where adaptivity is needed, a bounded lead-agent pass that inserts tasks via the existing `POST /sessions/{id}/tasks` suffices.
- **Building a microVM/Firecracker/gVisor layer into the harness.** That's hosted-platform territory (E2B, Modal, GKE); local-first SOTA (Claude Code, Codex CLI) is OS primitives + optional containers, which claw-forge already has. The cheap alternative — expose `agent.container.runtime: runsc` for users with gVisor — gets the isolation tier without new architecture. Any new layer must also clear the nested-sandbox matrix (`memory/project_nested_sandbox_landmine.md`).
- **Training a custom critic model now.** OpenHands' lesson is that critics need *production-scale* merge-survival signal, and theirs transferred poorly from benchmark traces. claw-forge lacks the volume; execution-grounded gates (improvement #6) and a cheap tiered reviewer deliver most of the value today, while training-trace capture accumulates the corpus for later.
- **Adopting Alembic/full migration framework.** The "no migrations, CREATE IF NOT EXISTS" convention is fine for a local single-writer SQLite DB; the generic column reconciler (#9) closes the actual hole at 5% of the cost. Similarly, don't invest in the PostgreSQL backend beyond gating the SQLite-specific startup paths or de-advertising it — multi-tenant PG deployment is not this product's job.
- **Flipping the state-service bind default to 127.0.0.1.** Tempting on security grounds, but the shared-box workflow (`memory/reference_remote_eyeworld_box.md`) depends on `0.0.0.0` and CLAUDE.md documents it as deliberate. The right moves are the UI-host quick win, requiring auth on mutating endpoints when non-loopback, and the existing `exposure_warning` — not a breaking default change.
- **Abandoning macOS sandbox-exec for its deprecation.** It remains deprecated-but-functional with no removal timeline and no headless replacement; Codex CLI and Claude Code still ship on it in 2026. Track Apple's Containerization framework as the eventual Layer-5 alternative; a docs/memory note, not code.
