Engineering Audit · Functional / Logical / Technical

nablr — Audit Report

Repo my_projects/01/nablr · branch dev · v4.3.25 (pyproject, unreconciled with tag v4.6.0 — see Wave 5) · audit baseline 2026-07-28 · Re-verified against code 2026-08-27 (HEAD 7113814) — Waves 0–1 complete, 16/35 backlog done
~40k src LOC / 132 modules · ~27.5k test LOC / 121 test files · 34 MCP tools · 18 personas · 241 commits

Verdict

The product is real; the enforcement is not yet. nablr's core — 18 personas, a phase state machine, 12 scan targets built on genuine AST analysis (tree-sitter), per-story git-worktree isolation, licensing with trial flow — all exists and runs. The architecture direction is sound and several subsystems are genuinely well-crafted.

But the audit's central finding is systemic: the governance layer is largely advisory in practice. Nearly every gate that is supposed to block an LLM agent fails open, is self-certifiable, or is unreachable. A second systemic finding: nablr does not pass its own bar. Its own health scan reports 93 critical findings; 73 files exceed complexity 30 while the product enforces ≤15 on users; CI runs coverage with --cov-fail-under=0 and only on main, while the shipped protocol demands ≥85% on changed files.

None of this is fatal — the failure modes are specific, enumerable, and mostly cheap to fix. The P0 list below is 7 items.

Update, 2026-08-27: all 7 P0 items are now fixed in code (Waves 0–1, verified this pass — see the execution sequence). The systemic risk today is narrower: the long tail of 24 Wave 2–5 stories, of which 14 remain untouched — notably scanner report-path pollution and the version/claims drift are actively recurring, not just historical.

Scorecard

Functional — does it do what it claims? Partial

Core loop works end-to-end. Headline claims inflated (19→18 personas, "40+"→~17 scanners); Jira sync doc says Shipped for reverted code; free-tier entry prompt is paywalled.

Logical — is the design sound? Weak enforcement

Good state-machine model; but gates fail open, artifacts self-certify, blockers/escalations are inert, phase-skip holes exist.
2026-08-27: fail-open + phase-skip fixed; blockers/escalations still inert (Wave 2, not started).

Technical — code quality Mixed

High type coverage, real AST engines, honest debt logging — alongside god modules, 1 atomic write in the codebase, no locking, 47 broad exception handlers.

Security posture Needs work

Air-gap architecture is a genuine strength. But no path confinement on MCP args, repo-supplied test commands executed verbatim, two shipped license bypasses.
2026-08-27: path confinement + license bypasses fixed.

Tests Real suite, hollow gates

1,548 mostly-real tests incl. strong e2e. Coverage gate disabled, full-suite report fabricates counts, generated tests are boilerplate.
2026-08-27: coverage gate (75% + diff-cover) + pyright now real; full-suite counts still partly fabricated.

Docs & requirements Heavy drift

public-docs tool page 100% stale; INDEX.md misparses; version split 3 ways (4.3.25 / tag v4.4.1 / changelog 4.3.12); foreign TD docs leaked in.

What you got right

Air-gap architecture with one disclosed outbound path. License client is the sole network code; everything else is local. The self-proof suite (scripts/airgap_proof.py, security scan) is a real differentiator — worth keeping honest (see Finding 4).
Real AST analysis, not regex theater. tree-sitter for Python/TS/Markdown; Python code-quality, debt, dead-code, complexity, and test-smell engines are genuine AST walks. The test-smell detector even has a working suppression mechanism (ref:allow) with justification comments.
Per-story worktree isolation ("Airlock") works. utils/worktree.py + merge_back with conflict-safe git reset --merge; three live worktrees confirm real use.
One shared artifact resolver feeds both the phase gate and the activation gate (transitions.py:366, guard_tools.py:102) — deliberate anti-drift design. Report freshness (2h health, signed full-suite markers) is enforced.
Audited bypasses instead of silent ones: *_deferred tokens write TECH_DEBT rows; force=true logs an L2 escalation and cannot bypass impact_reviewed_*.
Validator registry design: decorator registration, dispatcher that never raises, machine-readable gap codes, self-health endpoint (validate(role="__health__")), retry counter keyed on artifact hash.
Model-citizen module: full_suite_runner.py — the codebase's only atomic write, locked signer, correct double-checked singleton, subprocess timeouts. This is the pattern the rest of the state layer should copy.
Honest self-documentation: near every noqa carries a TECH_DEBT: + story ID; TECH_DEBT.md logs real, reproduced flaws against the project itself (e.g. DEBT-1a5fe7ac, DEBT-ec74396b).
Type discipline: full annotations, Literal action unions, pydantic report models, only 18 type: ignore in ~40k LOC.
Test suite is mostly real: 91 of 121 files use tmp_path; test_e2e_sdlc_flow.py drives the actual MCP tools through all 8 phases and asserts real gate refusals; several tests are named regression locks for specific prior bugs.
Rich audit trail: 1,027 handoff-log entries plus context-graph nodes per event.
Lean dependency set: 10 runtime deps, all justified.

Finding 1 — Governance gates fail open Critical · logical

The product's promise is enforcement. Today most enforcement paths can be skipped, self-certified, or silently pass when their inputs are missing.

Crit Arbitrary phase jump. workflow(action="validate_and_advance") accepts any next_phase with no adjacency or artifact check (state_machine.py:582-695). PHASE_0 → PHASE_7 in one call. Already bit you: DEBT-ec74396b. Fixed STORY-GATE-PHASESKIP-001 (c1bf456): validate_and_advance enforces legal phase transitions.
Crit Validators fail open on missing artifacts. Any gap code ending _unresolved returns allowed=True (validators/__init__.py:240-248) — and every validator returns exactly that when it can't find its file. No artifact ⇒ no validation ⇒ pass. The strictest-looking layer is the weakest. Fixed STORY-GATE-FAILCLOSED-001 (220db52): _unresolved gaps now block with remediation text; only no_project_root stays fail-open; gate exceptions in log(event="phase") also fail closed.
Crit Validators look in the wrong directory. PM/tester/scrum-master validators hardcode docs/requirements/{id}.md, but create_story writes to active/ (hierarchy_manager.py:25-31). Every story created through the supported path silently skips validation via the fail-open above. resolve_story_file_path exists but has only 3 call sites. Fixed e09f005 + STORY-GATE-FAILCLOSED-001: validators route through the shared resolver (flat/active/done/archive), locked by both-paths parity tests; platform_engineer now reads the real report path and timestamp key.
Crit Artifact tokens are self-certifiable and unaudited. log(event="artifact") sets any token true with no whitelist, no verification — including impact_reviewed_provisional, a HARD_REQUIRED gate (misc_tools.py:158-183). And the early return at server.py:428-431 means these grants never reach the handoff log: 0 artifact events in 1,027 log lines.
High "Unbypassable" gate is auto-granted. impact_reviewed_verified is both HARD_REQUIRED and a PHASE_4 auto-signal — set true unconditionally on developer phase-close with no evidence check (phase_signals.py:44-50).
High completed_phases is a permanent bypass. Once a phase is in the list, all its artifact checks short-circuit true forever (transitions.py:479-482); nothing removes entries, and log(event="phase", outcome="failed") has zero state effect. Rework loops auto-pass health/test/full-suite gates.
High Blockers and escalation levels are inert. add_blocker appends to state; the only consumer is can_transition, which production never calls. L1/L2/L3 write identical log rows; "HALT" is display text — nothing prevents the next tool call. HALT flags live in an in-memory set, gone on restart (session.py:86-92).
High Fix-flow certifies the gate it's supposed to pass. start_fix_flow pre-grants 5 gate artifacts including impact review, omits impact_architect from its injected sequence, and permanently hijacks active.json (never restored — live state shows FIX-611BC079 as active). AGENTS.md's claim that skipping impact review "is blocked at PR review" has no enforcing code.
Med Unknown phase string = ungated close. log(event="phase", phase="implementation") (name instead of key) resolves no phase → skips produce-gates and validators, reports success (server.py:438-449).
Med Closure never completes. close_workflow sets story_closed but never advances to PHASE_7 — closed stories report in-flight forever; 55 of 77 active/ stories still say Draft for shipped work.
Med Migration track is entirely ungated. migration_planner/parity_tester/cutover_engineer exist as personas but have no PHASE_CONFIG entries → activation with zero artifact requirements.
Med On-demand agents render "✅ Workflow Complete!" for stories that aren't (next_agent.py:189-191).

Finding 2 — Personas are never delivered Critical · functional

Crit agent(action="activate") injects no persona. It returns a generic "EXECUTION CONTRACT" string and never calls any get_*_prompt() (server.py:859-962). Personas exist only as MCP prompt endpoints, which AGENTS.md never tells the client to invoke. Following your own documented protocol activates a role with zero rule content. The only path that injects a persona is fix-flow. Fixed STORY-PERSONA-INJECT-001 (8f27090, PR #10): agent(activate) now delivers the persona.
High Tech-stack detection probes the wrong directory. _detect_tech_stack() uses Path.cwd() — the MCP client's launch dir, not the project root — then strips rule sections based on it (start_agent.py:124-140). Frontend/backend rules are dropped or kept essentially at random. Fixed (same story): tech-stack detection uses the project root; orphan rule files shipped into personas.
Med Rule coverage is uneven and partly orphaned. core_protocol.md missing from 8/18 personas (incl. architect, scrum_master, PM); 3 rule files loaded by nobody — including artifact_tokens.md, the doc that explains how to satisfy every gate token.
Med Registry drift. AGENTS.md lists 15 agents; code has 18; validators cover 17 (no test_designer). Phase numbering disagrees across three sources (QA_GATE is PHASE_2 but runs after PHASE_3).

Finding 3 — Scanner correctness High · technical

Crit Severity pipeline is broken end-to-end. Debt rules stamp emoji severities ("🔴 Critical"); the health scanner maps severity.lower() against plain names → never matches → every debt finding downgrades to MEDIUM (health_scanner.py:504-510). Confirmed live: all 659 debt findings in your own report are medium, including 45 bare-excepts configured critical. Debt can never trigger auto-remediation. Fixed STORY-SEVERITY-PIPELINE-001 (45e21f5): one severity vocabulary end-to-end; emoji-vs-name mapping bug removed in health + debt scanners.
High Same metric, two verdicts. Health flags file complexity against threshold 12 with multipliers (74% of your files flagged); scan(target="complexity") uses 80/150 and reports the same files OK. Per-function complexity for Python "isn't tracked yet" (complexity.py:259) despite being the advertised metric.
High High-noise detectors. Comment-smell (always HIGH) flags prose comments; dead-code DC002 flags 220 "never imported" files that are genuinely imported (candidate paths computed relative to scan root, not package root — dead_code_scanner.py:429-468); TS ternary regex matches every optional property; docs scanner flags Markdown headings like "# Get started" as code-echo comments; arg-count rule counts self → every method "missing 1 type hint".
High Portability: house rules hardcoded into the engine. PROJECT_SOURCE_PACKAGES=("nablr",) means UUT-mock detection can never fire in any customer project; import-order treats only nablr.* as local; validators hard-require structlog imports and literal FR-\d+ tokens in arbitrary user repos; branch name dev hardcoded in developer validator.
Med Remediation engine bookkeeping is wrong. Tier-1 marks all pending violations in a touched file AUTO_FIXED; relative-path findings never update; path-remap can rewrite a same-named file from a different tree (tier1_fixer.py:78-99); no backup/dirty-check/post-fix test run; completion always logs outcome="passed".
Med Five disagreeing exclusion lists; substring matching skips any path containing build/dist/target; only 2 scanners consult the gitignore matcher.
Med Config-driven claim partly decorative. YAML ast_node/condition parsed and never consumed; config/ai_patterns.yaml never read; framework merge silently resets complexity thresholds to hardcoded values and drops exclusions (config_models.py:141-167).
Med Report path bug (reproduced in this audit). Scanning src/nablr wrote reports and TECH_DEBT into src/nablr/nablr-reports/ and src/nablr/docs/ — inside the shipped package tree. Multiple stray nablr-reports//codemap/ dirs already litter src/.

Finding 4 — Security & input validation High

Crit No path confinement on MCP tool args. scan/remediate/full_suite accept arbitrary absolute paths. A prompt-injected assistant can point the secret scanner at ~/.ssh (masked prefixes written to a report), run tier-1 fixes (file mutation) anywhere, or run full_suite — which executes npm test/gradlew/mvn/cargo in any directory, with repo-supplied nablr-config.yaml extra_args appended verbatim to argv. Fixed STORY-PATH-CONFINE-001 (6675c74, PR #5): MCP tool inputs confined to project root (utils/path_confine.py), full_suite argv hardened.
High No ID validation → path traversal. story_id et al. interpolate straight into file paths (f"{story_id}.json"); pydantic fields have no pattern=. A ../-bearing ID writes outside the state dir. Fixed (same story): IDs validated.
High Two shipped license bypasses. NABLR_DEV_MODE=1 returns paid status before any check (license_client.py:420-421); transport auth defaults to a verifier accepting every token (auth.py:108-116). Key comparison non-constant-time; paid asset cache never expires offline.
Med The user-facing security scanner is your air-gap self-proof, not an app audit. Any real app fails "network imports" by design; URL allowlist is nablr's own domains and is bypassable by a backslash in the URL; secret scan has no entropy check, suppresses lines containing "test"; dependency audit reads a literal dependencies = [ line (reported 0 deps for this very repo), has no CVE check; file audit's "subprocess" detection can never match. Report titles itself "nablr Security Audit" for any project.
Med Side effects in getters. get_nablr_state_dir() edits the user's .vscode/settings.json and .cursorignore; a health scan mutates the scanned repo (writes TECH_DEBT.md, relocates docs).
Med demo_security_scan.sh claims "no network calls" then instructs the customer to run the exact grep that disproves it (urllib in license_client.py). README's "exactly one category of outbound call" is the honest framing — use it everywhere.

Finding 5 — State integrity High · technical

High One atomic write in the whole codebase. All workflow/remediation/graph/license state is written in place; a crash truncates JSON, corruption is read as None = "no such story", and re-init dead-ends in FileExistsError → story permanently unopenable (state_machine.py:239-265, init.py:303). Worse: a corrupted story file makes the scrum-master close-out gate pass (requirements fall back to []). Fixed STORY-STATE-ATOMIC-001 (PR #4, 06ed98e): atomic write/read for state layer.
High No locking on read-modify-write. Concurrent tester/security_owner completions lose updates (parallel.py:165); story_lock exists but guards 2 of ~9 mutation sites, and its own dict insertion races.
High merge_back deletes the audit trail. Worktree removal takes .nablr_state/ and handoff_log.jsonl with it — phase history vanishes at merge with no export (merge_back.py:206).
Med Four sources of truth for active story/agent (module global, SessionManager, disk, prompt globals) held together by manual sync calls. Graph singleton pins its first root forever (DEBT-1d576de9 — cross-worktree corruption already logged).
Med Naive-vs-UTC timestamp mix breaks touched_files_since_story_init by the UTC offset; second-resolution graph node IDs collide; git subprocesses have no timeouts (a credential prompt hangs the MCP handler).
Med Orphan accumulation. 138 fix-flow session files vs 11 FIX states; orphan story JSONs; 18MB .nablr_state/; 15MB unignored sef-reports/.

Finding 6 — CI & test enforcement High

Crit CI never runs where work happens. ci.yml triggers on main only; the active branch is dev; story-branch PRs into dev match nothing. Tests first run at the batched dev→main PR, long after stories are declared Done. Fixed STORY-CI-DEV-BRANCH-001 (c452af5): push+PR triggers on [main, dev]; paths-ignore narrowed to pure prose; lint/security/test parallel with build gating on all three.
Crit Coverage gate is --cov-fail-under=0 in a job named "Test & Coverage" (ci.yml:93), against a stated bar of ≥85% on changed files. No coverage config exists anywhere; coverage.xml is produced and discarded. The 85%-on-changed-files bar is currently uncomputable (no diff-cover). Fixed CI-DEV-BRANCH-001 uploads coverage.xml every run; STORY-COVERAGE-GATE-001 (97c4670, PR #11) sets a real floor (--cov-fail-under=75, ratchet-only policy) and adds diff-cover for the changed-files bar.
High The full-suite gate never verifies a test ran. pass_summary total/passed/failed are hardcoded 0 literals (full_suite_runner.py:689-693); evaluate_gate checks signature/freshness/exit codes only. A run collecting nothing passes. The shape-lock test asserts only that the key exists.
High Generated tests are boilerplate phantoms. All 7 files in tests/generated/ are assert symbol is not None (6 of 7 for the same symbol), regenerated by design, untracked in git and invisible to CI — yet counted in local green runs and already indexed in codemap.
Med 16 tests permanently skipped against scripts/propagate.sh, which doesn't exist — the "17 skipped" your status reports record as green. Two real red tests sit in .pytest_cache/lastfailed.
Med Own test-smell scanner runs in no pipeline (not CI, not nightly, not pre-commit), and has a patch.object blind spot — the strongest UUT-mock violation in your own suite (test_brownfield_populator.py:31-75) is invisible to it.
Med pyright configured, never executed — not in pre-commit, not in CI — despite "must exit 0" being a non-negotiable in the shipped protocol. bandit --exit-zero discards MEDIUM/LOW Fixed (CI-DEV-BRANCH-001: scripts/ci_bandit_gate.py — HIGH/CRITICAL fail, MEDIUM annotated). No branch protection; provenance check is a warning, not a block.

Finding 7 — Claims vs. reality High · functional

Claim	Reality	Verdict
19 expert personas	18 (`AVAILABLE_AGENTS`); the 19th prompt is the entry point	Inflated
40+ local scanners	12 scan targets, ~17 real engines; own docs page says both "40+" and "10"	Inflated
Rules/configs "Pro only"	All 34 rule files + 11 YAMLs ship in the free wheel; free tier reads them locally	Not enforced
Free entry: "activate nablr"	`activate_nablr` prompt is behind `_require_paid()` — free users get an upsell	Broken
Jira sync "Shipped"	Reverted; doc still says Shipped; 306-LOC export module survives with zero callers	Stale
Parallel orchestration	Gate only, hardcoded 2 agents at PHASE_6; never dispatches (uncommitted epic plans the generalization)	Partial
Codemap generation	Runs, but output has 176 `[Desc]` placeholders → FEATURES.md degenerates to "2 features" for a 34-tool product	Hollow output
`scan(target="compliance")`	Advertised in docstring + AGENTS.md; not in the dispatch — invalid target; 246-LOC dead implementation	Phantom
14-day trial / licensing	Implemented properly (mint, device dedup, offline grace)	Real
Worktree isolation, browser E2E stepper, air-gap proof	Implemented (browser fixed at `bcf0fcd`; was hardcoded-success before)	Real

Docs & requirements drift (16 items)

public-docs/tools/overview.mdx: all 18 documented tool names are pre-consolidation — none exist (scan_health → scan(target="health"), activate_key → set_license_key, …). remediate, agent, workflow, validate, browser etc. publicly undocumented.
Version three-way split: pyproject 4.3.25, latest tag v4.4.1, CHANGELOG 4.3.12. Tags v4.4.0/v4.4.1 are not ancestors of any branch — published to PyPI from unreachable commits.
CHANGELOG root cause: release.toml changelog mode=update with no insertion flag and no marker in CHANGELOG.md → semantic-release silently never updates it.
INDEX.md: claims 83 stories (actual 104), 6 epics (actual 11); status buckets include **|:** 11 from the fallback regex matching legacy metadata tables (requirements_serializer.py:137); 14 stories render "(untitled)"; TD/REF/FIX docs invisible to the generator.
docs/requirements/TD-011…TD-015 belong to a different product (education app: mindmaps, tutors, EPIC-19/20) — cross-project write consistent with the stale-root bug in DEBT-1d576de9.
Done stories still in active/ (STORY-011×3, BRANCH-RULESET, CI-MINUTES); archive/ never created despite the Done story specifying it; 22 md/json companion pairs split across directories; all 21 new uncommitted docs written to flat root, ignoring the subfolder convention.
docs/requirements/README.md 7 months stale; USER_GUIDE.md references renamed tools; CHANGELOG headed "SEF Agents"; README badge says Python 3.13+ vs requires-python >=3.12; quickstart points to nablr.nablrco.com while everything else says nablr.dev; dist/ holds builds from two identities (nablr, ref_agents).
TECH_DEBT.md: no table header row; one row's unescaped | corrupts its own columns (the exact bug the row describes); manual registry severities invisible in the auto-rollup below it.

Finding 8 — Code health, by your own tooling Medium

Ran scan(target="health") and scan(target="security") on src/nablr as part of this audit:

Metric	Result	Note
Critical / High / Medium	93 / 107 / 1,638	Product enforces "block on critical" for users
Files over complexity 30 (Tier-3 "full redesign")	73	Product mandates ≤15 per function; own worst: `server.py` CC 211, 1,793 LOC
`except Exception` without re-raise	34 critical hits	Own protocol forbids it; 29 handlers are body = `pass`
Security self-scan	FAIL (network imports)	Fails on its own allowlisted license client — allowlist matching bug
Dependency audit	"0 dependencies"	False — 10 runtime deps; parser missed its own pyproject format
Report placement	Wrote into `src/nablr/docs/` & `src/nablr/nablr-reports/`	Path-resolution bug pollutes shipped package tree

Other structural items: 19 latent import cycles broken by 166 function-local imports (33 in server.py); utils/ depends upward on session/workflow/tools; a leaf tool imports nablr.server and spawns a daemon thread from inside a getter (context_graph.py:830); orchestration decisions made by regex-parsing scanner markdown (server.py:321-324, :477 — if "❌" in result); 148 logger.info calls unreachable at the configured WARNING level; dead dispatch enums duplicating the Literal unions; stale hardcoded validator_version="3.13.2"; PyYAML used on a core path but declared dev-only.

Functional gaps (claimed or implied, not present)

No CVE/vulnerability scanning in the user-facing security target (planned: EPIC-SEC-HARDENING-001 — Bandit/Semgrep + pip-audit/OSV; good plan, right priority).
No real regression analysis — scan(target="regression") is filename-keyword matching wired to no gate.
No parallel dispatch, only a 2-slot PHASE_6 gate (planned: EPIC-PARALLEL-ORCHESTRATION-001).
No test_designer validator; conceptual-test plan sets tests_written three phases before executable tests exist (misc_tools.py:301).
No Jira integration (reverted; STORY-JIRA-SYNC-002 written against deleted code).
No closure-doc generator — the scrum-master validator's primary input is hand-written; 2 closure docs exist for 94 stories.
Playwright E2E gate is a warning, not a gate (parallel.py:197-219) — contradicts the tester persona's "will be BLOCKED".
TS/JS receive no AST quality analysis — code-quality scanner runs only for Python projects; TS coverage is regex-grade.
Uncommitted epics (AI-PROVENANCE, ATTACK-SURFACE, COGNITIVE-DEBT) are sensible, honest about limits, and correctly sequenced — but note guardian-ci.mdx markets the provenance story before the git-trailer bridge exists.

Addendum — 2026 AI Blocker Report coverage cross-check External source, re-verified 2026-08-27

A separate artifact ("sef vs. the 2026 AI Blocker Report — Coverage Audit", undated, from an earlier conversation) mapped nablr against "What's Still Blocking Developers in the Age of Frontier AI Coding Assistants" (July 2026), an external published landscape report on AI-coding-assistant failure modes. It is the source document behind four pre-existing epics in this backlog: EPIC-SEC-HARDENING-001, EPIC-ATTACK-SURFACE-001, EPIC-COGNITIVE-DEBT-001, EPIC-AI-PROVENANCE-001. Re-verified against code this pass rather than re-summarized from the artifact's prose.

Result: 1 of 12 recommended fixes shipped since that artifact was written. The one that landed (override-rate tracking, STORY-074) is real and well-executed. Everything else — including the two highest-priority items, a real SAST engine and a package-existence check — is still exactly as absent as the artifact found it.

Cluster	Verdict (unchanged unless noted)	2026-08-27 status
1 · Productivity paradox	Not covered	No telemetry mechanism added.
2 · Quality & tech debt	Partial	Codemap autofill mechanism shipped (STORY-CODEMAP-AUTOFILL-001, done/) but real-world coverage is incomplete — spot-checked `codemap/src_nablr_security.md` still reads the raw placeholder `[Describe the purpose of this module]`.
3 · Security regressions	Partial	Unchanged — no Semgrep/Bandit-class engine found; `security_owner.py` still validates the LLM's own report, not the code.
4 · New attack surfaces	Not covered	Unchanged. STORY-067 (package-existence/slopsquatting) and STORY-068 (prompt-injection) both still Draft, uncommitted to `active/`. No destructive-action interception found.
5 · Review bottleneck	Partial	Unchanged — no GitHub/GitLab merge-API integration found in `pr_reviewer.py`.
6 · Trust & governance	Partial	Unchanged — no AI-authorship git trailer found (would also unlock cluster 2's diff-aware-scanning recommendation).
7 · Cognitive / skill debt	Partial — strengthened	Real progress. STORY-074 shipped `utils/override_rate_report.py`: real force-bypass/escalation rates computed from `handoff_log.jsonl`, wired into the executive-summary report, 18+2 real tests, 96% coverage. The Socratic-interrogation validator still has no teeth — `specifier.py`'s verdict logic doesn't block on a missing "Open Questions" section.
8 · Legal / IP / licensing	Partial	Unchanged — `dependency_audit.py`'s `SAFE_LICENSES` check is still Python-only; no JS/TS license rules.

All 12 recommendations, verified individually

#	Recommendation	Status
1	Real SAST engine behind the security gate	Not started
2	Package-existence check (PyPI/npm) for new deps	Not started — STORY-067 Draft
3	Destructive-action interception at runtime	Not started
4	AI-authorship tagging (commit trailer)	Not started — same gap as this report's own EPIC-AI-PROVENANCE-001, STORY-069/070
5	Real PR/merge-API integration for pr_reviewer	Not started
6	npm/JS license scanning at parity with Python	Not started
7	Prompt-injection detection on ingested content	Not started — STORY-068 Draft
8	Diff-aware scanning for AI-generated code	Not started — blocked on #4
9	Real codemap summarization (LLM-filled purpose)	Partial — mechanism shipped, coverage incomplete
10	Opt-in productivity telemetry	Not started
11	Override-rate tracking (automation-complacency signal)	Done — STORY-074, `override_rate_report.py`
12	Validator teeth for Socratic-interrogation persona	Not started

Where this fits the rest of the backlog: this cross-check is a distinct axis from Findings 1–8 above — those are about whether nablr's own governance mechanisms work as designed; this is about whether nablr's scope covers what an external, independently-sourced landscape report says actually blocks AI-assisted teams. The two overlap once (cluster 3/Finding 4's security-scanner-is-self-proof point), otherwise they're additive. Recommendations 1, 2, 4, 5, 7 are the highest-leverage additions to the Wave 5+ backlog once Waves 0–2 land, since none require the gate-integrity work to land first — they're new scanning/integration surface, not enforcement-of-existing-scope.

Addendum (2026-07-28) — Token efficiency & First-Pass Yield Post-audit analysis

Question examined: how efficient is nablr in token terms — measured not as raw count but as outcome per token. Answer: token cost is moderate and bounded; token efficiency is poor, and the waste is caused by reliability failures, not verbose prompts. Fixing the Finding 1 gates saves more tokens than any prompt trimming.

Metric: First-Pass Yield (FPY)

North star: FPY — % of phases (or stories) that pass their gate on the first attempt with no retry, no rework loop, no fix-flow traced back later. Efficiency = FPY ÷ cost (nablr-controlled tokens + cycle time); concretely, cost-per-first-pass-phase. Token figures are an honest lower-bound proxy — an MCP server cannot see client-side reasoning tokens.

Cost side (measured, chars/4)

Surface	Cost	Note
AGENTS.md (always-on)	~3.5k tok	Reasonable
Persona payloads	2.0k–8.9k (developer max, mean ~5k)	Lean — but currently ~0 delivered (Finding 2)
Rule library	49.2k across 34 files	Loaded per-role only — good design
Health report / TECH_DEBT / INDEX	3.4k / 7.1k / 3.9k	Fine
`scan(debt, summary=false)`	Unbounded (659 rows)	Worst single offender

A full 10-phase story ≈ 30–60k nablr-controlled tokens once personas are actually delivered — the intended, acceptable price of governance.

Outcome side (day-zero baseline from live logs)

137 fix-flow sessions vs ~30 stories — ~4+ rework interventions per story; each burns a full forensic→strategist→developer token cycle.
0 of 10 in-flight stories closed; 10/10 stuck ≥3 days (several at PHASE_6/PHASE_7 — the closure defect, live).
Historical FPY uncomputable: phase outcome field is free text (34 non-enum values incl. multi-paragraph notes); validator retry counts die on restart.
339 escalations (L1 323 / L2 9 / L3 7, ~10 force overrides) — all inert today.

Synthesis: tokens-per-completed-story is effectively unbounded — the completions denominator is zero in current state. The leak is rework multiplying how many times the same tokens get spent. Efficiency-ordered levers: (1) Wave 0–1 gate fixes — the biggest token saver; (2) deliver personas on activation; (3) cap scan(debt) output; (4) prompt trimming — smallest lever.

Tracked by STORY-FPY-METRICS-001 (EPIC-TEAM-SCALE-001, commit 2c5e160; Jira NABLR-5 under epic NABLR-4): enum outcomes + persisted retries, per-payload token ledger, generate_efficiency_report → nablr-reports/efficiency/, today's numbers committed as the day-zero baseline.

Prioritized recommendations

P0 — the product's promise depends on these

Fail closed. *_unresolved gaps must block, not pass (validators/__init__.py:240-248); route all story-path lookups through resolve_story_file_path.
Close the phase-skip hole. validate_and_advance gets adjacency + artifact checks; remove or gate log(event="artifact") self-certification behind a verified whitelist, and log every grant.
Fix the severity mapping (emoji → enum) so critical debt is critical again — one small function, restores the entire remediation trigger chain.
Confine paths. Validate IDs (pattern= on pydantic fields), resolve every directory/file_path arg against the project root, refuse escapes; stop honoring repo-supplied extra_args without an allowlist.
Turn CI on: run on dev + PRs, set a real --cov-fail-under, add diff-cover for the 85%-changed-files bar, add pyright, wire the test-smell scanner in, and parse real pytest counts into pass_summary (require total > 0 in evaluate_gate).
Make state writes atomic + locked — copy the full_suite_runner pattern (tmp + os.replace, lock) into state_machine._save_state and friends; treat JSONDecodeError as corruption, not absence.
Deliver personas on activation — agent(activate) should return the persona prompt (or AGENTS.md should mandate the prompt endpoint); fix _detect_tech_stack to use the session root.

Code-verified 2026-08-27: all 7 done. #1 fail-closed Done — _unresolved gaps block (validators/__init__.py:286-299, "FR-1/FR-9") · #2 phase-skip Done — validate_and_advance now has an adjacency check with a logged override escape hatch (state_machine.py:856-871) · #3 severity mapping Done — emoji-vs-enum bug fixed at both call sites (health_scanner.py:515, :620) · #4 path confinement Done — new nablr.utils.path_confine.resolve_within_root, wired into server.py's directory/file_path args (server.py:55,388,410) · #5 CI Done — runs on [main, dev], pyright in the pipeline, real --cov-fail-under=75 + a separate 85% diff-cover step (ci.yml:9,46-49,118-141) · #6 atomic state writes Done — _save_state uses tmp+os.replace with a .bak recovery copy (state_machine.py:393-408), extracted into a shared atomic_write_json helper reused by the full-suite report writer too · #7 persona delivery Done — _detect_tech_stack now resolves via find_project_root(), not cwd() (start_agent.py:245-250). One residual: full_suite's pass_summary.total/passed/failed are still hardcoded 0/0/0 (tools/runner/report.py) — see Wave 2, FULLSUITE-COUNTS-001, still partial.

P1 — trust and hygiene

Export .nablr_state/ + handoff log before merge_back removes the worktree; restore the active-story pointer after fix-flow.
Make blockers/L3 actually halt (persist HALT flags; check blockers in suggest_next_agent); add revert_phase so failed phases can re-gate.
Fix the claims: 18 personas, honest scanner count, unpaywall activate_nablr (it's your funnel), delete or ship jira_confluence_export, remove the phantom compliance target, fix demo_security_scan.sh.
Rewrite public-docs/tools/overview.mdx against the real tool surface; fix INDEX serializer regexes; repair the release chain (changelog insertion marker; re-point tags at reachable commits).
Decompose server.py (registration vs orchestration vs watcher lifecycle) and give scanners a structured result type — stop regex-parsing your own markdown for control flow.
Remove the tracked/stray artifacts from the wheel (src/nablr/codemap/, stray report dirs); gitignore sef-reports/ or delete it.
Gate the migration personas (add M-phase PHASE_CONFIG entries) and add the test_designer validator.
Delete the foreign TD-011…015 docs; sweep active/ statuses (the close protocol works — Finding 1's closure bug is why they're stale).

P2 — polish

False-positive pass on comment-smell, DC002, TS ternary regex, arg-count self/cls; unify the five exclusion lists behind IgnorePatternMatcher.
Parameterize house rules (source packages, branch names, structlog/FR checks) so scanners work on customer repos.
Fill codemap purposes (planned STORY-072) so FEATURES.md stops reporting 2 features; regenerate AGENTS.md from code.
Timestamps → UTC-aware everywhere; git subprocess timeouts; graph node IDs with microseconds; expire the offline paid-asset cache.
Reduce the worst complexity offenders behind the P0 fixes (server.py, health_scanner.py, transitions.py resolve_artifact → token-resolver registry).
Ship artifact_tokens.md into the personas that need it — it's the manual for your own gates and nobody reads it.

Execution sequence — 35 stories in 6 waves

Principle: fix the tools you'll use to build everything else first, then correctness, then polish. Backlog committed as 5 new epics + stories linked into 7 existing epics (a8de8d4, 2c5e160).

Wave	Theme	Stories & verified status (2026-08-27)
0	Meta-blockers	WORKTREE-VENV-001 Done · GATE-FAILCLOSED-001 Done · CI-DEV-BRANCH-001 Done
1	P0 correctness	SEVERITY-PIPELINE-001 Done · GATE-PHASESKIP-001 Done · STATE-ATOMIC-001 Done · PATH-CONFINE-001 Done · COVERAGE-GATE-001 Done · PERSONA-INJECT-001 Done
2	Gate + state hardening	GATE-TOKEN-AUDIT-001 Done · FIXFLOW-INTEGRITY-001 Done (extended further by STORY-FIXFLOW-PERSIST-001, -REVIEW-PARITY-001, -WORKTREE-ISOLATION-001, all done/) · STATE-LOCKING-001 Done · CODEMAP-DEDUPE-001 Done (added to the backlog after the original audit) · FULLSUITE-COUNTS-001 Partial — wrong-tree provenance + per-kind status/exit-codes now real (tools/runner/gate.py); `pass_summary.total/passed/failed` still hardcoded `0` (tools/runner/report.py) · MERGEBACK-STATE-EXPORT-001 Not started — a related but distinct story, STORY-MERGE-BACK-ISOLATION-001 (done/), stops merge_back from mutating main_root's live tree mid-pipeline; it does not export worktree state/handoff-log before removal, which is what this story asks · GATE-PHASE-REVERT-001 Not started · GATE-BLOCKERS-001 Partial (corrected 2026-08-27) — `validate_and_advance` now blocks on open blockers and session HALT via `TransitionValidator.can_transition()`, landed as a byproduct of GATE-PHASESKIP-001; `workflow(next)` still doesn't check either. `resolve_blocker_tool` now bounds-checks the index cleanly (no more uncaught `IndexError`) but still isn't unified with `WorkflowManager.resolve_blocker`. `EscalationLevel` still unvalidated; HALT still doesn't persist across restart
3	Scanner correctness	SCAN-REPORT-PATHS-001 Not started — reproduced live: `src/nablr/nablr-reports/`, `src/nablr/tools/nablr-reports/` pollution is back on disk right now · REMEDIATION-BOOKKEEPING-001 Not started · SCANNER-FP-001 Not started · THRESHOLD-COHERENCE-001 Not started — confirmed: health_scanner's legacy `thresholds.complexity` and complexity.py's `file_complexity_warn` are still two separate systems · SCANNER-PORTABILITY-001 Partial — a prep module (`utils/source_packages.py`) now derives project packages instead of hardcoding `("nablr",)`, but per its own docstring it's consumed only by the Wave-0 wrong-tree guard so far, not the scanners · TESTSMELL-CI-001 Not started — no workflow file references test-smell scanning
4	Workflow completion + telemetry	CLOSE-PHASE7-001 Not started — confirmed: no `PHASE_7`/`update_phase` call anywhere in `fix_flow_close.py` (this function moved out of `fix_flow.py` since the story was written — re-verified on its current location 2026-08-27) · FPY-METRICS-001 Done (extended by STORY-DEBT-DELTA-REPORT-001, -DEBT-RATCHET-001, -FIX-METRICS-INTEGRITY-001, -FIX-LEDGER-DOUBLECOUNT-001, all done/ — the efficiency-report arc got real follow-on investment) · VALIDATOR-TESTDESIGNER-001 Not started · SESSION-COHERENCE-001 Addressed — superseded in substance by the STATELESS-CONTEXT-001/002/003 arc (all done/), which eliminates the same SessionManager mutable-singleton / active-story process-globals this story targeted · REQDOCS-SERIALIZER-001 Not started · REQDOC-LIFECYCLE-001 Not started (DEBT-a19ced2a still Open in TECH_DEBT.md) · MIGRATION-GATES-001 Not started — explicitly named as pending in a code comment (validators/__init__.py:376)
5	Truth & release	LICENSE-INTEGRITY-001 Done · RELEASE-CHAIN-001 Not started — regressed — version split is now wider: `pyproject.toml` still 4.3.25 against latest tag `v4.6.0` (3 releases behind), CHANGELOG.md still headed "SEF Agents" · FREE-ENTRY-001 Not started — `activate_nablr_prompt()` still calls `_require_paid("activate_nablr")` first thing (server_tools/prompts_impl.py:66-69) · DOCS-CLAIMS-001 Not started — README.md still reads "19 expert personas" / "40+ local code scanners" · PUBLICDOCS-TOOLS-001 Not started — 3 pre-consolidation tool names still in `public-docs/tools/overview.mdx` · SCAN-RESULTS-STRUCTURED-001 Partial — `full_suite_runner.py` was split into a typed `tools/runner/` package (`GateDecision` dataclass, structured report dict) via the unrelated STORY-COMPLEXITY-DEBT-001/STORY-LOC-DEBT-001 split — real progress, but scoped to full_suite only, not the general `scan()` dispatcher this story targets

Reading this table: Waves 0–1 (P0, the product's core promise) are fully shipped — genuine, verified progress in one month. Waves 2–5 are the long tail: 7 done, 4 partial, 13 untouched. The Not started — regressed/reproduced-live items (SCAN-REPORT-PATHS-001, RELEASE-CHAIN-001) are worth prioritizing next — both are cheap fixes for defects that are actively recurring, not just theoretical.

Concurrent-safe worktree batches File-footprint analysis

Method: for each not-yet-started or partial story, the file set it will touch was pulled from the acceptance criteria authored for it (this backlog), cross-checked against current code where files have since moved (e.g. full_suite_runner.py's split into tools/runner/report.py + tools/runner/gate.py). Two stories go in the same concurrent batch only if their footprints share zero files — a squash-merge of disjoint-file branches cannot conflict, regardless of story order.

Result: 11 of the 19 remaining stories are mutually disjoint — they can run in 11 separate worktrees right now with no merge-back risk between them. The other 8 cluster around a handful of hub files (server.py, workflow/state_machine.py, tools/workflow/merge_back.py, tools/validators/__init__.py) and need to run one at a time, in the order below.

Caveat: this is git-mechanical safety, not semantic independence. Two disjoint-file stories can still be logically coupled (e.g. GATE-PHASE-REVERT-001 and GATE-BLOCKERS-001 both extend the blocker/gate model even though only one shares a file with anything below). And a story's real diff can grow past its planned footprint during implementation — run git diff --name-only against the target branch before merge-back as a final check, especially for anything not marked high confidence below.

Verification note (2026-08-27): all 57 file paths below were confirmed to exist against the current tree; the specific functions/symbols each story targets were spot-checked for >70% of the footprint (state_machine.py, merge_back.py, hierarchy_manager.py, validators/__init__.py, tier1_fixer.py, compliance_remediation.py, comment_smell_scanner.py, dead_code_scanner.py, debt_ast_rules.py, test_smell_ast.py, both find_project_root duplicates, release.toml). One drift found and corrected: CLOSE-PHASE7-001's target, close_workflow, moved out of fix_flow.py (now split down to one function) into a new fix_flow_close.py since this story was originally scoped — its underlying finding (no PHASE_7 advance) still holds, re-verified on the correct file. Given the volume of concurrent work landing in this repo, re-run a footprint check like this before actually starting any story below, rather than trusting this table indefinitely.

Second pass (2026-08-27, same day): extended the spot-check to the remaining ~30 unverified symbols across all 19 stories. Two more real findings: GATE-BLOCKERS-001 is actually Partial, not Not-started — validate_and_advance picked up a working blockers/HALT check as an unplanned side effect of GATE-PHASESKIP-001's implementation (both modify the same can_transition() call path); and this story's own originally-authored footprint had a path error (workflow/phase_signals.py, not tools/workflow/). Two claims that looked like they might have been fixed turned out to be false alarms on closer read (config_models.merge_framework still drops exclusions and resets complexity-threshold fields to hardcoded defaults; complexity.py's ternary regex is still the exact r"\?\s*.*\s*:" originally cited). Every other footprint and verdict in this report held.

Read this section top to bottom — each step below assumes every step above it has already merged. (An earlier version of this table put the prerequisite step after the batch that depends on it. Fixed 2026-08-27; if you started from that ordering, stop and re-read from Step 1.)

Step 1 — run first, alone

Story	Footprint	Why first, why alone
`SCAN-REPORT-PATHS-001`	`utils/git_utils.py`, `tools/context_tools.py`, `tools/report_utils.py`, `tools/health_scanner.py`, `tools/debt_ast_rules.py`, `tools/dead_code_scanner.py`, `tools/docs_scanner.py`, `tools/complexity.py`, `tools/comment_smell_scanner.py`, `tools/code_quality_scanner.py`, `utils/ignore_matcher.py`, `tools/guard_tools.py`	Conflicts with two Step 2 stories (`SCANNER-FP-001`: 5 shared scanner files; `MIGRATION-GATES-001`: `guard_tools.py`). It's also the path-normalization convention the Step 2/3 scanner stories should build on — landing it first avoids rebasing their diffs onto a moving target.

Step 2 — once Step 1 merges, run these 11 concurrently (zero shared files)

Story	Footprint
`MERGEBACK-STATE-EXPORT-001`	`tools/workflow/merge_back.py`
`FULLSUITE-COUNTS-001`	`tools/runner/report.py`, `tools/runner/gate.py`, `tests/test_full_suite_runner.py`
`GATE-PHASE-REVERT-001`	`workflow/state_machine.py`, `workflow/transitions.py`, `server.py`, `workflow/phase_signals.py` (corrected 2026-08-27 — no `tools/` prefix)
`REMEDIATION-BOOKKEEPING-001`	`tools/tier1_fixer.py`, `tools/compliance_remediation.py`
`SCANNER-FP-001`	`tools/comment_smell_scanner.py`, `tools/dead_code_scanner.py`, `tools/docs_scanner.py`, `tools/debt_ast_rules.py`, `tools/complexity.py`, `tools/codemap_freshness.py`, `tools/invest_scorer.py`
`TESTSMELL-CI-001`	`.github/workflows/*.yml`, `tools/test_smell_ast.py`, `tests/test_brownfield_populator.py`, `rules/quality/testing_standards.md`
`REQDOCS-SERIALIZER-001`	`tools/requirements_serializer.py`, `tools/hierarchy_manager.py`, `tools/index_generator.py`, `docs/requirements/README.md`
`MIGRATION-GATES-001`	`constants.py`, `tools/guard_tools.py`, `tools/validators/__init__.py`
`RELEASE-CHAIN-001`	`release.toml`, `CHANGELOG.md`, `pyproject.toml`, `PUBLISH.md`, `core/validation_models.py`
`FREE-ENTRY-001`	`server_tools/prompts_impl.py`, root `README.md`, `public-docs/quickstart.mdx`, `public-docs/license.mdx`
`PUBLICDOCS-TOOLS-001`	`public-docs/tools/overview.mdx`, `docs/USER_GUIDE.md`, `docs/USP.md`

Note the two same-named-looking files that are not a clash: REQDOCS-SERIALIZER-001 touches docs/requirements/README.md, while FREE-ENTRY-001 touches the root README.md — genuinely different files.

Step 3 — once Step 2 merges, run each of these once its dependency clears

Story	Footprint	Waits on (shared file)
`THRESHOLD-COHERENCE-001`	`tools/health_scanner.py`, `tools/complexity.py`, `tools/complexity_ast.py`, `core/config_models.py`	`SCAN-REPORT-PATHS-001` (`health_scanner.py`) + `SCANNER-FP-001` (`complexity.py`)
`SCANNER-PORTABILITY-001`	`utils/source_packages.py`, `tools/test_smell_ast.py`, `tools/code_quality_scanner.py`, `validators/developer.py`	`SCAN-REPORT-PATHS-001` (`code_quality_scanner.py`) + `TESTSMELL-CI-001` (`test_smell_ast.py`)
`VALIDATOR-TESTDESIGNER-001`	new `tools/validators/test_designer.py`, `tools/validators/__init__.py`, `constants.py`, `tools/workflow/misc_tools.py`, `tools/workflow/parallel.py`	`MIGRATION-GATES-001` (`validators/__init__.py`, `constants.py`)
`DOCS-CLAIMS-001`	root `README.md`, `docs/USP.md`, `demo_security_scan.sh`, `docs/features/JIRA_SYNC.md`, `CHANGELOG.md`, `tool_names.py`, `server.py`, `tools/jira_confluence_export.py`	`RELEASE-CHAIN-001` (`CHANGELOG.md`) + `FREE-ENTRY-001` (`README.md`) + `PUBLICDOCS-TOOLS-001` (`USP.md`) + `GATE-PHASE-REVERT-001` (`server.py`)

THRESHOLD-COHERENCE-001 and SCANNER-PORTABILITY-001 (the first two rows above) can run concurrently with each other — they don't share a file with one another, only with things in Step 1/2.

Step 4 — sequential only, one at a time: the state-machine / merge_back hub

All three touch workflow/state_machine.py and/or tools/workflow/merge_back.py, which Step 2's GATE-PHASE-REVERT-001 and MERGEBACK-STATE-EXPORT-001 also touch. Run one at a time, after Step 2:

Order	Story	Footprint
1	`GATE-BLOCKERS-001` Partial already	`workflow/state_machine.py`, `workflow/transitions.py`, `tools/workflow/next_agent.py` (blockers/HALT still unchecked here), `tools/workflow/misc_tools.py`, `server.py`, `session.py` (HALT persistence)
2	`CLOSE-PHASE7-001`	`tools/workflow/fix_flow_close.py` (corrected 2026-08-27 — was `fix_flow.py` when this story was written; that file has since been split down to a single function, `close_workflow` now lives here), `workflow/state_machine.py`, `tools/workflow/merge_back.py`
3	`REQDOC-LIFECYCLE-001`	`tools/workflow/init.py`, `tools/hierarchy_manager.py`, `workflow/state_machine.py`, `tools/workflow/merge_back.py`, `.gitattributes`

Not scheduled here: SCAN-RESULTS-STRUCTURED-001's remaining scope (touches server.py plus, plausibly, several scanner return-value shapes) has the least-certain footprint of anything in the backlog — recommend re-mapping its actual diff once Batch 1 and the scanner cluster have landed, rather than pre-assigning it a slot now.

Method & caveats

Five parallel deep-read audits (architecture/core, scanners/validators, workflow engine/personas, test suite/CI, docs-vs-claims) over the full source tree, plus dogfooding nablr's own MCP scan(health) and scan(security) against src/nablr. Read-only — no code changed; the auto-started remediation session REM-8332E323 was left untouched.

Caveats: dev HEAD moved during the audit (e1a21d3 → bcf0fcd, a concurrent session's browser-stepper fix — findings reflect bcf0fcd where noted). Three story worktrees are live under .nablr-worktrees/. Line numbers are accurate as of scan time and may drift. Dogfood scan wrote reports into src/nablr/… (itself Finding 3's path bug) — those files are untracked and safe to delete.