# Handler: Phase 2 -- DEEP PLAN

## Purpose

Dispatch N parallel agents (one per sub-spec) to deep-explore the codebase and write detailed, implementation-ready plans. Each agent enriches its sub-spec with architectural context, ordered tasks, and exports/imports declarations. This is where plan quality is built -- implementation quality depends on it.

## Prerequisites

- Phase 1 (DECOMPOSE) is complete.
- Sub-spec directories exist at `.ai-engineering/runtime/autopilot/sub-NNN/` with `spec.md` (Scope + `files:` frontmatter) and `plan.md` (Plan placeholder).
- Manifest exists at `.ai-engineering/runtime/autopilot/manifest.md` with all sub-spec statuses set to `planning`.
- Parent spec is available at `.ai-engineering/specs/spec.md`.
- Decision store is loaded from `state/decision-store.json`.

## Procedure

### Step 0 — Stack context resolution (spec-139 M3)

Read `.ai-engineering/manifest.yml` ONCE here. Compute resolved stack
context (stacks list + test/format/lint commands per stack) via
`ai_engineering.autopilot.stack_context.resolve_stack_context()`. Write
the JSON to `.ai-engineering/runtime/autopilot/<active>/stack-context.json`.
Every subsequent dispatch prompt MUST include `STACK_CONTEXT=<contents>`
as a top-level variable. Agents read STACK_CONTEXT from the dispatch
prompt — they do NOT re-read manifest.yml.

Minimal invocation (Python; runs once per autopilot session):

```bash
.venv/bin/python -c "from ai_engineering.autopilot.stack_context import \
  resolve_stack_context, write_stack_context; \
  ctx = resolve_stack_context(); \
  write_stack_context(ctx, active='<active>')"
```

`<active>` is the autopilot session id (typically the parent spec slug
or `manifest.md` directory name). The resolver is fail-open: if the
manifest is missing or malformed, the JSON carries `"degraded": true`
and empty command tables; downstream agents still receive the variable
and degrade their own validation accordingly.

The serialised JSON is then embedded verbatim into every Phase 2 /
Phase 4 / Phase 5 dispatch prompt under the `STACK_CONTEXT=` key.
Dispatchers MUST NOT re-resolve from disk on a per-agent basis — that
is the N-reads-per-run regression spec-139 M3 closes.

### Step 1: Load Sub-Specs

1. Glob `.ai-engineering/runtime/autopilot/sub-*/spec.md`. Collect the full list of sub-spec directories.
2. For each `sub-NNN/spec.md`, extract:
   - `id` from frontmatter (e.g., `sub-001`)
   - `title` from frontmatter
   - `files:` list from frontmatter (Phase 1 best-guess file list)
   - `## Scope` section content
3. Verify all sub-specs are in `planning` status in the manifest. Skip any that are not.
4. Build a dispatch list: one entry per sub-spec with its content and metadata.

### Step 2: Dispatch the Explore and Plan Agents Per Sub-Spec

#### Concurrency cap (spec-139 M1)

Before dispatch, read the wave concurrency cap:

1. Read `AIENG_MAX_WAVE_AGENTS` from the environment (positive integer; clamped to `[1, 64]`).
2. If unset, read `performance.concurrency.max_wave_agents` from `.ai-engineering/manifest.yml`. Value `"auto"` (default) defers to the framework's host-capacity auto-tune; positive integers override.
3. If both unset, the framework derives `cap` from host capacity per D-139-01: floor `2`, ceiling `6`, dropping to `1` if the host probe reports memory pressure ≥ 50 %.

Dispatch in batches of `cap` agents. Await each batch before dispatching the next. After every batch boundary emit a framework event:

```
event: wave_dispatch_batched
phase: deep-plan
batch_index: K (zero-based)
batch_size: <number dispatched in this batch>
cap: <resolved cap>
total_sub_specs: N
```

The cap is consulted by the dispatching agent at runtime — read the env var first, then the manifest, then fall back to the framework default. The skill handler is prose; the cap value is not embedded here.

Dispatch agents within a batch in parallel (parallel in Claude Code, sequential in other IDEs). Each agent receives a self-contained prompt with:

- The full sub-spec content (frontmatter + scope)
- The parent spec's relevant section for this concern
- Decision-store constraints that apply to this sub-spec's scope
- Stack context file paths from the `context_paths` list resolved in Phase 0. Pass paths only — agents read on demand if they need stack guidance: `"Stack guidance available at: [context_paths]. Read these files if you need language/framework conventions for planning decisions."`
- `STACK_CONTEXT=<JSON>` — the resolved stack-context blob from Step 0 (`.ai-engineering/runtime/autopilot/<active>/stack-context.json`). Embed the JSON verbatim. Agents read STACK_CONTEXT from the dispatch prompt and MUST NOT re-read `manifest.yml` from disk (spec-139 M3).

Each agent executes the following five-step procedure:

#### 2a. Deep Explore

Read every file mentioned in the sub-spec's `files:` frontmatter. For each file:

- Summarize current state: key exports, public API surface, line count.
- Map imports: what does this file import, and who imports it.
- Map callers: trace usages from other modules.
- Identify test fixtures that cover this file.

Then read analogous implementations in the codebase -- files that solve a similar problem or follow the pattern this sub-spec should replicate.

#### 2b. Write Exploration Section

Populate `## Exploration` in `sub-NNN/spec.md` with the following required subsections:

```markdown
## Exploration

### Existing Files
[For each file in scope: path, summary of current state, key exports, line count.
If file does not exist yet, note "NEW FILE" with the module it belongs to.]

### Patterns to Follow
[Identify the exemplar file in the codebase that this sub-spec should replicate.
State what pattern to follow and why. Include file path and key structural elements.]

### Dependencies Map
[Import chains relevant to this sub-spec. What this code imports, what imports it.
External dependencies. Shared utilities.]

### Risks
[Discovered constraints, edge cases, or unknowns. Breaking changes to existing callers.
Assumptions that need validation. Missing test coverage.]
```

"Existing Files" and "Patterns to Follow" are mandatory. "Dependencies Map" and "Risks" are mandatory but may be brief if the sub-spec is self-contained.

#### 2c. Write Plan Section

Populate `## Plan` in `sub-NNN/plan.md` with ordered tasks using simplified checkbox format:

```markdown
## Plan

exports: [list of modules/classes/functions this sub-spec creates or exposes]
imports: [list of modules/classes/functions this sub-spec expects from other sub-specs]

- [ ] T-N.1: [Task title]
  - **Files**: [list of file paths]
  - **Done**: [verifiable condition]
- [ ] T-N.2: [Task title]
  - **Files**: [list of file paths]
  - **Done**: [verifiable condition]
```

Requirements for the plan:
- Minimum 2 tasks per sub-spec. No upper limit.
- Every task MUST use the canonical `- [ ] T-N.K:` checkbox prefix. Bullets without brackets (`- **T-N.K** —`), section headers (`### Task N`), and placeholder bodies (`[EMPTY — populated by Phase 2]`) all count as ZERO canonical tasks and will fail the gate (see Step 2f).
- Every task MUST have explicit file paths and a verifiable done condition.
- TDD pairs: where tests are needed, the test task precedes the implementation task.
- `exports:` declares modules, classes, or functions this sub-spec creates that other sub-specs may consume. Phase 3 uses these for DAG construction.
- `imports:` declares what this sub-spec expects from other sub-specs. Can be empty (`imports: []`) if there are no cross-sub-spec dependencies.
- Task IDs use the pattern `T-N.K` where N is the sub-spec number and K is the task sequence (e.g., T-3.1, T-3.2 for sub-003).

After writing the plan, run:

```bash
python .ai-engineering/scripts/plan_tasks.py validate .ai-engineering/runtime/autopilot/sub-NNN/plan.md
```

The script syncs the frontmatter `total` / `completed` with the real checkbox count and exits non-zero if fewer than two canonical `- [ ]` tasks are present. Non-zero exit means the plan is malformed — rewrite it before proceeding. Treat the script's exit code as the authoritative Step 2f gate.

#### 2d. Refine File List

Update `sub-NNN/spec.md` frontmatter `files:` field with the actual files discovered during exploration. This replaces Phase 1's best-guess list with a verified, complete list. Add files that were discovered during exploration but not in the original list. Remove files that turned out to be irrelevant.

#### 2e. Self-Assess

Append a brief assessment to `sub-NNN/plan.md` (after Plan, before Self-Report placeholder):

```markdown
### Confidence
- **Level**: high | medium | low
- **Assumptions**: [List any assumptions made during exploration]
- **Unknowns**: [List anything that could not be determined from the codebase]
```

Confidence criteria:
- **high**: all files read, patterns clear, no ambiguity in scope, done conditions are objective.
- **medium**: most files read, some assumptions about behavior or API contracts, minor unknowns.
- **low**: key files missing or inaccessible, significant assumptions, scope may need revision.

### Step 3: Collect Results

Wait for all dispatched agents to complete. For each agent:

1. Verify the sub-spec file was written (non-empty).
2. Read the enriched sub-spec content.
3. Record the outcome: success, timeout, crash, or empty output.

### Step 4: Validate Gate Criteria

For each sub-spec that completed successfully, validate the gate (see Gate section below). Track pass/fail per sub-spec.

If a sub-spec fails gate validation despite the agent completing (e.g., missing required subsections, fewer than 2 tasks, no exports/imports declaration), treat it as a failed agent and proceed to failure handling.

### Step 5: Update Manifest

For each sub-spec:

- If gate passed: update status in manifest to `planned`.
- If gate failed or agent failed: follow Failure Modes procedure. After retries, mark surviving failures as `plan-failed`.

Write a summary line to the manifest:

```markdown
## Deep Plan Summary
- Planned: N of M sub-specs
- Failed: K sub-specs [list IDs]
- Confidence distribution: X high, Y medium, Z low
```

## Output

- N enriched sub-spec directories at `.ai-engineering/runtime/autopilot/sub-NNN/`:
  - `spec.md` containing populated `## Exploration` and updated `files:` frontmatter reflecting actual discovered files.
  - `plan.md` containing `## Plan` with checkbox-formatted tasks, `exports:`/`imports:` declarations, and confidence assessment.
- Updated manifest with `planned` or `plan-failed` statuses.
- Summary report: N planned, M failed, confidence distribution.

## Gate

All of the following must pass for each sub-spec to be marked `planned`:

1. **Exploration completeness**: `sub-NNN/spec.md` contains `## Exploration` that is non-empty and includes at least the "Existing Files" and "Patterns to Follow" subsections with substantive content (not placeholders or TODOs).
2. **Plan minimum tasks**: `python .ai-engineering/scripts/plan_tasks.py validate sub-NNN/plan.md` exits zero. The script enforces at least two canonical `- [ ] T-N.K` items and syncs the frontmatter `total` / `completed` as a side effect, so the gate cannot pass while the frontmatter is drifted or invented.
3. **Dependency declarations**: `sub-NNN/plan.md` declares both `exports:` and `imports:` in the Plan section. Either can be an empty list (`[]`) if there are no cross-sub-spec dependencies, but the declaration must be present.

## Failure Modes

### Single Agent Failure (timeout, crash, empty output, gate fail)

1. **First attempt fails**: retry once with a narrower prompt. The retry prompt:
   - Reduces scope to the sub-spec's `## Scope` section only (drops parent spec context).
   - Explicitly names the missing sections: "You must produce ## Exploration with 'Existing Files' and 'Patterns to Follow' subsections, and ## Plan with at least 2 tasks containing file paths and done conditions."
   - Keeps `files:` frontmatter and decision-store constraints.
2. **Second attempt fails**: mark the sub-spec as `plan-failed` in manifest. Do not retry again.

### All Agents Fail

If every sub-spec is `plan-failed` after retries: **halt the pipeline**. Report:

```
HALT: Deep planning failed for all sub-specs.
None of the N agents produced a valid plan.
Pipeline cannot proceed to Phase 3 (Orchestrate).
Review the parent spec for clarity and scope issues.
```

### Partial Failure (some succeed, some fail)

The orchestrator evaluates the failed sub-specs against the parent spec:

1. Read the parent spec's requirements.
2. For each `plan-failed` sub-spec, determine if its scope covers critical or optional requirements.
3. **If failed sub-specs cover critical parent spec requirements**: halt the pipeline. Report which critical scope is unplannable and why.
4. **If failed sub-specs cover only optional or low-priority scope**: proceed to Phase 3 with the successful subset. Report the gap:

```
PARTIAL: N of M sub-specs planned successfully.
Proceeding without: [list of plan-failed sub-spec IDs and titles].
Gap: [description of scope not covered].
```

### Criticality Assessment Criteria

A sub-spec is **critical** if any of the following apply:
- It implements a requirement marked as mandatory or blocking in the parent spec.
- Other sub-specs declare an `imports:` dependency on modules it would `exports:`.
- Its scope covers the primary deliverable of the parent spec.

A sub-spec is **optional** if:
- It covers ancillary concerns (tests-only, docs-only, CI config, mirror updates).
- No other sub-spec depends on its exports.
- The parent spec can be considered partially delivered without it.
