# Agent-designed mini workflows — `matrx_ai.plans`

> Verified against code 2026-07-02 (module built + full test suite added the same day). Hardened 2026-07-07 after the first live test (see Change log).

**What this is.** A simplified scheme (`AgentPlan`) that LLM agents use to design simple ad-hoc multi-agent workflows, plus a deterministic converter (`compile_plan`) that turns a plan into a standard `matrx_graph.Definition`. **Once compiled, the workflow is identical to any other — the engine doesn't know or care that an agent designed it.** There is no separate runner: no new run system, no new checkpointing, no new persistence.

**What this is NOT.** A general workflow designer. Plans are agent→agent chains with fan-out and simple branching, bounded by `MAX_PLAN_STEPS=20` and `FOR_EACH_MAX_ITEMS=200` (CAPS in [types.py](types.py)). Anything richer is authored as a real workflow.

## The three parts

1. **`AgentPlan`** ([types.py](types.py)) — flat step list: `{name, reasoning, inputs, steps[]}`; each step `{step, agent_id, purpose, inputs, depends_on, for_each?, when?}`. `plan_json_schema()` is the output_schema a planner agent is given — routed through the `matrx_ai.schema` portable derivation, **never hand-massage provider rules**.
2. **Validation gate** ([validate.py](validate.py), [errors.py](errors.py)) — `validate_plan` (structural, pure) + `validate_plan_agents` (agents exist / access / variables / output_schema, via host-injected agx managers). ALL issues collected, `error_type="agent_plan_validation_gate"`. **An invalid plan fails before any money is spent or row is created.** `compile_graph(compile_plan(plan))` is the third, engine-level gate.
3. **Converter** ([compiler.py](compiler.py)) — `compile_plan(plan) -> Definition`. Pure, no LLM, no DB.

## Reference grammar (strict — any other `$`-string is a gate error)

```
$inputs.<path>              plan input, INLINED AT COMPILE TIME (re-run with new inputs = recompile)
$steps.<n>.output[.<path>]  a step's output payload
$item[.<path>]              the current item — for_each steps only
```

- `<path>` = dot-separated identifiers / list indices. **No wildcards, no `[]`** — per-item work is `for_each`, never array plucks.
- A plain agent step's payload is the `AiExecutionResult` dump — `final_text`, **`structured_output`** (parsed against the agent's output_schema; requires that agent to HAVE an output_schema — gate rule), `usage`, ids.
- A `for_each` step's payload is `{values: [per-item results], count}`.
- **A `for_each` step's inputs may use only `$item`, `$inputs`, literals** — cross-step refs can't vary per item; the gate rejects them.
- `when` = boolean expression over `$steps.M.output.<path>` where M is the step's single dependency; compiled onto a CONDITIONAL edge; false ⇒ step + dependents skipped, run completes.
- Input values are **any JSON literal** (string/number/bool/list/object) or a ref string — refs work inside nested lists/objects too. Explicit `null` for `inputs`/`depends_on` = omission (LLMs emit it; never reject it).
- **Payload paths are validated at the gate, two independent layers** (the 2026-07-07 extinction: `$steps.1.output.cards` passed, step 1's PAID call ran, step 2 died on `KeyError: 'cards'`): ① structural — `$steps.N.output.<first>` must be a real payload key (`AGENT_PAYLOAD_KEYS` / `FAN_OUT_PAYLOAD_KEYS` in types.py), with a "did you mean structured_output.…" hint; ② agent-aware — the path inside `structured_output` is walked through the source agent's output_schema (`{name, schema:{…}}` envelope unwrapped), unknown fields rejected with the available ones listed, and `for_each` targets must be schema arrays.
- Pydantic errors on a malformed plan render as gate-style issues (`issues_from_validation_error`) — no pydantic.dev URLs, union-branch noise collapsed.

## What compiles to what

| Plan construct | Synthesized graph |
|---|---|
| plain step N | `step_N_args` (data.transform assembler building `{user_input, variables}`) → `step_N` (action.ai.agent.start, `data.input={agent_id}`) |
| dependency M→N | edge from M's out-node, `mappings={"src_step_M": "$FULL_PAYLOAD"}` (namespaced so same-step merges can't collide) |
| unequal-depth join | pass-through `pad_M_to_N_k` transform nodes equalize arrival — **Pregel fires on ANY edge; without pads a join fires early with a partial payload** |
| for_each step N | args → `step_N_map` (control.map + `payload_template`) → sends → `step_N` workers (`output_channel="step_N_results"`, append channel) → edge-triggered `step_N_gather` (`join_policy: "any"` — with the default 'all' an EMPTY fan-out deadlocks: zero workers means the worker→gather edge never fires and never prunes; 'any' lets the delivering branch satisfy the join, and §4.4 group resolution still holds gather for ALL items in the non-empty case); plus `dispatched == 0` conditional empty-pad so gather always fires |
| when | CONDITIONAL first edge of the dependency chain |

Fan-in is **channel-only**: edge routing merges per-target last-writer-wins, so N parallel worker results over edges collapse to one payload. `output_channel` + `control.map payload_template` are the two matrx-graph platform capabilities this module rides (added 2026-07-02).

## Result contract

`definition.metadata["agent_plan"]` = `{version, plan, result_map, layers, max_steps_hint}`. `result_map[step] = {node_id, kind: single|list}` — the host reads the run's `last_outputs[node_id]` (`single` → AiExecutionResult dump; `list` → gather's `values`). A node id absent from `last_outputs` ⇒ the step was **skipped** (when-false) or **not reached** (run failed early). `definition.metadata["source"] == "agent_plan"` marks generated definitions.

## Consumers

- **aidream** — [`aidream/services/agent_plans/FEATURE.md`](../../../../aidream/services/agent_plans/FEATURE.md): persists the compiled Definition as a normal tagged `wf_definition`, starts a normal `wf_run` via the canonical detached executor, assembles per-step results from `result_map`, and exposes the `agent_plan` tool (run | status | cancel). New hosts follow the same shape; never fork the gate or the converter.

## Tests

`packages/matrx-ai/tests/plans/` — schema+grammar, one test per gate rule, compiler goldens (padding pinned), and `test_plan_engine_mock.py`: the full plan on the REAL scheduler via `mock_response` (zero LLM calls) covering fan-out, unequal-depth join, when-true/false, and the empty-list path.

## Change log

- 2026-07-07 (2) — companion guardrail honesty (`matrx_ai/tools/guardrails.py`): loop detection's md5-hex ">0.8 similarity" was mathematically always exactly equality (avalanche → ~1/16 char overlap for any non-identical args), so the "very similar arguments" message oversold it — now an explicit identical-args comparison (dead `_similarity` helper removed, no behavior change) with an honest message, plus a PER-TOOL recency window so a burst of exempt-tool polls (still recorded for rate/cost limits) can't evict an unrelated tool's calls from the window and blind its loop guard.
- 2026-07-07 — first-live-test hardening: null-tolerant schema, JSON-literal inputs with nested refs, payload-path gate rules (structural + schema-aware — the KeyError-at-runtime extinction), clean pydantic-error rendering. Companion platform fixes: errored runs persist partial `last_outputs` (matrx-graph run_store) and loop detection honors `dedupe_exempt` (matrx-ai guardrails).
- 2026-07-02 — module created: types/validate/compiler/errors + tests. Platform prerequisites landed the same day: matrx-graph `output_channel` binding + `control.map payload_template`; matrx-ai `AiExecutionResult.structured_output`.
