# Subagents — Agents Calling Agents

> **Status:** Planning + foundations. The simplest tool (`agent_call`, synchronous) is built and live. The no-cost **mock test provider** is built (this doc, §8). Everything else here is **design under review** — nothing in §5–§7 is implemented yet. This is the planning doc; update it as we go.

---

## 0. The goal, in one paragraph

An agent should be able to invoke other agents — synchronously, in parallel, or in the background — by ID (a saved agent) or ad-hoc (a manual one-shot agent). The hard part is **not** "call `run_agent` again." It's that a child agent run touches every fragile seam in the platform at once: the three identity scopes (`cx_user_request` / `cx_conversation` / `cx_request`), the fire-and-forget persistence coordinator, the single per-request stream, cost rollup, and the recursion/cost guards. This doc maps those seams (code-confirmed), enumerates every execution option, and fixes the rules **before** we build the async layers.

---

## 1. How it works today (code-confirmed)

**1. Single tool call** — `Call → full response → agent sees result → continues.`

**2. Multiple tool calls in one turn** — **concurrent, not sequential.** `executor.py::execute_batch` turns every call into a task and runs one `asyncio.gather(...)`:

```891:905:packages/matrx-ai/matrx_ai/tools/executor.py
        tasks = []
        for tc in tool_calls:
            ...
            tasks.append(self.execute(name, arguments, child_ctx, client_tools, allowed_tools))

        raw_results = await asyncio.gather(*tasks, return_exceptions=True)
```

Three `agent_call`s in one turn ⇒ three sub-agents run in parallel, the turn blocks until the **slowest** finishes (wall-clock ≈ `max`, not `sum`), then all results return together. This is already "parallel agents, join-all."

**3. The invariant that defines "today":** the parent turn **cannot advance until the whole tool batch completes.** A sub-agent runs *inside* the parent's gather, so the parent is frozen mid-loop, holding its inference slot. Sub-agents are fully synchronous w.r.t. the parent.

**The escape hatch that points the way:** **client-delegated tools already do non-blocking suspend/resume.** A delegated tool flips a `cx_tool_call` row to `status='delegated'`, ends the turn (loop suspends, stream closes), and `POST /resume` wakes it once results arrive — durable seconds→weeks, no idle loop held. *This is the template for background agents (§5, option 6).*

---

## 2. Vocabulary

- **Synchronous / blocking** — call, wait, get result. (today's default)
- **Parallel scatter-gather** — fire N at once, join on all. (today, within one turn)
- **Background / detached / async** — fire, get a *handle*, keep working, collect later. (the new thing)
- **Poll** — ask "is it done?" via a handle.
- **Join / await** — block on an outstanding handle when the answer is finally needed.
- **Suspend-resume** — parent ends its turn, is durably woken when the child finishes (no compute burned while waiting).
- **Observe / subscribe** — parent or human watches the child's progress live.

---

## 3. The full menu of options (all must be planned)

| # | Pattern | Parent while child runs | Substrate | Status |
|---|---|---|---|---|
| 0 | **Sync single** | frozen, waits | inline | ✅ built (`agent_call`) |
| 1 | **Parallel join-all** | frozen, waits for slowest | inline gather | ✅ built (N× `agent_call` in one turn) |
| 2 | **Background + poll** | keeps working same turn; `agent_check(job_id)`; never forced to wait | A (then B) | planned |
| 3 | **Background + explicit join** | keeps working, then `agent_await(job_id)` | A (then B) | planned |
| 4 | **Fan-out + collect** | spawns many, keeps working, `agent_collect([...])` | A (then B) | planned |
| 5 | **Fire-and-forget** | spawns, never reads; result persists for a human/another flow | B | planned |
| 6 | **Suspend-resume (durable)** | parent ends the turn; woken via `/resume` when child(ren) done | B | planned |
| 7 | **Observe / stream** | parent or human watches child progress events live | A/B + FE contract | planned |
| A | **Manual / ephemeral agent** (flavor A) | any of the above, but the child is an ad-hoc one-shot, not a saved agent | A/B | planned |
| B-search | **`agent_search`** | RLS-scoped discovery (own/org/shared/builtin) | read | planned |
| B-get | **`agent_get`** | variables + usage hints (list input) | read | planned |

Options 2–4 are "true concurrency" (parent loop alive alongside child). Option 6 is "suspend" (parent loop dead, durably resumed). Different cost, different machinery — both wanted.

---

## 4. The two execution substrates (the real decision)

Everything reduces to **where the spawned agent lives and who owns its lifecycle.**

**Substrate A — in-process detached task** (`asyncio.create_task` + an in-memory job registry).
- *Pro:* trivial; true parallelism with the live parent loop.
- *Con:* child life is tied to the parent request's event loop. The "disconnect never stops work" guarantee covers the **request's own detach supervisor** — an ad-hoc child task is **not** covered (`response.py::_supervise_detached_task` supervises the request task, not arbitrary children). If the parent turn ends, the task is orphaned. **No durability, no cross-restart, no suspend-resume.**
- **🔴 H1 — the foundational constraint:** Substrate A can satisfy **Rule 1 (durable result+cost capture)** ONLY for options where the parent is *structurally guaranteed* to join before the request ends — i.e. **options 3 (explicit join) and 4 (collect)**, and only if "the request cannot end with outstanding A-children" is enforced. **Option 2 (poll, never forced to wait), 5 (fire-and-forget), and 6 (suspend) CANNOT run on A** — an orphaned in-process child loses both its result and its cost record, the exact failure Rule 1 forbids. So "build A first" is valid **only for 3/4 against the mock**; 2/5/6 require Substrate B from the start.

**Substrate B — durable job + worker** (an `agent_job` row + a server-side worker lane).
- Fire → write a job row (agent_id, variables, `status='pending'`, parent linkage, depth, **its own `user_request_id`**) → a worker runs it through detached/streamed execution → result + cost persisted on the row.
- This is literally the `status='delegated'` + `/resume` pattern generalized to **"delegated to a server worker"** instead of "delegated to the client." Powers 5, 6, and a robust 2–4.
- *Con:* more to build.

**Plan (revised after adversarial review):** build **A** only for options **3/4** (parent always joins in-request, testable soon against the mock); build **B** for **2/5/6** (any "never-wait" / outlives-the-turn case) because only B gives a durable result+cost row and a live emitter (H1, H2). They coexist — `agent_check` reads whichever backend holds the job.

**Per-option cross-cutting concerns (apply to every row in §3):** cancellation propagation (parent cancel → children?), per-child + per-tree **timeout/answer-deadline**, **partial failure** in a fan-out batch (1 of 10 fails — fail-fast or collect-with-errors?), **retry** policy, **dedup** of identical concurrent spawns, **result-size limits** (a child returning 5 MB onto the parent's context), and **child output-directives / side-effects** (a child whose structured output triggers a durable side effect — does it apply, and under whose identity?). These are specified as rules in §7 / questions in §9, not per-row, to avoid duplication.

---

## 5. Confirmed architecture map (the seams we must respect)

### 5.1 The three identity scopes

| Scope | Table | Meaning | Key facts (code-confirmed) |
|---|---|---|---|
| **User request** | `cx_user_request` | ONE inbound backend API call (one user action) | PK **is** `AppContext.request_id`, resolved from the `X-Request-ID` header **or minted if absent/invalid** (`matrx_connect/middleware/auth.py:301-302`). **Has NO `conversation_id`** (dropped, migrations 0107–0110). Carries the cost/token rollup. ⚠️ Because a valid client-supplied `X-Request-ID` is **reused**, two separate top-level API calls can already share one `cx_user_request` *before subagents exist* — the cost-rollup design (Flags 1/4) must account for this existing reuse, not just child fan-out. |
| **Conversation** | `cx_conversation` | ONE back-and-forth thread (many turns) | Self-FK lineage columns **exist**: `parent_conversation_id` and `forked_from_id`/`forked_at_position`. **No `root_conversation_id`** — single-level pointers only. |
| **Provider request** | `cx_request` | ONE provider API call (message → response) | Carries **both** `user_request_id` AND `conversation_id`, both **NOT NULL**. This is the only durable join of request↔conversation (M2M). `cx_tool_call` also FKs both. |

**One-line model:** `cx_user_request (1) → (N) cx_request (N) → (1) cx_conversation`. `AppContext.request_id == cx_user_request.id`. Conversation↔request join lives only on `cx_request`.

### 5.2 AppContext + fork semantics

`AppContext` (frozen dataclass, `matrx_connect/context/app_context.py`) carries: `user_id`, `request_id` (= user-request id), `operation_id`, `conversation_id`, `parent_conversation_id`, `parent_request_id`, `is_internal_agent`, `agent_id`, `agent_version_id`, `organization_id`, `metadata` (dict), `emitter`. **There is NO `user_request_id` field** (`request_id` is it) and **NO `agent_call_depth` field** (it lives in `metadata["agent_call_depth"]`).

`fork_for_child_agent(new_conversation_id, operation_id)` (`app_context.py:177`):
- **New:** `conversation_id` (fresh uuid4), `is_internal_agent=True`, `memory_*` reset, `operation_id`.
- **Derived:** `parent_conversation_id ← parent.conversation_id`, `parent_request_id ← parent.request_id`.
- **Inherited unchanged:** `request_id` (so the child writes to the SAME `cx_user_request`), `user_id`, `emitter` (**shared by reference — same stream**; `app_context.py:317`), `agent_id`.
- **Shallow-copied (`dict(...)`/`list(...)`):** `metadata`, `api_keys`, `initial_variables`, `initial_overrides`, `client_tools`. Top-level copy only — nested values still alias. (The code's own docstring claims "deep-copy"; that's **wrong** — it's a shallow top-level copy. Fix the code docstring.)

> **Sibling fork to know about:** `fork_for_workflow_step` (`app_context.py:219`) sets `parent_conversation_id=None` AND `parent_request_id=None` while keeping `request_id`. A subagent built on the *workflow* path would get **no lineage at all** — only `fork_for_child_agent` carries parent links. Don't build subagents on the workflow fork.

`child_agent_context(label)` (the orchestration wrapper) mints a fresh conversation uuid + a new `operation_id`, gives the child its **own `ReservationTracker`**, runs registered child-fork hooks (matrx-ai forks the persistence coordinator here), emits a `sub_agent` `INIT`/`COMPLETION` pair, restores the parent on exit.

`run_agent`'s `independent_request=True` is the **only** way a child resets `request_id` (`overrides["request_id"] = uuid4()` → its own `cx_user_request`). **`agent_call` does NOT pass it today** — children share the parent's `cx_user_request`.

### 5.3 Persistence — fire-and-forget coordinator

- The matrx-ai `Coordinator` is a façade over `matrx_orm.Session`; ordering is **FK-tier topological** (parent rows before child rows), derived from declared FK fields — the `depends_on` arg is **ignored** (`coordinator.py:28`).
- **Scoping: per-async-task via ContextVars** (`_coordinator_cv`, plus `ExecutionState.writes`). Concurrent requests/children don't share a coordinator. **Each child gets its OWN coordinator/Session** via a registered child-fork hook (`queue_helpers.py::_child_coordinator_scope`).
- **Fork ordering guard:** before fan-out, the hook does `await parent_coord.finalize(reason="pre_fan_out")` so every ancestor row a child FK-depends on is durable first.
- **Process-wide cap (a POOL BUDGET, not a health signal):** `_MAX_INFLIGHT_COMMITS = 8` is a **single process-wide budget across ALL requests and children**; the headroom math is `2*cap + foreground_reads < pool_max(30)`. N concurrent children contend on the same 8 slots. Since 2026-08-18 a barrier **WAITS** for a slot (`await coord.acquire_commit_slot()` before `commit_async`, bounded by `_COMMIT_SLOT_WAIT_SECONDS = 30`) — saturation by HEALTHY ms-scale commits is backpressure to wait on, not a durability incident. `commit_async` still raises `PersistenceBarrierError` when the wait expires, i.e. commits genuinely are not draining. (Before that fix, wide fan-out failed at the cap: run `6266cb1b-…` fanned 25 auditor agents, 22 committed and 3 were refused with "DB not draining" while the DB was fine.)
- `finalize()` is the sanctioned sync blocking point at exactly: request-end, pre-client-suspend, sub-agent fork/join. `drain_and_confirm()` (never raises) runs on any anomaly. **`seal()`** (`coordinator.py:796`) terminally seals the coordinator after the lane's final drain so late writes one-shot instead of stranding as stuck `status='running'` rows — a Substrate-B worker that finalizes-but-never-seals reintroduces that stuck-row class.
- **Session-stack teardown:** the Coordinator pushes its Session onto matrx-orm's `_session_stack` and relies on the request task's Context being discarded to unwind (`coordinator.py:217`). A long-lived background worker (Substrate B) that is **not** a short-lived task needs an explicit teardown or it leaks stack entries.

### 5.4 Streaming — one stream per request

- `create_streaming_response` opens ONE `RequestLane` keyed to `ctx.request_id`, announces the conversation/user-request ids **once** up front (`ConversationIdData`, `RECORD_RESERVED`), installs the lane in a ContextVar so children inherit it.
- **Child agents reuse the PARENT emitter — there is exactly one stream** (`emitter = parent_ctx.emitter`, `app_context.py:317`). Child events multiplex onto the parent stream, tagged by `operation_id` via `sub_agent` `INIT`/`COMPLETION` and `STRUCTURED_OUTPUT`.
- **🔴 Fine for synchronous children, FATAL for background ones (H2):** a background child (options 5/6) outlives the parent turn — by the time it emits, the parent stream/emitter is **closed**. Reusing the parent emitter by reference means every event a backgrounded child emits drops onto a dead stream. Background children MUST own their own lane/emitter (⇒ Substrate B), not inherit the parent's.
- **There is NO event to announce a new id mid-stream / "also listen here."** Full `EventType` vocabulary: `chunk, reasoning_chunk, phase, warning, info, data, init, completion, error, tool_event, broker, heartbeat, end, render_block, record_reserved, record_update, resource_changed, context_analysis, structured_output, context_state, context_trimmed, injection_consumed`. The closest primitives are `INIT/COMPLETION(operation="sub_agent")` (same stream) and `RECORD_RESERVED` (a row-id hint). **`CHUNK`/`REASONING_CHUNK` payloads carry only `text`, no `operation_id`** — so streamed child *prose* can't currently be attributed to a specific child on the wire.

---

## 6. Cross-cutting concerns + required hardening

These four systems will be stressed by subagents. Each has a confirmed weakness or a needed new rule.

### 6.1 CTX / context: sharing, transition, new

We need explicit, named context modes for a child agent (proposed vocabulary — **for review**):

- **`isolated`** (default) — child gets a fresh conversation, no parent message history, no shared ctx context items. Safest; what `agent_call` does today.
- **`inherit_context`** — child receives selected ctx context items (the `__table_access` / context-injection machinery already exists; this is wiring) but still its own conversation.
- **`shared_thread`** — child continues *in the parent conversation* (no fork). Rare; dangerous (interleaving writes on one conversation). Likely **out of scope** for v1; flag if ever requested.

**The hard ID rules (must hold for every mode):**
1. `request_id` (= `cx_user_request.id`) is the *user action* scope. A child that should roll its own cost/usage MUST get `independent_request=True` (its own `cx_user_request`). See Flag 1.
2. A background/durable child (Substrate B) MUST get **its own `user_request_id`**, never inherit a parent request that may already be `completed`/`abandoned` (see Flag 4).
3. `conversation_id` is always fresh per child (except the out-of-scope `shared_thread`). `parent_conversation_id` links lineage.
4. Depth must be **stamped explicitly** on the child, never inferred from a shared-`metadata` bump (see Flag 3).

### 6.2 ORM persistence

- **Flag (cap):** the 8-commit process-wide ceiling will trip with 10+ concurrent children doing per-turn `commit_async`. **Required:** before any real fan-out, either (a) raise/scope the cap per-lane with backpressure that *queues* rather than *errors*, or (b) give background children their own lane/worker (Substrate B) so they don't contend on the parent request's commit budget. **Decision needed** (open question Q4).
- **Flag (FK ordering for durable jobs):** a Substrate-B child with its own `user_request_id` + `conversation_id` writes a self-consistent FK tree — *good*. A Substrate-A child sharing the parent request relies on the `pre_fan_out` finalize; that's proven for the in-turn gather but **not** for a detached task that forks its context at a different time.
- **Rule:** every spawned child runs inside a lane (its own or the parent's). `get_coordinator()` returning `None` is a DATA-LOSS path (`queue_helpers.py:333`) — a background spawn outside a lane silently drops writes. Guard against it.

### 6.3 Streaming → frontend

- **Confirmed gap:** no "new stream/id started, also listen here" event; child prose chunks aren't attributable on the wire.
- **Proposed (needs FE agreement, Q5):**
  - A new `EventType` (or a `RESOURCE_CHANGED` kind) — e.g. `SUBAGENT_STARTED { job_id, parent_operation_id, agent_id, conversation_id, mode }` — emitted when a child spawns, so the FE can render a child panel / optionally subscribe.
  - Add `operation_id` to `CHUNK`/`REASONING_CHUNK` payloads (additive field) so child prose is attributable. **OR** decide child prose stays silent on the parent stream and the FE reads the child conversation by id on completion.
  - Decide: do background children stream onto the **parent** stream (multiplexed, needs framing) or onto **their own** stream (FE opens a second listener via the new event)? This is the central FE design question.

### 6.4 DB storage — source / app / feature / subagent flag

- We already track the calling **app** and **feature** on requests. We must decide deliberately what a subagent run records:
  - **`source`/`feature`:** propagate the parent's, or a new value (e.g. `feature='subagent'` or `source='agent_call'`)? **Proposal:** keep the parent's app, add a dedicated marker so the UI/analytics can distinguish (Q6).
  - **Subagent marker:** the lineage already lives in `cx_conversation.parent_conversation_id`. **Proposal:** also stamp a typed marker (e.g. `cx_request.metadata` → a typed column later, or reuse `is_internal_agent`) so the FE renders child turns as nested, not as top-level conversation turns. **Must be discussed and agreed with the FE** before we pick the field.

---

## 7. Rules / invariants (airtight before async ships)

From the repo doctrine (a paid call is billed in full the instant it starts; a disconnect never stops work; no silent defaults) + the confirmed flags:

1. **A spawned agent persists its result + cost INDEPENDENTLY of whether the parent ever reads it.** A backgrounded paid agent whose handle is never joined must NOT vanish — it writes result+cost to its own durable row.
2. **Every handle has an abandonment backstop** (mirror `DELEGATED_CALL_ABANDON_AFTER_SECONDS`) — never-joined jobs are reaped to `abandoned` with result still captured.
3. **Depth is stamped explicitly on the child**, not inherited via a shared-`metadata` bump (the bump is racy under parallel fan-out — Flag 3). `MAX_AGENT_CALL_DEPTH` enforced on the stamped value.
4. **Per-request fan-out cap** (CAPS constant): cap outstanding background agents per request (e.g. `MAX_BACKGROUND_AGENTS = 8`), reject past it **loudly**, never silently drop. (Must be reconciled with the 8-commit persistence cap — Q4.)
5. **Handles are scoped to the requesting user + conversation.** `agent_check(job_id)` verifies ownership — a job_id is not a capability token. Same RLS gate as `agent_call`.
6. **Suspend (option 6) reuses the ONE delegation/resume path** — no second suspend mechanism.
7. **Poll is cheap and bounded; the model never busy-waits.** `agent_check` returns `pending|running|completed|failed|abandoned` + result-when-ready. "Must wait" = option 6 (suspend), not a spin loop against `max_iterations`.
8. **Cancellation is explicit and server-side** (reuse `POST /cancel/{request_id}` → `RequestControlRegistry`); even then, let the in-flight provider call finish and capture cost before exit.
9. **Cost-rollup correctness (Flag 1 + H4):** children that share a parent `user_request_id` must NOT clobber its aggregate; but giving children `independent_request=True` **destroys aggregation** because `cx_user_request` has **no lineage column** to join children back to the root (H4). Neither current option is free. The likely real fix: add a `parent_user_request_id` (or `root_user_request_id`) column AND re-derive the rollup from committed `cx_request` rows. **Decide before fan-out (Q3).** Note `agent_call` already surfaces child cost to the parent via `child_usages=list(result.usage_history)` (`agent_call.py:256`) — any new background path MUST replicate that or child cost silently disappears from the parent rollup.
10. **Lineage must be reconstructable (Flag 2):** the `pre_fan_out` finalize makes the parent conversation durable before a child reads it, but the "spawn on the parent's very first turn" race can still drop `parent_conversation_id` to NULL. Confirm the finalize closes it for spawns, or add a `root_conversation_id` / explicit lineage write.
11. **A real COST CEILING, not just a count (H3): ✅ LIVE.** Rule 4's count cap does NOT bound dollars — a tree can legally spawn `MAX_BACKGROUND_AGENTS ^ depth` paid agents with no money ceiling. The **tree-wide dollar budget** is now implemented in [`matrx_ai/orchestrator/cost_budget.py`](../orchestrator/cost_budget.py): one CAPS knob `TREE_COST_CEILING_USD` (conservative default; set `None` to disable), a process-local accumulator keyed by the shared `user_request_id` (parent turns + every sub-agent fold their cost in at the executor's `add_usage` chokepoint, including provider-billed cost on failure/cancel — pairs with Phase 3). Before each turn's tool calls the orchestrator reads `remaining_budget(request_id)` into `ToolContext.cost_budget_remaining`; the existing `GuardrailEngine._check_cost_budget` blocks new (expensive/spawning) tool calls **loudly** (red banner + model error + FE `stream.error` + failed `cx_tool_call` row) once the tree is out of budget. Released by the request owner on finalize/cancel. The budget is tree-wide, not per-child. **Caveat:** the accumulator is in-process, so it covers the current in-process (Substrate-A) fan-out; durable cross-worker jobs (Substrate B) will need the same total re-derived from committed `cx_request` rows (the SUM the Phase-2 rollup already computes).
12. **Cancellation propagates DOWN, completion captures UP (P1):** an explicit `POST /cancel/{request_id}` must cancel outstanding children of that request (let each in-flight provider call finish + capture cost, then stop — Rule 8). A child cancelled this way still persists its partial result+cost (Rule 1).
13. **Every child has a deadline (P1):** a per-child + per-tree timeout (a CAPS constant, far-future backstop like `DELEGATED_CALL_ABANDON_AFTER_SECONDS`) bounds a hung child; on expiry it's reaped to `abandoned` with whatever result/cost was captured. Distinct from the answer-deadline the model reasons about.
14. **Fan-out partial failure is COLLECT-WITH-ERRORS by default (P1):** one failed child in a batch does NOT fail the batch; `agent_collect` returns per-item `{success, result|error}`. Fail-fast is opt-in. A failed child still obeys Rule 1.
15. **A child's output-directive side-effects run under the CHILD's identity, gated (P1):** if a child agent emits an `output_directive` envelope, it applies under the child's `user_id`/RLS (which equals the parent's `user_id` today), after the child persists, exactly as a top-level run would — never silently suppressed, never escalated. Background children must run the directive on their own lane (H2).
16. **Abandon ⇄ join ⇄ cancel is a single atomic status transition (P1):** the terminal state of a job (`completed|failed|abandoned|cancelled`) is set once, compare-and-set; a late `agent_await` that races the abandonment backstop reads the settled row, never double-resolves. No status double-write (the `cx_message.status='error'` class of bug).

### Confirmed risk flags (the bug list to design against)

- **Flag 1 — cost rollup last-write-wins.** `persistence.py:912` computes the `cx_user_request` aggregate in-memory from this call's `req_list`; multiple sub-agents sharing one `user_request_id` clobber the headline totals (per-`cx_request` rows survive). 
- **Flag 2 — parent-conversation lineage dropped on first turn.** `conversation_gate.py::resolve_parent_conversation_lineage` sets `parent_conversation_id=NULL` when the parent row isn't committed yet. Combined with no `root_conversation_id`, deep hierarchies become unreconstructable.
- **Flag 3 — depth-guard race.** `agent_call` bumps a shared `metadata` dict; concurrent children mutate the same dict → racy `MAX_AGENT_CALL_DEPTH`. Background spawns would bypass it entirely.
- **Flag 4 — request reuse under a closed parent.** A detached child inheriting `request_id` can write `cx_request` under a `cx_user_request` already marked `completed`/`abandoned`. Durable jobs need their own `user_request_id`.
- **Flag 5 — 8-commit process-wide cap.** `_MAX_INFLIGHT_COMMITS=8` shared across the process. Heavy fan-out no longer *fails* on it (barriers wait for a slot, bounded 30s), but it does *serialize* commits — a fan-out far beyond 8 pays latency, and a truly stuck DB still stops requests loudly.
- **Flag 6 — single stream, no child framing.** Child prose chunks carry no `operation_id`; no "new stream" event.

---

## 8. The no-cost test provider (BUILT)

To test fan-out (10+ agents, deep nesting) without cost or flakiness, a fake provider follows the **exact** real path and only replaces the network call.

**What was built (all additive):**
- `packages/matrx-ai/matrx_ai/providers/mock/mock_api.py` — `MockChat`, a real chat client (reads emitter from `AppContext`, streams chunks, returns `UnifiedResponse` + `TokenUsage`, can emit real tool calls) that makes **no** network call.
- `unified_client.py` — `API_CLASS_TO_ENDPOINT["mock_standard"]="mock_chat"`, `_PROVIDER_FACTORIES["mock_chat"]="MockChat"`, dispatch `elif endpoint == "mock_chat"`.
- Capabilities come from the model row's `capabilities` jsonb — the mock declares text/vision/function_calling/structured_output/json_mode so tools aren't stripped and structured output works. (The old `capabilities.py` api_class map is deleted.)
- `providers/__init__.py` — exports `MockChat`.
- **DB:** real `ai_model` row — `name='mock-standard'`, offering → ai.api `translator_key='mock_chat'`, provider `AI Matrx`, `function_calling`. (id `282d1f0e-e564-4589-bb79-33a57d219ac0`.) **Pricing is a zero-price TIER** `[{input_price:0, output_price:0, cached_input_price:0}]`, **not an empty array** — an empty `[]` would make `warm_pricing_lookup` skip the model and `calculate_cost` scream "Pricing not found" on every call. Zeros = clean `$0`.

**How to drive it** — put a `mock` spec on the agent's `settings.metadata` (mirrors the extraction path's `metadata.extraction`):

```jsonc
{
  "mock": {
    "latency_ms": 800,        // total simulated wall-clock (capped 120s)
    "ttft_ms": 100,           // time-to-first-token
    "chunks": 8,              // stream chunk count
    "text": "final answer",   // omit → echoes the last user message
    "mode": "echo",           // echo | text | json
    "json": { "k": "v" },     // structured output when mode=json
    "input_tokens": 1200,     // omit → estimated from messages
    "output_tokens": 300,     // omit → estimated from text
    "tool_calls": [           // emitted as REAL tool calls on the first pass;
      { "name": "math_calculate", "arguments": { "expression": "2+2" } }
    ],                        // follow-up turn returns `text`
    "fail": false,            // true → runs the real error path
    "error_message": "..."
  }
}
```

With **no** spec, the mock echoes the last user message after a short delay — a zero-config smoke test. The tool-call round-trip lets you exercise the full loop (real tool execution, persistence, usage) for free.

> **Runtime verification:** the package needs `matrx_ai.configure()` (startup), so import-time checks can't run standalone — validate on next `python run.py` by assigning a test agent the `mock-standard` model and running it.

**Known follow-ups / caveats for the mock:**
- Deterministic per-iteration scripting (a different response per turn beyond the single tool round-trip).
- Simulated mid-stream cancellation; configurable reasoning chunks.
- Usage `api` field is set to the mock's own tag (`mock_standard`), not a canonical provider string — cost still resolves because the exact `matrx_model_name` (`mock-standard`) match hits first in `calculate_cost`.
- (Fixed) the tool-emitting first pass now estimates input tokens from the messages instead of reporting 0.

---

## 9. Open questions (for the user + the FE)

- **Q1 (model versioning):** ~99% current version, ~1% pinned. Confirm: optional `version_id` on the execute tools, current by default.
- **Q2 (context modes):** are the three modes in §6.1 (`isolated` / `inherit_context` / `shared_thread`) the right set? Is `shared_thread` ever needed, or explicitly out of scope?
- **Q3 (cost rollup, Flag 1 + H4):** for a sub-agent, do we want **its own `cx_user_request`** (accurate per-child cost, but *no lineage column exists* to aggregate the user action's total — H4), **shared parent request with a re-derived rollup** (Flag 1 clobber risk), or a **new `parent_user_request_id` column + re-derived rollup** (the likely correct answer)? This decides the identity model for every async option — answer first.
- **Q4 (concurrency cap, Flag 5):** raise/scope the 8-commit cap with queueing backpressure, or route background children to their own worker lane so they never contend on the parent's commit budget?
- **Q5 (streaming, FE):** background children stream onto the **parent** stream (multiplexed — needs `operation_id` on chunks + a `SUBAGENT_STARTED` event) or onto **their own** stream (FE opens a second listener)? **This needs an explicit FE agreement.**
- **Q6 (DB source/flag):** what `source`/`feature` does a subagent run record, and what typed marker tells the FE to render child turns as nested? **Needs FE agreement.**
- **Q7 (build order):** confirm: (a) `agent_search` + `agent_get` next (read-only, low risk), then (b) Substrate-A `agent_await`/`agent_collect` (options 3/4 only) behind the mock, then (c) Substrate-B durable jobs for options 2/5/6 + suspend-resume.
- **Q8 (cost ceiling, H3): ✅ RESOLVED + LIVE.** Tree-wide dollar budget implemented in [`cost_budget.py`](../orchestrator/cost_budget.py), threaded via `ToolContext.cost_budget_remaining`, default ceiling `TREE_COST_CEILING_USD = 25.0` (conservative; the single tuning knob). Remaining open piece is only the cross-worker (Substrate-B) re-derivation noted in Rule 11.
- **Q9 (cancellation/timeout, P1):** confirm cancel-propagates-down + per-child/per-tree deadline constants, and the fan-out partial-failure default (collect-with-errors).
- **Q10 (child side-effects, P1):** confirm a backgrounded child's `output_directive` side-effects should apply (under the child's identity) rather than be suppressed.

---

## 10. Recommended build order

0. **Settle H1–H4 + the identity model (Q3/Q8) FIRST.** The adversarial review blocks the build until the durability/identity/cost-ceiling rules are decided — otherwise we validate a foundation that silently loses money/data in production.
1. **`agent_search` + `agent_get`** — RLS-scoped discovery + variable/usage hints (read-only, additive, unblocks flavor B steps 1–2). Safe to start now.
2. **Substrate-A** `agent_await` / `agent_collect` — options **3/4 only** (parent always joins in-request), tested entirely against the **mock provider**, with results written durably (Rule 1), depth stamped (Rule 3), and a tree-wide cost budget (Rule 11).
3. **Substrate-B** durable `agent_job` + worker + own lane/emitter + **suspend-resume on the `/resume` path** — options **2/5/6**, plus the FE streaming contract (Q5).
4. **Flavor A** ephemeral `agent_run_manual` with curated defaults/menus.
5. **Hardening** the confirmed flags (cost rollup + lineage column, the commit cap) — must land *before* anything in (2)/(3) ships to real (non-mock) traffic.

### Known defects to file (per repo doctrine)
- `load_for_execution` enforces no RLS (agent_call gates it per-tool; the real fix is a visibility-checked load).
- Org-shared agents aren't runnable via `agent_call` yet (owner/public/admin only).
- Weekly background agent to write agent descriptions/variable-hints (makes search+get useful).
- Flags 1–6 above.
