# <a href="../README.md"><img src="https://raw.githubusercontent.com/skynetcmd/m3-memory/main/docs/m3_logo_icon.png" height="60" style="vertical-align: baseline; margin-bottom: -15px;"></a> Multi-Agent Orchestration

M3 Memory provides the persistent memory and coordination substrate for multi-agent workflows. Agents share knowledge through scoped memory, pass context through handoffs and inboxes, and coordinate work through tasks, notifications, and a recursive task tree.

M3 Memory is not an agent runtime — it does not schedule or execute agents. It is the memory layer underneath your orchestrator, whether that is the bundled `m3-team` CLI, a LangGraph pipeline, or your own polling loop.

> **Looking for a wire-up guide for Claude Code + Gemini CLI + OpenCode sharing one m3-memory store?** See the practical setup notes — subscription vs API token tradeoffs, unified tag schema across agents, per-agent install steps — at [Multi-Agent Subscription Models with m3-Memory](./multi_agent_subscription_models.htm) (saved page; covers the day-to-day workflow underneath the primitives below).

## 🧩 Primitives

### 👤 Agent registry

Every agent registers with an ID and description. The registry tracks liveness via heartbeats and provides a directory other agents can query.

| Tool | Purpose |
|---|---|
| `agent_register` | Register an agent with ID, description, and capabilities |
| `agent_heartbeat` | Signal liveness |
| `agent_list` | List registered agents and their status |
| `agent_get` | Get details for a specific agent |
| `agent_offline` | Mark an agent offline |

### 📁 Scoped memory

Memory is isolated by `scope` so agents can maintain private working notes while sharing project-level knowledge.

- **`agent`** (default) — private to the writing agent. Use for implementation notes, scratch work, and internal reasoning.
- **`org`** — shared across all agents. Use for requirements, decisions, contracts, and any fact the whole team should see.
- **`user`** — scoped to a specific user/data subject. Use for user preferences, profile data, and GDPR-relevant records.

All scopes support the same `memory_search`, `memory_write`, and `memory_update` operations. Scope filtering is applied at query time — an agent searching `scope="org"` sees only org-scoped memories.

### 🔐 Governance / Enforcement

M3 supports **true SQL-layer agent isolation** — access control injected into the query itself (`WHERE (scope != 'agent' OR agent_id = ?)`), never a post-fetch filter that can leak. It's opt-in, so trusted single-operator setups keep full visibility by default and gain a hard boundary the moment they need one.

By default, scope filtering is **caller-applied / advisory**: a search that supplies no `scope` or agent filter sees every agent's memories, including other agents' private `scope="agent"` notes. This is intentional — it fits a trusted, single-operator multi-agent setup where the operator (or the orchestrator itself) is allowed full visibility, and it keeps `memory_search` byte-identical to prior behavior for every existing caller.

For setups that want a real access-control boundary, enforcement is **opt-in**:

- Pass `requesting_agent="<id>"` on a `memory_search` call — the search returns only that agent's own `scope="agent"` memories, plus every shared-scope memory (`org`, `user`, `session`). It can never surface another agent's private notes.
- Or set `M3_ENFORCE_AGENT_ISOLATION=1` to make this the default whenever an `agent_filter` is present, without threading `requesting_agent` through every call site.

This is enforced at the **SQL layer** — a `WHERE (scope != 'agent' OR agent_id = ?)` clause added to the query itself — not as an app-layer post-filter over an already-fetched result set. An explicit `scope=` filter combined with `requesting_agent` only narrows further (e.g. `scope="org"` plus `requesting_agent="implementer"` still returns only org rows); enforcement never widens what an explicit scope filter already restricted.

**Example — planner vs. implementer:** a planner keeps private scratch notes in `scope="agent"` under `agent_id="planner"`. If the implementer searches with `requesting_agent="implementer"`, the planner's private notes are excluded from the result set — but both agents still see the same `scope="org"` requirements, since shared scopes are never gated by `requesting_agent`.

**Audit trail:** isolation is the access-control half of governance; observability is the other half. Every write already records `change_agent`, and the `memory_history` table (see `memory_history_impl`) keeps a full audit trail of which agent mutated which memory row and when — so even in a trusted setup without enforcement turned on, you can always answer "who changed this and when."

### 📨 Handoffs and inbox

When one agent finishes and the next needs to pick up, use `memory_handoff` to push context directly into the receiving agent's inbox. The receiver calls `memory_inbox` to read pending items and `memory_inbox_ack` to clear them.

```python
# Planner hands off to implementer
await tool(session, "memory_handoff", {
    "from_agent_id": "planner",
    "to_agent_id": "implementer",
    "title": "Implementation context",
    "content": "Contract is GET /health → 200 OK. Keep it dependency-free.",
    "conversation_id": "release-2026-04"
})

# Implementer reads inbox
await tool(session, "memory_inbox", {"agent_id": "implementer"})
```

### 📋 Tasks

Tasks are first-class objects with ownership, status, priority, and results. A planner creates tasks, assigns them to workers, and workers set results when done.

| Tool | Purpose |
|---|---|
| `task_create` | Create a task with title, description, priority |
| `task_assign` | Assign a task to an agent (triggers a `task_assigned` notification) |
| `task_update` | Update status, description, or priority |
| `task_set_result` | Record the output of a completed task |
| `task_get` | Read a single task |
| `task_list` | List tasks, optionally filtered by agent or status |
| `task_tree` | Render a recursive subtree rooted at a parent task |
| `task_delete` | Soft-delete a task |

### 🔔 Notifications

Agents discover work through a poll-based notification queue. The orchestrator polls each agent's queue and dispatches work when notifications arrive.

| Tool | Purpose |
|---|---|
| `notify` | Send a notification to an agent |
| `notifications_poll` | Read pending notifications for an agent |
| `notifications_ack` | Acknowledge a single notification |
| `notifications_ack_all` | Acknowledge all pending notifications |

### 💬 Conversation grouping

Tag related memories with a shared `conversation_id` to create a logical session boundary. This works across agents — a planner and implementer can both write to `conversation_id="release-2026-04"` and later search within that scope.

### ⏳ Bitemporal queries

Every memory records when it was created and when it was valid. The `as_of` parameter on `memory_search` enables time-travel queries — useful for debugging past decisions or reconciling conflicting reports across agents.

---

## 🔄 Workflow patterns

### ➡️ Turn-based (sequential handoff)

One agent finishes, then passes context to the next. Each agent reads the inbox, searches shared memory, does its work, and hands off to the successor.

```
Planner → (handoff) → Implementer → (handoff) → Reviewer
```

1. Planner writes requirements to `scope="org"`, creates a task, assigns it, and hands off context.
2. Implementer reads inbox, searches shared memory, writes private implementation notes to `scope="agent"`, and sets the task result.
3. Reviewer searches shared memory, verifies against the contract, and records approval.

### 🔀 Parallel (fan-out / fan-in)

Multiple agents work simultaneously on independent tasks, reading from the same shared memory.

```
Planner → assigns Task A to Agent 1
        → assigns Task B to Agent 2
                                    → Reviewer merges results
```

Agents read `scope="org"` for shared context while keeping private notes in `scope="agent"`. The reviewer searches across both scopes to verify consistency.

### 🌳 Hierarchical (task trees)

A parent task can have subtasks, forming a tree. Use `task_tree` to inspect the full hierarchy.

```python
parent = await tool(session, "task_create", {
    "agent_id": "planner",
    "title": "Ship release 2026-04",
    "priority": "high"
})

await tool(session, "task_create", {
    "agent_id": "planner",
    "title": "Implement /health endpoint",
    "parent_task_id": parent_id,
    "priority": "high"
})

await tool(session, "task_create", {
    "agent_id": "planner",
    "title": "Write documentation",
    "parent_task_id": parent_id,
    "priority": "medium"
})

# Inspect the full tree
await tool(session, "task_tree", {"root_task_id": parent_id})
```

### 📝 Blackboard (shared knowledge base)

All agents contribute facts and observations to `scope="org"` asynchronously, without a predefined sequence. Any agent can search the shared pool at any time. This is useful when agents are loosely coupled — each contributes what it knows, and others consume what they need.

#### Concurrency & scale

Concurrent writes from multiple agents do **not** fail on lock contention. Every SQLite connection runs in **WAL mode** (concurrent readers alongside a writer) with a **30-second `busy_timeout`**, a connection pool, and a write-path retry — so simultaneous writers serialize and wait rather than erroring. WAL is verified at init; M3 raises rather than silently running in a slower journal mode.

For **high-concurrency fleets** where many agents write to one shared pool continuously, there are two paths:

1. **PostgreSQL as the shared primary store** (`M3_DB_BACKEND=postgres` + `M3_PRIMARY_PG_URL`): every agent reads and writes the *same* Postgres database directly — no per-agent local store, no sync lag. Postgres has no single-writer constraint, so concurrent writers don't serialize the way one SQLite file does. This is the simplest topology when the agents share infrastructure.
2. **Local SQLite + bidirectional warehouse sync** (`bin/pg_sync.py`): each agent writes locally to its own WAL-mode SQLite store and syncs bidirectionally to a shared Postgres *warehouse* (`M3_CDW_PG_URL`). This suits agents that must keep working offline / air-gapped and reconcile later. See [SYNC.md](SYNC.md).

The warehouse (path 2) is a *sync tier*, distinct from using Postgres as the primary backend (path 1) — pick based on whether the fleet shares a live database or each agent owns a local one.

### ⚡ Reactive (notification-driven)

Agents react to events rather than following a fixed sequence. Add `task_completed` to the orchestrator's `notification_kinds` so a planner automatically wakes up when a subtask finishes. Agents can also use `notify` to broadcast custom signals.

---

## 🚀 Full example

The [`examples/multi-agent-team/`](../examples/multi-agent-team/) directory contains a complete, runnable orchestrator:

- **`team.yaml`** — declarative agent definitions with provider, model, role, capabilities, and tool allowlists
- **`orchestrator.py`** — polling loop that discovers work via notifications and dispatches to agents
- **`dispatch.py`** — provider-agnostic multi-turn MCP dispatch with bounded execution (max turns, tool calls, wall clock, loop detection)

```bash
pip install -e .
m3-team init team.yaml
m3-team check
m3-team run
```

Any MCP client (Claude Code, Gemini CLI, etc.) can queue work:

```
task_create("Summarize the README", created_by="you")
task_assign(<task_id>, "local-agent")
```

The agent picks it up on the next tick, executes tool calls against m3-memory, and writes results back.

See [`examples/multi-agent-team/README.md`](../examples/multi-agent-team/README.md) for provider setup, resilience knobs, and how to add agents without code changes.

---

## 🧭 Design principles

**Memory is the coordination layer.** Agents don't talk to each other directly — they read and write shared memory. This decouples agent execution from agent communication.

**Scope is the access control.** Private scratch work stays in `scope="agent"`. Shared decisions go to `scope="org"`. User data goes to `scope="user"` with GDPR primitives (`gdpr_export`, `gdpr_forget`) attached.

**The orchestrator is pluggable.** M3 Memory exposes primitives, not opinions about scheduling. The bundled `m3-team` is one orchestrator; you can build your own with the same tool catalog.
