# Agent Loop

> The per-turn control-flow hub: how an inbound message becomes a state-machine
> turn, runs the LLM-plus-tools iteration, persists the result, and publishes a
> reply — channel-agnostically, one turn at a time per session.
>
> Sibling internals docs: [memory/00_overview.md](memory/00_overview.md) for the
> memory subsystem, [skills/00_overview.md](skills/00_overview.md) for skills,
> [concurrency.md](concurrency.md) for the cross-process locking model,
> [observability.md](observability.md) for telemetry, [ux.md](ux.md) for the
> CLI/TUI/webui surfaces.

---

## 1. Purpose

The agent loop is the engine that turns a message into a response. Every channel
— CLI, Telegram, Slack, the webui, cron — speaks to it through one contract: it
publishes an `InboundMessage` to a queue and reads back `OutboundMessage`s. The
loop owns everything in between.

For each inbound message it:

1. routes the message to the right session and serializes it against any turn
   already running for that session,
2. drives the message through a fixed per-turn state machine
   (restore → compact → command → build → run → save → respond),
3. delegates the LLM-and-tools iteration to a shared `AgentRunner`,
4. applies permission-as-data agent modes to the tool surface,
5. persists the turn to the session transcript and schedules background memory
   consolidation,
6. assembles and publishes the reply.

It exists so that the rest of the system — channels, tools, providers, memory —
does not have to know about turn ordering, persistence, compaction, or
concurrency. Those concerns live here, behind the bus.

The core type is `AgentLoop` in
[`durin/agent/loop.py`](../../durin/agent/loop.py). A single `AgentLoop`
instance is shared by the whole gateway and handles every session concurrently.

---

## 2. Mental model

Three ideas explain almost everything the loop does.

**1. A per-turn state machine.** Each turn walks eight states —
`RESTORE → COMPACT → COMMAND → BUILD → RUN → SAVE → RESPOND → DONE` — defined by
the `TurnState` enum. Transitions are data, not branches: each state handler
(`_state_restore`, `_state_compact`, …) returns an *event* string, and the
`_TRANSITIONS` table maps `(state, event)` to the next state. The one branch
that matters is in `COMMAND`: a matched shortcut command emits `"shortcut"` and
jumps straight to `DONE`, skipping the build/run/save states; a non-command
emits `"dispatch"` and continues to `BUILD`.

**2. Session-scoped serial execution, cross-session parallel.** Turns for the
*same* session run one at a time; turns for *different* sessions run
concurrently. This is enforced by two layers: an in-process `asyncio.Lock` per
`session_key` (cheap, same-process) and a cross-process turn lease
(`session_turn_lease`, a `.turn.lock` flock) that prevents the gateway and a TUI
or cron run from executing the same session at once. While a turn is running,
follow-up messages for that session do not spawn a competing task — they are
routed to per-session pending queues. Routing is two-tier: explicit steers
(`metadata["steer"]`, or a literal `[steer]` prefix normalized by the consumer)
and system-origin results (subagent / background-workflow completions) go to an
*inject* queue that enters the running turn at the next checkpoint; plain user
messages go to a *deferred* queue and only enter the conversation after the
turn's final response, so typing mid-turn queues instead of derailing the work
in flight.

**3. The message bus is the only channel contract.** Channels push
`InboundMessage` to `bus.inbound` and read `OutboundMessage` from `bus.outbound`
([`durin/bus/queue.py`](../../durin/bus/queue.py),
[`durin/bus/events.py`](../../durin/bus/events.py)). `MessageBus` is two plain
`asyncio.Queue`s with no routing logic. The loop never imports a channel; a
channel never imports the loop. The session a message belongs to is derived from
the message itself (`InboundMessage.session_key` = `session_key_override` or
`"channel:chat_id"`).

---

## 3. Diagrams

### 3.1 The per-turn state machine

```mermaid
flowchart TD
    In([InboundMessage]) --> Restore["RESTORE<br/>extract media/docs,<br/>recover interrupted turn"]
    Restore -->|ok| Compact["COMPACT<br/>read pending summary"]
    Compact -->|ok| Command["COMMAND<br/>router.dispatch"]
    Command -->|dispatch| Build["BUILD<br/>history + context + prompt"]
    Command -->|shortcut| Done([DONE])
    Build -->|ok| Run["RUN<br/>AgentRunner: LLM + tools"]
    Run -->|ok| Save["SAVE<br/>append turn to .jsonl,<br/>schedule consolidation"]
    Save -->|ok| Respond["RESPOND<br/>assemble OutboundMessage"]
    Respond -->|ok| Done
    Command -.->|"shortcut persists via<br/>_persist_user_message_early<br/>+ early session save<br/>(NOT the SAVE state)"| Save
```

The dotted edge is a note, not a transition: a shortcut command still writes the
turn to the session — it just does it inline in `_state_command` rather than
through the `SAVE` state. `/new` is the one exception (it clears the session, so
it deliberately persists nothing).

### 3.2 Concurrency topology

```mermaid
flowchart LR
    Ch[Channels] -->|publish_inbound| Inb["bus.inbound<br/>(asyncio.Queue)"]
    Inb --> RunLoop["AgentLoop.run()<br/>consumer"]
    RunLoop -->|"is_priority?<br/>(/stop, /restart, /status)"| Prio["dispatch_priority<br/>(no lock)"]
    RunLoop -->|"session has<br/>pending queues?"| Inject["route to _pending_queues[key]<br/>(steer/system → inject,<br/>user → deferred)"]
    RunLoop -->|"new session turn"| NewTask["register _pending_queues[key]<br/>then create_task(_dispatch)"]
    NewTask --> Disp["_dispatch(msg, pending)"]
    Disp --> Lock["asyncio.Lock[session_key]<br/>+ interactive lane + ceiling<br/>(ResizableSemaphore x2)"]
    Lock --> Lease["turn lease<br/>(.turn.lock flock)"]
    Lease --> SM[per-turn state machine]
    Inject -.->|"drained by runner<br/>injection callback"| SM
    SM -->|publish_outbound| Outb["bus.outbound<br/>(asyncio.Queue)"]
    Outb --> Ch2[Channels]
```

`run()` is a single consumer. For each message it picks exactly one path:
dispatch a priority command without any lock, route a follow-up into an existing
session's pending queues, or spawn a new `_dispatch` task. The queues are
registered *before* `create_task` so a same-session message that arrives in the
gap cannot spawn a competing task.

### 3.3 Session lifecycle within a turn

```mermaid
sequenceDiagram
    participant D as _dispatch
    participant L as turn lease (.turn.lock)
    participant SM as SessionManager
    participant ST as state machine
    participant C as Consolidator

    D->>D: acquire asyncio.Lock + interactive lane + ceiling
    D->>L: acquire turn lease (inside the lock)
    L-->>D: held (or TimeoutError -> "session busy" notice)
    D->>SM: reload(session_key)  (load-per-turn from disk)
    SM-->>D: fresh Session
    D->>ST: run RESTORE..RESPOND
    ST->>SM: save(session)  (.jsonl rewrite + .meta.json sidecar)
    ST->>C: schedule maybe_consolidate_by_tokens (background)
    D->>L: release turn lease (finally)
    D->>D: drain leftover pending back to bus.inbound
```

---

## 4. How it works

### The consumer: `run()`

`AgentLoop.run()` is one `while`-loop that consumes `bus.inbound`. On startup it
connects configured MCP servers (lazily, once) and warms the memory embedding
model in the background. For each message it decides the routing in order:

- **Priority command?** `commands.is_priority(raw)` matches the exact-match,
  no-lock tier (`/stop`, `/restart`, `/status`). These are dispatched
  immediately via `_dispatch_command_inline` so `/stop` can cancel a running
  turn — it never queues behind the lock.
- **Pending answer?** If a turn is blocked on `ask_user_question`, a plain-text
  reply is consumed as the answer (`_maybe_resolve_pending_answer`).
- **Mid-turn follow-up?** If the effective session key already has pending
  queues, the message is routed there instead of starting a new turn (or, if it
  is itself a non-priority command, dispatched inline). Steers and system
  results go to the inject queue; plain user messages go to the deferred queue,
  and the sender's surface is told (a `message_queued` websocket ack drives the
  webui chip; the CLI/TUI print or toast the notice).
- **New turn.** Otherwise the loop registers fresh pending queues for the
  session and `create_task(self._dispatch(msg, pending))`. The task is tracked
  per session so `/stop` can find and cancel it.

The effective session key (`_effective_session_key`) collapses to a single
unified key when `unified_session` is enabled and the message carries no
override.

### The turn: `_dispatch`

`_dispatch` is where serialization happens. It acquires, in order, the
per-`session_key` `asyncio.Lock`, the **interactive lane** (`self._interactive_lane`,
a `ResizableSemaphore` sized by `agents.defaults.max_concurrent_interactive` —
`DURIN_MAX_CONCURRENT_REQUESTS` overrides it live), the **global ceiling**
(`self._ceiling`, sized by `agents.defaults.concurrency_ceiling`), and then —
*inside* the lock — the cross-process turn lease via `session_turn_lease`. The
lease is acquired inside the lock so an in-process turn never even attempts the
flock while a sibling task holds the same lock; if the lease times out (another
*process* holds the session), the turn returns early with a "session is busy"
notice. With the lease held, it calls `sessions.reload(session_key)` —
load-per-turn — so the turn always sees the freshest on-disk state rather than
a stale cached `Session`.

The ceiling is shared with `SubagentManager`, which acquires it around each
subagent's LLM run (`_run_subagent`) — so subagents count against the same
global cap as interactive turns rather than running unbounded alongside them.
Subagents remain fire-and-forget: `spawn()` schedules `_run_subagent` via
`asyncio.create_task` and returns immediately, so a parent turn holding the
lane/ceiling while spawning a subagent never waits on that subagent — only the
subagent's own later ceiling acquire can block, which resolves once any
in-flight turn releases its slot. `reload_app_config` re-applies the caps live —
the interactive lane and the ceiling via `set_limit()`, and the sub-agent cap by
re-reading `max_concurrent_subagents` — so a `ConfigService` write to a cap key
takes effect without a process restart. See [Concurrency](concurrency.md) for the
full lock-ordering picture.

By default a spawned subagent inherits the parent session's model/provider
(the same pinning-at-spawn-time snapshot described above). Setting
`agents.aux_models.subagents` (an aux-model handle — `preset` or inline
`model`/`provider`, same shape as `aux_models.vision`/`audio`/`memory`) runs
subagents on a different model instead, e.g. a cheaper one for fire-and-forget
background work. `SubagentManager` resolves it fresh on every `spawn()` call
via a live `app_config` getter from the loop, so a hot-reloaded change applies
to the next spawn without a restart. Resolution failure (bad preset, missing
key) logs a warning and falls back to the inherited session model — spawning
must never break on a misconfigured aux model.

It then runs `_process_message`, publishes the result, and in a `finally` block
releases the lease and re-publishes any messages still sitting in the pending
queue back onto `bus.inbound` so a late follow-up is processed as a fresh turn
rather than lost.

### The state loop: `_process_message`

`_process_message` builds a `TurnContext` (mutable per-turn state — message,
session key, history, messages, callbacks, the pending queue, a diagnostic
`trace`) and runs the state machine. The driver is generic: it looks up
`_state_<name>`, calls it, records a `StateTraceEntry` (state, duration, event,
error), and follows `_TRANSITIONS[(state, event)]` until it reaches `DONE`. A
missing handler or an unmapped `(state, event)` raises immediately — the table
is the spec.

The handlers, in order:

- **`_state_restore`** — splits documents out of media, then materializes any
  interrupted turn. A crashed or `/stop`-ped turn leaves a `runtime_checkpoint`
  (mid-turn tool state) and/or a `pending_user_turn` flag in session metadata;
  `_restore_runtime_checkpoint` / `_restore_pending_user_turn` fold those into
  history so the conversation is consistent before the new turn.
- **`_state_compact`** — reads the consolidator's archived-summary marker
  (`_format_pending_summary`) so the build step can prepend it.
- **`_state_command`** — runs `commands.dispatch`. If a handler matches it
  persists the user message (`_persist_user_message_early`) plus the command's
  reply (both tagged `_command` so they are filtered out of LLM history),
  saves the session, and returns `"shortcut"` → `DONE`. Otherwise `"dispatch"`
  → `BUILD`.
- **`_state_build`** — runs `maybe_consolidate_by_tokens` (compacting before
  building so the prompt fits), sets the per-tool request context, slices
  history (`session.get_history`), and assembles the LLM message list via
  `context.build_messages`. It also persists the user message early so an
  interrupted run is recoverable.
- **`_state_run`** — calls `_run_agent_loop`, which delegates to
  `AgentRunner.run`. The result tuple
  `(final_content, tools_used, all_messages, stop_reason, had_injections, tool_events)`
  is stored on the context. An overflow that aborted *before any tool ran*
  (the consolidator's trigger ceiling is held strictly under the runner's input
  budget, so a successful BUILD consolidation always fits — an iteration-0
  overflow means it failed) triggers one bounded retry: force a fresh
  consolidation, rebuild the context, re-run
  (`overflow_retry.forced_consolidation`); skipped once a tool has run, so
  side-effecting tools never re-fire.
- **`_state_save`** — finalizes plan/stall/goal bookkeeping, records skill-usage
  signals, appends only the new turn's messages to the session
  (`_save_turn` rewrites the `.jsonl` and mirrors derived/volatile metadata to
  the `.meta.json` sidecar), then schedules a background
  `maybe_consolidate_by_tokens`. A tool result too large for the persisted
  transcript is spilled to a recoverable file *before* it is truncated, so the
  truncated text left in the transcript carries a pointer back to the full
  output (`read_file` recovers it) instead of losing it.
- **`_state_respond`** — assembles the `OutboundMessage` (`_assemble_outbound`),
  suppressing it when the turn already streamed its answer through the
  `message` tool.

### The iteration core: `AgentRunner`

`_run_agent_loop` is the bridge from loop to runner. It builds the hook,
resolves the agent-mode provider, the compaction-grace probe, and any per-turn
model override, then calls `self.runner.run(AgentRunSpec(...))`.

`AgentRunner` ([`durin/agent/runner.py`](../../durin/agent/runner.py)) is the
shared, product-agnostic LLM loop. It iterates up to `max_iterations` (200 by
default): call the LLM → if the response has tool calls, execute them (with
topological batching) and loop; otherwise finalize the content and stop. Around
that core it layers guards and context governance — loop detection on repeated
failed calls, an unknown-tool breaker, an idle-timeout breaker, message
sanitization (dropping orphan tool results, backfilling missing ones),
per-result and per-turn tool-output budgets with spill-to-disk, and microcompact
/ media pruning of the in-flight message list. Crucially these reshape only the
copy sent to the model — the persisted transcript is untouched.

**Context budget.** The input budget reserves only a *capped* output headroom
(`_output_reservation`, not the full configured `max_tokens` ceiling), so a high
ceiling never collapses the usable input; the request then sends a *dynamic*
`max_tokens` sized to the room the prompt actually leaves (resolving the ceiling
from the provider when the spec leaves it unset). A mid-turn precheck estimates
the post-sanitize prompt each iteration: when it is over budget the runner
emergency-trims the largest string tool results on the model-facing copy and
proceeds if that fits (`mid_turn_precheck.recovered`); only when trimming can't
recover does it abort *before* the LLM call with
`stop_reason=mid_turn_precheck_overflow` and an overflow-specific placeholder.

**Microcompaction** is gated on pressure: results beyond the most recent
`_MICROCOMPACT_KEEP_RECENT` are only collapsed when the estimated prompt
already exceeds `_MICROCOMPACT_PRESSURE_RATIO` of the input budget, so a
turn with headroom to spare keeps stale tool results in full — they may
still hold the answer to a question the user hasn't asked yet. A second gate
is economic: rewriting a message invalidates every cached prompt prefix from
that point on, and this pass runs on every iteration, so a pass that could
reclaim less than `_MICROCOMPACT_MIN_RECLAIM_CHARS` in aggregate leaves the
messages alone rather than force a cache write that costs more than the freed
context is worth. When it does fire, the placeholder is informative rather
than opaque: it names the tool,
quotes a short head snippet of what the output began with (omitted when the
content is already a persisted reference, since its head is marker
boilerplate), and points at the recoverable file via `read_file`.

Two behaviors connect the runner back to the loop:

- **Mid-turn injection.** The runner calls the loop's `_drain_pending` callback
  at two kinds of checkpoint. After each tool batch it drains with
  `steer_only=True`: only the inject queue (steers, framed as mid-work
  guidance, plus sub-agent / workflow results) enters the running turn. After
  the final response it drains both queues — deferred user messages last, so
  the model answers them with all results already in context — and websocket
  clients get a `queued_consumed` ack. Drained messages are appended as user
  turns so the run continues without a new dispatch. Injection is bounded — at
  most `_MAX_INJECTIONS_PER_TURN` messages drained per cycle and
  `_MAX_INJECTION_CYCLES` cycles — so an injection chain cannot run forever.
- **Per-turn provider snapshot.** `AgentRunSpec.provider` carries the provider
  resolved for *this* turn. The gateway shares one runner, and a concurrent
  session's `/model` swap mutates `self.provider`; pinning the provider on the
  spec makes the turn immune to that mid-flight swap.

### Hooks

Every turn wires at least one hook. `_run_agent_loop` always constructs an
`AgentProgressHook`
([`durin/agent/progress_hook.py`](../../durin/agent/progress_hook.py)) for
streaming deltas, tool hints, reasoning, iteration counting, and cache-usage
capture. If the loop was built with extra hooks, they are wrapped together with
the progress hook in a `CompositeHook`
([`durin/agent/hook.py`](../../durin/agent/hook.py)), which fans out to each hook
with per-hook error isolation so a faulty hook cannot crash the turn.

### Personas & SOULs

#### SOUL library

A *SOUL* is a personality document that replaces the default `SOUL.md` in the
system prompt. `SoulStore` (`durin/souls/store.py`) is the file-backed library:
the `default` soul maps to the workspace-root `SOUL.md` (kept there for
backward compatibility and git-tracking); every other soul lives under
`workspace/souls/<slug>.md`. Souls are plain markdown — readable and editable
without any tooling.

On a fresh workspace, a set of example souls is pre-seeded under
`workspace/souls/` (`researcher`, `engineer`, `tutor-feynman`,
`tutor-socratic`). They back the seeded example personas and serve as templates
for user-defined souls.

#### Persona records

A *persona* pairs a soul with an optional model. `PersonaConfig` in
`durin/config/schema.py` has three fields: `soul` (a `SoulStore` slug; omit to
use `"default"`), `model` (a model picker ref — a preset name or
`"provider model"` string — or `None` to inherit the global default model), and
`description` (free text shown in `/persona` listings).

Personas live in the `personas` map in `config.json`, structurally identical to
`model_presets`. On first run, the example personas (`researcher`, `engineer`,
`tutor-feynman`, `tutor-socratic`) are seeded into that map
(`durin/personas/builtin.py`'s
`seed_example_personas`, guarded by the `agents.defaults.personas_seeded`
marker) as ordinary entries — fully editable and deletable; a user who removes
one keeps it removed across restarts (the marker prevents re-injection).

`Config.resolve_persona(name)` looks the name up in the `personas` map and
returns `None` when it is unknown (the caller falls back to the default SOUL and
default model). `Config.persona_names()` lists the configured personas. The
persona listing (`GET /api/v1/personas` and the webui pane) additionally
surfaces a synthetic `durin` entry — the base SOUL plus the default model —
last in the list, so the implicit fallback is visible and selectable under its
real name instead of an anonymous "default"; that synthetic entry alone is not
editable or deletable (`durin`/`default`/`none` are reserved persona names —
the latter two remain accepted as reset keywords for compatibility).

Example user config:

```json
{
  "personas": {
    "acme": {
      "soul": "researcher",
      "model": "fast",
      "description": "Research mode for Acme project"
    }
  }
}
```

#### Selection and precedence

The active persona for a turn is resolved once in `_state_build` by
`resolve_active_persona_name` (`durin/personas/resolve.py`):

1. **Cron job** — `job.payload.persona`, set per job in the cron panel or the
   `cron` tool; applies only to that job's run (mutually exclusive with the
   job's per-run model).
2. **Per-conversation** — `session.metadata["persona"]`, set by `/persona
   <name>` and cleared by `/persona durin` (`default`/`none` also work as reset
   keywords).
3. **Channel** — the inbound message's channel section:
   `channels.<name>.chat_personas[chat_id]` (per-chat) first, then
   `channels.<name>.persona` (the channel-wide default), so a transport can
   carry its own identity and refine it per conversation.
4. **Global default** — `agents.defaults.persona` in config (applies to every
   new conversation until overridden).
5. **No persona** — default `SOUL.md` and global default model, unchanged from
   pre-persona behavior.

The resolved persona's soul body replaces the default SOUL in the stable system-
prompt layer (via `ContextBuilder`). Its model ref is passed to the existing
per-turn model-override path alongside any explicit `/model` or cron per-job
model; the most-specific reference wins (`ctx.model_preset_override or
ctx.persona_model_ref`) — an explicit `/model` switch always overrides the
persona's model.

#### Surfaces

Souls and personas are manageable through three surfaces:

- **`/persona` slash command** — switches the active persona for the current
  conversation; `/persona durin` reverts to the base persona (`default`/`none`
  are also accepted as reset keywords).
- **`agents.defaults.persona` in `config.json`** — sets the global default persona
  applied to every new conversation.
- **Webui Personas settings section** — a SOUL library editor (create, edit,
  delete named soul files) and a Persona definitions panel (create/update
  personas with model picker and soul picker, delete user-defined personas, set
  or clear the global default). Backed by `PersonasService` (`durin/service/personas.py`)
  via the `/api/v1/souls` and `/api/v1/personas` route groups (see
  [api.md](api.md)).

  Two additional behaviors in this surface:

  - **Live model + SOUL test** — the persona form *and each persona row* (so
    any persona — including the seeded examples and the synthetic `durin` base
    persona — can be tested straight from the list) carry a "Test" action (`POST /api/v1/personas/test`) that
    runs the selected soul and model against a fixed short prompt and returns
    the model's reply. If the provider or model reference is invalid, the
    response carries an `ok: false` error message instead of raising an HTTP
    error; the webui shows it inline (below the form, or under the row).

  - **SOUL deletion** — every soul can be deleted except `default` (the
    workspace `SOUL.md`). A persona that still references a deleted soul falls
    back to the default SOUL at runtime (`SoulStore.read` returns empty →
    `_active_persona` yields `None` → the default SOUL is used), so deleting a
    referenced soul never breaks the agent. The webui still shows an "in use by
    N" hint on a referenced soul as a heads-up, but allows the delete.

#### Operating floor

durin's execution rules — tool-use discipline, planning hygiene, and operational
posture — live in `durin/templates/agent/operating_floor.md` and are injected
into the stable system-prompt layer by `ContextBuilder._build_operating_floor`
independent of which SOUL is active. Swapping a SOUL never removes these rules.
The only exception is backward compatibility: if the active SOUL already contains
a `## Execution Rules` section (a pre-existing workspace `SOUL.md` with embedded
rules), the floor injection is skipped to avoid duplication.

### Agent modes (permission-as-data)

Modes ([`durin/agent/agent_mode.py`](../../durin/agent/agent_mode.py)) are pure
data: a `name`, an optional `allowed` allowlist, a `denied` set, and a
`prompt_suffix`, plus an optional `icon` and a `builtin` flag carried for the
UI. The loop has no `if plan_mode` branches. Instead the runner
calls a `mode_provider` callback each iteration to read the active mode (from
`session.metadata["agent_mode"]`) and filters the tool definitions sent to the
LLM. Because it is read per iteration, a mid-run `/plan` or `/build` takes effect
on the very next iteration. If the model emits a tool call for a denied tool
anyway (e.g. a cached name), `_run_tool` returns a synthetic "not available in
this mode" result — tool-by-tool denial, not a stopped run. Built-ins:
`build` (full access), `plan` (read-only planning), `explore` (read-only, for
sub-agents), and `read` (read-only with a neutral posture — no interactive
framing, for workflow nodes that inspect or judge). `explore` and `read` share
one read-only tool surface by design; only their `prompt_suffix` posture differs.

The restricted built-ins carry **hand-curated** allowlists (only `build` grows
automatically as new tools are added). `plan` allows read-only investigation plus
project-memory recall and capability discovery (skills/workflows), so it can
plan against what already exists; `explore`/`read` allow read-only investigation
plus memory recall. A new tool does **not** join these allowlists on its own — the
list is maintained deliberately, because "side-effect-free" (`read_only`) is not
the same as "appropriate for a restricted mode" (e.g. the secret-read tools are
`read_only` yet excluded).

Two gates apply to a read-only mode's effective surface: the mode allowlist AND
the tool's `_scopes` (see [tools.md](tools.md)). A subagent or workflow node loads
only `subagent`-scoped tools, then the mode allowlist filters that set — so a tool
must carry the `subagent` scope to be reachable there at all (the memory-recall
tools do, which is why `explore`/`read` can surface them).

The registered modes are listed over `/api/v1/modes` (`ModesService`), which the
webui composer's mode picker renders by `name`. The picker is mode-agnostic: it
shows whatever the registry holds, so it follows new modes without UI changes. The
same service exposes `GET /api/v1/tools` — the catalog of tools a mode allowlist
can reference (name, description, `read_only`, built-in vs MCP, plus `background`:
whether an entry can ever apply to a sub-agent or workflow node — built-ins need
the `subagent` scope, MCP tools always qualify because a node opts into them via
its `mcps` field), read from the live loop's registry so it matches exactly what
the agent can call. The settings editor uses it to offer a checklist of the real
tool set (rather than free-text tool names), to surface read-only tools left out
of an allowlist, and to badge background-capable tools so a custom mode's author
can see which entries will reach workflow nodes and sub-agents.

User-defined modes are persisted as `ModeConfig` entries under `agent_modes` in
config and registered into the same registry at startup (`register_config_modes`,
which never shadows a built-in). The same `ModesService` route handles
`POST`/`DELETE` to create, edit, and remove them (the settings UI), re-registering
after each mutation so a change takes effect without a restart; the three
built-ins are immutable.

### Default model (live-applied from settings)

Changing `agents.defaults.model`/`provider` through the settings UI
(`POST /api/v1/settings`, `SettingsService`) applies to the running loop without a
restart. After saving config the service calls its `on_default_changed` hook,
which the gateway binds to `AgentLoop.apply_default_model_live`
(`durin/agent/loop.py`). That method reloads the config snapshot
(`reload_app_config`), which rebuilds the always-present `default` preset from
the new `agents.defaults`, then applies the new default so the change is live on
the next turn. The apply path depends on the provider:

- For an **explicit provider** the `"provider model"` picker pair is resolved
  through the same raw-pair-safe path the `/model` command uses —
  `resolve_preset_ref` (registers an ad-hoc preset) then `set_model_preset`.
- For `provider == "auto"` (the schema default, and what the settings provider
  picker emits when left empty) it applies the rebuilt `default` preset directly,
  the same path cold-start uses. A bare model string is not a registered preset,
  so a synthetic `provider model` ref would raise `KeyError` out of
  `set_model_preset`.

An unresolvable ref logs a warning and no-ops rather than escaping the
`on_default_changed` handler. A local provider (ollama and friends) is accepted
as the default when its `api_base` is set, not by `api_key`.

### Sessions

A `Session` ([`durin/session/manager.py`](../../durin/session/manager.py)) is an
append-only message list plus `metadata` and a `last_consolidated` cursor.
`SessionManager` persists it as a JSONL file: line 0 is the identity metadata
header (mode, todos, plan path, channel ownership), and the rest are messages.
Writes are atomic (`tmpfile` + `os.replace`) under a cross-process lock.

**The live file is bounded; the history is not destroyed.** Every save rewrites
the whole `.jsonl` (and regenerates the `.md` projection), and the full message
list lives in RAM — so the live file must stay bounded, which is what
`enforce_file_cap` does when the list crosses its message limit. The trimmed
prefix is not deleted, though: it is appended to per-session **archive
segments** (`sessions/archive/<key>.NNNNNN.jsonl`, plain message lines in the
live file's serialization). Segments are append-only and never rewritten, so
retention costs disk only — no per-turn cost. They rotate at a size bound and
the per-session total is capped as insurance, pruning oldest-first but never
the segment just written. Writes are serialized under a per-session
cross-process lock (segment selection is a read-modify-write; two processes
can legally own turns of the same workspace). The sink is best-effort: if the
archive write fails, the trim still proceeds — the availability of the live
session outranks retention, and a disk that fails the archive is failing
`save()` too. `session_search` transparently continues from the live list into
these segments (newest first, byte-budgeted, off the event loop), labelling
archived hits with their timestamp instead of a live index, since indexes are
rebased on every trim.

Line 0 also carries a **preview**: the text of the most recent user message,
which is what labels the session's row in the WebUI list. It is stored rather
than derived at read time because the row is sorted and dated by `updated_at`,
so its label has to describe the same moment — and computing that from the body
would mean reading every listed transcript on every refresh, on an endpoint the
sidebar re-hits whenever any session changes. Storing it is free, since `save()`
rewrites line 0 regardless. Sessions written before the field existed are read
from the tail of the file instead, and heal on their next save.

The distinction matters most for long-lived keys. A Slack channel session
(`slack:<channel>`, no thread timestamp) accumulates for weeks, so labelling it
by its *first* message left it under "Today" wearing whatever opened it a month
earlier.

Two metadata splits matter:

- **Derived vs identity.** `_DERIVED_METADATA_KEYS` (LLM-projected state —
  consolidation's entity/topic tag accumulation, skill-usage counts, and the
  legacy `_last_summary` field kept only for pre-migration sessions) and
  volatile per-turn keys (`runtime_checkpoint`, `pending_user_turn`) are
  written to the sibling `.meta.json` sidecar, not line 0. On load,
  `_merge_derived_from_sidecar` folds them back so consumers see one flat
  `metadata` dict. Keys *not* in those sets ride line 0 and survive
  compaction (the todo list, plan paths, goal state, mode, and the decision
  log — see below).
- **Compaction never edits messages in place.** When the consolidator archives
  a span it advances `last_consolidated` and appends the span's summary as a
  new block onto the session-summary projection (bounded; oldest blocks
  evicted as the cap is hit, their discovered-path trailers salvaged into a
  synthetic head block rather than lost) — it never mutates
  `session.messages`. `get_history` always returns `messages[last_consolidated:]`,
  so the model sees the unconsolidated tail and the raw transcript stays intact
  for recovery. See [memory/01_data_and_entities.md](memory/01_data_and_entities.md)
  for the summary's on-disk shape and
  [memory/06_prompts_and_instructions.md](memory/06_prompts_and_instructions.md)
  for what the archive prompt extracts.
- **The decision log resists eviction pressure differently than the summary.**
  `session.metadata["decision_log"]` is a line-0 identity key (not derived),
  so it rides every save and survives compaction untouched by the sidecar
  split. Each compaction round that advances the cursor — the primary
  token-triggered loop, the replay-window overflow path, and early-return
  compactions alike — runs a best-effort LLM extraction over the span just
  archived and appends any decisions/findings found. When the log's
  entry/char caps are hit, auto-extracted entries are evicted first;
  a manually authored (`note_decision`) entry is only dropped once no
  auto-extracted entry remains to take its place. Two rules keep that priority
  from silently swallowing writes: the entry just appended is never the one
  evicted (otherwise an `auto` entry that overflows the char cap is the only
  `auto` present and evicts *itself*, making the write a no-op), and an `auto`
  append that could only fit by evicting a manual anchor is rejected instead —
  still counted as a drop, so `decision_log.capped` records the loss.
- **A finished goal still leaves a trace in the anchor.** A session rarely ends
  when its goal does, and rendering only `status == "active"` erased the
  session's stated purpose the moment it succeeded — work continued with no
  objective in context at all. A completed goal now renders a compact
  `Goal (completed)` / `Outcome` pair (capped, `ui_summary` preferred over the
  full objective), and `complete_goal` additionally folds objective and recap
  into the decision log so they survive a later `long_task` overwriting the
  blob. Neither path touches `sustained_goal_active`, which gates the runner's
  wall-clock backstop and must stay false once a goal is done.

#### Compaction thresholds

Three numbers govern when the consolidator fires, and they are deliberately
distinct:

| Number | Formula | What it bounds |
|---|---|---|
| `_input_token_budget` | `window − max_completion_tokens − buffer` | Size of the text handed to the consolidation LLM. Reserves the *real* completion ceiling, because that call has to fit too. |
| `_preemptive_ceiling` | `window − min(max_completion_tokens, cap) − 2×buffer` | Hard upper bound on the trigger. Reserves only a *capped* output slice, mirroring the runner's `_output_reservation`, and stays strictly under the runner's input budget so the `_state_run` overflow invariant holds. |
| `_preemptive_trigger_tokens` | `min(window × effective_ratio, ceiling)` | Where compaction actually fires. |

Clamping the trigger against `_input_token_budget` instead of the ceiling is
what made `preemptive_compact_ratio` inert: a model whose catalog `max_tokens`
is a large fraction of its window (131,072 on a 231,072 window) pinned every
ratio above ~0.43 to the same trigger, and raising the knob changed nothing.

`effective_ratio` applies a **raise-only** floor below
`_SMALL_CTX_WINDOW_LIMIT`. On a small window the incompressible part of a
prompt (system + tool schemas + summary + task state) is a large fraction of
the whole, so a low ratio leaves almost no runway between the post-compaction
floor and the next trigger, and the session thrashes. An explicitly configured
higher ratio is always honoured; large-window models are untouched, where a
high ratio would mean shipping a huge prompt every turn.

**The real-usage veto.** `estimate_session_prompt_tokens` probes the *raw*
unconsolidated tail — it does not apply microcompaction or the tool-result
budget, which the runner does apply to what it ships. The estimate therefore
runs conservatively high by design, and a rough number over the trigger does
not prove the real prompt is over it. Two vetoes, both keyed on the provider's
own `usage_prompt_tokens` rather than the local estimator, are checked before
consolidating (`compaction.deferred` records either):

- `provider_fit` — the last real count came in under the trigger and the rough
  estimate has drifted less than the tolerance since. The estimate is measuring
  padding the provider never receives.
- `post_compaction` — a compaction just advanced the cursor and no LLM call has
  happened since, so the newest anchor still describes the *pre*-compaction
  prompt. Without this, reading that stale anchor fires a second compaction
  against an already-shortened conversation. Parked for exactly one turn.

### After DONE (post-processing in `_dispatch`)

Once the state machine returns, `_dispatch` publishes the outbound message,
serializes any pending interactive payloads for channels that cannot render
structured tool output, and for websocket clients emits a `_turn_end` signal
(carrying turn latency and goal state) and, for the webui, schedules background
title generation. It also emits a `turn.latency` breakdown — total wall-clock
split into `llm_ms` (model round-trips, accumulated in the runner and handed off
via `_pending_llm_ms`), `tools_ms`, and `local_ms` (everything else), plus the
per-state-machine durations from the `trace`. Finally it drains leftover pending
messages back to the bus and clears the per-session latency entry.

---

## 5. Key types & entry points

| Symbol | File | Role |
|---|---|---|
| `AgentLoop` | [`durin/agent/loop.py`](../../durin/agent/loop.py) | Core orchestrator; owns `run()`, `_dispatch`, the state handlers, and `from_config`. |
| `TurnContext` | [`durin/agent/loop.py`](../../durin/agent/loop.py) | Mutable per-turn state threaded through the state handlers. |
| `TurnState` | [`durin/agent/loop.py`](../../durin/agent/loop.py) | Enum of the eight turn states. |
| `StateTraceEntry` | [`durin/agent/loop.py`](../../durin/agent/loop.py) | Per-state diagnostic record (state, duration, event, error). |
| `MessageBus` | [`durin/bus/queue.py`](../../durin/bus/queue.py) | Two `asyncio.Queue`s decoupling channels from the loop. |
| `InboundMessage` | [`durin/bus/events.py`](../../durin/bus/events.py) | Channel input; derives `session_key`. |
| `OutboundMessage` | [`durin/bus/events.py`](../../durin/bus/events.py) | Agent reply with routing/trace metadata and optional buttons. |
| `CommandRouter` | [`durin/command/router.py`](../../durin/command/router.py) | Three-tier slash-command dispatch (priority / exact / prefix). |
| `CommandContext` | [`durin/command/router.py`](../../durin/command/router.py) | What a command handler receives (msg, session, key, raw, args, loop). |
| `AgentRunner` | [`durin/agent/runner.py`](../../durin/agent/runner.py) | Shared LLM-plus-tools iteration loop with guards and context governance. |
| `AgentRunSpec` | [`durin/agent/runner.py`](../../durin/agent/runner.py) | Per-run configuration, including the per-turn provider snapshot. |
| `Session` | [`durin/session/manager.py`](../../durin/session/manager.py) | Append-only conversation with metadata and a `last_consolidated` cursor. |
| `SessionManager` | [`durin/session/manager.py`](../../durin/session/manager.py) | Loads/saves sessions atomically; splits derived/volatile metadata to the sidecar. |
| `session_turn_lease` | [`durin/session/turn_lease.py`](../../durin/session/turn_lease.py) | Cross-process per-session turn lock held for the whole turn. |
| `ContextBuilder` | [`durin/agent/context.py`](../../durin/agent/context.py) | Builds the tiered system prompt and the LLM message list, including the active SOUL and operating floor. |
| `SoulStore` | [`durin/souls/store.py`](../../durin/souls/store.py) | File-backed SOUL library: `default` → `SOUL.md`, named souls → `souls/<slug>.md`. |
| `PersonaConfig` | [`durin/config/schema.py`](../../durin/config/schema.py) | A soul + optional model + description; lives in the `personas` config map. |
| `resolve_persona` / `persona_names` | [`durin/config/schema.py`](../../durin/config/schema.py) | Resolver (by name from the `personas` map) and listing method on `Config`. |
| `resolve_active_persona_name` | [`durin/personas/resolve.py`](../../durin/personas/resolve.py) | Precedence resolver: cron → per-conversation metadata → channel (chat map, then channel default) → global default → None. |
| `SEED_PERSONAS` / `seed_example_personas` | [`durin/personas/builtin.py`](../../durin/personas/builtin.py) | Example personas (`researcher`, `engineer`, `tutor-feynman`, `tutor-socratic`) seeded once into config on first run as ordinary editable/deletable entries. |
| `AgentMode` | [`durin/agent/agent_mode.py`](../../durin/agent/agent_mode.py) | Permission-as-data tool filter (build / plan / explore / read). |
| `AgentHook` / `CompositeHook` | [`durin/agent/hook.py`](../../durin/agent/hook.py) | Per-iteration lifecycle callbacks with fan-out and error isolation. |
| `AgentProgressHook` | [`durin/agent/progress_hook.py`](../../durin/agent/progress_hook.py) | The hook wired on every turn (streaming, tool hints, iteration count). |
| `Consolidator` | [`durin/agent/memory.py`](../../durin/agent/memory.py) | Memory archival; advances `last_consolidated` under a per-session lock. |
| `ToolRegistry` | [`durin/agent/tools/registry.py`](../../durin/agent/tools/registry.py) | Tool catalog and LLM-visible definitions; supports runtime MCP registration. |

---

## 6. Configuration & surfaces

### Config keys

Loop-relevant `agents.defaults.*` keys (see
[`durin/config/schema.py`](../../durin/config/schema.py)):

| Key | Default | Effect |
|---|---|---|
| `max_tool_iterations` | `200` | Hard cap on runner iterations per turn. |
| `max_messages` | `480` | History replay window (token-budget is the effective bound on large-context models). |
| `unified_session` | `false` | Collapse all channels to one shared session. |
| `consolidation_ratio` | `0.5` | How far each compaction round reduces the prompt. |
| `preemptive_compact_ratio` | `0.5` | Fraction of the window that triggers preemptive compaction. Clamped by the trigger ceiling and floored on small windows — see [Compaction thresholds](#compaction-thresholds). |
| `plan_stall_turns` | `8` | Turns of no todo progress on an executing plan before a "reassess" reminder (`0` disables). |
| `agents.defaults.persona` | `null` | Default persona name for interactive conversations. Overridden per-conversation via `/persona`. |
| `context_window_tokens`, `context_block_limit`, `max_tool_result_chars` | — | Token/size budgets used when building and persisting. |
| `max_concurrent_interactive` | `4` | Interactive-lane cap: human-facing turns in flight at once, across all sessions. `DURIN_MAX_CONCURRENT_REQUESTS` overrides this at runtime. |
| `concurrency_ceiling` | `12` | Global ceiling: total in-flight turns *and* subagents across all lanes (see [Concurrency](concurrency.md)). |
| `max_concurrent_subagents` | `3` | Process-wide subagent-lane cap, checked at spawn time (`spawn.py`); independent of the global ceiling above. |

### Environment knobs

Read at runtime (mostly in the runner / loop):

| Variable | Default | Effect |
|---|---|---|
| `DURIN_MAX_CONCURRENT_REQUESTS` | unset | Overrides `max_concurrent_interactive` (the interactive lane's cap) when set; `0` or negative = unlimited. |
| `DURIN_LLM_TIMEOUT_S` | unset | Wall-clock outer timeout per LLM request; an explicit value applies everywhere, `0` disables. Unset: `1800` backstop on natively-streaming providers (the idle watchdog is the primary hang detector), `300` on the rest. |
| `DURIN_STREAM_IDLE_TIMEOUT_S` | `90` | Provider idle-stall watchdog: seconds of stream silence before the request is declared stalled. `0` disables. Local endpoints default to disabled. |
| `DURIN_COMPACTION_GRACE_S` | `30` | One-shot deadline extension when consolidation is in flight. |
| `DURIN_MAX_CONSECUTIVE_IDLE_TIMEOUTS` | `1` | Idle-timeout circuit breaker (trips on the next timeout). |
| `DURIN_MAX_UNKNOWN_TOOL_ATTEMPTS` | `2` | Unknown-tool breaker (trips on the third call to a bad name). |
| `DURIN_TURN_BUDGET_CHARS` | `200000` | Aggregate per-turn tool-output budget (`0` disables). |
| `DURIN_HISTORY_IMAGE_PRESERVE_TURNS` | `3` | Keep media only in the most recent N turns. |
| `DURIN_HOME` | `~/.durin` | Workspace root for sessions, memory, and locks. |

### Surfaces

- **Slash commands** are registered in
  [`durin/command/builtin.py`](../../durin/command/builtin.py) via
  `register_builtin_commands`. Priority (no-lock) commands are `/stop`,
  `/restart`, `/status`. Mode/model controls relevant to the loop include
  `/plan`, `/build`, `/mode`, `/model`, `/effort`, `/new`, `/compact`.
  `/persona [name]` switches the active persona for the current conversation;
  `/persona durin` reverts to the base persona (the global default). Personas and souls are also
  managed through the webui Personas settings section — see the Surfaces
  subsection in "Personas & SOULs" above.
- **Bus** — any channel publishes/consumes through `MessageBus`; the loop is
  channel-agnostic. The CLI/TUI also use `process_direct` for a synchronous
  one-shot turn that mirrors `_dispatch`'s lease-and-reload semantics.
- **Webui/API** drive the same loop through the `websocket` channel, which adds
  streaming segments, the `_turn_end` signal, and background title generation.
  Generated titles are validated before persisting (reasoning models can leak
  meta text instead of a title); invalid output retries once, then falls back
  to the user's first message, and a stored implausible auto-title self-heals
  on the next turn.

---

## 7. Curated rationale

**Why session-scoped serial, cross-session parallel.** A single conversation
must stay linear — two turns interleaving the same transcript would corrupt
ordering and double-spend tokens. But unrelated conversations have no shared
state, so forcing them serial would waste the obvious parallelism of a
multi-channel agent. One lock per session key buys both: strict ordering where
it matters, full concurrency where it doesn't.

**Why the turn lease lives inside the in-process lock.** The `asyncio.Lock`
handles same-process contention cheaply and instantly; the cross-process flock
handles the rarer case of a second process (a TUI or a cron run) touching the
same session. Acquiring the flock only after the in-process lock is held means a
busy same-process session never pays the flock's blocking cost, and the lease
contends only across processes — exactly where it is needed.

**Why a per-turn provider snapshot.** The gateway shares one `AgentRunner` across
every concurrent session to keep connections and clients warm. That makes
`self.provider` shared mutable state, and a `/model` swap on one session would
otherwise change the provider under a turn already iterating on another session.
Pinning the provider onto the run spec converts shared state into a per-turn
value, so a model switch only affects turns that start after it.

**Why compaction advances a cursor instead of editing messages.** Treating the
transcript as append-only and immutable means an interrupted turn, a crash, or a
recovery read always has the full raw history to fall back on. The model sees a
compacted view through `get_history`, but the truth on disk is never destroyed —
the file cap moves old prefixes into append-only archive segments rather than
deleting them (see [Sessions](#sessions)) — which is also what lets the memory
subsystem reconstruct everything from the markdown source of truth.
