---
description: One pass of the loop in code terms — the check and submit phases, the single failure-attribution point, fan-out settlement, and selection.
---

# The orchestrator

`core/orchestrator.py` is one class with one interesting method. It takes plain
objects — stages in funnel order and a `{stage_name: backend}` mapping — and has
no idea a config file exists. {doc}`../explanation/scheduling-and-restart` covers
why it is shaped this way; this page is what the code does.

## Construction

The orchestrator derives its state machine at construction rather than
consulting config later. Mapped stage names take their output state from
`STAGE_OUTPUT_STATE`; unmapped names (test doubles) take the next chain state
positionally. Output states must advance strictly, so an out-of-order funnel is
a `ValueError` naming the problem instead of a run that quietly does the wrong
thing. Each stage's **input** state is the previous stage's output, with the
first stage consuming `NEW`.

Bad wiring fails fast, at construction: an unknown `only` stage, `only` combined
with `stop_after`, a stage with no backend.

## `run_pass()`

```text
release_stale_leases(lease_timeout_s)

check phase      for every recorded ledger job:
                     poll → terminal? → fetch
                     → interpret_result → validate_outputs
                     → complete / fail → release → clear_job

finalize gate    if this run goes through the configured last stage:
                     materials in the stop stage's output state → DONE

submit phase     for each active stage, in funnel order:
                     backlog gate → health gate
                     → claim(limit = max_parallel − live jobs)
                     → validate_inputs → build_jobspec → submit → record_job
```

Two orderings are deliberate. **Check before submit**, so free slots are counted
after finished work is cleared. **Backlog before health**, so a stage with
nothing waiting costs nothing at all.

A pass never raises. A `StageError` pauses that stage for the pass and fails no
material — a stage that cannot build jobs at all is not evidence about any
particular material.

Two smaller invariants worth knowing when reading the code: a job that cannot be
recorded in the ledger is released immediately, so nothing runs untracked; and a
job whose poll cannot be resolved is left recorded rather than dropped, so it is
retried next pass instead of leaking.

## `_fail` is the only attribution point

Every failure path in the pass funnels through one method, which is where the
verdict-vs-fault doctrine is applied exactly once:

| Cause | Treatment |
|---|---|
| `StageVerdict(terminal=True)` | Immediate terminal `FAILED`, budget 1 |
| `StageInputError` | Fault — spends `retry_budget` |
| Promised outputs missing | Fault |
| `BackendError` at submit | Fault |
| Anything unexpected | Fault |

Having one attribution point is why the rule can be stated as doctrine rather
than hoped for: there is a single place where "is this the science's answer or an
accident?" is decided.

## Fan-out settlement

For a `fan_out` stage, a finished job means a children manifest to settle:

1. Read and validate the manifest — unreadable, non-list, duplicated ids, unsafe
   ids (including path traversal), or a named child with no directory are all
   budgeted faults.
2. Adopt each child idempotently: move its directory unless the destination
   already exists (first adoption wins when two parents produce the same child),
   `enqueue(child, parent_id)`, and replay completions through the funnel prefix
   so the child lands in the right state with a real timeline.
3. Only when **every** child is adopted, expand the parent.

The parent is never moved through a claimable intermediate state on the way —
that window would let the next stage claim a parent that is really a container.
A partial adoption leaves the parent leased, so the lease timeout re-runs the
whole idempotent settlement. Zero children is a valid expansion, not a failure.

## Selection and single-stage runs

`materials=` restricts a run to named materials **and their descendants**, walked
through `parent_id` and memoized. The filter is applied behind the backlog gate,
and passed into `claim(among=…)` as a SQL `IN` clause rather than filtered in
Python. Finalize is scoped to the selected lineage, and an unknown id warns
rather than refuses.

`only=` restricts the submit phase to exactly one stage. The **check phase stays
unrestricted** on purpose: a job recorded by an earlier run still settles, even
if it belongs to a different stage. Finalize is suppressed unless the named stage
is the configured last one.

## Reading it alongside the tests

`tests/unit/test_orchestrator.py` is the specification in executable form: the
conveyor with files flowing between item directories, the stop-stage halt, the
requeue-to-terminal budget walk, a terminal verdict bypassing a budget of five,
and — the one to read first — restart re-attachment with `lease_timeout_s=0`,
which proves the ledger alone prevents resubmission.

## Related

- {doc}`state-and-storage` — the store the loop drives.
- {doc}`../api/orchestrator` — the generated reference.
