The orchestrator#

core/orchestrator.py is one class with one interesting method. It takes plain objects — stages in funnel order and a {stage_name: backend} mapping — and has no idea a config file exists. Scheduling and restart covers why it is shaped this way; this page is what the code does.

Construction#

The orchestrator derives its state machine at construction rather than consulting config later. Mapped stage names take their output state from STAGE_OUTPUT_STATE; unmapped names (test doubles) take the next chain state positionally. Output states must advance strictly, so an out-of-order funnel is a ValueError naming the problem instead of a run that quietly does the wrong thing. Each stage’s input state is the previous stage’s output, with the first stage consuming NEW.

Bad wiring fails fast, at construction: an unknown only stage, only combined with stop_after, a stage with no backend.

run_pass()#

release_stale_leases(lease_timeout_s)

check phase      for every recorded ledger job:
                     poll → terminal? → fetch
                     → interpret_result → validate_outputs
                     → complete / fail → release → clear_job

finalize gate    if this run goes through the configured last stage:
                     materials in the stop stage's output state → DONE

submit phase     for each active stage, in funnel order:
                     backlog gate → health gate
                     → claim(limit = max_parallel − live jobs)
                     → validate_inputs → build_jobspec → submit → record_job

Two orderings are deliberate. Check before submit, so free slots are counted after finished work is cleared. Backlog before health, so a stage with nothing waiting costs nothing at all.

A pass never raises. A StageError pauses that stage for the pass and fails no material — a stage that cannot build jobs at all is not evidence about any particular material.

Two smaller invariants worth knowing when reading the code: a job that cannot be recorded in the ledger is released immediately, so nothing runs untracked; and a job whose poll cannot be resolved is left recorded rather than dropped, so it is retried next pass instead of leaking.

_fail is the only attribution point#

Every failure path in the pass funnels through one method, which is where the verdict-vs-fault doctrine is applied exactly once:

Cause

Treatment

StageVerdict(terminal=True)

Immediate terminal FAILED, budget 1

StageInputError

Fault — spends retry_budget

Promised outputs missing

Fault

BackendError at submit

Fault

Anything unexpected

Fault

Having one attribution point is why the rule can be stated as doctrine rather than hoped for: there is a single place where “is this the science’s answer or an accident?” is decided.

Fan-out settlement#

For a fan_out stage, a finished job means a children manifest to settle:

  1. Read and validate the manifest — unreadable, non-list, duplicated ids, unsafe ids (including path traversal), or a named child with no directory are all budgeted faults.

  2. Adopt each child idempotently: move its directory unless the destination already exists (first adoption wins when two parents produce the same child), enqueue(child, parent_id), and replay completions through the funnel prefix so the child lands in the right state with a real timeline.

  3. Only when every child is adopted, expand the parent.

The parent is never moved through a claimable intermediate state on the way — that window would let the next stage claim a parent that is really a container. A partial adoption leaves the parent leased, so the lease timeout re-runs the whole idempotent settlement. Zero children is a valid expansion, not a failure.

Selection and single-stage runs#

materials= restricts a run to named materials and their descendants, walked through parent_id and memoized. The filter is applied behind the backlog gate, and passed into claim(among=…) as a SQL IN clause rather than filtered in Python. Finalize is scoped to the selected lineage, and an unknown id warns rather than refuses.

only= restricts the submit phase to exactly one stage. The check phase stays unrestricted on purpose: a job recorded by an earlier run still settles, even if it belongs to a different stage. Finalize is suppressed unless the named stage is the configured last one.

Reading it alongside the tests#

tests/unit/test_orchestrator.py is the specification in executable form: the conveyor with files flowing between item directories, the stop-stage halt, the requeue-to-terminal budget walk, a terminal verdict bypassing a budget of five, and — the one to read first — restart re-attachment with lease_timeout_s=0, which proves the ledger alone prevents resubmission.