---
description: Why the orchestrator is a single-threaded loop over durable state, why every job is written down before it is polled, and why that makes an interrupted run resumable.
---

# Scheduling and restart

MatMelt's scheduler is deliberately boring: one loop, one thread, no queue
server, no daemon. Everything interesting about it is in what it writes down and
when.

```{image} /_static/diagrams/orchestrator-loop-light.svg
:alt: One pass of the orchestrator loop as a cycle: release stale leases, then a check phase that polls every recorded job and settles the terminal ones, then a finalize gate, then a submit phase gated by backlog and backend health that claims work, builds a JobSpec, submits it and records the handle in the in-flight ledger before the next pass begins.
:class: only-light
:width: 100%
```

```{image} /_static/diagrams/orchestrator-loop-dark.svg
:alt: One pass of the orchestrator loop as a cycle: release stale leases, then a check phase that polls every recorded job and settles the terminal ones, then a finalize gate, then a submit phase gated by backlog and backend health that claims work, builds a JobSpec, submits it and records the handle in the in-flight ledger before the next pass begins.
:class: only-dark
:width: 100%
```

## One pass

Each pass does the same four things, in this order:

1. **Release stale leases** — claims older than `lease_timeout_s` go back to
   their queue.
2. **Check phase** — poll every recorded job; settle the terminal ones by
   fetching the result, asking the stage to interpret it, validating the
   promised outputs, then completing or failing the material, releasing the job
   and clearing it from the ledger.
3. **Finalize gate** — retire materials as `DONE`, but only if this run goes
   through the configured last stage ({doc}`funnel-and-stop-after`).
4. **Submit phase** — for each active stage: check the backlog, check backend
   health, claim up to the free slots, validate inputs, build a `JobSpec`,
   submit, and record the handle.

Then it sleeps for `pass_interval_s` and does it again. A pass never raises: a
problem becomes a fault on a material or a paused stage, not a crashed run.

## Check before submit, and gates ordered by cost

Settling finished work before starting new work is what keeps the number of jobs
in flight honest — free slots are computed after the finished ones are cleared,
not before.

The two submit gates are ordered by what they cost. The **backlog gate** is a
dictionary lookup: if no material is waiting for a stage, that stage costs
nothing this pass — not even a health check. Only stages with work reach the
**health gate**, which asks the backend whether it is usable. An unhealthy
backend pauses its stage for that pass and fails nothing, because a cluster
being down is not a statement about anybody's chemistry.

## The ledger is the restart mechanism

Every submitted job is written to an in-flight ledger **at the submit boundary**
— before anything polls it, before the pass ends, before anything can go wrong.

That ordering is the whole trick. A pass that finds a recorded job re-polls it
instead of submitting a second one. So if the tool dies — Ctrl-C, a closed
laptop, a killed process — the jobs it started keep running, and the next run
picks them up:

```text
pass 1: completed=2
```

Two relaxations settled with nothing submitted, because they had been running
the whole time under no supervision at all. Work is not repeated, and nothing is
orphaned.

For that to work, a handle must be **durable**: a process id, or a scheduler job
id — never a socket, a `Popen` object, or anything else that lives only inside
the process that created it. A fresh interpreter must be able to poll a job it
did not start, from the ledger alone. Everything the backend needs later travels
as JSON-safe values.

The local backend goes further than a `Popen` wrapper for the same reason: each
job gets a directory holding a wrapper script, an atomically written exit-code
sentinel, both captured streams and a manifest. `poll` and `fetch` read only
those files, and the recorded process start time defeats PID reuse — so a
restarted tool resolves jobs it never launched instead of assuming they died.

## Leases, not locks

A claimed material carries a timestamp rather than a lock. If a run vanishes
between claiming and submitting, the claim expires and the material returns to
its queue. Claims are atomic — a material is never worked twice — and materials
owned by a live ledger entry are never swept, so a slow job is not mistaken for
an abandoned one.

This is also why the tool is safe to simply re-run. Every state transition is
idempotent; replaying a completion is a no-op; seeding twice adds nothing.

## Single-threaded on purpose

The loop has no threads. The parallelism is real, but it lives one level down:
jobs are OS processes, and `max_parallel` decides how many run at once.

That choice buys crash isolation — one bad calculation cannot take down the
tool, because it is not in the tool's process — and true parallelism without the
GIL, since the work is in someone else's interpreter anyway. What it buys most
is debuggability: a single-threaded loop over durable state can be stopped,
inspected, and restarted, and a bug reproduces instead of interleaving
differently every time.

Concurrency is used where it is needed and nowhere else. A heavy runner may
parallelise its own per-item work, bounded by the configured cores; the
orchestrator stays sequential.

## What this costs

It is not free, and the trade is worth naming. Polling has latency: a job that
finishes just after a poll waits until the next pass, so `pass_interval_s` sets
a floor on how quickly a fast stage turns around. A single process means
throughput is bounded by one machine's ability to shepherd jobs — fine for a
laptop driving a cluster, not a replacement for a workflow engine at ten
thousand concurrent tasks. And the state store is SQLite, which is a deliberate
single-user choice: zero setup, no server, one researcher's workspace.

In exchange there is nothing to deploy, nothing to keep running, and no state
that exists only in memory.

## Related

- {doc}`../reference/states` — the state machine the loop advances.
- {doc}`../how-to/stop-and-resume` — interrupting and resuming in practice.
- {doc}`stage-contract` — who decides whether a finished job succeeded.
