Scheduling and restart#

MatMelt’s scheduler is deliberately boring: one loop, one thread, no queue server, no daemon. Everything interesting about it is in what it writes down and when.

One pass of the orchestrator loop as a cycle: release stale leases, then a check phase that polls every recorded job and settles the terminal ones, then a finalize gate, then a submit phase gated by backlog and backend health that claims work, builds a JobSpec, submits it and records the handle in the in-flight ledger before the next pass begins. One pass of the orchestrator loop as a cycle: release stale leases, then a check phase that polls every recorded job and settles the terminal ones, then a finalize gate, then a submit phase gated by backlog and backend health that claims work, builds a JobSpec, submits it and records the handle in the in-flight ledger before the next pass begins.

One pass#

Each pass does the same four things, in this order:

  1. Release stale leases — claims older than lease_timeout_s go back to their queue.

  2. Check phase — poll every recorded job; settle the terminal ones by fetching the result, asking the stage to interpret it, validating the promised outputs, then completing or failing the material, releasing the job and clearing it from the ledger.

  3. Finalize gate — retire materials as DONE, but only if this run goes through the configured last stage (The funnel, and what stopping means).

  4. Submit phase — for each active stage: check the backlog, check backend health, claim up to the free slots, validate inputs, build a JobSpec, submit, and record the handle.

Then it sleeps for pass_interval_s and does it again. A pass never raises: a problem becomes a fault on a material or a paused stage, not a crashed run.

Check before submit, and gates ordered by cost#

Settling finished work before starting new work is what keeps the number of jobs in flight honest — free slots are computed after the finished ones are cleared, not before.

The two submit gates are ordered by what they cost. The backlog gate is a dictionary lookup: if no material is waiting for a stage, that stage costs nothing this pass — not even a health check. Only stages with work reach the health gate, which asks the backend whether it is usable. An unhealthy backend pauses its stage for that pass and fails nothing, because a cluster being down is not a statement about anybody’s chemistry.

The ledger is the restart mechanism#

Every submitted job is written to an in-flight ledger at the submit boundary — before anything polls it, before the pass ends, before anything can go wrong.

That ordering is the whole trick. A pass that finds a recorded job re-polls it instead of submitting a second one. So if the tool dies — Ctrl-C, a closed laptop, a killed process — the jobs it started keep running, and the next run picks them up:

pass 1: completed=2

Two relaxations settled with nothing submitted, because they had been running the whole time under no supervision at all. Work is not repeated, and nothing is orphaned.

For that to work, a handle must be durable: a process id, or a scheduler job id — never a socket, a Popen object, or anything else that lives only inside the process that created it. A fresh interpreter must be able to poll a job it did not start, from the ledger alone. Everything the backend needs later travels as JSON-safe values.

The local backend goes further than a Popen wrapper for the same reason: each job gets a directory holding a wrapper script, an atomically written exit-code sentinel, both captured streams and a manifest. poll and fetch read only those files, and the recorded process start time defeats PID reuse — so a restarted tool resolves jobs it never launched instead of assuming they died.

Leases, not locks#

A claimed material carries a timestamp rather than a lock. If a run vanishes between claiming and submitting, the claim expires and the material returns to its queue. Claims are atomic — a material is never worked twice — and materials owned by a live ledger entry are never swept, so a slow job is not mistaken for an abandoned one.

This is also why the tool is safe to simply re-run. Every state transition is idempotent; replaying a completion is a no-op; seeding twice adds nothing.

Single-threaded on purpose#

The loop has no threads. The parallelism is real, but it lives one level down: jobs are OS processes, and max_parallel decides how many run at once.

That choice buys crash isolation — one bad calculation cannot take down the tool, because it is not in the tool’s process — and true parallelism without the GIL, since the work is in someone else’s interpreter anyway. What it buys most is debuggability: a single-threaded loop over durable state can be stopped, inspected, and restarted, and a bug reproduces instead of interleaving differently every time.

Concurrency is used where it is needed and nowhere else. A heavy runner may parallelise its own per-item work, bounded by the configured cores; the orchestrator stays sequential.

What this costs#

It is not free, and the trade is worth naming. Polling has latency: a job that finishes just after a poll waits until the next pass, so pass_interval_s sets a floor on how quickly a fast stage turns around. A single process means throughput is bounded by one machine’s ability to shepherd jobs — fine for a laptop driving a cluster, not a replacement for a workflow engine at ten thousand concurrent tasks. And the state store is SQLite, which is a deliberate single-user choice: zero setup, no server, one researcher’s workspace.

In exchange there is nothing to deploy, nothing to keep running, and no state that exists only in memory.