Architecture diagrams#

Five diagrams describe how MatMelt is put together. Each one is generated by a script under docs/diagrams/ and rendered twice — once per theme — so the version you are reading matches the page around it. To regenerate them after an architectural change:

$ python docs/diagrams/render.py
$ python docs/diagrams/render.py --check   # what CI asks: is anything stale?

Graphviz is needed only for that regeneration. Building this site never runs dot: the rendered SVGs are committed.

Note

The diagrams land before the prose that will surround them. Each one moves into its explanation page as that page is written; this gallery stays as the index.

Reading them#

The five share one vocabulary, so a colour means the same thing on every page. Colour is never the only signal: each category owns a shape as well, and the diagrams stay readable in greyscale or with any colour vision.

Shape

Category

Meaning

rounded box

stage

the what — builds a portable JobSpec

component

backend

the where — runs jobs, judges no science

3-D box

host

a machine a backend reaches

tab

queue

a scheduler queue on such a machine

note

artifact

a value object, or a file on disk

cylinder

state

durable state — SQLite, the in-flight ledger

parallelogram

config

what you write in matmelt.yaml

ellipse

step

one action the orchestrator or a command takes

diamond

gate

a decision it makes

Lines carry meaning too: solid is a handoff inside the tool, dashed crosses a machine boundary, dotted is a read that changes nothing, and a red dashed line — always labelled — is a connection the architecture does not permit. A dashed outline on a node means one thing only: optional.

What vs. where#

The rule the rest of the architecture hangs from. A stage decides what to compute and hands over a portable JobSpec — an argv, the files it needs, the files it promises, and how much machine it wants. A backend decides where that runs. The spec names no host, key, queue or path, so the same stage runs on a laptop or a cluster depending on one line of YAML; and the backend never judges science, so a non-zero exit comes back as an ordinary JobResult for the stage to interpret.

A stage builds a JobSpec that names no machine; either backend can run it, and the result comes back for the stage to judge. An arrow from the stage to a host is marked as never permitted. A stage builds a JobSpec that names no machine; either backend can run it, and the result comes back for the stage to judge. An arrow from the stage to a host is marked as never permitted.

The red arrow is the point: it is the connection the architecture does not permit. No stage anywhere in the codebase reaches a machine.

The stage funnel#

Five stages, one of them optional. The shape matters as much as the order: the first two stages fan out — an element combination becomes several plausible compositions, and each composition becomes several candidate crystals, one per dimensionality and space group — and the pipeline then narrows, because screening is where a material can be rejected on the science.

A dashed outline means optional: calculate runs only if your config asks for it, and a funnel without it behaves exactly as it did before the stage existed.

The five-stage funnel: compose and structure fan one seed out into several compositions and then several candidate crystals; screen narrows them, sending rejects to a terminal FAILED state; the optional calculate stage and then extract carry survivors to DONE. A stop_after setting rests materials at a stage rather than retiring them. The five-stage funnel: compose and structure fan one seed out into several compositions and then several candidate crystals; screen narrows them, sending rejects to a terminal FAILED state; the optional calculate stage and then extract carry survivors to DONE. A stop_after setting rests materials at a stage rather than retiring them.

Three terminal outcomes are drawn rather than left to be inferred. DONE is a material that rode the whole funnel. EXPANDED is a parent that fanned out — terminal, but emphatically not a failure. FAILED is either a terminal science verdict or a fault that spent its retry budget, and matmelt retry brings those materials back.

Note what stop_after does not do: a mid-funnel stop rests materials at that stage’s output state, live and claimable. Raising the setting, removing it, or running matmelt run --only <stage> continues them on the same workspace.

Local vs. remote topology#

The same run, drawn twice. On the left every stage runs on one machine. On the right screen — usually the expensive one — runs on a cluster instead. Between the two panels not one line of Python changes: what differs is a backend entry in matmelt.yaml and the single line that wires the stage to it.

The same pipeline drawn twice. On the left every stage runs on one workstation through the local backend. On the right the screen stage is wired to an ssh_exec backend, which pushes the runner script and the declared input files to a per-job directory on a cluster, runs them there through SLURM, and pulls the declared outputs back. Only the YAML differs. The same pipeline drawn twice. On the left every stage runs on one workstation through the local backend. On the right the screen stage is wired to an ssh_exec backend, which pushes the runner script and the declared input files to a per-job directory on a cluster, runs them there through SLURM, and pulls the declared outputs back. Only the YAML differs.

Two things in the right-hand panel are worth reading carefully, because both are easy to assume wrongly.

The remote holds no MatMelt code. A job ships the stage’s self-contained runner script and the files that stage declared — nothing else. There is no package to install there, no PYTHONPATH to arrange, and no assumption that anything of ours is importable on that machine. It is a pure calculation executor with your own science stack in remote_python.

Data moves by declarative staging. Inputs are pushed into a per-job scratch directory and the declared outputs are pulled back, so nothing depends on the two machines sharing a filesystem or agreeing on paths. If they do share one, shared_filesystem: true skips the transfers — an optimisation you opt into, never an assumption the tool makes for you.

The orchestrator loop#

One pass, drawn as the cycle it is. run_pass() releases stale leases, settles whatever is already running, decides whether anything may retire, and only then starts new work — then sleeps and does it again. It is single-threaded and it never raises.

One pass of the orchestrator loop as a cycle: release stale leases, then a check phase that polls every recorded job and settles the terminal ones, then a finalize gate, then a submit phase gated by backlog and backend health that claims work, builds a JobSpec, submits it and records the handle in the in-flight ledger before the next pass begins. One pass of the orchestrator loop as a cycle: release stale leases, then a check phase that polls every recorded job and settles the terminal ones, then a finalize gate, then a submit phase gated by backlog and backend health that claims work, builds a JobSpec, submits it and records the handle in the in-flight ledger before the next pass begins.

The parallelism is real, but it is not in this loop: it is the jobs in flight, each an OS process the backend started. Keeping the loop itself single-threaded is what makes a run debuggable and safe to interrupt.

Two things carry that restart safety. Every job is written to the in-flight ledger at the submit boundary, so a pass that finds a recorded job re-polls it rather than starting a second one — kill a run mid-flight and its work survives on the compute plane, waiting to be re-attached. And a Handle holds a durable id, a PID or a SLURM job id, never a socket or a Popen, so a fresh process can poll a job it did not start.

The gates are ordered by cost. The backlog gate is a dictionary lookup and comes first, so a stage with nothing waiting costs nothing at all — not even a health check. An unhealthy backend pauses its stage for that pass and fails nothing.

The monitor’s data flow#

matmelt monitor is a dashboard over a run it never touches. One idea repeats at every layer: read, compute, render — and never write.

The monitor's data flow: a read-only state store, a CPU and GPU usage sampler, and a cached backend health probe feed the pure build_snapshot function, which returns a MonitorSnapshot value; the Rich renderers turn that one value into the dashboard panels. Nothing on the path writes to the run. The monitor's data flow: a read-only state store, a CPU and GPU usage sampler, and a cached backend health probe feed the pure build_snapshot function, which returns a MonitorSnapshot value; the Rich renderers turn that one value into the dashboard panels. Nothing on the path writes to the run.

The store is opened read_only=True, which sets SQLite’s query_only pragma — the refusal is enforced by the engine, not by our good intentions. Watching a run cannot disturb it, and the same dashboard works just as well over a finished workspace with nothing running at all.

build_snapshot is pure: a store and some plain values in, one MonitorSnapshot out, with no knowledge of Rich, SQL or config. Every panel is a function of that single value, which is what makes the whole dashboard testable without a terminal.

Telemetry is best-effort by construction. CPU and GPU sampling can fail, time out, or simply not exist on a given machine, and the dashboard shows a dash: an unknown reading must never look like zero, and must never take the view down. The rates differ on purpose — the store every tick, local CPU every tick, a remote at most every 10 seconds, backend health at most every 30. A dashboard must not become load.