---
description: The MatMelt architecture in five diagrams, generated from committed scripts.
---

# Architecture diagrams

Five diagrams describe how MatMelt is put together. Each one is generated by a
script under `docs/diagrams/` and rendered twice — once per theme — so the
version you are reading matches the page around it. To regenerate them after an
architectural change:

```console
$ python docs/diagrams/render.py
$ python docs/diagrams/render.py --check   # what CI asks: is anything stale?
```

Graphviz is needed only for that regeneration. Building this site never runs
`dot`: the rendered SVGs are committed.

:::{note}
The diagrams land before the prose that will surround them. Each one moves into
its explanation page as that page is written; this gallery stays as the index.
:::

## Reading them

The five share one vocabulary, so a colour means the same thing on every page.
Colour is never the only signal: each category owns a shape as well, and the
diagrams stay readable in greyscale or with any colour vision.

| Shape | Category | Meaning |
|---|---|---|
| rounded box | stage | the *what* — builds a portable `JobSpec` |
| component | backend | the *where* — runs jobs, judges no science |
| 3-D box | host | a machine a backend reaches |
| tab | queue | a scheduler queue on such a machine |
| note | artifact | a value object, or a file on disk |
| cylinder | state | durable state — SQLite, the in-flight ledger |
| parallelogram | config | what you write in `matmelt.yaml` |
| ellipse | step | one action the orchestrator or a command takes |
| diamond | gate | a decision it makes |

Lines carry meaning too: **solid** is a handoff inside the tool, **dashed**
crosses a machine boundary, **dotted** is a read that changes nothing, and a
**red dashed** line — always labelled — is a connection the architecture does
*not* permit. A dashed outline on a node means one thing only: optional.

(diagram-what-vs-where)=

## What vs. where

The rule the rest of the architecture hangs from. A **stage** decides *what* to
compute and hands over a portable `JobSpec` — an argv, the files it needs, the
files it promises, and how much machine it wants. A **backend** decides *where*
that runs. The spec names no host, key, queue or path, so the same stage runs on
a laptop or a cluster depending on one line of YAML; and the backend never
judges science, so a non-zero exit comes back as an ordinary `JobResult` for the
stage to interpret.

```{image} /_static/diagrams/what-vs-where-light.svg
:alt: A stage builds a JobSpec that names no machine; either backend can run it, and the result comes back for the stage to judge. An arrow from the stage to a host is marked as never permitted.
:class: only-light
:width: 100%
```

```{image} /_static/diagrams/what-vs-where-dark.svg
:alt: A stage builds a JobSpec that names no machine; either backend can run it, and the result comes back for the stage to judge. An arrow from the stage to a host is marked as never permitted.
:class: only-dark
:width: 100%
```

The red arrow is the point: it is the connection the architecture does not
permit. No stage anywhere in the codebase reaches a machine.

(diagram-stage-funnel)=

## The stage funnel

Five stages, one of them optional. The shape matters as much as the order: the
first two stages **fan out** — an element combination becomes several plausible
compositions, and each composition becomes several candidate crystals, one per
dimensionality and space group — and the pipeline then **narrows**, because
screening is where a material can be rejected on the science.

A dashed outline means *optional*: `calculate` runs only if your config asks for
it, and a funnel without it behaves exactly as it did before the stage existed.

```{image} /_static/diagrams/stage-funnel-light.svg
:alt: The five-stage funnel: compose and structure fan one seed out into several compositions and then several candidate crystals; screen narrows them, sending rejects to a terminal FAILED state; the optional calculate stage and then extract carry survivors to DONE. A stop_after setting rests materials at a stage rather than retiring them.
:class: only-light
:width: 100%
```

```{image} /_static/diagrams/stage-funnel-dark.svg
:alt: The five-stage funnel: compose and structure fan one seed out into several compositions and then several candidate crystals; screen narrows them, sending rejects to a terminal FAILED state; the optional calculate stage and then extract carry survivors to DONE. A stop_after setting rests materials at a stage rather than retiring them.
:class: only-dark
:width: 100%
```

Three terminal outcomes are drawn rather than left to be inferred. `DONE` is a
material that rode the whole funnel. `EXPANDED` is a parent that fanned out —
terminal, but emphatically not a failure. `FAILED` is either a terminal science
verdict or a fault that spent its retry budget, and `matmelt retry` brings those
materials back.

Note what `stop_after` does *not* do: a mid-funnel stop **rests** materials at
that stage's output state, live and claimable. Raising the setting, removing it,
or running `matmelt run --only <stage>` continues them on the same workspace.

(diagram-local-vs-remote)=

## Local vs. remote topology

The same run, drawn twice. On the left every stage runs on one machine. On the
right `screen` — usually the expensive one — runs on a cluster instead. Between
the two panels **not one line of Python changes**: what differs is a backend
entry in `matmelt.yaml` and the single line that wires the stage to it.

```{image} /_static/diagrams/local-vs-remote-light.svg
:alt: The same pipeline drawn twice. On the left every stage runs on one workstation through the local backend. On the right the screen stage is wired to an ssh_exec backend, which pushes the runner script and the declared input files to a per-job directory on a cluster, runs them there through SLURM, and pulls the declared outputs back. Only the YAML differs.
:class: only-light
:width: 100%
```

```{image} /_static/diagrams/local-vs-remote-dark.svg
:alt: The same pipeline drawn twice. On the left every stage runs on one workstation through the local backend. On the right the screen stage is wired to an ssh_exec backend, which pushes the runner script and the declared input files to a per-job directory on a cluster, runs them there through SLURM, and pulls the declared outputs back. Only the YAML differs.
:class: only-dark
:width: 100%
```

Two things in the right-hand panel are worth reading carefully, because both are
easy to assume wrongly.

**The remote holds no MatMelt code.** A job ships the stage's self-contained
runner script and the files that stage declared — nothing else. There is no
package to install there, no `PYTHONPATH` to arrange, and no assumption that
anything of ours is importable on that machine. It is a pure calculation
executor with your own science stack in `remote_python`.

**Data moves by declarative staging.** Inputs are pushed into a per-job scratch
directory and the declared outputs are pulled back, so nothing depends on the
two machines sharing a filesystem or agreeing on paths. If they *do* share one,
`shared_filesystem: true` skips the transfers — an optimisation you opt into,
never an assumption the tool makes for you.

(diagram-orchestrator-loop)=

## The orchestrator loop

One pass, drawn as the cycle it is. `run_pass()` releases stale leases, settles
whatever is already running, decides whether anything may retire, and only then
starts new work — then sleeps and does it again. It is single-threaded and it
never raises.

```{image} /_static/diagrams/orchestrator-loop-light.svg
:alt: One pass of the orchestrator loop as a cycle: release stale leases, then a check phase that polls every recorded job and settles the terminal ones, then a finalize gate, then a submit phase gated by backlog and backend health that claims work, builds a JobSpec, submits it and records the handle in the in-flight ledger before the next pass begins.
:class: only-light
:width: 100%
```

```{image} /_static/diagrams/orchestrator-loop-dark.svg
:alt: One pass of the orchestrator loop as a cycle: release stale leases, then a check phase that polls every recorded job and settles the terminal ones, then a finalize gate, then a submit phase gated by backlog and backend health that claims work, builds a JobSpec, submits it and records the handle in the in-flight ledger before the next pass begins.
:class: only-dark
:width: 100%
```

The parallelism is real, but it is not in this loop: it is the jobs in flight,
each an OS process the backend started. Keeping the loop itself single-threaded
is what makes a run debuggable and safe to interrupt.

Two things carry that restart safety. Every job is written to the **in-flight
ledger at the submit boundary**, so a pass that finds a recorded job re-polls it
rather than starting a second one — kill a run mid-flight and its work survives
on the compute plane, waiting to be re-attached. And a `Handle` holds a
**durable id**, a PID or a SLURM job id, never a socket or a `Popen`, so a fresh
process can poll a job it did not start.

The gates are ordered by cost. The backlog gate is a dictionary lookup and comes
first, so a stage with nothing waiting costs nothing at all — not even a health
check. An unhealthy backend pauses its stage for that pass and fails nothing.

(diagram-monitor-flow)=

## The monitor's data flow

`matmelt monitor` is a dashboard over a run it never touches. One idea repeats
at every layer: read, compute, render — and never write.

```{image} /_static/diagrams/monitor-flow-light.svg
:alt: The monitor's data flow: a read-only state store, a CPU and GPU usage sampler, and a cached backend health probe feed the pure build_snapshot function, which returns a MonitorSnapshot value; the Rich renderers turn that one value into the dashboard panels. Nothing on the path writes to the run.
:class: only-light
:width: 100%
```

```{image} /_static/diagrams/monitor-flow-dark.svg
:alt: The monitor's data flow: a read-only state store, a CPU and GPU usage sampler, and a cached backend health probe feed the pure build_snapshot function, which returns a MonitorSnapshot value; the Rich renderers turn that one value into the dashboard panels. Nothing on the path writes to the run.
:class: only-dark
:width: 100%
```

The store is opened `read_only=True`, which sets SQLite's `query_only` pragma —
the refusal is enforced by the engine, not by our good intentions. Watching a
run cannot disturb it, and the same dashboard works just as well over a finished
workspace with nothing running at all.

`build_snapshot` is pure: a store and some plain values in, one `MonitorSnapshot`
out, with no knowledge of Rich, SQL or config. Every panel is a function of that
single value, which is what makes the whole dashboard testable without a
terminal.

Telemetry is best-effort by construction. CPU and GPU sampling can fail, time
out, or simply not exist on a given machine, and the dashboard shows a dash: an
unknown reading must never look like zero, and must never take the view down.
The rates differ on purpose — the store every tick, local CPU every tick, a
remote at most every 10 seconds, backend health at most every 30. A dashboard
must not become load.
