T2 — The config file in depth#

Goal: understand every block of matmelt.yaml well enough to change a run’s scope, cost and destination without guessing.

Prerequisites: a working install and a scaffolded config (T1, or Your first run). Nothing here requires a scientific stack — every command on this page is a config check, not a calculation.

Time: about 15 minutes.

The four blocks#

A config has exactly four top-level sections, and the split is the architecture in miniature:

run:        # what to explore, and where the workspace lives
pipeline:   # how the orchestrator behaves
backends:   # WHERE work runs — machines
stages:     # WHAT runs — the funnel, and which backend each stage uses

Anything else is rejected. Try adding a key that does not exist and MatMelt tells you exactly where it is:

$ matmelt doctor
Config error: matmelt.yaml: 2 configuration errors:
  - pipeline.retry_budget: Input should be greater than or equal to 1
  - pipeline.stop_afterr: Extra inputs are not permitted

Two things worth noticing. Every problem is reported at once, not one per run, and each is addressed by its dotted path — pipeline.stop_afterr is a typo for stop_after, and the message says so by refusing to accept an unknown key rather than silently ignoring it. A config error always exits 2.

`run:` — what to explore#

run:
  name: nacl-binaries
  work_dir: ./materials-work
  elements: [Na, Cl, O]
  combination_sizes: [2]

elements is the set the funnel explores, and combination_sizes decides how they are combined into seeds. This is the single most expensive decision in the file, because it is combinatorial:

`combination_sizes`	Seeds from `[Na, Cl, O]`
`[1]`	3 — the elemental solids
`[2]`	3 — the binaries
`[2, 3]`	4
omitted, or `[]`	7 — every size from 1 to 3

Omitting the key means all sizes, which is rarely what you want on a wide element set. You never have to work this out on paper: doctor reports the enumeration before any work is possible.

$ matmelt doctor
Seed enumeration: 3 element(s) -> 3 combination(s) (size 2: 3).

Check that number after every change to elements or combination_sizes. Ten elements with sizes omitted is 1023 seeds, each fanning out into compositions and then into candidate structures — the funnel multiplies from there.

work_dir is the workspace: one SQLite state database plus one directory per material. It is safe to inspect while a run is in progress, and deleting it deletes the run.

`pipeline:` — how the orchestrator behaves#

pipeline:
  stop_after: extract     # last stage this run executes
  retry_budget: 3         # fault retries per stage before FAILED
  lease_timeout_s: 900    # when an abandoned claim is reclaimed
  pass_interval_s: 1.0    # sleep between orchestrator passes

stop_after is a pause, not a finish line. A run stops after the named stage, and materials rest at that stage’s output state — live, claimable, and picked up by any later run that goes further. Only the funnel’s last configured stage retires a material as DONE. This is what makes a staged workflow possible: generate thousands of candidates one day, screen them the next.

retry_budget counts faults, not rejections. The distinction is the one that matters most when reading a failure:

A fault is something that might work next time — a full disk, an unreachable host, a killed process. It spends one unit of budget and the material is requeued.
A terminal verdict is a scientific answer. A structure that relaxed into something unphysical is not retried, because running it again gives the same answer. It goes straight to FAILED regardless of remaining budget.

Raising retry_budget therefore buys tolerance of flaky infrastructure, never better science.

lease_timeout_s is how long a claimed material may sit untouched before another pass may reclaim it — the recovery path when a run is killed. Lower it and an interrupted run resumes sooner; raise it if legitimate jobs run longer than the default 15 minutes without being polled.

`backends:` — where work runs#

backends:
  laptop:
    kind: local
    options:
      work_dir: ./materials-jobs
      max_parallel: 2

Each entry is a machine, named by you. max_parallel is the cap on jobs in flight for that backend — the concurrency dial. The orchestrator itself stays single-threaded; parallelism is the jobs, each an OS process.

Names are referenced by the stages block, and a typo is caught before anything runs:

$ matmelt doctor
Config error: matmelt.yaml: 1 configuration error:
  - <config>: Value error, stages.screen.backend: references undefined backend
'gpu-box'; defined backends: ['laptop']

The other backend kind, ssh_exec, is T3.

`stages:` — the funnel#

stages:
  compose:
    backend: laptop
    options:
      python: python3
      threshold: 2
  structure:
    backend: laptop
    options:
      python: python3
      dimensions: ["3D"]
  screen:
    backend: laptop
    options:
      python: python3
      device: cpu
      model_size: small
      fmax: 0.05
      max_steps: 200
  extract:
    backend: laptop
    options:
      python: python3

Two rules govern this block.

Every stage names a backend. That one line is the whole “move this stage to another machine” mechanism — no code changes, no stage rewrite.

The configured stages must form the funnel in order, omitting only optional ones. Leave out a required stage and the run is refused before it starts:

$ matmelt doctor
Config error: stages: the configured stages must follow the funnel (compose ->
structure -> screen -> [calculate] -> extract), omitting only optional stages;
got [compose, screen] -- missing [structure]

The [calculate] in that message is bracketed because it is the one optional stage: a funnel without it is complete and valid.

Each stage’s options are that stage’s own — threshold means something to compose and nothing to anyone else. An unknown option is refused with the same dotted-path clarity, so a renamed or misspelled option cannot sit silently in a config for weeks.

Running part of a funnel on demand#

stop_after sets a run’s ceiling; --only runs exactly one stage:

$ matmelt run --only screen

This is how you push everything currently waiting at one state through a single stage — after adding an element set, or after fixing whatever made a stage fail. Because both express “how far this run goes”, they are mutually exclusive and say so rather than picking a winner:

$ matmelt run --only screen --yes
run: --only and pipeline.stop_after are mutually exclusive -- remove stop_after
from the config or drop --only.

Exit code 2, before any work starts.

Keeping secrets out of the file#

Any string value may be an environment reference instead of a literal:

backends:
  cluster:
    kind: ssh_exec
    options:
      host: cluster.example.org
      user: env:CLUSTER_USER

env:NAME is resolved when the config is loaded. An unset variable is an error that names both the variable and the YAML path that wanted it — never an empty string that fails confusingly later. This is what lets a config be committed to a repository without carrying a username or a host into version control.

What you learned#

The four blocks, and that unknown keys are refused with their dotted path.
combination_sizes is the cost dial, and doctor reports the blast radius before any work starts.
stop_after pauses; only the last configured stage retires materials.
retry_budget buys tolerance of infrastructure faults, never better science — a terminal scientific verdict is never retried.
Every stage names its backend, which is the entire mechanism behind T3.

Next: T3 moves the expensive stage onto another machine, using exactly one of the lines you just read. Every key, with its default, is in the configuration reference.

T2 — The config file in depth#

The four blocks#

run: — what to explore#

pipeline: — how the orchestrator behaves#

backends: — where work runs#

stages: — the funnel#