What vs. where#

The rule everything else follows: a stage decides what to compute; a backend decides where it runs. Neither knows the other’s business, and the only thing that crosses between them is a portable description of a job.

A stage builds a JobSpec that names no machine; either backend can run it, and the result comes back for the stage to judge. An arrow from the stage to a host is marked as never permitted. A stage builds a JobSpec that names no machine; either backend can run it, and the result comes back for the stage to judge. An arrow from the stage to a host is marked as never permitted.

The thing in the middle#

A stage’s whole output is a JobSpec: an argv to execute, the files the job needs, the files it promises to produce, an environment, a working directory, and a coarse statement of how much machine it wants. That is all.

What a JobSpec deliberately does not contain is a hostname, a username, an SSH key, a queue name, a scheduler directive, or an absolute path on somebody else’s machine. It cannot express those things, which is what makes it portable: the same spec is runnable by a backend that forks a subprocess and by one that rsyncs to a cluster and calls sbatch.

The red arrow on the diagram is the point. No stage anywhere in the codebase reaches a machine. If one did, moving that stage to a cluster would become a code change, and the property this whole design exists to protect would be gone.

Why it is worth the indirection#

Because the interesting question in a materials funnel is where the expensive stage runs, and that answer changes constantly — laptop today, the group’s GPU box next week, a SLURM allocation for the big run. With the split, that is one line of YAML:

stages:
  screen:
    backend: cluster

Without it, it is an edit to the screening code, a new code path to test, and a second way for the stage to be wrong. The stage does not become more correct by knowing about SSH; it becomes harder to test and impossible to reuse.

The same split is what lets the test suite drive the entire pipeline with a fake backend that never runs anything, and what lets every stage’s real work be run by hand as a plain script when you are debugging.

The other half: backends never judge science#

The split cuts both ways, and the second half is the one people get wrong.

A backend runs a job and reports what happened. It does not decide whether what happened was good. A job that exits non-zero has not failed as far as the backend is concerned — it ran, and its exit code was 3. That comes back as an ordinary JobResult for the stage to interpret.

Backends raise only for infrastructure problems: the host is unreachable, the scratch directory is not writable, the spec was malformed. The distinction is not pedantry — it decides what the tool does next. A scientific rejection should retire a material permanently; an unreachable host should pause the stage and try again later. Conflating them either retries physics forever or permanently discards a material because a network blipped.

There is a third state worth naming, because it is a real source of silent bugs elsewhere: a job that ran but whose exit code could not be recovered reports None, never 0. “I don’t know” and “it succeeded” are different answers, and defaulting the first to the second is how a broken job becomes a fabricated result.

What this buys you, concretely#

  • Backend choice is configuration. The registry maps a kind string to an implementation; adding a backend adds a config option, not a branch in a stage.

  • Stages are testable without compute. A fake backend fabricates the declared outputs, so a stage’s logic is exercised with no science stack installed.

  • A backend is testable without science. Its contract is “run this argv, bring back these files”, which is checkable with /bin/echo.

  • The same conformance tests run against every backend, so a new one either satisfies the contract or fails a test that already exists.