Remote execution#

Running a stage on another machine is a config change (What vs. where). This page is about what happens after that line is written, and about two decisions that shape everything: the remote holds none of our code, and nothing is assumed about shared storage.

The same pipeline drawn twice. On the left every stage runs on one workstation through the local backend. On the right the screen stage is wired to an ssh_exec backend, which pushes the runner script and the declared input files to a per-job directory on a cluster, runs them there through SLURM, and pulls the declared outputs back. Only the YAML differs. The same pipeline drawn twice. On the left every stage runs on one workstation through the local backend. On the right the screen stage is wired to an ssh_exec backend, which pushes the runner script and the declared input files to a per-job directory on a cluster, runs them there through SLURM, and pulls the declared outputs back. Only the YAML differs.

The remote is a calculation executor, nothing more#

A remote machine never receives MatMelt. Not the package, not a checkout, not a PYTHONPATH pointing at one. What travels is the stage’s self-contained runner script and the files that stage declared — and what comes back are the declared outputs.

This is the same model AiiDA uses, and it was chosen for reasons that survive contact with real clusters:

Nothing to install or keep in step. The remote needs your science stack and nothing else. There is no second installation to upgrade in lockstep, and no class of bug where the cluster is three versions behind your laptop.

No assumption of admin rights. Plenty of machines let you run jobs and not install packages system-wide. A per-job script that only needs an interpreter works there.

The code that ran is the code you can inspect. A job directory contains the exact script that executed, so reproducing a failure means running that file by hand — no orchestrator, no config, no database.

A private project stays private. Our source is never copied onto machines we do not control.

The cost is that runners must be genuinely self-contained: standard library plus their own science package, never an import from MatMelt. That is a real constraint on how stages are written, and it is enforced by tests that parse each runner’s imports.

Declarative staging, not a shared filesystem#

Each job gets a fresh remote directory. Declared inputs are pushed into it, the command runs there, declared outputs are pulled back into the local material directory, and the directory is removed when the job is released.

The alternative — assume both machines see the same path over NFS — is simpler and was the earlier default. It was inverted deliberately: a shared filesystem is a property of some clusters, and building it into the data model excludes every machine that is merely reachable by SSH. Staging works on any of them, and where storage genuinely is shared, shared_filesystem: true skips the transfers as an optimisation you opt into.

Two consequences worth knowing:

Undeclared files do not travel. This is where The stage contract’s “declare everything” rule earns its keep — an output a stage forgot to declare simply is not there afterwards. That has happened: a fan-out stage declaring only its manifest, and not the subtree beside it, worked locally and failed every material remotely.

A missing output is data, not an error. If a declared output was not produced, it is omitted and the stage judges what that means. The transport does not get to decide that a job failed.

Both directions go through one small seam — a stage-in call and a stage-out call — so a different transport (object storage, a data-transfer service) can be added later without touching any stage.

Scheduler or bare process#

With use_slurm: true a job goes through sbatch --parsable and the recorded handle is the SLURM job id; otherwise it is a remote PID. Everything else is identical, because both are just durable ids in the ledger (Scheduling and restart) — which is why an interrupted run re-attaches to jobs still sitting in a cluster queue.

Polling maps scheduler states conservatively. An empty accounting record means “still running”, never a guessed terminal state: a scheduler that has not answered yet is not a job that failed. And the true exit code comes from a sentinel file the wrapper writes, so a job whose exit status cannot be recovered reports “unknown” rather than “fine”.

Safety at the boundary#

Every token crossing the SSH boundary is quoted, so nothing in a path, an environment value or a material id can splice a shell command. Connections use BatchMode=yes — key-based access must already work, and MatMelt never prompts for, stores, or logs a password. Health checks report an unreachable host as a status rather than raising, and never include anything secret in the detail.

Credentials belong in the environment, not the config file: any value may be written env:CLUSTER_HOST, and an unset variable is a config error naming the variable rather than an empty string quietly substituted.

What is not remoted this way#

The optional calculate stage is the exception, and deliberately so. Its remoteness belongs to AiiDA: the code you configure already names a computer and a queue, and the plugin’s WorkChain handles submission and healing there. Wrapping that in an SSH backend would mean two systems dispatching the same calculation. See Add the DFTB+ calculate stage.