---
description: Why a remote machine never receives MatMelt's code, why data moves by declarative staging rather than a shared filesystem, and what that makes possible.
---

# Remote execution

Running a stage on another machine is a config change ({doc}`what-vs-where`).
This page is about what happens after that line is written, and about two
decisions that shape everything: the remote holds none of our code, and nothing
is assumed about shared storage.

```{image} /_static/diagrams/local-vs-remote-light.svg
:alt: The same pipeline drawn twice. On the left every stage runs on one workstation through the local backend. On the right the screen stage is wired to an ssh_exec backend, which pushes the runner script and the declared input files to a per-job directory on a cluster, runs them there through SLURM, and pulls the declared outputs back. Only the YAML differs.
:class: only-light
:width: 100%
```

```{image} /_static/diagrams/local-vs-remote-dark.svg
:alt: The same pipeline drawn twice. On the left every stage runs on one workstation through the local backend. On the right the screen stage is wired to an ssh_exec backend, which pushes the runner script and the declared input files to a per-job directory on a cluster, runs them there through SLURM, and pulls the declared outputs back. Only the YAML differs.
:class: only-dark
:width: 100%
```

## The remote is a calculation executor, nothing more

A remote machine never receives MatMelt. Not the package, not a checkout, not a
`PYTHONPATH` pointing at one. What travels is the stage's **self-contained runner
script** and the files that stage declared — and what comes back are the declared
outputs.

This is the same model AiiDA uses, and it was chosen for reasons that survive
contact with real clusters:

**Nothing to install or keep in step.** The remote needs your science stack and
nothing else. There is no second installation to upgrade in lockstep, and no
class of bug where the cluster is three versions behind your laptop.

**No assumption of admin rights.** Plenty of machines let you run jobs and not
install packages system-wide. A per-job script that only needs an interpreter
works there.

**The code that ran is the code you can inspect.** A job directory contains the
exact script that executed, so reproducing a failure means running that file by
hand — no orchestrator, no config, no database.

**A private project stays private.** Our source is never copied onto machines we
do not control.

The cost is that runners must be genuinely self-contained: standard library plus
their own science package, never an import from MatMelt. That is a real
constraint on how stages are written, and it is enforced by tests that parse
each runner's imports.

## Declarative staging, not a shared filesystem

Each job gets a fresh remote directory. Declared inputs are pushed into it, the
command runs there, declared outputs are pulled back into the local material
directory, and the directory is removed when the job is released.

The alternative — assume both machines see the same path over NFS — is simpler
and was the earlier default. It was inverted deliberately: a shared filesystem is
a property of *some* clusters, and building it into the data model excludes every
machine that is merely reachable by SSH. Staging works on any of them, and where
storage genuinely is shared, `shared_filesystem: true` skips the transfers as an
optimisation you opt into.

Two consequences worth knowing:

**Undeclared files do not travel.** This is where {doc}`stage-contract`'s
"declare everything" rule earns its keep — an output a stage forgot to declare
simply is not there afterwards. That has happened: a fan-out stage declaring only
its manifest, and not the subtree beside it, worked locally and failed every
material remotely.

**A missing output is data, not an error.** If a declared output was not
produced, it is omitted and the stage judges what that means. The transport does
not get to decide that a job failed.

Both directions go through one small seam — a stage-in call and a stage-out call
— so a different transport (object storage, a data-transfer service) can be
added later without touching any stage.

## Scheduler or bare process

With `use_slurm: true` a job goes through `sbatch --parsable` and the recorded
handle is the SLURM job id; otherwise it is a remote PID. Everything else is
identical, because both are just durable ids in the ledger
({doc}`scheduling-and-restart`) — which is why an interrupted run re-attaches to
jobs still sitting in a cluster queue.

Polling maps scheduler states conservatively. An empty accounting record means
"still running", never a guessed terminal state: a scheduler that has not
answered yet is not a job that failed. And the true exit code comes from a
sentinel file the wrapper writes, so a job whose exit status cannot be recovered
reports "unknown" rather than "fine".

## Safety at the boundary

Every token crossing the SSH boundary is quoted, so nothing in a path, an
environment value or a material id can splice a shell command. Connections use
`BatchMode=yes` — key-based access must already work, and MatMelt never prompts
for, stores, or logs a password. Health checks report an unreachable host as a
status rather than raising, and never include anything secret in the detail.

Credentials belong in the environment, not the config file: any value may be
written `env:CLUSTER_HOST`, and an unset variable is a config error naming the
variable rather than an empty string quietly substituted.

## What is not remoted this way

The optional `calculate` stage is the exception, and deliberately so. Its
remoteness belongs to **AiiDA**: the code you configure already names a computer
and a queue, and the plugin's WorkChain handles submission and healing there.
Wrapping that in an SSH backend would mean two systems dispatching the same
calculation. See {doc}`../how-to/configure-calculate`.

## Related

- {doc}`../how-to/remote-backend` — the configuration, step by step.
- {doc}`../tutorials/t3-offload-a-stage` — the same move as a tutorial.
- {doc}`../reference/config` — every `ssh_exec` option.
