---
description: Move one stage onto another machine over SSH or SLURM by adding a backend and changing one line — no code, and no MatMelt installed on the remote.
---

# Run a stage on another machine

Screening is usually the expensive stage. Moving it to a bigger machine is two
edits to `matmelt.yaml`: describe the machine under `backends:`, then point the
stage at it. No Python changes, and **nothing of MatMelt is installed on the
remote**.

## Before you start

Passwordless SSH must already work, non-interactively — MatMelt connects with
`BatchMode=yes` and never prompts:

```console
$ ssh alice@cluster.example.org true && echo ok
```

The remote needs `rsync` and a Python environment holding the science package
for the stage you are offloading — nothing else. Create it once:

```console
$ ssh alice@cluster.example.org
$ python3 -m venv ~/science-venv && ~/science-venv/bin/pip install mace-torch
```

## Add the backend and rewire the stage

```yaml
backends:
  laptop:
    kind: local
    options:
      work_dir: ./materials-jobs
  cluster:
    kind: ssh_exec
    options:
      host: cluster.example.org
      user: alice
      remote_work_dir: /scratch/alice/matmelt-jobs
      max_parallel: 4
```

```yaml
stages:
  screen:
    backend: cluster
    options:
      python: /home/alice/science-venv/bin/python
      device: cuda
```

Only two things changed: `backend: laptop` became `backend: cluster`, and
`python` now names the interpreter **on that machine**. Everything else —
compose, structure, extract — still runs locally.

Keep credentials out of the file with `env:` references:

```yaml
backends:
  cluster:
    kind: ssh_exec
    options:
      host: env:CLUSTER_HOST
      user: env:CLUSTER_USER
```

## Check it before running

```console
$ matmelt doctor
```

`doctor` reaches across the network for both tables. When something is wrong it
says so in the SSH client's own words:

```text
│ cluster │ SshExecBackend │ DOWN    │ alice@cluster.example.org: work dir     │
│         │                │         │ /tmp/matmelt-jobs not usable (ssh exit  │
│         │                │         │ 255) ssh: Could not resolve hostname    │
│         │                │         │ cluster.example.org: Name or service    │
│         │                │         │ not known                               │
```

When it is right, the readiness table shows the probe location and, if the
package is missing, an `ssh` command you can paste:

```text
│ screen    │ mace        │ /opt/scien… │ alice@clust… │ MISSING │ ssh: Could  │
...
To fix the missing requirements, run:
  ssh alice@cluster.example.org /opt/science-venv/bin/python -m pip install
mace-torch
```

`Where` naming `alice@cluster.example.org` rather than `local` is the
confirmation that the check went to the right machine.

## Run it

```console
$ matmelt run --yes
```

Nothing about the command changes. Per job, MatMelt creates a fresh directory
under `remote_work_dir`, pushes the stage's **self-contained runner script** and
the files that stage declared, runs it there, and pulls the declared outputs
back into the local material directory. The remote directory is removed when the
job is released.

```text
/scratch/alice/matmelt-jobs/mcli-4f8b2a1c9d3e/
├── screen_runner.py      # shipped per job
├── structure.cif         # the declared input
├── job.rc  job.out  job.err
├── screen_result.json    # pulled back
└── relaxed.cif           # pulled back
```

There is no `pip install matmelt` on that machine, no `PYTHONPATH`, and no
assumption that our package is importable there. That is why the remote needs
only your science stack.

## Submit through SLURM

```yaml
backends:
  cluster:
    kind: ssh_exec
    options:
      host: cluster.example.org
      user: alice
      use_slurm: true
      partition: gpu
      max_parallel: 8
```

Jobs then go through `sbatch --parsable`, and the handle MatMelt records is the
**SLURM job id** rather than a PID — so an interrupted run re-attaches to jobs
still queued or running. `Resources` become `--cpus-per-task` and `--time`.

## Skip staging on a shared filesystem

If the remote already sees your working directory at the same absolute path
(NFS, SSHFS), skip the transfers:

```yaml
backends:
  cluster:
    kind: ssh_exec
    options:
      host: cluster.example.org
      shared_filesystem: true
```

This is an optimisation you opt into. The default assumes nothing is shared,
which is what makes an arbitrary machine usable.

## When it goes wrong

| Symptom | Usual cause |
|---|---|
| Backend `DOWN`, `ssh exit 255` | Host, user, or key — the detail line quotes SSH |
| Backend healthy, stage `MISSING` | The science package is not in the remote `python` |
| Jobs submit but outputs never come back | The stage produced nothing; check `job.err` in the remote job dir before the run releases it |
| Everything slow, nothing parallel | `max_parallel` is 1 by default on every backend |

:::{note}
The end-to-end output on this page comes from a recorded run on our own
hardware (the Phase 4 acceptance gate: a real `screen` stage staged to a compute
server, tracked by remote PID, `relaxed.cif` pulled back, verdict `RELAXED` in
17 s). Two machines cannot be reproduced while writing docs. The `doctor`
failure output above **is** live, and every config key, flag and message shape
on this page is checked against the code by the test suite.
:::

## Related

- {doc}`../tutorials/t3-offload-a-stage` — the same move as a tutorial.
- {doc}`../reference/config` — every `ssh_exec` option.
- {doc}`configure-calculate` — why the DFTB+ stage is *not* offloaded this way.
