---
description: Run the screening stage on a remote machine over SSH while every other stage stays local — a config change, not a code change.
---

# T3 — Offloading a stage to a remote machine

**Goal:** move `screen` onto a more powerful machine over SSH, leaving
everything else on your laptop, and understand exactly what crosses the network.

**Prerequisites:**

- A completed local run (T1), so you have a working config to modify.
- A remote machine you can reach with **non-interactive SSH** — key-based auth,
  no password prompt, no passphrase prompt.
- `rsync` on both machines.
- A Python on the remote with `mace-torch` installed, and either a MACE model
  file there or the willingness to let one download into that machine's cache.

**Time:** about 20 minutes, most of it verifying SSH.

:::{admonition} About the output on this page
:class: note

Unlike T1 and T2, this tutorial cannot be reproduced on a single machine, so its
outputs come from a recorded run on our own lab machine (`wisdom@192.168.1.83`,
a CPU server) rather than from a session captured while writing. Substitute your
own host, user and paths throughout. The command surface, the option names and
the message formats are checked against the code by the documentation tests.
:::

## 1. Prove SSH works non-interactively

MatMelt never prompts for a password and never allocates a terminal. If this
command does not print the hostname immediately, fix that before touching any
config:

```console
$ ssh -o BatchMode=yes you@remote.example.org hostname
```

`BatchMode=yes` is exactly how MatMelt connects. A password prompt, a
passphrase prompt or a host-key confirmation each mean the same thing here: the
connection is not usable by an automated tool yet.

Then confirm the remote has the science stack, in the interpreter you intend to
use:

```console
$ ssh -o BatchMode=yes you@remote.example.org /opt/science-venv/bin/python -c "import mace; print('ok')"
```

## 2. Add the backend

A backend is a machine. Add one beside your existing `laptop`:

```yaml
backends:
  laptop:
    kind: local
    options:
      work_dir: ./materials-jobs
      max_parallel: 2
  cluster:
    kind: ssh_exec
    options:
      host: remote.example.org
      user: env:CLUSTER_USER
      remote_work_dir: /tmp/matmelt-jobs
      max_parallel: 4
```

`user: env:CLUSTER_USER` reads the username from the environment, so the config
stays committable. `remote_work_dir` is where per-job scratch directories are
created; the default shown is also what you get if you omit it.

## 3. Point one stage at it

This is the whole offloading mechanism — two lines:

```yaml
stages:
  screen:
    backend: cluster
    options:
      python: /opt/science-venv/bin/python
      device: cpu
      model_path: /data/models/mace.model
```

:::{important}
The stage's `python` is now an interpreter **on the remote machine**. It is the
command that runs there, so it must exist there — a local path that happens to
work on your laptop is a common first mistake. Nothing else in the config
changes, and no Python of ours changes at all: `ScreenStage` is the same class
doing the same thing.
:::

## 4. Let `doctor` check the remote

```console
$ matmelt doctor
```

The readiness table now answers a different question for `screen`, because the
probe runs **over SSH on the remote**:

```text
│ screen    │ mace        │ /opt/science-… │ you@remote.example.org │ ok │ ... │
```

The **Where** column is the point. A dependency answer is only as good as the
machine it was obtained on, so MatMelt reports where it asked. If the package is
missing there, the fix it prints is a command you can run as-is:

```text
ssh you@remote.example.org /opt/science-venv/bin/python -m pip install mace-torch
```

The model check is location-aware in the same way. With `model_path` set and the
stage wired to a remote, `doctor` stats the file **on that machine**:

```text
Screen model: ok -- MACE model on you@remote.example.org: /data/models/mace.model
```

Backend reachability is checked too: MatMelt creates and tests its remote work
directory, and an unreachable or unwritable host is reported here rather than
discovered halfway through a funnel.

## 5. Run it

```console
$ matmelt run --yes
```

Nothing about the command changes. What changes is where the work happens, and
you can watch it from the remote side while a job is in flight:

```console
$ ssh you@remote.example.org ls /tmp/matmelt-jobs/
mcli-9c1f4a2be07d/
$ ssh you@remote.example.org ls /tmp/matmelt-jobs/mcli-9c1f4a2be07d/
job.err  job.out  job.rc  screen_runner.py  structure.cif
```

That listing is the architecture made visible. The remote received **two
things**: the stage's self-contained runner script, and the input file the stage
declared. There is no MatMelt package there, no `PYTHONPATH`, nothing to
install — the remote is a pure calculation executor. When the job finishes, the
declared outputs (`relaxed.cif`, `screen_result.json`) are pulled back into the
material's local directory, and the local funnel continues as if the work had
been done at home.

From the recorded gate run: a real `Cl1Na1` candidate staged up, screened on the
remote and settled to `DONE` in **17 seconds**, with the verdict `RELAXED`
(converged in 56 steps, −3.376 eV/atom, residual force 0.044 eV/Å) pulled back
into the local workspace. Afterwards the remote job directory was gone —
MatMelt removes it when it releases the job.

## 6. If the two machines share a filesystem

When the workspace is already visible on both sides — a shared `/scratch`, for
instance — the file transfers are pure overhead:

```yaml
backends:
  cluster:
    kind: ssh_exec
    options:
      host: remote.example.org
      shared_filesystem: true
```

The job then runs in place, with no staging at all. Note the direction of the
default: staging is what happens **unless you say otherwise**, because assuming
a shared filesystem that does not exist fails in confusing ways, while staging
when you did not need it merely copies a few files.

## 7. Submitting through SLURM

If the remote is a cluster login node rather than a machine you may compute on,
let the scheduler place the work:

```yaml
backends:
  cluster:
    kind: ssh_exec
    options:
      host: login.cluster.example.org
      use_slurm: true
      partition: compute
      max_parallel: 8
```

Jobs are then submitted with `sbatch --parsable`, and the handle MatMelt stores
is the **SLURM job id** rather than a process id. Resources declared by the
stage become `--cpus-per-task` and `--time`. Everything else — polling,
fetching, the funnel — is unchanged.

## What restart looks like now

The handle is a durable id: a remote PID, or a SLURM job id. Kill your local
`matmelt run` mid-flight and the remote work carries on; run it again and it
re-attaches to the recorded job instead of submitting a second one. This is why
a long remote run survives a closed laptop lid.

## What you learned

- Offloading a stage is a backend entry plus one `backend:` line — no Python
  changes anywhere.
- The stage's `python` becomes an interpreter *on the remote*.
- `doctor` probes dependencies and the model file **where the job will run**, and
  reports the location alongside the answer.
- A remote receives only the runner script and the declared inputs; declared
  outputs come back. It never receives MatMelt itself.
- Staging is the default; `shared_filesystem: true` is an opt-in optimisation.
- `use_slurm: true` swaps process ids for scheduler job ids and changes nothing
  else.

Next: {doc}`T4 <t4-monitor>` watches a run like this one while it drains. For
the same move as a short recipe, see {doc}`../how-to/remote-backend`.
