T3 — Offloading a stage to a remote machine#

Goal: move screen onto a more powerful machine over SSH, leaving everything else on your laptop, and understand exactly what crosses the network.

Prerequisites:

  • A completed local run (T1), so you have a working config to modify.

  • A remote machine you can reach with non-interactive SSH — key-based auth, no password prompt, no passphrase prompt.

  • rsync on both machines.

  • A Python on the remote with mace-torch installed, and either a MACE model file there or the willingness to let one download into that machine’s cache.

Time: about 20 minutes, most of it verifying SSH.

About the output on this page

Unlike T1 and T2, this tutorial cannot be reproduced on a single machine, so its outputs come from a recorded run on our own lab machine (wisdom@192.168.1.83, a CPU server) rather than from a session captured while writing. Substitute your own host, user and paths throughout. The command surface, the option names and the message formats are checked against the code by the documentation tests.

1. Prove SSH works non-interactively#

MatMelt never prompts for a password and never allocates a terminal. If this command does not print the hostname immediately, fix that before touching any config:

$ ssh -o BatchMode=yes you@remote.example.org hostname

BatchMode=yes is exactly how MatMelt connects. A password prompt, a passphrase prompt or a host-key confirmation each mean the same thing here: the connection is not usable by an automated tool yet.

Then confirm the remote has the science stack, in the interpreter you intend to use:

$ ssh -o BatchMode=yes you@remote.example.org /opt/science-venv/bin/python -c "import mace; print('ok')"

2. Add the backend#

A backend is a machine. Add one beside your existing laptop:

backends:
  laptop:
    kind: local
    options:
      work_dir: ./materials-jobs
      max_parallel: 2
  cluster:
    kind: ssh_exec
    options:
      host: remote.example.org
      user: env:CLUSTER_USER
      remote_work_dir: /tmp/matmelt-jobs
      max_parallel: 4

user: env:CLUSTER_USER reads the username from the environment, so the config stays committable. remote_work_dir is where per-job scratch directories are created; the default shown is also what you get if you omit it.

3. Point one stage at it#

This is the whole offloading mechanism — two lines:

stages:
  screen:
    backend: cluster
    options:
      python: /opt/science-venv/bin/python
      device: cpu
      model_path: /data/models/mace.model

Important

The stage’s python is now an interpreter on the remote machine. It is the command that runs there, so it must exist there — a local path that happens to work on your laptop is a common first mistake. Nothing else in the config changes, and no Python of ours changes at all: ScreenStage is the same class doing the same thing.

4. Let doctor check the remote#

$ matmelt doctor

The readiness table now answers a different question for screen, because the probe runs over SSH on the remote:

│ screen    │ mace        │ /opt/science-… │ you@remote.example.org │ ok │ ... │

The Where column is the point. A dependency answer is only as good as the machine it was obtained on, so MatMelt reports where it asked. If the package is missing there, the fix it prints is a command you can run as-is:

ssh you@remote.example.org /opt/science-venv/bin/python -m pip install mace-torch

The model check is location-aware in the same way. With model_path set and the stage wired to a remote, doctor stats the file on that machine:

Screen model: ok -- MACE model on you@remote.example.org: /data/models/mace.model

Backend reachability is checked too: MatMelt creates and tests its remote work directory, and an unreachable or unwritable host is reported here rather than discovered halfway through a funnel.

5. Run it#

$ matmelt run --yes

Nothing about the command changes. What changes is where the work happens, and you can watch it from the remote side while a job is in flight:

$ ssh you@remote.example.org ls /tmp/matmelt-jobs/
mcli-9c1f4a2be07d/
$ ssh you@remote.example.org ls /tmp/matmelt-jobs/mcli-9c1f4a2be07d/
job.err  job.out  job.rc  screen_runner.py  structure.cif

That listing is the architecture made visible. The remote received two things: the stage’s self-contained runner script, and the input file the stage declared. There is no MatMelt package there, no PYTHONPATH, nothing to install — the remote is a pure calculation executor. When the job finishes, the declared outputs (relaxed.cif, screen_result.json) are pulled back into the material’s local directory, and the local funnel continues as if the work had been done at home.

From the recorded gate run: a real Cl1Na1 candidate staged up, screened on the remote and settled to DONE in 17 seconds, with the verdict RELAXED (converged in 56 steps, −3.376 eV/atom, residual force 0.044 eV/Å) pulled back into the local workspace. Afterwards the remote job directory was gone — MatMelt removes it when it releases the job.

6. If the two machines share a filesystem#

When the workspace is already visible on both sides — a shared /scratch, for instance — the file transfers are pure overhead:

backends:
  cluster:
    kind: ssh_exec
    options:
      host: remote.example.org
      shared_filesystem: true

The job then runs in place, with no staging at all. Note the direction of the default: staging is what happens unless you say otherwise, because assuming a shared filesystem that does not exist fails in confusing ways, while staging when you did not need it merely copies a few files.

7. Submitting through SLURM#

If the remote is a cluster login node rather than a machine you may compute on, let the scheduler place the work:

backends:
  cluster:
    kind: ssh_exec
    options:
      host: login.cluster.example.org
      use_slurm: true
      partition: compute
      max_parallel: 8

Jobs are then submitted with sbatch --parsable, and the handle MatMelt stores is the SLURM job id rather than a process id. Resources declared by the stage become --cpus-per-task and --time. Everything else — polling, fetching, the funnel — is unchanged.

What restart looks like now#

The handle is a durable id: a remote PID, or a SLURM job id. Kill your local matmelt run mid-flight and the remote work carries on; run it again and it re-attaches to the recorded job instead of submitting a second one. This is why a long remote run survives a closed laptop lid.

What you learned#

  • Offloading a stage is a backend entry plus one backend: line — no Python changes anywhere.

  • The stage’s python becomes an interpreter on the remote.

  • doctor probes dependencies and the model file where the job will run, and reports the location alongside the answer.

  • A remote receives only the runner script and the declared inputs; declared outputs come back. It never receives MatMelt itself.

  • Staging is the default; shared_filesystem: true is an opt-in optimisation.

  • use_slurm: true swaps process ids for scheduler job ids and changes nothing else.

Next: T4 watches a run like this one while it drains. For the same move as a short recipe, see Run a stage on another machine.