Run a stage on another machine#

Screening is usually the expensive stage. Moving it to a bigger machine is two edits to matmelt.yaml: describe the machine under backends:, then point the stage at it. No Python changes, and nothing of MatMelt is installed on the remote.

Before you start#

Passwordless SSH must already work, non-interactively — MatMelt connects with BatchMode=yes and never prompts:

$ ssh alice@cluster.example.org true && echo ok

The remote needs rsync and a Python environment holding the science package for the stage you are offloading — nothing else. Create it once:

$ ssh alice@cluster.example.org
$ python3 -m venv ~/science-venv && ~/science-venv/bin/pip install mace-torch

Add the backend and rewire the stage#

backends:
  laptop:
    kind: local
    options:
      work_dir: ./materials-jobs
  cluster:
    kind: ssh_exec
    options:
      host: cluster.example.org
      user: alice
      remote_work_dir: /scratch/alice/matmelt-jobs
      max_parallel: 4

stages:
  screen:
    backend: cluster
    options:
      python: /home/alice/science-venv/bin/python
      device: cuda

Only two things changed: backend: laptop became backend: cluster, and python now names the interpreter on that machine. Everything else — compose, structure, extract — still runs locally.

Keep credentials out of the file with env: references:

backends:
  cluster:
    kind: ssh_exec
    options:
      host: env:CLUSTER_HOST
      user: env:CLUSTER_USER

Check it before running#

$ matmelt doctor

doctor reaches across the network for both tables. When something is wrong it says so in the SSH client’s own words:

│ cluster │ SshExecBackend │ DOWN    │ alice@cluster.example.org: work dir     │
│         │                │         │ /tmp/matmelt-jobs not usable (ssh exit  │
│         │                │         │ 255) ssh: Could not resolve hostname    │
│         │                │         │ cluster.example.org: Name or service    │
│         │                │         │ not known                               │

When it is right, the readiness table shows the probe location and, if the package is missing, an ssh command you can paste:

│ screen    │ mace        │ /opt/scien… │ alice@clust… │ MISSING │ ssh: Could  │
...
To fix the missing requirements, run:
  ssh alice@cluster.example.org /opt/science-venv/bin/python -m pip install
mace-torch

Where naming alice@cluster.example.org rather than local is the confirmation that the check went to the right machine.

Run it#

$ matmelt run --yes

Nothing about the command changes. Per job, MatMelt creates a fresh directory under remote_work_dir, pushes the stage’s self-contained runner script and the files that stage declared, runs it there, and pulls the declared outputs back into the local material directory. The remote directory is removed when the job is released.

/scratch/alice/matmelt-jobs/mcli-4f8b2a1c9d3e/
├── screen_runner.py      # shipped per job
├── structure.cif         # the declared input
├── job.rc  job.out  job.err
├── screen_result.json    # pulled back
└── relaxed.cif           # pulled back

There is no pip install matmelt on that machine, no PYTHONPATH, and no assumption that our package is importable there. That is why the remote needs only your science stack.

Submit through SLURM#

backends:
  cluster:
    kind: ssh_exec
    options:
      host: cluster.example.org
      user: alice
      use_slurm: true
      partition: gpu
      max_parallel: 8

Jobs then go through sbatch --parsable, and the handle MatMelt records is the SLURM job id rather than a PID — so an interrupted run re-attaches to jobs still queued or running. Resources become --cpus-per-task and --time.

Skip staging on a shared filesystem#

If the remote already sees your working directory at the same absolute path (NFS, SSHFS), skip the transfers:

backends:
  cluster:
    kind: ssh_exec
    options:
      host: cluster.example.org
      shared_filesystem: true

This is an optimisation you opt into. The default assumes nothing is shared, which is what makes an arbitrary machine usable.

When it goes wrong#

Symptom	Usual cause
Backend `DOWN`, `ssh exit 255`	Host, user, or key — the detail line quotes SSH
Backend healthy, stage `MISSING`	The science package is not in the remote `python`
Jobs submit but outputs never come back	The stage produced nothing; check `job.err` in the remote job dir before the run releases it
Everything slow, nothing parallel	`max_parallel` is 1 by default on every backend

Note

The end-to-end output on this page comes from a recorded run on our own hardware (the Phase 4 acceptance gate: a real screen stage staged to a compute server, tracked by remote PID, relaxed.cif pulled back, verdict RELAXED in 17 s). Two machines cannot be reproduced while writing docs. The doctor failure output above is live, and every config key, flag and message shape on this page is checked against the code by the test suite.