Built-in backends

Running jobs on this machine, or on a machine you own over SSH.

The local backend: run a JobSpec as a subprocess on this machine.

The zero-setup default – and the reference implementation of the durability contract. Registered under kind = "local" at import time.

Execution model (the why behind the shape):

• submit is non-blocking: it stages inputs into a fresh per-job directory, launches the command through a tiny job.sh wrapper, and returns a Handle immediately. Progress is observed via poll; results via fetch.

• The ``.rc`` sentinel is the durability mechanism. The wrapper runs the command, then atomically writes its exit code to job.rc (tmp-file + mv). Everything poll/fetch need afterwards lives on disk (job.rc, stdout.log/stderr.log, a job.json manifest), so a restarted tool resolves the same handle with no in-memory state: rc file present -> terminal with a faithful code; rc absent + process alive -> RUNNING; rc absent + process gone -> FAILED with exit_code=None (“ran, code unknown” – never defaulted to 0).

• PID reuse is defeated by a start-time check. The handle records the wrapper’s PID and its /proc/<pid>/stat start time; a live PID only counts as our job if the start time still matches (best-effort: on hosts without /proc the check degrades to PID-aliveness).

• Two failure tiers. A malformed spec raises BackendError (programming error, nothing launched). A job-level problem is never raised: a missing input artifact becomes a durable already-failed job (detail in stderr.log, no process), and a command-not-found simply exits 127 through the wrapper – one bad item fails while the backend stays healthy.

Stdlib-only (subprocess/pathlib/shutil/shlex/tempfile). No compute detail is hardcoded: work_dir/gpu/max_parallel come from the backend’s configured options.

class matmelt.backends.local.LocalBackend(name, options)

Bases: ComputeBackend

Run jobs as child processes on this machine.

The default backend, and the one that needs no setup: a bare {kind: local} works.

Parameters:

• name (str) – The backend’s configured name, carried into every Handle it issues.

• options (dict[str, Any]) –

  The config block. All keys are optional:

  work_dir

  Base directory for per-job directories. Defaults to a private temp directory – zero setup, but those job directories die with the OS temp cleaner, so configure a persistent path to keep in-flight jobs resumable across restarts.

  gpu

  Whether this host exposes a GPU to jobs. Default False, and config-driven rather than auto-detected, so capabilities stay predictable.

  max_parallel

  How many jobs this backend advertises it can run at once. Default 1; advertised here, enforced by the orchestrator.

Raises:

BackendError – For an option of the wrong type or an unusable work_dir.

Notes

Durability is what distinguishes this from a plain Popen wrapper. Each job gets a directory holding a wrapper script, an atomically-written exit-code sentinel, both captured streams, and a manifest of what the job promised to produce. poll and fetch read only those, so a restarted tool resolves jobs it never launched – no in-memory map is required, and an interrupted run resumes rather than orphaning its work.

capabilities()

Report what this backend offers (cpu/gpu/slurm, max_parallel).

Pure introspection – nothing is submitted, nothing mutates. A stage’s Resources needs are asserted against this before any work is handed over, so an impossible placement (a GPU stage on a CPU-only backend) is caught up front rather than as a late job failure.

Returns:

What this target offers. max_parallel is advertised here and enforced by the orchestrator, not the backend.

Return type:

Capabilities

health_check()

Probe that the job root is writable. Never raises.

Returns:

Whether a file can be created and removed under work_dir, with the reason when it cannot.

Return type:

HealthStatus

sample_cpu_percents(handles)

Sample CPU usage per job – the monitor’s duck-typed CPU hook.

Parameters:

handles (Sequence[Mapping[str, Any]]) – The ledger’s serialized handles, in order. The wrapper PID lives at extra.pid.

Returns:

One reading per handle, in the same order, as a percentage of one core. Readings come from the shared LocalCpuTracker – /proc tick deltas over each job’s whole process subtree – so the first call after startup answers None for every job, that being the delta baseline. A handle without an integer PID, or a PID no longer running, is also None.

Return type:

list

Notes

Never raises: this is cosmetic telemetry, and its contract is that a broken reading can never take the dashboard down.

sample_gpu_percents(handles)

Sample GPU usage per job – the monitor’s duck-typed GPU hook.

Parameters:

handles (Sequence[Mapping[str, Any]]) – The ledger’s serialized handles, in order.

Returns:

One SM-utilisation reading per handle, summed over that job’s process subtree. A machine without nvidia-smi, or one where it fails or times out, answers all None – unknown, never zero. A live job with no pmon entry is an honest 0.0: running, but not on the GPU.

Return type:

list

Notes

One nvidia-smi pmon -c 1 -s u call answers the whole batch. Never raises.

submit(spec)

Start spec through a shell wrapper and return a durable handle.

Parameters:

spec (JobSpec) – The job to run. Its cwd is the material’s directory, so outputs land beside the science data; scratch_inputs stage into this backend’s private job directory instead, and the matching command token is rewritten to point there – which is what keeps runner scripts out of material directories.

Returns:

Carrying the job directory, the wrapper PID, and that PID’s /proc start time.

Return type:

Handle

Raises:

BackendError – Only for a structurally malformed spec, which is a programming error. A job-level problem – an unstageable input, for instance – becomes a durable already-failed job instead, whose handle still resolves: one bad item must not look like a sick backend.

Notes

The wrapper writes the exit code to a sentinel file via tmp-then-rename after the command returns. That sentinel, not the process, is what poll and fetch believe – and recording the PID start time alongside the PID is what makes re-attachment safe against PID reuse.

poll(handle)

Report status from durable evidence alone.

Parameters:

handle (Handle) – The job to poll, possibly reloaded from the ledger in a later process.

Returns:

DONE or FAILED when the exit-code sentinel exists; RUNNING when the wrapper process is alive and still ours; FAILED otherwise – no exit code and no live process means the job never started, died before writing the sentinel, or could not write it.

Return type:

JobStatus

Raises:

BackendError – When the handle’s job directory does not exist: unknown, or already released.

Notes

Works identically for a job this instance launched and one re-attached after a restart, because it consults the sentinel and /proc rather than any in-memory state.

fetch(handle)

Collect the faithful result from the job directory. Restart-safe.

Parameters:

handle (Handle) – The job to collect.

Returns:

The sentinel’s exit code – None when absent or garbled, which means “ran, code unknown” and is never defaulted to 0 – both captured streams, and the declared outputs that were actually produced, resolved against the job’s recorded cwd.

Return type:

JobResult

Raises:

BackendError – When the job directory is gone, or when the job is still running and was launched by a previous process, which cannot be waited on. Poll until terminal instead.

Notes

A still-running job this instance owns is waited for, so fetch never returns a half-finished result.

release(handle)

Stop the job if it is ours, then remove its directory.

Parameters:

handle (Handle) – The job to clean up. Releasing an unknown or already-released handle is a no-op, so restart-time cleanup is always safe.

Return type:

None

Notes

Idempotent and never raises. A process this instance launched is SIGTERMed then SIGKILLed; a re-attached one is killed only after the start-time check has proven the PID is still our wrapper and not a recycled stranger.

The job directory holds this backend’s scratch, including the staged runner script, so removing it is what keeps our implementation files out of the material directory – which only ever received the item’s science data and outputs.

The ssh_exec backend: run a JobSpec on a remote host over SSH (4.1).

Modelled on MVP_V0’s ssh_exec (its durable-native-id + .rc-sentinel + single-SSH-seam shape is proven) but with the data model inverted per the 2026-07-08 decisions: MVP_V0 assumes a shared filesystem and stages nothing; here declarative staging is the default and shared_filesystem: true is the opt-out optimization.

• Remotes never receive our package code. A remote is a pure calculation executor: the only things there are the user’s environment (remote_python with the science deps) and what we push per job. Every job’s declared input_artifacts – which, by the Phase-3 stage discipline, always include the stage’s self-contained runner script – are staged into a fresh per-job remote directory; the command runs there (artifact names resolve in the cwd, same as local); declared output_artifacts are pulled back into the job’s local cwd (the item dir), where the stage judges them. Our runners are stdlib+science only, so there is no PYTHONPATH plumbing at all.

• The staging transport is a seam. _stage_in / _stage_out wrap rsync-over-SSH today; an S3/Globus transport plugs in behind the same two calls with zero stage changes. With shared_filesystem: true both become no-ops: the job runs in spec.cwd (the same absolute path on both sides) and outputs are collected locally, exactly like local.

• A durable native id. Handle.job_id is the remote PID (or the SLURM job id under use_slurm); everything poll/fetch/release need (scheme, remote job dir, log/sentinel paths, output names, local cwd) rides in Handle.extra as plain JSON-safe strings – no in-memory job map, so a restarted orchestrator re-attaches from the persisted handle alone.

• Exit-code fidelity via the ``.rc`` sentinel. The remote wrapper appends echo $? > <rc>; fetch reads it back. A missing/garbled sentinel is exit_code=None (“ran, code unknown”) – never a silent 0. The SLURM path prefers sacct’s ExitCode and falls back to the sentinel.

Stdlib + the system ssh/rsync (and remote sbatch/sacct) only. Key-based auth must already work: every SSH runs BatchMode=yes and never prompts. All compute details come from options – no host, user, path, or queue literal in code, and health_check never leaks a secret.

class matmelt.backends.ssh_exec.SshExecBackend(name, options)

Bases: ComputeBackend

Run jobs on a remote host over SSH, staging files per job by default.

This is the backend that makes “run the screening stage on my GPU box” a config change rather than a code change.

Parameters:

• name (str) – The backend’s configured name, carried into every Handle it issues.

• options (dict[str, Any]) –

  The config block. Only host is required:

  host

  An ~/.ssh/config alias, a hostname, or user@host.

  user

  SSH user, when host carries none.

  ssh_binary, rsync_binary

  Client binaries, defaulting to ssh and rsync.

  remote_python

  Interpreter to substitute for argv[0] remotely. Normally unnecessary – the stage’s own python option already names the remote interpreter – but this override wins when both are set.

  remote_work_dir

  Root for per-job remote directories, default /tmp/matmelt-jobs. Each job gets a fresh subdirectory, removed by release.

  shared_filesystem

  true skips data staging, assuming the remote sees the job’s cwd at the same absolute path over NFS or SSHFS. Scratch inputs – the runner script – are still pushed into the per-job directory and referenced by absolute path, because they are deliberately kept out of the material directory. Default false.

  use_slurm

  Submit through sbatch and track the SLURM job id instead of a bare PID. partition sets the queue, and JobSpec.resources becomes --cpus-per-task and --time. Default false.

  gpu, max_parallel

  Advertised in capabilities.

Raises:

BackendError – For a missing host or any option of the wrong type.

Notes

Remotes never receive MatMelt’s code. A remote is a pure calculation executor: what travels is the job’s self-contained runner script plus its declared input files, and what comes back are the declared outputs. There is no pip install on the remote, no PYTHONPATH, and no assumption that our package is importable there.

Declarative staging is therefore the default, and shared_filesystem is an optimization that skips it. Both paths run through the same two seams, _stage_in and _stage_out, so a future S3 or Globus transport needs no stage changes at all.

Everything poll, fetch and release need rides in Handle.extra as JSON-safe values – there is no in-memory job map, so a restarted run re-attaches to live remote jobs from the ledger alone. Every token crossing the SSH boundary is quoted with shlex.quote.

capabilities()

Report what this backend offers (cpu/gpu/slurm, max_parallel).

Pure introspection – nothing is submitted, nothing mutates. A stage’s Resources needs are asserted against this before any work is handed over, so an impossible placement (a GPU stage on a CPU-only backend) is caught up front rather than as a late job failure.

Returns:

What this target offers. max_parallel is advertised here and enforced by the orchestrator, not the backend.

Return type:

Capabilities

health_check()

Check that the remote is reachable and its work root writable. Never raises.

Returns:

Down is a normal answer, reported with a short, secret-free detail. The orchestrator pauses this backend’s stages on a down report and resumes them by itself once it recovers.

Return type:

HealthStatus

property probe_target: str

Where this backend’s remote probes run.

Returns:

The SSH target. doctor reports it as the location of a readiness answer, and builds the ssh-prefixed install command from it – an answer is only as good as where it was obtained.

Return type:

str

check_remote_import(python, module)

Ask whether the remote interpreter can import a module. Never raises.

Closes the preflight’s blind spot for remotely-wired stages: the science stack must exist in the interpreter that will actually run the staged runner, not in ours.

Parameters:

• python (str) – The remote interpreter path.

• module (str) – The import name to try.

Returns:

(satisfied, detail). Transport failure, a missing remote interpreter, and a failing import all come back as (False, why).

Return type:

tuple

Notes

Duck-typed rather than part of the six-method ABC: core/preflight looks for this hook, so any future remote backend opts in simply by providing it and probe_target.

check_remote_file(path)

Ask whether a path exists as a regular file on the remote. Never raises.

The file-existence analogue of check_remote_import. A stage option naming a path – the screen stage’s model_path – points at where the stage runs, so stat-ing our own disk would report a false problem and could refuse a perfectly valid run.

Parameters:

path (str) – The remote path to test.

Returns:

(present, detail). Transport failure and a missing file both come back as (False, why).

Return type:

tuple

sample_cpu_percents(handles)

Sample remote CPU usage per job – the monitor’s duck-typed CPU hook.

Parameters:

handles (Sequence[Mapping[str, Any]]) – The ledger’s serialized handles, in order.

Returns:

One reading per handle, summed over that job’s remote process subtree. These are ps pcpu semantics – a lifetime average rather than an instantaneous rate, which is honest enough for long-running science jobs. None for a SLURM-scheme handle, where the scheduler owns the process and we hold only its job id; for a PID no longer in the table; and for any transport failure.

Return type:

list

Notes

One batched ps round-trip answers every handle at once, and the caller caches the result behind a TTL so a dashboard refresh does not mean an SSH round-trip. Never raises.

sample_gpu_percents(handles)

Sample remote GPU usage per job – the monitor’s duck-typed GPU hook.

Parameters:

handles (Sequence[Mapping[str, Any]]) – The ledger’s serialized handles, in order.

Returns:

One SM-utilisation reading per handle, summed over that job’s remote process subtree. A remote without nvidia-smi answers all None – unknown, never zero – as do SLURM-scheme handles and any transport failure. A live job with no pmon entry is an honest 0.0.

Return type:

list

Notes

One combined round-trip fetches the process table and the pmon table together, separated by a sentinel line. Never raises.

submit(spec)

Stage inputs, launch remotely, and return a durable handle. Non-blocking.

Parameters:

spec (JobSpec) – The job to run.

Returns:

Carrying the remote job directory, the stream and sentinel paths, the declared outputs, the local cwd, and the tracking scheme – all JSON-safe, so the ledger alone is enough to resolve the job after a restart.

Return type:

Handle

Raises:

BackendError – For a malformed spec, an unstageable input, an uncreatable remote directory, a failed launch, or a launch that yielded no trackable id.

Notes

Under the staged model (the default) a fresh per-job directory is created, every declared input is pushed into it, and the command runs there – so artifact names, the runner included, resolve in the cwd exactly as they do under local.

Under shared_filesystem, data inputs are not staged at all, since the remote already sees them at spec.cwd. The scratch inputs are still pushed into the per-job directory, because the runner is deliberately kept out of the material directory, and the matching command tokens are rewritten to that absolute remote path.

Either way the launch wrapper appends an exit-code sentinel, and the job is tracked by remote PID – or by SLURM job id under use_slurm, where JobSpec.resources becomes scheduler directives.

poll(handle)

Map the remote job’s state to a portable JobStatus. Idempotent.

Parameters:

handle (Handle) – The job to poll, possibly reloaded from the ledger in a later process.

Returns:

Under the PID scheme, a live process is RUNNING and anything else is DONE – the true outcome is fetch’s business, read from the sentinel. Under SLURM, sacct states map conservatively, and an empty accounting line is ``RUNNING``: a job not yet visible to the accounting database must never be guessed into a terminal state.

Return type:

JobStatus

Raises:

BackendError – For a handle this backend did not issue, or transport failure.

fetch(handle)

Collect the finished job: streams, true exit code, pulled outputs.

Parameters:

handle (Handle) – The job to collect.

Returns:

The exit code comes from the sentinel, or from sacct first under SLURM with the sentinel as fallback. A missing or garbled sentinel is None – “ran, code unknown”, never a silent 0 – and a non-zero code is reported faithfully rather than raised. Outputs are pulled back through the staging seam, or collected locally under shared_filesystem; absent ones are omitted for the stage to judge.

Return type:

JobResult

Raises:

BackendError – For a handle this backend did not issue, or transport failure.

release(handle)

Terminate a still-running job and remove its remote directory.

Parameters:

handle (Handle) – The job to clean up: kill under the PID scheme, scancel under SLURM, then the per-job directory removed.

Return type:

None

Notes

Idempotent and best-effort by contract. An unreachable host or an already-gone job never raises, because restart-time cleanup must always be safe. Never touches material state – that is the caller’s business.