Value objects#

The vocabulary that crosses the stage/backend boundary. Every other page refers to these types.

The value objects that cross the stage <-> backend boundary (Phase 1.1).

This module is pure data – stdlib only, no I/O, no third-party imports – so everything else (stages, backends, the orchestrator, tests) can depend on it without dragging anything in.

The types encode the tool’s central thesis: separate *what* to compute from *where* it runs. A stage describes work as a portable JobSpec (“run this command; here are its input files; it needs these resources”) and never names a host, a queue, or a transport. A backend turns that spec into a real execution – a local subprocess today, a remote SSH/SLURM job later – and reports back through these same types. Keeping them free of any backend detail is what makes “run the screening stage on my GPU box instead” a one-line config change.

The second load-bearing idea is durability: matmelt run is an on-demand tool the user may interrupt at any time. Every reference to in-flight work (Handle) must therefore be plain persistable identity – a PID, a SLURM id – never a live object, so an interrupted run can be resumed and its jobs re-polled instead of orphaned.

exception matmelt.core.jobspec.BackendError[source]#

Bases: Exception

An infrastructure failure in the compute plane.

Raised for what the backend itself could not do: an unreachable host, a handle it does not recognise, a malformed JobSpec, a submission the target rejected. It is never used for a job that ran and exited non-zero – that is a normal JobResult for the stage to judge. The distinction is what lets the orchestrator treat a sick backend (pause, retry later) differently from a failed calculation (a per-material verdict).

class matmelt.core.jobspec.JobStatus(value)[source]#

Bases: str, Enum

Where a submitted job is in its lifecycle, as reported by poll.

A str enum: members compare equal to their plain-string values and serialise as-is, so a status can be persisted to SQLite or a log line without custom conversion. The four states are deliberately backend-agnostic – a local subprocess and a SLURM job both map onto exactly these (e.g. SLURM PD -> PENDING, R -> RUNNING, CD -> DONE, F/CA/TO -> FAILED).

DONE and FAILED are terminal: once reached, poll keeps returning the same status and fetch becomes meaningful. DONE means “the backend ran the job to completion”, not “the science succeeded” – inspect the JobResult for that.

Examples

>>> JobStatus.DONE == "DONE"
True
>>> str(JobStatus.RUNNING.value)
'RUNNING'
property is_terminal: bool#

Whether this status can still change.

Returns:

True for DONE and FAILED – the job is over and fetch is meaningful. False for PENDING and RUNNING.

Return type:

bool

Examples

>>> JobStatus.RUNNING.is_terminal
False
>>> JobStatus.FAILED.is_terminal
True
class matmelt.core.jobspec.Resources(cores=1, gpu=False, walltime_s=None)[source]#

Bases: object

A portable hint of what a job needs – carried by JobSpec.resources.

A stage declares needs here; a backend asserts them against its own Capabilities before accepting work, so an impossible placement (a GPU stage on a CPU-only backend) is caught up front rather than as a late job failure.

cores#

CPU cores the job should be given. A SLURM-wrapping backend maps this onto --cpus-per-task.

Type:

int

gpu#

Whether the job needs a GPU, asserted against Capabilities.gpu.

Type:

bool

walltime_s#

Wall-clock limit in seconds, or None when the stage makes no promise. A SLURM-wrapping backend may translate it into a time limit; a local backend may ignore it.

Type:

int or None

Examples

>>> Resources()
Resources(cores=1, gpu=False, walltime_s=None)
>>> Resources(cores=8, gpu=True, walltime_s=3600).gpu
True
class matmelt.core.jobspec.JobSpec(command, input_artifacts=<factory>, output_artifacts=<factory>, resources=<factory>, env=<factory>, cwd=None, scratch_inputs=<factory>)[source]#

Bases: object

An immutable description of what to run – handed to submit.

Built by a stage (Stage.build_jobspec is the only place a command is constructed); the only thing that crosses into a backend. It must be runnable by any backend, so it names no host, queue, or transport.

command#

The argv to execute – list form, never a shell string, so there are no quoting or injection surprises across local vs remote execution.

Type:

list of str

input_artifacts#

Name -> path of files the job needs staged before it runs; a local backend links them in place, a remote one uploads them. The name is where the file appears relative to the job’s effective cwd (symmetric with output_artifacts), so a command may reference a staged file – e.g. its runner script – by bare name. Paths, never contents: file bodies do not travel inside the spec.

Type:

dict

output_artifacts#

Names of files the job is expected to produce, located and collected by fetch afterwards. A name may denote a directory tree, which is staged back whole.

Type:

list of str

resources#

The portable resource hint, asserted against the backend’s Capabilities.

Type:

Resources

env#

Environment overlay applied to the job’s process.

Type:

dict

cwd#

Working directory the command runs in – the item’s directory, so outputs land beside its science data. None lets the backend choose.

Type:

str or None

scratch_inputs#

The subset of input_artifacts names that are the job’s own implementation (its self-contained runner script), not the item’s science data. A backend stages these into its private per-job dir rather than the effective cwd, and rewrites the matching command token to that staged path, so the material’s working directory keeps only science artifacts. Empty by default (every input staged into cwd).

Type:

frozenset of str

Examples

A stage builds a spec that names a runner, its input, and its promised output – and nothing about where any of it runs:

>>> spec = JobSpec(
...     command=["python3", "screen_runner.py", "--structure", "structure.cif"],
...     input_artifacts={"screen_runner.py": "/pkg/screen_runner.py",
...                      "structure.cif": "/work/mat-1/structure.cif"},
...     scratch_inputs=frozenset({"screen_runner.py"}),
...     output_artifacts=["screen_result.json", "relaxed.cif"],
...     cwd="/work/mat-1",
... )
>>> spec.resources.cores
1
class matmelt.core.jobspec.Handle(backend_name, job_id, extra=<factory>)[source]#

Bases: object

An opaque, durable reference to one submitted job.

Returned by submit; passed back to poll / fetch / release. Deliberately not a live object – just enough identity to find the job again.

backend_name#

Which configured backend issued it, so a handle is never resolved against the wrong one.

Type:

str

job_id#

That backend’s native id (a PID, a SLURM job id). Must keep resolving after the tool restarts – never a socket, Popen, or any in-memory object. This is what lets an interrupted matmelt run resume and re-poll its in-flight jobs instead of orphaning or resubmitting them.

Type:

str

extra#

Free-form backend bookkeeping (the job dir, stream paths) needed to reconstruct the job later without any in-memory map. Values must be JSON-serialisable: the whole handle is persisted to the state store’s in-flight ledger.

Type:

dict

Examples

>>> handle = Handle(backend_name="gpubox", job_id="41231",
...                 extra={"job_dir": "/tmp/matmelt-jobs/mcli-41231"})
>>> handle.job_id
'41231'
class matmelt.core.jobspec.JobResult(exit_code, stdout='', stderr='', output_artifacts=<factory>)[source]#

Bases: object

What a finished job produced – returned by fetch.

Meaningful once poll reports a terminal status.

exit_code#

Load-bearing semantics – do not weaken. 0 is conventional success; non-zero means the job ran and failed, reported faithfully, never raised (judging scientific success is the stage’s job); None means the job ran but its code could not be obtained (killed before exiting, a missing exit-code sentinel after a crash). “Ran, code unknown” must never be defaulted to 0 – a stage treats None as not-proven-successful.

Type:

int or None

stdout#

The job’s captured standard output. A backend may truncate very large streams.

Type:

str

stderr#

The job’s captured standard error, which is where a runner’s actionable failure messages arrive.

Type:

str

output_artifacts#

Name -> path of the files named in JobSpec.output_artifacts that were actually found. A missing expected file is simply absent from the dict, for the stage to judge.

Type:

dict

Examples

>>> JobResult(exit_code=0, output_artifacts={"record.json": "/work/m/record.json"}).exit_code
0
>>> JobResult(exit_code=None).exit_code is None   # ran, code unknown -- not success
True
class matmelt.core.jobspec.Capabilities(cpu=True, gpu=False, slurm=False, max_parallel=1, extra=<factory>)[source]#

Bases: object

What a backend can offer – returned by capabilities.

The counterpart a stage’s Resources needs are asserted against before any work is submitted.

cpu#

Whether the backend can run CPU work at all. Effectively always True.

Type:

bool

gpu#

Whether a GPU is available to jobs on this backend.

Type:

bool

slurm#

Whether jobs are submitted through a SLURM queue rather than launched directly.

Type:

bool

max_parallel#

How many jobs the backend will run at once – advertised here, enforced by the orchestrator.

Type:

int

extra#

Open-ended extras (memory, partitions, accelerators) so a backend can describe itself more richly without changing this type.

Type:

dict

class matmelt.core.jobspec.HealthStatus(ok, detail='')[source]#

Bases: object

Whether a backend is reachable right now – returned by health_check.

health_check itself never raises: being down is the normal answer it reports.

ok#

Whether the backend can take work right now.

Type:

bool

detail#

Short, human-readable reason, shown verbatim in logs and doctor output. Must never contain a secret – no keys, tokens, or passwords.

Type:

str

Examples

>>> HealthStatus(ok=False, detail="ssh: connect to host gpubox port 22: No route to host").ok
False