The two contracts#

MatMelt has exactly two extension points. Both are abstract base classes with a registry in front of them, and both are covered by conformance tests you inherit by implementing them.

ComputeBackend — six methods#

submit(spec) -> Handle          # start a job, return a durable identity
poll(handle) -> JobStatus       # what is it doing now?
fetch(handle) -> JobResult      # what did it do?
release(handle) -> None         # clean up; idempotent
health_check() -> HealthStatus  # usable right now? never raises
capabilities() -> Capabilities  # what does this target offer?

The obligations that matter are not in the signatures:

Two failure tiers. A non-zero exit is a normal JobResult. Only infrastructure failures raise BackendError — an unreachable host, an unusable scratch directory, a malformed spec. A backend that raises because a job exited 1 has broken the contract, and the tool will treat a scientific rejection as a transport problem.

Durable handles. poll, fetch and release must work from a persisted Handle alone, in a process that never called submit. No in-memory map may be required. The local backend keeps one, but only to reap child processes — never to answer a poll.

health_check never raises, and never leaks a secret. A backend that is down returns an unhealthy status saying so.

release is idempotent and driver-owned. Calling it twice, or on a job that already finished, is fine. The orchestrator decides when a job is done with; the backend does not release itself.

Construction is (name, options). The name is unique per instance and rides into every Handle; the options dict is copied and validated by the backend itself, which is why an invalid option is caught while wiring the run.

Register with register(kind, cls) — explicit, not an import-time decorator, so the set of known kinds does not depend on which modules happened to be imported. Re-registering the same class is a no-op; shadowing a different one is an error.

Stage — five abstract members, two concrete enforcers#

name                            # the stage's key in config and the registry
requires  -> (StageInputSpec…)  # declared inputs
produces  -> (StageOutputSpec…) # declared outputs
build_jobspec(item) -> JobSpec  # the ONLY place a command is constructed
interpret_result(item, result) -> StageVerdict

plus two the base class implements for you, and which you should not override:

validate_inputs(item)           # refuse clearly, naming what is missing
validate_outputs(item, result)  # a zero exit without its files is not success

build_jobspec being the only place a command is built is what keeps What vs. where true: there is one place to look to confirm no stage names a machine.

Class attributes complete the contract: fan_out (does one item become many?), runtime_requirements (what the preflight probes), and runner_python (the interpreter, from config).

Registration mirrors backends: register(name, cls), with create_stage(name, options) doing cls(**options) — the typed constructor signature is the option contract, so validation lives in one place and the documentation of options can be checked against it automatically.

Three module-level facts in core/stages/registry.py define the funnel: FUNNEL_ORDER, OPTIONAL_STAGES, and STAGE_OUTPUT_STATE — the map from stage name to the state a completed material lands in. The orchestrator derives input states from that map and the configured funnel, which is what makes an optional stage possible without special cases (The funnel, and what stopping means).

Conformance tests come with the contract#

tests/unit/test_backend_contract.py runs the same ten checks against every backend through a parametrized fixture: lifecycle, a faithful non-zero exit, outputs collected, a stable fetch, a sticky terminal poll, an idempotent release, unknown-handle behaviour. A new backend joins by adding one branch to that fixture — and then either satisfies the contract or fails tests that already exist.

Stages get the same treatment from the other direction: FakeBackend fabricates each declared output, so a stage’s whole lifecycle — validate, build, submit, interpret, validate — runs with no science stack installed.

Adding one#

A backend: subclass ComputeBackend, implement the six methods, register it, add the fixture branch. If it can probe a remote interpreter, also provide check_remote_import / check_remote_file; the preflight uses them when present, and the six-method ABC stays untouched — an opt-in by duck typing rather than a wider mandatory interface.

A stage: write the runner script first (it is the part with the science, and it is runnable by hand), then the thin Stage class around it, then register it and add it to FUNNEL_ORDER and STAGE_OUTPUT_STATE. The state machine may need a new member; adding one is additive, as CALCULATED was.