State and storage#

A run’s durable state is one SQLite file and a directory per material. There is no server, no daemon, and no state that exists only in memory.

The schema#

Four tables, applied idempotently on every open — every statement is IF NOT EXISTS, which is the migration mechanism: adding a table or an index is a change to the schema constant, and an existing workspace picks it up when next opened.

materials(material_id PK, state, claimed_at, parent_id, attempts,
          error, fail_stage, fail_reason, created_at, updated_at)

stage_completions(material_id, stage, completed_at,
                  PRIMARY KEY (material_id, stage))

jobs(job_key PK, stage, backend_name, handle, submitted_at)

job_materials(material_id PK, job_key)

Each carries an invariant in its shape rather than in code that must remember:

  • stage_completions’s composite primary key makes completion idempotent. Replaying a completion is a no-op at the storage layer, which is what makes an interrupted run safe to simply re-run.

  • job_materials.material_id is the primary key, so “one live job per material” is enforced by the schema. A second submission for the same material cannot be recorded, rather than being prevented by a check someone could forget.

  • handle is a JSON blob, which is why Handle.extra must be JSON-safe (Value objects).

  • claimed_at is a lease, not a lock — a timestamp, so an abandoned claim ages out instead of requiring a cleanup process.

Two indexes exist for named hot paths: claiming filters on (state, claimed_at), and the monitor’s throughput window filters on completed_at.

Concurrency discipline#

WAL mode, a busy_timeout, and BEGIN IMMEDIATE on every read-then-write. The last one is what makes claim atomic: read-then-write under SQLite’s default deferred transaction is a race, and claiming is exactly that pattern. A test drives two store instances against one file from two threads to prove a material is never claimed twice.

The store is deliberately leaner than a multi-worker design: there is no claimed_by, because a single-process tool’s lease is just claimed_at, and no artifacts table, because files live in the material’s directory where you can look at them.

The workspace on disk#

materials-work/                  # run.work_dir
├── state.sqlite                 # everything above
├── Cl1Na1/                      # a fan-out parent
└── Cl1Na1_3D-6/                 # one material, one directory
    ├── structure.cif            # from structure
    ├── structure.json           # generation provenance
    ├── relaxed.cif              # from screen
    ├── screen_result.json       # the verdict
    └── record.json              # from extract

Job scratch lives elsewhere — under the backend’s own work_dir — so a material directory holds results rather than a copy of the tool. That separation was a deliberate fix: runner scripts used to be staged into the material directory and left symlinks among the science files.

The layout is meant to be readable with ls and cat. A researcher should be able to answer “what happened to this material” without the tool, and a workspace should be movable and archivable as an ordinary directory tree.

Read-only observers#

StateStore(read_only=True) sets SQLite’s query_only pragma, so writes are refused by the engine, not by our good intentions. A missing file is an error rather than a silently created empty database — asking about a run that does not exist should say so.

status, results, monitor and cif all open the store this way. Watching a run cannot disturb it, and there is a test asserting the database file is byte-identical after a read-only command.

The monitor's data flow: a read-only state store, a CPU and GPU usage sampler, and a cached backend health probe feed the pure build_snapshot function, which returns a MonitorSnapshot value; the Rich renderers turn that one value into the dashboard panels. Nothing on the path writes to the run. The monitor's data flow: a read-only state store, a CPU and GPU usage sampler, and a cached backend health probe feed the pure build_snapshot function, which returns a MonitorSnapshot value; the Rich renderers turn that one value into the dashboard panels. Nothing on the path writes to the run.

The observer path is worth reading as a pattern for any future view. Sources — the read-only store, the usage sampler, a cached health probe — feed build_snapshot, a pure function with no knowledge of Rich, SQL or config, which returns one MonitorSnapshot. Every panel is then a function of that single value, which is why the whole dashboard is testable without a terminal.

Sampling rates differ on purpose: the store and local CPU every tick, a remote at most every 10 s, backend health at most every 30 s. A dashboard must not become load on the thing it is watching. Telemetry is best-effort throughout — it can fail, time out, or not exist, and the result is a dash rather than a zero and never an exception.