---
description: The four-table SQLite schema, the concurrency discipline behind it, what a workspace looks like on disk, and how read-only observers work.
---

# State and storage

A run's durable state is one SQLite file and a directory per material. There is
no server, no daemon, and no state that exists only in memory.

## The schema

Four tables, applied idempotently on every open — every statement is
`IF NOT EXISTS`, which *is* the migration mechanism: adding a table or an index
is a change to the schema constant, and an existing workspace picks it up when
next opened.

```sql
materials(material_id PK, state, claimed_at, parent_id, attempts,
          error, fail_stage, fail_reason, created_at, updated_at)

stage_completions(material_id, stage, completed_at,
                  PRIMARY KEY (material_id, stage))

jobs(job_key PK, stage, backend_name, handle, submitted_at)

job_materials(material_id PK, job_key)
```

Each carries an invariant in its shape rather than in code that must remember:

- **`stage_completions`'s composite primary key makes completion idempotent.**
  Replaying a completion is a no-op at the storage layer, which is what makes an
  interrupted run safe to simply re-run.
- **`job_materials.material_id` is the primary key**, so "one live job per
  material" is enforced by the schema. A second submission for the same material
  cannot be recorded, rather than being prevented by a check someone could
  forget.
- **`handle` is a JSON blob**, which is why `Handle.extra` must be JSON-safe
  ({doc}`value-objects`).
- **`claimed_at` is a lease, not a lock** — a timestamp, so an abandoned claim
  ages out instead of requiring a cleanup process.

Two indexes exist for named hot paths: claiming filters on
`(state, claimed_at)`, and the monitor's throughput window filters on
`completed_at`.

## Concurrency discipline

WAL mode, a `busy_timeout`, and **`BEGIN IMMEDIATE` on every read-then-write**.
The last one is what makes `claim` atomic: read-then-write under SQLite's default
deferred transaction is a race, and claiming is exactly that pattern. A test
drives two store instances against one file from two threads to prove a material
is never claimed twice.

The store is deliberately leaner than a multi-worker design: there is no
`claimed_by`, because a single-process tool's lease is just `claimed_at`, and no
artifacts table, because files live in the material's directory where you can
look at them.

## The workspace on disk

```text
materials-work/                  # run.work_dir
├── state.sqlite                 # everything above
├── Cl1Na1/                      # a fan-out parent
└── Cl1Na1_3D-6/                 # one material, one directory
    ├── structure.cif            # from structure
    ├── structure.json           # generation provenance
    ├── relaxed.cif              # from screen
    ├── screen_result.json       # the verdict
    └── record.json              # from extract
```

Job scratch lives elsewhere — under the backend's own `work_dir` — so a material
directory holds results rather than a copy of the tool. That separation was a
deliberate fix: runner scripts used to be staged into the material directory and
left symlinks among the science files.

The layout is meant to be readable with `ls` and `cat`. A researcher should be
able to answer "what happened to this material" without the tool, and a
workspace should be movable and archivable as an ordinary directory tree.

## Read-only observers

`StateStore(read_only=True)` sets SQLite's `query_only` pragma, so writes are
refused **by the engine**, not by our good intentions. A missing file is an
error rather than a silently created empty database — asking about a run that
does not exist should say so.

`status`, `results`, `monitor` and `cif` all open the store this way. Watching a
run cannot disturb it, and there is a test asserting the database file is
byte-identical after a read-only command.

```{image} /_static/diagrams/monitor-flow-light.svg
:alt: The monitor's data flow: a read-only state store, a CPU and GPU usage sampler, and a cached backend health probe feed the pure build_snapshot function, which returns a MonitorSnapshot value; the Rich renderers turn that one value into the dashboard panels. Nothing on the path writes to the run.
:class: only-light
:width: 100%
```

```{image} /_static/diagrams/monitor-flow-dark.svg
:alt: The monitor's data flow: a read-only state store, a CPU and GPU usage sampler, and a cached backend health probe feed the pure build_snapshot function, which returns a MonitorSnapshot value; the Rich renderers turn that one value into the dashboard panels. Nothing on the path writes to the run.
:class: only-dark
:width: 100%
```

The observer path is worth reading as a pattern for any future view. Sources —
the read-only store, the usage sampler, a cached health probe — feed
**`build_snapshot`**, a pure function with no knowledge of Rich, SQL or config,
which returns one `MonitorSnapshot`. Every panel is then a function of that
single value, which is why the whole dashboard is testable without a terminal.

Sampling rates differ on purpose: the store and local CPU every tick, a remote
at most every 10 s, backend health at most every 30 s. A dashboard must not
become load on the thing it is watching. Telemetry is best-effort throughout —
it can fail, time out, or not exist, and the result is a dash rather than a zero
and never an exception.

## Related

- {doc}`../reference/states` — the state machine stored in `materials.state`.
- {doc}`orchestrator` — the writer.
- {doc}`../api/state` and {doc}`../api/monitoring` — generated reference.
