Watch a run live#
$ matmelt monitor
A read-only dashboard that refreshes in place. It never writes: the store is
opened with SQLite’s query_only pragma, so watching a run cannot disturb it,
and the same command works on a finished workspace. Ctrl-C leaves with exit 0.
$ matmelt monitor --interval 5 # refresh every 5 seconds
$ matmelt monitor --once # one frame, then exit (scripts, logs)
$ matmelt monitor --max-ticks 20 # a bounded watch
The overview#
╭─ Run overview ──────────────────────────────────────────────────────────────╮
│ Run nacl-demo │
│ Elapsed 25m 26s │
│ Total materials 39 ██░░░ │
│ Done 4 (10.3%) ░ ░ │
│ In flight 0 ░ 10% ░ │
│ Failed 3 ░ ░ │
│ Throughput — ░░░░░ │
│ ETA — │
│ Stop stage (full funnel) │
│ │
│ ███████████████████████████████████████ │
│ █ STRUCTURED 28 (72%) █ SCREENED 2 (5%) █ DONE 4 (10%) █ EXPANDED 2 (5%) │
╰─────────────────────────────────────────────────────────────────────────────╯
The bar is the whole population by state, so you can see at a glance where
materials are piling up. EXPANDED is not a failure — it is a fan-out parent
that finished its job.
The stage pipeline#
┃Stage ┃ Backend ┃ Ready ┃ Running ┃ Done ┃ Failed ┃ Speed ┃
│compose │ laptop (local) │ 0 │ 0 │ 38 │ 0 │ — │
│structure │ laptop (local) │ 0 │ 0 │ 37 │ 0 │ — │
│screen │ laptop (local) │ 26 │ 2 │ 6 │ 3 │ — │
│extract │ laptop (local) │ 2 │ 0 │ 4 │ 0 │ — │
This is the stall diagnostic. A large Ready with zero Running, holding
still, is never a science problem — it is a backend that is down, a dependency
that is missing, or a stage paused by an unhealthy health check. Go to
Check that a machine can do the work. A small Ready with steady Running is simply a slow stage.
Speed is completions per minute once there are enough of them to mean
anything.
Running now#
╭─ Running now (2) ───────────────────────────────────────────────────────────╮
┃Material ┃ Stage ┃ Backend ┃ Elapsed ┃ Try ┃ CPU ┃ GPU ┃
│Cl1Na1_3D-83 │ screen │ laptop │ 7s │ #1 │ — │ — │
│Cl1Na1_3D-89 │ screen │ laptop │ 7s │ #1 │ — │ — │
╰─────────────────────────────────────────────────────────────────────────────╯
Try #2 or higher means this material has already faulted once — useful early
warning that something is wrong before it exhausts its budget.
A dash means unknown, never zero. In the frame above both jobs had just been sighted for the first time, and a first sighting has no earlier sample to measure against; later frames show percentages. A machine with no GPU shows dashes in that column forever, and a remote is sampled at most every ten seconds. CPU is a process-subtree figure, so a job using three cores honestly reads above 100%.
Telemetry is best-effort by construction: it can fail or time out, and when it does you get a dash. It can never take the dashboard down or slow the run.
Completions and failures#
╭─ Recent completions ────────────────────────────────────────────────────────╮
│✓ Cl1Na1_3D-25 │ 5m 52s │ 00:53:35 │
│✓ Cl1Na1_3D-16 │ 5m 52s │ 00:53:35 │
╰─────────────────────────────────────────────────────────────────────────────╯
╭─ Recent failures ───────────────────────────────────────────────────────────╮
│Cl1Na1_3D-3 │ screen │ screened out: energy -578480.6344 eV/atom outsid… │
│Cl1Na1_3D-2 │ screen │ stage 'screen' cannot run on item 'Cl1Na1_3D-2': … │
╰─────────────────────────────────────────────────────────────────────────────╯
Reasons are truncated to fit. For the full text — and the attempt count that
tells a verdict from a fault — use matmelt status
(Diagnose a failed material).
Follow named materials#
Swap the failures panel for a detail table on materials you care about:
$ matmelt monitor --track Cl1Na1_3D-83,Cl1Na1_3D-1
$ matmelt monitor --track-file picked.txt
╭─ Tracked materials ─────────────────────────────────────────────────────────╮
┃Material ┃ State ┃ Stage ┃ Backend ┃ In state ┃ Info ┃
│Cl1Na1_3D-83 │ STRUCTURED │ screen │ laptop │ 9s │ laptop:33ba286…│
│Cl1Na1_3D-1 │ FAILED │ — │ — │ 24m 35s │ stage 'screen' │
│Cl1Na1_3D-999 │ not found │ — │ — │ — │ — │
╰─────────────────────────────────────────────────────────────────────────────╯
An id that does not exist renders as not found rather than being dropped,
so a typo in your list is visible instead of silently ignored. Rows keep the
order you gave them.