T4 — Watching a run with monitor#

Goal: read every panel of the live dashboard, and follow named materials through a run in progress.

Prerequisites: a run you can start (T1). You will want two terminals: one running the funnel, one watching it.

Time: about 10 minutes.

Every frame on this page is real, captured from the run it describes.

1. Start a run, then watch it#

In the first terminal:

$ matmelt run --yes

In the second, from the same directory:

$ matmelt monitor

The dashboard redraws on an interval (default 2 seconds) until you press Ctrl-C, which exits cleanly with code 0. To take a single frame instead — handy in a script, or over SSH — use --once:

$ matmelt monitor --once

Important

monitor opens the state database read-only. It cannot start, stop or alter a run, and watching cannot disturb what you are watching. It is equally happy pointed at a finished workspace or one where nothing is running at all.

2. The overview panel#

╭─ Run overview ───────────────────────────────────────────────────────────────╮
│ Run              monitor-demo                                                │
│ Elapsed          21s                                ░░░░░                    │
│ Total materials  39                               ░       ░                  │
│ Done             0 (0.0%)                         ░  0%   ░                  │
│ In flight        2                                ░       ░                  │
│ Failed           0                                  ░░░░░                    │
│ Throughput       —                                                           │
│ ETA              —                                                           │
│ Stop stage       extract                                                     │
│                                                                              │
│ ██████████████████████████████████████████████                               │
│ █ STRUCTURED 37 (95%)   █ EXPANDED 2 (5%)                                    │
╰──────────────────────────────────────────────────────────────────────────────╯

The ring is completion; the bar underneath is where every material currently sits, proportionally. Early in a run it is one colour — everything waiting at STRUCTURED — and it drains left to right as the funnel advances.

Throughput and ETA show — until there is enough history to mean anything. A dash always means unknown on this dashboard, never zero.

3. The stage pipeline panel#

┃Stage      ┃ Backend          ┃  Ready ┃  Running ┃  Done ┃ Failed ┃   Speed ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━┩
│compose    │ laptop (local)   │      0 │        0 │    38 │      0 │ 7.6/min │
│structure  │ laptop (local)   │      0 │        0 │    37 │      0 │ 7.4/min │
│screen     │ laptop (local)   │     35 │        2 │     0 │      0 │       — │
│extract    │ laptop (local)   │      0 │        0 │     0 │      0 │       — │

This is the funnel with numbers on it, and it is where you diagnose a stalled run. Ready is the backlog waiting for that stage; Running is what is in flight; Backend names the machine each stage is wired to, which matters most when some stages are remote.

The shape above is the normal one: compose and structure have finished and handed 35 candidates to screen, which is working through them two at a time — max_parallel: 2 on that backend.

If Ready is large while Running stays at 0, something is blocking: an unhealthy backend, or a stage whose dependencies are missing. Run doctor.

4. Running now#

┃Material                   ┃ Stage       ┃ Backend      ┃ Elapsed ┃ Try ┃ CPU ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━╇━━━━━┩
│Cl1Na1_3D-1                │ screen      │ laptop       │     12s │  #1 │ —   │
│Cl1Na1_3D-2                │ screen      │ laptop       │     12s │  #1 │ —   │

One row per job in flight, with how long it has been running and which attempt this is. Try #2 or higher means the material previously hit a fault and is being retried — worth noticing early.

CPU and GPU are best-effort. The sampler measures the job’s process subtree, and the first sighting of a process has nothing to compare against, so a dash is normal on the tick a job appears; readings follow. A machine with no GPU shows dashes forever in that column, which is honest — it is not zero usage, it is no measurement. Sampling can never slow down or crash the dashboard: local processes are read every tick, a remote backend is asked at most every 10 seconds, and backend health at most every 30.

5. Completions and failures#

As the run progresses, two panels fill in:

╭─ Recent completions ─────────────────────────────────────────────────────────╮
│ ┃Material                   ┃      Took ┃              Finished at┃          │
│ │✓ Cl1Na1_3D-221            │    6m 37s │                 14:28:23│          │
│ │✓ Cl1Na1_3D-215            │    6m 36s │                 14:28:22│          │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Recent failures ────────────────────────────────────────────────────────────╮
│ ┃Material         ┃ Stage  ┃ Reason                            ┃ Failed at┃   │
│ │Cl1Na1_3D-177    │ screen │ screened out: volume changed by … │ 14:27:24 │   │
│ │Cl1Na1_3D-164    │ screen │ screened out: residual stress 7.… │ 14:26:48 │   │
╰──────────────────────────────────────────────────────────────────────────────╯

Took is measured from when the material entered the run, not from when its last job started — which is why these read in minutes while an individual relaxation takes seconds.

The failures panel is the one to watch. A stream of screened out: reasons is the funnel working. A stream of anything else — missing files, unreachable hosts — means something is wrong with the setup rather than the science.

6. Tracking specific materials#

When you care about particular candidates, name them:

$ matmelt monitor --track Cl1Na1_3D-221,Cl1Na1_3D-215

The failures panel is replaced by a tracked-materials table showing each id in the order you listed them:

┃Material         ┃ State     ┃ Stage ┃ Backend ┃ In state ┃ Info              ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩
│Cl1Na1_3D-221    │ DONE      │ —     │ —       │   5m 25s │ —                 │
│Cl1Na1_3D-6      │ FAILED    │ —     │ —       │  11m 36s │ screened out: vo… │
│Cl1Na1_3D-999    │ not found │ —     │ —       │        — │ —                 │

A material that failed carries its reason in Info, and an id that does not exist is reported as not found rather than silently dropped — so a typo is visible instead of looking like a material that simply is not doing anything. For a longer list, keep the ids in a file:

$ matmelt monitor --track-file candidates.txt

7. Useful flags#

$ matmelt monitor --interval 5     # redraw every 5 seconds
$ matmelt monitor --once           # one frame, then exit
$ matmelt monitor --max-ticks 20   # redraw 20 times, then exit

--interval is worth raising when watching a run over SSH, or when the run is long and you want less redraw traffic. Nothing about the interval changes the run itself; it only changes how often the dashboard reads.

What you learned#

  • monitor is strictly an observer: read-only by construction, safe on a live run, and usable on a finished workspace.

  • The pipeline panel is the stall diagnostic — a large Ready with zero Running points at a backend or a dependency, not at the science.

  • A dash means unknown, never zero: unmeasured CPU, no GPU, or not enough history for a rate.

  • Try #N exposes retries as they happen.

  • --track follows named materials; unknown ids are reported, not ignored.

Next: T5 picks up an existing workspace. The panels are summarised as a recipe in Watch a run live.