---
description: Watch a run live with the monitor dashboard, and track specific materials by id.
---

# T4 — Watching a run with `monitor`

**Goal:** read every panel of the live dashboard, and follow named materials
through a run in progress.

**Prerequisites:** a run you can start (T1). You will want two terminals: one
running the funnel, one watching it.

**Time:** about 10 minutes.

Every frame on this page is real, captured from the run it describes.

## 1. Start a run, then watch it

In the first terminal:

```console
$ matmelt run --yes
```

In the second, from the same directory:

```console
$ matmelt monitor
```

The dashboard redraws on an interval (default 2 seconds) until you press
Ctrl-C, which exits cleanly with code 0. To take a single frame instead — handy
in a script, or over SSH — use `--once`:

```console
$ matmelt monitor --once
```

:::{important}
`monitor` opens the state database **read-only**. It cannot start, stop or alter
a run, and watching cannot disturb what you are watching. It is equally happy
pointed at a finished workspace or one where nothing is running at all.
:::

## 2. The overview panel

```text
╭─ Run overview ───────────────────────────────────────────────────────────────╮
│ Run              monitor-demo                                                │
│ Elapsed          21s                                ░░░░░                    │
│ Total materials  39                               ░       ░                  │
│ Done             0 (0.0%)                         ░  0%   ░                  │
│ In flight        2                                ░       ░                  │
│ Failed           0                                  ░░░░░                    │
│ Throughput       —                                                           │
│ ETA              —                                                           │
│ Stop stage       extract                                                     │
│                                                                              │
│ ██████████████████████████████████████████████                               │
│ █ STRUCTURED 37 (95%)   █ EXPANDED 2 (5%)                                    │
╰──────────────────────────────────────────────────────────────────────────────╯
```

The ring is completion; the bar underneath is where every material currently
sits, proportionally. Early in a run it is one colour — everything waiting at
`STRUCTURED` — and it drains left to right as the funnel advances.

`Throughput` and `ETA` show `—` until there is enough history to mean anything.
A dash always means *unknown* on this dashboard, never zero.

## 3. The stage pipeline panel

```text
┃Stage      ┃ Backend          ┃  Ready ┃  Running ┃  Done ┃ Failed ┃   Speed ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━┩
│compose    │ laptop (local)   │      0 │        0 │    38 │      0 │ 7.6/min │
│structure  │ laptop (local)   │      0 │        0 │    37 │      0 │ 7.4/min │
│screen     │ laptop (local)   │     35 │        2 │     0 │      0 │       — │
│extract    │ laptop (local)   │      0 │        0 │     0 │      0 │       — │
```

This is the funnel with numbers on it, and it is where you diagnose a stalled
run. **Ready** is the backlog waiting for that stage; **Running** is what is in
flight; **Backend** names the machine each stage is wired to, which matters most
when some stages are remote.

The shape above is the normal one: compose and structure have finished and
handed 35 candidates to `screen`, which is working through them two at a time —
`max_parallel: 2` on that backend.

If `Ready` is large while `Running` stays at 0, something is blocking: an
unhealthy backend, or a stage whose dependencies are missing. Run `doctor`.

## 4. Running now

```text
┃Material                   ┃ Stage       ┃ Backend      ┃ Elapsed ┃ Try ┃ CPU ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━╇━━━━━┩
│Cl1Na1_3D-1                │ screen      │ laptop       │     12s │  #1 │ —   │
│Cl1Na1_3D-2                │ screen      │ laptop       │     12s │  #1 │ —   │
```

One row per job in flight, with how long it has been running and which attempt
this is. `Try #2` or higher means the material previously hit a fault and is
being retried — worth noticing early.

**CPU and GPU are best-effort.** The sampler measures the job's process subtree,
and the first sighting of a process has nothing to compare against, so a dash is
normal on the tick a job appears; readings follow. A machine with no GPU shows
dashes forever in that column, which is honest — it is not zero usage, it is no
measurement. Sampling can never slow down or crash the dashboard: local
processes are read every tick, a remote backend is asked at most every 10
seconds, and backend health at most every 30.

## 5. Completions and failures

As the run progresses, two panels fill in:

```text
╭─ Recent completions ─────────────────────────────────────────────────────────╮
│ ┃Material                   ┃      Took ┃              Finished at┃          │
│ │✓ Cl1Na1_3D-221            │    6m 37s │                 14:28:23│          │
│ │✓ Cl1Na1_3D-215            │    6m 36s │                 14:28:22│          │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Recent failures ────────────────────────────────────────────────────────────╮
│ ┃Material         ┃ Stage  ┃ Reason                            ┃ Failed at┃   │
│ │Cl1Na1_3D-177    │ screen │ screened out: volume changed by … │ 14:27:24 │   │
│ │Cl1Na1_3D-164    │ screen │ screened out: residual stress 7.… │ 14:26:48 │   │
╰──────────────────────────────────────────────────────────────────────────────╯
```

`Took` is measured from when the material entered the run, not from when its
last job started — which is why these read in minutes while an individual
relaxation takes seconds.

The failures panel is the one to watch. A stream of `screened out:` reasons is
the funnel working. A stream of anything else — missing files, unreachable
hosts — means something is wrong with the setup rather than the science.

## 6. Tracking specific materials

When you care about particular candidates, name them:

```console
$ matmelt monitor --track Cl1Na1_3D-221,Cl1Na1_3D-215
```

The failures panel is replaced by a tracked-materials table showing each id in
the order you listed them:

```text
┃Material         ┃ State     ┃ Stage ┃ Backend ┃ In state ┃ Info              ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩
│Cl1Na1_3D-221    │ DONE      │ —     │ —       │   5m 25s │ —                 │
│Cl1Na1_3D-6      │ FAILED    │ —     │ —       │  11m 36s │ screened out: vo… │
│Cl1Na1_3D-999    │ not found │ —     │ —       │        — │ —                 │
```

A material that failed carries its reason in `Info`, and an id that does not
exist is reported as **`not found`** rather than silently dropped — so a typo is
visible instead of looking like a material that simply is not doing anything.
For a longer list, keep the ids in a file:

```console
$ matmelt monitor --track-file candidates.txt
```

## 7. Useful flags

```console
$ matmelt monitor --interval 5     # redraw every 5 seconds
$ matmelt monitor --once           # one frame, then exit
$ matmelt monitor --max-ticks 20   # redraw 20 times, then exit
```

`--interval` is worth raising when watching a run over SSH, or when the run is
long and you want less redraw traffic. Nothing about the interval changes the
run itself; it only changes how often the dashboard reads.

## What you learned

- `monitor` is strictly an observer: read-only by construction, safe on a live
  run, and usable on a finished workspace.
- The pipeline panel is the stall diagnostic — a large `Ready` with zero
  `Running` points at a backend or a dependency, not at the science.
- A dash means *unknown*, never zero: unmeasured CPU, no GPU, or not enough
  history for a rate.
- `Try #N` exposes retries as they happen.
- `--track` follows named materials; unknown ids are reported, not ignored.

Next: {doc}`T5 <t5-start-mid-funnel>` picks up an existing workspace. The
panels are summarised as a recipe in {doc}`../how-to/read-the-monitor`.
