Metadata-Version: 2.5
Name: stablemate-groom
Version: 1.3.0
Summary: Local push-driven dashboard for workhorse agent-workflow operator gates — tends the horses while they run.
Project-URL: Homepage, https://github.com/GabrielCpp/stablemate/tree/main/groom
Project-URL: Repository, https://github.com/GabrielCpp/stablemate
Project-URL: Issues, https://github.com/GabrielCpp/stablemate/issues
Author: Gabriel Côté
License-Expression: MIT
License-File: LICENSE
Keywords: agent,claude,dashboard,litestar,operator,preact,workflow
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Build Tools
Classifier: Topic :: Utilities
Requires-Python: >=3.12
Requires-Dist: jinja2>=3.1
Requires-Dist: litestar[standard]>=2.10
Requires-Dist: markdown-it-py>=3
Requires-Dist: opentelemetry-proto>=1.25
Requires-Dist: platformdirs>=4.0
Requires-Dist: pydantic>=2.0
Requires-Dist: watchfiles>=0.21
Requires-Dist: websockets>=15
Requires-Dist: workhorse-agent
Provides-Extra: test
Requires-Dist: playwright>=1.44; extra == 'test'
Requires-Dist: pytest>=8.0; extra == 'test'
Description-Content-Type: text/markdown

# groom

A local, single-process web dashboard for `workhorse` agent-workflow operator
gates. Run `groom serve` on your host while `author`/`coder` (or any other
`workhorse`-based workflow) runs in the background. `groom` shows every running
workflow, pages you the moment one blocks on an operator gate, and lets you answer
the gate from the browser. The shared `await_operator` node blocks in place rather
than exiting, so the run keeps running. The answer travels over the run's own
workhorse control socket and the run wakes at once. The gate file stays the durable
record, and the fallback channel when nothing is listening.

![groom's run list with a blocked coder run selected: the gate question rendered on the right, an answer typed, ready to send](../docs/features/groom/gui/screenshots/operator-answers-blocked-gate-answer-typed.png)

Beyond the gates, the same dashboard is the fleet's telemetry reader — every run's
spans, per-node timings and error status, filterable without leaving the browser:

![The telemetry view: one run's spans with started/node/duration/status columns, two of them flagged ERROR](../docs/features/groom/gui/screenshots/groom-dashboard-telemetry.png)

## How it works

- Each run pushes its telemetry to `groom` over OTLP. A run on groom's own host also
  advertises its run directory, workspace and pid. `groom` reads both from local disk
  to serve the run's Files and Diff panels and to answer its gates. A run with no
  `groom` listening behaves the same.
- `groom` holds live fleet and browser state in memory and pushes
  JSON state to open browser tabs over a websocket; the browser renders it with
  Preact + htm (the vendored `htm/preact` standalone build — no build step, no
  `node_modules`). Every shape the socket pushes is also fetchable over HTTP, so a
  tab whose socket has gone quiet resyncs instead of going stale.
  Those pushes are edge-triggered — something changed, so tell the
  tabs — which covers everything except the half of a run row that is derived from
  the clock: the liveness dot, `silent 4m`, `in node 12m`. The event that should
  turn a row dead is the run *ceasing* to emit, and an absence cannot be pushed, so
  the run list is additionally re-rendered and broadcast every `GROOM_LIVE_TICK_S`
  (5s), skipped entirely when no tab is connected. That is what makes the dashboard
  safe to leave open and read without refreshing.
  A selected run loads its span summary, recent logs, and metrics from SQLite
  once when its watch subscription opens. A loading indicator covers that initial
  read. The history lists timestamped log records, completed spans, and metric
  observations, including runs with no script logs. The type filter shows up to
  60 recent records per type; an explicit run selection includes older spans beyond
  the fleet’s activity window. Committed OTLP batches update the history in
  memory and push it immediately to the watching tabs; the clock tick only
  refreshes elapsed labels, with no database reads. Tabs share the watched history,
  which is released when the last tab leaves. Reopening the run loads a fresh
  snapshot, and reconnecting restores the subscription automatically.
- Fleet rows and the detail pane project the same received telemetry. Operator and
  machine wait gauges carry their gate path and question; node, activity, elapsed
  time and terminal state come from the run's exports. The freshness label shows
  how old that evidence is. Silence leaves the last reported state visible; groom
  does not infer completion by reading checkpoints, `run.json`, or the host PID.
  A resumed process exports a new session identity, replacing the prior session's
  live values as its telemetry arrives.
  The gate pane shows only the latest question set, in full, including nested
  headings. Questions render as Markdown (`marked`, sanitized with `DOMPurify`
  before insertion since the content is LLM-authored). The automatic attendant
  receives that same latest question set. Earlier exchanges stay in the gate file;
  the pane has no additional context disclosure. Each workflow row can expand a
  `git diff` of its working tree (rendered
  with `diff2html`). All front-end assets are vendored locally; nothing is loaded
  from a CDN at runtime.
- Gates travel over the run's control socket, with the file as the record. An
  answer goes socket-first — `groom` asks the run what it is waiting on and
  delivers the answer on the run's own path spelling; the run persists it into
  the gate file before acknowledging — and falls back to writing the gate file
  from outside only when nothing answers on the control socket.
  Wait telemetry updates native gate discovery, including after groom reconnects.
  Socket questions are queried when submitting an answer so the write uses the
  waiting process's exact path. Machine waits show their pending result without
  an operator answer form.

## Install

groom will publish to PyPI as `stablemate-groom`, because the name `groom` belongs to an
unrelated project. Until that release, install it from a checkout of the
[stablemate](https://github.com/GabrielCpp/stablemate) workspace:

```bash
pipx install ./groom        # isolated CLI on your PATH
```

Requires Python ≥ 3.12. groom is an optional add-on — no base workflow requires it — and
nothing has to be configured on the workflow side: `workhorse` probes for a collector at
the default port when a run starts.

## Usage

```bash
groom serve                       # binds 127.0.0.1:8787 — loopback only, the default
groom serve --host 0.0.0.0       # all interfaces: for runs on another machine (see note)
```

From a checkout of this workspace, prefix each command with `uv run` (`uv run groom serve`)
to use the working tree instead of the installed copy.

> **Binding.** groom binds loopback by default because it has **no
> authentication**, and it answers operator gates. A run on another machine needs
> `--host 0.0.0.0`. That choice prints a one-line exposure warning, and
> `--allow-non-loopback` acknowledges it. Only do that on a trusted network.

## Telemetry collector (OTLP) + AFK alerting

`groom` is also the default local **OpenTelemetry collector** for `workhorse`
runs. The same uvicorn process
and port expose standard OTLP/HTTP receivers — `POST /v1/traces`,
`POST /v1/metrics` and `POST /v1/logs` — so an ordinary run

```bash
workhorse-coder run                # probes the endpoint at start; exports if groom answers
```

(No env var is needed while `groom serve` is up on the default port — workhorse
enables telemetry when it finds a collector listening. Point elsewhere with
`OTEL_EXPORTER_OTLP_ENDPOINT`, or opt out with `WORKHORSE_OTEL=0`.)

It streams node/agent-turn spans, gas/heartbeat metrics, and the log records of the
engine and every node it runs in-process into `groom`. Because a
pushed span carries its own identity, **every run appears with no discovery
step**. Spans and metrics persist in an embedded SQLite file
(`groom.db` in the platform data dir; override with `GROOM_DB`), searchable
from the dashboard's *Telemetry* pane, via `GET /traces?run=…&node=…&status=…&
slower_than=…`, or with raw `sqlite3` queries. Rows older than
`GROOM_RETENTION_DAYS` (30) are pruned at startup and on a periodic tick — but
only for a run whose telemetry is already **archived** to disk (below), so the
window bounds the *database*, not the evidence.

A batch the store cannot take is answered `503` with a `Retry-After`, not `500`: the
exporter re-sends it, where an unhandled error would have said "stored" by omission.
That is a deliberate trade of duplicates for losses — `spans` and `turns` are keyed and
absorb a re-send, `metrics` and `logs` are plain appends and a retried batch can land
twice. Duplicate rows are cosmetic in tables read by recency; a dropped batch is gone.
The connection behind it recycles itself on a SQLite error and retries once, so a
wedged handle heals on the next request instead of at the next restart. Whether any of
that has happened is in `GET /api/state` under `store` — `reopens`, `failures`,
`last_error`, and `ok`, which is false only while a failure is *newer* than the last
statement that worked.

The liveness counters — `workhorse.run.heartbeat`, `workhorse.turn.heartbeat`,
`workhorse.cap_wait.heartbeat` — are **never stored at all**: `insert_metrics`
drops them at the door. They tick every ~10s for every open node, so
on long runs they outgrow everything else combined: in one real store the run
heartbeat alone was 1.77M of 2.21M metric rows, and a later one hit 2.8 GB with
heartbeats ~80% of 18M rows. Nothing reads their history — the alert rules fold
them into memory at ingest, and `groom status` asks the running server for that
in-memory picture over HTTP (`/api/live`) — so persistence bought nothing and
cost most of the file. Rows written before this change drain on age alone: the
never-archived series are swept by the ordinary prune without waiting for an
archive behind them, because there is none. The seven liveness *gauges*
(`turn.active`, `turn.idle_s`, `turn.elapsed_s`, `wait.active`, `wait.elapsed_s`,
`node.active`, `node.elapsed_s`) are stored and read live but not archived, on the
same reasoning: climbing idle on an active turn diagnoses a wedged agent *now*,
while an explicit wait is expected parked work, and neither question is asked of a
run that ended a month ago. `models.UNARCHIVED_DELETABLE_METRICS` names them, in
one place, because what may be deleted without a copy is a rule that has to be
readable rather than inferred.

### Archival: SQLite is the live tier, disk is the permanent one

A run's spans and logs are not lost when they age out. Once a run has been quiet
longer than `GROOM_RETENTION_DAYS`, an archival pass writes everything it holds —
every span, every log, and the four budget metric series — as one
`telemetry.jsonl` beside that run's transcripts, and only then does prune delete
the rows. Write, move, delete, in that order: prune is fail-closed and refuses to
touch a run the archive has not reported back.

```
<data dir>/transcripts/<run-id>/…      live runs, the harvester's only target
<data dir>/archives/<run-id>/          frozen: transcripts + telemetry.jsonl
```

Two roots on one filesystem, so promotion is a single `os.rename`. **An archived
run is frozen** — never re-archived, never resumed, never written again. A second
run claiming a name already under `archives/` is archived under a derived name
(`R-a1b2c3d4`) rather than overwriting it.

The file is one JSON object per line, ordered by `ts`, discriminated by `kind`
(`span` / `log` / `metric`), with line 1 a manifest naming the run and its row
counts. Records carry `run_id`, `node`, `generation` and `seq`, which is exactly
the key the per-node transcripts are classified by — so the telemetry joins the
transcripts with no correlation step. The heartbeats and the liveness gauges never
reach it; they are for the in-memory picture, not for history.

The archive has **no retention of its own** and **no index table**: the set of
archived runs is what is on disk under `archives/`, read by listing it. A sweep
interrupted halfway resumes from the manifest rather than rewriting the file.

```bash
groom archive status         # root, frozen runs, pending, held, last sweep
groom archive ls             # what is frozen (--long for the manifests)
groom archive show RUN       # stream one run's telemetry.jsonl
groom archive now --dry-run  # force a pass; --dry-run writes nothing
```

The pass runs in its own task at `groom serve` launch, seeded already-expired so
a restart archives immediately, then every `GROOM_ARCHIVE_EVERY_S` (6h) for at
most `GROOM_ARCHIVE_RUNS_PER_PASS` (25) runs — oldest first, the rest reported as
pending. Nothing in it may raise into groom's tick.

**The pane shows the runs connected right now.** A month of retention means the
unfiltered strip is mostly runs that ended days ago, and a dashboard is for
watching, so a run card (and its spans — a span table listing a hidden run's nodes
is telemetry from nowhere) appears only while the run is `live` by the same
heartbeat predicate the fleet rows use. History is one tick of *show ended* away
(`GET /traces?show_ended=1`), and searching an explicit `run=` always finds it:
naming a run is asking for that run, finished or not.

**Native runs are first-class dashboard rows, not just telemetry.** A run on
groom's own host advertises its `run_dir`, workspace path, pid, and per-node
`activity` label on the OTLP resource; groom materializes a fleet row from that
(keyed by `run_id`), shows what it is doing ("coder · reviewing ACME-A2JX"), and —
because it shares the host — serves the row's Files/Diff panels and answers its
operator gates straight from the local filesystem (`groom.localfs`). The native test
is self-validating. groom draws the row only for a run whose `run_dir` it can read
locally, so a run whose paths don't resolve on this host never double-lists.

Alert rules run on every ingest plus a periodic tick, and page you through
browser notifications **and** an away-from-keyboard push — configure
`GROOM_NTFY_TOPIC` (posts to `https://ntfy.sh/<topic>`; override the server
with `GROOM_NTFY_URL`) and/or `GROOM_WEBHOOK_URL` (JSON `{"title","message"}`):

| Rule | Fires when | Knob (default) |
|---|---|---|
| STALL | a live run emits **nothing** — no span, no heartbeat | `GROOM_STALL_MIN` (90) |
| STUCK | a live agent turn is silent too long, or deterministic work has sat in one node too long; explicit waits are excluded | `GROOM_STUCK_MIN` (75) |
| CHURN | the same node span completes again and again under an unchanged `labels()` signature — i.e. on the same unit of work. A visit a live reload cut short (`workhorse.cut`) is an interruption, not a repeat, and does not count | `GROOM_CHURN_REPEATS` (5) |
| WATCHDOG | a `watchdog_kill` span event arrives | — |
| GAVE-UP | a give-up node's span arrives | `GROOM_GIVEUP_NODES` (qa_give_up,fix_give_up) |
| ENDED | the run's root span arrives — the run is over, whatever the verdict, and nothing is executing for it now | — |
| BLOCKED | an operator gate opens — the run is parked until someone answers it. A cap wait does not count | — |
| WAITING | that gate is still unanswered later | `GROOM_WAIT_MIN` (30) |

Groom deliberately does not alert on total run age: resumptions reuse a run identity and multi-day
runs are normal. Use workhorse's `WORKHORSE_MAX_RUNTIME_S` when a run needs a hard wall-clock limit.

The crux: workhorse **heartbeats** for as long as its process lives, so silence
and slowness are different observations. STALL means the run stopped emitting
(dead/killed/frozen); STUCK means it is alive and parked. Alerts dedupe per
`(run, rule)`.

ENDED covers the case neither absence rule can, because a terminal is precisely
what retires a run from both of them: the run *finished*. That is the loudest
event on a queue — nothing is executing until someone launches the next run — and
until it existed it was the only one that paged nobody. It fires on every ending,
naming the terminal and the error class, because from outside "it crashed" and
"it succeeded" are the same silence. A resume reuses the run id and clears the
fired set, so the next session's ending pages on its own.

A process killed before it exports its terminal remains in its last reported state
with an aging freshness label. STALL reports the missing telemetry; completion
requires the run's terminal export.

BLOCKED and WAITING cover the mirror image: a run parked on an operator gate is
behaving correctly, so no rule described it, and STUCK skips an open wait by
design. Yet it is the only alert whose subject can end it — the human reading the
page *is* the missing input. Answering the gate closes the wait and retires both.
Run `groom serve` under nohup/systemd as the always-on collector
(the consuming repo's `make groom-serve` target does exactly that).

## Where is my run right now?

Spans export **on completion**, so a run's current node — the only one that
matters when it will not finish — has no row in `spans` and never will while it
hangs. The live picture therefore comes from metrics, which ship on a timer
regardless of span state:

```bash
uv run groom status              # every live run: open node, node age, agent idleness
uv run groom status --run <id>   # one run
uv run groom status --json       # same data, machine-readable
```

Read it like this:

| Reading | Means |
|---|---|
| `alive`, node age small | working normally |
| `alive`, explicit wait | operator/cap/retry wait — parked deliberately |
| `alive`, active turn age large, agent `idle` small | a long but **streaming** turn — healthy |
| `alive`, active turn, agent `idle` large | **wedged** agent/tool/API inside the node |
| `alive`, no turn or wait, node age large | **wedged deterministic work** inside the node |
| no heartbeat (`DEAD?`) | the process is gone — SIGKILL, OOM, crashed host |

## Wait until a run needs attention

```bash
uv run --all-packages groom wait --run RUN --json
uv run --all-packages groom wait --run RUN --until blocked,ended
```

`wait` holds one WebSocket connection to `groom serve` and stays silent during
normal progress. It returns one event and exits when the named run is blocked,
ended, stuck, stalled, or gave up. `--until` selects a comma-separated set from
`blocked,ended,stuck,stalled,gave-up,churn,watchdog,died,waiting`.
There is no timeout; Ctrl+C cancels the monitor without stopping the workflow.

Current conditions are checked in the initial server snapshot, including runs
hidden from the dashboard. A run not yet known to the server is watched until
it reports a matching condition. This uses the server's current cache, not an
archive search: an ending evicted from that cache is no longer available.
Machine, capacity and retry waits do not count as operator gates. Starting a
new waiter while the same condition remains active returns that condition again.

`--run` takes the exact telemetry run ID. `GROOM_URL` selects the collector, as for `groom status`.
Both CLI and server must support the wait protocol; an older server is reported
as a monitoring error. JSON output contains `run_id`, `event`, `node`, `question`,
`gate_path`, `terminal`, and `message`; unavailable context is an empty string.
Exit `0` means a matching event, even if the workflow failed; `1` means the
connection or protocol failed, `2` means invalid arguments, and `130` means
cancellation. Disconnections are reported instead of silently reconnecting
across a gap in monitoring.

## What was it saying? (`groom logs`)

`status` says *where* a run is; logs say what it was doing on the way there.

```bash
uv run groom logs --run <id>                 # everything, oldest-first
uv run groom logs --run <id> --node select_item
uv run groom logs --level WARNING            # a FLOOR: WARNING + ERROR + FATAL
uv run groom logs --contains "over budget"
```

Deterministic (non-agent) nodes appear here because workhorse **runs them in-process**
and hands each one a per-node `logger`, preserving the node's decisions alongside its
spans and artifacts.

Records carry the same `run_id`/`run_dir` resource as the spans, so a log line
joins to its node span and its on-disk artifacts with no correlation step. They
are correlated by an explicit `node` attribute rather than `trace_id`, which is
zeroes: workhorse never makes its node spans current, so there is no ambient
context for the SDK to attach.

No alert rule fires on logs, deliberately — liveness is already answered by the
heartbeat metrics, and paging on log *content* would mean guessing which strings
are worth waking someone for, per workflow. Logs are for reading once a metric has
told you where to look. They expire on the same window as everything else and
are archived with it — one signal, one clock, one file.

### Test runs are not telemetry

A test suite is not a run anyone comes back to, and on a machine where
`groom serve` is up it is by far the loudest producer: one `make test` of the
workflows suite wrote a six-figure number of spans, burying every real run in
the fleet view. So test telemetry is kept out at both ends — workhorse declines
to auto-enable export from a test process (`WORKHORSE_OTEL=1` still forces it
on), and the receivers drop any record whose `run_dir` is *certainly* a test
dir — one containing `pytest-of-` or `.workhorse-test/`. Neither undoes what an
older producer already wrote:

```bash
uv run groom purge-tests --dry-run   # what would go
uv run groom purge-tests             # delete it, then VACUUM to shrink the file
```

`purge-tests` casts the wider net: it also evicts runs whose dir is a
`tempfile.mkdtemp` scratch dir (`/tmp/tmpXXXXXXXX/…`), which is how the largest
single junk run in a real store got there. That one is a guess — a genuine run
launched from a mkdtemp directory matches too — so it is confined to a command
you run deliberately and can preview with `--dry-run`, rather than to the ingest
path, where the same guess would discard evidence unasked. Runs are identified
by their run dir in both cases, never by name.

## Reading the store afterwards

The same file the dashboard reads answers the after-the-fact questions — what a run cost
and where it looped, what it would have cost on another model set, what a node actually
said, and where the wall clock went:

```bash
uv run groom cost --run <id>          # money and rework per node, with /work turns-per-item
uv run groom prices --run <id>        # re-price the same turns against another model set
uv run groom loops --run <id>         # which review→rework loops converge, and which do not
uv run groom transcript --run <id>    # the reasoning and tool trace prompt.md omits
uv run groom profile --run <id>       # what occupied the wall clock
uv run groom db-path                  # …and the SQLite file, for anything the CLI does not ask
```

Cost coverage is not uniform — it is harness × *provider*, and the two ways of not pricing
a turn (nothing reported, versus a reported `0`) differ in how visible they are. Every
command above says how much of itself is real rather than presenting a partial sum as a
complete one.

Those commands, the schema and its one footgun, and worked SQL are in
[docs/STORE.md](docs/STORE.md).

See `docs/features/groom/` at the repo root for the full design — groom's own OKF book,
queryable with `ostler graph --surface groom`.

## Map

- `groom/`: the package behind the `groom` command: the server, the collector, the store and the dashboard assets.
