# vllmstat v0.6.0 — auto-discover + new metrics — Implementation Plan

> REQUIRED SUB-SKILL: superpowers:subagent-driven-development. TDD, checkbox steps.

**Goal:** (A) auto-discover Docker vLLM containers when the default endpoint is down, and add four
metric views confirmed present on the live V1 server: (B) latency phase breakdown, (C) request-shape
token-length distributions, (D) finish-reasons + goodput, (E) energy efficiency (tok/W).

**Conventions:** Python ≥3.10, `from __future__ import annotations`, ruff+pyright clean, pytest
(`source .venv/bin/activate`; aiohttp installed). Commit per task. The metrics engine already has
`_quantiles(fam, base)` (windowed histogram p50/p90/p99), `get_buckets`/`hist_sum`/`hist_count`
(parse), and `histogram_quantile`/`windowed_buckets` (histogram).

Confirmed metric families on the server: `vllm:request_prefill_time_seconds`,
`vllm:request_decode_time_seconds`, `vllm:request_prompt_tokens`, `vllm:request_generation_tokens`,
`vllm:request_success_total{finished_reason=…}`, `vllm:time_to_first_token_seconds`,
`vllm:request_time_per_output_token_seconds`.

---

## Task 1: Snapshot fields + `histogram_fraction_below`

**Files:** Modify `src/vllmstat/core/state.py`, `src/vllmstat/core/histogram.py`; Test
`tests/test_state.py`, `tests/test_histogram.py`.

- [ ] `state.py` — add to `Snapshot` (after the existing `queue` latency field and near related groups):
```python
    # latency phases (seconds)
    prefill: Quantiles = field(default_factory=Quantiles)
    decode: Quantiles = field(default_factory=Quantiles)
    # request shape (token counts; .mean = average)
    prompt_len: Quantiles = field(default_factory=Quantiles)
    gen_len: Quantiles = field(default_factory=Quantiles)
    # outcomes
    finish_reasons: dict[str, float] = field(default_factory=dict)  # windowed fraction by reason
    goodput_ttft: float | None = None  # fraction of requests with TTFT <= ttft_slo_s
    goodput_tpot: float | None = None  # fraction with TPOT <= tpot_slo_s
    ttft_slo_s: float = 1.0
    tpot_slo_s: float = 0.05
```
- [ ] `histogram.py` — add:
```python
def histogram_fraction_below(buckets: list[tuple[float, float]], threshold: float) -> float | None:
    """Fraction of observations with value <= threshold, interpolated within the bucket.

    `buckets` are cumulative [(le, cum_count), ...] sorted ascending (Prometheus style).
    Returns None when there are no observations.
    """
    if not buckets:
        return None
    total = buckets[-1][1]
    if total <= 0:
        return None
    prev_le, prev_count = 0.0, 0.0
    for le, count in buckets:
        if threshold < le:
            if le == float("inf"):
                return prev_count / total
            bucket_count = count - prev_count
            if bucket_count <= 0:
                return prev_count / total
            frac = (threshold - prev_le) / (le - prev_le)
            return min(1.0, (prev_count + frac * bucket_count) / total)
        prev_le, prev_count = le, count
    return 1.0
```
- [ ] Tests: `test_state.py` — a `Snapshot(ts=0, connected=True)` has `prefill`/`decode`/`prompt_len`/
  `gen_len` as empty `Quantiles`, `finish_reasons == {}`, `goodput_ttft is None`, `ttft_slo_s == 1.0`.
  `test_histogram.py`:
```python
def test_histogram_fraction_below():
    from vllmstat.core.histogram import histogram_fraction_below
    b = [(0.1, 10.0), (0.5, 50.0), (1.0, 90.0), (float("inf"), 100.0)]
    assert histogram_fraction_below(b, 0.5) == 0.5            # exactly at a boundary
    assert abs(histogram_fraction_below(b, 0.3) - 0.40) < 1e-9  # interpolated within (0.1,0.5]
    assert histogram_fraction_below(b, 5.0) == 0.9             # beyond last finite le
    assert histogram_fraction_below([], 1.0) is None
    assert histogram_fraction_below([(1.0, 0.0)], 1.0) is None
```
- [ ] Commit: `feat(metrics): Snapshot fields for phases/shape/outcomes + histogram_fraction_below`.

---

## Task 2: metrics engine computes the new fields

**Files:** Modify `src/vllmstat/core/metrics.py`; Test `tests/test_metrics.py`.

- [ ] Add to `_LAT`: `"prefill": "vllm:request_prefill_time_seconds"`,
  `"decode": "vllm:request_decode_time_seconds"`. Add module bases:
  `_PROMPT_LEN = "vllm:request_prompt_tokens"`, `_GEN_LEN = "vllm:request_generation_tokens"`,
  `_SUCCESS = "vllm:request_success_total"`.
- [ ] `MetricsEngine.__init__`: add params `ttft_slo_s: float = 1.0`, `tpot_slo_s: float = 0.05`
  stored on self.
- [ ] Refactor the windowing out of `_quantiles` into a helper and reuse it:
```python
    def _window_buckets(self, fam: Families, base: str) -> list[tuple[float, float]]:
        cur = get_buckets(fam, base)
        if not cur:
            return []
        if self._prev is not None:
            prev = get_buckets(self._prev, base)
            if prev:
                delta = windowed_buckets(prev, cur)
                if delta and delta[-1][1] > 0:
                    return delta
        return cur

    def _quantiles(self, fam: Families, base: str) -> Quantiles:
        buckets = self._window_buckets(fam, base)
        if not buckets:
            return Quantiles()
        return Quantiles(
            p50=histogram_quantile(buckets, 0.50),
            p90=histogram_quantile(buckets, 0.90),
            p99=histogram_quantile(buckets, 0.99),
        )

    def _hist_stats(self, fam: Families, base: str) -> Quantiles:
        """p50/p90/p99 (windowed) plus a windowed mean (sum/count delta)."""
        q = self._quantiles(fam, base)
        cnt, s = hist_count(fam, base), hist_sum(fam, base)
        mean: float | None = None
        if self._prev is not None and cnt is not None and s is not None:
            pc, ps = hist_count(self._prev, base), hist_sum(self._prev, base)
            if pc is not None and ps is not None and cnt > pc:
                mean = (s - ps) / (cnt - pc)
        if mean is None and cnt and s is not None and cnt > 0:
            mean = s / cnt
        return Quantiles(p50=q.p50, p90=q.p90, p99=q.p99, mean=mean)

    def _finish_reasons(self, fam: Families) -> dict[str, float]:
        cur = {lbl.get("finished_reason", "?"): v for lbl, v in fam.get(_SUCCESS, [])}
        if not cur:
            return {}
        base = cur
        if self._prev is not None:
            prev = {lbl.get("finished_reason", "?"): v for lbl, v in self._prev.get(_SUCCESS, [])}
            delta = {k: max(0.0, cur.get(k, 0.0) - prev.get(k, 0.0)) for k in cur}
            if sum(delta.values()) > 0:
                base = delta
        total = sum(base.values())
        return {k: v / total for k, v in base.items()} if total > 0 else {}
```
  (Add `hist_count, hist_sum` to the `core.parse` import and `histogram_fraction_below` to the
  `core.histogram` import.)
- [ ] In `derive(...)`, compute and pass to the `Snapshot(...)`:
```python
        prefill=self._quantiles(fam, _LAT["prefill"]),
        decode=self._quantiles(fam, _LAT["decode"]),
        prompt_len=self._hist_stats(fam, _PROMPT_LEN),
        gen_len=self._hist_stats(fam, _GEN_LEN),
        finish_reasons=self._finish_reasons(fam),
        goodput_ttft=histogram_fraction_below(self._window_buckets(fam, _LAT["ttft"]), self.ttft_slo_s),
        goodput_tpot=histogram_fraction_below(self._window_buckets(fam, _LAT["tpot"]), self.tpot_slo_s),
        ttft_slo_s=self.ttft_slo_s,
        tpot_slo_s=self.tpot_slo_s,
```
- [ ] Tests (`test_metrics.py`): build a two-sample Prometheus text fixture (prev → cur) exposing
  `request_prefill_time_seconds_bucket/_sum/_count`, `request_generation_tokens_bucket/_sum/_count`,
  `request_success_total{finished_reason="stop"|"length"}`, and `time_to_first_token_seconds_bucket`.
  Derive twice and assert: `snap.prefill.p50` is a positive float; `snap.gen_len.mean` ≈ expected
  windowed average; `snap.finish_reasons` sums to ~1.0 with the right split; `0 <= snap.goodput_ttft
  <= 1`. (Follow the existing `test_metrics.py` fixture style — reuse helpers there.)
- [ ] Commit: `feat(metrics): latency phases, request-shape, finish-reasons, goodput`.

---

## Task 3: renderers

**Files:** Modify `src/vllmstat/render.py`; Test `tests/test_render.py`. Read the existing
`latency`, `efficiency`, `session` renderers first to match style (plain strings; `fmt_dur`,
`fmt_pct`, `fmt_si` helpers already imported).

- [ ] **Latency phases** — extend `render.latency(s)` to add `prefill` and `decode` rows after the
  existing `queue` row (same `_q(label, quantiles)` table formatting). So the panel shows TTFT, TPOT,
  e2e, queue, prefill, decode.
- [ ] **Request shape** — new `render.request_shape(s) -> str` (returns "" when no data, i.e.
  `s.prompt_len.mean is None and s.gen_len.mean is None`):
```
REQUEST SHAPE (recent)
 prompt  avg {fmt_si(mean)}  p50 {fmt_si(p50)}  p90 {fmt_si(p90)} tok
 gen     avg {fmt_si(mean)}  p50 {fmt_si(p50)}  p90 {fmt_si(p90)} tok
```
  (Use `fmt_si` for token counts; show `—` for None components.)
- [ ] **Outcomes** — new `render.outcomes(s) -> str` (returns "" when `not s.finish_reasons` and both
  goodput values are None):
```
OUTCOMES (recent)
 {reason fractions joined: "stop 92% · length 6% · abort 2%"}
 goodput  TTFT<{fmt_dur(s.ttft_slo_s)} {fmt_pct(s.goodput_ttft)} · TPOT<{fmt_dur(s.tpot_slo_s)} {fmt_pct(s.goodput_tpot)}
```
  Sort reasons by descending fraction; the goodput line only shows a side when its value is not None.
- [ ] **Energy efficiency** — extend `render.efficiency(s)`: compute total GPU power
  `pw = sum(g.power_w for g in s.gpu.gpus if g.power_w)`; when `pw > 0 and s.gen_tps > 0`, append
  `f"{s.gen_tps / pw:.1f} tok/W"` and `f"{pw / s.gen_tps:.2f} J/tok"` to the parts list. Broaden the
  panel's visibility: it should render when `s.eff_active` **or** that tok/W pair is available (adjust
  the early-return guard accordingly).
- [ ] Tests (`test_render.py`): a snapshot with `prefill`/`decode` populated → `latency` output
  contains "prefill"/"decode"; `request_shape` with `prompt_len=Quantiles(p50=1400, mean=2100)` shows
  "prompt" + "avg"; empty → ""; `outcomes` with `finish_reasons={"stop":0.92,"length":0.08}`,
  `goodput_ttft=0.88` shows "stop" and "TTFT<"; empty → ""; `efficiency` with `gen_tps=1400` and a
  GPU `power_w=238` shows "tok/W".
- [ ] Commit: `feat(ui): latency phases, request-shape, outcomes, tok/W renderers`.

---

## Task 4: app wiring (new panels)

**Files:** Modify `src/vllmstat/app.py`; Test `tests/test_app.py`. Read `app.py` first.

- [ ] In `compose`, add two panels inside the `#detail` `Vertical`, placed right after `p_lat`
  (latency) for the shape panel and after `p_session`/near `p_eff` for outcomes — pick a sensible
  order: header, row1(conc/tput/lat), cache, **shape**, session, **outcomes**, eff, spec, gpu, tee:
  `self.p_shape = Panel(id="shape")` and `self.p_outcomes = Panel(id="outcomes")`.
- [ ] In `_refresh_detail`, drive them exactly like the existing conditional `eff`/`spec` panels:
```python
        shape = render.request_shape(snap)
        self.p_shape.display = bool(shape)
        self.p_shape.update(shape)
        outcomes = render.outcomes(snap)
        self.p_outcomes.display = bool(outcomes)
        self.p_outcomes.update(outcomes)
```
- [ ] Test (`test_app.py`, mock mode): after a few ticks the app has `#shape` and `#outcomes`
  panels queryable, and the app doesn't crash. (Mock data may or may not populate them; assert the
  panels exist and `app.snapshot is not None`. If the mock provider doesn't emit the new histograms,
  just assert no crash + panels present with `display in (True, False)`.)
- [ ] Commit: `feat(ui): mount request-shape and outcomes panels`.

---

## Task 5: auto-discover Docker when the default endpoint is down

**Files:** Modify `src/vllmstat/cli.py`; Test `tests/test_cli_once.py`. Read `cli.py`
(`resolve_instances`) first.

- [ ] Add a probe helper (TCP connect; never raises):
```python
def port_responding(url: str, timeout: float = 0.4) -> bool:
    import socket
    from urllib.parse import urlparse
    p = urlparse(url if "://" in url else "http://" + url)
    host, port = p.hostname or "localhost", p.port or (443 if p.scheme == "https" else 80)
    try:
        with socket.create_connection((host, port), timeout=timeout):
            return True
    except OSError:
        return False
```
- [ ] In `resolve_instances`, BEFORE the final `resolve_fleet(...)`: when no explicit source was given
  (`not config_instances and not docker_instances and not cfg.urls`) AND not `cfg.mock`, probe the
  default. If it's down, try discovery and use it (with a one-line stderr note). Concretely, after
  computing `config_instances` and `docker_instances = discover_docker() if cfg.discover_docker else []`:
```python
    default_url = "http://localhost:8000"
    if not config_instances and not docker_instances and not cfg.urls and not cfg.mock:
        if not port_responding(default_url):
            found = discover_docker()
            if found:
                docker_instances = found
                print(
                    f"vllmstat: {default_url} not responding — "
                    f"found {len(found)} vLLM container(s) via Docker",
                    file=sys.stderr,
                )
    cfg.instances = resolve_fleet(config_instances, docker_instances, cfg.urls, ...)
```
  (Keep the existing `--logs` propagation and config-globals logic. `resolve_fleet` still falls back
  to the bare default when `found` is empty — unchanged behavior when nothing is discovered.)
- [ ] Tests (`test_cli_once.py`): monkeypatch `vllmstat.cli.port_responding` and
  `vllmstat.cli.discover_docker` (or inject) to assert: default **up** → instances == [localhost:8000]
  and discovery NOT consulted; default **down** + discovery returns `[Instance("c","http://localhost:9001")]`
  → `cfg.instances` is that discovered instance; default **down** + discovery returns `[]` →
  falls back to localhost:8000. Use `monkeypatch.setattr` on the `vllmstat.cli` names.
- [ ] Commit: `feat(cli): auto-discover Docker vLLM when the default endpoint is down`.

---

## Task 6 (controller): docs, version, screenshot, live test, release

- [ ] Bump to `0.6.0` (pyproject + `__init__`).
- [ ] README: "What it shows" — add latency-phases, request-shape, outcomes/goodput, tok/W bullets;
  document auto-discover (zero-config behavior) and that goodput SLOs default to TTFT<1s / TPOT<50ms.
- [ ] Regenerate the demo GIF generator with the new panels (optional) or add a static screenshot.
- [ ] Live test against the real `:8000`: confirm the new panels populate under load; confirm
  `vllmstat` with `:8000` stopped auto-discovers the `vllm-xpu` container.
- [ ] Full `ruff && pyright && pytest` green; final review; merge; build; twine upload `0.6.0`.

## Self-review notes
- Goodput SLO thresholds are baked into `MetricsEngine` defaults (1.0s / 0.05s) — CLI flags are a
  future follow-up; `Snapshot` carries them for display.
- New histogram fields reuse the existing windowed-bucket machinery; no per-request state.
- Auto-discover only triggers in the true zero-config case (no `--url`/`--config`/`--discover-docker`,
  not `--mock`), so explicit usage and the test suite are unaffected.
