Metadata-Version: 2.5
Name: llamastat
Version: 0.3.0
Summary: nvtop for llama.cpp: an interactive terminal dashboard for llama-server inference monitoring
Project-URL: Homepage, https://github.com/bryanvine/llamastat
Project-URL: Issues, https://github.com/bryanvine/llamastat/issues
Author-email: Bryan Vine <bryan@bryanvine.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: gguf,llama-server,llama.cpp,llm,monitoring,tui
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27
Requires-Dist: prometheus-client>=0.20
Requires-Dist: textual>=0.60
Provides-Extra: dev
Requires-Dist: pyright>=1.1; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Description-Content-Type: text/markdown

# llamastat

**nvtop for llama.cpp**: a zero-infrastructure terminal dashboard for `llama-server` inference. Point it at a running server and watch slots, throughput, and context usage live.

![llamastat](https://raw.githubusercontent.com/bryanvine/llamastat/main/docs/screenshot.svg)

---

## Why llamastat?

`llama-server` tells you what it is doing, but only in raw JSON. llamastat turns that into a live picture:

- **Per-slot activity**: which slots are processing prompts, which are generating, and how full each context window is.
- **Throughput without `--metrics`**: llamastat diffs slot token counters between polls, so you get generation and prompt tokens/s even on servers started without the Prometheus endpoint.
- **Zero infrastructure**: no Prometheus, no Grafana, no agents. One `pip install`, one command, works over any HTTP reachable server (including an SSH port-forward to a production box).
- **Degrades gracefully**: if the server flaps, is still loading the model, or has `/slots` disabled, llamastat shows what it can instead of crashing.

Sibling project: [vllmstat](https://github.com/bryanvine/vllmstat) does the same for vLLM.

## Install

```bash
pip install llamastat
```

## Usage

```bash
llamastat                          # local server on the default port
llamastat -u http://host:8080      # remote server
llamastat -u http://host:8080 -i 0.5
llamastat --once --json            # one snapshot as JSON, for scripting
llamastat --mock                   # synthetic data, no server needed
```

### Key bindings

| Key | Action |
| --- | ------ |
| `q` | Quit |
| `p` | Pause/resume polling |
| `+` | Poll faster (halve interval, min 0.1s) |
| `-` | Poll slower (double interval, max 10s) |

### Flags

| Flag | Default | Meaning |
| ---- | ------- | ------- |
| `-u, --url` | `http://localhost:8080` | llama-server base URL |
| `-i, --interval` | `1.0` | poll interval in seconds |
| `--api-key` | `$LLAMACPP_API_KEY` | bearer token for servers started with `--api-key` |
| `--metrics-path` | `/metrics` | Prometheus metrics path |
| `--no-host` | off | disable local host sensors (hwmon temps, RAPL power) |
| `--once` | off | poll once, print, exit |
| `--json` | off | with `--once`, print machine-readable JSON |
| `--mock` | off | run against synthetic data |

## What it shows

- **Header**: model alias, quantization, parameter count, file size, context per slot, training context, server build, health.
- **Slots**: one row per slot with phase (`prompt`/`gen`/`idle`), task id, prompt progress, generated tokens, per-slot tokens/s, and a context-usage bar.
- **Throughput**: EWMA generation and prompt tokens/s with session peaks and totals, plus a braille history plot.
- **Requests**: running/peak concurrency, completed tasks, aggregate context in use.
- **Host sensors** (Linux, when llamastat runs on the inference box itself): aggregate CPU utilization bar with load average, RAM usage, and CPU, RAM, motherboard, NVMe, and GPU temperatures from hwmon, grouped per chip, plus power draw from hwmon sensors and RAPL package counters. The CPU bar reflects the whole machine, so standalone tools like `llama-bench` show up here even though they never touch llama-server. RAPL wattage is derived by diffing the energy counters between polls. On hardened kernels `energy_uj` is root-only; run `sudo chmod a+r /sys/class/powercap/intel-rapl*/energy_uj` (or run llamastat as root) to expose package power, and llamastat will say so in the panel. Disable everything with `--no-host`.
- **Server metrics**: when the server was started with `--metrics`, llamastat also shows the server-side Prometheus counters (total tokens, deferred requests, context high-water mark, busy slots per decode). The panel hides itself otherwise.
- **Sampling defaults**: temperature, top_p, top_k, min_p, max_tokens from `/props`.

llamastat only issues read-only GET requests (`/health`, `/slots`, `/props`, `/v1/models`, and optionally `/metrics`), so it is safe to point at a production server.

## Requirements

- Python 3.10+
- A llama.cpp `llama-server` new enough to expose `/slots` (enabled by default; some deployments disable it, in which case llamastat falls back to header and metrics panels)

## Development

```bash
git clone https://github.com/bryanvine/llamastat
cd llamastat
pip install -e ".[dev]"
pytest -q
```

See [CONTRIBUTING.md](CONTRIBUTING.md).

## License

Apache-2.0. See [LICENSE](LICENSE).
