Metadata-Version: 2.4
Name: infermon
Version: 0.1.0rc3
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Rust
Requires-Dist: vllm==0.31.0 ; extra == 'vllm'
Provides-Extra: vllm
Summary: Real-time monitoring, control and auto-tuning for vLLM
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

# infermon

Real-time monitoring and live knob control for vLLM 0.31.0. Closed source. Free to monitor, paid to change.

- Per-step engine data, not scraped histograms: exact TTFT / ITL / TPOT / e2e / queue quantiles, throughput, running / waiting / skipped, KV usage, preemptions, prefix cache hit rate, CUDA graph padding waste, spec decode acceptance, batch mix.
- Binding-limit diagnosis for each step: `KV_PRESSURE`, `SEQ_CAP`, `TOKEN_BUDGET`, `DEFERRED`, `UNDERLOADED`. SLA attainment and goodput.
- Live control (Pro): `max_num_active_seqs` and `max_num_scheduled_tokens` at the next step boundary, no restart. Bounds = the startup values.
- Local web UI, `infermon top` in the terminal, Prometheus `/metrics`.

Stock vLLM wheel, no fork. Two probes (a stat-logger plugin and a scheduler mixin) write to a shared-memory ring; one static Rust binary (`infermon-agent`) reads it. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).

## Install (2 minutes)

Linux x86_64 or aarch64. vLLM **0.31.0** exactly (other versions run as passthrough, see FAQ).

### A. Fresh: Docker image

```bash
docker run --gpus all --network host --ipc host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  ghcr.io/bardavid/infermon:0.31.0-cuda \
  Qwen/Qwen2.5-7B-Instruct \
  --scheduler-cls infermon.sched.TunableAsyncScheduler \
  --enable-logging-iteration-details
```

- The image is `vllm/vllm-openai:v0.31.0` plus the infermon wheel. Arguments go to `vllm serve`.
- The entrypoint starts the agent next to vLLM. UI: http://127.0.0.1:8765/.
- `INFERMON_AGENT=0` turns the built-in agent off. `INFERMON_LISTEN=0.0.0.0:8765` with `INFERMON_TOKEN=<t>` exposes it.
- CPU: `ghcr.io/bardavid/infermon:0.31.0-cpu` (from `vllm/vllm-openai-cpu:v0.31.0`).
- Kubernetes: [deploy/helm/infermon](deploy/helm/infermon) (agent as a sidecar).

### B. Attach to an existing vLLM

```bash
pip install infermon            # into the same env as vllm==0.31.0; no dependencies
                                # (or: pip install infermon-0.1.0-cp310-abi3-manylinux_2_28_$(uname -m).whl from the GitHub release)
infermon doctor                 # vLLM version, compat canary, flags, agent binary
```

Monitoring (Free) needs no vLLM flag: the stat-logger plugin loads automatically. Restart `vllm serve` once.
For live control (Pro), add the scheduler class:

```bash
vllm serve <model> --scheduler-cls infermon.sched.TunableAsyncScheduler \
  --enable-logging-iteration-details      # batch mix + step time panels
infermon up                     # starts the bundled agent; prints the UI URL
```

`infermon serve -- <vllm serve args>` adds `--scheduler-cls` for you. Use `TunableScheduler` if you run with `--no-async-scheduling`.

Optional flags for more panels: `--cudagraph-metrics` (CUDA graph waste), `--enable-mfu-metrics`, `--kv-cache-metrics`. Do not pass `--disable-log-stats`.

### Try it with no GPU

```bash
pip install "infermon[vllm]"    # pulls vllm==0.31.0
infermon demo                   # real vLLM scheduler + synthetic model + agent; prints the UI URL
```

The demo runs the real vLLM 0.31.0 `Scheduler` with our mixin on CPU, with Poisson load and a regime shift. No model download, no network.

### License (Pro and up)

```bash
infermon license install license.json   # saves to ~/.config/infermon/license.json, reloads the agent
```

Offline Ed25519 check. No phone-home. Expired license + grace → Free; vLLM never stops.

### Share a trace, get 30 days of Pro

```bash
infermon trace record --hours 24            # all probes in INFERMON_DIR -> infermon-trace-<time>.trace.gz
infermon trace info infermon-trace-*.trace.gz
infermon trace share infermon-trace-*.trace.gz --email you@example.com   # prints a Stripe Checkout URL
```

- A trace holds ring records only: counts and timings. No prompt or output text exists in them. Ring/socket paths are dropped, the hostname is replaced by its sha256, a local model path keeps only its last component.
- The reward is a 30-day Pro trial for the GPUs in the trace. A card is required. It auto-renews into paid Pro unless you cancel. One reward per customer.
- Only `trace share` sends data, and only when you run it. Format: [docs/SPEC.md](docs/SPEC.md) section 14.

## Tiers

| | Free | Pro | Fleet | Enterprise |
|---|:-:|:-:|:-:|:-:|
| Real-time monitor, UI, `top`, all internal panels | ✓ | ✓ | ✓ | ✓ |
| Binding-limit diagnosis, SLA attainment, goodput, hints | ✓ | ✓ | ✓ | ✓ |
| Prometheus export | ✓ | ✓ | ✓ | ✓ |
| History retention | 24 h | 30 d* | 90 d | custom |
| Live knob control (manual), audit log | – | ✓ | ✓ | ✓ |
| Tuning recommendations, trace replay, alerts | – | later | later | later |
| Multi-host fleet view, OTLP, RBAC/SSO | – | – | later | later |
| Closed-loop auto-tune | – | – | – | later |
| Price | $0 | $15 / GPU / month | $25 / GPU / month | $40 / GPU / month |

\* v0.1 keeps history in memory only (Pro capped at 7 d). The disk store comes later.

## Security and privacy

- Prompt and output content never leave the vLLM process. The probes record counts and times only.
- The agent makes no outbound connections. No telemetry. `infermon trace share` uploads a trace only when you run it. Licenses are verified offline.
- Rings, manifests and control sockets are files with mode `0600` in a `0700` directory (`INFERMON_DIR`, default `/dev/shm/infermon-<uid>`). The control socket checks the peer uid (`SO_PEERCRED`).
- The agent API binds `127.0.0.1:8765`. A non-loopback `--listen` requires `--token`; mutating requests then need `Authorization: Bearer <token>`.
- Knob changes go through a governor: schema bounds (never above startup), ≤ 1 change per knob per 2 s, audit log.

## FAQ

**Is it safe in my serving process?**
- Probe-F (stat logger) never raises into vLLM. On an error it logs one WARNING and turns itself off; after 10 errors in a row in `record()` it disables itself.
- Probe-E (scheduler mixin) runs only if the version check and the AST canary over the vLLM scheduler source pass. Otherwise it is a passthrough to the stock `Scheduler`: no ring, no socket, vLLM serves.
- Knob writes apply only at a step boundary, from a lock-protected swap. The values are bounded by the startup values, so no buffer or CUDA graph is resized.
- If the agent dies, the knobs keep their last value and vLLM keeps serving. `INFERMON_DISABLE=1` turns every probe off.
- Lowering `max_num_active_seqs` gates admission; it does not preempt running requests. They drain to the new cap.

**What does it cost in latency?**
Measured on CPU (`tests/py/bench_probe.py`, Python 3.13, vLLM 0.31.0):

| Hook | Load | Cost per step |
|---|---|---|
| Probe-F `record()` (API server, off the engine loop) | 32 running / 256 running, 0 finished | 2.7 µs / 5.7 µs |
| Probe-F `record()` | 32 running / 256 running, 2 finished | 5.4 µs / 8.4 µs |
| Probe-E `schedule()` mixin (EngineCore) | 32 running | +1.5 µs on a 280 µs `schedule()` (0.5 %) |
| Probe-E `schedule()` mixin | 256 running | +3.2 µs on a 2.0 ms `schedule()` (0.2 %) |

GPU step times are 10–50 ms, so the engine-side cost is far below 0.1 % of TPOT. GPU numbers come from the owner GPU run ([docs/gpu-test.md](docs/gpu-test.md)). The optional vLLM flags (`--enable-logging-iteration-details`, `--cudagraph-metrics`, ...) have their own cost inside vLLM; measure them with the same script.

**Which vLLM versions?** 0.31.0. On another version, Probe-E is a passthrough and logs an ERROR; Probe-F runs if its canary passes. `INFERMON_ALLOW_UNTESTED=1` overrides the version check (not the canary). A nightly job tests each new vLLM release.

**Does it work with Grafana?** Yes. Scrape `http://127.0.0.1:8765/metrics` (`infermon_<metric>{engine,window}`, latencies in ms).

**Multi-node, PD disaggregation?** Not in v0.1. Multiple engines and DP ranks on one host: yes.

## Build from source (maintainers)

```bash
scripts/build_wheel.sh                     # static musl agent + manylinux_2_28 wheel in dist/ (off a manylinux host: add --zig)
cargo test --workspace && pytest tests/py -q
```

CI: `.github/workflows/` — `ci` (T0–T2), `e2e-cpu` (T3: real `vllm serve` on CPU), `release` (wheels + images on `v*` tags), `nightly-gate` (new vLLM releases).

