Metadata-Version: 2.4
Name: inference-profiler
Version: 0.1.0
Summary: A profiling tool for LLM inference servers: measure, diagnose, and visualize performance bottlenecks.
Author: Abhilash Sarnad
License: MIT
Project-URL: Homepage, https://github.com/SarnadAbhilash/inference_profiler
Keywords: llm,inference,profiling,benchmark,vllm,performance
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: System :: Benchmark
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.5
Requires-Dist: numpy>=1.24
Requires-Dist: pandas>=2.0
Requires-Dist: plotly>=5.18
Requires-Dist: psutil>=5.9
Requires-Dist: jinja2>=3.1
Provides-Extra: gpu
Requires-Dist: nvidia-ml-py>=12.535; extra == "gpu"
Provides-Extra: dev
Requires-Dist: pytest>=7.4; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: mypy>=1.8; extra == "dev"
Requires-Dist: pre-commit>=3.6; extra == "dev"
Dynamic: license-file

# Inference Profiler

A profiling tool for LLM inference servers — the `perf` / PyTorch Profiler equivalent for serving.
Point it at any OpenAI-compatible endpoint (vLLM, SGLang, TensorRT-LLM, TGI, llama.cpp server, …)
and it measures where time goes, diagnoses bottlenecks, and renders a self-contained HTML report.

Everything runs locally. No cloud backend, no auth, no telemetry.

## Why

Engineers usually know latency is bad, but not *why*. Inference Profiler answers:

- Why is TTFT high — prefill compute, or requests stuck in the scheduler queue?
- Is batching actually working, or is the GPU idle between requests?
- Where is the throughput/latency knee as concurrency grows?
- Are a few long prompts ruining tail latency for everyone?
- Is the server compute-saturated, or starved for work?

## Install

```bash
pip install inference-profiler          # core
pip install "inference-profiler[gpu]"   # + NVML GPU metrics (NVIDIA)
```

## Quick start

```bash
inference-profiler profile \
    --endpoint http://localhost:8000/v1 \
    --model llama3 \
    --requests 500 \
    --concurrency 32 \
    --stream
```

Console output shows per-stage latency/throughput, findings, and an ASCII waterfall.
Full artifacts land in `results/`:

```text
results/
├── config.json      # exact run configuration (reproducible)
├── metrics.jsonl    # one line per request: timestamps, tokens, errors
├── system.jsonl     # GPU/CPU/RAM/network samples over time
├── summary.json     # aggregated metrics + derived estimates
├── timeline.json    # per-request phase spans (queue/waiting/decode)
├── findings.json    # automatic analysis results with evidence
├── report.html      # self-contained interactive report
└── charts/          # each chart as a standalone HTML file
```

### No GPU? Try the mock server

```bash
python examples/mock_server.py --port 8399 --batch-size 8 &
inference-profiler profile --endpoint http://127.0.0.1:8399/v1 --model demo \
    --requests 100 --concurrency 4,8,16,32
```

The mock server queues requests beyond its batch size — the profiler will find the knee.

## What it measures

| Category | Metrics |
|---|---|
| Latency | TTFT, inter-token latency, end-to-end · P50/P90/P95/P99 |
| Throughput | requests/s, output tokens/s, total tokens/s |
| Tokens | prompt/output totals, means, min/max, per-request decode speed |
| System | GPU utilization & memory (NVML), CPU, RAM, network |
| Derived estimates | batch utilization, prefill vs scheduler-wait split, GPU idle %, saturation knee |

Measured numbers and heuristic estimates are kept strictly separate — estimates are
suffixed `*_estimate` and their method is documented in
[docs/methodology.md](docs/methodology.md). Findings are only ever generated from
measured data.

## Common recipes

```bash
# Concurrency sweep: find the throughput/latency knee
inference-profiler profile --endpoint http://localhost:8000/v1 --model llama3 \
    --requests 200 --concurrency 8,16,32,64,128

# Open-loop (Poisson arrivals at 25 req/s) — reveals queueing under overload
inference-profiler profile --endpoint http://localhost:8000/v1 --model llama3 \
    --requests 500 --arrival-mode open --rate 25

# Burst traffic
inference-profiler profile --endpoint http://localhost:8000/v1 --model llama3 \
    --requests 300 --arrival-mode burst --rate 50 --burst-size 40 --burst-idle 5

# Your own prompts (JSONL with a "prompt" field), controlled output lengths
inference-profiler profile --endpoint http://localhost:8000/v1 --model llama3 \
    --dataset examples/prompts.jsonl --output-tokens 64-256 --requests 200

# Everything in a config file (CLI flags override it)
inference-profiler profile --config examples/sweep_config.json

# Re-generate the report from raw data (tweak nothing, re-analyze)
inference-profiler report --results results/
```

## Reading the report

- **Findings** — ranked observations with the evidence that triggered them, e.g.
  *“GPU utilization averaged only 42%”* or *“Throughput saturates near concurrency 16”*.
- **Timeline** — a waterfall of every request split into client-queue, waiting
  (TTFT), and decode phases. Growing gray/yellow bands = queueing; long blue = decode-bound.
- **Charts** — latency/TTFT histograms, percentile curves, tokens/s over time,
  in-flight concurrency, GPU utilization & memory, prompt/output length vs latency,
  and concurrency scaling (on sweeps).

## Architecture

```text
cli ─▶ config (pydantic) ─▶ runner ──▶ workload   (synthetic / JSONL prompts)
                              │    ──▶ scheduler  (closed / open-poisson / burst)
                              │    ──▶ collector  (httpx + SSE timing per request)
                              │    ──▶ gpu_monitor (NVML/psutil background sampling)
                              ▼
                    metrics ─▶ timeline ─▶ analysis ─▶ charts ─▶ report.html
```

See [docs/architecture.md](docs/architecture.md). Adding a non-OpenAI backend means
implementing one `execute_request` variant; everything downstream operates on
`RequestRecord`s and is backend-agnostic.

## Development

```bash
git clone https://github.com/SarnadAbhilash/inference_profiler
cd inference-profiler
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest            # no GPU or server required — uses a mock transport
ruff check .
pre-commit install
```

## License

MIT
