Metadata-Version: 2.4
Name: inflame-trace
Version: 0.1.0
Summary: Where does your LLM inference latency live? vLLM/TGI/Triton traces in, flamegraph-style HTML out.
Author-email: Sophie Nguyen <sophie.nguyenthuthuy@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/sophie-nguyenthuthuy/inflame
Keywords: llm,inference,latency,vllm,tgi,triton,flamegraph,observability
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# inflame

**Where does your LLM inference latency live?**

Feed it vLLM / TGI / Triton request traces; get a self-contained HTML report
that shows, per request and fleet-wide, how wall time splits across
**tokenization → queue → prefill → decode → data io → network**. Zero
dependencies, stdlib only.

```bash
pip install inflame-trace

inflame report traces.jsonl -o report.html   # flamegraph-style HTML
inflame summary traces.jsonl                 # terminal table
inflame formats                              # how to produce each input
```

The report has three layers:

1. **Where latency lives** — one icicle bar: share of total wall time per phase.
2. **Slowest requests** — a waterfall of stacked per-request bars, phases in
   request order; hover any segment for exact timings, hover the legend to
   isolate a phase.
3. **Phase percentiles** — p50/p95/p99 per phase, as a table.

The terminal `summary` gives you the same phase split as a bar chart you can
paste into an incident channel.

## Getting traces out of your stack

| Stack | How | Phases you get |
|---|---|---|
| **vLLM** | `vllm serve MODEL --otlp-traces-endpoint http://collector:4318/v1/traces`, collector writes a JSON/file export | queue, prefill, decode |
| **TGI** | `text-generation-launcher --json-output ... 2>&1 \| tee tgi.log` | tokenization, queue, inference |
| **Triton** | `tritonserver --trace-config mode=triton --trace-config triton,file=trace.json --trace-config rate=1 --trace-config level=TIMESTAMPS` | network, queue, data io, inference |

vLLM inputs are accepted as raw OTLP exports, JSON arrays, or JSONL of spans —
whatever your collector writes. Format is auto-detected; force it with
`--format vllm|tgi|triton`.

## Reading the report

- **queue dominating** → batch scheduler is saturated: add replicas, cap
  concurrency upstream, or shrink max batch tokens.
- **prefill dominating** → long prompts: cache system prompts
  (prefix caching), trim context, or split prefill across GPUs.
- **decode dominating** → normal for long generations; compare decode tok/s
  against your quantization/kernel expectations — a drop here is where
  quantization overhead or a mis-tuned kernel shows up.
- **data io / network visible at all** → payloads are too fat (send token ids,
  not logits; compress embeddings).

## Honest limitations

- TGI does not expose a prefill/decode split; its model compute is one
  `inference` phase (completion tokens are estimated from `time_per_token`).
- vLLM's periodic `Avg prompt throughput` stats log has no per-request data
  and is rejected with a pointer to the OTLP setup.
- Triton phases describe the server's view (queue/compute/marshalling); for
  in-model phase splits use an ensemble that traces per step.

## Demo

```bash
python3 scripts/make_demo.py   # deterministic 240-request sample -> demo/index.html
```

## Dev

```bash
python3 -m pytest -q   # 60+ tests, no network, no deps
```

MIT.
