Metadata-Version: 2.4
Name: llmtrafficlens
Version: 0.2.0
Summary: LLM gateway traffic profiler: prefix-reuse analysis, cache-hit-rate upper bounds, and anonymized replayable trace export
Project-URL: Homepage, https://github.com/GMISWE/llmtrafficlens
Author: Jason Zhu
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: benchmark,kv-cache,llm,mooncake,prefix-cache,trace,traffic-analysis,workload-characterization
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Benchmark
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# llmtrafficlens

Point it at your LLM gateway's request log. It tells you **how much a
prefix cache could save on your real traffic** — before you build anything —
and turns the log into an **anonymized dataset you can load-test with**.

```
                       ┌────────────────────────────────────────────┐
  gateway log (CSV) ──►│ llmtrafficlens                             │
  · request_json       │  chunk → block-hash → prefix tree          │
  · response_json      │  (all local; prompts never leave the box)  │
  · timestamp, status  └──┬─────────────┬───────────────┬───────────┘
                          ▼             ▼               ▼
                     report.html   trace.jsonl     SGLang bench args
                     "worth it?"   "test with it"  "or synthesize it"
```

## Input → Output

| Command | Input | Output | The question it answers |
|---|---|---|---|
| `profile` | gateway log CSV | `report.html` + `.json` | Is my traffic worth prefix-aware routing? What's the ceiling? |
| `export` | gateway log CSV | anonymized `trace.jsonl` | How do I load-test / share this traffic without leaking a single prompt? |
| `fit` | gateway log CSV | SGLang bench CLI args | What generator settings reproduce my traffic shape? |
| `validate` | public trace | hit-rate numbers | Is this tool's math correct? (compare with the Mooncake simulator) |

**Input format**: a CSV with a `request_json` column holding the
OpenAI-format request body. Optional columns picked up automatically:
`response_json`, `model_name`, `status_code`, and a timestamp column
(`timestamp`/`ts`/`created_at`/... — epoch or ISO-8601; or set
`--ts-column`). Public Mooncake / qwen-bailian traces are also accepted
as input (`--format mooncake|bailian`).

## What you get

### 1. `profile` — the decision numbers

```bash
pip install llmtrafficlens
llmtrafficlens profile gateway.csv -o report
```

`report.html` shows, on *your* traffic:

- **Ideal hit rate (upper bound)** — the ceiling on prefix-cache savings:
  the share of input tokens that would hit under an infinite cache with
  perfect routing (the metric from
  [KVCache in the Wild, ATC'25](https://arxiv.org/abs/2506.02634)).
  If this is 3%, stop here; if it's 37%, keep reading.
- **LRU capacity curve** — hit rate at each cache size, so you can judge
  how close to the ceiling a real deployment can get.
- **Dual-metric prefix leaderboard** — top prefixes *by reuse count* vs
  *by reusable token volume*. The heads differ: a 15-token greeting reused
  870× is worthless to a cache; a 19K-token system prompt reused 95× is
  the whole prize. This tells you what your routing actually has to keep
  resident.
- **Session-identifier presence** — 0% means your reuse is cross-request
  shaped and session-affinity routing cannot capture it; prefix-aware
  routing is required.
- Input/output token distributions, streaming ratio, model mix, QPS.

### 2. `export` — an anonymized, replayable dataset

```bash
llmtrafficlens export gateway.csv --to bailian -o trace.jsonl
```

Each request becomes one line — **no text, only structure**:

```json
{"chat_id": "4abc41ba209004a0", "timestamp": 1753340000123,
 "input_length": 1994, "output_length": 117, "type": "glm-4", "turn": 1,
 "hash_ids": [4251731047194047120, 8125214104179782736, ...],
 "stream": true, "status": 200, "ts_exact": true}
```

`hash_ids` is a salted chained block-hash fingerprint (qwen-bailian
convention, 16-token blocks): two requests share the first N hashes
exactly when they share their first N blocks. Replay tools that support
this format regenerate synthetic prompts from the hashes — same hash,
same synthetic block — reproducing the **prefix-sharing structure** of
production while the content is fake. The export targets NVIDIA AIPerf's
documented `bailian_trace` mode:

```bash
aiperf profile --custom-dataset-type bailian_trace --input-file trace.jsonl ...
```

Honesty note: our extra fields (`stream`, `status`, `ts_exact`) ride on
top of the bailian schema, and we have not yet round-trip-tested the
export against AIPerf itself — that verification is on the roadmap. If a
strict parser rejects the extra fields, drop them with `jq`.

### 3. `fit` — generator settings instead of hand-tuning

```bash
llmtrafficlens fit gateway.csv --target sglang-gsp
```

Emits ready-to-run SGLang `bench_serving` arguments (group count, zipf
popularity, prompt lengths) fitted to your measured histogram, plus a
**fidelity report**: it simulates the request stream the fitted generator
would emit and scores it with the same hit-rate metric as your real
traffic. A small gap means the generator is a faithful stand-in; a large
gap means it flattens your traffic and you should replay the exported
trace instead.

## Try it on the bundled sample (no data of your own needed)

The repository (not the pip package) ships a fully synthetic sample log —
360 requests, three shared system prompts with deliberately skewed
popularity, sparse session keys, real timestamps
(`examples/sample_gateway.csv`, regenerate with
`python examples/generate_sample.py`):

```bash
git clone https://github.com/GMISWE/llmtrafficlens && cd llmtrafficlens
llmtrafficlens profile examples/sample_gateway.csv -o sample-report
```

What it finds (and why each number matters):

| Metric | Value | Reading |
|---|---|---|
| Ideal hit rate (upper bound) | **66.4%** | worth building prefix-aware routing for |
| Top prefix by count | 16 tok × 180 reqs = 2.3% of reusable volume | popularity ≠ value: not worth pinning |
| Top prefix by volume | 2400 tok × 30 reqs = **55.6%** of reusable volume | this is what routing must keep resident |
| Session-id presence | 4.2% | reuse is cross-request shaped; affinity alone can't catch it |

And `fit` on the same sample demonstrates the fidelity report doing its
job — the gap is large, which is the tool telling you to replay the trace
instead of trusting a generator:

```bash
llmtrafficlens fit examples/sample_gateway.csv --target sglang-gsp
# → zipf_alpha 1.56, fidelity gap +0.30  ← generator can't represent
#   this mix of shared groups and singletons; use export + replay
```

## Scenarios

### "Should we build prefix-cache-aware routing?"

You run a multi-tenant gateway and someone proposes cache-aware routing.
Instead of arguing from intuition:

```bash
llmtrafficlens profile last-week.csv -o report
```

Three numbers settle it. **Ideal hit rate 4%** → decline the proposal
with evidence; the traffic has no shared structure worth routing for.
**Upper bound 40%, session-id presence 85%** → cheap session affinity
captures most of it; no prefix machinery needed. **Upper bound 40%,
session-id presence 0%** → reuse lives in cross-request shared prefixes;
only prefix-aware routing reaches it — fund the project, and attach the
report to the design doc as its quantitative basis.

### "How much cache memory does each worker need?"

The LRU capacity curve in the report shows hit rate at each cache size.
On the public Mooncake conversation trace (run the Verify section below to
reproduce): 4.2% at 64K tokens, 5.6% at 1M, 18.5% at 4M, 27.1% at 8M,
34.2% at 16M, against a 37.4% ceiling — on that workload the curve only
takes off past ~2M tokens, so a small cache buys almost nothing and the
sizing answer sits where your curve flattens toward its ceiling. Multiply
tokens by per-token KV size for GB, and note the curve models one global
cache shared by all traffic (see Methodology).

### "Our load test says the cache works great. Production disagrees."

A hand-written benchmark with uniform prompt groups inflates hit rates —
every group stays warm. Two honest alternatives:

```bash
# closest to production: replay the real structure at the real pace
llmtrafficlens export last-week.csv --to bailian -o trace.jsonl
aiperf profile --custom-dataset-type bailian_trace --input-file trace.jsonl --fixed-schedule

# open-loop rate sweeps: fit a generator, but obey the fidelity report
llmtrafficlens fit last-week.csv --target sglang-gsp
# small gap (illustrative: +0.02) → generator is a faithful stand-in
# large gap (the bundled sample measures +0.30) → generator flattens
#   your traffic; use replay above
```

### "The vendor asks what our workload looks like. Legal says no."

`export` produces lengths, timestamps, anonymized session ids, and keyed
block-hash fingerprints — enough for the recipient to reproduce your
cache behavior and tune against it, while prompts are unrecoverable
(salted keyed hashing; the salt never leaves you). What they learn: your
request sizes, arrival pattern, and prefix-sharing structure. What they
cannot learn: a single word of any prompt.

## Verify the math (60 seconds, public data)

```bash
curl -LO https://raw.githubusercontent.com/kvcache-ai/Mooncake/main/FAST25-release/traces/conversation_trace.jsonl
llmtrafficlens validate conversation_trace.jsonl --format mooncake
# → ideal_hit_rate_upper_bound: 0.3738 on the Mooncake conversation trace
llmtrafficlens profile conversation_trace.jsonl --format mooncake -o mooncake-report
```

To cross-check, feed the same trace to the official [KV Cache Hit Rate
Simulator](https://kvcache.ai/tools/kv-cache-hit-rate-simulator/) and
compare its infinite-capacity ceiling with the number above. Our 37.4% is
consistent with third-party analyses of this trace (a prefix-caching study
on the Mooncake traces reports ≈40% for the conversation workload —
[arXiv:2505.21919](https://arxiv.org/abs/2505.21919)); an automated
side-by-side check against the official simulator is on the roadmap and
has not been run yet.

## Privacy model

All processing is local. Reports contain aggregates only. Exports contain
lengths, timestamps, anonymized session ids, and keyed block hashes —
prompts are unrecoverable. The hash key is generated at `.ltl-salt`
(keep it stable across runs, treat it as a secret).

## Methodology and provenance

Every metric follows a published convention; deviations are listed, not
hidden:

- **Block hashing** — chained per-block hashing (`hash(parent, block)`,
  16-token blocks) follows the vLLM prefix-cache block scheme and the
  [qwen-bailian trace](https://github.com/alibaba-edu/qwen-bailian-usagetraces-anon)
  convention. *Deviation*: in approximate mode we hash fixed-size
  character chunks, not tokenizer output; two prompts match only if
  byte-identical. Exact-tokenizer mode is planned.
- **Ideal hit rate upper bound** — the construction defined by
  [KVCache in the Wild (ATC'25)](https://arxiv.org/abs/2506.02634) and the
  Mooncake simulator's infinite-capacity ceiling: replay requests in
  timestamp order, a block hits if any earlier request produced it.
  It ignores in-flight timing (a block is reusable the instant it first
  appears), which keeps it a true upper bound.
- **LRU capacity curve** — same construction as the
  [Mooncake KV Cache Hit Rate Simulator](https://kvcache.ai/blog/calculate-kvcache-cache-budge/);
  LRU is the default eviction policy in vLLM and SGLang. The curve
  models a **single global cache** (one worker, or perfect routing): with
  N workers behind hash routing each sees a partition, so treat it as
  fleet-level guidance, not per-worker sizing. Remember the engine's KV
  pool also holds *active* requests — the reuse pool is what remains.
- **Zipf fit** — least-squares slope on log(count) vs log(rank); the
  standard quick estimator, not the stricter MLE (Clauset et al.).
- **Fidelity gap** — no bespoke statistic: we simulate the request stream
  the fitted generator would emit (per SGLang gsp semantics, group sizes
  at their zipf expectations) and score both the real and the simulated
  stream with the same ideal-hit-rate metric above. The gap is the
  difference between the two scores.

## Notes on precision

- Token counts use a chars/4 approximation unless the log carries `usage`
  (exact counts are then used and marked in the report). CJK-heavy text
  tokenizes closer to 1–2 chars/token: absolute counts skew low, ratios
  (ideal hit rate, leaderboard shares) stay usable.
- Without a timestamp column the profile still works, but arrival stats
  are omitted and exports are marked unfit for timestamped replay.
- The ideal hit rate upper bound covers *cross-request* sharing under infinite cache,
  perfect routing, no eviction. Real deployments land below it;
  intra-session reuse is only visible if the log carries session ids.

## Positioning

Observability platforms (Langfuse, Helicone, Datadog LLM) count tokens and
cost but never look inside the prompt structure. Trace analyzers (Mooncake
simulator, AIPerf `analyze-trace`) require pre-hashed traces. Capacity
simulators (Vidur, SplitwiseSim) consume traces nobody produces from
production logs (our survey: July 2026). `llmtrafficlens` is the missing pipeline: **raw gateway
logs → structural analysis → decision numbers → standard replayable
trace**.

## License

Apache-2.0
