Metadata-Version: 2.4
Name: llmtrafficlens
Version: 0.1.5
Summary: LLM gateway traffic profiler: prefix-reuse analysis, cache-hit-rate upper bounds, and anonymized replayable trace export
Project-URL: Homepage, https://github.com/GMISWE/llmtrafficlens
Author: Jason Zhu
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: benchmark,kv-cache,llm,mooncake,prefix-cache,trace,traffic-analysis,workload-characterization
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Benchmark
Requires-Python: >=3.10
Provides-Extra: tokenizer
Requires-Dist: tokenizers>=0.15; extra == 'tokenizer'
Description-Content-Type: text/markdown

# llmtrafficlens

Point it at your LLM gateway's request log. It tells you **how much a
prefix cache could save on your real traffic** — before you build anything —
and turns the log into an **anonymized dataset you can load-test with**.

```
                       ┌────────────────────────────────────────────┐
  gateway log (CSV) ──►│ llmtrafficlens                             │
  · request_json       │  tokenize → block-hash → prefix tree       │
  · response_json      │  (all local; prompts never leave the box)  │
  · timestamp, status  └──┬─────────────┬───────────────┬───────────┘
                          ▼             ▼               ▼
                     report.html   trace.jsonl     SGLang bench args
                     "worth it?"   "test with it"  "or synthesize it"
```

## Input → Output

| Command | Input | Output | The question it answers |
|---|---|---|---|
| `profile` | gateway log CSV | `report.html` + `.json` | Is my traffic worth prefix-aware routing? What's the ceiling? |
| `export` | gateway log CSV | anonymized `trace.jsonl` | How do I load-test / share this traffic without leaking a single prompt? |
| `fit` | gateway log CSV | SGLang bench CLI args | What generator settings reproduce my traffic shape? |
| `validate` | public trace | hit-rate numbers | Is this tool's math correct? (checked against the Mooncake simulator) |

**Input format**: a CSV with a `request_json` column holding the
OpenAI-format request body. Optional columns picked up automatically:
`response_json`, `model_name`, `status_code`, and a timestamp column
(`timestamp`/`ts`/`created_at`/... — epoch or ISO-8601; or set
`--ts-column`). Public Mooncake / qwen-bailian traces are also accepted
as input (`--format mooncake|bailian`).

## What you get

### 1. `profile` — the decision numbers

```bash
pip install llmtrafficlens
llmtrafficlens profile gateway.csv -o report
```

`report.html` shows, on *your* traffic:

- **ECHR upper bound** — the ceiling on prefix-cache savings: the share of
  input tokens that would hit under an infinite cache with perfect
  routing. If this is 3%, stop here; if it's 37%, keep reading.
- **LRU capacity curve** — hit rate at each cache size, so you can judge
  how close to the ceiling a real deployment can get.
- **Dual-metric prefix leaderboard** — top prefixes *by reuse count* vs
  *by reusable token volume*. The heads differ: a 15-token greeting reused
  870× is worthless to a cache; a 19K-token system prompt reused 95× is
  the whole prize. This tells you what your routing actually has to keep
  resident.
- **Session-identifier presence** — 0% means your reuse is cross-request
  shaped and session-affinity routing cannot capture it; prefix-aware
  routing is required.
- Input/output token distributions, streaming ratio, model mix, QPS.

### 2. `export` — an anonymized, replayable dataset

```bash
llmtrafficlens export gateway.csv --to bailian -o trace.jsonl
```

Each request becomes one line — **no text, only structure**:

```json
{"chat_id": "4abc41ba209004a0", "timestamp": 1753340000123,
 "input_length": 1994, "output_length": 117, "type": "glm-4", "turn": 1,
 "hash_ids": [4251731047194047120, 8125214104179782736, ...],
 "stream": true, "status": 200}
```

`hash_ids` is a salted chained block-hash fingerprint (qwen-bailian
convention, 16-token blocks): two requests share the first N hashes
exactly when they share their first N blocks of tokens. Replay tools
regenerate synthetic prompts from the hashes — same hash, same synthetic
block — so **cache behavior under replay matches production**, while the
content is fake. Replay with NVIDIA AIPerf:

```bash
aiperf profile --custom-dataset-type bailian_trace --input-file trace.jsonl ...
```

### 3. `fit` — generator settings instead of hand-tuning

```bash
llmtrafficlens fit gateway.csv --target sglang-gsp
```

Emits ready-to-run SGLang `bench_serving` arguments (group count, zipf
popularity, prompt lengths) fitted to your measured histogram, plus a
**fidelity report**: the theoretical hit rate the synthetic config
produces vs your measured ceiling, so you know when a generator is a
faithful stand-in and when you should replay the trace instead.

## Try it on the bundled sample (no data of your own needed)

The repository (not the pip package) ships a fully synthetic sample log —
360 requests, three shared system prompts with deliberately skewed
popularity, sparse session keys, real timestamps
(`examples/sample_gateway.csv`, regenerate with
`python examples/generate_sample.py`):

```bash
git clone https://github.com/GMISWE/llmtrafficlens && cd llmtrafficlens
llmtrafficlens profile examples/sample_gateway.csv -o sample-report
```

What it finds (and why each number matters):

| Metric | Value | Reading |
|---|---|---|
| ECHR upper bound | **66.4%** | worth building prefix-aware routing for |
| Top prefix by count | 16 tok × 180 reqs = 2.3% of reusable volume | popularity ≠ value: not worth pinning |
| Top prefix by volume | 2400 tok × 30 reqs = **55.6%** of reusable volume | this is what routing must keep resident |
| Session-id presence | 4.2% | reuse is cross-request shaped; affinity alone can't catch it |

And `fit` on the same sample demonstrates the fidelity report doing its
job — the gap is large, which is the tool telling you to replay the trace
instead of trusting a generator:

```bash
llmtrafficlens fit examples/sample_gateway.csv --target sglang-gsp
# → zipf_alpha 1.56, fidelity gap +0.30  ← generator can't represent
#   this mix of shared groups and singletons; use export + replay
```

## Scenarios

### "Should we build prefix-cache-aware routing?"

You run a multi-tenant gateway and someone proposes cache-aware routing.
Instead of arguing from intuition:

```bash
llmtrafficlens profile last-week.csv -o report
```

Three numbers settle it. **ECHR upper bound 4%** → decline the proposal
with evidence; the traffic has no shared structure worth routing for.
**Upper bound 40%, session-id presence 85%** → cheap session affinity
captures most of it; no prefix machinery needed. **Upper bound 40%,
session-id presence 0%** → reuse lives in cross-request shared prefixes;
only prefix-aware routing reaches it — fund the project, and attach the
report to the design doc as its quantitative basis.

### "How much cache memory does each worker need?"

The LRU capacity curve in the report shows hit rate at each cache size.
A typical shape: 8% at 64K tokens, 24% at 1M, 31% at 4M, ceiling 37%.
The knee (here ~1–4M tokens) is your sizing answer; provisioning past it
buys almost nothing. Multiply tokens by per-token KV size for GB.

### "Our load test says the cache works great. Production disagrees."

A hand-written benchmark with uniform prompt groups inflates hit rates —
every group stays warm. Two honest alternatives:

```bash
# closest to production: replay the real structure at the real pace
llmtrafficlens export last-week.csv --to bailian -o trace.jsonl
aiperf profile --custom-dataset-type bailian_trace --input-file trace.jsonl --fixed-schedule

# open-loop rate sweeps: fit a generator, but obey the fidelity report
llmtrafficlens fit last-week.csv --target sglang-gsp
# fidelity gap +0.02 → generator is a faithful stand-in, sweep away
# fidelity gap +0.30 → generator flattens your traffic; use replay above
```

### "The vendor asks what our workload looks like. Legal says no."

`export` produces lengths, timestamps, anonymized session ids, and keyed
block-hash fingerprints — enough for the recipient to reproduce your
cache behavior and tune against it, while prompts are unrecoverable
(salted keyed hashing; the salt never leaves you). What they learn: your
request sizes, arrival pattern, and prefix-sharing structure. What they
cannot learn: a single word of any prompt.

## Verify the math (60 seconds, public data)

```bash
curl -LO https://raw.githubusercontent.com/kvcache-ai/Mooncake/main/FAST25-release/traces/conversation_trace.jsonl
llmtrafficlens validate conversation_trace.jsonl --format mooncake
# → echr_upper_bound: 0.3738 on the Mooncake conversation trace
llmtrafficlens profile conversation_trace.jsonl --format mooncake -o mooncake-report
```

To cross-check, feed the same trace to the official [KV Cache Hit Rate
Simulator](https://kvcache.ai/tools/kv-cache-hit-rate-simulator/) and
compare its infinite-capacity ceiling with the number above. Our 37.4% is
consistent with third-party analyses of this trace (a prefix-caching study
on the Mooncake traces reports ≈40% for the conversation workload —
[arXiv:2505.21919](https://arxiv.org/abs/2505.21919)); an automated
side-by-side check against the official simulator is on the roadmap and
has not been run yet.

## Privacy model

All processing is local. Reports contain aggregates only. Exports contain
lengths, timestamps, anonymized session ids, and keyed block hashes —
prompts are unrecoverable. The hash key is generated at `.ltl-salt`
(keep it stable across runs, treat it as a secret).

## Methodology and provenance

Every metric follows a published convention; deviations are listed, not
hidden:

- **Block hashing** — chained per-block hashing (`hash(parent, block)`,
  16-token blocks) follows the vLLM prefix-cache block scheme and the
  [qwen-bailian trace](https://github.com/alibaba-edu/qwen-bailian-usagetraces-anon)
  convention. *Deviation*: in approximate mode we hash fixed-size
  character chunks, not tokenizer output; two prompts match only if
  byte-identical. Exact-tokenizer mode is planned.
- **ECHR upper bound** — the "ideal hit rate" construction used by
  [KVCache in the Wild (ATC'25)](https://arxiv.org/abs/2506.02634) and the
  Mooncake simulator's infinite-capacity ceiling: replay requests in
  timestamp order, a block hits if any earlier request produced it.
  It ignores in-flight timing (a block is reusable the instant it first
  appears), which keeps it a true upper bound.
- **LRU capacity curve** — same construction as the
  [Mooncake KV Cache Hit Rate Simulator](https://kvcache.ai/blog/calculate-kvcache-cache-budge/);
  LRU is the eviction policy vLLM and SGLang actually use. The curve
  models a **single global cache** (one worker, or perfect routing): with
  N workers behind hash routing each sees a partition, so treat it as
  fleet-level guidance, not per-worker sizing. Remember the engine's KV
  pool also holds *active* requests — the reuse pool is what remains.
- **Zipf fit** — least-squares slope on log(count) vs log(rank); the
  standard quick estimator, not the stricter MLE (Clauset et al.).
- **Fidelity gap** — our own definition (no industry standard exists):
  theoretical hit share of the fitted generator config vs the measured
  upper bound.

## Notes on precision

- Token counts use a chars/4 approximation unless the log carries `usage`
  (exact counts are then used and marked in the report). CJK-heavy text
  tokenizes closer to 1–2 chars/token: absolute counts skew low, ratios
  (ECHR, leaderboard shares) stay usable.
- Without a timestamp column the profile still works, but arrival stats
  are omitted and exports are marked unfit for timestamped replay.
- ECHR upper bound covers *cross-request* sharing under infinite cache,
  perfect routing, no eviction. Real deployments land below it;
  intra-session reuse is only visible if the log carries session ids.

## Positioning

Observability platforms (Langfuse, Helicone, Datadog LLM) count tokens and
cost but never look inside the prompt structure. Trace analyzers (Mooncake
simulator, AIPerf `analyze-trace`) require pre-hashed traces. Capacity
simulators (Vidur, SplitwiseSim) consume traces nobody produces from
production logs. `llmtrafficlens` is the missing pipeline: **raw gateway
logs → structural analysis → decision numbers → standard replayable
trace**.

## License

Apache-2.0
