Metadata-Version: 2.4
Name: llmtrafficlens
Version: 0.3.1
Summary: Profile LLM gateway logs for prefix-cache reuse potential, and export anonymized replayable traces
Project-URL: Homepage, https://github.com/GMISWE/llmtrafficlens
Author: Jason Zhu
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: benchmark,kv-cache,llm,mooncake,prefix-cache,trace,traffic-analysis,workload-characterization
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Benchmark
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# llmtrafficlens

Point it at your LLM gateway's request log. It answers **how much a prefix
cache could save on your real traffic** — before you build anything — and
turns the log into an **anonymized trace the standard benchmark tools can
replay**.

Two commands. Everything downstream of the trace is left to the tools that
already do it well.

```
  gateway log CSV ──► llmtrafficlens profile ──► report.html / .json
   · request_json                                 decision numbers
   · timestamp, status
                  └──► llmtrafficlens export  ──► trace.jsonl
                                                  AIPerf · SGLang · AIBrix
                                                  · Mooncake simulator
```

## Install

```bash
pip install llmtrafficlens
```

Zero runtime dependencies. Python ≥ 3.10.

## `profile` — the decision numbers

```bash
llmtrafficlens profile gateway.csv -o report
```

**Input**: a CSV with a `request_json` column holding the OpenAI-format
request body. Picked up automatically when present: `response_json`,
`model_name`, `status_code`, and a timestamp column (`timestamp`, `ts`,
`created_at`, ...; epoch or ISO-8601 — or name it with `--ts-column`).
Public Mooncake and qwen-bailian traces also work as input
(`--format mooncake|bailian`).

**Output** (`report.html` + `report.json`):

- **Hit rate by cache size** — one LRU simulation per capacity, ending
  with the unbounded case: your **ceiling**. If the ceiling is 3%, stop
  here. If it's 37%, the curve tells you how much cache memory it takes to
  approach it.
- **Dual-metric prefix leaderboard** — top prefixes *by reuse count* vs
  *by reusable token volume*. The heads differ, and only the second one
  matters: a 15-token greeting reused 870× is worthless to a cache; a
  19K-token system prompt reused 95× is the whole prize.
- **Session-identifier presence** — 0% means reuse is cross-request
  shaped, so session-affinity routing cannot capture it and prefix-aware
  routing is required.
- Input/output token distributions, streaming ratio, model mix, QPS.

## `export` — an anonymized, replayable trace

```bash
llmtrafficlens export gateway.csv --to mooncake -o trace.jsonl   # or --to bailian
```

Each request becomes one line — **no text, only structure**:

```json
{"timestamp": 1753340000123, "input_length": 1994, "output_length": 117,
 "hash_ids": [4251731047194047120, 8125214104179782736, ...]}
```

`hash_ids` is a salted chained block-hash fingerprint: two requests share
the first N hashes exactly when they share their first N blocks. Replay
tools regenerate synthetic prompts from the hashes — same hash, same
synthetic block — so the **prefix-sharing structure** of production is
reproduced while the content is fake.

Then hand it to the tools built for replay:

```bash
# faithful replay at the recorded pace
aiperf profile --custom-dataset-type mooncake_trace --input-file trace.jsonl --fixed-schedule ...
# same trace, 2× the pace (rate sweeps without leaving replay)
aiperf profile --custom-dataset-type mooncake_trace --input-file trace.jsonl --synthesis-speedup-ratio 2.0 ...
# or SGLang
python -m sglang.bench_serving --dataset-name mooncake --dataset-path trace.jsonl ...
```

`--to bailian` adds `chat_id` / `parent_chat_id` / `turn` for AIPerf's
`bailian_trace` mode; use it when your log carries session identifiers.
Tell the consumer which block size you exported with — AIPerf defaults to
512 for mooncake and 16 for bailian (`--prompt-input-tokens-block-size`).

Untested claim, stated plainly: we have not round-tripped an export
through AIPerf or SGLang ourselves. The formats match their documented
schemas; actual replay verification is on the roadmap.

## Try it on a public trace

No data of your own required, and it doubles as a correctness check:

```bash
curl -LO https://raw.githubusercontent.com/kvcache-ai/Mooncake/main/FAST25-release/traces/conversation_trace.jsonl
llmtrafficlens profile conversation_trace.jsonl --format mooncake -o mooncake-report
# → ceiling 0.3738 over 12,031 requests
```

Feed the same trace to the official [KV Cache Hit Rate
Simulator](https://kvcache.ai/tools/kv-cache-hit-rate-simulator/) and
compare its infinite-capacity ceiling. Our 37.4% is consistent with
third-party analysis of this trace (a prefix-caching study reports ≈40%
for the conversation workload,
[arXiv:2505.21919](https://arxiv.org/abs/2505.21919)); an automated
side-by-side check against the official simulator has not been run yet.

A second, fully synthetic sample lives in the source repository (not in
the pip package): 360 requests, three shared system prompts with
deliberately skewed popularity, sparse session keys, real timestamps.
`llmtrafficlens profile examples/sample_gateway.csv` reports a 66.4%
ceiling and, more interestingly, a leaderboard whose two heads disagree —
16 tokens × 180 requests is 2.3% of the reusable volume, while 2400
tokens × 30 requests is 55.6% of it. Regenerate it with
`python examples/generate_sample.py`.

## Scenarios

**"Should we build prefix-cache-aware routing?"** Run `profile` on last
week's log. Ceiling 4% → decline the proposal with evidence. Ceiling 40%
with 85% session-id presence → cheap session affinity captures most of it.
Ceiling 40% with 0% presence → reuse lives in cross-request shared
prefixes and only prefix-aware routing reaches it; attach the report to
the design doc as its quantitative basis.

**"How much cache memory per worker?"** Read the curve. On the public
Mooncake conversation trace it runs 4.2% at 64K tokens, 5.6% at 1M, 18.5%
at 4M, 27.1% at 8M, 34.2% at 16M, against a 37.4% ceiling — that workload
only takes off past ~2M tokens, so a small cache buys almost nothing.
Multiply tokens by per-token KV size for GB, and mind the caveats in
Methodology: the curve models one global cache, and the engine's KV pool
also holds in-flight requests.

**"Our load test says the cache works great; production disagrees."**
Hand-written benchmarks with uniform prompt groups inflate hit rates —
every group stays warm. Export your real trace and replay that instead;
`--synthesis-speedup-ratio` covers rate sweeps.

**"A vendor wants to see our workload; legal says no."** The export
carries lengths, timestamps, and keyed block hashes. They can reproduce
your cache behavior and tune against it; they cannot recover a word of any
prompt.

## Methodology

Every number follows a published construction; deviations are listed, not
hidden.

- **Hit rate** — replay requests in timestamp order against an LRU block
  cache. Unbounded capacity is the "ideal hit rate" of
  [KVCache in the Wild (ATC'25)](https://arxiv.org/abs/2506.02634) and the
  Mooncake simulator's infinite-capacity ceiling; bounded capacities give
  the [Mooncake simulator's capacity
  curve](https://kvcache.ai/blog/calculate-kvcache-cache-budge/). LRU is
  the default eviction policy in vLLM and SGLang. The simulation models a
  **single global cache** (one worker, or perfect routing): with N workers
  behind hash routing each sees a partition, so treat it as fleet-level
  guidance. The engine's KV pool also holds in-flight requests — the reuse
  pool is what remains.
- **Block hashing** — chained per-block hashing (`hash(parent, block)`,
  16-token blocks) follows the vLLM prefix-cache block scheme and the
  [qwen-bailian trace](https://github.com/alibaba-edu/qwen-bailian-usagetraces-anon)
  convention. *Deviation*: in approximate mode we hash fixed-size
  character chunks, not tokenizer output, so two prompts match only if
  byte-identical.
- **Token counts** — chars/4 unless the log carries `usage` (exact counts
  are then used and marked in the report). CJK-heavy text tokenizes closer
  to 1–2 chars/token: absolute counts skew low, ratios stay usable.
- **Coverage** — the ceiling covers *cross-request* sharing only.
  Intra-session reuse is invisible unless the log carries session ids.
  Without a timestamp column, row order stands in: arrival stats are
  omitted and exports are flagged unfit for fixed-schedule replay.

## Privacy

All processing is local. Reports contain aggregates only. Exports contain
lengths, timestamps, anonymized session ids, and keyed block hashes. The
hash key is generated at `.ltl-salt` — keep it stable across runs and
treat it as a secret.

## Scope

Observability platforms (Langfuse, Helicone, Datadog LLM) count tokens and
cost but never look inside prompt structure. Trace analyzers (Mooncake
simulator, AIPerf `analyze-trace`) require pre-hashed traces. Load
generators and capacity simulators (AIPerf, SGLang, Vidur) consume traces
nobody produces from production logs (our survey: July 2026).
`llmtrafficlens` fills exactly that gap and stops there: **raw gateway
logs → structure → decision numbers → a standard trace**. Replay, rate
control, and deployment search belong to the tools above.

## License

Apache-2.0
