Metadata-Version: 2.4
Name: llmtrafficlens
Version: 0.3.5
Summary: Profile LLM gateway logs for prefix-cache reuse potential, and export anonymized replayable traces
Project-URL: Homepage, https://github.com/GMISWE/llmtrafficlens
Author: Jason Zhu
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: benchmark,kv-cache,llm,mooncake,prefix-cache,trace,traffic-analysis,workload-characterization
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Benchmark
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# llmtrafficlens

Analyze an LLM gateway's request log for prefix-cache reuse, then export
the log as an anonymized trace that standard benchmark tools can replay.

Two commands:

- **`profile`** — how much a prefix cache could save on this traffic, and
  how much cache memory that takes.
- **`export`** — the same log as a Mooncake- or bailian-format trace, with
  all text removed.

## Install

```bash
pip install llmtrafficlens
```

No runtime dependencies. Python ≥ 3.10.

## profile

```bash
llmtrafficlens profile gateway.csv -o report
```

**Input.** A CSV with a `request_json` column holding the OpenAI-format
request body. These columns are used when present: `response_json`,
`model_name`, `status_code`, and a timestamp column (`timestamp`, `ts`,
`created_at`, ...; epoch or ISO-8601, or set `--ts-column`). Mooncake and
qwen-bailian traces are accepted as input too
(`--format mooncake|bailian`).

**Output.** `report.html` and `report.json`, containing:

- hit rate at each cache size, ending with the unbounded case — the
  ceiling;
- top prefixes by reuse count, and separately by reusable token volume;
- share of requests carrying a session identifier;
- input/output token distributions, streaming ratio, model mix, QPS.

**How to read it.** The ceiling bounds what prefix-aware routing can
achieve on this traffic. A few percent means the work will not pay off.

The session-identifier share says whether session affinity alone would
reach that ceiling. If most requests carry an identifier, affinity
suffices. If none do, the reuse sits in prefixes shared between unrelated
requests, and only prefix-aware routing can exploit it.

The two leaderboards rank differently, and the token-volume one determines
savings: a short prefix reused very often contributes almost no reusable
volume, while a long prefix reused a few dozen times can account for most
of it.

The capacity curve is the input to cache sizing: each row is the hit rate
if the engine can keep that many tokens of KV resident. Read where the
curve flattens toward the ceiling, then convert tokens to bytes as shown
in the next section. Two limits apply to that reading; both are in
Methodology.

## export

```bash
llmtrafficlens export gateway.csv --to mooncake -o trace.jsonl   # or --to bailian
```

A benchmark written by hand, with equally popular prompt groups, reports a
higher hit rate than production reaches, because every group stays warm.
Replaying the real log avoids that, and the export can be shared, because
each request is reduced to one line of structure:

```json
{"timestamp": 1753340000123, "input_length": 1994, "output_length": 117,
 "hash_ids": [4251731047194047120, 8125214104179782736, ...]}
```

`hash_ids` is a salted chained block hash: two requests share their first
N hashes exactly when they share their first N blocks. Replay tools
generate one synthetic block per hash, so the prefix-sharing structure is
preserved while the content is not real.

```bash
# replay at the recorded pace
aiperf profile --custom-dataset-type mooncake_trace --input-file trace.jsonl --fixed-schedule ...
# replay at 2x the pace, for rate sweeps
aiperf profile --custom-dataset-type mooncake_trace --input-file trace.jsonl --synthesis-speedup-ratio 2.0 ...
# or SGLang
python -m sglang.bench_serving --dataset-name mooncake --dataset-path trace.jsonl ...
```

`--to bailian` adds `chat_id`, `parent_chat_id` and `turn` for AIPerf's
`bailian_trace` mode; use it when the log carries session identifiers.
Tell the consumer which block size the export used — AIPerf defaults to
512 for mooncake and 16 for bailian (`--prompt-input-tokens-block-size`).

We have not round-tripped an export through AIPerf or SGLang yet. The
formats match their documented schemas, but replay is unverified.

## Try it on a public trace

This needs no data of your own, and doubles as a correctness check:

```bash
curl -LO https://raw.githubusercontent.com/kvcache-ai/Mooncake/main/FAST25-release/traces/conversation_trace.jsonl
llmtrafficlens profile conversation_trace.jsonl --format mooncake -o mooncake-report
```

The curve over those 12,031 requests:

| cache size (tokens) | hit rate |
|---|---|
| 16K | 3.3% |
| 64K | 4.2% |
| 256K | 4.3% |
| 1M | 5.6% |
| 4M | 18.5% |
| 16M | 34.2% |
| unbounded | **37.4%** (ceiling) |

Cache size is how many tokens' worth of KV blocks the engine can keep
resident at once; a block is evicted (LRU) once the cache is full. The
unit is tokens because bytes depend on the model: multiply by
`layers × kv_heads × head_dim × 2 (K and V) × dtype_bytes`, which is
128 KB/token for a 32-layer, 8-KV-head, 128-dim model in fp16. At that
rate the 4M row is about 512 GB.

So this workload gains almost nothing below 1M tokens, and reaching most
of its ceiling takes KV storage in the terabytes — more than device memory
alone provides, which is the case tiered KV stores (host memory, SSD) are
built for.

To cross-check, feed the same trace to the official [KV Cache Hit Rate
Simulator](https://kvcache.ai/tools/kv-cache-hit-rate-simulator/) and
compare its infinite-capacity ceiling. Our figure agrees with third-party
analysis of this trace (a prefix-caching study reports ≈40% for the
conversation workload,
[arXiv:2505.21919](https://arxiv.org/abs/2505.21919)). We have not run an
automated comparison against the simulator itself.

The repository also contains a synthetic sample log (`examples/`, not
shipped in the pip package): 360 requests, three shared system prompts
with skewed popularity, sparse session identifiers, real timestamps. It
reports a 66.4% ceiling, and it shows the two leaderboards disagreeing —
16 tokens × 180 requests is 2.3% of the reusable volume, while 2400
tokens × 30 requests is 55.6% of it.

## Methodology

Each metric follows a published construction. Deviations are listed.

- **Hit rate** — requests are replayed in timestamp order against an LRU
  block cache. Unbounded capacity gives the "ideal hit rate" of
  [KVCache in the Wild (ATC'25)](https://arxiv.org/abs/2506.02634), which
  is also the Mooncake simulator's infinite-capacity ceiling; bounded
  capacities give that simulator's [capacity
  curve](https://kvcache.ai/blog/calculate-kvcache-cache-budge/). LRU is
  the default eviction policy in vLLM and SGLang.
  - The simulation models one global cache, i.e. a single worker or
    perfect routing. With N workers behind hash routing, each sees a
    partition, so read the curve as fleet-level guidance.
  - An engine's KV pool also holds in-flight requests. Only the remainder
    is available for reuse.
- **Block hashing** — chained per-block hashing (`hash(parent, block)`,
  16-token blocks) follows the vLLM prefix-cache block scheme and the
  [qwen-bailian trace](https://github.com/alibaba-edu/qwen-bailian-usagetraces-anon)
  convention. Deviation: in approximate mode we hash fixed-size character
  chunks rather than tokenizer output, so two prompts match only if they
  are byte-identical.
- **Token counts** — characters divided by four, unless the log carries
  `usage`, in which case exact counts are used and the report says so.
  CJK-heavy text tokenizes closer to 1–2 characters per token, so absolute
  counts run low while ratios remain usable.
- **Coverage** — the ceiling covers sharing between requests only.
  Intra-session reuse is not measurable without session identifiers in the
  log. Without a timestamp column, row order is used instead: arrival
  statistics are omitted and the export is flagged as unfit for
  fixed-schedule replay.

## Privacy

Processing is local. Reports contain aggregates only. An export contains
lengths, timestamps, anonymized session identifiers and salted block
hashes; prompts cannot be reconstructed from it. The hash key is written
to `.ltl-salt` on first run — keep it unchanged so runs stay comparable,
and treat it as a secret.

## Related tools and scope

We do not replay traffic, control request rate, or search deployment
configurations. AIPerf, SGLang `bench_serving` and simulators such as
Vidur do that, and all of them take a trace as input.

Observability platforms (Langfuse, Helicone, Datadog LLM Observability)
record token counts and cost per request, not prefix structure. Trace
analyzers such as the Mooncake simulator and `aiperf analyze-trace`
require an already-hashed trace. As of July 2026 we found no tool that
goes from a raw gateway log to a hashed trace, which is the step this
fills.

## License

Apache-2.0
