Metadata-Version: 2.4
Name: llmtrafficlens
Version: 0.1.0
Summary: LLM gateway traffic profiler: prefix-reuse analysis, cache-hit-rate upper bounds, and anonymized replayable trace export
Project-URL: Homepage, https://github.com/Belyenochi/llmtrafficlens
Author: Jason Zhu
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: benchmark,kv-cache,llm,mooncake,prefix-cache,trace,traffic-analysis,workload-characterization
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Benchmark
Requires-Python: >=3.10
Provides-Extra: tokenizer
Requires-Dist: tokenizers>=0.15; extra == 'tokenizer'
Description-Content-Type: text/markdown

# llmtrafficlens

**LLM gateway traffic profiler** — analyze prefix-reuse structure in your
gateway logs, estimate the prefix-cache hit-rate ceiling *before* you build
KV-cache-aware routing, and export anonymized replayable traces in
community-standard formats.

Every KV-cache product tells you how much you saved *after* deployment.
`llmtrafficlens` answers the prior question: **is your traffic worth it, and
what is the upper bound?**

## What it does

1. **Traffic profile** — input/output token distributions, streaming ratio,
   model mix, error rates.
2. **Prefix structure** — distinct prefixes, shared-prefix groups, and a
   **dual-metric leaderboard**: top prefixes *by reuse count* vs *by reusable
   token volume*. The two heads routinely differ: a 15-token prefix reused
   870× is worthless to a cache, a 19K-token prefix reused 95× dominates it.
3. **Session-shape detection** — presence rate of session identifiers
   (`prompt_cache_key`, `metadata.*`): tells you whether reuse is
   *cross-request shared-prefix* shaped (needs prefix-aware routing) or
   *intra-session* shaped (session affinity suffices).
4. **Reuse potential** — ECHR (engine cache hit rate) upper bound under
   infinite cache + perfect routing, plus an LRU capacity/hit-rate curve.
5. **Anonymized export** — qwen-bailian-superset or Mooncake JSONL (salted
   chained block hashes; no text leaves your machine), directly replayable
   by NVIDIA AIPerf, SGLang bench_serving, and AIBrix.
6. **Generator fitting** — emit SGLang `generated-shared-prefix` CLI args
   (zipf popularity fitted to your histogram) with a fidelity report.

## Install

```bash
pip install llmtrafficlens
```

Zero runtime dependencies. Python ≥ 3.10.

## Quickstart on a public trace

Validate the analyzer against the public [Mooncake
trace](https://github.com/kvcache-ai/Mooncake) (FAST'25) — the result should
match the official [KV Cache Hit Rate
Simulator](https://kvcache.ai/tools/kv-cache-hit-rate-simulator/)'s
infinite-capacity ceiling:

```bash
curl -LO https://raw.githubusercontent.com/kvcache-ai/Mooncake/main/FAST25-release/traces/conversation_trace.jsonl
llmtrafficlens validate conversation_trace.jsonl --format mooncake
llmtrafficlens profile  conversation_trace.jsonl --format mooncake -o mooncake-report
```

## Profile your own gateway logs

Input: a CSV with a `request_json` column holding the OpenAI-format request
body (optional `response_json`, `model_name`, `status_code` columns are
picked up when present).

```bash
llmtrafficlens profile gateway.csv -o report
# -> report.json + report.html
```

All processing is local. The report contains aggregates only — raw prompts
never appear in any output.

Token counts use a chars/4 approximation unless the log carries `usage`
(then exact counts are used and marked in the report). A keyed hash salt is
generated at `.ltl-salt` on first run; keep it stable to make runs and
exports comparable, and treat it as a secret.

## Export a replayable anonymized trace

```bash
# primary: bailian superset (16-token blocks, session structure, stream/status)
llmtrafficlens export gateway.csv --to bailian -o trace.jsonl

# downgrade: Mooncake four-field format
llmtrafficlens export gateway.csv --to mooncake -o mooncake.jsonl
```

Replay examples:

```bash
# NVIDIA AIPerf (timestamped replay)
aiperf profile --custom-dataset-type mooncake_trace --input-file mooncake.jsonl --fixed-schedule ...

# SGLang bench_serving
python -m sglang.bench_serving --dataset-name mooncake --dataset-path mooncake.jsonl ...
```

## Fit a load generator to your traffic

```bash
llmtrafficlens fit gateway.csv --target sglang-gsp
```

Emits `--gsp-*` arguments (group count, zipf alpha, prompt lengths) plus a
**fidelity report**: the theoretical ECHR the synthetic config would
produce vs your measured upper bound, so the flattening loss is explicit.

## Reading the numbers

- **ECHR upper bound** counts *cross-request* shared prefixes only, under
  infinite cache / perfect routing / no eviction. Real deployments land
  below it; intra-session reuse is invisible unless your sample carries
  session identifiers.
- **Session identifier presence 0%** means session-affinity routing cannot
  capture any of the observed reuse — a prefix-aware policy is required.
- Block size defaults to 16 tokens (qwen-bailian convention; finer and
  engine-page aligned). Mooncake input implies 512.

## Positioning

Observability platforms (Langfuse, Helicone, Datadog LLM) stop at
token/cost counting. Trace analyzers (Mooncake simulator, AIPerf
`analyze-trace`) require pre-hashed traces. Simulators (Vidur,
SplitwiseSim) consume traces nobody produces from production logs.
`llmtrafficlens` covers the missing pipeline: **raw gateway logs →
structural analysis → decision numbers → standard replayable trace**.

## License

Apache-2.0
