Metadata-Version: 2.4
Name: llmtrafficlens
Version: 0.1.1
Summary: LLM gateway traffic profiler: prefix-reuse analysis, cache-hit-rate upper bounds, and anonymized replayable trace export
Project-URL: Homepage, https://github.com/GMISWE/llmtrafficlens
Author: Jason Zhu
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: benchmark,kv-cache,llm,mooncake,prefix-cache,trace,traffic-analysis,workload-characterization
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Benchmark
Requires-Python: >=3.10
Provides-Extra: tokenizer
Requires-Dist: tokenizers>=0.15; extra == 'tokenizer'
Description-Content-Type: text/markdown

# llmtrafficlens

**LLM gateway traffic profiler** — analyze prefix-reuse structure in your
gateway logs, estimate the prefix-cache hit-rate ceiling *before* you build
KV-cache-aware routing, and export anonymized replayable traces in
community-standard formats.

Every KV-cache product tells you how much you saved *after* deployment.
`llmtrafficlens` answers the prior question: **is your traffic worth it, and
what is the upper bound?**

## What it does

1. **Traffic profile** — input/output token distributions, streaming ratio,
   model mix, error rates.
2. **Prefix structure** — distinct prefixes, shared-prefix groups, and a
   **dual-metric leaderboard**: top prefixes *by reuse count* vs *by reusable
   token volume*. The two heads routinely differ: a 15-token prefix reused
   870× is worthless to a cache, a 19K-token prefix reused 95× dominates it.
3. **Session-shape detection** — presence rate of session identifiers
   (`prompt_cache_key`, `metadata.*`): tells you whether reuse is
   *cross-request shared-prefix* shaped (needs prefix-aware routing) or
   *intra-session* shaped (session affinity suffices).
4. **Reuse potential** — ECHR (engine cache hit rate) upper bound under
   infinite cache + perfect routing, plus an LRU capacity/hit-rate curve.
5. **Anonymized export** — qwen-bailian-superset JSONL (salted chained
   block hashes; no text leaves your machine), replayable with NVIDIA
   AIPerf's `bailian_trace` mode. A Mooncake downgrade export is planned.
6. **Generator fitting** — emit SGLang `generated-shared-prefix` CLI args
   (zipf popularity fitted to your histogram) with a fidelity report.

## Install

```bash
pip install llmtrafficlens
```

Zero runtime dependencies. Python ≥ 3.10.

## Quickstart on a public trace

Validate the analyzer against the public [Mooncake
trace](https://github.com/kvcache-ai/Mooncake) (FAST'25) — the result should
match the official [KV Cache Hit Rate
Simulator](https://kvcache.ai/tools/kv-cache-hit-rate-simulator/)'s
infinite-capacity ceiling:

```bash
curl -LO https://raw.githubusercontent.com/kvcache-ai/Mooncake/main/FAST25-release/traces/conversation_trace.jsonl
llmtrafficlens validate conversation_trace.jsonl --format mooncake
llmtrafficlens profile  conversation_trace.jsonl --format mooncake -o mooncake-report
```

## Profile your own gateway logs

Input: a CSV with a `request_json` column holding the OpenAI-format request
body (optional `response_json`, `model_name`, `status_code` columns are
picked up when present).

```bash
llmtrafficlens profile gateway.csv -o report
# -> report.json + report.html
```

All processing is local. The report contains aggregates only — raw prompts
never appear in any output.

Token counts use a chars/4 approximation unless the log carries `usage`
(then exact counts are used and marked in the report). The chars/4 rule is
tuned for English-dominant text; CJK-heavy traffic tokenizes closer to 1–2
chars per token, so absolute token counts skew low while ratios (ECHR,
leaderboard shares) stay usable. A keyed hash salt is generated at
`.ltl-salt` on first run; keep it stable to make runs and exports
comparable, and treat it as a secret.

Timestamps are auto-detected from common column names (`timestamp`, `ts`,
`created_at`, ...; epoch seconds/ms or ISO-8601) or set explicitly with
`--ts-column`. Without one, row order stands in: the profile still works,
but arrival stats are omitted and exports are marked unfit for
timestamped replay.

## Export a replayable anonymized trace

```bash
# bailian superset: 16-token blocks, session structure, stream/status
llmtrafficlens export gateway.csv --to bailian -o trace.jsonl
```

Replay with NVIDIA AIPerf (`--fixed-schedule` requires real timestamps in
the source log; the export warns when timestamps are synthetic):

```bash
aiperf profile --custom-dataset-type bailian_trace --input-file trace.jsonl ...
```

## Fit a load generator to your traffic

```bash
llmtrafficlens fit gateway.csv --target sglang-gsp
```

Emits `--gsp-*` arguments (group count, zipf alpha, prompt lengths) plus a
**fidelity report**: the theoretical ECHR the synthetic config would
produce vs your measured upper bound, so the flattening loss is explicit.

## Reading the numbers

- **ECHR upper bound** counts *cross-request* shared prefixes only, under
  infinite cache / perfect routing / no eviction. Real deployments land
  below it; intra-session reuse is invisible unless your sample carries
  session identifiers.
- **Session identifier presence 0%** means session-affinity routing cannot
  capture any of the observed reuse — a prefix-aware policy is required.
- Block size defaults to 16 tokens (qwen-bailian convention; finer and
  engine-page aligned). Mooncake input implies 512.

## Positioning

Observability platforms (Langfuse, Helicone, Datadog LLM) stop at
token/cost counting. Trace analyzers (Mooncake simulator, AIPerf
`analyze-trace`) require pre-hashed traces. Simulators (Vidur,
SplitwiseSim) consume traces nobody produces from production logs.
`llmtrafficlens` covers the missing pipeline: **raw gateway logs →
structural analysis → decision numbers → standard replayable trace**.

## License

Apache-2.0
