Metadata-Version: 2.4
Name: fbc-cache-lint
Version: 0.1.0
Summary: GPU-free, cross-engine inference-cache linter for vLLM /metrics and llama.cpp slot logs
Author: Daniel Stanley
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/the10kclub/fbc
Project-URL: Repository, https://github.com/the10kclub/fbc
Project-URL: Bug Tracker, https://github.com/the10kclub/fbc/issues
Project-URL: Codec benchmark, https://github.com/the10kclub/fbc/blob/main/bench/README.md
Keywords: vllm,llama.cpp,kv-cache,prefix-cache,inference,observability,linter,llm,prometheus,cache
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Topic :: System :: Monitoring
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Provides-Extra: codec
Requires-Dist: numpy; extra == "codec"
Requires-Dist: zstandard; extra == "codec"
Requires-Dist: lz4; extra == "codec"
Requires-Dist: pyarrow; extra == "codec"
Requires-Dist: xxhash; extra == "codec"
Requires-Dist: constriction; extra == "codec"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: numpy; extra == "dev"
Requires-Dist: zstandard; extra == "dev"
Requires-Dist: lz4; extra == "dev"
Requires-Dist: pyarrow; extra == "dev"
Requires-Dist: xxhash; extra == "dev"
Requires-Dist: constriction; extra == "dev"
Dynamic: license-file

# fbc-cache-lint

**Lint your LLM inference cache. Find the money it's leaking.**

`fbc-cache-lint` is a GPU-free, cross-engine linter for inference-cache telemetry.
Point it at a recorded **vLLM `/metrics`** dump or a **llama.cpp `llama-server`** slot
log and it prints ranked, opinionated findings — each quantified in wasted tokens / KV
bytes / dollars and paired with a concrete remediation naming the exact flag to change.
It reads *recorded* dumps: no GPU, no live server, nothing to attach to production.

Metrics dashboards tell you the hit rate is 55%. cache-lint tells you *why it fell from
80%, what it's costing you, and which flag to flip.*

## Install

```bash
pipx install fbc-cache-lint      # isolated CLI
# or
pip install fbc-cache-lint
```

Requires Python 3.11+. **Zero third-party dependencies** — the linter is pure standard
library, so the install is small and fast. That's the whole thing; no clone needed.

## Quickstart — run it now, on real data

The package ships **two real** `llama-server` captures (phi-3-mini Q4\_K\_M, build 8227,
same tool/model/machine) so you can see it work with no dump of your own. Start with the
**healthy** run:

```bash
fbc-cache-lint demo
```

```text
cache-lint report — llama_server_slots.REAL.phi3-mini.log
source: llama.cpp slot log (251 events)
trace: 27 requests, 83.6% prefix reuse (4,172/4,989 prompt tokens), 2 cold starts

No cache findings above the configured thresholds. OK.

----------------------------------------------------------------------------
Costs are estimates: $0.05/1k prefill tokens, 131,072 B/token KV,
512 recompute tokens/preemption (override with --price-* flags).
```

That is a **healthy** run — 83.6% of prompt tokens served straight from the KV cache,
sub-2 s decodes, no eviction churn — and the linter says so, with numbers. (A linter that
only ever shouts is noise; confirming a cache *is* healthy is half the job.)

Now the **same tool, same model, same machine — deliberately misconfigured** (prefix
caching disabled + an oversized generation cap on ~1.1k-token prompts) so there is a real
problem to catch. It is a *real* run, not a hand-authored fixture; the report says so:

```bash
fbc-cache-lint demo misconfig --fail-on medium   # exit 1 so CI can gate on it
```

```text
cache-lint report — llama_server_slots.REAL.INDUCED-MISCONFIG.phi3-mini.log
source: llama.cpp slot log (55 events)
NOTE: REAL run, DELIBERATELY MISCONFIGURED to demonstrate detections — not a found-in-the-wild trace.
trace: 6 requests, 0.0% prefix reuse (0/6,746 prompt tokens), 6 cold starts
1 finding  |  843.2 MiB KV

[1] MEDIUM long_decode_kv_pin  (llama.cpp)
    Stranded/pinned blocks from long-decode holds
    impact: 843.2 MiB KV
    6 slot(s) held a large KV prefix through a long decode: worst was slot 0
    — 1,126-token prefix pinned for 25s of decode. 843 MiB of KV pinned
    across these holds.
    fix: Long decodes hold their prompt KV non-evictable and can starve
         other requests. Cap generation length per request (`--n-predict` /
         `n_predict`), reduce `--parallel` so one long decode does not
         monopolise the cache, or offload idle slot KV with `--slot-save-
         path` + the `/slots/{id}/save` endpoint.
```

Two real runs, same linter: 84%-reuse healthy → *OK*; caching-off + uncapped generation →
a flagged, quantified problem naming the exact knob (`--n-predict`).

## What it finds

| Finding | What it catches | Remediation names | Needs |
|---|---|---|---|
| `prefix_cache_hit_rate_decay` | Hit rate sliding over time (scaling event, prompt-template drift) | session-affinity / stable shared prefix | vLLM `/metrics` series |
| `eviction_churn` | KV cache oversubscribed, preempting + recomputing running requests | `--gpu-memory-utilization`, `DYN_KVBM_CPU_CACHE_GB` | vLLM `/metrics` series |
| `route_utilisation_skew` | Round-robin routing giving ~1/N hit rate across replicas/tenants | sticky routing, shared prefix tier (LMCache) | vLLM `/metrics` (per-route) |
| `long_decode_kv_pin` | Long decodes pinning a big KV prefix, starving other requests | `--n-predict`, `--parallel`, slot offload | llama.cpp slot log |
| `prompt_order_breaks_prefix` | Shared content that isn't a *prefix*, so it can't be reused | move static content to the front | llama.cpp slot log (LCS) |

Findings are ranked by estimated cost (dollars, then wasted tokens, then KV bytes).

## Honest scope: which engine surfaces which finding

A finding can only fire on the telemetry that carries its signal:

- **A llama.cpp slot log** (like the two shipped demos) surfaces the llama.cpp-sourced
  findings — the long-decode KV pin above, and (on builds that log LCS token counts) the
  prompt-order break. It does **not** carry the counters the dollar-quantified findings
  read, so those stay silent on a slot log — a coverage boundary, not a clean bill.
- **A vLLM `/metrics` series** surfaces the three time-series findings — hit-rate decay,
  eviction churn, and per-route skew, the ones quantified in **dollars**. Diffing
  consecutive scrapes is how the windowed rates are computed, so record a series, not a
  single snapshot.

This dev box has no vLLM cluster to record a real `/metrics` series from, so those three
findings are demonstrated on a **labelled synthetic** dump (real metric names/types/labels,
hand-authored values). Its report says so in the footer:

```text
cache-lint report — vllm_metrics_scrapes.SYNTHETIC.prom
source: vLLM /metrics (5 scrapes)
2 findings  |  est. $11.77 wasted  |  235,324 tokens  |  28.7 GiB KV

[1] HIGH   eviction_churn  (vllm)     impact: $10.29 / 205,824 tokens / 25.1 GiB KV
[2] HIGH   prefix_cache_hit_rate_decay  (vllm)   impact: $1.48 / 29,500 tokens / 3.6 GiB KV
... footer: NOTE: SYNTHETIC demo data — hand-authored fixtures, not a live capture.
```

Point the tool at your own recorded dump for real numbers on your workload.

## Use your own data

**vLLM** — record a `/metrics` time-series (the linter diffs consecutive scrapes to get
windowed hit-rate and eviction trends):

```bash
while :; do echo "# scrape_unix_ms $(date +%s%3N)"; curl -s localhost:8000/metrics; sleep 15; done > scrapes.prom
fbc-cache-lint report scrapes.prom
```

A single snapshot works too, but time-dependent findings need a series. You can also pipe
straight in:

```bash
curl -s localhost:8000/metrics | fbc-cache-lint report -
```

**llama.cpp** — capture `llama-server` stderr (slot logs need verbose: `-lv 1`):

```bash
llama-server -m model.gguf -lv 1 2> slots.log
fbc-cache-lint report slots.log
```

The format is auto-detected; force it with `--source vllm` or `--source llama.cpp`.

## JSON + CI

`--json` emits the same findings as a machine-readable document (summary + ranked
findings, each with `currency`, `remediation`, and raw `evidence`). The **exit code**
makes it a CI gate:

| Code | Meaning |
|---|---|
| `0` | ran cleanly; no finding at/above the `--fail-on` severity |
| `1` | ran cleanly; a finding met/exceeded `--fail-on` (default: `high`) |
| `2` | usage / IO / unrecognised-input error |

```bash
fbc-cache-lint report scrapes.prom                 # fail only on HIGH (default)
fbc-cache-lint report scrapes.prom --fail-on none  # report, never fail
fbc-cache-lint report scrapes.prom --fail-on medium --json
```

The dollar and KV-byte figures are estimates from `--price-per-1k-tokens`,
`--kv-bytes-per-token`, and `--recompute-tokens-per-preemption`; set them to your model
and pricing for accurate numbers.

## About the demo data

The two `demo` captures are **real**: byte-identical slices of actual `llama-server`
runs (phi-3-mini, build 8227), so their reports carry no synthetic disclaimer. The
misconfigured one is labelled *DELIBERATELY MISCONFIGURED … not a found-in-the-wild
trace* — a real run staged to demonstrate a detection, never presented as one caught in
the wild. The `.SYNTHETIC.` vLLM example is format-faithful but hand-valued, and every
report over it says so in the footer.

## The bigger picture

cache-lint is the shippable wedge of a larger project. The same repository also contains
the **FBC codec** — a factored block codec for inference-cache *metadata* (a separate,
gated technical component; the linter shares no code with it and never pulls its
scientific-stack dependencies). The codec's one honest, reproducible compression result —
and, just as prominently, the domains where it loses — is written up in the
[codec benchmark](https://github.com/the10kclub/fbc/blob/main/bench/README.md).

Source, issues, and the full roadmap: <https://github.com/the10kclub/fbc>.
Licensed under Apache-2.0.
