Metadata-Version: 2.5
Name: lifeboatcache
Version: 0.1.6
Summary: Lifeboat Cache — tiered, compressed, persistent KV cache for vLLM, SGLang and Lifeboat, with a built-in dashboard
Author-email: "Iterate.ai" <brian@iterate.ai>
License: Proprietary
Requires-Python: >=3.10
Requires-Dist: numpy>=1.24
Requires-Dist: pydantic<3,>=2.7
Requires-Dist: pyyaml<7,>=6
Requires-Dist: zstandard>=0.22
Provides-Extra: dev
Requires-Dist: cython; extra == 'dev'
Requires-Dist: maturin<2,>=1.7; extra == 'dev'
Requires-Dist: moto<6,>=5; extra == 'dev'
Requires-Dist: openai; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest<9,>=8; extra == 'dev'
Provides-Extra: engines
Requires-Dist: lmcache<0.6,>=0.5.6; extra == 'engines'
Requires-Dist: nixl<2,>=1.5.0; extra == 'engines'
Requires-Dist: sglang<0.6,>=0.5.19; extra == 'engines'
Requires-Dist: vllm<0.33,>=0.32.0; extra == 'engines'
Provides-Extra: s3
Requires-Dist: boto3>=1.28; extra == 's3'
Provides-Extra: sglang
Provides-Extra: vllm
Requires-Dist: lmcache<0.6,>=0.5.6; extra == 'vllm'
Description-Content-Type: text/markdown

# Lifeboat Cache

Tiered, compressed, persistent KV cache for **vLLM**, **SGLang**, and **Lifeboat** (which
embeds SGLang), with a built-in dashboard to configure, monitor and observe it.

```bash
pip install "lifeboatcache[vllm]"   # into the environment that runs vLLM
pip install lifeboatcache           # into the environment that runs SGLang / Lifeboat
```

Installing registers engine plugins; nothing else is needed to start. With no config file,
Lifeboat writes KV to local disk under `~/.lifeboat_cache/kv`, compresses it with fp8, and
serves the dashboard on <http://127.0.0.1:9400>.

## Turn it on

| Engine | How |
|---|---|
| vLLM | `LIFEBOAT_CACHE=1 vllm serve <model>` — or explicitly `--kv-transfer-config '{"kv_connector":"LifeboatConnector","kv_role":"kv_both"}'` |
| SGLang | `LIFEBOAT_CACHE=1 python -m sglang.launch_server --model-path <model>` |
| Lifeboat | set `LIFEBOAT_CACHE=1` in the Lifeboat container's environment, with `lifeboatcache` installed in its Python env |

What happens underneath:

- **vLLM** (default, `engine.vllm_mode: native`) — Lifeboat is a secondary tier of vLLM's own
  KV offloading (`OffloadingConnector` + `TieringOffloadingSpec`). vLLM moves GPU ↔ CPU for every
  KV cache group, including hybrid models' Mamba / linear-attention state; Lifeboat persists
  those blocks. **No LMCache needed.**
- **vLLM** (`engine.vllm_mode: lmcache`) — `LifeboatConnector` is LMCache's multiprocess
  connector with Lifeboat defaults: it auto-starts a local LMCache server (port 6555) whose L2
  storage is the Lifeboat adapter. Standard-attention models only.
- **SGLang / Lifeboat** — Lifeboat is a HiCache storage backend (`--hicache-storage-backend
  dynamic`). With `LIFEBOAT_CACHE=1` the plugin enables HiCache and selects the backend unless
  the launch already chose a storage backend. Explicit flags work too:

  ```bash
  python -m sglang.launch_server --model-path <model> --enable-hierarchical-cache \
    --hicache-storage-backend dynamic \
    --hicache-storage-backend-extra-config '{"backend_name":"lifeboat","module_path":"lifeboat_cache.integrations.sglang_backend","class_name":"LifeboatHiCacheStorage"}'
  ```

## vLLM: install and start

```bash
# into the Python environment that runs vLLM (vLLM >= 0.32; LMCache not required)
pip install lifeboatcache

# zero-flag: the plugin wires vLLM's native offloading -> CPU tier -> Lifeboat Cache
LIFEBOAT_CACHE=1 vllm serve /models/Qwen3.8-27B --served-model-name my-model \
  --kv-cache-dtype fp8 --enable-prefix-caching

# explicit (same result, no env var)
vllm serve /models/Qwen3.8-27B --served-model-name my-model --enable-prefix-caching \
  --kv-transfer-config '{"kv_connector":"OffloadingConnector","kv_role":"kv_both",
    "kv_connector_extra_config":{"spec_name":"TieringOffloadingSpec","cpu_bytes_to_use":21474836480,
      "secondary_tiers":[{"type":"LifeboatTier","module_path":"lifeboat_cache.integrations.vllm_tier"}]}}'
```

How it attaches, native mode (no vLLM source changes):

```
vLLM (GPU KV + Mamba state, prefix cache)
  └─ OffloadingConnector / TieringOffloadingSpec                       [vllm.general_plugins]
       └─ CPU primary tier (/dev/shm, engine.native_cpu_gb, default 20 GB)
            └─ secondary tier: LifeboatTier                 [module_path plugin]
                 └─ tiers: ONTAP / NFS / S3 / local disk (fallback), index, eviction,
                    dashboard :9400, Prometheus :9402
```

The CPU tier lives in `/dev/shm`: make sure it has `native_cpu_gb` free (a vLLM killed with
SIGKILL can leave a stale `/dev/shm/vllm_offload_*.mmap` behind — delete it if nothing maps it).
Blocks are stored raw (KV is already fp8 with `--kv-cache-dtype fp8`; recurrent state must stay
lossless), keyed by vLLM's storage namespace (model, parallel layout, dtype, KV groups) so they
are never served to an incompatible configuration.

LMCache mode (`engine.vllm_mode: lmcache`, `pip install "lifeboatcache[vllm]"`, or
`--kv-transfer-config '{"kv_connector":"LifeboatConnector","kv_role":"kv_both"}'`):

```
vLLM (GPU KV, prefix cache)
  └─ LifeboatConnector  = LMCache MP connector + Lifeboat defaults      [vllm.general_plugins]
       └─ LMCache MP server (auto-started, localhost:6555)  L1 = host RAM
            └─ L2 adapter: LifeboatL2Adapter                 [--l2-adapter plugin]
                 └─ tiers: ONTAP / NFS / S3 / local disk (fallback), compression, index,
                    eviction policy, dashboard :9400, Prometheus :9402
```

Hybrid (Mamba / linear-attention) models such as Qwen3.5/3.8: **use native mode** (the
default) — 32× faster first token after a restart on Qwen3.8-27B at 256K, see below. Keep
`--max-num-seqs` below the Mamba cache-block count. Under LMCache (0.1.4+) hybrid models restore
too, but LMCache's format for them is ≈3.7× larger (≈131 KB/token vs ≈35 KB native), so its 20 GB
RAM tier cannot stage a 256K prompt.

Sizing native mode: `engine.native_cpu_gb` (default 20) is vLLM's host-RAM staging tier; KV
restored from disk passes through it, so it bounds how much several returning users can restore at
once. vLLM's offloading also allocates ≈19 GB of its own next to it — budget ≈ `native_cpu_gb` +
20 GB of host RAM. On attention models set `engine.native_block_tokens: 256` (vLLM otherwise
offloads 16-token blocks).

### If vLLM already uses LMCache

Lifeboat Cache never replaces a KV connector you configured — an explicit
`--kv-transfer-config` always wins, `LIFEBOAT_CACHE=1` included.

| Your current setup | With `lifeboatcache` installed |
|---|---|
| No KV connector | `LIFEBOAT_CACHE=1` adds the native `OffloadingConnector` + `LifeboatTier` (or `LifeboatConnector` with `engine.vllm_mode: lmcache`); without the variable nothing changes |
| `--kv-transfer-config` with `LMCacheMPConnector` + your own `lmcache server` | **Untouched** — an explicit KV-transfer config always wins, `LIFEBOAT_CACHE=1` included. Add Lifeboat to *your* server: `lmcache server ... --l2-adapter '{"type":"plugin","module_path":"lifeboat_cache.integrations.lmcache_adapter","class_name":"LifeboatL2Adapter","adapter_params":{"lifeboat_config":"/etc/lifeboat.yaml"}}'` — `--l2-adapter` is repeatable, so existing LMCache L2 adapters stay |
| `LifeboatConnector` pointed at an existing server (`lmcache.mp.host`/`port` set) | Connects to it; no second server is auto-started |
| In-process `LMCacheConnectorV1` (non-MP, legacy) | Not supported — use the MP connector |

## Storage targets and the local fallback

Targets are tried in order; the first usable one takes new writes. Each is write-probed at
startup. One that can't be reached (ONTAP/NFS export not mounted, S3 without credentials) is
skipped with the reason shown on the dashboard, and if nothing persistent is left, KV goes to
`storage.fallback_local` on local disk.

```yaml
storage:
  tiers:
    - {name: ontap, type: ontap, enabled: true, mount: /mnt/kvcache, protocol: pnfs_rdma}
    - {name: nfs,   type: nfs,   enabled: true, mount: /mnt/nfs, path: /mnt/nfs/lifeboat/kv}
    - {name: s3,    type: s3,    enabled: true, bucket: kv-archive, region: us-east-1}
  fallback_local: {enabled: true, path: ~/.lifeboat_cache/kv, capacity_gb: 50}
```

Persisted KV survives restarts: on startup the index is rebuilt from disk in the background.

## Dashboard

`http://127.0.0.1:9400` on the node (use `ssh -L 9400:localhost:9400 <node>` from your
laptop). It shows the cache hit rate, KV served, bytes saved by compression and the codec's
sampled reconstruction error, per-target status (circuit breaker, usage, queue), evictions,
and startup recovery. The **Configure** tab edits storage targets, the local fallback, the
compression codec per rung, the eviction policy and the write mode; changes are validated,
applied live, and saved to the config file (startup-bound sections are flagged "restart
required"). `/metrics` serves Prometheus text (also on port 9402).

To expose it beyond localhost, set `dashboard.host: 0.0.0.0` **and** `dashboard.auth_token`;
open the page once with `?token=<token>`.

CLI: `lifeboat-cache status` prints the same numbers; `lifeboat-cache init` writes a starter
config; `lifeboat-cache server` runs a standalone Lifeboat-backed LMCache server for several
vLLM instances to share.

## Configuration

One YAML file, resolved as: `--config` / `$LIFEBOAT_CACHE_CONFIG` →
`~/.lifeboat_cache/lifeboat.yaml` → built-in defaults (`$LIFEBOAT_CACHE_HOME` moves the home
directory). `lifeboat-cache config show` prints the resolved config. Presets ship with the
package: `extends: single_node_gpu`.

## Measured results (vLLM 0.32 / SGLang 0.5.19, 1× RTX PRO 6000, FP8 KV)

Time to first token for the same prompt **after a full server restart** (real source-code prompts):

| Model | Prompt | Without | With Lifeboat Cache | |
|---|---|---|---|---|
| Qwen3-30B-A3B (standard attention MoE) | 8K | 0.289 s | 0.161 s | 1.8× |
| | 16K | 0.470 s | 0.183 s | 2.6× |
| | 32K | 1.426 s | 0.339 s | **4.2×** |
| Qwen3.8-27B (hybrid), **SGLang 0.5.19** | 8K | 1.28 s | 0.70 s | 1.8× |
| | 32K | 5.61 s | 1.15 s | 4.9× |
| | 64K | 8.64 s | 1.49 s | **5.8×** |
| | 128K | 20.65 s | 5.41 s | 3.8× |
| Qwen3.8-27B (hybrid), **vLLM 0.32 native** | 8K | 1.264 s | 0.239 s | 5.3× |
| | 32K | 4.486 s | 0.599 s | 7.5× |
| | 64K | 7.826 s | 0.800 s | 9.8× |
| | 128K | 20.758 s | 1.248 s | **16.6×** |
| Qwen3.8-27B (hybrid), vLLM 0.32 + LMCache 0.5.6 | 8K / 128K | 1.26 s / 20.8 s | 1.43 s / 22.2 s | no gain (vLLM external hit rate 0%) |

**Full benchmark (revision 2, 2026-10-25, `docs/reports/Lifeboat_Cache_Benchmark_Report.pdf`)** —
six configurations including **vLLM's and LMCache's own disk tiers** as baselines, 1/4/16
concurrent users, 1K–256K context, page cache dropped before every disk read, same NVMe and 250 GB
budget for every disk tier:

| After a full restart, 1 user | vLLM only | LMCache (RAM) | LMCache + LMCache fs | LMCache + Lifeboat | vLLM offload + vLLM fs | vLLM offload + Lifeboat |
|---|---|---|---|---|---|---|
| Regular (Qwen3-30B-A3B), 256K | 90.4 s | 90.6 s | 6.11 s | 3.68 s | **2.88 s** | 5.72 s |
| Hybrid (Qwen3.8-27B), 128K | 34.4 s | 34.4 s | 6.54 s | 4.40 s | **1.29 s** | 1.85 s |
| Hybrid, 256K | 98.0 s | 98.2 s | 97.1 s (0%) | 97.5 s (0%) | **1.95 s** | 3.07 s |

| Returning users (mean TTFT, share cached) | LMCache (RAM) | LMCache + LMCache fs | LMCache + Lifeboat | vLLM offload + vLLM fs | vLLM offload + Lifeboat |
|---|---|---|---|---|---|
| Regular, 16 users × 32K | 12.4 s, 56% | 6.48 s, 81% | 6.53 s, 81% | **3.11 s, 93%** | 7.27 s, 81% |
| Regular, 4 users × 128K | 42.7 s, 50% | 24.8 s, 75% | 24.7 s, 75% | **0.98 s, 100%** | 24.8 s, 75% |
| Hybrid, 16 users × 32K | 52.6 s, 5% | 35.2 s, 30% | 43.5 s, 30% | **4.07 s, 96%** | **4.07 s, 96%** |
| Hybrid, 4 users × 128K | 88.5 s, 0% | 62.2 s, 25% | 70.8 s, 25% | **1.32 s, 99%** | 1.45 s, 99% |

The two tables above were measured with 0.1.4. At that point, as LMCache's disk tier Lifeboat
restored 1.4–1.7× faster than LMCache's own fs adapter, and as vLLM's offload tier it was
1.3–2.0× **slower** than vLLM's own fs tier. 0.1.5–0.1.6 rebuilt the read and write paths
(table below); the end-to-end tables have not been re-run since. The large gains over
"vLLM only" come from persistence itself, which both engines already ship.

**Storage layer head-to-head (0.1.6, 2026-10-25).** Both adapters were driven through LMCache's
L2 interface on the same host and NVMe (Intel Core Ultra 9 285K; 6.9 GB/s O_DIRECT, 5.3 GB/s
buffered). The data was 6.4 GB of Qwen3-30B-A3B-shaped chunks (48 layers × 512 × 256 tokens),
read 32 chunks per request. "Cold" means the page cache was dropped before reading; "hot" means
just written. Throughput is in GB/s of KV restored:

| KV cache, requests at once | Lifeboat cold | LMCache fs cold | LMCache fs + O_DIRECT cold | Lifeboat hot | LMCache fs hot | Lifeboat store | LMCache store |
|---|---|---|---|---|---|---|---|
| FP8, 1 | **6.4** | 2.9 | 5.5 | **14.6** | 5.0 | **5.4–6.1** | 2.9–3.1 |
| FP8, 4 | **6.8** | 4.7 | 6.8 | **15.2** | 11.0 | 1.9–2.2¹ | 2.9–3.2 |
| BF16, 1 | **6.5** | 3.2 | 5.7 | **16.3** | 5.3 | 3.0¹ | 3.1 |
| BF16, 4 | **6.8** | 4.7 | 6.8 | **17.0** | 10.9 | 2.2¹ | 3.3 |
| BF16 stored as fp8 (half the disk), 1 / 4 | 6.3 / 6.6 | — | — | 7.2 / 7.1 | — | 3.2 / 3.3¹ | — |

¹ Measured before the store path lost its extra copies; only the FP8, 1-request row was re-run
after. Store runs vary ±30% with page-cache writeback from earlier runs.

- **One request:** 2.0–2.2× faster than LMCache's default fs adapter, and 14–16% faster than
  LMCache with O_DIRECT.
- **4 concurrent requests:** 1.4× faster than the default adapter. LMCache with O_DIRECT ties,
  because both run at the drive's limit.
- **Just-written KV:** 1.4–3.1× faster.
- **Stores:** 1.8–2.1× faster (FP8, re-measured).
- **Limits:**
  - At the drive's limit, only reading fewer bytes goes faster.
  - A block read with O_DIRECT is not left in the page cache. A repeat read that LMCache's
    buffered adapter serves from RAM (13.6–17.3 GB/s at 4 requests) runs at disk speed here.
    `direct_io: false` on a tier restores page-cache behavior.

Native vLLM tier, same host, 1 GB of blocks, cold: 1.38 GB/s (0.1.4) → 4.04 (0.1.5) → **6.3–6.4
GB/s** (0.1.6). vLLM's own fs tier measured 3.2 GB/s there.

What changed in 0.1.6:
- Cold reads go O_DIRECT. New files align the payload to 4 KiB for this, and a read is
  bounced through an aligned buffer when the engine's buffer isn't 64-byte aligned.
- Blocks already in the page cache are served from it first (`RWF_NOWAIT`).
- Uncompressed chunks are read straight into LMCache's buffers.
- Store and load threads hash single-threaded. BLAKE3's own thread pool oversubscribed the
  cores (6–9 ms vs 0.7 ms per chunk).
- Stores encode, hash and write each chunk on its own worker, with one copy instead of three.

Lifeboat's other additions are operational: capacity limit, free-disk reserve, integrity,
indexed manifest, ONTAP/NFS/S3 fallback, SGLang and the dashboard.

Storage on that host: Lifeboat restore 4.8–5.2 GB/s from NVMe (raw device 6.7 GB/s), 17–19 GB/s
from RAM (above a 100 Gb/s RDMA link), GPU H2D 28.7 GB/s; recovery of 100K blocks from the SQLite
manifest 0.07 s vs 4.8 s by directory scan.

Qwen3-30B-A3B rows use LMCache mode; the native rows restored 95/95 blocks (4.9 GB) with a 95.5%
external prefix-cache hit rate; greedy outputs matched the pre-restart outputs in 3/4 prompts (the
4th diverged mid-sentence — vLLM with fp8 KV is not bit-deterministic even without a cache).

- 99.7% of the prompt's KV served from Lifeboat after restart on the standard-attention model.
- Cold-request overhead ≈1% (LMCache mode); native mode +0.1–2% at 32K–128K, +9% at 8K
  (1.37 s vs 1.25 s); GPU-warm repeats unchanged.
- Restore throughput 12.2 GiB/s (zero-copy mmap, parallel BLAKE3, one copy); index rebuild for
  16 GB of KV in 0.06 s.
- Hybrid (Mamba / linear-attention) models: on **SGLang** Lifeboat persists the attention KV *and*
  the recurrent state (HiCache v2 sidecar pools, state stored losslessly) → 1.8–5.8× after restart.
  On **vLLM**, native mode stores both through vLLM's own offloading → 5.3–32× after restart.
  Under LMCache they restore from 0.1.4 on (0.1.3 failed every hybrid restore), but LMCache's 20 GB
  L1 cannot stage a 256K hybrid prompt — prefer native mode for hybrid models.
- After a long prompt, allow SGLang's write-through (~30 s for 128K tokens) to finish before a
  restart; an unfinished tail is recomputed after the restart, never corrupted.

## Compression — what the numbers say

The lossy codecs apply to 16-bit KV. fp8 is the default for every stored rung. The dashboard
shows the sampled reconstruction error of whatever codec is live.

**Lossless (`lossless`, type `zkv`, 0.1.6).** It splits each element's bytes into planes and
codes each plane with zstd. Decoding is exact.

| | fp8 KV | bf16 KV |
|---|---|---|
| Size (Qwen3-4B KV) | 1.21× smaller (entropy bound ≈1.26×) | 1.46× smaller |
| Decode speed, benchmark host's x86 CPU | 1.4 GB/s per core; 9 GB/s on 8 threads, 18 GB/s on 12 | |

- **fp8 KV caches** (`--kv-cache-dtype fp8`) can't be shrunk by the lossy codecs.
  - On shared and archive rungs they are coded losslessly. Those links are slower than decode.
  - On local NVMe they are stored raw. Decoding there cost more time than the 21% of bytes it
    saved (cold 4.6–5.1 vs 6.4–6.8 GB/s).
  - `compression.lossless_8bit: on | remote | off` controls this.
- **`max_accuracy`** uses `lossless` on its local rungs.

**LBQ (0.1.5)** is Lifeboat's low-bit codec. For each stored chunk it:
- centers the keys per chunk (attention ignores a shared key offset);
- applies a randomized Hadamard rotation;
- stores a per-vector fp16 scale and codes values with a Lloyd-Max codebook;
- uses separate key / value bit widths;
- keeps the first / last layer and the chunk that opens a sequence (the attention sink) at fp8.

Qwen3-4B, BF16 KV. Code and prose prompts are stored and restored through the live path in
256-token chunks, then a 512-token continuation is scored against the uncompressed cache
(`benchmarks/accuracy/codec_accuracy.py`, `docs/reports/codec_accuracy_qwen3-4b.json`):

| codec | size vs BF16 (2K / 8K / 16K) | KL vs uncompressed (2K / 8K / 16K) | top-1 agreement |
|---|---|---|---|
| fp8 (default) | 2.0× | 0.008 / 0.003 / 0.014 | 98% |
| `lbq4` (`turboquant4`) | 3.5× / 3.7× / 3.8× | 0.004 / 0.005 / 0.006 | 98% |
| `lbq3` (`turboquant3`) | 4.3× / 4.8× / 5.0× | 0.016 / 0.012 / 0.019 | 96–97% |
| `lbq2` (`oscar2`) | 5.0× / 6.0× / 6.2× | 0.10 / 0.11 / 0.15 (lossy) | 90–93% |
| old group-wise turboquant3 | 4.0× | 7.8 / 7.0 / 7.0 (breaks the model) | 4–6% |

The old algorithms on Qwen3-30B-A3B (2K / 8K / 24K): turboquant4 2.7× at +0–4% perplexity;
turboquant3 +44–64%; 2-bit OSCAR per sequence +955–1,379%. Per stored chunk, OSCAR compressed
nothing (1.0×). LBQ has not yet been measured on the 30B model.

When to use which:
- **LBQ** decodes slower than fp8. Per 25 MB chunk (raw-equivalent): ~1 GB/s vs 4.3 GB/s for fp8
  on eqprod's x86 CPU with 8 threads, and ~1.5 GB/s vs ~10 GB/s on an Apple M4 Max. Use it where
  bytes cost more than CPU: shared / archive rungs and capacity-bound disks.
- **fp8** stays the default for local NVMe, where restore time matters most.
- **`storage_efficient`** uses `lbq3` everywhere.
