Metadata-Version: 2.5
Name: lifeboatcache
Version: 0.1.2
Summary: Lifeboat Cache — tiered, compressed, persistent KV cache for vLLM, SGLang and Lifeboat, with a built-in dashboard
Author-email: "Iterate.ai" <brian@iterate.ai>
License: Proprietary
Requires-Python: >=3.10
Requires-Dist: numpy>=1.24
Requires-Dist: pydantic<3,>=2.7
Requires-Dist: pyyaml<7,>=6
Provides-Extra: dev
Requires-Dist: cython; extra == 'dev'
Requires-Dist: maturin<2,>=1.7; extra == 'dev'
Requires-Dist: moto<6,>=5; extra == 'dev'
Requires-Dist: openai; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest<9,>=8; extra == 'dev'
Provides-Extra: engines
Requires-Dist: lmcache<0.6,>=0.5.6; extra == 'engines'
Requires-Dist: nixl<2,>=1.5.0; extra == 'engines'
Requires-Dist: sglang<0.6,>=0.5.19; extra == 'engines'
Requires-Dist: vllm<0.33,>=0.32.0; extra == 'engines'
Provides-Extra: s3
Requires-Dist: boto3>=1.28; extra == 's3'
Provides-Extra: sglang
Provides-Extra: vllm
Requires-Dist: lmcache<0.6,>=0.5.6; extra == 'vllm'
Description-Content-Type: text/markdown

# Lifeboat Cache

Tiered, compressed, persistent KV cache for **vLLM**, **SGLang**, and **Lifeboat** (which
embeds SGLang), with a built-in dashboard to configure, monitor and observe it.

```bash
pip install "lifeboatcache[vllm]"   # into the environment that runs vLLM
pip install lifeboatcache           # into the environment that runs SGLang / Lifeboat
```

Installing registers engine plugins; nothing else is needed to start. With no config file,
Lifeboat writes KV to local disk under `~/.lifeboat_cache/kv`, compresses it with fp8, and
serves the dashboard on <http://127.0.0.1:9400>.

## Turn it on

| Engine | How |
|---|---|
| vLLM | `LIFEBOAT_CACHE=1 vllm serve <model>` — or explicitly `--kv-transfer-config '{"kv_connector":"LifeboatConnector","kv_role":"kv_both"}'` |
| SGLang | `LIFEBOAT_CACHE=1 python -m sglang.launch_server --model-path <model>` |
| Lifeboat | set `LIFEBOAT_CACHE=1` in the Lifeboat container's environment, with `lifeboatcache` installed in its Python env |

What happens underneath:

- **vLLM** (default, `engine.vllm_mode: native`) — Lifeboat is a secondary tier of vLLM's own
  KV offloading (`OffloadingConnector` + `TieringOffloadingSpec`). vLLM moves GPU ↔ CPU for every
  KV cache group, including hybrid models' Mamba / linear-attention state; Lifeboat persists
  those blocks. **No LMCache needed.**
- **vLLM** (`engine.vllm_mode: lmcache`) — `LifeboatConnector` is LMCache's multiprocess
  connector with Lifeboat defaults: it auto-starts a local LMCache server (port 6555) whose L2
  storage is the Lifeboat adapter. Standard-attention models only.
- **SGLang / Lifeboat** — Lifeboat is a HiCache storage backend (`--hicache-storage-backend
  dynamic`). With `LIFEBOAT_CACHE=1` the plugin enables HiCache and selects the backend unless
  the launch already chose a storage backend. Explicit flags work too:

  ```bash
  python -m sglang.launch_server --model-path <model> --enable-hierarchical-cache \
    --hicache-storage-backend dynamic \
    --hicache-storage-backend-extra-config '{"backend_name":"lifeboat","module_path":"lifeboat_cache.integrations.sglang_backend","class_name":"LifeboatHiCacheStorage"}'
  ```

## vLLM: install and start

```bash
# into the Python environment that runs vLLM (vLLM >= 0.32; LMCache not required)
pip install lifeboatcache

# zero-flag: the plugin wires vLLM's native offloading -> CPU tier -> Lifeboat Cache
LIFEBOAT_CACHE=1 vllm serve /models/Qwen3.8-27B --served-model-name my-model \
  --kv-cache-dtype fp8 --enable-prefix-caching

# explicit (same result, no env var)
vllm serve /models/Qwen3.8-27B --served-model-name my-model --enable-prefix-caching \
  --kv-transfer-config '{"kv_connector":"OffloadingConnector","kv_role":"kv_both",
    "kv_connector_extra_config":{"spec_name":"TieringOffloadingSpec","cpu_bytes_to_use":21474836480,
      "secondary_tiers":[{"type":"LifeboatTier","module_path":"lifeboat_cache.integrations.vllm_tier"}]}}'
```

How it attaches, native mode (no vLLM source changes):

```
vLLM (GPU KV + Mamba state, prefix cache)
  └─ OffloadingConnector / TieringOffloadingSpec                       [vllm.general_plugins]
       └─ CPU primary tier (/dev/shm, engine.native_cpu_gb, default 20 GB)
            └─ secondary tier: LifeboatTier                 [module_path plugin]
                 └─ tiers: ONTAP / NFS / S3 / local disk (fallback), index, eviction,
                    dashboard :9400, Prometheus :9402
```

The CPU tier lives in `/dev/shm`: make sure it has `native_cpu_gb` free (a vLLM killed with
SIGKILL can leave a stale `/dev/shm/vllm_offload_*.mmap` behind — delete it if nothing maps it).
Blocks are stored raw (KV is already fp8 with `--kv-cache-dtype fp8`; recurrent state must stay
lossless), keyed by vLLM's storage namespace (model, parallel layout, dtype, KV groups) so they
are never served to an incompatible configuration.

LMCache mode (`engine.vllm_mode: lmcache`, `pip install "lifeboatcache[vllm]"`, or
`--kv-transfer-config '{"kv_connector":"LifeboatConnector","kv_role":"kv_both"}'`):

```
vLLM (GPU KV, prefix cache)
  └─ LifeboatConnector  = LMCache MP connector + Lifeboat defaults      [vllm.general_plugins]
       └─ LMCache MP server (auto-started, localhost:6555)  L1 = host RAM
            └─ L2 adapter: LifeboatL2Adapter                 [--l2-adapter plugin]
                 └─ tiers: ONTAP / NFS / S3 / local disk (fallback), compression, index,
                    eviction policy, dashboard :9400, Prometheus :9402
```

Hybrid (Mamba / linear-attention) models such as Qwen3.5/3.8: **use native mode** (the
default) — 5.3–16.6× faster first token after a restart on Qwen3.8-27B, see below. Keep
`--max-num-seqs` below the Mamba cache-block count. Behind LMCache 0.5.6 these models reuse
nothing (external hit rate 0%) because LMCache does not pass the recurrent state down.

### If vLLM already uses LMCache

Lifeboat Cache never replaces a KV connector you configured — an explicit
`--kv-transfer-config` always wins, `LIFEBOAT_CACHE=1` included.

| Your current setup | With `lifeboatcache` installed |
|---|---|
| No KV connector | `LIFEBOAT_CACHE=1` adds the native `OffloadingConnector` + `LifeboatTier` (or `LifeboatConnector` with `engine.vllm_mode: lmcache`); without the variable nothing changes |
| `--kv-transfer-config` with `LMCacheMPConnector` + your own `lmcache server` | **Untouched** — an explicit KV-transfer config always wins, `LIFEBOAT_CACHE=1` included. Add Lifeboat to *your* server: `lmcache server ... --l2-adapter '{"type":"plugin","module_path":"lifeboat_cache.integrations.lmcache_adapter","class_name":"LifeboatL2Adapter","adapter_params":{"lifeboat_config":"/etc/lifeboat.yaml"}}'` — `--l2-adapter` is repeatable, so existing LMCache L2 adapters stay |
| `LifeboatConnector` pointed at an existing server (`lmcache.mp.host`/`port` set) | Connects to it; no second server is auto-started |
| In-process `LMCacheConnectorV1` (non-MP, legacy) | Not supported — use the MP connector |

## Storage targets and the local fallback

Targets are tried in order; the first usable one takes new writes. Each is write-probed at
startup. One that can't be reached (ONTAP/NFS export not mounted, S3 without credentials) is
skipped with the reason shown on the dashboard, and if nothing persistent is left, KV goes to
`storage.fallback_local` on local disk.

```yaml
storage:
  tiers:
    - {name: ontap, type: ontap, enabled: true, mount: /mnt/kvcache, protocol: pnfs_rdma}
    - {name: nfs,   type: nfs,   enabled: true, mount: /mnt/nfs, path: /mnt/nfs/lifeboat/kv}
    - {name: s3,    type: s3,    enabled: true, bucket: kv-archive, region: us-east-1}
  fallback_local: {enabled: true, path: ~/.lifeboat_cache/kv, capacity_gb: 50}
```

Persisted KV survives restarts: on startup the index is rebuilt from disk in the background.

## Dashboard

`http://127.0.0.1:9400` on the node (use `ssh -L 9400:localhost:9400 <node>` from your
laptop). It shows the cache hit rate, KV served, bytes saved by compression and the codec's
sampled reconstruction error, per-target status (circuit breaker, usage, queue), evictions,
and startup recovery. The **Configure** tab edits storage targets, the local fallback, the
compression codec per rung, the eviction policy and the write mode; changes are validated,
applied live, and saved to the config file (startup-bound sections are flagged "restart
required"). `/metrics` serves Prometheus text (also on port 9402).

To expose it beyond localhost, set `dashboard.host: 0.0.0.0` **and** `dashboard.auth_token`;
open the page once with `?token=<token>`.

CLI: `lifeboat-cache status` prints the same numbers; `lifeboat-cache init` writes a starter
config; `lifeboat-cache server` runs a standalone Lifeboat-backed LMCache server for several
vLLM instances to share.

## Configuration

One YAML file, resolved as: `--config` / `$LIFEBOAT_CACHE_CONFIG` →
`~/.lifeboat_cache/lifeboat.yaml` → built-in defaults (`$LIFEBOAT_CACHE_HOME` moves the home
directory). `lifeboat-cache config show` prints the resolved config. Presets ship with the
package: `extends: single_node_gpu`.

## Measured results (vLLM 0.32 / SGLang 0.5.19, 1× RTX PRO 6000, FP8 KV)

Time to first token for the same prompt **after a full server restart** (real source-code prompts):

| Model | Prompt | Without | With Lifeboat Cache | |
|---|---|---|---|---|
| Qwen3-30B-A3B (standard attention MoE) | 8K | 0.289 s | 0.161 s | 1.8× |
| | 16K | 0.470 s | 0.183 s | 2.6× |
| | 32K | 1.426 s | 0.339 s | **4.2×** |
| Qwen3.8-27B (hybrid), **SGLang 0.5.19** | 8K | 1.28 s | 0.70 s | 1.8× |
| | 32K | 5.61 s | 1.15 s | 4.9× |
| | 64K | 8.64 s | 1.49 s | **5.8×** |
| | 128K | 20.65 s | 5.41 s | 3.8× |
| Qwen3.8-27B (hybrid), **vLLM 0.32 native** | 8K | 1.264 s | 0.239 s | 5.3× |
| | 32K | 4.486 s | 0.599 s | 7.5× |
| | 64K | 7.826 s | 0.800 s | 9.8× |
| | 128K | 20.758 s | 1.248 s | **16.6×** |
| Qwen3.8-27B (hybrid), vLLM 0.32 + LMCache 0.5.6 | 8K / 128K | 1.26 s / 20.8 s | 1.43 s / 22.2 s | no gain (vLLM external hit rate 0%) |

Qwen3-30B-A3B rows use LMCache mode; the native rows restored 95/95 blocks (4.9 GB) with a 95.5%
external prefix-cache hit rate; greedy outputs matched the pre-restart outputs in 3/4 prompts (the
4th diverged mid-sentence — vLLM with fp8 KV is not bit-deterministic even without a cache).

- 99.7% of the prompt's KV served from Lifeboat after restart on the standard-attention model.
- Cold-request overhead ≈1% (LMCache mode); native mode +0.1–2% at 32K–128K, +9% at 8K
  (1.37 s vs 1.25 s); GPU-warm repeats unchanged.
- Restore throughput 12.2 GiB/s (zero-copy mmap, parallel BLAKE3, one copy); index rebuild for
  16 GB of KV in 0.06 s.
- Hybrid (Mamba / linear-attention) models: on **SGLang** Lifeboat persists the attention KV *and*
  the recurrent state (HiCache v2 sidecar pools, state stored losslessly) → 1.8–5.8× after restart.
  On **vLLM**, native mode stores both through vLLM's own offloading → 5.3–16.6× after restart.
  Behind LMCache 0.5.6 they reuse nothing (no recurrent state) — don't use `vllm_mode: lmcache`
  for hybrid models.
- After a long prompt, allow SGLang's write-through (~30 s for 128K tokens) to finish before a
  restart; an unfinished tail is recomputed after the restart, never corrupted.

## Compression — what the numbers say

Measured on real Qwen3 KV (2,048-token prefix, see `docs/`):

| codec | bits/value | Qwen3-4B perplexity (base 2.297) |
|---|---|---|
| fp8 (default) | 8.0 | 2.293 |
| turboquant4, keep first/last 2 layers | 6.2 | 2.300 |
| turboquant3 | 4.0 | 824 (breaks the model) |
| oscar2 on 2,048-token chunks | 5.0 | 72 (breaks the model) |

fp8 is the default for every stored rung. The dashboard shows the sampled reconstruction error
of whatever codec is live.
