Metadata-Version: 2.5
Name: lifeboatcache
Version: 0.1.19
Summary: Lifeboat Cache — tiered, compressed, persistent KV cache for vLLM, SGLang and Lifeboat, with a built-in dashboard
Author-email: "Iterate.ai" <brian@iterate.ai>
License: Proprietary
Requires-Python: >=3.10
Requires-Dist: numpy>=1.24
Requires-Dist: pydantic<3,>=2.7
Requires-Dist: pyyaml<7,>=6
Requires-Dist: zstandard>=0.22
Provides-Extra: dev
Requires-Dist: cython; extra == 'dev'
Requires-Dist: maturin<2,>=1.7; extra == 'dev'
Requires-Dist: moto<6,>=5; extra == 'dev'
Requires-Dist: openai; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest<9,>=8; extra == 'dev'
Provides-Extra: engines
Requires-Dist: lmcache<0.6,>=0.5.6; extra == 'engines'
Requires-Dist: nixl<2,>=1.5.0; extra == 'engines'
Requires-Dist: sglang<0.6,>=0.5.19; extra == 'engines'
Requires-Dist: vllm<0.33,>=0.32.0; extra == 'engines'
Provides-Extra: s3
Requires-Dist: boto3>=1.28; extra == 's3'
Provides-Extra: sglang
Provides-Extra: vllm
Requires-Dist: lmcache<0.6,>=0.5.6; extra == 'vllm'
Description-Content-Type: text/markdown

# Lifeboat Cache

Tiered, compressed, persistent KV cache for **vLLM**, **SGLang**, and **Lifeboat** (which
embeds SGLang), with a built-in dashboard to configure, monitor and observe it.

```bash
pip install "lifeboatcache[vllm]"   # into the environment that runs vLLM
pip install lifeboatcache           # into the environment that runs SGLang / Lifeboat
```

Installing registers engine plugins; nothing else is needed to start. With no config file,
Lifeboat writes KV to local disk under `~/.lifeboat_cache/kv`, compresses it with fp8, and
serves the dashboard on <http://127.0.0.1:9400>.

## Turn it on

| Engine | How |
|---|---|
| vLLM | `LIFEBOAT_CACHE=1 vllm serve <model>` — or explicitly `--kv-transfer-config '{"kv_connector":"LifeboatConnector","kv_role":"kv_both"}'` |
| SGLang | `LIFEBOAT_CACHE=1 python -m sglang.launch_server --model-path <model>` |
| Lifeboat | set `LIFEBOAT_CACHE=1` in the Lifeboat container's environment, with `lifeboatcache` installed in its Python env |

What happens underneath:

- **vLLM** (default, `engine.vllm_mode: native`) — Lifeboat is a secondary tier of vLLM's own
  KV offloading (`OffloadingConnector` + `TieringOffloadingSpec`). vLLM moves GPU ↔ CPU for every
  KV cache group, including hybrid models' Mamba / linear-attention state; Lifeboat persists
  those blocks. **No LMCache needed.**
- **vLLM** (`engine.vllm_mode: lmcache`) — `LifeboatConnector` is LMCache's multiprocess
  connector with Lifeboat defaults: it auto-starts a local LMCache server (port 6555) whose L2
  storage is the Lifeboat adapter. Standard-attention models only.
- **SGLang / Lifeboat** — Lifeboat is a HiCache storage backend (`--hicache-storage-backend
  dynamic`). With `LIFEBOAT_CACHE=1` the plugin enables HiCache and selects the backend unless
  the launch already chose a storage backend. Explicit flags work too:

  ```bash
  python -m sglang.launch_server --model-path <model> --enable-hierarchical-cache \
    --hicache-storage-backend dynamic \
    --hicache-storage-backend-extra-config '{"backend_name":"lifeboat","module_path":"lifeboat_cache.integrations.sglang_backend","class_name":"LifeboatHiCacheStorage"}'
  ```

## Where it runs

Lifeboat Cache plugs into the engine's own KV offload path, so it runs where that path exists.
Everywhere else `LIFEBOAT_CACHE=1` leaves the launch exactly as it would be without it, and logs
why.

| Engine and backend | Lifeboat Cache |
|---|---|
| vLLM on NVIDIA GPUs (CUDA) | Supported; measured below |
| vLLM on AMD (ROCm) and Intel (XPU) GPUs | vLLM's offloading supports them; not measured yet |
| vLLM on CPU (Apple silicon, x86 / ARM CPU servers) | Supported (native mode, Lifeboat's host offload spec); LMCache mode needs a GPU |
| SGLang on NVIDIA GPUs (HiCache) | Supported |
| SGLang on AMD GPUs (HiCache) | Expected to work through HiCache; not measured yet |
| SGLang on Apple silicon (MLX backend) | Supported as SGLang's prefix-cache backend (`--radix-cache-backend lifeboat`, set automatically): full-attention models, and hybrid Mamba / linear-attention models from 0.1.17 (one request at a time: concurrent hybrid requests crash inside SGLang's MLX runtime) |
| MLX (`mlx-lm`), llama.cpp, Ollama | Not integrated |

Measured on an Apple M4 Max (2026-10-27), Qwen3-0.6B, a 6,123-token prompt asked again after the
engine's prefix cache was cleared and after a full restart (same answer each time):

| Engine on the Mac | First visit | Returning (cache cleared) | After restart |
|---|---|---|---|
| vLLM 0.32.0, CPU build | 15.9 s | 0.91 s (6,016 tokens from cache) | 0.94 s (from Lifeboat) |
| SGLang 0.5.22, MLX (GPU) | 3.6 s | 0.91 s (6,080 from Lifeboat) | 1.63 s (6,080 from Lifeboat) |

Several users returning at once, measured with the public verification tool
([LifeboatCacheTest](https://github.com/IterateAI/LifeboatCacheTest), lifeboatcache from PyPI, same
Mac and model; CSVs in `docs/reports/verify-mac-m4max-0.1.14/` (vLLM) and `-0.1.15/` (SGLang)):

| Engine on the Mac | Returning users | Recompute | Lifeboat Cache | After restart |
|---|---|---|---|---|
| vLLM 0.32.0, CPU build | 3 x 8K | 49.8 s | 1.11 s (99% from cache, 45x) | 0.55 s |
| SGLang 0.5.22, MLX (0.1.15) | 2 x 16K | 7.9 s | 1.48 s (100% from cache, 5.4x) | 0.88 s |

On MLX, Lifeboat stores full 64-token chunks uncompressed when a request finishes (copied out of
the pool on the GPU, written by a background thread), and loads them back when a prefix match
stops short. Reads start when a request arrives, so several returning users' reads overlap
(0.1.15); the match then writes the chunks into the pool in one update, about 0.3 s per 16K user
on this model. 4 users x 16K returning together: 2.0 s to the last user's first token, 2.5-2.6 s
without the prefetch. Hybrid (Mamba / sliding-window) models keep SGLang's cache.

## vLLM: install and start

```bash
# into the Python environment that runs vLLM (vLLM >= 0.32; LMCache not required)
pip install lifeboatcache

# zero-flag: the plugin wires vLLM's native offloading -> CPU tier -> Lifeboat Cache
LIFEBOAT_CACHE=1 vllm serve /models/Qwen3.8-27B --served-model-name my-model \
  --kv-cache-dtype fp8 --enable-prefix-caching

# explicit (same result, no env var)
vllm serve /models/Qwen3.8-27B --served-model-name my-model --enable-prefix-caching \
  --kv-transfer-config '{"kv_connector":"OffloadingConnector","kv_role":"kv_both",
    "kv_connector_extra_config":{"spec_name":"TieringOffloadingSpec","cpu_bytes_to_use":21474836480,
      "secondary_tiers":[{"type":"LifeboatTier","module_path":"lifeboat_cache.integrations.vllm_tier"}]}}'
```

How it attaches, native mode (no vLLM source changes):

```
vLLM (GPU KV + Mamba state, prefix cache)
  └─ OffloadingConnector / TieringOffloadingSpec                       [vllm.general_plugins]
       └─ CPU primary tier (/dev/shm, engine.native_cpu_gb, default 20 GB)
            └─ secondary tier: LifeboatTier                 [module_path plugin]
                 └─ tiers: ONTAP / NFS / S3 / local disk (fallback), index, eviction,
                    dashboard :9400, Prometheus :9402
```

The CPU tier lives in `/dev/shm`: make sure it has `native_cpu_gb` free (a vLLM killed with
SIGKILL can leave a stale `/dev/shm/vllm_offload_*.mmap` behind — delete it if nothing maps it).
Blocks are stored raw (KV is already fp8 with `--kv-cache-dtype fp8`; recurrent state must stay
lossless), keyed by vLLM's storage namespace (model, parallel layout, dtype, KV groups) so they
are never served to an incompatible configuration.

LMCache mode (`engine.vllm_mode: lmcache`, `pip install "lifeboatcache[vllm]"`, or
`--kv-transfer-config '{"kv_connector":"LifeboatConnector","kv_role":"kv_both"}'`):

```
vLLM (GPU KV, prefix cache)
  └─ LifeboatConnector  = LMCache MP connector + Lifeboat defaults      [vllm.general_plugins]
       └─ LMCache MP server (auto-started, localhost:6555)  L1 = host RAM
            └─ L2 adapter: LifeboatL2Adapter                 [--l2-adapter plugin]
                 └─ tiers: ONTAP / NFS / S3 / local disk (fallback), compression, index,
                    eviction policy, dashboard :9400, Prometheus :9402
```

Hybrid (Mamba / linear-attention) models such as Qwen3.5/3.8: **use native mode** (the
default) — 32× faster first token after a restart on Qwen3.8-27B at 256K, see below. Keep
`--max-num-seqs` below the Mamba cache-block count. Under LMCache (0.1.4+) hybrid models restore
too, but LMCache's format for them is ≈3.7× larger (≈131 KB/token vs ≈35 KB native), so its 20 GB
RAM tier cannot stage a 256K prompt.

Sizing native mode: `engine.native_cpu_gb` (default 20) is vLLM's host-RAM staging tier; KV
restored from disk passes through it, so it bounds how much several returning users can restore at
once. vLLM's offloading also allocates ≈19 GB of its own next to it — budget ≈ `native_cpu_gb` +
20 GB of host RAM. On attention models set `engine.native_block_tokens: 256` (vLLM otherwise
offloads 16-token blocks).

### If vLLM already uses LMCache

Lifeboat Cache never replaces a KV connector you configured — an explicit
`--kv-transfer-config` always wins, `LIFEBOAT_CACHE=1` included.

| Your current setup | With `lifeboatcache` installed |
|---|---|
| No KV connector | `LIFEBOAT_CACHE=1` adds the native `OffloadingConnector` + `LifeboatTier` (or `LifeboatConnector` with `engine.vllm_mode: lmcache`); without the variable nothing changes |
| `--kv-transfer-config` with `LMCacheMPConnector` + your own `lmcache server` | **Untouched** — an explicit KV-transfer config always wins, `LIFEBOAT_CACHE=1` included. Add Lifeboat to *your* server: `lmcache server ... --l2-adapter '{"type":"plugin","module_path":"lifeboat_cache.integrations.lmcache_adapter","class_name":"LifeboatL2Adapter","adapter_params":{"lifeboat_config":"/etc/lifeboat.yaml"}}'` — `--l2-adapter` is repeatable, so existing LMCache L2 adapters stay |
| `LifeboatConnector` pointed at an existing server (`lmcache.mp.host`/`port` set) | Connects to it; no second server is auto-started |
| In-process `LMCacheConnectorV1` (non-MP, legacy) | Not supported — use the MP connector |

## Storage targets and the local fallback

Targets are tried in order; the first usable one takes new writes. Each is write-probed at
startup. One that can't be reached (ONTAP/NFS export not mounted, S3 without credentials) is
skipped with the reason shown on the dashboard, and if nothing persistent is left, KV goes to
`storage.fallback_local` on local disk.

```yaml
storage:
  tiers:
    - {name: ontap, type: ontap, enabled: true, mount: /mnt/kvcache, protocol: pnfs_rdma}
    - {name: nfs,   type: nfs,   enabled: true, mount: /mnt/nfs, path: /mnt/nfs/lifeboat/kv}
    - {name: s3,    type: s3,    enabled: true, bucket: kv-archive, region: us-east-1}
  fallback_local: {enabled: true, path: ~/.lifeboat_cache/kv, capacity_gb: 50}
```

Persisted KV survives restarts: on startup the index is rebuilt from disk in the background.

Several engine processes on one node (one vLLM per GPU behind a load balancer) can share one
tier directory: give them the same path and set `storage.shared_lookup: true`, and a block one
copy wrote is found by the others (a user returning on a different GPU is restored, not
recomputed). Size `capacity_gb` for the whole directory; see G23 in `docs/KNOWN_GAPS.md`.

## Dashboard

`http://127.0.0.1:9400` on the node (use `ssh -L 9400:localhost:9400 <node>` from your
laptop). It shows the cache hit rate, KV served, bytes saved by compression and the codec's
sampled reconstruction error, per-target status (circuit breaker, usage, queue), evictions,
and startup recovery. The **Configure** tab edits storage targets, the local fallback, the
compression codec per rung, the eviction policy and the write mode; changes are validated,
applied live, and saved to the config file (startup-bound sections are flagged "restart
required"). `/metrics` serves Prometheus text (also on port 9402).

To expose it beyond localhost, set `dashboard.host: 0.0.0.0` **and** `dashboard.auth_token`;
open the page once with `?token=<token>`.

CLI: `lifeboat-cache status` prints the same numbers; `lifeboat-cache init` writes a starter
config; `lifeboat-cache server` runs a standalone Lifeboat-backed LMCache server for several
vLLM instances to share.

## Configuration

One YAML file, resolved as: `--config` / `$LIFEBOAT_CACHE_CONFIG` →
`~/.lifeboat_cache/lifeboat.yaml` → built-in defaults (`$LIFEBOAT_CACHE_HOME` moves the home
directory). `lifeboat-cache config show` prints the resolved config. Presets ship with the
package: `extends: single_node_gpu`.

## Measured results (vLLM 0.32 / SGLang 0.5.19, 1× RTX PRO 6000, FP8 KV)

Time to first token for the same prompt **after a full server restart** (real source-code prompts):

| Model | Prompt | Without | With Lifeboat Cache | |
|---|---|---|---|---|
| Qwen3-30B-A3B (standard attention MoE) | 8K | 0.289 s | 0.161 s | 1.8× |
| | 16K | 0.470 s | 0.183 s | 2.6× |
| | 32K | 1.426 s | 0.339 s | **4.2×** |
| Qwen3.8-27B (hybrid), **SGLang 0.5.19** | 8K | 1.28 s | 0.70 s | 1.8× |
| | 32K | 5.61 s | 1.15 s | 4.9× |
| | 64K | 8.64 s | 1.49 s | **5.8×** |
| | 128K | 20.65 s | 5.41 s | 3.8× |
| Qwen3.8-27B (hybrid), **vLLM 0.32 native** | 8K | 1.264 s | 0.239 s | 5.3× |
| | 32K | 4.486 s | 0.599 s | 7.5× |
| | 64K | 7.826 s | 0.800 s | 9.8× |
| | 128K | 20.758 s | 1.248 s | **16.6×** |
| Qwen3.8-27B (hybrid), vLLM 0.32 + LMCache 0.5.6 | 8K / 128K | 1.26 s / 20.8 s | 1.43 s / 22.2 s | no gain (vLLM external hit rate 0%) |

**Full benchmark (revision 2, 2026-10-25, `docs/reports/Lifeboat_Cache_Benchmark_Report.pdf`)** —
six configurations including **vLLM's and LMCache's own disk tiers** as baselines, 1/4/16
concurrent users, 1K–256K context, page cache dropped before every disk read, same NVMe and 250 GB
budget for every disk tier:

| After a full restart, 1 user | vLLM only | LMCache (RAM) | LMCache + LMCache fs | LMCache + Lifeboat | vLLM offload + vLLM fs | vLLM offload + Lifeboat |
|---|---|---|---|---|---|---|
| Regular (Qwen3-30B-A3B), 256K | 90.4 s | 90.6 s | 6.11 s | 3.68 s | **2.88 s** | 5.72 s |
| Hybrid (Qwen3.8-27B), 128K | 34.4 s | 34.4 s | 6.54 s | 4.40 s | **1.29 s** | 1.85 s |
| Hybrid, 256K | 98.0 s | 98.2 s | 97.1 s (0%) | 97.5 s (0%) | **1.95 s** | 3.07 s |

| Returning users (mean TTFT, share cached) | LMCache (RAM) | LMCache + LMCache fs | LMCache + Lifeboat | vLLM offload + vLLM fs | vLLM offload + Lifeboat |
|---|---|---|---|---|---|
| Regular, 16 users × 32K | 12.4 s, 56% | 6.48 s, 81% | 6.53 s, 81% | **3.11 s, 93%** | 7.27 s, 81% |
| Regular, 4 users × 128K | 42.7 s, 50% | 24.8 s, 75% | 24.7 s, 75% | **0.98 s, 100%** | 24.8 s, 75% |
| Hybrid, 16 users × 32K | 52.6 s, 5% | 35.2 s, 30% | 43.5 s, 30% | **4.07 s, 96%** | **4.07 s, 96%** |
| Hybrid, 4 users × 128K | 88.5 s, 0% | 62.2 s, 25% | 70.8 s, 25% | **1.32 s, 99%** | 1.45 s, 99% |

The two tables above were measured with 0.1.4. At that point, as LMCache's disk tier Lifeboat
restored 1.4–1.7× faster than LMCache's own fs adapter, and as vLLM's offload tier it was
1.3–2.0× **slower** than vLLM's own fs tier. 0.1.5–0.1.6 rebuilt the read and write paths
(tables below). The large gains over "vLLM only" come from persistence itself, which both
engines already ship.

**End-to-end re-run (0.1.6–0.1.8, 2026-10-25).** Same harness, host, GPU, prompts and 250 GB
budget as revision 2; only the Lifeboat columns were re-run (vLLM 0.32 and LMCache 0.5.6 are
unchanged), with the cells the tables above quote. Time to first token in seconds, share of
the prompt served from cache in brackets where it is not ~100%
(`docs/reports/e2e_0.1.6.txt`):

| After a full restart, 1 user | vLLM fs | Lifeboat native 0.1.4 → **0.1.6** | LMCache fs | LMCache + Lifeboat 0.1.4 → **0.1.6** |
|---|---|---|---|---|
| Regular, 32K | 0.40 | 0.72 → **0.37** | 0.80 | 0.48 → **0.38** |
| Regular, 128K | 1.53 | 2.84 → **1.36** | 3.07 | 1.82 → **1.46** |
| Regular, 256K | 2.88 | 5.72 → **2.58** | 6.11 | 3.68 → **2.60** |
| Hybrid, 32K | 0.53 | 0.69 → **0.53** | 1.77 | 1.24 → **1.09** |
| Hybrid, 128K | 1.29 | 1.85 → **1.23** | 6.54 | 4.40 → **3.54** |
| Hybrid, 256K | 1.95 | 3.07 → **1.84** | 97.1 (0%) | 97.3 (0%) |

| Returning users (mean TTFT) | vLLM fs | Lifeboat native 0.1.4 → 0.1.7 → **0.1.8** |
|---|---|---|
| Regular, 16 users × 32K | 3.11 (93%) | 7.27 (81%) → 3.16 (93%) → **1.50 (99%)** |
| Regular, 4 users × 128K | 0.98 (100%) | 24.8 (75%) → 24.6 (75%) → **0.99 (100%)** |
| Hybrid, 16 users × 32K | 4.07 (96%) | 4.07 → 4.06 → **4.04** (96%) |
| Hybrid, 4 users × 128K | 1.32 (99%) | 1.45 → 1.37 → **1.34** (99%) |

- **vs LMCache's disk tier** (LMCache mode): restarts 1.6–2.35× faster on the regular model and
  1.4–1.85× on the hybrid.
- **vs vLLM's disk tier** (native mode): restarts went from up to 2× slower (0.1.4) to level or
  up to 11% faster (5–11% at 128K–256K). Most of the remaining time is GPU work both share, so faster disk reads show up
  only partly in TTFT.
- **Returning users** on the regular model:
  - 16 users × 32K: **2.07× faster than vLLM's fs tier** (1.50 s vs 3.11 s; 99% vs 93% of
    the prompt from cache).
  - 4 users × 128K: level (0.99 s vs 0.98 s); 0.1.4–0.1.7 recomputed one user (24.6 s).
  - How:
    - 0.1.7: lookups are asynchronous, through vLLM's own file-tier protocol.
    - 0.1.8: when vLLM's CPU tier is momentarily full, a request whose blocks are on disk
      waits for room (up to `engine.native_promotion_wait_s`, 5 s) instead of being
      recomputed. vLLM's fs tier recomputes it.
- **Hybrid on LMCache** stays unsupported at 256K (LMCache doesn't pass the recurrent state);
  use native mode.

**CPU-tier size (0.1.8, 2026-10-26).** The overflow cells re-run with vLLM's CPU tier
(`engine.native_cpu_gb`) at 20 GB and 30 GB, the most a 64 GB host can spare next to the
engine, and with `engine.native_promotion_wait_s` at 5 s and 30 s. Returning users, mean TTFT
in seconds (share from cache); – = not run (`docs/reports/ram_tier_0.1.8.txt`):

| Returning users | vLLM fs 20 GB | vLLM fs 30 GB | Lifeboat 20 GB | Lifeboat **30 GB** | Lifeboat 30 GB, wait 30 s |
|---|---|---|---|---|---|
| Regular, 16 × 32K | 3.11 (93%) | 1.60 (99%) | 1.36 (99%) | 1.67 (99%) | 1.60 (99%) |
| Regular, 16 × 64K | 17.99 (81%) | 10.75 (93%) | 12.57 (87%) | **3.40 (99.6%)** | 2.93 (99.6%) |
| Regular, 4 × 128K | 0.98 | 1.22 | 0.99 | 1.27 | 1.23 |
| Regular, 4 × 256K | 160.59 (50%) | 93.00 (75%) | 74.65 (75%) | 76.66 (75%) | 122.34 (75%) |
| Hybrid, 16 × 64K | 29.31 (86%) | 18.86 (92%) | – | **5.23 (98%)** | 5.24 (98%) |
| Hybrid, 4 × 256K | 56.52 (75%) | 2.83 (100%) | – | 2.77 (100%) | 2.97 (100%) |

- **30 GB fixes 16 users × 64K**: 3.40 s, 3.2× faster than vLLM's fs tier given the same 30 GB.
- **4 users × 256K is about 51 GB of KV** and doesn't fit 30 GB: one user is recomputed in
  every configuration. Lifeboat is still 1.2× faster than vLLM fs at 30 GB (76.66 s vs 93.00 s).
  Fitting all four needs a tier of about 55 GB, i.e. more than 64 GB of RAM.
- **Keep the 5 s default wait.** 30 s helps only when everything fits (2.93 s vs 3.40 s). When
  the tier can't hold everyone, the request is recomputed anyway, just later (31.41 s vs 12.57 s
  at 20 GB; 122.34 s vs 76.66 s at 30 GB).
- **Hybrid model at 30 GB**: 16 × 64K is 3.6× faster than vLLM fs at the same 30 GB (5.23 s vs
  18.86 s). At 4 × 256K both fit at 30 GB and are level (2.77 s vs 2.83 s): that gain over
  20 GB (56.52 s) comes from the bigger tier, not from Lifeboat.
- Single-user and restart times don't depend on tier size and were unchanged.

**Storage layer head-to-head (0.1.6, 2026-10-25).** Both adapters were driven through LMCache's
L2 interface on the same host and NVMe (Intel Core Ultra 9 285K; 6.9 GB/s O_DIRECT, 5.3 GB/s
buffered). The data was 6.4 GB of Qwen3-30B-A3B-shaped chunks (48 layers × 512 × 256 tokens),
read 32 chunks per request. "Cold" means the page cache was dropped before reading; "hot" means
just written. Throughput is in GB/s of KV restored:

| KV cache, requests at once | Lifeboat cold | LMCache fs cold | LMCache fs + O_DIRECT cold | Lifeboat hot | LMCache fs hot | Lifeboat store | LMCache store |
|---|---|---|---|---|---|---|---|
| FP8, 1 | **6.4** | 2.9 | 5.5 | **14.6** | 5.0 | **5.4–6.1** | 2.9–3.1 |
| FP8, 4 | **6.8** | 4.7 | 6.8 | **15.2** | 11.0 | 1.9–2.2¹ | 2.9–3.2 |
| BF16, 1 | **6.5** | 3.2 | 5.7 | **16.3** | 5.3 | 3.0¹ | 3.1 |
| BF16, 4 | **6.8** | 4.7 | 6.8 | **17.0** | 10.9 | 2.2¹ | 3.3 |
| BF16 stored as fp8 (half the disk), 1 / 4 | 6.3 / 6.6 | — | — | 7.2 / 7.1 | — | 3.2 / 3.3¹ | — |

¹ Measured before the store path lost its extra copies; only the FP8, 1-request row was re-run
after. Store runs vary ±30% with page-cache writeback from earlier runs.

- **One request:** 2.0–2.2× faster than LMCache's default fs adapter, and 14–16% faster than
  LMCache with O_DIRECT.
- **4 concurrent requests:** 1.4× faster than the default adapter. LMCache with O_DIRECT ties,
  because both run at the drive's limit.
- **Just-written KV:** 1.4–3.1× faster.
- **Stores:** 1.8–2.1× faster (FP8, re-measured).
- **Limits:**
  - At the drive's limit, only reading fewer bytes goes faster.
  - A block read with O_DIRECT is not left in the page cache. A repeat read that LMCache's
    buffered adapter serves from RAM (13.6–17.3 GB/s at 4 requests) runs at disk speed here.
    `direct_io: false` on a tier restores page-cache behavior.

Native vLLM tier, same host, 1 GB of blocks, cold: 1.38 GB/s (0.1.4) → 4.04 (0.1.5) → **6.3–6.4
GB/s** (0.1.6). vLLM's own fs tier measured 3.2 GB/s there.

What changed in 0.1.6:
- Cold reads go O_DIRECT. New files align the payload to 4 KiB for this, and a read is
  bounced through an aligned buffer when the engine's buffer isn't 64-byte aligned.
- Blocks already in the page cache are served from it first (`RWF_NOWAIT`).
- Uncompressed chunks are read straight into LMCache's buffers.
- Store and load threads hash single-threaded. BLAKE3's own thread pool oversubscribed the
  cores (6–9 ms vs 0.7 ms per chunk).
- Stores encode, hash and write each chunk on its own worker, with one copy instead of three.

Lifeboat's other additions are operational: capacity limit, free-disk reserve, integrity,
indexed manifest, ONTAP/NFS/S3 fallback, SGLang and the dashboard.

Storage on that host: Lifeboat restore 4.8–5.2 GB/s from NVMe (raw device 6.7 GB/s), 17–19 GB/s
from RAM (above a 100 Gb/s RDMA link), GPU H2D 28.7 GB/s; recovery of 100K blocks from the SQLite
manifest 0.07 s vs 4.8 s by directory scan.

Qwen3-30B-A3B rows use LMCache mode; the native rows restored 95/95 blocks (4.9 GB) with a 95.5%
external prefix-cache hit rate; greedy outputs matched the pre-restart outputs in 3/4 prompts (the
4th diverged mid-sentence — vLLM with fp8 KV is not bit-deterministic even without a cache).

- 99.7% of the prompt's KV served from Lifeboat after restart on the standard-attention model.
- Cold-request overhead ≈1% (LMCache mode); native mode +0.1–2% at 32K–128K, +9% at 8K
  (1.37 s vs 1.25 s); GPU-warm repeats unchanged.
- Restore throughput 12.2 GiB/s (zero-copy mmap, parallel BLAKE3, one copy); index rebuild for
  16 GB of KV in 0.06 s.
- Hybrid (Mamba / linear-attention) models: on **SGLang** Lifeboat persists the attention KV *and*
  the recurrent state (HiCache v2 sidecar pools, state stored losslessly) → 1.8–5.8× after restart.
  On **vLLM**, native mode stores both through vLLM's own offloading → 5.3–32× after restart.
  Under LMCache they restore from 0.1.4 on (0.1.3 failed every hybrid restore), but LMCache's 20 GB
  L1 cannot stage a 256K hybrid prompt — prefer native mode for hybrid models.
- After a long prompt, allow SGLang's write-through (~30 s for 128K tokens) to finish before a
  restart; an unfinished tail is recomputed after the restart, never corrupted.

## Compression — what the numbers say

The lossy codecs apply to 16-bit KV. fp8 is the default for every stored rung. The dashboard
shows the sampled reconstruction error of whatever codec is live.

**Lossless (`lossless`, type `zkv`, 0.1.6).** It splits each element's bytes into planes and
codes each plane with zstd. Decoding is exact.

| | fp8 KV | bf16 KV |
|---|---|---|
| Size (Qwen3-4B KV) | 1.21× smaller (entropy bound ≈1.26×) | 1.46× smaller |
| Decode speed, benchmark host's x86 CPU | 1.4 GB/s per core; 9 GB/s on 8 threads, 18 GB/s on 12 | |

- **fp8 KV caches** (`--kv-cache-dtype fp8`) can't be shrunk by the lossy codecs.
  - On shared and archive rungs they are coded losslessly. Those links are slower than decode.
  - On local NVMe they are stored raw. Decoding there cost more time than the 21% of bytes it
    saved (cold 4.6–5.1 vs 6.4–6.8 GB/s).
  - `compression.lossless_8bit: on | remote | off` controls this.
- **`max_accuracy`** uses `lossless` on its local rungs.

**LBQ (0.1.5)** is Lifeboat's low-bit codec. For each stored chunk it:
- centers the keys per chunk (attention ignores a shared key offset);
- applies a randomized Hadamard rotation;
- stores a per-vector fp16 scale and codes values with a Lloyd-Max codebook;
- uses separate key / value bit widths;
- keeps the first / last layer and the chunk that opens a sequence (the attention sink) at fp8.

Qwen3-4B, BF16 KV. Code and prose prompts are stored and restored through the live path in
256-token chunks, then a 512-token continuation is scored against the uncompressed cache
(`benchmarks/accuracy/codec_accuracy.py`, `docs/reports/codec_accuracy_qwen3-4b.json`):

| codec | size vs BF16 (2K / 8K / 16K) | KL vs uncompressed (2K / 8K / 16K) | top-1 agreement |
|---|---|---|---|
| fp8 (default) | 2.0× | 0.008 / 0.003 / 0.014 | 98% |
| `lbq4` (`turboquant4`) | 3.5× / 3.7× / 3.8× | 0.004 / 0.005 / 0.006 | 98% |
| `lbq3` (`turboquant3`) | 4.3× / 4.8× / 5.0× | 0.016 / 0.012 / 0.019 | 96–97% |
| `lbq2` (`oscar2`) | 5.0× / 6.0× / 6.2× | 0.10 / 0.11 / 0.15 (lossy) | 90–93% |
| old group-wise turboquant3 | 4.0× | 7.8 / 7.0 / 7.0 (breaks the model) | 4–6% |

The old algorithms on Qwen3-30B-A3B (2K / 8K / 24K): turboquant4 2.7× at +0–4% perplexity;
turboquant3 +44–64%; 2-bit OSCAR per sequence +955–1,379%. Per stored chunk, OSCAR compressed
nothing (1.0×). LBQ has not yet been measured on the 30B model.

When to use which:
- **LBQ** decodes slower than fp8. Per 25 MB chunk (raw-equivalent): ~1 GB/s vs 4.3 GB/s for fp8
  on eqprod's x86 CPU with 8 threads, and ~1.5 GB/s vs ~10 GB/s on an Apple M4 Max. Use it where
  bytes cost more than CPU: shared / archive rungs and capacity-bound disks.
- **fp8** stays the default for local NVMe, where restore time matters most.
- **`storage_efficient`** uses `lbq3` everywhere.
