Metadata-Version: 2.5
Name: lifeboatcache
Version: 0.1.0
Summary: Lifeboat Cache — tiered, compressed, persistent KV cache for vLLM, SGLang and Lifeboat, with a built-in dashboard
Author-email: "Iterate.ai" <brian@iterate.ai>
License: Proprietary
Requires-Python: >=3.10
Requires-Dist: numpy>=1.24
Requires-Dist: pydantic<3,>=2.7
Requires-Dist: pyyaml<7,>=6
Provides-Extra: dev
Requires-Dist: cython; extra == 'dev'
Requires-Dist: maturin<2,>=1.7; extra == 'dev'
Requires-Dist: moto<6,>=5; extra == 'dev'
Requires-Dist: openai; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest<9,>=8; extra == 'dev'
Provides-Extra: engines
Requires-Dist: lmcache<0.6,>=0.5.6; extra == 'engines'
Requires-Dist: nixl<2,>=1.5.0; extra == 'engines'
Requires-Dist: sglang<0.6,>=0.5.19; extra == 'engines'
Requires-Dist: vllm<0.33,>=0.32.0; extra == 'engines'
Provides-Extra: s3
Requires-Dist: boto3>=1.28; extra == 's3'
Provides-Extra: sglang
Provides-Extra: vllm
Requires-Dist: lmcache<0.6,>=0.5.6; extra == 'vllm'
Description-Content-Type: text/markdown

# Lifeboat Cache

Tiered, compressed, persistent KV cache for **vLLM**, **SGLang**, and **Lifeboat** (which
embeds SGLang), with a built-in dashboard to configure, monitor and observe it.

```bash
pip install "lifeboatcache[vllm]"   # into the environment that runs vLLM
pip install lifeboatcache           # into the environment that runs SGLang / Lifeboat
```

Installing registers engine plugins; nothing else is needed to start. With no config file,
Lifeboat writes KV to local disk under `~/.lifeboat_cache/kv`, compresses it with fp8, and
serves the dashboard on <http://127.0.0.1:9400>.

## Turn it on

| Engine | How |
|---|---|
| vLLM | `LIFEBOAT_CACHE=1 vllm serve <model>` — or explicitly `--kv-transfer-config '{"kv_connector":"LifeboatConnector","kv_role":"kv_both"}'` |
| SGLang | `LIFEBOAT_CACHE=1 python -m sglang.launch_server --model-path <model>` |
| Lifeboat | set `LIFEBOAT_CACHE=1` in the Lifeboat container's environment, with `lifeboatcache` installed in its Python env |

What happens underneath:

- **vLLM** — `LifeboatConnector` is LMCache's multiprocess connector with Lifeboat defaults: it
  auto-starts a local LMCache server (port 6555) whose L2 storage is the Lifeboat adapter.
  GPU ↔ host movement is LMCache's; everything below host RAM is Lifeboat.
- **SGLang / Lifeboat** — Lifeboat is a HiCache storage backend (`--hicache-storage-backend
  dynamic`). With `LIFEBOAT_CACHE=1` the plugin enables HiCache and selects the backend unless
  the launch already chose a storage backend. Explicit flags work too:

  ```bash
  python -m sglang.launch_server --model-path <model> --enable-hierarchical-cache \
    --hicache-storage-backend dynamic \
    --hicache-storage-backend-extra-config '{"backend_name":"lifeboat","module_path":"lifeboat_cache.integrations.sglang_backend","class_name":"LifeboatHiCacheStorage"}'
  ```

## Storage targets and the local fallback

Targets are tried in order; the first usable one takes new writes. Each is write-probed at
startup. One that can't be reached (ONTAP/NFS export not mounted, S3 without credentials) is
skipped with the reason shown on the dashboard, and if nothing persistent is left, KV goes to
`storage.fallback_local` on local disk.

```yaml
storage:
  tiers:
    - {name: ontap, type: ontap, enabled: true, mount: /mnt/kvcache, protocol: pnfs_rdma}
    - {name: nfs,   type: nfs,   enabled: true, mount: /mnt/nfs, path: /mnt/nfs/lifeboat/kv}
    - {name: s3,    type: s3,    enabled: true, bucket: kv-archive, region: us-east-1}
  fallback_local: {enabled: true, path: ~/.lifeboat_cache/kv, capacity_gb: 50}
```

Persisted KV survives restarts: on startup the index is rebuilt from disk in the background.

## Dashboard

`http://127.0.0.1:9400` on the node (use `ssh -L 9400:localhost:9400 <node>` from your
laptop). It shows the cache hit rate, KV served, bytes saved by compression and the codec's
sampled reconstruction error, per-target status (circuit breaker, usage, queue), evictions,
and startup recovery. The **Configure** tab edits storage targets, the local fallback, the
compression codec per rung, the eviction policy and the write mode; changes are validated,
applied live, and saved to the config file (startup-bound sections are flagged "restart
required"). `/metrics` serves Prometheus text (also on port 9402).

To expose it beyond localhost, set `dashboard.host: 0.0.0.0` **and** `dashboard.auth_token`;
open the page once with `?token=<token>`.

CLI: `lifeboat-cache status` prints the same numbers; `lifeboat-cache init` writes a starter
config; `lifeboat-cache server` runs a standalone Lifeboat-backed LMCache server for several
vLLM instances to share.

## Configuration

One YAML file, resolved as: `--config` / `$LIFEBOAT_CACHE_CONFIG` →
`~/.lifeboat_cache/lifeboat.yaml` → built-in defaults (`$LIFEBOAT_CACHE_HOME` moves the home
directory). `lifeboat-cache config show` prints the resolved config. Presets ship with the
package: `extends: single_node_gpu`.

## Compression — what the numbers say

Measured on real Qwen3 KV (2,048-token prefix, see `docs/`):

| codec | bits/value | Qwen3-4B perplexity (base 2.297) |
|---|---|---|
| fp8 (default) | 8.0 | 2.293 |
| turboquant4, keep first/last 2 layers | 6.2 | 2.300 |
| turboquant3 | 4.0 | 824 (breaks the model) |
| oscar2 on 2,048-token chunks | 5.0 | 72 (breaks the model) |

fp8 is the default for every stored rung. The dashboard shows the sampled reconstruction error
of whatever codec is live.
