Metadata-Version: 2.4
Name: memkv-vllm
Version: 1.0.4
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Rust
Classifier: Operating System :: POSIX :: Linux
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Dist: pytest >=7 ; extra == 'test'
Provides-Extra: test
Summary: vLLM native KV offloading secondary tier backed by the MemKV context memory store (TieringOffloadingSpec).
Author: MinIO, Inc.
License: LicenseRef-Proprietary
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/miniohq/memkv
Project-URL: Documentation, https://docs.min.io/memkv/

# memkv-vllm

MemKV secondary tier for vLLM's **native KV offloading** feature
(`OffloadingConnector` + `TieringOffloadingSpec`). vLLM offloads completed KV
blocks from GPU to a pinned CPU pool; this plugin adds a `memkv` tier behind
that pool, so blocks evicted from CPU (or produced by another vLLM instance)
are served from a shared MemKV cluster over RDMA instead of being recomputed.

This is MemKV's third vLLM-adjacent integration and does not replace the
others: `lmcache-plugin/` (LMCache connector path, current production
recommendation) and `plugin/` (NIXL storage backend for Dynamo KVBM).

## Requirements

- vLLM main at or after commit `6cf7b26bd` (v0.23.1rc1.dev962). The tiering
  framework (`vllm.v1.kv_offload.tiering`) has shipped in releases since
  v0.22.0, but the secondary-tier API this plugin implements (LookupResult
  lookups, ScheduleEndContext, get_stats/build_metric_definitions) is newer
  than the latest stable release (v0.24.0); it first appears in the v0.25.0
  release candidates. The offloading spec API is experimental upstream; this
  plugin pins to the commit range noted in
  `plans/VLLM-TIERING-PLUGIN-PLAN.md`.
- Linux with RDMA userspace (`libibverbs`, `ibverbs-providers`) for the data
  path. The client falls back to TCP where RDMA is unavailable.
- `PYTHONHASHSEED` pinned (e.g. `0`) on every vLLM instance that should share
  KV — vLLM's block content hashes are seeded per process otherwise.

## Install

```bash
pip install memkv-vllm
```

The wheel registers itself via the `vllm.general_plugins` entry point
`memkv_tier` — no vLLM patches, no extra flags. If you restrict plugins with
`VLLM_PLUGINS`, include `memkv_tier` in the list.

## Configure

MemKV connection settings come from the standard `MEMKV_CONFIG` yaml or
`MEMKV_*` env-var chain (`MEMKV_SERVERS`, `MEMKV_AUTH_KEY`, …) — identical to
the other MemKV plugins. The tier entry in `secondary_tiers` carries only
tier tuning:

```bash
PYTHONHASHSEED=0 vllm serve <model> \
  --kv-transfer-config '{
    "kv_connector": "OffloadingConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "spec_name": "TieringOffloadingSpec",
      "cpu_bytes_to_use": 10737418240,
      "block_size": 256,
      "secondary_tiers": [
        {"type": "memkv", "n_read_threads": 8, "n_write_threads": 8}
      ]
    }
  }'
```

| Tier config key | Default | Meaning |
| --- | --- | --- |
| `type` | — | Must be `"memkv"`. |
| `prefix` | `""` | Key-namespace prefix inside MemKV (multi-fleet separation). |
| `n_read_threads` | `8` | Load-priority I/O threads (promotions, TTFT-critical). |
| `n_write_threads` | `8` | Store-priority I/O threads (cascades). |
| `blocks_per_op` | `8` | Max blocks per batched client call. |
| `load_failure_grace_s` | `60` | Withhold a failed load's completion from vLLM for this long, keeping its target slots pinned so a server write abandoned by a client timeout cannot land in reused memory. `0` disables. |
| `max_store_backlog_mb` | `2048` | Shed store jobs once this much store work is queued/in flight, so a slow MemKV cannot pin the CPU pool solid. Shed blocks just skip MemKV; loads are never shed. Keep below `cpu_bytes_to_use`/2. |

Sizing notes: `block_size` (offloaded block, in tokens) controls the MemKV
value size — prefer values ≥ 1 MiB (long blocks) so reads are
bandwidth-bound, not per-key-overhead-bound. `cpu_bytes_to_use` is the CPU
tier working set; MemKV only sees traffic once blocks cascade (immediately on
store) and promote (on CPU-tier misses).

## How it maps

One MemKV value per (block hash, KV cache group): the full offloaded block
across all TP ranks (`stride` bytes of the CPU pool). Keys reuse vLLM's
`FileMapper` naming — `<prefix>/<model>_<configdigest>…/<hash>.bin` — so runs
with the same model/layout share blocks and incompatible layouts can't
collide. Lookups are batched async `exists` probes (one round per scheduler
step); stores/loads are chunked `batch_put_views`/`batch_get_into` calls
straight against the pool slots (no Python-side copies). Capacity is governed
server-side by MemKV lane eviction; a value evicted between lookup and load
degrades to recompute, never corruption.

## Test

```bash
cargo build -p memkv            # loopback server binary for the fixture
maturin develop --manifest-path vllm-plugin/Cargo.toml
pytest vllm-plugin/tests -v    # requires vllm importable; Linux for data path
```

