Metadata-Version: 2.4
Name: memkv-vllm
Version: 1.0.11
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Rust
Classifier: Operating System :: POSIX :: Linux
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Dist: pytest >=7 ; extra == 'test'
Provides-Extra: test
Summary: MemKV backend for vLLM native KV offloading: a secondary tier behind the CPU pool (TieringOffloadingSpec) or MemKV as the offload medium itself (MemKVOffloadingSpec).
Author: MinIO, Inc.
License: LicenseRef-Proprietary
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/miniohq/memkv
Project-URL: Documentation, https://docs.min.io/memkv/

# memkv-vllm

Two ways to back vLLM's **native KV offloading** with a shared MemKV cluster.
Both ship in this one wheel and both are selected purely through
`--kv-transfer-config`. Neither needs a patched vLLM to run; the tier's
optional `lookup_timeout_s` deadline is the one exception, noted where it
appears below.

| Mode | Selected by | KV path | Use when |
| --- | --- | --- | --- |
| **Secondary tier** | `"spec_name": "TieringOffloadingSpec"` with a `memkv` entry in `secondary_tiers` | GPU → vLLM's pinned CPU pool → MemKV. RDMA, TCP fallback | You want vLLM's CPU pool in front as a residency layer. |
| **Direct offload** | `"spec_name": "MemKVOffloadingSpec"`, `"spec_module_path": "memkv_vllm.spec"` | GPU ↔ MemKV, staged through a small scratch buffer. RDMA only | You want no CPU pool holding a second copy of everything. |

They are alternatives — pick one per engine. The two modes write different
key layouts, so a cluster written by one is not read by the other.

This is MemKV's third vLLM-adjacent integration and does not replace the
others: `memkv-lmcache` (the LMCache connector path, currently the
production recommendation) and the MemKV NIXL storage backend for Dynamo
KVBM, which ships as a shared library rather than a wheel.

## Requirements

- **vLLM 0.25.0 or newer.** Tested against 0.25.0, 0.25.1, 0.26.0, 0.27.0,
  and 0.27.1. Releases before 0.25.0 do not work: the tiering framework
  itself has shipped since 0.22.0, but the secondary-tier interfaces this
  plugin implements — `LookupResult` lookups, `ScheduleEndContext`, and
  `get_stats`/`build_metric_definitions` — arrived in 0.25.0. vLLM reworked the offloading-spec API again
  in 0.26.0; the plugin detects which side it is running against and adapts,
  so no version pin is needed in either direction. The offloading-spec API
  is still experimental upstream, so releases past 0.27.1 are untested.
- **Python 3.10 or newer, on Linux** (x86-64 or aarch64). The wheels on PyPI
  are `manylinux_2_28`, so glibc 2.28 or newer — RHEL/Rocky 8+, Debian 10+,
  Ubuntu 18.10+. (The copies bundled in the MemKV deb/rpm are built without a
  manylinux tag and target the same hosts.)
- **RDMA userspace** (`libibverbs`, `ibverbs-providers`) for the data path.
  The wheels load it lazily, so they import on any Linux host. What happens
  without a reachable NIC differs by mode: the secondary tier falls back to
  TCP, while direct offload is RDMA-only and fails at startup rather than
  running slowly.
- **`PYTHONHASHSEED` pinned** (e.g. `0`) on every vLLM instance that should
  share KV. Without it, vLLM 0.25 and earlier seed block content hashes per
  process, as do later versions with the `xxhash` prefix-cache algorithm; the
  plugin warns at startup when that applies.
- **`--enable-cumem-allocator` when `PYTORCH_CUDA_ALLOC_CONF` has
  `expandable_segments:True`.** vLLM refuses to start `OffloadingConnector`
  under that setting otherwise. The flag puts the model weights and the KV
  cache in vLLM's own CuMem pool, where expandable segments are off; other
  allocations keep `PYTORCH_CUDA_ALLOC_CONF` as set.
- **A MemKV license.** The client verifies one when it builds its engine, so
  a plugin with none available fails at startup with `no license found`
  rather than degrading. Sources, in order: the `license:` field of
  `MEMKV_CONFIG`, then `MEMKV_LICENSE` (an inline JWT or a path to a file
  holding one), then `MINIO_LICENSE` / `AISTOR_LICENSE` and the standard
  `minio.license` file locations. Contact MinIO to obtain one.

## Install

```bash
pip install memkv-vllm
```

The wheel registers itself via the `vllm.general_plugins` entry point
`memkv_tier` — no vLLM patches, no extra flags. If you restrict plugins with
`VLLM_PLUGINS`, include `memkv_tier` in the list.

## Configure — secondary tier (`TieringOffloadingSpec`)

MemKV connection settings come from the standard `MEMKV_CONFIG` yaml or
`MEMKV_*` env-var chain (`MEMKV_SERVERS`, `MEMKV_AUTH_KEY`, …) — identical to
the other MemKV plugins. The tier entry in `secondary_tiers` carries only
tier tuning:

```bash
PYTHONHASHSEED=0 vllm serve <model> \
  --kv-transfer-config '{
    "kv_connector": "OffloadingConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "spec_name": "TieringOffloadingSpec",
      "cpu_bytes_to_use": 10737418240,
      "block_size": 256,
      "secondary_tiers": [
        {"type": "memkv", "n_read_threads": 8, "n_write_threads": 8}
      ]
    }
  }'
```

| Tier config key        | Default | Meaning                                                                                                                                                                                                                                                                                   |
| ---------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type`                 | —       | Must be `"memkv"`.                                                                                                                                                                                                                                                                        |
| `prefix`               | `""`    | Key-namespace prefix inside MemKV (multi-fleet separation).                                                                                                                                                                                                                               |
| `n_read_threads`       | `8`     | Load-priority I/O threads (promotions; these gate time-to-first-token).                                                                                                                                                                                                                   |
| `n_write_threads`      | `8`     | Store-priority I/O threads (cascades).                                                                                                                                                                                                                                                    |
| `blocks_per_op`        | `8`     | Max blocks per batched client call.                                                                                                                                                                                                                                                       |
| `load_failure_grace_s` | `60`    | Withhold a failed load's completion from vLLM for this long, keeping its target slots pinned so a server write abandoned by a client timeout cannot land in reused memory. `0` disables.                                                                                                  |
| `max_store_backlog_mb` | `2048`  | Shed store jobs once this much store work is queued/in flight, so a slow MemKV cannot pin the CPU pool solid. Shed blocks just skip MemKV; loads are never shed. Keep below `cpu_bytes_to_use`/2.                                                                                         |
| `lookup_timeout_s`     | `5`, inactive without the patch | Deadline for async exists probes: a lookup still unresolved past it resolves as MISS (recompute) instead of holding the request deferred on RETRY. Any value `<= 0` disables. Requires a vLLM carrying the async-lookup-deadline patch, which no vLLM release includes — contact MinIO support to obtain it. Ignored, with a warning, on an unpatched engine. |

Sizing notes: `block_size` (offloaded block, in tokens) controls the MemKV
value size — prefer values ≥ 1 MiB (long blocks) so reads are
bandwidth-bound, not per-key-overhead-bound. `cpu_bytes_to_use` is the CPU
tier working set; MemKV only sees traffic once blocks cascade (immediately on
store) and promote (on CPU-tier misses).

## Configure — direct offload (`MemKVOffloadingSpec`)

No CPU pool in front: the worker is handed the live GPU KV tensors, gathers
each block into one scratch slot, and MemKV servers move it with one-sided
RDMA. Connection settings come from the same `MEMKV_*` chain.

```bash
PYTHONHASHSEED=0 vllm serve <model> \
  --kv-transfer-config '{
    "kv_connector": "OffloadingConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "spec_name": "MemKVOffloadingSpec",
      "spec_module_path": "memkv_vllm.spec"
    }
  }'
```

All keys sit directly in `kv_connector_extra_config`. Do **not** set
`block_size` here — the offloaded block is the GPU block.

Direct offload supports tensor parallelism only: startup fails with
pipeline parallelism or prefill context parallelism. Block keys include a
fingerprint of the loaded weights (the hub snapshot commit, or a hash of a
local checkpoint), so a new checkpoint under the same model name does not
restore the old one's KV. At startup each worker publishes its key prefix
and the scheduler adopts it, so the two sides cannot drift apart; startup
fails only if ranks disagree (`worker ranks key KV blocks differently`) or
a node publishes nothing. See the gpudirect docs for the details and for
which models each integration supports.

| Config key                    | Default  | Meaning                                                                                                                                                                                                              |
| ----------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `spec_name`                   | —        | Must be `"MemKVOffloadingSpec"`.                                                                                                                                                                                     |
| `spec_module_path`            | —        | Must be `"memkv_vllm.spec"`.                                                                                                                                                                                         |
| `memkv_scratch_medium`        | `"auto"` | Where staging slots live: `"auto"`, `"host"` (pinned host memory) or `"gpu"` (HBM). See below.                                                                                                                       |
| `memkv_num_threads`           | `4`      | Transfer threads.                                                                                                                                                                                                    |
| `memkv_scratch_mb_per_thread` | `128`    | Target scratch per thread. The chunk is whole blocks, at least one and at most 64, so this is a target rather than an exact allocation: a small-block model uses less, and a model whose block is larger than this budget still gets one full block per thread — actual scratch then exceeds the value set. The startup log reports value length, chunk size, and the resulting total.          |
| `memkv_wait_timeout_s`        | `120`    | Bound on `wait()` for outstanding transfers.                                                                                                                                                                         |
| `memkv_lookup_miss_ttl`       | `30`     | Negative-cache seconds for lookups.                                                                                                                                                                                  |
| `memkv_lookup_hit_ttl`        | `30`     | Positive-cache seconds for lookups. A block is a hit only when every TP rank's shard exists; after this many seconds it is checked again, which bounds how long a server-side eviction goes unnoticed. It does not guarantee the load: a shard evicted after the last check still fails it. |
| `memkv_spec_prefix`           | `""`     | Extra key-namespace salt, mixed into the key prefix digest. Set it when two deployments of the same model share a MemKV cluster; it also separates the records the startup check reads.                         |

Each rank drives its RDMA through the NIC on its own GPU's PCIe switch,
read from sysfs at startup; `MEMKV_RDMA_DEVICES` narrows the candidates if
some NICs belong to another fabric. `memkv net topo` prints the layout.

`memkv_scratch_medium` follows from the same read. GPUDirect (`"gpu"`)
reaches line rate only when the NIC and that rank's GPU sit on the same PCIe
switch; otherwise peer-to-peer climbs to the CPU root complex and back. The
`"auto"` default takes `"gpu"` on the ranks that have such a NIC and
`"host"` on the rest, so a host with more ranks than NICs needs no decision.
Pin it to either value to override.

## How it maps

**Secondary tier.** One MemKV value per (block hash, KV cache group): the full
offloaded block across all TP ranks, `stride` bytes of the CPU pool. Keys reuse
vLLM's `FileMapper` naming — `<prefix>/<model>_<configdigest>…/<hash>.bin` — so
runs with the same model and layout share blocks, and incompatible layouts
cannot collide.

**Direct offload.** One value per (block, rank), gathered from the live GPU KV
tensors, so its length comes from the KV layout — the widest cache group's
per-block size — not from a CPU-pool stride. Keys are binary and
rank-qualified, under a digest of model, dtype, TP size, block size and layout;
that is why a restore needs the same TP topology that stored it, and why a
cluster written in this mode is not readable by the tier.

Both modes share the rest. Lookups are batched async `exists` probes, one round
per scheduler step; stores and loads are chunked `batch_put_views` /
`batch_get_into` calls with no Python-side copies. Capacity is governed
server-side by MemKV lane eviction; a value evicted between lookup and load
degrades to recompute, never corruption.

## Test

```bash
cargo build -p memkv            # loopback server binary for the fixture
maturin develop --manifest-path vllm-plugin/Cargo.toml
pytest vllm-plugin/tests -v    # requires vllm importable; Linux for data path
```

