Metadata-Version: 2.4
Name: memkv-vllm
Version: 1.0.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Rust
Classifier: Operating System :: POSIX :: Linux
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Dist: pytest >=7 ; extra == 'test'
Provides-Extra: test
Summary: MemKV backend for vLLM native KV offloading: a secondary tier behind the CPU pool (TieringOffloadingSpec) or MemKV as the offload medium itself (MemKVOffloadingSpec).
Author: MinIO, Inc.
License: LicenseRef-Proprietary
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/miniohq/memkv
Project-URL: Documentation, https://docs.min.io/memkv/

# memkv-vllm

Two ways to back vLLM's **native KV offloading** with a shared MemKV cluster.
Both ship in this one wheel and both are selected purely through
`--kv-transfer-config`. Neither needs a patched vLLM to run; the tier's
optional `lookup_timeout_s` deadline is the one exception, noted where it
appears below.

| Mode | Selected by | KV path | Use when |
| --- | --- | --- | --- |
| **Secondary tier** | `"spec_name": "TieringOffloadingSpec"` with a `memkv` entry in `secondary_tiers` | GPU → vLLM's pinned CPU pool → MemKV. RDMA, TCP fallback | You want vLLM's CPU pool in front as a residency layer. |
| **Direct offload** | `"spec_name": "MemKVOffloadingSpec"`, `"spec_module_path": "memkv_vllm.spec"` | GPU ↔ MemKV, staged through a small scratch buffer. RDMA only | You want no CPU pool holding a second copy of everything. |

They are alternatives — pick one per engine. The two modes write different
key layouts, so a cluster written by one is not read by the other.

This is MemKV's third vLLM-adjacent integration and does not replace the
others: `memkv-lmcache` (the LMCache connector path, currently the
production recommendation) and the MemKV NIXL storage backend for Dynamo
KVBM, which ships as a shared library rather than a wheel.

## Requirements

- **vLLM 0.25.0 or newer.** Tested against 0.25.0, 0.25.1, 0.26.0, 0.27.0,
  and 0.27.1. Releases before 0.25.0 do not work: the tiering framework
  itself has shipped since 0.22.0, but the secondary-tier interfaces this
  plugin implements — `LookupResult` lookups, `ScheduleEndContext`, and
  `get_stats`/`build_metric_definitions` — arrived in 0.25.0. vLLM reworked the offloading-spec API again
  in 0.26.0; the plugin detects which side it is running against and adapts,
  so no version pin is needed in either direction. The offloading-spec API
  is still experimental upstream, so releases past 0.27.1 are untested.
- **Python 3.10 or newer, on Linux** (x86-64 or aarch64). The wheels on PyPI
  are `manylinux_2_28`, so glibc 2.28 or newer — RHEL/Rocky 8+, Debian 10+,
  Ubuntu 18.10+. (The copies bundled in the MemKV deb/rpm are built without a
  manylinux tag and target the same hosts.)
- **RDMA userspace** (`libibverbs`, `ibverbs-providers`) for the data path.
  The wheels load it lazily, so they import on any Linux host. What happens
  without a reachable NIC differs by mode: the secondary tier falls back to
  TCP, while direct offload is RDMA-only and fails at startup rather than
  running slowly.
- **`PYTHONHASHSEED` pinned** (e.g. `0`) on every vLLM instance that should
  share KV — vLLM's block content hashes are seeded per process otherwise.
- **A MemKV license.** The client verifies one when it builds its engine, so
  a plugin with none available fails at startup with `no license found`
  rather than degrading. Sources, in order: the `license:` field of
  `MEMKV_CONFIG`, then `MEMKV_LICENSE` (an inline JWT or a path to a file
  holding one), then `MINIO_LICENSE` / `AISTOR_LICENSE` and the standard
  `minio.license` file locations. Contact MinIO to obtain one.

## Install

```bash
pip install memkv-vllm
```

The wheel registers itself via the `vllm.general_plugins` entry point
`memkv_tier` — no vLLM patches, no extra flags. If you restrict plugins with
`VLLM_PLUGINS`, include `memkv_tier` in the list.

## Configure — secondary tier (`TieringOffloadingSpec`)

MemKV connection settings come from the standard `MEMKV_CONFIG` yaml or
`MEMKV_*` env-var chain (`MEMKV_SERVERS`, `MEMKV_AUTH_KEY`, …) — identical to
the other MemKV plugins. The tier entry in `secondary_tiers` carries only
tier tuning:

```bash
PYTHONHASHSEED=0 vllm serve <model> \
  --kv-transfer-config '{
    "kv_connector": "OffloadingConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "spec_name": "TieringOffloadingSpec",
      "cpu_bytes_to_use": 10737418240,
      "block_size": 256,
      "secondary_tiers": [
        {"type": "memkv", "n_read_threads": 8, "n_write_threads": 8}
      ]
    }
  }'
```

| Tier config key        | Default | Meaning                                                                                                                                                                                                                                                                                   |
| ---------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type`                 | —       | Must be `"memkv"`.                                                                                                                                                                                                                                                                        |
| `prefix`               | `""`    | Key-namespace prefix inside MemKV (multi-fleet separation).                                                                                                                                                                                                                               |
| `n_read_threads`       | `8`     | Load-priority I/O threads (promotions; these gate time-to-first-token).                                                                                                                                                                                                                   |
| `n_write_threads`      | `8`     | Store-priority I/O threads (cascades).                                                                                                                                                                                                                                                    |
| `blocks_per_op`        | `8`     | Max blocks per batched client call.                                                                                                                                                                                                                                                       |
| `load_failure_grace_s` | `60`    | Withhold a failed load's completion from vLLM for this long, keeping its target slots pinned so a server write abandoned by a client timeout cannot land in reused memory. `0` disables.                                                                                                  |
| `max_store_backlog_mb` | `2048`  | Shed store jobs once this much store work is queued/in flight, so a slow MemKV cannot pin the CPU pool solid. Shed blocks just skip MemKV; loads are never shed. Keep below `cpu_bytes_to_use`/2.                                                                                         |
| `lookup_timeout_s`     | `5`, inactive without the patch | Deadline for async exists probes: a lookup still unresolved past it resolves as MISS (recompute) instead of holding the request deferred on RETRY. Any value `<= 0` disables. Requires a vLLM carrying the async-lookup-deadline patch, which no vLLM release includes — contact MinIO support to obtain it. Ignored, with a warning, on an unpatched engine. |

Sizing notes: `block_size` (offloaded block, in tokens) controls the MemKV
value size — prefer values ≥ 1 MiB (long blocks) so reads are
bandwidth-bound, not per-key-overhead-bound. `cpu_bytes_to_use` is the CPU
tier working set; MemKV only sees traffic once blocks cascade (immediately on
store) and promote (on CPU-tier misses).

## Configure — direct offload (`MemKVOffloadingSpec`)

No CPU pool in front: the worker is handed the live GPU KV tensors, gathers
each block into one scratch slot, and MemKV servers move it with one-sided
RDMA. Connection settings come from the same `MEMKV_*` chain.

```bash
PYTHONHASHSEED=0 vllm serve <model> \
  --kv-transfer-config '{
    "kv_connector": "OffloadingConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "spec_name": "MemKVOffloadingSpec",
      "spec_module_path": "memkv_vllm.spec",
      "memkv_scratch_medium": "host"
    }
  }'
```

All keys sit directly in `kv_connector_extra_config`. Do **not** set
`block_size` here — the offloaded block is the GPU block.

| Config key                    | Default  | Meaning                                                                                                                                                                                                              |
| ----------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `spec_name`                   | —        | Must be `"MemKVOffloadingSpec"`.                                                                                                                                                                                     |
| `spec_module_path`            | —        | Must be `"memkv_vllm.spec"`.                                                                                                                                                                                         |
| `memkv_scratch_medium`        | `"host"` | Where staging slots live: `"host"` (pinned host memory) or `"gpu"`. See below.                                                                                                                                       |
| `memkv_num_threads`           | `4`      | Transfer threads.                                                                                                                                                                                                    |
| `memkv_scratch_mb_per_thread` | `128`    | Target scratch per thread. The chunk is whole blocks, at least one and at most 64, so this is a target rather than an exact allocation: a small-block model uses less, and a model whose block is larger than this budget still gets one full block per thread — actual scratch then exceeds the value set. The startup log reports value length, chunk size, and the resulting total.          |
| `memkv_wait_timeout_s`        | `120`    | Bound on `wait()` for outstanding transfers.                                                                                                                                                                         |
| `memkv_lookup_miss_ttl`       | `30`     | Negative-cache seconds for lookups.                                                                                                                                                                                  |
| `memkv_spec_prefix`           | `""`     | Extra key-namespace salt, mixed into the key prefix digest (multi-fleet separation).                                                                                                                                 |

`memkv_scratch_medium` is the one deployment-specific choice. GPUDirect
(`"gpu"`) reaches line rate only when the NIC and that rank's GPU sit on the
same PCIe switch; otherwise peer-to-peer climbs to the CPU root complex and
back, and one such rank drags the whole rail down. On a host with more ranks
than NICs — the ordinary case — keep the `"host"` default. Set `"gpu"` on
rail-optimised nodes that pair one NIC per GPU under a shared switch.

## How it maps

**Secondary tier.** One MemKV value per (block hash, KV cache group): the full
offloaded block across all TP ranks, `stride` bytes of the CPU pool. Keys reuse
vLLM's `FileMapper` naming — `<prefix>/<model>_<configdigest>…/<hash>.bin` — so
runs with the same model and layout share blocks, and incompatible layouts
cannot collide.

**Direct offload.** One value per (block, rank), gathered from the live GPU KV
tensors, so its length comes from the KV layout — the widest cache group's
per-block size — not from a CPU-pool stride. Keys are binary and
rank-qualified, under a digest of model, dtype, TP size, block size and layout;
that is why a restore needs the same TP topology that stored it, and why a
cluster written in this mode is not readable by the tier.

Both modes share the rest. Lookups are batched async `exists` probes, one round
per scheduler step; stores and loads are chunked `batch_put_views` /
`batch_get_into` calls with no Python-side copies. Capacity is governed
server-side by MemKV lane eviction; a value evicted between lookup and load
degrades to recompute, never corruption.

## Test

```bash
cargo build -p memkv            # loopback server binary for the fixture
maturin develop --manifest-path vllm-plugin/Cargo.toml
pytest vllm-plugin/tests -v    # requires vllm importable; Linux for data path
```

