Metadata-Version: 2.4
Name: memkv-vllm
Version: 1.0.6
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Rust
Classifier: Operating System :: POSIX :: Linux
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Dist: pytest >=7 ; extra == 'test'
Provides-Extra: test
Summary: vLLM native KV offloading secondary tier backed by the MemKV context memory store (TieringOffloadingSpec).
Author: MinIO, Inc.
License: LicenseRef-Proprietary
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/miniohq/memkv
Project-URL: Documentation, https://docs.min.io/memkv/

# memkv-vllm

MemKV secondary tier for vLLM's **native KV offloading** feature
(`OffloadingConnector` + `TieringOffloadingSpec`). vLLM offloads completed KV
blocks from GPU to a pinned CPU pool; this plugin adds a `memkv` tier behind
that pool, so blocks evicted from CPU (or produced by another vLLM instance)
are served from a shared MemKV cluster over RDMA instead of being recomputed.

This is MemKV's third vLLM-adjacent integration and does not replace the
others: `lmcache-plugin/` (LMCache connector path, current production
recommendation) and `plugin/` (NIXL storage backend for Dynamo KVBM).

## Requirements

- vLLM main at or after commit `6cf7b26bd` (v0.23.1rc1.dev962). The tiering
  framework (`vllm.v1.kv_offload.tiering`) has shipped in releases since
  v0.22.0, but the secondary-tier API this plugin implements (LookupResult
  lookups, ScheduleEndContext, get_stats/build_metric_definitions) is newer
  than the latest stable release (v0.24.0); it first appears in the v0.25.0
  release candidates. The offloading spec API is experimental upstream and
  has already been reworked once (`a9531edfa`, between v0.25.1 and v0.26.0 —
  renamed `OffloadingSpec` construction and several fields this plugin
  reads). Rather than pinning to one side of that boundary, `_vllm_compat.py`
  detects and handles both; confirmed working on v0.25.0, v0.25.1, v0.26.0,
  v0.27.0, and v0.27.1.
- Linux with RDMA userspace (`libibverbs`, `ibverbs-providers`) for the data
  path. The client falls back to TCP where RDMA is unavailable.
- `PYTHONHASHSEED` pinned (e.g. `0`) on every vLLM instance that should share
  KV — vLLM's block content hashes are seeded per process otherwise.

## Install

```bash
pip install memkv-vllm
```

The wheel registers itself via the `vllm.general_plugins` entry point
`memkv_tier` — no vLLM patches, no extra flags. If you restrict plugins with
`VLLM_PLUGINS`, include `memkv_tier` in the list.

## Configure

MemKV connection settings come from the standard `MEMKV_CONFIG` yaml or
`MEMKV_*` env-var chain (`MEMKV_SERVERS`, `MEMKV_AUTH_KEY`, …) — identical to
the other MemKV plugins. The tier entry in `secondary_tiers` carries only
tier tuning:

```bash
PYTHONHASHSEED=0 vllm serve <model> \
  --kv-transfer-config '{
    "kv_connector": "OffloadingConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "spec_name": "TieringOffloadingSpec",
      "cpu_bytes_to_use": 10737418240,
      "block_size": 256,
      "secondary_tiers": [
        {"type": "memkv", "n_read_threads": 8, "n_write_threads": 8}
      ]
    }
  }'
```

| Tier config key        | Default | Meaning                                                                                                                                                                                                                                                                                   |
| ---------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type`                 | —       | Must be `"memkv"`.                                                                                                                                                                                                                                                                        |
| `prefix`               | `""`    | Key-namespace prefix inside MemKV (multi-fleet separation).                                                                                                                                                                                                                               |
| `n_read_threads`       | `8`     | Load-priority I/O threads (promotions; these gate time-to-first-token).                                                                                                                                                                                                                   |
| `n_write_threads`      | `8`     | Store-priority I/O threads (cascades).                                                                                                                                                                                                                                                    |
| `blocks_per_op`        | `8`     | Max blocks per batched client call.                                                                                                                                                                                                                                                       |
| `load_failure_grace_s` | `60`    | Withhold a failed load's completion from vLLM for this long, keeping its target slots pinned so a server write abandoned by a client timeout cannot land in reused memory. `0` disables.                                                                                                  |
| `max_store_backlog_mb` | `2048`  | Shed store jobs once this much store work is queued/in flight, so a slow MemKV cannot pin the CPU pool solid. Shed blocks just skip MemKV; loads are never shed. Keep below `cpu_bytes_to_use`/2.                                                                                         |
| `lookup_timeout_s`     | `5`     | Deadline for async exists probes: a lookup still unresolved past it resolves as MISS (recompute) instead of holding the request deferred on RETRY. Any value `<= 0` disables. Requires a vLLM with the async-lookup-deadline patch (`upstream/0002`); ignored (with a warning) otherwise. |

Sizing notes: `block_size` (offloaded block, in tokens) controls the MemKV
value size — prefer values ≥ 1 MiB (long blocks) so reads are
bandwidth-bound, not per-key-overhead-bound. `cpu_bytes_to_use` is the CPU
tier working set; MemKV only sees traffic once blocks cascade (immediately on
store) and promote (on CPU-tier misses).

## How it maps

One MemKV value per (block hash, KV cache group): the full offloaded block
across all TP ranks (`stride` bytes of the CPU pool). Keys reuse vLLM's
`FileMapper` naming — `<prefix>/<model>_<configdigest>…/<hash>.bin` — so runs
with the same model/layout share blocks and incompatible layouts can't
collide. Lookups are batched async `exists` probes (one round per scheduler
step); stores/loads are chunked `batch_put_views`/`batch_get_into` calls
straight against the pool slots (no Python-side copies). Capacity is governed
server-side by MemKV lane eviction; a value evicted between lookup and load
degrades to recompute, never corruption.

## Test

```bash
cargo build -p memkv            # loopback server binary for the fixture
maturin develop --manifest-path vllm-plugin/Cargo.toml
pytest vllm-plugin/tests -v    # requires vllm importable; Linux for data path
```

