Metadata-Version: 2.5
Name: kvplan
Version: 0.2.0
Summary: Plan vLLM deployments without the GPU: will it start, how much KV cache, how many users.
Project-URL: Homepage, https://github.com/ShawnNotFound/kvplan
Project-URL: Repository, https://github.com/ShawnNotFound/kvplan
Project-URL: Issues, https://github.com/ShawnNotFound/kvplan/issues
Project-URL: Changelog, https://github.com/ShawnNotFound/kvplan/blob/master/CHANGELOG.md
Author: Shawn (Shuaijie Shao)
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: gpu,inference,kv-cache,llm,memory,vllm
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Provides-Extra: test
Requires-Dist: pytest>=8; extra == 'test'
Description-Content-Type: text/markdown

# kvplan

Plan vLLM deployments without the GPU.

Tell kvplan a model, your GPUs and your vLLM settings. It tells you whether vLLM will start, how much KV cache you will get, and how many full-length requests fit at once. It reads the model's real files from Hugging Face (only the headers, never the weights), so it works for new models the day they are uploaded.

```
$ kvplan check unsloth/Meta-Llama-3.1-8B-Instruct --gpu L4

unsloth/Meta-Llama-3.1-8B-Instruct on 1 x L4 (max_model_len 131072, max_num_seqs 256, gpu_memory_utilization 0.92)

  budget         20.69 GiB of 22.49 GiB per GPU
  weights        14.96 GiB per GPU (exact, from the weight files)
  runtime        0.83 GiB estimated: activation peak 0.22 GiB, torch.compile 0.11 GiB, CUDA graphs 0.25 GiB, other 0.25 GiB
  KV cache       4.91 GiB available (131,072 bytes per token per GPU, auto, block size 16)
  capacity       40,208 tokens; max concurrency at 131,072 tokens per request: 0.31x

  RESULT         vLLM will not start
  error          one request at max_model_len=131072 needs 16.00 GiB of KV cache, but only 4.91 GiB is available; vLLM's estimate of the longest length that fits is 40192
  try            --max-model-len 40192 (or lower)
  try            --kv-cache-dtype fp8 halves the KV cache
  try            --tensor-parallel-size 2 on 2 GPUs splits weights and KV further
```

> **Status: alpha.** Weight and KV sizes are computed exactly from the model files. The runtime overheads (activation peak, torch.compile buffers, CUDA graphs) are fitted to vLLM 0.32 startup logs from text-model runs on RTX A6000 and A5000 GPUs, and the multimodal encoder memory is derived from vLLM's code and checked on 27 more runs; see [Accuracy](https://github.com/ShawnNotFound/kvplan#accuracy) for what that does and does not cover.

## Why

vLLM decides its KV cache size at startup, on the real GPU: it reserves `gpu_memory_utilization` of the **total** GPU memory, loads the weights, profiles a forward pass, and gives whatever is left to the KV cache. If one request at `max_model_len` doesn't fit, it refuses to start. You find this out after renting or buying the GPU.

Other tools approximate this with parameter count × bytes per parameter plus a fixed overhead. That misses quantisation scales, padded vocabularies, tied embeddings, sliding-window layers, latent (MLA) caches and how tensor parallelism splits or replicates KV heads. kvplan reproduces vLLM's own accounting instead. See [docs/accounting.md](https://github.com/ShawnNotFound/kvplan/blob/master/docs/accounting.md) for every rule and the vLLM source it comes from.

## Install

```
pip install kvplan
```

or the latest development version: `pip install git+https://github.com/ShawnNotFound/kvplan`.

No dependencies beyond Python 3.10+. For gated models (Llama, Gemma), run `kvplan login` or set `HF_TOKEN`; see [Gated models](#gated-models).

## Use

```
kvplan check MODEL [--gpu NAME | --gpu-mem-gib GIB] [vLLM options] [--download]
kvplan inspect MODEL
kvplan recommend [--gpu NAME] [--gpus N|any] [--task chat|code|vision|audio|omni] [--priority quality|balanced|speed]
kvplan search WORDS...
kvplan download MODEL [--local-dir DIR]
kvplan login
```

`MODEL` is a Hugging Face repo id, a local directory or file, or just a model name. For a name without an owner, such as `Qwen2.5-7B-Instruct`, kvplan uses the most downloaded repo with exactly that name and says which. If nothing matches, or a repo id is misspelled, it lists the closest popular models instead (and in a terminal, lets you pick one). Without `--gpu` or `--gpu-mem-gib`, kvplan plans for the GPUs on this machine (`--gpu auto`). `check` understands the vLLM options that change memory: `-tp/--tensor-parallel-size`, `--max-model-len`, `--max-num-seqs`, `--max-num-batched-tokens`, `--gpu-memory-utilization`, `--kv-cache-dtype`, `--block-size`, `--enforce-eager`. Add `--json` for scripts and CI. The exit code is 0 when vLLM should start and 1 when it won't. It is 3 ("start not confirmed", `"starts": null` in the JSON) when a term kvplan cannot size could decide the outcome: an encoder whose startup profiling it cannot size, an architecture vLLM has no native implementation of, or attention geometry read from weight shapes. The output then names the unsized term, the memory left under kvplan's estimates, and what to do. Exit code 2 is a kvplan error.

In a terminal the result is coloured: green when vLLM should start, red when it won't, yellow when the start is not confirmed, with errors in red and warnings in yellow. `--color never` (or the `NO_COLOR` environment variable) turns colour off and `--color always` forces it; files, pipes and the JSON, CSV and Markdown formats are never coloured.

For the most precise result, give the GPU memory exactly as CUDA reports it (`--gpu-mem-gib`); the `--gpu` presets use the slightly larger `nvidia-smi` totals.

On the machine that will run vLLM, `--gpu auto` reads the GPUs from `nvidia-smi`, including memory other processes already hold. vLLM reserves a fraction of *total* memory, so a desktop session on a laptop GPU or another job on a shared server can stop it from starting even when the model fits:

```
$ kvplan check Qwen/Qwen2.5-1.5B-Instruct --gpu auto
...
  gpu 0          NVIDIA GeForce RTX 5090 Laptop GPU: 24,463 MiB total, 4,478 MiB in use (nvidia-smi)
  RESULT         vLLM will not start
  error          gpu_memory_utilization=0.92 asks for 21.98 GiB, but only about 19.02 GiB is free after the CUDA context and other processes. ...
  try            lower --gpu-memory-utilization to 0.79
```

With tensor parallelism it uses the first `-tp` GPUs, or the ones you name with `--gpu-index 2,3`.

```
$ kvplan inspect openai/gpt-oss-20b
openai/gpt-oss-20b
  architecture   gpt_oss (read from config)
  layers         24: 12 sliding, 12 full (window 128)
  attention      64 heads, 8 KV heads, head_dim 64
  experts        32 (all resident in GPU memory)
  context        131072
  KV per token   49,152 bytes per GPU at tp=1 (auto)
  weights        12.82 GiB per GPU at tp=1 (checkpoint 12.82 GiB in 3 file(s))
```

When vLLM should start, `check` ends with the command to download the model and the matching `vllm serve` line:

```
  RESULT         vLLM should start

  download       kvplan download Qwen/Qwen2.5-1.5B-Instruct   (or add --download to this check)
  serve          vllm serve Qwen/Qwen2.5-1.5B-Instruct --max-model-len 8192
```

`kvplan download` fetches the weight files kvplan sized plus the configs, tokenizer and chat template vLLM needs, and nothing else (no duplicate `.bin`, `.pth` or GGUF copies). With `huggingface_hub` installed (any machine with vLLM has it) the files go into the Hugging Face cache, so `vllm serve REPO_ID` finds them; otherwise, or with `--local-dir`, into a directory, resuming interrupted downloads.

## Find a model

`kvplan recommend` lists popular Hugging Face models that vLLM should start on your GPUs, each one checked the same way as `kvplan check`. In a terminal it asks what you need: how many GPUs to use (one, all, or the fewest that work), the task, quality or speed, and the context length. Pass the answers as options, or `--no-input` for the defaults, to skip the questions.

```
$ kvplan recommend --gpu A5000 --task vision --priority speed --max-model-len 4096

vision models for 1 x A5000, speed (smaller models first: faster replies and more requests at once), 4,096-token context

   #  model                                                  size GPUs weights/GPU   KV tokens requests at once
   1  Qwen/Qwen2.5-VL-3B-Instruct-AWQ                        3.8B    1    3.29 GiB     441,140           107.7x
   2  Qwen/Qwen2.5-VL-3B-Instruct                            3.8B    1    7.12 GiB     348,207            85.0x
   3  Qwen/Qwen2.5-VL-7B-Instruct                            8.3B    1   15.57 GiB      65,440            16.0x

  Checked 28 of 32 candidates from Hugging Face's most downloaded image-text-to-text models. Ranked by size and downloads, not by benchmark scores.
  ...
Number for full details, d<number> to download, Enter to finish:
```

Pick a number to see the full `kvplan check` output, or `d` and a number to download it. Candidates are the most downloaded models for the task on Hugging Face. kvplan leaves out models vLLM has no native implementation of, GGUF and MLX conversions, base and draft models, and anything whose weights cannot fit, then orders the rest by size: larger first for `quality`, smaller first for `speed`, about a third of the memory for `balanced`. Size and popularity are only rough signs of quality, so treat the list as a shortlist to try, not a ranking. Image generation (Stable Diffusion, FLUX) is not served by vLLM, so kvplan does not recommend those.

`kvplan search llama 3.1 8b instruct` finds repos by a loose name, most downloaded first.

## Gated models

Gated models (Llama, Gemma and others) need a Hugging Face token, and you need to accept the model's licence on its Hugging Face page. Either:

- run `kvplan login` and paste a read token from https://huggingface.co/settings/tokens. kvplan checks it and saves it where `huggingface-cli login` does (`$HF_HOME/token`, or `HF_TOKEN_PATH`), so vLLM and `huggingface-cli` use it too; or
- set `HF_TOKEN` in your environment (for example in your shell profile). It takes precedence over a saved token.

`--token` on any command overrides both. With a token, `recommend` also considers gated models.

## Use it from scripts and apps

**Output.** Every command that prints results takes `--format text|json|jsonl|csv|markdown` (`--json` is short for `--format json`) and `-o/--output PATH`. The format follows the file name when you don't give one (`-o results.csv` writes CSV). A path ending in `/`, or an existing directory, gets one file per model. With `--append`, CSV and JSON Lines output is added to the file, so repeated runs build a log. JSON holds everything kvplan computed; the other formats hold one flat row per model (model, GPU, tp, context, verdict, weights, KV cache, KV tokens, concurrency, problems, `vllm serve` command).

**Many models at once.** `kvplan check --models FILE` checks every model in a file and prints one table, or writes one CSV, JSON or Markdown file. The file is a text file with one model per line, a CSV with a `model` column plus any option columns (`gpu`, `tp`, `max_model_len`, `kv_cache_dtype`, ...), or a JSON list of names or objects; `-` reads standard input. A model that cannot be read gets an `error` row instead of stopping the run. The exit code is the worst result: 0 all start, 3 some not confirmed, 1 some will not start, 2 some could not be checked.

```
$ kvplan check --models models.csv -o results/ --format csv
kvplan: wrote results/kvplan-check.csv
```

**Config file.** Put defaults in `~/.config/kvplan/config.toml` (or `config.json`, or any file named by `KVPLAN_CONFIG` or `--config`). Top-level keys apply to every command; a table named after a command applies to that command only. Options on the command line still win.

```toml
gpu = "A6000"
format = "csv"
output = "results/"

[check]
max_model_len = 8192
tp = 2

[recommend]
task = "code"
priority = "speed"
```

TOML needs Python 3.11 or newer; JSON works everywhere.

**Python.** The command line is a thin layer over `kvplan`'s Python API, which returns plain data and never prints or asks:

```python
import kvplan

r = kvplan.check("Qwen/Qwen2.5-7B-Instruct", gpu="A6000", max_model_len=8192)
r.starts            # True, False, or None for "start not confirmed"
r.kv_tokens         # KV-cache capacity in tokens
r.to_dict()         # everything, as JSON-ready data (the same as --format json)
r.serve_command     # "vllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 8192"

kvplan.inspect("openai/gpt-oss-20b")["kv_bytes_per_token"]
kvplan.search("llama 3.1 8b instruct", limit=5)
kvplan.recommend(task="chat", priority="quality", gpu="A6000", gpus="any")["recommendations"]
kvplan.download("Qwen/Qwen2.5-7B-Instruct")      # returns where the files went
```

Without `gpu` or `gpu_mem_gib`, `check` plans for the GPUs on the machine it runs on. Errors are `kvplan.KvplanError` (a tuple to catch); an unknown model raises `kvplan.ModelNotFound`, whose `suggestions` lists the closest popular models.

## What it covers

| | |
|---|---|
| Model metadata | config.json, including nested language-model configs (`text_config`, `thinker_config`, `llm_config`, `language_config`) resolved as vLLM does; safetensors headers; GGUF headers. A tensor-shape fallback when config.json lacks fields (flagged as uncertain), and a refusal when config and weights disagree |
| Multimodal | checkpoint parts vLLM does not load (Omni speech generators) are skipped; vLLM's startup encoder profiling is reproduced for the Qwen2-VL family (dummy items, encoder memory), approximated for Whisper-style audio and flagged as uncertain for other encoders; `--limit-mm-per-prompt`, `--mm-processor-kwargs` and `--skip-mm-profiling` as in vLLM |
| Attention | MHA, GQA, MQA, sliding-window and hybrid full/sliding layouts (Gemma 2/3, gpt-oss, Cohere2, Qwen2/3 with sliding window), MLA (DeepSeek V2/V3, Kimi K2) |
| Weights | exact bytes per tensor, 16-bit loading of fp32 checkpoints, quantised formats at their stored size, vocab padding, tied embeddings, MTP layers excluded, tensor-parallel sharding and KV-head replication |
| vLLM | utilisation budget, free-memory check, KV block grouping and rounding, max concurrency, the max_model_len check and its suggested length |

In a sample of the 300 most-downloaded text-generation models on Hugging Face, kvplan's sources describe 93% without a token and 98% with one (details in [docs/coverage.md](https://github.com/ShawnNotFound/kvplan/blob/master/docs/coverage.md)).

**Not yet supported:** models with Mamba or linear-attention layers (Qwen3-Next, Jamba, Nemotron-H, Falcon-H1), DeepSeek V3.2 sparse attention, Gemma 4 (full-attention layers with their own head size), encoder-decoder models, pipeline/expert/data parallelism, LoRA, speculative decoding, and llama.cpp. kvplan says so instead of guessing.

## Accuracy

Checked against vLLM 0.32.0 on RTX A6000 and RTX A5000 GPUs.

- **Text models** (Qwen2.5 1.5B, Gemma 2 2B, Llama 3.2 3B, Llama 3.1 8B; CUDA graphs and eager; tensor parallelism 1 and 2): kvplan's KV-cache token count is within 1.9% of vLLM's in all 15 runs (mean 0.5%). These runs were used for calibration, so they are in-sample. On two RTX A6000s at tensor parallelism 2, which were not used, Llama 3.1 8B is within 0.22%.
- **Multimodal models:** kvplan reproduces the dummy images and videos vLLM profiles at startup for Qwen2-VL, Qwen2.5-VL and Qwen2.5-Omni, including the effect of `--mm-processor-kwargs` max_pixels and `--limit-mm-per-prompt`, and derives the vision encoder's memory from vLLM's code. On 16 fit runs (Qwen2.5-VL 7B, Qwen2.5-Omni 3B and 7B, Qwen2-Audio 7B on an A6000) it is within 0.72%; on 11 held-out runs (A5000, tensor parallelism 2) within 2.24%. Other encoders are not sized: for MiniCPM-o 2.6 kvplan was 19–23% high, and it now warns that the figure is uncertain for any encoder it cannot size.
- **MoE and quantised models:** on 12 runs of Granite 3.1 1B-A400M (MoE), Qwen2.5 1.5B AWQ and GPTQ, and Qwen3 1.7B FP8, none of them used for fitting, kvplan is within 1.01% (mean 0.33%). An fp8 KV cache did not start on the RTX A6000 or A5000 (compute capability 8.6) with vLLM 0.32, so kvplan says "start not confirmed" for it on GPUs older than 8.9.

Not measured yet: large MoE and MLA models, the size of an fp8 KV cache, other vLLM versions, Hopper or Blackwell GPUs. The full table, method and known gaps are in [docs/validation.md](https://github.com/ShawnNotFound/kvplan/blob/master/docs/validation.md).

## Roadmap

- v0.2 (released): model search and recommendations, downloads, gated-model login, output formats and files, config files and a Python API
- v0.3: a measurement script anyone can run on their GPU, llama.cpp profile, published validation table
- v0.4: `place`: several models across several machines with mixed GPUs

## Licence

Apache-2.0
