Metadata-Version: 2.4
Name: llm-vram-fit
Version: 0.1.0
Summary: VRAM calculation and local LLM fit analyzer
Author: an1zy
License: MIT
Project-URL: Homepage, https://github.com/an1zy/vram-fit
Project-URL: Repository, https://github.com/an1zy/vram-fit
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: typer
Requires-Dist: rich
Requires-Dist: pydantic>=2.0
Requires-Dist: httpx
Requires-Dist: psutil
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Dynamic: license-file

# vram-fit

Stop guessing which local LLM fits your GPU. Scan VRAM, calculate GGUF + KV-cache math, and get exact Ollama & llama.cpp commands.

[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg?style=flat-square)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg?style=flat-square)](LICENSE)
[![Agent Skill](https://img.shields.io/badge/agent_skill-vram--fit-purple.svg?style=flat-square)](skills/vram-fit/SKILL.md)
[![Slop Score: 0%](https://img.shields.io/badge/slop_score-0%25_clean-brightgreen.svg?style=flat-square)](https://github.com/an1zy/vram-fit)

![VRAM Calculation Architecture](assets/vram-breakdown.svg)

---

## Quick Start

Install via pip:

```bash
pip install llm-vram-fit
```

Or run directly without installation:

```bash
uvx llm-vram-fit fit
```

Install as an Agent Skill for coding assistants (Claude Code, Cursor, Codex):

```bash
npx skills add an1zy/vram-fit
```

---

## Features

- **Hardware Detection**: Automatically detects NVIDIA GPUs via `nvidia-smi`, Apple Silicon unified memory via macOS metrics, or CPU-only profiles with system RAM inspection.
- **Accurate GQA Memory Formula**: Computes exact Grouped-Query Attention (GQA) KV-cache footprints across context lengths (4k to 128k) and KV data types (`fp16`, `q8_0`, `q4_0`).
- **GGUF Weight Math**: Pre-calculated Bits-Per-Weight (BPW) tables for `IQ4_XS`, `Q4_K_M`, `Q5_K_M`, `Q6_K`, and `Q8_0` with metadata overhead factors.
- **Offload Status Categorization**: Classifies models into `FULL_GPU` (100% VRAM), `HYBRID` (partial GPU offloading), or `OOM` (out of memory).
- **Execution Ready Commands**: Emits copy-pasteable environment variables and flags for PowerShell, Bash, Ollama, and llama.cpp `llama-server`.
- **Benchmark Utility**: Built-in latency and tokens-per-second (`prompt_tps`, `gen_tps`) benchmarking against local Ollama instances.

---

## CLI Usage

### 1. Fit Analysis

Analyze memory fit for curated models on current hardware:

```bash
vram-fit fit
```

Filter by category (`coding`, `reasoning`, `general`) or view only models that fit 100% in VRAM:

```bash
vram-fit fit --category coding --only-gpu
```

![vram-fit fit demo](assets/demo-fit.svg)

### 2. Hardware Simulation

Simulate target hardware setups before purchasing GPUs or renting cloud instances:

```bash
vram-fit fit --vram 24 --ram 64 --quant Q5_K_M --ctx 16384
```

### 3. Generate Launch Commands

Generate production launch commands for a specific model ID or Ollama tag:

```bash
vram-fit cmd qwen2.5-coder-7b --vram 8 --ctx 8192 --kv q8_0
```

![vram-fit cmd demo](assets/demo-cmd.svg)

### 4. Hardware Scan

Inspect host GPU VRAM, system RAM, CPU cores, and unified memory configuration:

```bash
vram-fit scan
```

### 5. Benchmark Ollama

Benchmark prompt evaluation and token generation speed on a running Ollama server:

```bash
vram-fit bench qwen2.5-coder:7b --url http://127.0.0.1:11434
```

---

## Agent and Machine-Readable JSON Mode

Every command supports `--json` for integration with autonomous agents, workflows, and automated pipelines:

```bash
vram-fit scan --json
vram-fit fit --only-gpu --json
```

Example JSON payload:

```json
{
  "hardware": {
    "gpu_name": "NVIDIA GeForce RTX 3060 Ti",
    "gpu_vendor": "NVIDIA",
    "vram_total_gb": 8.0,
    "vram_free_gb": 5.23,
    "ram_total_gb": 31.95,
    "ram_free_gb": 17.09,
    "cpu_cores": 8,
    "unified_memory": false
  },
  "results": [
    {
      "model_id": "qwen2.5-coder-7b",
      "model_name": "Qwen 2.5 Coder 7B",
      "category": "coding",
      "status": "FULL_GPU",
      "weights_gb": 4.84,
      "kv_cache_gb": 0.23,
      "total_gb": 5.57,
      "gpu_layers_pct": 100.0,
      "max_ctx_full_gpu": 32768,
      "ollama_cmd": "OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama run qwen2.5-coder:7b",
      "llama_cpp_cmd": "llama-cli -m qwen2.5-coder-7b.gguf -ngl 28 -c 8192 -ctk q8_0 -ctv q8_0 --flash-attn"
    }
  ]
}
```

---

## Model Catalog

The built-in catalog includes 10 popular local models across size categories:
- **4B–9B**: `gemma3:4b`, `qwen2.5-coder:7b`, `qwen3:8b`, `deepseek-r1:8b`
- **12B–14B**: `gemma3:12b`, `phi4:14b`, `qwen2.5-coder:14b`
- **MoE**: `qwen3:30b-a3b` (Active: 3.3B)
- **27B–32B**: `gemma3:27b`, `qwen2.5-coder:32b`

---

## License

MIT © 2026 [an1zy](https://github.com/an1zy)
