Metadata-Version: 2.4
Name: ggufit
Version: 0.1.2
Summary: Check whether a local LLM can run on your machine (CPU-only inference).
Author: Mouad
License: Ggufit Use-Only License
        
        Copyright (c) 2026 Mouad
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to use
        and run the Software for any purpose, including commercial purposes, and to
        redistribute unmodified copies of the Software, subject to the following
        conditions:
        
        1. The above copyright notice and this permission notice shall be included
           in all copies of the Software.
        
        2. You may NOT modify, adapt, translate, or create derivative works based
           on the Software.
        
        3. You may NOT redistribute the Software, or any portion of it, in modified
           form.
        
        4. You may NOT sublicense the Software under different terms.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
        ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
        WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
        
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Utilities
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: psutil>=5.9
Requires-Dist: numpy>=1.24
Dynamic: license-file

# ggufit

Check whether a local LLM can run on your machine, CPU-only — from PowerShell or any terminal.

Uses real formulas (model memory footprint, KV cache size, and a measured memory-bandwidth
micro-benchmark) rather than guesses, to estimate whether a model fits in RAM and roughly
how fast it'll generate tokens. Correctly handles MoE models (speed driven by active
experts, not total params) and SSM/MLA architectures (Mamba, DeepSeek's MLA) that don't
use standard multi-head attention.

## Install

```bash
# Recommended - works everywhere, gives you a global `ggufit` command
pipx install ggufit

# Or plain pip (Windows PowerShell, or inside a venv on Linux/macOS)
pip install ggufit
```

### Installing from source (for development)

```bash
git clone <this repo>   # or unzip the source
cd ggufit/               # the folder containing pyproject.toml
pipx install -e .        # editable install - code changes apply without reinstalling
# or: pip install -e .   (inside a venv, or with --break-system-packages)
```

## Usage

```bash
# Full hardware scan: shows your CPU/RAM/bandwidth and which models fit
ggufit scan

# Check one specific model
ggufit llama3.1
ggufit mistral
ggufit qwen2.5-14b

# Force a specific quantization level
ggufit llama3.1 --quant q4

# Evaluate at a longer context length
ggufit qwen2.5-32b --context 16384
```

## How it works

For each model:
1. **Model size (RAM)** = total params × bytes-per-parameter (varies by quant: FP16, Q8, Q6, Q5, Q4, Q3, Q2)
2. **KV cache** = `2 × layers × hidden_size × context_len × bytes_per_param × kv_cache_multiplier`
   - `kv_cache_multiplier` defaults to 1.0 (standard MHA), and is set lower for architectures
     that don't use full attention at every layer: `0.0` for pure SSM/Mamba (no attention at all),
     `~0.15` for MLA (DeepSeek-V2/V3/R1, MiniCPM3), `~0.125` for hybrid Mamba+attention (Jamba).
3. **Total RAM needed** = `weights × overhead_multiplier(weights) + KV cache`
   - The overhead multiplier accounts for compute buffers, activations, and allocator
     overhead. It is **size-dependent**, not a flat percentage: a fixed ~1GB baseline
     buffer plus ~7.6% of weight size. This means it's high for tiny models (~2x for a
     1GB model, which genuinely needs ~1GB of buffers on top) but asymptotes to ~1.08x
     for very large models. Calibrated against real-world llama.cpp memory reports across
     7B/13B/70B/671B models — e.g. DeepSeek-V3 671B at Q4 comes out to ~405GB, matching
     reality, instead of the ~450GB an old flat-20% rule would have predicted.
4. **Speed estimate** = `measured_memory_bandwidth / active_size`, where `active_size` is the
   total model size for dense models, or just the active experts' size for MoE models
   (`active_params_billion`) — since only those weights are streamed from RAM per token.

`ggufit` runs a quick real memory-bandwidth benchmark (a large array copy) instead of guessing
from RAM specs, since achievable bandwidth depends heavily on channel configuration.

## Adding models

Edit `ggufit/models_db.py` and add an entry to the `MODELS` dict with `params_billion`,
`layers`, and `hidden_size` (found in the model's Hugging Face `config.json`). Optional:
- `moe: True` + `active_params_billion: X` for Mixture-of-Experts models
- `kv_cache_multiplier: X` for non-standard attention architectures (see above)

## Known limitations

- Speed estimates assume batch size 1, single-user chat.
- The bandwidth benchmark is single-threaded; real inference engines use multiple threads.
- Quant byte-per-param values are approximations of real GGUF file sizes.
- **KV cache assumes full hidden_size for standard (non-flagged) models, even though most
  modern ones use GQA** (Grouped-Query Attention) with far fewer KV heads than query heads.
  This over-estimates the KV cache — a Mistral 7B's real KV cache is roughly 1/4 of what
  the formula computes. It's intentionally conservative (never under-estimates RAM), and it
  doesn't change the fit verdict for most models at short context, but it's the largest
  remaining source of RAM over-estimation. Full per-model GQA accounting (using each model's
  actual `num_kv_heads`) is planned for a future release.
- `kv_cache_multiplier` values for MLA/hybrid architectures are approximate, not
  per-model-measured.
- The overhead multiplier is calibrated against a handful of real-world data points
  (7B/13B/70B/671B at Q4); it's a strong approximation but not exact for every model/runtime.

## Changelog

- **0.1.2** — Overhead is now size-dependent (fixed ~1GB baseline + ~7.6% of weights),
  calibrated against real llama.cpp memory reports. Fixes large-model RAM over-estimation
  (DeepSeek-V3 671B Q4 now estimates ~405GB, matching reality, vs ~450GB before). Also fixes
  a wording bug where the "try a lower quant" hint could suggest the same quant that just failed.
- **0.1.1** — Custom use-only license; corrected install instructions.
- **0.1.0** — Initial release (MoE active-param speed, SSM/MLA KV cache handling, 217 models).

## Notes

This is a side-project CLI. The full hardware-scan desktop app (Rust/Tauri) is a separate,
more thorough tool still in development.

## Q&A

A comprehensive FAQ — from "what is a GGUF" to the exact formulas behind every number
`ggufit` prints. Organized beginner → expert.

### Basics

**What is a GGUF?**
GGUF (GPT-Generated Unified Format) is a file format for storing LLM weights, designed
by the llama.cpp project for fast loading and CPU/GPU-flexible inference. It bundles
the model's tensors and metadata (architecture, tokenizer, etc.) into a single file.
It replaced the older GGML format. If you've downloaded a `.gguf` file from Hugging
Face to run in `llama.cpp`, Ollama, or LM Studio, that's what `ggufit` is estimating
compatibility for.

**What is quantization?**
Shrinking a model's weights from their original precision (usually 16-bit floats) down
to smaller representations (8-bit, 4-bit, etc.) to save memory and speed up inference,
at the cost of some accuracy. A "Q4" model uses roughly a quarter of the memory of the
same model at full 16-bit precision.

**What do Q4, Q5, Q6, Q8 mean?**
The number is roughly the average bits per weight after quantization (not exactly,
see next question). Lower number = smaller file, faster inference, more quality loss.
Q4 is the most common "sweet spot" for CPU-only local inference. Q8 is close to
lossless but nearly as large as full precision.

**Why isn't Q4 exactly 4 bits then?**
Modern GGUF "K-quants" (the `_K_M`, `_K_S` suffixes you see on Hugging Face) don't use
a uniform bit-width across the whole model — they mix precision per tensor, using
slightly higher precision for the parts most sensitive to quality loss. So "Q4" is
really an *average* around 4-5 bits/weight in practice. `ggufit` uses effective
per-quant byte values that reflect this real-world average, not naive N-bit math:

| Quant | Effective bytes/param |
|---|---|
| FP32 | 4.0 |
| FP16/BF16 | 2.0 |
| Q8 | 1.05 |
| Q6 | 0.8 |
| Q5 | 0.7 |
| Q4 | 0.6 |
| Q3 | 0.5 |
| Q2 | 0.4 |

**What's the difference between RAM and VRAM, and why does ggufit only care about RAM?**
VRAM is memory on a dedicated GPU; RAM is your system's main memory, used by the CPU.
`ggufit` is CPU-only by design — it answers "can my CPU and system RAM handle this,"
not "can my GPU handle this." If you have a GPU, tools like `nvidia-smi` and the model
card's VRAM requirements are what you want instead.

**What is context length?**
The number of tokens (roughly, chunks of a word) the model can "see" at once —
your prompt plus its response so far. Longer context means the model can process
longer documents, conversations, and code, but it also means more memory used for
the KV cache (see below).

**What are tokens/sec, and what's a "good" number?**
How many tokens the model generates per second. For a comfortable reading pace, most
people find 5-15 tok/s tolerable for chat; below ~2 tok/s feels quite slow; above 20
tok/s feels close to instant. It's highly subjective and task-dependent though —
background batch jobs can tolerate much lower throughput than an interactive chat.

---

### Installing and running ggufit

**How do I install it?**
```bash
pipx install ggufit    # recommended
pip install ggufit     # inside a venv, or with --break-system-packages
```

**Why do I get "externally-managed-environment"?**
Modern Debian/Ubuntu (PEP 668) blocks system-wide `pip install` to protect the OS's
own Python packages. Use `pipx` instead — it installs into an isolated environment
while still giving you a global command.

**Why does `ggufit` say "command not found" right after installing?**
Either your venv isn't activated (`source venv/bin/activate`), or `pipx`'s bin
directory isn't on your PATH yet (run `pipx ensurepath` and reopen your terminal).

**Why did a model that should fit show as "does not fit"?**
`ggufit` checks *currently available* RAM, not total installed RAM. If other programs
are using most of your memory, an otherwise-fine model can fail the check. Run `free -h`
(Linux/macOS) to see what's actually free before assuming your hardware is the problem.

**Can I check a model at a specific quant instead of letting ggufit auto-pick?**
Yes — `ggufit <model> --quant q4`. Without `--quant`, it auto-picks the highest-quality
quant that fits.

---

### Understanding the numbers (intermediate)

**How is "Model size (RAM)" calculated?**
```
model_size_bytes = num_parameters × bytes_per_param
```
`bytes_per_param` comes from the quant table above. This is the number of bytes the
weights occupy once loaded — the dominant factor in "will this even load."

**What is the KV cache, and why does it matter?**
During generation, the model caches the Key and Value tensors from every previous
token in the conversation so it doesn't have to recompute them each step. This cache
grows with context length. The standard formula:
```
kv_cache_bytes = 2 × num_layers × hidden_size × seq_len × batch_size × bytes_per_param
```
The `2×` covers storing both K and V. This is why a model that fits fine at a 4K
context can stop fitting at 32K — the KV cache scales linearly with context length
while the model weights stay fixed.

**Why is there a 1.2x "overhead factor" on top of model size + KV cache?**
Real inference isn't just raw tensor storage — the runtime, OS, and memory allocator
all need working space too (buffers, fragmentation, temporary activations). 1.2x is a
practical rule-of-thumb headroom, not a value measured per specific runtime.

**Why is CPU inference speed based on memory bandwidth instead of raw compute (FLOPS)?**
At batch size 1 (the normal case for a single person chatting), generating each new
token requires reading every single model weight from RAM once. The CPU spends far
more time waiting on memory than doing arithmetic — so the bottleneck is how fast
data can move from RAM to the CPU, not how many operations per second the CPU can do.
That's why the formula is:
```
tokens_per_sec ≈ memory_bandwidth (GB/s) / model_size (GB)
```

**Why does ggufit benchmark my memory bandwidth instead of just knowing it from my RAM specs?**
Achievable bandwidth depends on RAM generation, channel configuration
(single/dual/quad-channel), and platform quirks — none of which are reliably
detectable across Windows/Mac/Linux without vendor-specific tools. So `ggufit` times
a real large in-memory array copy on your actual machine, right now, and uses that
measured number instead of guessing.

**Why does the bandwidth number change slightly every time I run ggufit?**
It's a live micro-benchmark, not a cached constant — normal system load, thermal
throttling, and other processes competing for memory access all cause small
run-to-run variance. That's expected and not a bug.

---

### Architecture-specific math (expert / advanced)

**What is a Mixture-of-Experts (MoE) model, and why does it need special handling?**
An MoE model has many "expert" sub-networks, but only a subset of them are activated
for any given token (a small router network decides which experts to use). This means:
- **All experts must be loaded into RAM** — because any token could route to any
  expert, so the *fit* check uses the full `params_billion` (total, all experts).
- **Only the active experts are actually read from RAM per token** — so the *speed*
  estimate uses `active_params_billion` instead. Using total params for speed
  would make big MoE models look absurdly slow (DeepSeek-V3 at 671B total but only
  37B active would look ~18x slower than it really is if you used the total).

```
fit check:   uses params_billion (total) — all experts must be resident
speed check: uses active_params_billion — only active experts are read per token
```

**What is GQA (Grouped-Query Attention), and why doesn't ggufit fully account for it?**
Most modern transformer models don't give every attention head its own K/V
projection — they group multiple query heads to share a smaller number of K/V heads
(`num_kv_heads < num_attention_heads`). This makes the *real* KV cache smaller than
the standard formula (which assumes `hidden_size` worth of K/V per layer) predicts.
`ggufit` currently uses the full `hidden_size` for standard models — this is
intentionally conservative (it over-estimates KV cache, never under-estimates), and
since total model weight size is still the dominant term in "does it fit," this
doesn't change the fit verdict in most cases. It mainly matters at very long context
lengths. Full per-model GQA accounting (using each model's actual `num_kv_heads`) is
a planned improvement, not yet implemented.

**What is MLA (Multi-head Latent Attention), and why the ~0.15 multiplier?**
MLA (used by DeepSeek-V2/V3/R1 and MiniCPM3) compresses the K/V representations into
a much smaller latent vector before caching them, then reconstructs full K/V on the
fly. This shrinks the real KV cache to roughly 1/6 to 1/8 of what standard
multi-head attention would need for the same layer count and hidden size. `ggufit`
applies a `kv_cache_multiplier: 0.15` to these models so the KV cache estimate
reflects that compression instead of wildly overestimating it.

**Why do pure Mamba/SSM models have a KV cache multiplier of 0.0?**
State-Space Models (Mamba, used in Codestral Mamba and Falcon-Mamba) don't use
attention at all — they maintain a fixed-size recurrent state instead of caching
every previous token's K/V. That state doesn't grow with context length. So their
effective "KV cache" for the purposes of this formula is zero, regardless of how
long the context gets. This is why `ggufit falcon-mamba --context 65536` still shows
~0GB KV cache even at a huge context length.

**What about hybrid models like Jamba, and why 0.125?**
Jamba interleaves Mamba blocks with regular attention blocks — only about 1 in every
8 layers actually uses attention (the rest are Mamba). So its effective KV cache is
roughly 1/8th of what you'd get if every layer used standard attention, hence the
`0.125` multiplier.

**Are the MoE/SSM/MLA multipliers exact?**
No — they're architecture-level approximations based on published compression
ratios, not measured per-model-per-config values. They're a large improvement over
assuming standard attention everywhere (which would be wrong by 5-40x for these
architectures), but treat them as "much closer estimate," not "exact number."

**Why does `ggufit` need real `layers` and `hidden_size` values instead of estimating them from param count?**
Because two models with the same parameter count can have very different KV cache
sizes depending on how those parameters are distributed across layers and hidden
dimension — there's no reliable shortcut from param count alone. `ggufit` stores
these values explicitly per model (sourced from each model's Hugging Face
`config.json`) to keep the KV cache estimate accurate.

---

### Project & contributing

**How do I add a model that isn't in the database?**
Edit `ggufit/models_db.py` and add an entry to the `MODELS` dict with `params_billion`,
`layers`, `hidden_size` (all from the model's Hugging Face `config.json`). Add
`moe: True` + `active_params_billion` for MoE models, or `kv_cache_multiplier` for
non-standard attention architectures.

**Why was the project renamed from moscan to ggufit?**
The name `moscan` was already taken on PyPI. `ggufit` was chosen instead — pun on
GGUF (the file format) + "fit" (does the model fit on your machine).

**What license is ggufit under?**
A custom "use-only" license: you're free to install and run it for any purpose,
including commercial use, but you may not modify it or redistribute a modified
version. See the `LICENSE` file for exact terms. Note: versions published before
this license was adopted (0.1.0) remain under their original MIT terms for anyone
who already obtained that specific release — license changes only apply going
forward, not retroactively.

**Is this a substitute for actually running the model to see how it performs?**
No — treat every number here as an estimate to guide a decision (e.g. "should I even
attempt downloading this 40GB file"), not a guarantee. Real-world speed depends on
your specific inference engine (llama.cpp, Ollama, etc.), thread count settings, OS
scheduler behavior, and quantization implementation quality, none of which `ggufit`
can measure without you actually running the model.
