Metadata-Version: 2.4
Name: ggufit
Version: 0.2.0
Summary: Check whether a local LLM can run on your machine, on CPU or with a GPU.
Author: Mouad
License: Ggufit Use-Only License
        
        Copyright (c) 2026 Mouad
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to use
        and run the Software for any purpose, including commercial purposes, and to
        redistribute unmodified copies of the Software, subject to the following
        conditions:
        
        1. The above copyright notice and this permission notice shall be included
           in all copies of the Software.
        
        2. You may NOT modify, adapt, translate, or create derivative works based
           on the Software.
        
        3. You may NOT redistribute the Software, or any portion of it, in modified
           form.
        
        4. You may NOT sublicense the Software under different terms.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
        ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
        WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
        
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Utilities
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

<div align="center">

# ggufit

### Will this LLM actually run on my machine?

Answer it in two seconds, before downloading 40 GB to find out.

[![PyPI](https://img.shields.io/pypi/v/ggufit)](https://pypi.org/project/ggufit/)
[![Python](https://img.shields.io/pypi/pyversions/ggufit)](https://pypi.org/project/ggufit/)
![Dependencies](https://img.shields.io/badge/dependencies-none-brightgreen)
![Platform](https://img.shields.io/badge/platform-Linux%20%7C%20macOS%20%7C%20Windows-lightgrey)
![Models](https://img.shields.io/badge/models-218-blue)

</div>

---

`ggufit` sizes local LLMs against **your** hardware: it measures your real memory bandwidth,
computes each model's true memory footprint from its actual architecture, and tells you what
fits and roughly how fast it will generate, on CPU and on your GPU if you have one worth
using.

It has **zero dependencies**. The model and GPU catalogues ship with it, so a catalogued
model is answered entirely offline. Name a model it has never heard of and it asks
HuggingFace for the specs rather than giving up; `--offline` refuses that outright.

```bash
pipx install ggufit     # recommended: isolated env, global `ggufit` command
pip install ggufit      # or inside a venv
```

## Not sure what to type?

Run `ggufit` with no arguments and it asks instead of printing a wall of flags:

```console
$ ggufit

  ggufit  v0.2.0  ·  what can this machine run?
  Answer a few questions, or press Esc at any point to stop.

  Language / Langue
   > English    answers in English
     Français   réponses en français

  What would you like to do?
   > See everything this machine can run   scan the whole catalogue
     Get one recommendation per use case   the short answer
     Check one specific model              does it fit, and how fast
     Just look at my hardware              CPU, RAM and GPU, with a real bandwidth test
     List the GPUs ggufit knows            names to use for comparisons
```

Arrow keys and Enter, or type the number. Every answer maps onto a flag that already
exists, and the last thing it prints is the command it just built:

```
  Same thing, without the questions:
    ggufit scan --for code --context 32768 --gpu "rtx 4090" --fastest --top 5
```

That line is the point. Use it twice and you have learned the flags for the thing you
actually do, and stop needing the questions. A wizard that leaves you dependent on the
wizard would be worse than the help text it replaced.

Piped or redirected output prints the help instead, so `ggufit > report.txt` never blocks
waiting for a keystroke that cannot arrive.

## In English, or in French

The first thing bare `ggufit` asks is which language to answer in, and it remembers what
you pick:

```console
  Language / Langue
   > English    answers in English
     Français   réponses en français
```

English is the default and stays the default until you choose otherwise. The choice is
remembered in the same cache directory as the HuggingFace lookups, so the next
`ggufit scan` answers the same way without being asked again (and `--clear-cache`, which
is about network lookups, leaves it alone).

```bash
ggufit language            # what it is answering in now, and how to change it
ggufit language fr         # switch, and remember it
ggufit language en         # switch back
ggufit language reset      # forget the preference; English again

ggufit --lang fr scan      # this one run only, nothing remembered
GGUFIT_LANG=fr             # this terminal session only
```

Three scopes, because "just this once", "for as long as this shell lives" and "from now
on" are genuinely different wishes. First match wins: `--lang`, then `GGUFIT_LANG`, then
the remembered preference, then English.

**The commands stay in English in every language.** `scan`, `--for code`, `--gpu-layers`,
`-ngl`: all of it is the same in French, and only the answers change. A flag is a thing
you type, and a French user reading a blog post, a README or their own shell history needs
the words in it to be the words the tool accepts. Translating the interface you type at
would break every copied command line to spare nobody.

```console
$ ggufit --lang fr llama3.1

  Llama 3.1 8B                                           8 Md de paramètres · contexte 4096
  ────────────────────────────────────────────────────────────────────────────────────
  RAM libérée      26,75 Go utilisables, tout sauf 4,72 Go réservés au système
  Maintenant       15,7 Go libres, sans rien fermer de ce qui tourne

  ✓ Maintenant   Q8 · 9,95 Go · ~2,6–4,9 jetons/s                                ▇▇▇·······
                 poids 7,85 Go · cache 0,5 Go · 9,95 Go sur 15,7 Go

  ✓ RAM libérée  FP16 · 17,59 Go · ~1,4–2,6 jetons/s                             ▇▇········
                 poids 14,96 Go · cache 0,5 Go · 17,59 Go sur 26,75 Go
                 limité par la bande passante mémoire, plage non calibrée de ±30 %
                 → mieux que le Q8 qui tient maintenant
```

Numbers and units follow the language too, since "6.14 GB" reads to a French speaker about
the way "6,14 GB" reads to an English one. The help text is translated as well, so
`ggufit --lang fr --help` is a French help page describing English flags.

`tools/check_translations.py` (or `make check-translations`) fails the build on a key that
one language has and the other does not, or on a placeholder that drifted between them. A
missing translation does not crash anything, it just quietly prints English into a French
screen, which is exactly the kind of bug a build step should catch rather than a user.

## See it work

```console
$ ggufit scan --for code --fastest --top 4

  Hardware
  ────────────────────────────────────────────────────────────────────────────────
  OS                Windows 11
  CPU               Intel Core Ultra 7 155H · 14 logical cores
  Memory            31.47 GB installed · 15.72 GB free now
  Bandwidth         36.7 GB/s · measured here, multi-process

  What runs here, code
  ────────────────────────────────────────────────────────────────────────────────
  RAM cleared  26.75 GB usable, everything but 4.72 GB reserved for the OS
  Right now    15.72 GB free, with what you have open left alone
  Sorted by estimated speed, quickest first.

  ✓ Runs right now  ·  28 models
    Model                                 Quant     Needs             Speed
    Qwen2.5-Coder 0.5B                    FP16    2.04 GB  ~23.7–44.1 tok/s    ▇▇▇▇▇▇··
    DeepSeek-Coder-V2-Lite 16B (MoE/MLA)  Q6     13.71 GB  ~12.2–22.7 tok/s    ▇▇▇▇····
    DeepSeek-Coder 1.3B                   FP16    4.46 GB   ~8.7–16.1 tok/s    ▇▇▇▇····
    Yi-Coder 1.5B                         FP16    4.71 GB   ~7.9–14.8 tok/s    ▇▇▇·····
    and 24 more, raise --top to see them

  ! Usable once you free up RAM  ·  1 model
    Granite Code 34B                      Q3    26.14 GB    ~1.4–2.6 tok/s    ▇·······

  ✕ Will not fit  ·  2 models
    Code Llama 70B                     DeepSeek-Coder-V2 236B (MoE/MLA)
```

Every row is one model, and every column answers one question: which quantization, how
much memory it wants, how fast it will generate. The meter on the right is that speed at a
glance, on a fixed scale, so the same model draws the same bar wherever it is listed.

Or drill into one model:

```console
$ ggufit llama3.1

  Llama 3.1 8B                                                     8B params · 4096 ctx
  ────────────────────────────────────────────────────────────────────────────────
  RAM cleared  26.75 GB usable, everything but 4.72 GB reserved for the OS
  Right now    15.72 GB free, with what you have open left alone

  ✓ Right now     Q8 · 9.95 GB · ~2.8–5.1 tok/s                              ▇▇▇·······
                  weights 7.85 GB · cache 0.5 GB · 9.95 GB of 15.72 GB

  ✓ RAM cleared   FP16 · 17.59 GB · ~1.5–2.7 tok/s                           ▇▇········
                  weights 14.96 GB · cache 0.5 GB · 17.59 GB of 26.75 GB
                  memory-bandwidth bound, uncalibrated ±30% range
                  → better than the Q8 that fits right now
```

One line answers the question, and it is the same line on every model: quantization,
memory, speed. The arithmetic behind it sits underneath, for arguing with the answer
rather than for finding it.

## Why not just multiply parameters by bytes?

Because that gets three things wrong, and each one is worth gigabytes:

| | The naive answer | What `ggufit` does |
|---|---|---|
| **Attention cache** | assume every model caches per attention head | reads each model's real K/V head counts, a **4-8x** difference on modern models |
| **Mixture-of-experts** | score a 671B model as if all 671B are read per token | fits on total weights, but times on **active** experts only |
| **Memory bandwidth** | look up the RAM spec sheet | **measures** it, multi-process, on your machine |

Concretely: Llama 2 70B at 32K context needs **10 GB** of attention cache, not the 80 GB a
per-head assumption gives you. That is the difference between "you need a server" and "this
runs on a workstation."

Mamba and DeepSeek's MLA are handled as themselves too, a pure state-space model has *no*
attention cache at any context length, and MLA caches one compressed vector per layer.

## Two answers, not one

Every result is reported against **two** memory figures, because "can this machine run it"
and "can it run it without closing anything first" are different questions, and one number
quietly conflated them:

- **Right now**: what is actually free at this instant, with everything you have open left
  alone. A live reading, so it moves between runs.
- **With RAM cleared**: total installed RAM minus a reserve for the OS (2 GB or 15%,
  whichever is larger). Fixed and reproducible: the machine's real ceiling.

That is why `scan` has a middle group. **Usable if you free up RAM** means the model fits
your hardware fine, just not alongside your browser. Where free memory cannot be read, the
two groups collapse into one rather than guessing.

## Using your GPU

If a usable GPU is detected, every result gains a second verdict: what the same model does
with the GPU helping, and the exact `-ngl` value to run it with.

```console
$ ggufit llama3.1 --gpu "rtx 4070"

  ✓ With the GPU  FP16 · 21 of 33 layers offloaded · ~3.5–6.5 tok/s          ▇▇▇·······
                  GPU  9.52 GB weights + 0.33 GB cache → 10.92 GB of 11.4 GB
                  RAM  5.44 GB weights + 0.17 GB cache → 7.02 GB of 26.75 GB
                  → 2.4× faster than the CPU alone, which manages ~1.5–2.7 tok/s
                  › 88% of each token is still spent reading weights from RAM. That
                    remainder is the bottleneck.
                  › run with  -ngl 21
```

Three things are happening there that a simpler tool gets wrong.

**RAM and VRAM add up.** A model too big for either pool alone can still run split across
both, so a machine the CPU-only path calls hopeless often is not. That is why the 70B above
goes from "does not fit at any quant" to runnable.

**A partial offload is timed properly.** Generation reads every active weight once per token,
so the two halves add: `t = W_gpu / BW_gpu + W_cpu / BW_cpu`. The CPU term dominates the
moment it is non-zero, which is why moving 21 of 33 layers to a 500 GB/s card buys 2.4x and
not 10x. `ggufit` reports that 88% figure rather than leaving you to infer a speedup from the
layer count, which suggests the opposite conclusion.

**Integrated graphics are called what they are.** An Intel or AMD iGPU has no memory of its
own: it reads the same DRAM at the same bandwidth the CPU already has. Since generation is
bandwidth-bound, offloading to it cannot help, and `ggufit` says so instead of printing a
speedup. Windows will happily report such an adapter as having "2 GB of VRAM" (a shared
aperture, not dedicated memory), and taking that number at face value is exactly how a
confident wrong answer gets made.

### Where GPU bandwidth comes from

It is looked up from `ggufit/gpus.json` (123 entries), not measured, and that is deliberate.

System RAM bandwidth has to be measured because it depends on channel configuration and how
the sticks are populated, which no spec sheet knows. A discrete GPU has no such variance: bus
width and memory clock are fixed properties of the part, so every RTX 4090 ever made moves
1008 GB/s. For this hardware the table is the ground truth, and measuring it would mean
shipping a CUDA or Metal runtime to learn a number the vendor already publishes exactly.

Capacity still comes from detection, since `nvidia-smi` knows both what is installed and what
is in use right now. A card that is not in the table gets reported with its VRAM, and no speed
estimate at all.

```bash
# Size a model against hardware you do not own yet
ggufit llama3.1-70b --gpu "rtx 3090"
ggufit --list-gpus

# Pin the split, or ignore the GPU entirely
ggufit llama3.1 --gpu-layers 20
ggufit llama3.1 --no-gpu
```

Detection uses `nvidia-smi` for NVIDIA, `sysctl` for Apple Silicon (where the GPU budget is a
slice of unified memory), `rocm-smi` or DRM sysfs for AMD, and the display-class registry on
Windows. All standard library, no new dependencies.

## More than one person at a time

Every figure above is for one person generating. `--users N` sizes the same model for a
team, and the answer changes in two directions at once:

```console
$ ggufit llama3.1 --users 4

  Llama 3.1 8B                                          8B params · 4096 ctx · 4 users
  ──────────────────────────────────────────────────────────────────────────────
  RAM cleared  26.75 GB usable, everything but 4.72 GB reserved for the OS
  Right now    15.47 GB free, with what you have open left alone
  Users        4 at once. The weights are loaded once and shared; the attention cache
               is not, so it is counted 4 times.

  ✓ Right now     Q8 · 11.45 GB · ~2.2–4 tok/s each                       ▇▇········
                  weights 7.85 GB · cache 4 × 0.5 GB · 11.45 GB of 15.47 GB
                  → 4 users at once: ~8.7–16.1 tok/s across all of them
```

**Memory goes up, but only the cache does.** The weights are loaded once and shared by
everyone; the attention cache belongs to a sequence, so it is the one term that multiplies.
At 4K context that is half a gigabyte a head and barely registers. At 32K it is 4 GB a
head, so eight people need 32 GB of cache on top of the model, and that, not the weights,
is usually what decides whether a machine can serve a team.

**Speed splits into two numbers, and they move in opposite directions.** One decoding step
reads every active weight once and emits a token for *each* sequence in flight, so the
expensive read is shared: per-user speed falls a little, total throughput rises a lot. Both
are printed. The `--best` bar is applied to the per-user figure, because a machine serving
eight people at 1 tok/s each has excellent throughput and is not worth sitting in front of.

`--users` is llama.cpp's `-np` slot count, and the run-with line says so:

```console
                  › run with  -ngl 20 -np 4
```

One assumption worth stating plainly: `--users 1` is identical to every version before
this one, and each *additional* concurrent sequence is charged one more full cache read
per step. Whatever traffic the first sequence causes is already inside the
runtime-efficiency constant the single-user estimate was built on, so counting it again
would have moved every number in the tool to model something it already modelled. That
assumption sits on exactly the same footing as the constant itself, inside the same ±30%
band.

## One pick per use case

`scan` lists everything that fits and leaves the choosing to you. `scan --best` does the
choosing:

```console
$ ggufit scan --best

  Best model per use case
  ────────────────────────────────────────────────────────────────────────────────
  RAM cleared  26.75 GB usable, everything but 4.72 GB reserved for the OS
  Right now    15.7 GB free, with what you have open left alone
  Best = the largest model that fits and still generates at 5+ tok/s.

    Use case   Model                                 Quant             Speed  Params
    code       DeepSeek-Coder-V2-Lite 16B (MoE/MLA)  Q6     ~11.4–21.2 tok/s   15.7B
    reasoning  DeepSeek-R1-Distill 14B (Qwen base)   Q2       ~3.7–6.9 tok/s   14.8B
    maths      Qwen2.5-Math 7B                       Q6       ~3.6–6.7 tok/s    7.6B
    vision     LLaVA 1.5 13B (Vicuna)                Q2       ~4.1–7.6 tok/s   13.4B
    chat       WizardCoder 15B (StarCoder)           Q2       ~3.5–6.6 tok/s   15.5B
    general    Qwen3 30B-A3B (MoE)                   Q2     ~16.6–30.9 tok/s   30.5B
```

"Best" means the largest model that is still fast enough to sit through, not simply the
largest that fits. Those give different answers, and the difference is the whole point: a
70B squeezed to Q2 at 0.8 tok/s fits and is unusable.

Getting this right also means quantising before ruling a model out. Scoring a 14B at FP16,
watching it miss the speed bar and discarding it would recommend something far smaller than
the machine can run, when the same 14B at Q4 clears the bar comfortably. Move the bar with
`--best-speed`.

## Models the catalogue has never heard of

Name a model that is not among the 218 and `ggufit` offers to ask HuggingFace for its
architecture instead of giving up. It asks first, every time, because a tool that made no
network calls should not start making them without being told to:

```console
$ ggufit granite-3.1-8b

  'granite-3.1-8b' is not in the catalogue that ships with ggufit.

  Looking it up means contacting huggingface.co over the internet, sending
  the model name you typed. Nothing about your machine or its hardware is sent.

  Fetch it from HuggingFace? [y/N]: y

  'granite-3.1-8b' is not in the catalogue, so it was fetched from HuggingFace.
  Resolved to: ibm-granite/granite-3.1-8b-base
```

The prompt defaults to no. `--yes` answers it in advance, `--offline` refuses without
asking, and a catalogued model never triggers it at all. Where there is nobody to ask
(output piped, or a CI job) the answer is no rather than yes: a run must not pick up a
network dependency just because no terminal was attached to object.

Short names are searched; an exact repo id (`Qwen/Qwen3-4B`) is fetched directly and is the
way to be certain which model you got. The resolved repo is always printed, because search
is fuzzy and picking the wrong one silently would be worse than not answering.

A few things this handles that a naive fetch does not:

- **Gated repos.** Llama and Gemma return HTTP 401 for their config. The parameter count is
  still available from the API, so only the architecture is missing, and `ggufit` sources it
  from an ungated re-upload carrying the same model name. It says so when it does.
- **Multimodal configs.** Gemma 3, Qwen-VL and Llava nest the language model under
  `text_config`, leaving no layer count at the top level. Reading the top level regardless
  produces a record with no layers, which fails quietly.
- **Mixture-of-experts.** Three vendors spell the expert count three different ways
  (`num_local_experts`, `n_routed_experts`, `num_experts`). Missing one scores the model as
  dense, which for Qwen3-30B-A3B means timing 30B per token instead of 3B.

A `config.json` cannot say how many parameters are active per token, so for a fetched MoE
model the speed estimate falls back to the full parameter count and is **too low**. That is
the safe direction, and `ggufit` says so rather than presenting it as a measurement. Memory
figures are unaffected.

Lookups are cached, so naming the same model twice hits the network once. `--offline`
refuses network lookups entirely and restores the pre-0.1.4 behaviour; `--clear-cache`
empties the cache.

## Sizing a machine that is not yours

The web form asks a client what hardware they have and hands them a `rapport.json`.
That file is now an input, not just a record: point `ggufit` at it and every command
works against the machine it describes.

```bash
ggufit json rapport.json                  # what that machine can run
ggufit json rapport.json --for code       # only coding models
ggufit json rapport.json --best           # one pick per use case
ggufit llama3.1 --from-json rapport.json  # one model in detail
ggufit hardware --from-json rapport.json  # just what the report says it has
```

```console
$ ggufit json rapport.json

  From report
  File              rapport.json
  Stated use        Coding assistance
  Every figure below describes the machine in this report, not the one running
  this command.
  > 2 channel(s) x 6000 MT/s x 8 bytes = 96.0 GB/s theoretical, 62.4 GB/s after
    real-world efficiency.

  Hardware
  OS                Windows
  CPU               AMD Ryzen 9 7950X
  Memory            64 GB installed
  Bandwidth         62.4 GB/s | derived from the stated memory configuration, not measured here
  GPU               GeForce RTX 4070 | 12 GB VRAM
```

Three things this path is careful about, because each one is a way to quietly answer
the wrong question:

**The bandwidth is derived, not measured.** A local run benchmarks it multi-process.
A report computes it from the stick count and memory speed the client typed. Those are
different claims, so the line beside the number says which one it is holding rather
than inheriting the words "measured here".

**The GPU comes from the report, never from your machine.** Detecting the local card
while sizing someone else's hardware would credit their report with whatever happens to
be in the computer reading it. The form's free-text answer ("RTX 4070, 12 GB") is
resolved through the same name table a detected card goes through, and an unrecognised
string yields no speed rather than a guess.

**The report's own answers are the defaults.** Context length, concurrent sessions and
the speed floor implied by "how fast does it need to feel" all come from what the client
told the form. Anything you type still wins, so it is a starting point rather than a cage.

What a report cannot say, it does not: free memory at this instant is unknowable from a
file written earlier elsewhere, and the core count is never asked, so both are reported
as unstated instead of filled in.

## Usage

```bash
# Not sure what you want? Answer questions instead
ggufit

# What runs on this machine
ggufit scan

# Just this machine's CPU, RAM, GPU and real memory bandwidth
ggufit hardware

# Filter by what you want the model for
ggufit scan --for code          # also: reasoning, math, vision, chat, general

# Sort by speed, cap the output
ggufit scan --fastest
ggufit scan --for code --fastest --top 5

# One recommendation per use case, as a table
ggufit scan --best
ggufit scan --best --best-speed 10

# Drill into one model (short names, HF-style and Ollama-style all work)
ggufit llama3.1
ggufit llama-3.1-8b
ggufit llama3.1:8b

# Force a quant level, or test a longer context
ggufit llama3.1 --quant q4
ggufit qwen2.5-32b --context 16384

# Size it for several people generating at the same time
ggufit llama3.1 --users 4
ggufit scan --users 8 --context 8192

# Answer in French, for one run, this shell, or from now on
ggufit --lang fr scan
GGUFIT_LANG=fr ggufit scan
ggufit language fr
```

Output is plain text when piped (`ggufit scan > report.txt`), coloured when it is a
terminal. `NO_COLOR` and `FORCE_COLOR` are both honoured.

The layout adapts to the terminal rather than assuming one. Tables drop their least
important column before they will wrap, prose is re-wrapped to the window, and the box
characters fall back to ASCII when the output stream cannot encode them, which on Windows
is what a redirect into a file often is. `GGUFIT_ASCII=1` forces that fallback everywhere.

### Installing from source

```bash
cd ggufit/                # the folder containing pyproject.toml
pipx install -e .         # editable install
# or: pip install -e .    (inside a venv, or with --break-system-packages)
```

## How it works

For each model:
1. **Model size (RAM)** = total params × bytes-per-parameter (varies by quant: FP16, Q8, Q6, Q5, Q4, Q3, Q2)
2. **KV cache** uses the model's real attention architecture, not a flat guess:
   - **Standard / GQA models**: `2 × layers × num_kv_heads × head_dim × context_len × bytes_per_elem`,
     using each model's *actual* KV head count (sourced from its Hugging Face `config.json`), not
     `hidden_size`. Most models since Llama 2 70B use Grouped-Query Attention, where `num_kv_heads`
     is far smaller than the number of query heads, using `hidden_size` there over-estimates the
     KV cache by 4-8x (e.g. Llama 2 70B at 32K context: 10 GB real vs 80 GB under the old formula).
   - **MLA** (DeepSeek-V2/V3/R1, MiniCPM3): `layers × (kv_lora_rank + qk_rope_head_dim) × context_len
     × bytes_per_elem`, MLA caches one compressed latent vector per layer, not one per head.
   - **Pure SSM/Mamba** (no attention at all): `0`, a fixed-size recurrent state that doesn't grow
     with context length.
   - A handful of older/override-only models have no published head-count data; those still fall
     back to the conservative `hidden_size`-based estimate.
3. **Total RAM needed** = `weights × overhead_multiplier(weights) + KV cache`
   - The overhead multiplier accounts for compute buffers, activations, and allocator
     overhead. It is **size-dependent**, not a flat percentage: a fixed ~1GB baseline
     buffer plus ~7.6% of weight size. This means it's high for tiny models (~2x for a
     1GB model, which genuinely needs ~1GB of buffers on top) but asymptotes to ~1.08x
     for very large models. Calibrated against real-world llama.cpp memory reports across
     7B/13B/70B/671B models, e.g. DeepSeek-V3 671B at Q4 comes out to ~405GB, matching
     reality, instead of the ~450GB an old flat-20% rule would have predicted.
4. **RAM budget**: two of them, see "Two answers, not one" above. The reproducible one is total
   installed RAM minus `max(2 GB, 15% of total)` reserved for the OS; the live one is whatever is
   free at this instant. The reproducible figure is what the fit verdict is anchored to, since a
   live reading alone used to make `ggufit` report a different verdict for the same machine seconds
   apart.
5. **Speed estimate** = `measured_memory_bandwidth / active_size`, shown as a range (±30%, since the
   runtime-efficiency constant behind it is not yet calibrated against real measurements, see
   Known limitations). `active_size` is the total model size for dense models, or just the active
   experts' size for MoE models (`active_params_billion`), since only those weights are streamed
   from RAM per token.

`ggufit` runs a real memory-bandwidth benchmark (several worker processes copying a buffer
concurrently, STREAM-style) instead of guessing from RAM specs, since achievable bandwidth depends
heavily on channel configuration. It has no dependencies, standard library only.

## Model catalogue

The 218-model catalogue (`ggufit/models.json`) is generated from HuggingFace `config.json` /
safetensors metadata, not hand-typed, see the `ggufit` repo's `tools/fetch_model_specs.py`. This is
what makes the per-model KV head counts above trustworthy: a hand-maintained file couldn't carry that
data without it silently going stale. A few fields (MoE `active_params_billion`, and specs for
repos HuggingFace can't resolve directly) come from a small manual overrides file instead, since
those genuinely aren't derivable from a config.

## Known limitations

- Speed estimates are shown as an uncalibrated ±30% range, the runtime-efficiency constant behind
  them is a placeholder pending real tokens/sec measurements.
- Concurrency (`--users N`) is modelled as: the cache multiplies, the weights do not, and each
  sequence beyond the first adds one more full cache read per decoding step. It assumes continuous
  batching, ignores scheduling and prefill entirely, and treats the cache as full at the stated
  context, which is the worst case rather than the average one.
- Quant byte-per-param values are approximations of real GGUF file sizes.
- A handful of older or override-only models have no published attention head-count data and fall
  back to the conservative `hidden_size`-based KV estimate (over-estimates, never under-estimates).
- GPU bandwidth is **looked up, not measured** (see "Using your GPU" above for why that is the
  right call for a discrete card). A GPU that isn't in the table gets a fit verdict and no speed.
- The GPU runtime-efficiency and VRAM-overhead constants are placeholders on the same footing as
  the CPU ones, and the ±30% band applies to GPU estimates for the same reason.
- Partial-offload speed assumes weights split evenly across `layers + 1` units. For MoE models it
  also assumes an offloaded fraction of the weights serves about that fraction of the reads per
  token, which routing makes approximate.
- A model fetched from HuggingFace is assembled from its `config.json`, not curated like the
  catalogue. Its use-case tag is guessed from the repo name, and for an MoE model the active
  parameter count is unavailable, so the speed estimate is conservative and labelled as such.
- Resolving a short name uses HuggingFace search, which is fuzzy. The resolved repo is always
  printed; pass an exact repo id when it matters which one you get.
- The overhead multiplier is calibrated against a handful of real-world data points
  (7B/13B/70B/671B at Q4); it's a strong approximation but not exact for every model/runtime.
- Prompt processing time isn't modeled, only generation speed.

## Changelog

Upgrading from 0.1.2? `UPDATE.md`, shipped in the source distribution, covers what changed
and why, the measured before/after figures, and the library-API breaking changes.

- **0.2.0**: A `rapport.json` from the web form is now an input. `ggufit json PATH`, or
  `--from-json PATH` on any command, sizes the machine the report describes instead of
  this one, with the report's own context, session count and speed floor as defaults. The
  bandwidth is labelled derived rather than measured, the GPU comes from the report rather
  than from local detection, and a budget that disagrees with the one the client was shown
  is reported instead of silently replaced. The wizard gained a "size a report someone sent
  me" path. New `hardware` command output states what a report cannot know.
- **0.1.4**: GPU support. Every model now gets a second verdict: what it does with the GPU
  helping, and the `-ngl` value to run it with. RAM and VRAM count as one pool, so models too
  big for either alone are correctly reported as runnable. Partial offload is timed with each
  half divided by its own bandwidth. Integrated graphics are identified and excluded from
  speedup claims, since an iGPU reads the same DRAM at the same bandwidth. Bandwidth comes from
  a new 123-entry table (`ggufit/gpus.json`); capacity still comes from detection, and an
  unrecognised card gets no speed estimate. New `--gpu`, `--vram`, `--gpu-layers`, `--no-gpu`
  and `--list-gpus`. Still zero dependencies. Also new: `scan --best` picks one model per use
  case, quantising before ruling anything out so it does not recommend far below what the
  machine can run. And a model outside the catalogue is now fetched from HuggingFace instead
  of rejected, handling gated repos via ungated mirrors, multimodal configs that nest the
  decoder, and the three different spellings of an MoE expert count. That lookup asks
  permission first and defaults to no; `--yes` answers in advance and `--offline` restores the
  no-network behaviour. Bare `ggufit` now runs an interactive wizard rather than printing the
  help, and echoes the command it built so the questions become unnecessary. New `hardware`
  command for the machine profile on its own. See `UPDATE.md` for the reasoning and the math.
  Finally, the output was redesigned around a single-line verdict carrying quantization,
  memory and speed together, with aligned tables that adapt to the terminal width, and
  ggufit now answers in **French** as well as English: bare `ggufit` asks which, the choice
  is remembered, and `ggufit language`, `--lang` and `GGUFIT_LANG` change it at three
  different scopes. The commands and flags stay English in both. And `--users N` sizes a model
  for several people generating at once: the attention cache multiplies while the shared weights
  do not, and speed is reported both per user and as total throughput, since batching moves those
  two in opposite directions. `--users 1` is identical to every earlier version.
- **0.1.3**: KV cache now uses each model's real attention head counts (GQA) or MLA-specific fields
  instead of a flat `hidden_size` estimate, fixing a 4-8x over-estimation on most modern models
  (e.g. Llama 2 70B at 32K context: 80 GB → 10 GB, the correct figure). The memory-bandwidth
  benchmark is now a real multi-process STREAM-style measurement instead of a single-threaded numpy
  copy that read a fraction of true bandwidth. Results are now reported against two RAM figures,   what's free right now and the machine's ceiling with RAM cleared, with a new **Usable if you free
  up RAM** group in `scan` for models that fit the hardware but not the current moment. New
  `--for <use>` filter (code, reasoning, math, vision, chat, general), `--fastest` sort and `--top N`
  limit. Speed is now shown as a range, not a false-precision single number. Dropped the
  `psutil`/`numpy` dependencies, the CLI is standard-library only now. Model catalogue expanded from
  217 hand-typed entries to 218 generated from HuggingFace.
- **0.1.2**: Overhead is now size-dependent (fixed ~1GB baseline + ~7.6% of weights),
  calibrated against real llama.cpp memory reports. Fixes large-model RAM over-estimation
  (DeepSeek-V3 671B Q4 now estimates ~405GB, matching reality, vs ~450GB before). Also fixes
  a wording bug where the "try a lower quant" hint could suggest the same quant that just failed.
- **0.1.1**: Custom use-only license; corrected install instructions.
- **0.1.0**: Initial release (MoE active-param speed, SSM/MLA KV cache handling, 217 models).

## License

`ggufit` uses a custom **use-only** license, not an open-source one. You may install and
run it for any purpose including commercial use, and redistribute unmodified copies. You
may **not** modify it, redistribute modified versions, or sublicense it. Full terms in the
`LICENSE` file.

Version 0.1.0 was published under MIT; that release stays MIT for anyone who already has
it. License changes apply going forward, not retroactively.

## Q&A

A comprehensive FAQ, from "what is a GGUF" to the exact formulas behind every number
`ggufit` prints. Organized beginner → expert.

### Basics

**What is a GGUF?**
GGUF (GPT-Generated Unified Format) is a file format for storing LLM weights, designed
by the llama.cpp project for fast loading and CPU/GPU-flexible inference. It bundles
the model's tensors and metadata (architecture, tokenizer, etc.) into a single file.
It replaced the older GGML format. If you've downloaded a `.gguf` file from Hugging
Face to run in `llama.cpp`, Ollama, or LM Studio, that's what `ggufit` is estimating
compatibility for.

**What is quantization?**
Shrinking a model's weights from their original precision (usually 16-bit floats) down
to smaller representations (8-bit, 4-bit, etc.) to save memory and speed up inference,
at the cost of some accuracy. A "Q4" model uses roughly a quarter of the memory of the
same model at full 16-bit precision.

**What do Q4, Q5, Q6, Q8 mean?**
The number is roughly the average bits per weight after quantization (not exactly,
see next question). Lower number = smaller file, faster inference, more quality loss.
Q4 is the most common "sweet spot" for CPU-only local inference. Q8 is close to
lossless but nearly as large as full precision.

**Why isn't Q4 exactly 4 bits then?**
Modern GGUF "K-quants" (the `_K_M`, `_K_S` suffixes you see on Hugging Face) don't use
a uniform bit-width across the whole model, they mix precision per tensor, using
slightly higher precision for the parts most sensitive to quality loss. So "Q4" is
really an *average* around 4-5 bits/weight in practice. `ggufit` uses effective
per-quant byte values that reflect this real-world average, not naive N-bit math:

| Quant | Effective bytes/param |
|---|---|
| FP32 | 4.0 |
| FP16/BF16 | 2.0 |
| Q8 | 1.05 |
| Q6 | 0.8 |
| Q5 | 0.7 |
| Q4 | 0.6 |
| Q3 | 0.5 |
| Q2 | 0.4 |

**What's the difference between RAM and VRAM, and why does ggufit only care about RAM?**
VRAM is memory on a dedicated GPU; RAM is your system's main memory, used by the CPU.
`ggufit` is CPU-only by design, it answers "can my CPU and system RAM handle this,"
not "can my GPU handle this." If you have a GPU, tools like `nvidia-smi` and the model
card's VRAM requirements are what you want instead.

**What is context length?**
The number of tokens (roughly, chunks of a word) the model can "see" at once, your prompt plus its response so far. Longer context means the model can process
longer documents, conversations, and code, but it also means more memory used for
the KV cache (see below).

**What are tokens/sec, and what's a "good" number?**
How many tokens the model generates per second. For a comfortable reading pace, most
people find 5-15 tok/s tolerable for chat; below ~2 tok/s feels quite slow; above 20
tok/s feels close to instant. It's highly subjective and task-dependent though, background batch jobs can tolerate much lower throughput than an interactive chat.

---

### Installing and running ggufit

**How do I install it?**
```bash
pipx install ggufit    # recommended
pip install ggufit     # inside a venv, or with --break-system-packages
```

**Why do I get "externally-managed-environment"?**
Modern Debian/Ubuntu (PEP 668) blocks system-wide `pip install` to protect the OS's
own Python packages. Use `pipx` instead, it installs into an isolated environment
while still giving you a global command.

**Why does `ggufit` say "command not found" right after installing?**
Either your venv isn't activated (`source venv/bin/activate`), or `pipx`'s bin
directory isn't on your PATH yet (run `pipx ensurepath` and reopen your terminal).

**Why did a model that should fit show as "does not fit"?**
Check which of the two verdicts you're reading. "Right now" reflects what's free at this
instant, so other programs holding memory will fail it; "with RAM cleared" is the machine's
real ceiling. A model in the **Usable if you free up RAM** group fits your hardware fine, it just doesn't fit alongside what you currently have open.

**Can I check a model at a specific quant instead of letting ggufit auto-pick?**
Yes, `ggufit <model> --quant q4`. Without `--quant`, it auto-picks the highest-quality
quant that fits.

**How do I find models for a specific job, or just the fastest ones?**
`ggufit scan --for code` filters to coding models; `--for` also takes `reasoning`, `math`,
`vision`, `chat` and `general`. Add `--fastest` to sort by estimated speed and `--top N`
to cap the list, e.g. `ggufit scan --for code --fastest --top 5`. Tags are editorial
groupings held in `tools/aliases.json`, not something the model files declare, so treat
them as a helpful index rather than an authoritative capability list, plenty of
"general" models write perfectly good code.

---

### Understanding the numbers (intermediate)

**How is "Model size (RAM)" calculated?**
```
model_size_bytes = num_parameters × bytes_per_param
```
`bytes_per_param` comes from the quant table above. This is the number of bytes the
weights occupy once loaded, the dominant factor in "will this even load."

**What is the KV cache, and why does it matter?**
During generation, the model caches the Key and Value tensors from every previous
token in the conversation so it doesn't have to recompute them each step. This cache
grows with context length. The standard formula:
```
kv_cache_bytes = 2 × num_layers × hidden_size × seq_len × batch_size × bytes_per_param
```
The `2×` covers storing both K and V. This is why a model that fits fine at a 4K
context can stop fitting at 32K, the KV cache scales linearly with context length
while the model weights stay fixed.

**Why is there an "overhead factor" on top of model size + KV cache?**
Real inference isn't just raw tensor storage, the runtime, OS, and memory allocator
all need working space too (buffers, fragmentation, temporary activations). It's a
fixed ~1GB baseline plus ~7.6% of weight size (not a flat multiplier, see "How it
works" above), calibrated against real llama.cpp memory reports.

**Why is CPU inference speed based on memory bandwidth instead of raw compute (FLOPS)?**
At batch size 1 (the normal case for a single person chatting), generating each new
token requires reading every single model weight from RAM once. The CPU spends far
more time waiting on memory than doing arithmetic, so the bottleneck is how fast
data can move from RAM to the CPU, not how many operations per second the CPU can do.
That's why the formula is:
```
tokens_per_sec ≈ memory_bandwidth (GB/s) / model_size (GB)
```

**Why does ggufit benchmark my memory bandwidth instead of just knowing it from my RAM specs?**
Achievable bandwidth depends on RAM generation, channel configuration
(single/dual/quad-channel), and platform quirks, none of which are reliably
detectable across Windows/Mac/Linux without vendor-specific tools. So `ggufit` runs
several worker processes copying a buffer concurrently on your actual machine, right
now (a single thread can't saturate a memory controller multiple cores can drive), and
uses that measured number instead of guessing.

**Why does the bandwidth number change slightly every time I run ggufit?**
It's a live micro-benchmark, not a cached constant, normal system load, thermal
throttling, and other processes competing for memory access all cause small
run-to-run variance. That's expected and not a bug.

---

### Architecture-specific math (expert / advanced)

**What is a Mixture-of-Experts (MoE) model, and why does it need special handling?**
An MoE model has many "expert" sub-networks, but only a subset of them are activated
for any given token (a small router network decides which experts to use). This means:
- **All experts must be loaded into RAM**: because any token could route to any
  expert, so the *fit* check uses the full `params_billion` (total, all experts).
- **Only the active experts are actually read from RAM per token**: so the *speed*
  estimate uses `active_params_billion` instead. Using total params for speed
  would make big MoE models look absurdly slow (DeepSeek-V3 at 671B total but only
  37B active would look ~18x slower than it really is if you used the total).

```
fit check:   uses params_billion (total), all experts must be resident
speed check: uses active_params_billion, only active experts are read per token
```

**What is GQA (Grouped-Query Attention), and how does ggufit account for it?**
Most modern transformer models don't give every attention head its own K/V
projection, they group multiple query heads to share a smaller number of K/V heads
(`num_kv_heads < num_attention_heads`). This makes the *real* KV cache smaller than
a formula assuming `hidden_size` worth of K/V per layer would predict, over-estimating
by 4-8x for models like Llama 2 70B or Mistral 7B. `ggufit` uses each model's actual
`num_key_value_heads` and `head_dim` (sourced from its Hugging Face `config.json`) to
compute the real KV cache size, not a blanket correction factor, that matters because
some models (Phi-3-mini) are genuinely full multi-head attention, and a global
multiplier would break those while fixing everything else. Only a handful of older or
override-only models still fall back to the conservative `hidden_size` estimate,
because no published head-count data exists for them.

**What is MLA (Multi-head Latent Attention)?**
MLA (used by DeepSeek-V2/V3/R1 and MiniCPM3) compresses the K/V representations into
a much smaller latent vector before caching them, then reconstructs full K/V on the
fly. `ggufit` computes this directly from the model's real `kv_lora_rank` and
`qk_rope_head_dim` fields, one compressed latent vector cached per layer, not one
per head, rather than applying an approximate correction factor.

**Why do pure Mamba/SSM models show a 0 GB KV cache?**
State-Space Models (Mamba, used in Codestral Mamba and Falcon-Mamba) don't use
attention at all, they maintain a fixed-size recurrent state instead of caching
every previous token's K/V. That state doesn't grow with context length. So their
KV cache is genuinely zero, regardless of how long the context gets. This is why
`ggufit falcon-mamba --context 65536` still shows ~0GB KV cache even at a huge
context length.

**What about hybrid models like Jamba?**
Jamba interleaves Mamba blocks with regular attention blocks, only a fraction of
layers actually use attention. `ggufit` doesn't yet have published per-layer
attention-head data for these hybrid architectures, so they fall back to the
conservative `hidden_size`-based estimate rather than a wrong precise-looking number.

**Why does `ggufit` need real `layers` and `hidden_size` values instead of estimating them from param count?**
Because two models with the same parameter count can have very different KV cache
sizes depending on how those parameters are distributed across layers and hidden
dimension, there's no reliable shortcut from param count alone. `ggufit` stores
these values explicitly per model (sourced from each model's Hugging Face
`config.json`) to keep the KV cache estimate accurate.

---

### Project & contributing

**How do I add a model that isn't in the database?**
The catalogue (`ggufit/models.json`) is generated, not hand-edited, see the `ggufit`
repo's `tools/fetch_model_specs.py`. Add the model's HuggingFace repo id to
`tools/repos.txt`, re-run the fetcher, and (for MoE models, or repos HuggingFace can't
resolve directly) add an entry to `data/overrides.json` for whatever the fetcher
couldn't derive on its own, most commonly `active_params_billion`.

**Why was the project renamed from moscan to ggufit?**
The name `moscan` was already taken on PyPI. `ggufit` was chosen instead, pun on
GGUF (the file format) + "fit" (does the model fit on your machine).

**What license is ggufit under?**
A custom "use-only" license: you're free to install and run it for any purpose,
including commercial use, but you may not modify it or redistribute a modified
version. See the `LICENSE` file for exact terms. Note: versions published before
this license was adopted (0.1.0) remain under their original MIT terms for anyone
who already obtained that specific release, license changes only apply going
forward, not retroactively.

**Is this a substitute for actually running the model to see how it performs?**
No, treat every number here as an estimate to guide a decision (e.g. "should I even
attempt downloading this 40GB file"), not a guarantee. Real-world speed depends on
your specific inference engine (llama.cpp, Ollama, etc.), thread count settings, OS
scheduler behavior, and quantization implementation quality, none of which `ggufit`
can measure without you actually running the model.
