Metadata-Version: 2.4
Name: ggufit
Version: 0.1.1
Summary: Check whether a local LLM can run on your machine (CPU-only inference).
Author: Mouad
License: Ggufit Use-Only License
        
        Copyright (c) 2026 Mouad
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to use
        and run the Software for any purpose, including commercial purposes, and to
        redistribute unmodified copies of the Software, subject to the following
        conditions:
        
        1. The above copyright notice and this permission notice shall be included
           in all copies of the Software.
        
        2. You may NOT modify, adapt, translate, or create derivative works based
           on the Software.
        
        3. You may NOT redistribute the Software, or any portion of it, in modified
           form.
        
        4. You may NOT sublicense the Software under different terms.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
        ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
        WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
        
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Utilities
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: psutil>=5.9
Requires-Dist: numpy>=1.24
Dynamic: license-file

# ggufit

Check whether a local LLM can run on your machine, CPU-only — from PowerShell or any terminal.

Uses real formulas (model memory footprint, KV cache size, and a measured memory-bandwidth
micro-benchmark) rather than guesses, to estimate whether a model fits in RAM and roughly
how fast it'll generate tokens. Correctly handles MoE models (speed driven by active
experts, not total params) and SSM/MLA architectures (Mamba, DeepSeek's MLA) that don't
use standard multi-head attention.

## Install

```bash
# Recommended - works everywhere, gives you a global `ggufit` command
pipx install ggufit

# Or plain pip (Windows PowerShell, or inside a venv on Linux/macOS)
pip install ggufit
```

### Installing from source (for development)

```bash
git clone <this repo>   # or unzip the source
cd ggufit/               # the folder containing pyproject.toml
pipx install -e .        # editable install - code changes apply without reinstalling
# or: pip install -e .   (inside a venv, or with --break-system-packages)
```

## Usage

```bash
# Full hardware scan: shows your CPU/RAM/bandwidth and which models fit
ggufit scan

# Check one specific model
ggufit llama3.1
ggufit mistral
ggufit qwen2.5-14b

# Force a specific quantization level
ggufit llama3.1 --quant q4

# Evaluate at a longer context length
ggufit qwen2.5-32b --context 16384
```

## How it works

For each model:
1. **Model size (RAM)** = total params × bytes-per-parameter (varies by quant: FP16, Q8, Q6, Q5, Q4, Q3, Q2)
2. **KV cache** = `2 × layers × hidden_size × context_len × bytes_per_param × kv_cache_multiplier`
   - `kv_cache_multiplier` defaults to 1.0 (standard MHA), and is set lower for architectures
     that don't use full attention at every layer: `0.0` for pure SSM/Mamba (no attention at all),
     `~0.15` for MLA (DeepSeek-V2/V3/R1, MiniCPM3), `~0.125` for hybrid Mamba+attention (Jamba).
3. **Total RAM needed** = `(model size + KV cache) × 1.2` overhead factor
4. **Speed estimate** = `measured_memory_bandwidth / active_size`, where `active_size` is the
   total model size for dense models, or just the active experts' size for MoE models
   (`active_params_billion`) — since only those weights are streamed from RAM per token.

`ggufit` runs a quick real memory-bandwidth benchmark (a large array copy) instead of guessing
from RAM specs, since achievable bandwidth depends heavily on channel configuration.

## Adding models

Edit `ggufit/models_db.py` and add an entry to the `MODELS` dict with `params_billion`,
`layers`, and `hidden_size` (found in the model's Hugging Face `config.json`). Optional:
- `moe: True` + `active_params_billion: X` for Mixture-of-Experts models
- `kv_cache_multiplier: X` for non-standard attention architectures (see above)

## Known limitations

- Speed estimates assume batch size 1, single-user chat.
- The bandwidth benchmark is single-threaded; real inference engines use multiple threads.
- Quant byte-per-param values are approximations of real GGUF file sizes.
- KV cache still assumes full hidden_size for standard (non-flagged) models even though most
  modern ones use GQA with fewer KV heads than query heads — this is intentionally
  conservative (over-estimates KV, doesn't affect the dominant "does it load" answer).
- `kv_cache_multiplier` values for MLA/hybrid architectures are approximate, not
  per-model-measured.

## Notes

This is a side-project CLI. The full hardware-scan desktop app (Rust/Tauri) is a separate,
more thorough tool still in development.
