Metadata-Version: 2.4
Name: quantfit
Version: 0.4.0
Summary: Quantize an LLM and check it still refuses what it should — a GPU-aware quantization CLI that measures the safety drift of the quantization it just performed.
Author: Sahil Kadadekar
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/Sahil170595/quantfit
Project-URL: Repository, https://github.com/Sahil170595/quantfit
Keywords: llm,quantization,awq,gptq,gguf,fp8,gpu,safety
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.4
Requires-Dist: transformers>=4.56
Requires-Dist: huggingface_hub>=0.25
Requires-Dist: datasets>=3.0
Requires-Dist: accelerate>=1.0
Requires-Dist: llmcompressor<0.13,>=0.5
Requires-Dist: psutil>=5.9
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: ruff>=0.3; extra == "dev"
Requires-Dist: scipy>=1.11; extra == "dev"
Provides-Extra: gguf
Requires-Dist: gguf>=0.10; extra == "gguf"
Dynamic: license-file

# quantfit

**Quantize an LLM — and check it still refuses what it should.**

Quantization makes a model cheaper to serve. It can also quietly strip safety
behavior: a 4-bit model that answers prompts the full-precision model refused is a regression
you will not see in a perplexity number. `quantfit` quantizes across the SOTA method
matrix, is honest about whether a model fits your GPU, and — uniquely — measures the
**safety drift** of the quantization it just performed.

```bash
pip install quantfit

quantfit check        --model Qwen/Qwen2.5-7B-Instruct                 # will it fit? (no download)
quantfit plan         --model Qwen/Qwen2.5-7B-Instruct                 # what config would it pick? + why
quantfit quantize     --model Qwen/Qwen2.5-1.5B-Instruct --method awq --out ./out
quantfit probe        --model Qwen/Qwen2.5-1.5B-Instruct --bits 4 8    # per-bit-width quant sensitivity
quantfit verify-safety --baseline Qwen/Qwen2.5-1.5B-Instruct --quant ./out  # did quantization break refusals?
```

## The safety check — what nothing else does

`verify-safety` generates from both the unquantized baseline (at its native dtype)
and the quantized model over a
curated probe set, judges each response refusal/compliance with a local classifier,
and reports the drift as a **vector**, the way it actually matters:

```
safety drift over 40 probes — REGRESSION DETECTED (over-refusal axis)
  refusal-robustness (expected-unsafe n=12): baseline refused 12 -> quant 12
    harmful-compliance regressions: 0/12 at-risk pairs flipped (95% CI upper 24.2%; ~13pp detectable at 80% power)
  over-refusal       (expected-safe   n=28): baseline refused 18 -> quant 18
    over-refusal regressions: 2/10 at-risk pairs flipped (20.0%, 95% CI 5.7-51.0%)
  by zone (baseline->quant refusals / n): borderline[10->10/16] clear_safe[8->8/12] clear_unsafe[12->12/12]
  note: 40 curated probes; a no-detection result bounds the drift, it does not certify safety.
```

Two axes, not one number:
- **refusal-robustness drift** — on prompts that *should* be refused, did the quant
  start complying? (the dangerous direction)
- **over-refusal drift** — on prompts that *should* be answered, did the quant start
  refusing? (the usability direction)

A scalar refusal-delta can read 0 while both axes move in opposite directions; the
vector + per-zone breakdown catches it. Local judge, curated public probes, no
external API and no raw harmful corpora — so the check is distributable.

Verdicts are **bounded, never absolute**: each axis is a binomial over its *at-risk
pairs* (probes the baseline got right), reported with a Wilson 95% CI and — on
zero flips — the minimum detectable effect at 80% power. The intervals are
cross-checked against scipy in CI. At the shipped probe set's n, a pass bounds the
dangerous flip rate below ~24pp; it does not certify safety. (Why "drift" and not
"tax": in the alignment literature a safety/alignment *tax* is capability paid FOR
safety — nearly the inverse of what this measures.)

Add `--report drift.json` to write the run as an **auditable artifact** (schema v1):
judge + probe-set revision pins, the pinned judge input contract, decode params,
resolved per-arm dtypes (never "auto"), an environment fingerprint, per-arm
runtimes, and the full drift vector with CIs — enough to audit, diff against a
rerun, or cite.

## GPU-aware quantization

**3-tier capacity.** `check` reads HF metadata (no download) to estimate the footprint:
fits VRAM (and RAM — weights always stage in CPU RAM first) → fast; too big for VRAM
but fits RAM+disk → same mechanism, slower (weights
load into CPU RAM and llm-compressor's default **sequential onloading** streams one
layer at a time to the GPU — no accelerate `device_map`; validated at in-GPU sizes,
the exceeds-VRAM case is the design target — see *What it is — and isn't*); won't fit
even in RAM → refuse, naming the real limit. No OOM 20 minutes into a job.

**Method × scheme matrix** (one llm-compressor backend, vLLM-loadable):

| method | what | default scheme |
|---|---|---|
| `awq` | activation-aware weight quant (best 4-bit quality) | W4A16_ASYM |
| `gptq` | Hessian/OBQ weight quant | W4A16 |
| `smoothquant` | activation smoothing + W8A8 | W8A8 |
| `fp8` | FP8 E4M3 dynamic, no calibration | FP8_DYNAMIC |
| `rtn` | round-to-nearest baseline | W4A16 |

Schemes (`--scheme`): `W4A16`, `W4A16_ASYM`, `W8A16`, `W8A8`, `INT8`, `W4A8`,
`FP8_DYNAMIC`, `NVFP4`, `MXFP4`. Defaults are the validated paths; FP4 schemes need
Blackwell to *serve* (quantfit can still produce them anywhere).

**GGUF** (`--method gguf`) for Ollama / llama.cpp: `Q2_K`..`Q8_0` + `IQ4_XS`.
Auto-provisions the prebuilt `llama-quantize` binary + convert script (override with
`QUANTFIT_LLAMACPP`).

One frozen packed calibration (wikitext-103, 128 samples, seq-len 2048, seed 42,
group-size 128) is shared across the calibrated methods, so they are comparable.

## What it is — and isn't

- It **quantizes** (wrapping llm-compressor + llama.cpp) and **checks safety
  preservation**. Both run end-to-end and are validated on small models (Qwen-1.5B,
  Llama-1B); exceeds-VRAM quantization (llm-compressor's sequential onloading) is the
  intended design, not yet validated at scale.
- It ships **transparent config help**, not auto-quantization: `quantfit plan --model <id>`
  shows the config a heuristic would pick and *why* (instant, no quantize); `quantfit
  probe --model <id>` measures per-bit-width quantization sensitivity (forward-only RTN-KL,
  a conservative upper bound — see the caveat in `policy/probe.py`).
- It does **not** *auto-pick the method and quantize* for you — you pass `--method`.
  Learned routing ([AMQ](https://arxiv.org/abs/2509.12019),
  [KL-Lens](https://arxiv.org/abs/2604.13440)) exists as published research, but it is
  explicitly out of scope here (see `ROADMAP.md`): quantfit's bet is honest
  measurement, and `plan`/`probe` stay transparent diagnostics.

## Docker

`Dockerfile` builds an isolated CUDA image. For GGUF in Docker, the official
`ghcr.io/ggml-org/llama.cpp:full` image carries the convert + quantize tooling.

## License

Apache-2.0.
