Metadata-Version: 2.4
Name: llm-dequant
Version: 0.1.0
Summary: Dequantize compressed-tensors NVFP4 LLM checkpoints to dense safetensors (streaming, constant memory)
Author: Victor Cruz
License: Apache-2.0
Project-URL: Repository, https://github.com/vcruz305/llm-dequant
Keywords: nvfp4,quantization,dequantization,safetensors,llm,compressed-tensors,gguf
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.0
Requires-Dist: safetensors>=0.4
Requires-Dist: numpy>=1.24
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Dynamic: license-file

<div align="center">

<img src="llm-dequant.png" alt="llm-dequant — stream NVFP4 compressed-tensors checkpoints back to dense safetensors" width="50%">

**Turn NVFP4-only model releases back into dense safetensors.**

Streaming, constant-memory dequantization of
[compressed-tensors](https://github.com/neuralmagic/compressed-tensors)
`nvfp4-pack-quantized` LLM checkpoints — a 300B-parameter MoE converts
on a few hundred MB of RAM.

[![tests](https://github.com/vcruz305/llm-dequant/actions/workflows/tests.yml/badge.svg)](https://github.com/vcruz305/llm-dequant/actions/workflows/tests.yml)
[![license](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
[![python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](pyproject.toml)

</div>

---

## Why this exists

A growing number of community models — abliterations, fine-tunes, supertunes —
are published **only** in NVFP4 compressed-tensors form. That single release
format locks out most of the ecosystem:

- `llama.cpp`'s `convert_hf_to_gguf.py` can't read compressed-tensors
  checkpoints → **no GGUFs**
- fine-tuning frameworks expect dense weights → **no further training**
- weight surgery (MTP grafts, merges, pruning) needs real tensors → **no surgery**

The reference decompression path in `compressed-tensors`/`transformers`
materializes the whole model in memory, which is a non-starter at 300B scale.

`llm-dequant` fixes this with a purpose-built converter:

| | |
|---|---|
| 🌊 **Streaming** | module-by-module; peak RSS = one module + one shard buffer |
| 🎯 **Exact** | output is bit-identical to the reference implementation (CI-checked) |
| 🧾 **Standard output** | sharded safetensors + index, `quantization_config` stripped, tokenizer files copied — a normal dense HF checkpoint |
| 🪶 **Dependency-light** | `torch`, `safetensors`, `numpy`. No compressed-tensors, no transformers |
| 🔍 **Self-verifying** | `verify` proves round-trip byte-equality against *your actual checkpoint* before you commit to a 600GB write |
| 🛡️ **Defensive** | untrusted `index.json` path names rejected, NaN/Inf scales refuse loudly, unsupported formats never convert silently |

Born from converting a 295B MoE released as NVFP4-only into a full GGUF quant
sweep ([SuperHY3-abliterated-GGUF](https://huggingface.co/vcruz305/SuperHY3-abliterated-GGUF)).

## Install

```bash
pip install llm-dequant            # from PyPI (published releases)
```

or straight from GitHub (works on any commit, no release needed):

```bash
pip install git+https://github.com/vcruz305/llm-dequant.git
```

Both pull in `torch`, `safetensors`, and `numpy` automatically. For a
CPU-only torch (much smaller download):

```bash
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install llm-dequant
```

## Quickstart

```bash
# 1. what is this checkpoint?
llm-dequant inspect /models/SomeModel-NVFP4

# 2. prove the math round-trips on THIS checkpoint (seconds, no writes)
llm-dequant verify /models/SomeModel-NVFP4 --sample 16

# 3. convert (bf16 default; fp16/fp32 available)
llm-dequant convert /models/SomeModel-NVFP4 /models/SomeModel-BF16
```

Then use the output like any dense checkpoint:

```bash
python llama.cpp/convert_hf_to_gguf.py /models/SomeModel-BF16 \
    --outfile SomeModel-BF16.gguf --outtype bf16
```

### Python API

```python
from llm_dequant.model import convert, verify, inspect_checkpoint
from llm_dequant.nvfp4 import dequantize_nvfp4, unpack_fp4, pack_fp4

convert("/models/in-nvfp4", "/models/out-bf16", dtype="bf16")
```

## The NVFP4 format, in one table

`nvfp4-pack-quantized` (emitted by
[llm-compressor](https://github.com/vllm-project/llm-compressor)) stores, per
quantized Linear:

| tensor | dtype | shape | meaning |
|---|---|---|---|
| `weight_packed` | uint8 | `[m, n/2]` | two FP4-E2M1 codes per byte, low nibble first |
| `weight_scale` | fp8-e4m3 | `[m, n/16]` | one scale per group of 16 columns |
| `weight_global_scale` | fp32 | scalar | per-module global scale *(optional)* |

An FP4 code is 1 sign bit + 3 bits indexing the E2M1 magnitude table
`[0, 0.5, 1, 1.5, 2, 3, 4, 6]`, and

```
W[i, j] = e2m1(code[i, j]) · scale[i, j//16] / weight_global_scale
```

**Dequantization is exact.** Every recovered value sits precisely on the
scaled FP4 lattice (computed in fp32, cast once to the output dtype). Nothing
the NVFP4 checkpoint encodes is lost. What the original 4-bit quantization
already destroyed is of course not recoverable — the output is *dense*, not
*restored*. If you re-quantize the result (e.g. to GGUF), disclose the FP4
provenance: tiers at or below ~4 bpw lose essentially nothing, higher tiers
only improve the never-quantized tensors.

## How correctness is established

Two independent layers, both of which you can run yourself:

**1. `llm-dequant verify <src>` — against your checkpoint.**
Samples random packed modules, dequantizes, renormalizes by the effective
scale, re-packs with nearest-E2M1 rounding, and requires **byte equality**
with the original `weight_packed`. Any error in nibble order, scale layout,
group mapping, or global-scale handling breaks byte equality. (This proves
self-consistency against the real bytes; spec conformance comes from layer 2.)

**2. The test suite — against the reference implementation.**
Hand-computed vectors, exhaustive FP4 code tables, property-based round-trips,
end-to-end synthetic checkpoints (sharded + single-file, with/without global
scale), corruption and path-traversal rejection, and a bit-exactness test
against upstream `compressed-tensors` (runs in CI).

```bash
pip install -e .[dev] && pytest
```

## Scope

Supported: `format: nvfp4-pack-quantized`, symmetric, `group_size: 16` — the
only layout llm-compressor emits for NVFP4 today. Fused-MoE expert tensors
(`[experts, m, n/2]`) work unchanged.

Rejected loudly, never converted silently: other compressed-tensors formats,
asymmetric quantization (`weight_zero_point`), NaN/Inf scales, shard names
that escape the checkpoint directory, colliding output tensor names.

Dropped by design: activation-quantization metadata (`input_global_scale`) —
the output is dense; runtime activation quantization no longer applies.

## License

Apache-2.0. The FP4/E2M1 constant table follows the format specification
established by the vLLM / compressed-tensors projects (Apache-2.0).
