Metadata-Version: 2.4
Name: gguf2mlx
Version: 2.2.0
Summary: Convert verified GGUF models to MLX-LM safetensors on Apple Silicon
License: Apache-2.0
Project-URL: Homepage, https://github.com/barrontang/gguf2mlx
Project-URL: Documentation, https://github.com/barrontang/gguf2mlx#readme
Project-URL: Repository, https://github.com/barrontang/gguf2mlx
Project-URL: Issues, https://github.com/barrontang/gguf2mlx/issues
Project-URL: Changelog, https://github.com/barrontang/gguf2mlx/blob/main/CHANGELOG.md
Keywords: gguf,mlx,mlx-lm,apple-silicon,safetensors,llama.cpp,model-conversion
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Environment :: MacOS X
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Build Tools
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: gguf>=0.18.0
Requires-Dist: safetensors
Requires-Dist: numpy
Requires-Dist: tqdm
Requires-Dist: mlx>=0.18.0; sys_platform == "darwin" and platform_machine == "arm64"
Requires-Dist: mlx-lm>=0.29.0; sys_platform == "darwin" and platform_machine == "arm64"
Provides-Extra: mlx
Requires-Dist: mlx>=0.18.0; (sys_platform == "darwin" and platform_machine == "arm64") and extra == "mlx"
Requires-Dist: mlx-lm>=0.29.0; (sys_platform == "darwin" and platform_machine == "arm64") and extra == "mlx"
Provides-Extra: vlm
Requires-Dist: huggingface-hub>=1.0.1; extra == "vlm"
Provides-Extra: publish
Requires-Dist: huggingface-hub>=1.0.1; extra == "publish"
Provides-Extra: ppl
Requires-Dist: llama-cpp-python==0.3.36; extra == "ppl"
Provides-Extra: all
Requires-Dist: mlx>=0.18.0; (sys_platform == "darwin" and platform_machine == "arm64") and extra == "all"
Requires-Dist: mlx-lm>=0.29.0; (sys_platform == "darwin" and platform_machine == "arm64") and extra == "all"
Requires-Dist: huggingface-hub>=1.0.1; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: pyyaml; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Requires-Dist: tokenizers>=0.20; extra == "dev"
Requires-Dist: jinja2>=3.1.6; extra == "dev"
Requires-Dist: transformers>=5.19.0; extra == "dev"
Dynamic: license-file

# gguf2mlx — convert GGUF to MLX without guessing

<div align="center">

[![PyPI](https://img.shields.io/pypi/v/gguf2mlx.svg)](https://pypi.org/project/gguf2mlx/)
[![CI](https://github.com/barrontang/gguf2mlx/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/barrontang/gguf2mlx/actions/workflows/ci.yml)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-green.svg)](LICENSE)

**GGUF → HF-style safetensors that MLX-LM and `mlx-vlm` can load.
Explicit architecture contracts, no silent precision downgrade, fail-closed when a layout is not supported.**

[Quick start](#quick-start) · [Supported models](#supported-models) · [Vision](#vision-language-models) · [Docs](#documentation)

</div>

---

## Why this exists

GGUF is a distribution format; MLX-LM and `mlx-vlm` expect a Hugging Face-style
directory with `config.json`, tokenizer assets and safetensors weights. `gguf2mlx`
bridges that gap for **explicitly supported** architectures by reading GGUF
metadata and tensors, rebuilding the MLX-compatible layout, and **failing closed**
when a model layout is not actually supported — no silent "Llama fallback".

Use it when a model ships GGUF but no MLX checkpoint, or when you need an
inspectable HF-style directory on an M1–M4 Mac. If the original Hugging Face
weights are available, prefer `mlx_lm.convert` — it avoids inheriting the source
GGUF's quantization error.

Conversion itself only needs `numpy`, `gguf` and `safetensors`, so it runs fine
on Linux servers and CI; MLX is required only for `--quantize`.

> **Important:** the default output is FP16/BF16 safetensors. `--quantize`
> **re-quantizes** from that FP16 copy. To keep the bytes that are already in the
> GGUF, use `--quant-source preserve`, which re-expresses the source blocks as
> MLX affine weights for `llama`, `qwen2`, `qwen3` and `mistral` (see
> [Preserve GGUF quantization](#preserve-gguf-quantization)).

## Install

```bash
pip install gguf2mlx            # installs MLX + MLX-LM on Apple Silicon
pipx install gguf2mlx           # isolated CLI
uv tool install gguf2mlx        # or with uv
```

Extras: `[vlm]`/`[publish]` add Hugging Face Hub, `[all]` everything, `[dev]`
the test/lint toolchain.

## Quick start

```bash
# GGUF -> FP16 MLX-LM directory
gguf2mlx --input model.gguf --output ./mlx-model

# GGUF -> 4-bit MLX-LM output
gguf2mlx --input model-Q4_K_M.gguf --output ./mlx-model-4bit \
  --quantize --q-bits 4 --q-group-size 64

# Bounded-memory direct quantization (no FP16 staging directory)
gguf2mlx --input model-Q4_K_M.gguf --output ./mlx-model-4bit \
  --quantize --direct-quant --q-bits 4 --q-group-size 64 --q-mode affine

# Reuse the GGUF blocks instead of quantizing the FP16 copy again
gguf2mlx convert --input model-Q4_K_M.gguf --output ./mlx-model-preserved \
  --quant-source preserve

# Inspect metadata without writing weights
gguf2mlx --input model.gguf --skip-weights

# Smaller shards (payload bytes, not source bytes)
gguf2mlx --input model.gguf --output ./mlx-model --max-shard-gb 2

# Package / verify a directory as a .mlx bundle
gguf2mlx package --model-dir ./mlx-model --output ./mlx-model.mlx
gguf2mlx verify --bundle ./mlx-model.mlx
```

Load the result:

```python
from mlx_lm import load, generate
model, tok = load("./mlx-model")
print(generate(model, tok, prompt="Hello from MLX", max_tokens=32))
```

See [How conversion works](docs/how-conversion-works.md) for streaming, sharding
and offline tokenizer reconstruction.

## Is the output trustworthy?

Every conversion writes `conversion_report.json` and gates on **weight
fidelity**: each written tensor is compared position-wise against an
independent decode of the source GGUF, with a per-dtype relative-L2 tolerance.
Fused or permuted tensors are reported as `transformed` rather than silently
counted as verified.

```bash
# Re-audit an existing artifact against the GGUF it came from
gguf2mlx verify --model-dir ./mlx-model --weights --source model.gguf

# Load it with mlx_lm and require finite logits from one forward pass
gguf2mlx verify --model-dir ./mlx-model --load

# Skip the (roughly 2x) audit cost
gguf2mlx convert --input model.gguf --output ./mlx-model --no-weight-audit
```

Perplexity is recorded but never gated: cross-runtime PPL mixes quantization
noise with conversion error, so it cannot prove a converter correct. Nightly,
the pinned [load matrix](tests/load_matrix.json) converts real GGUF models and
proves each one loads and runs.

## Preserve GGUF quantization

`--quant-source preserve` re-expresses the GGUF blocks that are already there as
MLX affine groups (`w = scale * code + bias`) on the 32-value group grid the
blocks use, instead of decoding to FP16 and quantizing again. The integer codes
are kept; only the fp16 rounding of the scale and bias is new. Measured against
`gguf.quants.dequantize`, per-tensor relative L2 is 2.0-2.4e-4 for Q4_K, 0.0
(bit-exact) for Q8_0, and 2.2e-4 for Q6_K when a 32-value group's two 16-value
sub-block scales agree.

Q6_K has a structural limit: its sub-block scales are 16 values wide, so when
they disagree inside a 32-value group it cannot be represented exactly. Those
tensors are flagged `exact: false` and kept in **FP16** rather than silently
requantized, and `conversion_report.json` records the decision plus per-tensor
max abs error, relative L2 and an aggregate 1e-3 gate. Modules end up with mixed
4/6/8-bit widths, which `mlx-lm` loads via per-module `quantization` entries.

Preserve fails closed: `llama`, `qwen2`, `qwen3` and `mistral` are supported, and
any other architecture errors out naming the arch instead of falling back to
requantization. `--preserve-source-quant` still works as a deprecated alias.

```bash
gguf2mlx convert --input model-Q8_0.gguf --output ./mlx-model-q8 --quant-source preserve
```

## Publish to the Hugging Face Hub

`gguf2mlx publish` downloads a hosted GGUF, converts it, and uploads the result
with a provenance record and the source license files. Split GGUFs are merged;
a partial shard set is rejected before anything is downloaded, and the
destination repository is only created once conversion produced weights.

```bash
pip install "gguf2mlx[publish]"
gguf2mlx publish --source bartowski/Model-GGUF --gguf Model-Q8_0.gguf \
  --dest my-org/Model-MLX --source-revision main
# add --quantize --q-bits 4 --q-group-size 64 for affine 4-bit (needs MLX)
```

The reusable `publish-mlx.yml` workflow is a thin shell around the same module:
float16 runs on `ubuntu-latest`, and only `quantize: true` needs a macOS Apple
Silicon runner. Details: [Publishing](docs/publishing.md).

## Supported models

Conversion is adapter-based, not "any GGUF". Rows marked ⚠️ convert but still
need opt-in real-checkpoint validation.

| Family | GGUF architecture IDs | Convert | End-to-end evidence |
|---|---|---|---|
| Llama | `llama`, `mistral` | ✅ + preserve | synthetic direct-quant; SmolLM2-135M HF parity |
| Qwen | `qwen2`, dense `qwen3` | ✅ + preserve | synthetic direct-quant |
| Qwen3.5 family | `qwen35` + aliases | ✅ | portable fixtures |
| Qwen MoE | `qwen2moe`, `qwen3moe` | ✅ | real GGUF: Qwen1.5-MoE, Qwen3-30B-A3B |
| DeepSeek | `deepseek2`, `deepseek3` | ✅ | real GGUF: V2 Lite Q2_K |
| GLM | `glm4moe`, `glm-dsa` | ✅ / ⚠️ experimental | none |
| Gemma | `gemma`, `gemma2`, `gemma3`, text `gemma4` | ✅ | portable fixtures |
| Phi | `phi3` (Phi-3, Phi-4) | ✅ | synthetic GGUF |
| StableLM | `stablelm` | ✅ | none |
| LLaVA / Qwen2-VL / MiniCPM-V | mmproj pairs | ✅ | Qwen2-VL-2B real-image acceptance |

`preserve` in the Convert column means `--quant-source preserve` is available
(Q4_K, Q6_K, Q8_0). Other architectures convert, but must requantize.

**Inspection-only** (recognized via metadata, rejected on conversion): `arctic`,
`bert`, `bitnet`, `bloom`, `dbrx`, `falcon`, `gpt2`, `gptneox`, `mpt`, `olmo`,
`phi`, `phi2`, `t5` and others — full list in the
[support matrix](docs/support-matrix.md).

## Vision-language models

Three explicit adapters emit directories for `mlx-vlm`: **LLaVA-1.5**,
**Qwen2-VL**, and **MiniCPM-V 4.7 1B**. Language weights and the projector come
from GGUF; hybrid mode takes only the vision tower from a matching HF
checkpoint. Qwen2.5-VL, Qwen3-VL full VLM, LLaVA-NeXT and SigLIP layouts are
**not** supported by these adapters.

```bash
pip install "gguf2mlx[vlm]"
gguf2mlx convert --type vlm \
  --input qwen2-vl.gguf --mmproj qwen2-vl-mmproj.gguf \
  --hf-model Qwen/Qwen2-VL-2B-Instruct --output ./qwen2-vl-mlx
```

Flags, hybrid/offline paths, quantization policy and the Q4_K
`--preserve-source-quant` mode: [VLM conversion](docs/vlm-conversion.md). Native
acceptance results: [Qwen2-VL report](docs/qwen2-vl-real-validation.md).

Qwen3-VL **text-only** extraction is a separate `--type text-encoder` contract
(BF16 by default, mRoPE preserved, vision tensors skipped): see
[text conditioning](docs/text-encoder.md).

## Limits worth knowing before you start

- **`--quantize` is a second quantization.** Use `--quant-source preserve` to
  keep the original blocks; for Q6_K tensors whose sub-block scales disagree,
  that still means FP16.
- **Conversion cannot restore precision** lost in the source GGUF.
- **Tokenizer fidelity depends on GGUF metadata**; embedded HF tokenizer JSON is
  preserved when present, otherwise reconstructed from the header.
- **Mixed-precision MoE output is experimental** and needs a JANG-compatible
  runtime; use uniform output when interoperability is unverified.
- **No universal speed claim** versus llama.cpp — measure on your hardware.

## Documentation

[Quality gates](docs/quality-gates.md) · [Support matrix](docs/support-matrix.md) · [How conversion works](docs/how-conversion-works.md) ·
[VLM conversion](docs/vlm-conversion.md) · [Text conditioning](docs/text-encoder.md) · [Publishing](docs/publishing.md) ·
[`.mlx` bundles](docs/mlx-bundle.md) · [Capabilities](docs/capabilities.md) · [Benchmarks](docs/benchmarks.md) ·
[FAQ](docs/faq.md) · [Roadmap](docs/roadmap.md) · [Validation logs](docs/validation/) · [Development](docs/development.md)

## Development

```bash
git clone https://github.com/barrontang/gguf2mlx.git && cd gguf2mlx
uv sync --all-extras
pytest -q                                   # portable suite
GGUF2MLX_RUN_E2E=1 pytest tests/test_e2e.py # Apple Silicon, real GGUF
ruff check src/ tests/ benchmarks/
```

CI runs Python 3.10/3.12/3.13 with an 80% coverage floor; a separate macOS
workflow runs the native VLM suites. Details and opt-in fixture paths are in
[Development](docs/development.md).

## Contributing

New architecture adapters must ship with a tensor manifest and a load test.
When requesting a family, include the exact GGUF architecture, model name,
quantization type, tokenizer type, and a public fixture URL.

## License

Apache-2.0. The traffic-tracking dashboard that used to live here was GPL-3.0 and
now lives in its own repository.
