Metadata-Version: 2.4
Name: turboquant-mlx-full
Version: 0.25.0
Summary: Extreme weight and KV cache compression for LLMs on Apple Silicon (MLX implementation of Google's TurboQuant)
Author-email: Manjunath Janardhan <manjunath.shiva@gmail.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/manjunathshiva/turboquant-mlx
Project-URL: Repository, https://github.com/manjunathshiva/turboquant-mlx
Project-URL: Issues, https://github.com/manjunathshiva/turboquant-mlx/issues
Keywords: mlx,quantization,llm,kv-cache,apple-silicon,turboquant,polarquant,moe,long-context
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Operating System :: MacOS
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: LICENSE-MIT
License-File: NOTICE
Requires-Dist: mlx>=0.20.0
Requires-Dist: mlx-lm>=0.31.3
Requires-Dist: transformers!=5.13.*
Requires-Dist: numpy
Provides-Extra: eval
Requires-Dist: datasets; extra == "eval"
Requires-Dist: transformers; extra == "eval"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Provides-Extra: vlm
Requires-Dist: mlx-vlm>=0.6.12; extra == "vlm"
Provides-Extra: kimi
Requires-Dist: tiktoken; extra == "kimi"
Requires-Dist: blobfile; extra == "kimi"
Dynamic: license-file

# TurboQuant-MLX

Extreme **weight** and **KV cache** compression for LLMs on Apple Silicon. MLX implementation of Google's [TurboQuant](https://arxiv.org/abs/2504.19874) (Zandieh et al., 2025) — Hadamard rotation + Lloyd-Max codebooks applied both to weights (compile time) and the KV cache (run time).

Supports dense models (LLaMA, Qwen, Mistral), **Mixture-of-Experts** (Qwen-MoE, GPT-OSS, Qwen3.5-MoE, Qwen3.6-35B-A3B, Qwen3-235B-A22B, DeepSeek-V2/V3), **Mamba/attention hybrids** (Nemotron-3-Nano-4B, Nemotron-3-Super-120B), and **multimodal** models (Muse Glimmer, Qwen3.8-27B). Compatible with hybrid attention architectures, attention sinks, sliding-window attention, and linear attention layers (including Gated DeltaNet).

**With both weight and KV cache compression at 3-bit, GPT-OSS-120B fits its full 131K context window in 50 GB on a 64 GB MacBook — and KV cache compression actually makes generation *faster* on the 120B (8.7 vs 6.4 tok/s) because the smaller cache cuts memory bandwidth more than dequant costs.**

**Expert streaming (v0.4.0)** runs MoE models whose weights exceed available RAM by paging only the router-selected experts from disk per token — e.g. the 35B-parameter Qwen3.6-35B-A3B runs on a **16 GB Mac mini** in under 4 GB of RAM, with output bit-identical to the fully-resident model. See [Qwen3.6-35B-A3B on a 16 GB Mac mini](#qwen36-35b-a3b-on-a-16-gb-mac-mini-expert-streaming).

**Local coding** — [Qwen3.6-27B](https://huggingface.co/manjunathshiva/Qwen3.6-27B-tq3-g32), a dense SWE-bench-grade coder, runs **fully resident on a 48 GB Mac** at 3-bit (~13 GB on disk, ~17.5 GB at runtime) and serves to Cursor / VS Code over an OpenAI-compatible endpoint. See [Qwen3.6-27B](#qwen36-27b-dense-coding-model-for-a-48-gb-mac).

## Key Results — Weight Compression

| Model | Method | Bits | PPL | Size | Gen Speed (M4 Max) |
|-------|--------|------|-----|------|---------------------|
| Qwen2.5-7B | TurboQuant | 3 | 8.92 | 3.5 GB | — |
| Qwen2.5-7B | Affine | 3 | 13.37 | 3.3 GB | — |
| GPT-OSS-20B | Affine (mlx-lm) | 4 | — | 11.2 GB | 148 tok/s |
| GPT-OSS-20B | MXFP4 (original) | 4 | 83.04 | 12.8 GB | — |
| GPT-OSS-20B | TurboQuant | 4 | 72.63 | 11.2 GB | — |
| GPT-OSS-20B | TurboQuant | 3 | 78.60 | 9.3 GB | **73 tok/s** |
| GPT-OSS-120B | [Affine 4-bit (mlx-community)](https://huggingface.co/mlx-community/gpt-oss-120b-4bit) | 4 | — | 65.8 GB | *Doesn't fit 64GB* |
| GPT-OSS-120B | MXFP4 (original) | 4 | — | 63.5 GB | *Doesn't fit 64GB* |
| GPT-OSS-120B | TurboQuant | 3 | — | 48 GB | **44 tok/s** |
| **[GPT-OSS-120B (hybrid for 48GB)](https://huggingface.co/manjunathshiva/gpt-oss-120b-tq3a-tq2e-g32)** | **TQ 3-attn / 2-experts, gs=32** | **2/3 mix** | **—** | **~35 GB** | **42–50 tok/s** |
| GPT-OSS-120B | TurboQuant | 2 | — | 32 GB | 51 tok/s (poor quality) |
| Qwen3.5-122B-A10B | BF16 (original) | 16 | — | ~240 GB | *Doesn't fit 64GB* |
| **Qwen3.5-122B-A10B** | **TurboQuant** | **3** | **—** | **~50 GB** | **26.5 tok/s (64 GB) · streams on a 16 GB Mac mini** |
| **[Qwen3.6-35B-A3B](https://huggingface.co/manjunathshiva/Qwen3.6-35B-A3B-tq3-g32)** | **TurboQuant, gs=32** | **3** | **—** | **~16 GB** | **~60 tok/s (resident) · runs in <4 GB via streaming** |
| **[Qwen3.6-27B (dense coder)](https://huggingface.co/manjunathshiva/Qwen3.6-27B-tq3-g32)** | **TurboQuant, gs=32** | **3** | **—** | **~13 GB** | **~14 tok/s (resident) · fits 48 GB, SWE-bench coder** |
| **[Qwen3-235B-A22B-Instruct-2507 (hybrid)](https://huggingface.co/manjunathshiva/Qwen3-235B-A22B-Instruct-2507-tq3a-tq2e-g32)** | **TQ 3-attn / 2-experts, gs=32** | **2/3 mix** | **—** | **70.5 GB** | **~4–6 tok/s (64 GB, 40 GB cache) · converts + streams on a 16 GB Mac mini** |
| **[Qwen3-235B-A22B-Instruct-2507 (full 3-bit)](https://huggingface.co/manjunathshiva/Qwen3-235B-A22B-Instruct-2507-tq3-g32)** | **TurboQuant, gs=32** | **3** | **—** | **103 GB** | **~1.3 tok/s (64 GB, 40 GB cache) · recall-critical sibling, passes 6/6 stress** |
| **Qwen3-235B-A22B-Instruct-2507 (ternary experts, 1.58-bit)** | **TQ 3-attn / ternary trit-packed experts, gs=64** | **1.6/3 mix** | **—** | **53 GB** | **5.6 tok/s (64 GB) · fully resident, no streaming** |
| **Nemotron-3-Nano-4B** | **TurboQuant** | **3** | **—** | **~2.2 GB** | **75.6 tok/s** |
| Nemotron-3-Super-120B-A12B | BF16 (original) | 16 | — | ~240 GB | *Doesn't fit 64GB* |
| **Nemotron-3-Super-120B-A12B** | **TurboQuant** | **3** | **—** | **~50 GB** | **18.7 tok/s** |
| **[Nemotron-3-Super-120B-A12B (hybrid for 48GB)](https://huggingface.co/manjunathshiva/Nemotron-3-Super-120B-A12B-tq3a-tq2e-g32)** | **TQ 3-attn / 2-experts, gs=32** | **2/3 mix** | **—** | **~36 GB** | **~27.2 tok/s** |
| **[Laguna-XS.2 (33B agentic coder)](https://huggingface.co/manjunathshiva/Laguna-XS.2-tq3-g64)** | **TurboQuant, gs=64** | **3** | **—** | **13.8 GB** | **46.8 tok/s · 32 GB Mac · Opencode pass** |
| Laguna-S-2.1 (118B) | BF16 (original) | 16 | — | ~219 GB | *Doesn't fit 64GB* |
| Laguna-S-2.1 (118B) | [Affine 4-bit](https://huggingface.co/mlx-community/Laguna-S-2.1-4bit) | 4 | — | ~64 GB | *Exceeds Metal's ~56 GB working set — won't load* |
| **[Laguna-S-2.1 (118B, ternary experts)](https://huggingface.co/manjunathshiva/Laguna-S-2.1-tqTe-g64)** | **TQ 3-attn / ternary trit-packed experts, gs=64** | **1.6/3 mix** | **—** | **~27 GB** | **~12.5 tok/s · fully resident on 64 GB with ~25 GB spare · Opencode pass** |
| **Kimi K3 (2.8T MoE)** | **TQ 3-attn / ternary experts / 4-bit expert-down, gs=64** | **1.6/3/4 mix** | **—** | **931 GB** | **2.3 tok/s (Mac Studio 512 GB, top-8) · 1.2 tok/s at native top-16 · streams experts from disk** |
| Muse Glimmer (30B dense VLM) | [Affine 4-bit](https://huggingface.co/mlx-community/Muse-Glimmer-30B-4bit) | 4 | 4.3798 | 19.88 GiB | 26.1 tok/s · leaves embedding + vision tower at bf16 |
| **Muse Glimmer (30B dense VLM)** | **TurboQuant, gs=64** | **4** | **4.3315** | **14.88 GiB** | **9.4 tok/s · better PPL than affine 4-bit in 25% less space** |
| **Muse Glimmer (30B dense VLM)** | **TurboQuant, gs=64** | **3** | **5.0454** | **12.53 GiB** | **10.9 tok/s · smallest coherent build** |
| Qwen3.8-27B (dense 27.8B hybrid-GDN VLM) | BF16 (original) | 16 | — | 55.6 GB | *Needs a 64 GB Mac just to load* |
| Qwen3.8-27B | [Affine 4-bit (mlx-community)](https://huggingface.co/mlx-community/Qwen3.8-27B-4bit) | 4 | 7.2685 | 14.95 GiB | **29.7 tok/s** — fastest 4-bit, but peaks 16.16 GiB: *won't fit 16 GB* |
| **[Qwen3.8-27B](https://huggingface.co/manjunathshiva/Qwen3.8-27B-tq4-g64)** | **TurboQuant, gs=64** | **4** | **7.2496** | **15.15 GiB** | **11.3 tok/s · best PPL of the three · Opencode agentic pass · vision 4/4** |
| **[Qwen3.8-27B (16 GB build)](https://huggingface.co/manjunathshiva/Qwen3.8-27B-tq3-mini-g64)** | **TurboQuant, gs=64** | **3** | **8.0052** | **11.55 GiB** | **12–13.6 tok/s (M4 Max) · verified on a real 16 GB Mac mini: 13.61 GiB peak, 3.7 tok/s, vision passes** |

## Key Results — KV Cache Compression

| Model | KV cache config | KV size | Speed | Notes |
|-------|----------------|---------|-------|-------|
| GPT-OSS-20B (FP16 weights) | FP16 KV | 27.0 MB | 90.6 tok/s | baseline |
| GPT-OSS-20B (FP16 weights) | TQ 3-bit KV | 7.79 MB | 29.9 tok/s | **3.5x cache savings** |
| GPT-OSS-120B (TQ 3-bit weights) | FP16 KV | 45.0 MB | 6.4 tok/s | baseline |
| **GPT-OSS-120B (TQ 3-bit weights)** | **TQ 3-bit KV** | **11.83 MB** | **8.7 tok/s** | **3.8x cache savings — and *faster* than FP16** |
| GPT-OSS-120B (TQ 3-bit weights) | TQ 4-bit KV | 12.21 MB | 16.0 tok/s | also clean |
| Qwen3.5-122B (TQ 3-bit weights) | FP16 KV | 161.06 MB | 5.4 tok/s | baseline |
| **Qwen3.5-122B (TQ 3-bit weights)** | **TQ 3-bit KV** | **150.17 MB** | **5.7 tok/s** | output identical to FP16 |

KV cache compression projects to ~7 GB RAM saved at 131K context on GPT-OSS-120B and ~5 GB at 262K on Qwen3.5-122B. Roundtrip cosine similarity vs FP16: 0.983 at 3-bit, 0.995 at 4-bit.

> **On the Qwen3.5-122B KV rows:** these were measured with **symmetric 3-bit KV** (`demo_kv.py --tq-bits 3`, short prompt), and the rates sit well below the model's fully-resident decode. With the Metal wired cap raised so the ~50 GB model stays fully resident, decode with the recommended **mixed K8/V3** cache is ~24–25 t/s — and there **fp16 edges out compression** at short-to-moderate context, the gap widening as context grows. On this model KV compression is a *memory* win, not a speed win; the genuine decode speed-up is **GPT-OSS-120B-specific**. See the resident long-context sweep in [#19](https://github.com/manjunathshiva/turboquant-mlx/pull/19).

## Key Results — Apple M5 Pro (48 GB, Metal4)

First Metal4 / `MTLGPUFamilyApple10` data point, contributed by [@sbayer2](https://github.com/sbayer2) (#14, #16) — a new 48 GB tier between the 16 GB Mac mini and the 64 GB M4 Max. Reproduce with [`benchmarks/bench_m5_pro.py`](benchmarks/bench_m5_pro.py).

| Model | Mode | Gen t/s | Peak Memory | Notes |
|-------|------|---------|-------------|-------|
| [Qwen3.6-35B-A3B](https://huggingface.co/manjunathshiva/Qwen3.6-35B-A3B-tq3-g32) tq3-g32 | resident | **52.0** | 18.1 GB | fp16 KV |
| [Nemotron-3-Super-120B-A12B](https://huggingface.co/manjunathshiva/Nemotron-3-Super-120B-A12B-tq3a-tq2e-g32) tq3a/tq2e-g32 | resident | **20.2** | 41.1 GB | needs `sudo sysctl iogpu.wired_limit_mb=49152` |
| [Qwen3.5-122B-A10B](https://huggingface.co/manjunathshiva/qwen3.5-122b-tq3) tq3 | streaming (30 GB cache) | **9.4** (7.3 e2e) | 34.2 GB | 89.9% expert hit-rate |

**KV cache sweep on the 35B (resident, 3-run averages):**

| KV config | Prompt t/s | Gen t/s | Peak |
|-----------|-----------|---------|------|
| fp16 baseline | 47.8 | **52.0** | 18.131 GB |
| K8 / V3 | 75.5 | 45.7 | 18.123 GB |
| K8 / V3 + sink128 | **76.1** | 45.7 | 18.124 GB |
| K3 / V3 | 75.6 | 45.2 | 18.122 GB |

**122B expert streaming — parallel prefetch** (`--cache-budget-gb 30`, 256 tokens):

| `--prefetch-workers` | Gen t/s | E2E t/s | Disk read | Hit rate |
|----------------------|---------|---------|-----------|----------|
| 1 (serial) | 7.4 | 5.7 | 44.3 GB | 89.5% |
| **8 (parallel)** | **9.1** | **7.6** | 41.9 GB | 90.1% |
| **Speedup** | **1.23×** | **1.33×** | | |

A 1.23× decode speedup from `--prefetch-workers 8`, landing between the Mac mini (1.3×) and the M4 Max (1.67×). The M5 Pro MacBook Pro SSD is the limiter — parallel prefetch helps but doesn't saturate the way the M4 Max's higher-bandwidth SSD does.

**122B expert streaming — cache-budget sweep** (`--prefetch-workers 8`, 256 tokens):

| Budget | Hit rate | Gen t/s | E2E t/s | Peak Metal | Disk read |
|--------|----------|---------|---------|------------|-----------|
| 20 GB | 80.9% | 7.1 | 6.1 | 24.2 GB | 80.7 GB |
| **30 GB** | **90.3%** | **9.1** | **7.6** | 34.2 GB | 40.9 GB |
| 38 GB | 91.0% | 8.9 | 7.5 | 42.2 GB | 38.0 GB |

The hit-rate curve flattens hard past 30 GB (only +0.7% for +8 GB), and throughput actually dips at 38 GB as peak Metal (42.2 GB) crowds the wired cap. **30 GB is the sweet spot on the 48 GB tier** — 90%+ hit rate with ~14 GB of headroom for OS stability; pushing to 38 GB gives negligible gain while peaking uncomfortably close to the wired limit.

KV compression gives a consistent **~1.6× prompt-processing speedup** for a ~12% decode cost (long-context decode behavior is in [The speed flip](#the-speed-flip)). Expert-streaming hit-rate scales with the cache budget — **44.6% at 4 GB (16 GB mini) → 89.9% at 30 GB (48 GB)**, a ~7× throughput jump that fills the gap between the 16 GB and 64 GB tiers.

> **Stability near the memory ceiling:** long-context *prompt prefill* close to the wired cap can starve the kernel watchdog (a `watchdogd` / `AppleARMWatchdogTimer` panic). A rapidly-growing KV cache makes Metal commit pages continuously (`AGXG17XFamilyResidencySet _commitAddedAllocations`), so the binding limit is allocation *rate*, not peak — a static 41 GB resident model (Nemotron) is stable, while a growing ~30 GB KV during a 63K-token prefill can panic. Practical limits on the 48 GB tier (#14): contexts up to ~14.5K are safe on the 35B with either KV config; 63K is feasible with K8/V3 (28.7 GB peak) but not fp16 (30.8 GB → panic). Keep headroom and close other apps for long-context runs near the cap.

## Install

```bash
pip install turboquant-mlx-full
```

The package is published as `turboquant-mlx-full` on PyPI, but importable as
`turboquant_mlx` (without the `-full` suffix) — this matches the original
project name and the examples in the Medium articles.

```python
import turboquant_mlx
from turboquant_mlx.layers import TurboQuantKVCache, convert_cache_to_turboquant
```

### Requirements

- macOS with Apple Silicon (M1/M2/M3/M4)
- Python 3.10+
- 64 GB unified memory recommended for 20B+ models

The Metal kernels are JIT-compiled by MLX at first use, so no Xcode / CMake
toolchain is required to install the package.

### Install from source (for development)

```bash
git clone https://github.com/manjunathshiva/turboquant-mlx.git
cd turboquant-mlx
pip install -e .
```

Optional extras:

```bash
pip install "turboquant-mlx-full[eval]"   # perplexity benchmarking (datasets, transformers)
pip install "turboquant-mlx-full[vlm]"    # multimodal / diffusion models via mlx-vlm
pip install "turboquant-mlx-full[kimi]"   # Kimi K3's tiktoken-based tokenizer
```

> **Muse Glimmer needs mlx-vlm >= 0.6.12**, the first release carrying its MLX
> model classes ([Blaizzy/mlx-vlm#1838](https://github.com/Blaizzy/mlx-vlm/pull/1838),
> merged 2026-08-10). `[vlm]` pins that floor, so the one-liner above is all you
> need — the git pin previously documented here is obsolete.

## Quick Start

### 0. Will it run on my Mac? (`turboquant-plan`)

Before a multi-GB download, ask:

```bash
turboquant-plan --model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64
```

```
Projection at 16,384 tokens of context
  weights              12.59 GB
  KV cache             0.34 GB  (20.0 KB/token, hybrid: 10/40 full-attention layers)
  prefill workspace    1.07 GB  (estimate, at --prefill-step-size 2048)
  runtime reserve      1.00 GB  (buffer cache, activations, fragmentation)
                       ----------------------------------
  peak                 15.00 GB of 61.85 GB usable   46.85 GB headroom

Verdict: ✅ RESIDENT — fits fully in memory
```

It reads **only safetensors headers** — no tensors are loaded, no engine starts.
A HuggingFace repo id is planned **over the network**, from range requests against
the shard headers: the 12.6 GB repo above answers in ~2.5 s and pulls **232 KB**
into a cold cache, so "will it fit?" is answered *before* the download, not after.
Weights are
exact; KV is derived from the config's attention geometry (hybrid models only
grow KV on their full-attention layers); the prefill workspace is an estimate
that scales with *both* chunk size and context.

Planning for a machine you're not sitting at, and asking for the flags:

```bash
turboquant-plan --model <repo> --wired-gb 10.5 --ram-gb 16 --context 21000 --kv-bits 8
```

```
Verdict: ⚠️  RESIDENT — fits, but only after raising the Metal wired cap

Recommended:
  sudo sysctl -w iogpu.wired_limit_mb=13721   (raises the 10.50 GB Metal cap —
      the binding limit here, not your 16 GB of RAM; resets on reboot)
  --prefill-step-size 128   (the default 2048 needs 1.38 GB of transient
      workspace at this context)
  --kv-bits 8
```

That is the 16 GB Mac mini recipe, *derived* rather than found by trial and error
— the projection is calibrated against measurements on that machine (it predicts
10.44 GB where the 9.4 GB ternary build measures 10.42, and recommends a wired
limit within 1% of the one that actually works).

### A note on reproducibility and `--prefill-step-size`

Changing the prefill chunk size can change a greedy answer by a token — measured
on a ~3.9K-token prompt, `--prefill-step-size 2048` produced "...which *explains
why* the sky reads blue" where 512/256/128 produced "...which *is why* the sky
reads blue". Both are valid argmax continuations; the chunk size only changes the
order the GPU reduces in, and fp16 rounding occasionally lands on the other side
of a near-tie.

**This is a property of chunked prefill on Metal, not of TurboQuant.** The same
test on stock `mlx-community/Llama-3.2-1B-Instruct-4bit` — plain mlx-lm, plain
affine 4-bit, none of our kernels — forks the same way. Our own decode and
prefill MoE kernels are *bit-identical* to each other
(`tests/test_kernel_determinism.py`), which is what keeps `--disk-cache`
checkpoint restores consistent.

Practical upshot: for byte-reproducible greedy output, hold `--prefill-step-size`
fixed across runs. Everything else — quality, correctness, coherence — is
unaffected.

`turboquant-doctor` runs the same projection plus a read-only readiness check —
model files, tokenizer, quantization block, mlx/mlx-lm imports, machine limits —
with stable check ids and exit codes (0 ok/warn, 1 won't fit or files missing,
2 bad usage). Both take `--json` for automation.

### 1. Convert a model to TurboQuant format

```bash
# Dense model (e.g., LLaMA 3.2 1B at 3-bit)
python -m turboquant_mlx.convert \
    --hf-path meta-llama/Llama-3.2-1B \
    --mlx-path ./llama-3.2-1b-tq3 \
    --bits 3 --group-size 64

# MoE model (e.g., GPT-OSS-20B at 2-bit)
python -m turboquant_mlx.convert \
    --hf-path openai/gpt-oss-20b \
    --mlx-path ./gpt-oss-20b-tq2 \
    --bits 2 --group-size 64

# Very large model whose quantized form won't fit in RAM (200B+): --streaming
# writes each layer to a shard and frees it, so peak memory stays ~one shard
# (5 GB) + one layer — letting 235B/671B-class MoEs convert on a 64 GB Mac.
python -m turboquant_mlx.convert \
    --hf-path Qwen/Qwen3-235B-A22B-Instruct-2507 \
    --mlx-path ./qwen3-235b-tq3 \
    --bits 3 --group-size 64 --streaming

# Ternary (1.58-bit) experts — sub-2-bit MoE experts on the data-free {-c, 0, +c}
# codebook, packed as genuine base-3 trits (20 per uint32, ~1.6 bpw vs 2.0 for the
# 2-bit slot). Attention + lm_head stay at --bits; only the routed experts go
# ternary. On a 128-expert model this shrinks Qwen3-235B to 53 GB (from 70.5 GB),
# so it runs FULLY RESIDENT on a 64 GB Mac at ~5.6 tok/s (vs ~2 tok/s streaming).
# Needs enough expert redundancy — great on 128 experts, breaks below ~64.
python -m turboquant_mlx.convert \
    --hf-path Qwen/Qwen3-235B-A22B-Instruct-2507 \
    --mlx-path ./qwen3-235b-tq3a-tqTe-g64 \
    --bits 3 --group-size 64 --ternary-experts --streaming
# Resident load of a ~53 GB model needs a wired-memory bump on a 64 GB Mac:
#   sudo sysctl -w iogpu.wired_limit_mb=60416

# Asymmetric experts — ternary up/gate + higher-bit down_proj. The down
# projection is the SwiGLU write-back into the residual stream, and spending
# 4 bits on just that one expert matrix is what turns a sub-2-bit build
# agent-capable: on Qwen3.6-35B-A3B, pure ternary fails an Opencode coding
# task 0/4 while ternary+down4 passes 3/3 — for +3.2 GB (9.4 -> 12.6 GB),
# still resident on a 16 GB Mac mini.
python -m turboquant_mlx.convert \
    --hf-path Qwen/Qwen3.6-35B-A3B \
    --mlx-path ./Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64 \
    --bits 3 --group-size 64 --ternary-experts --expert-down-bits 4
```

### 2. Generate text

```bash
turboquant-generate \\
    --model ./gpt-oss-20b-tq2 \
    --prompt "Why is the sky blue? Explain in simple terms." \
    --max-tokens 200
```

### 3. Evaluate perplexity

```bash
python -m turboquant_mlx.evaluate \
    --hf-path openai/gpt-oss-20b \
    --bits 2 3 4 \
    --num-samples 256 --seq-len 512
```

### 4. Generate with KV cache compression

The production `turboquant-generate` CLI accepts KV-cache flags directly (v0.2+).
Use mixed K/V precision (`--kv-k-bits 8 --kv-v-bits 3`) — required for
TurboQuant-quantized weights, and lossless on stock fp16 weights:

```bash
# v0.2 recommended default: mixed K8/V3 + 128-token fp16 sink
turboquant-generate \
    --model ./gpt-oss-120b-tq3 \
    --prompt "Why is the sky blue?" \
    --max-tokens 1024 --temp 0.7 \
    --kv-k-bits 8 --kv-v-bits 3 --kv-min-tokens 128

# Symmetric (legacy) — only safe on fp16 weights
turboquant-generate \
    --model openai/gpt-oss-20b \
    --prompt "Why is the sky blue?" \
    --max-tokens 200 --kv-bits 3

# Side-by-side comparison harness (4 configs in one run)
python -m turboquant_mlx.benchmarks.demo_kv_v02 \
    --model ./gpt-oss-120b-tq3 \
    --prompt "Why is the sky blue?" \
    --max-tokens 1024 --temp 0.7 --top-p 0.9 --repetition-penalty 1.1
```

### 5. Serve a TurboQuant model over an OpenAI-compatible API

`turboquant-serve` wraps `mlx_lm.server` and patches its loader so any
TurboQuant model (`quantization.mode = "turboquant"` in `config.json`)
loads through the PolarQuant path. Non-TurboQuant models pass through
unchanged, so this is a drop-in replacement for `mlx_lm.server`.

```bash
# Serve a local TQ model
turboquant-serve \
    --model ./NVIDIA-Nemotron-3-Super-120B-A12B-BF16-tq3 \
    --port 8080

# Or serve directly from the Hugging Face Hub
turboquant-serve \
    --model manjunathshiva/Nemotron-3-Super-120B-A12B-tq3 \
    --port 8080
```

Then call it like any OpenAI-compatible endpoint. The `model` field in
the request must match the string passed to `--model`:

```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "./NVIDIA-Nemotron-3-Super-120B-A12B-BF16-tq3",
    "messages": [{"role": "user", "content": "Why is the sky blue?"}],
    "max_tokens": 4096,
    "temperature": 0.7
  }'
```

For Nemotron-3 reasoning models, prefer `max_tokens >= 2048` so the
`<think>` trace and the final answer both fit. mlx-lm splits them into
`message.reasoning` (the thinking) and `message.content` (the answer).

From Python via the OpenAI SDK:

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model="./NVIDIA-Nemotron-3-Super-120B-A12B-BF16-tq3",
    messages=[{"role": "user", "content": "Why is the sky blue?"}],
    max_tokens=4096,
    temperature=0.7,
    stream=True,
)
for chunk in resp:
    print(chunk.choices[0].delta.content or "", end="", flush=True)
```

All `mlx_lm.server` flags forward unchanged — see `turboquant-serve --help`
for `--host`, `--temp`, `--top-p`, `--prompt-cache-size`, etc.

#### Serve a multimodal model (`turboquant-serve-vlm`)

`turboquant-serve` wraps `mlx_lm.server`, which knows nothing about multimodal
architectures. For a VLM — Muse Glimmer, DiffusionGemma — use
**`turboquant-serve-vlm`**, which drives *mlx-vlm's* server instead (OpenAI
**and** Anthropic routes, per-model tool parsers, continuous batching):

```bash
turboquant-serve-vlm \
    --model manjunathshiva/Muse-Glimmer-30B-tq4-g64 \
    --reasoning-strength low --port 8080
```

All `mlx_vlm.server` flags forward unchanged. Two things are handled for you:

- **The reasoning channel is split out of `content`.** Muse Glimmer answers in a
  harmony-style channel format, and mlx-vlm's ATEM parser only strips that
  envelope when a tool call was parsed — so an ordinary turn would hand your
  agent the model's private deliberation as its reply. The right delimiters are
  supplied per architecture; the thinking lands in `reasoning_content` where it
  belongs. An explicit `--thinking-start-token`/`--thinking-end-token` still wins.
- **`--reasoning-strength` actually reaches the template.** OpenAI clients send
  `reasoning_effort`, but Muse Glimmer's template reads `reasoning_strength` and
  otherwise deliberates at its `high` default. Requests are translated, and this
  flag sets the default for clients that ask for nothing — which is most agent
  harnesses. Measured on `tq4-g64`: a tool-result turn cost **54 completion
  tokens at `low` versus 106 at the default `high`**, for the same answer.

#### Serve a model bigger than RAM (expert streaming)

Passing `--cache-budget-gb` routes the loader through the streaming path, so a
MoE whose weights exceed RAM can be **served** over the OpenAI API — only the
router-selected experts are paged from disk per token. This is how you put a
~50 GB **122B on a 16 GB Mac mini** behind Claude Code / Aider:

```bash
turboquant-serve \
    --model manjunathshiva/qwen3.5-122b-tq3 \
    --cache-budget-gb 4 \
    --kv-k-bits 8 --kv-v-bits 3 --kv-min-tokens 128 \
    --prompt-concurrency 1 --port 8080
```

The [Flash-MoE streaming levers](#tuning-the-streaming-reader-v061) ride along:
`--max-active-experts` (K-reduction, default `4` → ~2× less disk I/O) and
`--use-page-cache` / `--no-page-cache` (auto by model-size-vs-RAM — trust-OS is
~2.4× faster decode when the model fits free RAM, `F_NOCACHE` otherwise). Pair
with `--kv-*` to compress the growing KV cache of long agentic loops, and
`--prompt-concurrency 1` since streaming is a single-user path.

`--cache-budget-gb auto` sizes the expert cache from the machine (80% of
Metal's recommended working set, minus resident weights and a KV reserve),
and a `hot_experts.json` shipped in the model directory (the pin output of
`calibrate_experts.py`) is auto-discovered: those experts are pinned **and
preloaded** at startup — measured on the 35B ternary, a 1.1 GB / 1,500-expert
profile preloads in 0.3 s and lifts the cache hit rate 74 → 82%, cutting
critical-path disk reads 30%. `--no-hotlist` opts out; `--wire-memory`
(opt-in) wires weights + expert cache against memory pressure.

That 35B lift is the roomy case (a large cache budget relative to the experts).
The gain shrinks when the cache is a small fraction of a much larger expert set —
on a genuinely disk-bound 122B it falls to ~4% fewer disk reads and **no speed
change**; see the measured note under [Tuning the streaming reader](#tuning-the-streaming-reader-v061).
Pinning helps hit-rate and disk, not throughput; disk bandwidth is the wall.

> **Note**: `mlx_lm.server` is intended for development and local use, not
> production. It does not implement authentication or rate limiting.

#### Memory tuning when serving near the unified-memory ceiling

Serving a 50 GB model on a 64 GB Mac (or any TQ model that fills most of
RAM on a 48 GB / 96 GB Mac) leaves very little headroom for Metal command
buffers and accumulating prompt caches. After 3-4 multi-turn requests the
server can crash with:

```
libc++abi: terminating due to uncaught exception of type std::runtime_error:
[METAL] Command buffer execution failed: Insufficient Memory
(00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
```

`mlx_lm.server` keeps a **persistent prompt cache per role/conversation**
to speed up follow-up turns. Each new prompt grows that pool, and once
caches + model weights + decode workspace exceed Metal's wired-memory
budget, the next allocation aborts the process.

Since 0.13, `turboquant-serve` applies two auto-guards on tight machines
(both no-ops when working-set headroom is ≥ 8 GB):

- **`--metal-cache-limit-gb auto`** (default) caps MLX's GPU buffer reuse
  cache after the model loads. During a long chunked prefill the attention
  workspace has a new, larger shape every chunk, so the unbounded cache
  grows ~80 MB per 1K prompt tokens — measured on a 16 GB mini serving a
  resident 12.6 GB 35B, that alone OOM-crashed a 21K-token prefill at 14K
  while *active* memory stayed flat. With the cap the same prefill runs
  flat end to end.
- **`--prompt-cache-max-gb auto`** (default) puts a **byte** budget on the
  LRU prompt cache, enforced on every insert. Upstream's
  `--prompt-cache-bytes` exists but is only applied on the *batched*
  serving path — TurboQuant KV-quant serving is sequential, where an agent
  conversation retained 8 KV states (1.14 GB) on the mini, paged the
  system, and got the server killed by the GPU watchdog. Evictions are
  cheap when `--disk-cache` is on (states restore from disk).

On 16 GB machines also pass **`--prefill-step-size 256`** (or 128 near the
context ceiling): the *per-chunk* prefill workspace scales with chunk size,
and mlx-lm's default of 2048 OOMs tight boxes on the first chunk.

For bigger machines serving near the ceiling, the two manual levers:

**1. Raise Metal's wired-memory ceiling** (biggest lever, requires sudo,
resets on reboot):

```bash
# 64 GB Mac → leave ~7 GB for macOS
sudo sysctl iogpu.wired_limit_mb=57344

# 48 GB Mac → leave ~5 GB for macOS
sudo sysctl iogpu.wired_limit_mb=43008
```

To make it permanent, append `iogpu.wired_limit_mb=57344` to
`/etc/sysctl.conf`.

**2. Cap the prompt cache** (works without sudo, evicts oldest cached
prompts to stay under the cap):

```bash
turboquant-serve \
    --model ./NVIDIA-Nemotron-3-Super-120B-A12B-BF16-tq3 \
    --port 8080 \
    --prompt-cache-bytes 2147483648    # 2 GB hard cap
```

Tighter caps (`536870912` = 512 MB, or `--prompt-cache-size 1` to keep
only one sequence) trade follow-up prefix-cache speedup for stability.

Recommended combo for **Nemotron-3-Super-120B-A12B-tq3 on a 64 GB Mac**:

```bash
sudo sysctl iogpu.wired_limit_mb=57344
turboquant-serve \
    --model ./NVIDIA-Nemotron-3-Super-120B-A12B-BF16-tq3 \
    --port 8080 \
    --prompt-cache-bytes 2147483648
```

Also close any other GPU users (Chrome/Electron apps, Final Cut, Xcode
simulators) before launching — even an idle Chrome can be holding 1-2 GB
of unified memory.

#### Compress the KV cache while serving

`--prompt-cache-bytes` caps how much the *reuse pool* across requests can
grow, but each in-flight request still holds an **fp16** KV cache that
scales with context length. On a memory-constrained box — e.g. a streaming
120B on 16 GB driven by an agentic loop (Aider, Claude Code) whose prompts
grow every turn — that per-request cache is usually what runs you out of
memory first. `turboquant-serve` adds the same KV-quant flags as
`turboquant-generate` to shrink it ~4x in place:

```bash
turboquant-serve \
    --model manjunathshiva/Nemotron-3-Super-120B-A12B-tq3 \
    --port 8080 \
    --kv-k-bits 8 --kv-v-bits 3 \   # mixed-precision KV (recommended default)
    --kv-min-tokens 128 \           # keep first 128 tokens fp16 (attention sinks)
    --prompt-concurrency 1
```

| Flag | Meaning |
| --- | --- |
| `--kv-k-bits N` / `--kv-v-bits M` | Mixed-precision KV; K8/V3 is the recommended default |
| `--kv-bits N` | Symmetric K=V=N (legacy; not recommended below 3) |
| `--kv-min-tokens N` | Keep first N cached tokens in fp16 (sink protection) |
| `--kv-group-size G` | Hadamard rotation group size (default 64) |
| `--kv-fused` | Use the fused decode+attend Metal kernel (experimental; needs `--kv-min-tokens 0`; see [Fused decode+attend](#fused-decodeattend-kernel-experimental)) |

These are processed by `turboquant-serve` and stripped before the remaining
flags forward to `mlx_lm.server`. See [KV Cache Compression](#kv-cache-compression)
below for how to choose bit-widths and the speed/quality trade-offs.

> **Single-stream when KV-quant is on.** TurboQuant KV caches don't support
> the cross-request `merge` that `mlx_lm.server`'s batch generator needs, so
> enabling any `--kv-*` flag makes the server serve requests sequentially.
> That's the right trade-off for a single-user setup; for a multi-client
> server it means concurrent requests queue rather than batch.

#### Persistent prompt cache across restarts (`--disk-cache`)

`--disk-cache [DIR]` makes the prompt-prefix cache a disk citizen: KV-cache
checkpoints are written per conversation (background writer, LRU byte-budget
eviction, default 10 GB) and any later request — including the first one
after a server **restart or crash** — resumes from the longest saved token
prefix instead of re-prefilling from scratch. Measured: a 16.3K-token
conversation's first turn after restart drops **21.1 s → 3.0 s** on a 64 GB
Mac (6.9×, output byte-identical), and on a 16 GB mini a crashed 21K-token
prefill resumed from token 18,432 (249 MB checkpoint loaded in 2.3 s).

Checkpoints are also taken **mid-prefill** every `--disk-cache-save-every`
tokens (default 1024). That ladder is what makes persistence work for
hybrid GDN/Mamba models whose recurrent state can't be trimmed: chat
templates re-render the assistant turn differently in history (e.g. Qwen's
empty `<think>` block), so the end-of-request checkpoint is never an exact
prefix of the next turn — the ladder restores from the newest checkpoint
below the divergence instead. Measured on the mini: turn 2 of a 21K
conversation reused **20,480 of 21,250 tokens** on a non-trimmable cache
that previously got zero.

#### Agent harnesses on low-bit builds (`--tool-syntax-greedy`)

Agent harnesses need temperature (greedy decoding perseverates across turns
on low-bit builds), but sampled *structure* is where those builds fabricate
tool calls. `--tool-syntax-greedy` masks logits to argmax only while the
generation is inside a `<tool_call>`...`</tool_call>` block — braces, keys,
tags — and leaves the sampler in charge of JSON value strings and of the
decision to call a tool at all. Tags are configurable via
`--tool-syntax-tags "<open>,</close>"`.

#### Recipe: a coding agent on a 16 GB Mac mini

The asymmetric-expert build
[Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64](https://huggingface.co/manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64)
(ternary up/gate + 4-bit down_proj, 12.6 GB) is the smallest 35B that
survives an agent loop — validated end-to-end on a base M-series 16 GB
Mac mini:

```bash
sudo sysctl -w iogpu.wired_limit_mb=14336   # 14 GiB; 13824 is enough below ~16K context

turboquant-serve \
    --model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64 \
    --host 0.0.0.0 --port 8080 \
    --kv-bits 8 --tool-syntax-greedy --disk-cache \
    --prefill-step-size 128 \
    --temp 0.7 --top-p 0.8 --top-k 20 \
    --chat-template-args '{"enable_thinking": false}' \
    --prompt-concurrency 1
```

Measured on the mini (model on an external SSD): **15.2–15.6 tok/s** decode
at 13.6 GB peak (flat to 1800 generated tokens), 21K-token cold prefill
~234 s (~90 tok/s), and the Opencode fix-the-failing-test task passes
**3/3** (3:37 / 2:25 / 1:54 — faster each run as the disk cache warms; the
identical task fails 0/4 on the pure-ternary build). Practical context
ceiling is ~16–18K at the default wired cap; the 128-token prefill step +
14 GiB wired configuration above is what carries 21K+. Point Opencode (or
any OpenAI-compatible harness) at `http://<host>:8080/v1` with model id
`default_model`, keep temperature ~0.7, and disable auto-discovered skills.

---

## KV Cache Compression

TurboQuant KV cache compression applies the same Hadamard rotation + Lloyd-Max codebook pipeline to KV vectors at runtime. The compressed cache is dequantized to float16 only when attention needs it, so it routes through MLX's standard `scaled_dot_product_attention` and is compatible with attention sinks, sliding windows, and linear attention layers.

### Programmatic usage

```python
from turboquant_mlx.layers import convert_cache_to_turboquant
from mlx_lm.models.cache import make_prompt_cache

# 1. Build per-layer cache (correct types for hybrid models)
cache = make_prompt_cache(model)

# 2. Convert to TurboQuant KV cache (v0.2 mixed K/V + sink protection)
cache = convert_cache_to_turboquant(
    cache,
    k_bits=8, v_bits=3,           # K-precision-critical, V tolerates 3-bit
    min_tokens_before_quant=128,  # keep first 128 tokens fp16 (attention sinks)
    group_size=64,
)

# 3. Process the prompt and generate — cache is compressed from token 128+
model(prompt_tokens, cache=cache)
for token in generate_loop(model, cache):
    ...
```

> **v0.1 → v0.2 migration:** `tq_bits=3` still works (symmetric K=V=3) but is
> not recommended on TurboQuant-quantized weights. Pass `k_bits=8, v_bits=3`
> instead. Pre-existing checkpoints and code paths are fully backward compatible.

### Choosing a bit-width (v0.2)

K precision matters far more than V precision: softmax amplifies any K error,
while V tolerates aggressive quantization. Mixed K8/V3 is the new default.

| Weights | K bits | V bits | sink | When to use |
|---------|-------:|-------:|-----:|-------------|
| FP16 / BF16 | 8 | 3 | 128 | Default — lossless quality, ~4× smaller cache |
| FP16 / BF16 | 4 | 3 | 128 | More aggressive; small quality dip on dense attention |
| **TurboQuant-quantized** | **8** | **3** | **128** | **Required on tq3 weights — symmetric K3 collapses past ~1k generated tokens** |
| Any | 8 | 4 | 128 | Highest fidelity TQ KV setting |

**Why K8 specifically on TurboQuant weights:** stacking 3-bit K cache on top of
already-3-bit weight quantization compounds the noise enough to break long-form
generation on GPT-OSS-20B (we observed total output collapse past ~800 tokens
with `K3_V3` on tq3 weights, while `K8_V3` is clean). The same `K3_V3` cache is
fine on stock fp16 weights — the failure mode is co-compression, not the cache
alone.

### CLI flags

`turboquant-generate` exposes the same controls:

```bash
turboquant-generate --model ./model-tq3 --prompt "..." \
    --kv-k-bits 8 --kv-v-bits 3 \
    --kv-min-tokens 128 \
    --kv-group-size 64
```

| Flag | Purpose |
|------|---------|
| `--kv-bits N` | Symmetric K=V=N (legacy v0.1) |
| `--kv-k-bits` / `--kv-v-bits` | Mixed precision (v0.2 recommended) |
| `--kv-min-tokens N` | Keep the first N cached tokens in fp16 (sink protection) |
| `--kv-group-size N` | Hadamard rotation group size (default 64) |

### The speed flip

Whether KV compression speeds up or slows down decode depends on the **per-token KV cache size**, not the parameter count. When the per-token KV is large (many KV heads and/or long context), its 4x smaller footprint cuts memory bandwidth more than dequant adds, and decode is *faster* than FP16. When it is small (few active params, short context), dequant overhead dominates and compression is *slower* — a pure memory optimization.

| Model | FP16 KV | TQ 3-bit KV | Direction |
|-------|---------|-------------|-----------|
| GPT-OSS-20B | 90.6 tok/s | 29.9 tok/s | TQ is 3x **slower** |
| Qwen3.6-35B-A3B (3B active) | 52.0 tok/s | 45.7 tok/s | TQ is 1.1x **slower** |
| GPT-OSS-120B | 6.4 tok/s | 8.7 tok/s | TQ is 1.4x **faster** |

The penalty also **grows with context** on small-KV models. A community long-context benchmark on M5 Pro (#14) measured Qwen3.6-35B-A3B-tq3-g32 decode at four context lengths (256 gen tokens, except 63K which used 128):

| Context | KV Config | Prompt t/s | Decode t/s | Peak Metal | Memory Saved |
|---------|-----------|-----------|-----------|------------|--------------|
| ~65 tok | fp16 | 47.8 | 52.0 | 18.13 GB | — |
| ~65 tok | K8/V3 | 76.1 | 45.7 | 18.12 GB | 0.01 GB |
| ~2.5K tok | fp16 | 131.9 | 51.5 | 20.50 GB | — |
| ~2.5K tok | K8/V3 | 133.4 | 38.2 | 20.55 GB | -0.05 GB |
| ~14.5K tok | fp16 | 132.8 | 46.0 | 22.50 GB | — |
| ~14.5K tok | K8/V3 | 132.6 | 16.7 | 21.94 GB | 0.56 GB |
| ~63K tok | fp16 | 124.5 | 34.5 | 30.79 GB | — |
| ~63K tok | K8/V3 (no sink) | 123.6 | 5.2 | 28.71 GB | 2.08 GB |

The fp16 decode advantage *widens* with context — 1.14× at 65 tokens → 1.35× at 2.5K → 2.75× at 14.5K → **6.6× at 63K**. So on small-active MoEs, use KV compression to *fit* longer contexts in less RAM (2.08 GB saved at 63K) — not to speed them up. The flip to *faster* shows up only on **GPT-OSS-120B** (8.7 vs 6.4 t/s). The similarly-sized **Qwen3.5-122B does *not* flip** — run resident on a 64 GB M4 Max, fp16 KV beats mixed K8/V3 at every context (1.07× at 256 → 1.20× at 4096; [#19](https://github.com/manjunathshiva/turboquant-mlx/pull/19)) — so the speed-up isn't a general 120B-class property; it's specific to GPT-OSS-120B's KV geometry.

### Fused decode+attend kernel (experimental)

The "speed flip" penalty above is almost entirely the **dequantize** step: at
each decode token the compressed cache is expanded to fp16 before
`scaled_dot_product_attention` runs, and at long context that materialization —
not the attention math — is the wall. The fused kernel removes it.

`turboquant_mlx.kernels.kv_decode_attend` is a hand-written Metal kernel that
reads the **packed** cache directly and runs a FlashAttention-style
online-softmax pass, decoding K/V on the fly and never building the fp16
tensors. The inverse-Hadamard rotation is kept outside the kernel via an
orthonormal identity (rotate the query once, un-rotate the output once), so the
per-token rotation cost collapses too. It is numerically equivalent to the
dequantize+SDPA path (cosine 1.0 vs it, same fp16-rounding error as always).

Opt in per prompt-cache — it patches the model's SDPA seam and flips a flag on
each `TurboQuantKVCache`:

```python
from turboquant_mlx.layers import convert_cache_to_turboquant, enable_fused_attend
from mlx_lm.models.cache import make_prompt_cache

cache = convert_cache_to_turboquant(make_prompt_cache(model),
                                    k_bits=8, v_bits=3, group_size=64)
enable_fused_attend(cache)          # decode steps now use the fused kernel
```

Measured decode speedup **over the standard TQ-KV (dequant+SDPA) path**, greedy,
byte-identical output:

| Model | Context | TQ-KV shipped | TQ-KV fused | Speedup |
|-------|---------|--------------:|------------:|--------:|
| Llama-3.2-1B (D=64, full-attn) | 4.2K | 70.5 t/s | 105.2 t/s | **1.49×** |
| Qwen3.6-35B-A3B (hybrid, 10/40 full-attn) | 2.1K | 40.2 t/s | 43.9 t/s | 1.09× |

The isolated attention-layer speedup grows with context (≈10× over the shipped
dequant path at 32K in a per-layer microbenchmark); the end-to-end gain scales
with the full-attention-layer fraction × context length, so hybrid MoEs
(Qwen3.5's GatedDeltaNet layers are untouched) see less than dense models.

Or enable it while serving with `--kv-fused` (see the
[serve KV flags](#compress-the-kv-cache-while-serving)):

```bash
turboquant-serve --model ./model-tq3 --kv-k-bits 8 --kv-v-bits 3 \
    --kv-min-tokens 0 --kv-fused
```

**Scope / limits.** Decode only (`S_q=1`), batch 1, single-tier
(`min_tokens_before_quant=0`, no fp16 sink window), bit-packed K/V, and equal
K/V head dims that are a multiple of 32. Prefill and any non-applicable step
transparently fall back to the standard path. The kernel performs plain causal
attention, so on a layer that needs attention sinks or a sliding-window mask it
**gracefully falls back** to the standard dequantize+SDPA path for that step
(correct output, just without the fused speedup) — so it is safe to enable on
any model, including GPT-OSS. (Sink/sliding layers mostly use `RotatingKVCache`
and never convert to `TurboQuantKVCache`; the fallback covers the GPT-OSS
full-attention-with-sinks case.)

### Compatibility

| Feature | Supported | Notes |
|---------|-----------|-------|
| Attention sinks | Yes | GPT-OSS sink vectors flow through standard SDPA |
| Sliding window attention | Yes | `RotatingKVCache` layers are left untouched |
| Linear attention | Yes | `ArraysCache` (Qwen3.5 GatedDeltaNet) is left untouched |
| Hybrid architectures | Yes | Per-layer cache type is preserved |
| Prompt-first conversion | Yes | Process prompt with FP16, convert before generation |
| Fused decode+attend | Experimental | Opt-in (`enable_fused_attend` / serve `--kv-fused`); standard causal decode, graceful dequant fallback on sink/sliding layers (see above) |

---

## Running GPT-OSS MoE Models on Apple Silicon

### GPT-OSS-20B (21B total, 32 experts, 3.6B active)

**Hardware:** Apple M4 Max 64GB (or any Apple Silicon with 16GB+ unified memory at 3-bit)

#### Step 1: Convert to TurboQuant 3-bit (recommended)

```bash
python -m turboquant_mlx.convert \
    --hf-path openai/gpt-oss-20b \
    --mlx-path ./gpt-oss-20b-tq3 \
    --bits 3 --group-size 32
```

**Model size:** 9.3 GB (vs 12.8 GB MXFP4 original — 28% smaller, lower perplexity)

The converter automatically:
- Detects MoE architecture (SwitchLinear / QuantizedSwitchLinear layers)
- Dequantizes MXFP4 expert weights to float
- Applies Hadamard rotation + Lloyd-Max codebook quantization
- Keeps router weights and attention at full precision
- Handles blockwise Hadamard for 2880-dim experts (2880 = 9 x 320)

#### Step 2: Generate text

```bash
turboquant-generate \\
    --model ./gpt-oss-20b-tq3 \
    --prompt "Explain quantum entanglement to a 10-year-old." \
    --max-tokens 256
```

**Expected:** ~73 tok/s generation, ~85 tok/s prefill on M4 Max

#### Step 3: Run a quick quality check

```bash
python -m turboquant_mlx.evaluate \
    --hf-path openai/gpt-oss-20b \
    --bits 3 \
    --no-affine --no-qjl \
    --num-samples 64 --seq-len 512
```

#### All bit-widths for GPT-OSS-20B

| Method | Bits | Size | Peak RAM | Gen Speed | Quality |
|--------|------|------|----------|-----------|---------|
| Affine (mlx-lm) | 4 | 11.2 GB | ~14 GB | 148 tok/s | Coherent (but see note below) |
| TurboQuant | 4 | 11.2 GB | ~14 GB | — | Best (PPL 72.63, beats MXFP4) |
| **TurboQuant** | **3** | **9.3 GB** | **~12 GB** | **73 tok/s** | **Recommended (PPL 78.60, beats MXFP4, coherent)** |
| TurboQuant | 2 | 7.5 GB | ~10 GB | — | Poor (incoherent generation on pre-quantized models) |

> **Speed vs quality tradeoff:** Affine 4-bit is ~2x faster on the 20B model due to simpler dequantization, but TurboQuant 3-bit is 28% smaller with lower perplexity than both affine 4-bit and OpenAI's own MXFP4. Crucially, affine 4-bit **cannot scale to 120B** on 64GB hardware — TurboQuant 3-bit is the only option there.

```bash
# 4-bit (best quality, beats OpenAI's MXFP4)
python -m turboquant_mlx.convert \
    --hf-path openai/gpt-oss-20b \
    --mlx-path ./gpt-oss-20b-tq4 \
    --bits 4 --group-size 32
```

---

### GPT-OSS-120B (120B total, 128 experts, ~13B active)

**Hardware:** Apple M4 Max 64GB — neither the original MXFP4 (63.5 GB) nor the [mlx-community 4-bit affine](https://huggingface.co/mlx-community/gpt-oss-120b-4bit) (65.8 GB) fit on a 64GB machine. TurboQuant 3-bit is the only way to run this model on consumer hardware.

#### Step 1: Convert to TurboQuant 3-bit (recommended)

```bash
python -m turboquant_mlx.convert \
    --hf-path openai/gpt-oss-120b \
    --mlx-path ./gpt-oss-120b-tq3 \
    --bits 3 --group-size 64
```

**Model size:** 48 GB

> **Note:** The default converter materializes the full quantized model in RAM before saving, so peak memory ≈ the quantized model size (~50–55 GB for a 120B). On a 64 GB machine that caps conversion at ~130B params. For anything larger, add **`--streaming`**: it writes each quantized layer to a shard and frees it, keeping peak memory to ~one 5 GB shard plus the layer being processed — so 200B+ models (Qwen3-235B, DeepSeek-V3) convert on a 64 GB Mac. Output is byte-identical to the in-memory path.

#### Step 2: Generate text

```bash
turboquant-generate \\
    --model ./gpt-oss-120b-tq3 \
    --prompt "Explain quantum computing in simple terms." \
    --max-tokens 200
```

**Expected:** ~44 tok/s generation, ~9.5 tok/s prefill, 52 GB peak memory on M4 Max 64GB

#### Step 3: Quick quality check

```bash
python -m turboquant_mlx.evaluate \
    --hf-path openai/gpt-oss-120b \
    --bits 3 \
    --no-affine --no-qjl \
    --num-samples 32 --seq-len 512
```

#### All bit-widths for GPT-OSS-120B

| Method | Bits | Size | Peak RAM | Gen Speed | Fits 64 GB? | Quality |
|--------|------|------|----------|-----------|-------------|---------|
| [mlx-community 4-bit](https://huggingface.co/mlx-community/gpt-oss-120b-4bit) | 4 (affine) | 65.8 GB | — | — | **No** | — |
| MXFP4 (original) | 4 (mxfp) | 63.5 GB | ~70 GB | — | **No** | — |
| **TurboQuant** | **3** | **48 GB** | **52.3 GB** | **44 tok/s** | **Yes** | **Coherent, well-structured** |
| TurboQuant | 2 | 32 GB | 34.9 GB | 51 tok/s | Yes | Incoherent after ~20 tokens |

> Neither the original MXFP4 format (63.5 GB) nor the mlx-community affine 4-bit re-quantization (65.8 GB) fit on a 64GB Mac. TurboQuant 3-bit (48 GB) is the **only** way to run GPT-OSS-120B on consumer hardware — and at 44 tok/s, it's interactive speed. At 2-bit, the model fits easily but generation quality degrades rapidly — **3-bit is the minimum for coherent output on pre-quantized MoE models.**

---

### Qwen3.5-122B-A10B (122B total, 256 experts, 8 active, ~10B active)

**Hardware:** Apple M4 Max 64GB — the original BF16 model is ~240 GB. TurboQuant 3-bit compresses it to ~50 GB, fitting on a 64GB machine.

This is a brand-new architecture featuring **256 MoE experts** (the most of any model we've tested), **hybrid attention** (GatedDeltaNet linear attention + standard softmax attention), and **thinking/reasoning** capability. The model also has a shared expert per layer alongside the routed experts.

#### Step 1: Convert to TurboQuant 3-bit

```bash
python -m turboquant_mlx.convert \
    --hf-path Qwen/Qwen3.5-122B-A10B \
    --mlx-path ./qwen3.5-122b-tq3 \
    --bits 3 --group-size 64
```

**Model size:** ~50 GB | **Conversion time:** ~90 seconds

> **Note:** Conversion requires ~55 GB peak memory. Close all other applications before running. The converter uses memory-efficient processing — each expert layer is replaced immediately after quantization with aggressive garbage collection to handle the 256 experts per layer.

#### Step 2: Generate text

```bash
turboquant-generate \\
    --model ./qwen3.5-122b-tq3 \
    --prompt "Why is the sky blue? Explain in simple terms." \
    --max-tokens 200
```

**Expected:** ~26.5 tok/s generation, 55 GB peak memory on M4 Max 64GB

#### Benchmark

| Method | Bits | Size | Peak RAM | Gen Speed | Fits 64 GB? | Quality |
|--------|------|------|----------|-----------|-------------|---------|
| BF16 (original) | 16 | ~240 GB | — | — | **No** | — |
| **TurboQuant** | **3** | **~50 GB** | **54.9 GB** | **26.5 tok/s** | **Yes** | **Coherent reasoning with structured thinking** |

> Qwen3.5-122B-A10B is the largest and most complex model TurboQuant has been tested on: 122B parameters, 256 experts (8 active per token), hybrid GatedDeltaNet + softmax attention, and a shared expert per MoE layer. At 3-bit, the model produces structured reasoning with proper analysis steps — demonstrating that TurboQuant preserves thinking capability at extreme compression.

#### Run it on a 16 GB Mac mini (expert streaming)

This 122B model — ~54 GB on disk — also runs on a **16 GB Mac mini** via expert streaming (the same mechanism as [Qwen3.6-35B-A3B](#qwen36-35b-a3b-on-a-16-gb-mac-mini-expert-streaming)). Only the router-selected experts are paged from disk per token (LRU-cached), so the resident footprint stays well under the machine's GPU wired-memory cap, and output is bit-identical to the fully-resident model. Requires `turboquant-mlx-full>=0.4.1`.

```bash
python -m turboquant_mlx.stream.stream_generate \
    --model manjunathshiva/qwen3.5-122b-tq3 \
    --prompt "Explain why the sky is blue." \
    --max-tokens 128 --cache-budget-gb 4
```

Measured on a **base Apple M4 Mac mini, 16 GB**:

| Cache budget | Expert hit-rate | Disk read / token | Decode | Peak (mlx) |
|---|---|---|---|---|
| `--cache-budget-gb 1` | 0% | ~1.78 GB | ~0.6 tok/s | 6.0 GB |
| `--cache-budget-gb 4` *(recommended)* | **44.6%** | ~0.93 GB | **~1.1 tok/s** | 9.0 GB |

On a 16 GB machine the binding limit is the **Metal GPU wired-memory cap (~10.5 GB)**, not total RAM — and the expert cache counts against it, so `mlx_peak ≈ 5 GB + cache_budget`. `--cache-budget-gb 4` is the sweet spot (~9 GB peak, safe margin); higher budgets risk a Metal out-of-memory error. Throughput is disk-bandwidth-bound (~10B active params/token) → ~1 tok/s on a single mini SSD. Slow, but **a 122B model running on a 16 GB Mac** is the result.

---

### Qwen3.6-35B-A3B on a 16 GB Mac mini (expert streaming)

**Hardware:** Apple M4 Max 64GB to convert; runs **fully resident on 64 GB** or on a **16 GB Mac mini via expert streaming**. Qwen3.6-35B-A3B is a hybrid linear-attention (`qwen3_5_moe`, qwen3_next-style) + MoE model — **256 routed experts (top-8) + 1 shared**, ~35B total / ~3B active. The text-only language model is extracted (the vision tower is dropped during conversion).

A pre-converted 3-bit (group-size 32) model is on the Hub:

→ [`manjunathshiva/Qwen3.6-35B-A3B-tq3-g32`](https://huggingface.co/manjunathshiva/Qwen3.6-35B-A3B-tq3-g32) — ~16 GB on disk; ~60 tok/s at ~18 GB peak when fully resident on a 64 GB Mac.

#### Run it fully resident (64 GB)

```bash
turboquant-generate \
    --model manjunathshiva/Qwen3.6-35B-A3B-tq3-g32 \
    --prompt "Explain why the sky is blue." \
    --max-tokens 512
```

#### Run it on a 16 GB Mac mini (expert streaming)

The model is ~16 GB on disk, so it won't fit fully resident in 16 GB alongside the OS (resident decode peaks ~18 GB). Expert streaming pages only the router-selected experts from disk per token (LRU-cached), keeping resident memory to a few GB. Output is **bit-identical** to the fully-resident model. (`os.pread` + macOS `F_NOCACHE` keep the OS page cache from ballooning while streaming.)

Since `v0.5.0` the missing experts for each layer are read **in parallel** on a thread pool (`--prefetch-workers`, default `8`), hiding SSD latency behind compute — ~1.9× faster decode at a tight cache budget, still bit-identical. Pass `--prefetch-workers 1` for the serial baseline.

```bash
python -m turboquant_mlx.stream.stream_generate \
    --model manjunathshiva/Qwen3.6-35B-A3B-tq3-g32 \
    --prompt "Explain why the sky is blue." \
    --max-tokens 512 --cache-budget-gb 8
```

#### Benchmark (base Apple M4 Mac mini, 16 GB)

| Config | Expert hit-rate | Disk read / token | Decode | Peak RSS |
|--------|-----------------|-------------------|--------|----------|
| `--cache-budget-gb 2` | ~60% | ~175 MB | ~3.0 tok/s | **3.9 GB** |
| `--cache-budget-gb 8` *(recommended)* | **91%** | ~41 MB | **~4.5 tok/s** | 9.4 GB |

A larger cache keeps more experts resident, raising the hit-rate and cutting SSD reads — the throughput limiter when streaming. `--cache-budget-gb 8` is the sweet spot on a 16 GB machine; drop to `2` if RAM is tight. Streaming targets the SwitchGLU expert layout used by `qwen3_5_moe` and the DeepSeek MLA+MoE family (`deepseek_v2`/`v3`); the loader auto-detects the model's layer-key prefix.

> **Note:** Qwen3.6 is a thinking-mode model — it emits a reasoning trace before the final answer, so give it a generous `--max-tokens` (512+) for tasks that need a concluding answer.

#### Tuning the streaming reader (`v0.6.1`)

Once the cache policy is reasonable, **disk bandwidth is the wall** — for MoE decode the LRU + 8-worker parallel-read pool is already near-optimal, so the big levers are faster storage (Thunderbolt/NVMe) and fewer bytes/token (a hybrid build, a bigger `--cache-budget-gb`), not the read algorithm. A few knobs squeeze the rest:

> **How much can a smarter cache policy actually buy? (measured, 122B on a 16 GB mini.)** The 0.15.0 **learning expert cache** (`learn_experts=True`, default) records which experts *your* traffic routes to and pins from that profile once mature — the adaptive cousin of a shipped `hot_experts.json`. On the genuinely disk-bound case it was built for — `Qwen3.5-122B-A10B-tq3a-tqTe-g64` (30.9 GB, `F_NOCACHE`, auto budget 4.3 GB), learned pins vs pure LRU over identical prompts — it lifted the cache hit rate **54.0 → 55.9%** and cut expert bytes read **84.4 → 81.0 GB (−4.0%)**, but decode speed stayed **inside run-to-run noise** (two byte-identical unpinned runs themselves varied ±7%). The reason is structural, and it is the same wall: 4.3 GB of cache against **27 GB of experts is 16%**, and 256-expert top-8 routing is too spread for a small pin to beat recency. At **~0.47 GB/token off the SSD**, trimming 4% is nothing. **The pin keeps better experts and reads a little less disk; it does not make streaming faster.** Faster storage and fewer bytes/token do — the learned profile's real value is warm-starting the cache (skipping the cold-record run), not throughput.


| Knob | Default | What it does |
|------|---------|--------------|
| `--max-active-experts K` | `4` | **K-reduction** — caps router `top_k` to `min(native, K)` per MoE block, so the switch streams fewer experts/token. `argpartition` selects fewer and `norm_topk_prob` renormalizes the gates, so it stays a clean reduced-K MoE. On Qwen3.6-35B-A3B (native top-8) **K=8→4 is byte-identical** on the 6-test stress harness and cuts streamed disk reads **~2.09×** (1.4× faster decode in the disk-bound regime); K=2 collapses (broken JSON). `4` = safe floor; `0` = native routing. |
| `--use-page-cache` / `--no-page-cache` | **auto** | **Trust-OS** — whether expert reads use the OS page cache (vs `F_NOCACHE`). On a roomy machine where the model fits in free RAM, leaving the page cache on returns LRU-eviction re-reads from warm RAM instead of disk: **2.44× faster decode** on the 35B at a small budget (7.58 → 18.50 tok/s), same hit-rate/RSS. Auto-enables only when model files are `< 0.6× total RAM`, so a 16 GB mini on a 70 GB MoE keeps `F_NOCACHE` and never thrashes. |
| read-coalescing | **on** | Merges contiguous missed experts into one `os.pread`. Bit-identical, free, ~5% faster when disk-bound. No flag. |
| `--prefetch-ahead N` | `0` (off) | Speculatively prefetch the next *N* layers' experts (predicted from the previous token's routing) on a background thread. ~+6% on fast NVMe with spare bandwidth; **self-disables** if the drive proves bandwidth-bound (e.g. a saturated USB bus), so it's safe to set `1`. |
| `--pin-file pin.json` | none | Keep a calibrated hot-expert set permanently resident. **Experimental** — measured net-negative vs pure LRU on a 122B (static pinning costs LRU's adaptivity). For experimentation only. |

The `--max-active-experts` and page-cache levers are ports of [Flash-MoE](https://github.com/danveloper/flash-moe)'s K-reduction and "trust the OS" findings (blueprint: Apple [*LLM in a Flash*](https://arxiv.org/abs/2312.11514)), measured on the TurboQuant streaming path.

Generate `pin.json` (and a co-activation `perm.json` for the optional `stream/repack_experts.py` relayout) with `python -m turboquant_mlx.stream.calibrate_experts`.

---

### Qwen3.6-27B (dense coding model for a 48 GB Mac)

**Hardware:** converts on a 64 GB Mac — or off slow USB storage with `v0.6.2+`, which forces the disk read ahead of GPU compute so conversion doesn't trip the Metal GPU watchdog — and runs **fully resident** on a **48 GB** Mac with headroom to spare. [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) is a **dense** (`qwen3_5`) long-context coder — 64 layers, hybrid attention (**48 GatedDeltaNet linear-attention + 16 full-attention** layers), head_dim 256, 262K context — that Qwen positions as competitive on SWE-bench Verified / SWE-bench Pro. Being dense, it has **no experts to stream**: it loads once and stays in RAM, so storage only matters for load time.

A pre-converted 3-bit (group-size 32) build is on the Hub:

→ [`manjunathshiva/Qwen3.6-27B-tq3-g32`](https://huggingface.co/manjunathshiva/Qwen3.6-27B-tq3-g32) — **~13 GB on disk**, **~17.5 GB peak** at runtime (fits 48 GB with ~30 GB free for KV), ~14 tok/s decode.

#### Run it

```bash
turboquant-generate \
    --model manjunathshiva/Qwen3.6-27B-tq3-g32 \
    --prompt "Write a Python function that merges overlapping intervals." \
    --max-tokens 512 --temp 0.7
```

#### Serve it to Cursor / VS Code (OpenAI-compatible)

```bash
turboquant-serve --model manjunathshiva/Qwen3.6-27B-tq3-g32 --port 8080
```

Point the IDE's custom OpenAI base URL at `http://localhost:8080/v1`. Stock `mlx_lm.server` **can't** load a TurboQuant model (`KeyError: 'turboquant'`) — `turboquant-serve` patches the loader so the weights load through the PolarQuant path.

#### Convert it yourself

```bash
python -m turboquant_mlx.convert \
    --hf-path Qwen/Qwen3.6-27B \
    --mlx-path ./Qwen3.6-27B-tq3-g32 \
    --bits 3 --group-size 32 --streaming
```

> **Note:** Qwen3.6 is a thinking-mode model — it emits a reasoning trace before the answer, so give it a generous `--max-tokens` (512+). Only the **16 full-attention layers** keep a growing KV cache (the 48 linear-attention layers use a fixed-size state), so it stays KV-light for long coding context; compress further with `--kv-k-bits 8 --kv-v-bits 3`.

---

### Qwen3-235B-A22B-Instruct-2507 — a 235B MoE that converts on a 16 GB Mac (hybrid + streaming)

**Hardware:** converts on a **16 GB Mac mini** via `--streaming`; runs on a **64 GB Mac** (expert streaming) or fully resident on **96 GB+**. Qwen3-235B-A22B is a `qwen3_moe` Mixture-of-Experts — **94 layers, 128 routed experts (top-8)**, ~235B total / ~22B active.

This is a **hybrid tq3a-tq2e** build: **3-bit attention** (the always-on path, kept safer) + **2-bit experts** (where the parameters — and the savings — live), routers full precision. The 128-expert / top-8 routing carries enough redundancy to absorb 2-bit experts cleanly — the same reason gpt-oss-120b holds at 2-bit while gpt-oss-20b (32 experts) collapses. Result: **~470 GB BF16 → 70.5 GB** (15 shards, 6.7×).

A pre-converted build is on the Hub:

→ [`manjunathshiva/Qwen3-235B-A22B-Instruct-2507-tq3a-tq2e-g32`](https://huggingface.co/manjunathshiva/Qwen3-235B-A22B-Instruct-2507-tq3a-tq2e-g32)

#### Convert it yourself (streaming, fits in ~8–12 GB RAM)

```bash
python -m turboquant_mlx.convert \
    --hf-path Qwen/Qwen3-235B-A22B-Instruct-2507 \
    --mlx-path /Volumes/SSD/qwen3-235b-tq3a-tq2e-g32 \
    --bits 3 --mlp-bits 2 -g 32 --streaming
```

`--mlp-bits 2` drops experts to 2-bit while `--bits 3` keeps attention at 3-bit; `--streaming` writes each quantized layer to a shard and frees it, so the full 235B converts in **~8–12 GB of RAM** — it was produced on a **16 GB Mac mini in ~18 minutes**. Point `--mlx-path` at a drive with ≥70 GB free.

#### Run it (expert streaming)

```bash
python -m turboquant_mlx.stream.stream_generate \
    --model manjunathshiva/Qwen3-235B-A22B-Instruct-2507-tq3a-tq2e-g32 \
    --prompt "Explain why the sky is blue." \
    --max-tokens 512 --cache-budget-gb 40
```

#### Quality + streaming benchmark

A 6-probe stress run passes **5/6**: coherent long-form essay, **correct multi-step math** ($142.80 with a 15% bulk discount), correct memoized Fibonacci, strict-JSON formatting, and clean 1–15 enumeration. The one miss was **exact factual recall** — an in-context password came back with a single flipped digit (`RAVEN-stone-91` → `-51`). Math/reasoning held; verify outputs where an exact literal value matters.

> **Need exact recall?** The full-3-bit sibling [**Qwen3-235B-A22B-Instruct-2507-tq3-g32**](https://huggingface.co/manjunathshiva/Qwen3-235B-A22B-Instruct-2507-tq3-g32) (3-bit experts, 103 GB) fixes the needle flip and passes **6/6** — at ~1.3 tok/s / 86.3% hit-rate on a 64 GB Mac (slower and bigger than this hybrid; the cost of full 3-bit experts). Pick the hybrid for the smallest footprint, the tq3 build when exact literal recall matters.

| Machine | Cache budget | Expert hit-rate | Disk read / token | Decode | Peak memory |
|---------|-------------|-----------------|-------------------|--------|-------------|
| M4 Mac mini, 16 GB | `--cache-budget-gb 6` | ~38% | ~3.2 GB | ~0.2 tok/s | 10.1 GB |
| 64 GB Mac | `--cache-budget-gb 40` | **94.1%** | ~0.28 GB | **~4–6 tok/s** (warm) | 46 GB |

On 64 GB a 40 GB cache holds ~60% of the ~67 GB of experts, but temporal locality lifts the hit-rate to **94.1%**, so warm decode runs at the compute-bound ~4–6 tok/s. Throughput is **bursty**: the first generation and tasks that route into a colder slice of experts stall on the SSD until their experts page in. Bump `sudo sysctl iogpu.wired_limit_mb=57344` to raise the cache past the ~48 GB default Metal wired cap.

---

### Nemotron-3 (Mamba/attention hybrid)

Nemotron-3 is NVIDIA's hybrid Mamba2 + attention architecture. Two variants are tested:

- **Nano-4B** — dense (Mamba + MLP + attention), 42 layers
- **Super-120B-A12B** — hybrid MoE (Mamba + 512-expert latent-MoE + attention), 88 layers, ~12B active per token

Both require **mlx-lm ≥ 0.31.3** for upstream Nemotron-H support (installed automatically).

#### Convert

```bash
# Nano-4B
python -m turboquant_mlx.convert \
    --hf-path nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 \
    --mlx-path ./nemotron-3-nano-4b-tq3 \
    --bits 3 --group-size 64

# Super-120B
python -m turboquant_mlx.convert \
    --hf-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
    --mlx-path ./nemotron-3-super-120b-tq3 \
    --bits 3 --group-size 64
```

#### Generate

Nemotron-3's chat template ends in a `<think>\n` scaffold that primes EOS as the top-1 logit at the start of the assistant turn. Pass `--min-tokens` to mask EOS for the first N tokens so the model enters the think phase:

```bash
turboquant-generate \\
    --model ./nemotron-3-super-120b-tq3 \
    --prompt "Why is the sky blue?" \
    --max-tokens 200 --min-tokens 50
```

#### Benchmarks (M4 Max)

| Model | Bits | Size | Peak RAM | Gen Speed | Quality |
|-------|------|------|----------|-----------|---------|
| **Nemotron-3-Nano-4B** | **3** | **~2.2 GB** | **4.3 GB** | **75.6 tok/s** | **Coherent** |
| **Nemotron-3-Super-120B-A12B** | **3** | **~50 GB** | **54.7 GB** | **18.7 tok/s** | **Coherent with structured `<think>` reasoning (974-token answer w/ self-correction, formulas, formatted structure)** |
| **Nemotron-3-Super-120B-A12B (hybrid)** | **3-attn / 2-experts, gs=32** | **~36 GB** | **~40.8 GB** | **~27.2 tok/s** | **Coherent prose, code, format, and long-context recall; math accuracy degraded — see Phase-1 note below** |

#### 48 GB-RAM target: hybrid (3-bit attention / 2-bit experts) at group-size 32

The standard 3-bit Super-120B (~50 GB) needs ~55 GB peak and only fits a 64 GB
Mac after raising `iogpu.wired_limit_mb`. For users on a 64 GB Mac who want
headroom for other applications — or for users on **48 GB Macs** — there is
a **hybrid quantization** that keeps attention at 3-bit (where precision
matters most) and pushes experts to 2-bit (where the bulk of the weights
live), at a smaller group size (g=32) that improves per-group fit.

**Pre-converted model on Hugging Face:**

```bash
hf download manjunathshiva/Nemotron-3-Super-120B-A12B-tq3a-tq2e-g32 \
    --local-dir ~/models/nemotron-3-super-120b-tq3a-tq2e-g32
```

→ [`manjunathshiva/Nemotron-3-Super-120B-A12B-tq3a-tq2e-g32`](https://huggingface.co/manjunathshiva/Nemotron-3-Super-120B-A12B-tq3a-tq2e-g32)
on the Hub: ~36 GB on disk, ~40.8 GB peak memory, ~27.2 tok/s decode, fits the
default 48 GB `iogpu.wired_limit_mb` cap.

**Or convert from BF16 source yourself:**

```bash
python -m turboquant_mlx.convert \
    --hf-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
    --mlx-path ./nemotron-3-super-120b-tq3a-tq2e-g32 \
    --bits 2 --attn-bits 3 --mlp-bits 2 --group-size 32
```

For long-form generation, the model needs a small repetition penalty to
avoid degenerate tail loops at >1500 tokens. The recommended decode config
(empirically validated to keep essay, code, format, and long-context
recall clean):

```bash
turboquant-generate \\
    --model ./nemotron-3-super-120b-tq3a-tq2e-g32 \
    --prompt "Why is the sky blue?" \
    --max-tokens 4096 --min-tokens 50 \
    --temp 0.7 --rep-penalty 1.04 --rep-ctx 256
```

> **Phase-1 known limitation: math accuracy.** Step-by-step arithmetic
> on the hybrid degrades under any non-zero `--rep-penalty`. For
> numeric/math prompts in Phase 1, **omit `--rep-penalty`** (you may
> see long-gen tail loops on long prompts, but the arithmetic will land
> correctly more often). A permanent fix is planned for Phase 2 — likely
> first/last-layer bit protection, a calibration-data codebook, or a
> fused QJL Metal kernel. Until Phase 2, use the hybrid for prose,
> coding, format, and long-context tasks; use the standard 3-bit model
> for serious numeric work.

The fused MoE decode kernel transparently chunks expert routings on long
prompts, so this hybrid handles long-context retrieval (e.g. password-
recall over 4000+ tokens of context) without the kernel argument-validation
crash that affected earlier builds.

### Poolside Laguna (33B and 118B MoE for agentic coding)

Two models share this architecture: **XS.2** (33B) and **S-2.1** (118B). The port
covers both; the licence does **not** — XS.2 is Apache-2.0, S-2.1 is OpenMDW-1.1.
Check per repo before redistributing derived weights.

#### Laguna-XS.2 (33B)

[Laguna-XS.2](https://huggingface.co/poolside/Laguna-XS.2) is Poolside's 33B
Mixture-of-Experts (256 experts, top-8, 3B active) built for **agentic coding
and long-horizon work on a local machine**. It is a custom architecture —
mixed sliding-window / global attention in a 3:1 ratio, per-head attention
gating, QK-norm, partial+YaRN RoPE on the global layers, and a sigmoid router
with a selection-only correction bias. `mlx-lm` has **no native `laguna`
model**; the MLX port and loader ship here (registered via `compat.py`), with
logit-parity of 2e-7 against the transformers reference.

```bash
# Convert bf16 -> TurboQuant 3-bit. bf16 (~67 GB) exceeds 64 GB RAM, so use --streaming.
python -m turboquant_mlx.convert \
    --hf-path poolside/Laguna-XS.2 \
    --mlx-path ./laguna-xs2-tq3-g64 \
    --bits 3 --group-size 64 --streaming

# Generate. Laguna's generation_config ships top_p 1.0; ALWAYS pass --top-p 0.9
# or the untruncated 100k-vocab tail injects rare junk tokens.
python -m turboquant_mlx.generate \
    --model ./laguna-xs2-tq3-g64 \
    --prompt "Write a mergesort in Python." --temp 0.7 --top-p 0.9

# Serve (agentic use). Do NOT pass --kv-bits: 30/40 layers are sliding-window(512)
# so KV is already tiny (peak 16.8 GB @2K -> 19.4 GB @31K); KV-quant saves nothing
# here and can cost up to ~4x decode.
turboquant-serve --model ./laguna-xs2-tq3-g64 --port 8080 \
    --temp 0.7 --top-p 0.9 --prompt-concurrency 1 \
    --chat-template-args '{"enable_thinking": false}'
```

- **Runs on a 32 GB Mac** at 13.8 GB on disk / ~16 GB peak; MMLU-Redux 76.9 %,
  GSM8K 79.0 % (3-bit, no-think), and passes the Opencode observe→read→edit→verify
  loop.
- **Tool calls are GLM-XML**, not Hermes-JSON; `mlx-lm`'s `glm47` parser is
  auto-selected and patched here for a trailing-newline function-name bug.
- **3-bit vs 4-bit**: this data-free codebook 3-bit build is 4 GB smaller than
  MLX-native affine 4-bit but ~2× slower per token (codebook decode + online
  Hadamard rotation, not bandwidth). Prefer it when memory-bound; prefer affine
  4-bit for the fastest interactive loop.

#### Laguna-S-2.1 (118B) — ternary experts, resident on a 64 GB Mac

[Laguna-S-2.1](https://huggingface.co/poolside/Laguna-S-2.1) is the 118B sibling
(256 experts, top-10 + 1 shared, `moe_intermediate` 1024, 48 layers). At bf16 it
is ~219 GB. **Ternary (1.58-bit) experts bring it to ~27 GB, which is the only
build of this model that runs resident on a 64 GB Mac** — affine 4-bit (~64 GB)
exceeds Metal's ~56 GB working set and won't load at all, and TurboQuant's own
3-bit (48 GB) peaks at 52.7 GB and starves the OS.

```bash
python -m turboquant_mlx.convert --hf-path poolside/Laguna-S-2.1 \
    --mlx-path ./laguna-s21-tqTe-g64 \
    --attn-bits 3 --ternary-experts --group-size 64 --streaming

# Resident on 64 GB. --no-think matters: Laguna reasons out loud by default, so
# a plain generate call can spend its whole budget deliberating and never answer.
python -m turboquant_mlx.generate --model ./laguna-s21-tqTe-g64 \
    --prompt "Write a mergesort in Python." \
    --temp 0.7 --top-p 0.9 --max-tokens 1024 --no-think
```

- **~30 GB peak resident, ~12.5 tok/s** on an M4 Max, leaving ~25 GB for the OS.
- **MMLU-Redux 79.5 %, GSM8K 82.5 %** — at 1.58-bit experts this 118B still edges
  the 33B XS.2 at 3-bit (76.9 / 79.0). The larger, wider expert pool absorbs the
  deeper quantization.
- **Passes Opencode agentic coding at 1.58-bit**, where the 512-wide-expert 35B
  ternary build fails the same harness. Expert **width** (1024 vs 512), not just
  count, is what sustains sub-2-bit — see [Sub-2-bit ternary experts](#key-results--weight-compression).

**Expert streaming** works on Laguna too, so 27 GB resident is not the floor —
only 2.28 GB of this model has to stay resident, the other 26.42 GB pages from
disk on demand:

```bash
python -m turboquant_mlx.stream.stream_generate --model ./laguna-s21-tqTe-g64 \
    --cache-budget-gb 4 --max-active-experts 0 --prompt "..." --max-tokens 256
```

Measured on an M4 Max with a 4 GB expert cache: **peak 7.3 GB** (RSS 6.6 GB),
141 of 144 expert projections streamed, ~5 tok/s, 56.8 % cache hit rate — a
footprint that fits a **16 GB Mac mini**. Use `--max-active-experts 0` to keep
native top-10 routing (lowering it reads less from disk but changes which experts
run, and breaks agentic tool-calling).

> Requires `turboquant-mlx-full >= 0.18.0`. Earlier versions matched expert
> tensors by the `switch_mlp` name only, so Laguna's `mlp.experts` container
> counted as resident: `turboquant-plan` reported "won't run" on small Macs and
> `--cache-budget-gb auto` sized itself from a bad figure. Explicit
> `--cache-budget-gb` was unaffected.

---

### Kimi K3 (2.8T MoE) — the largest model TurboQuant-MLX has run

[Kimi K3](https://huggingface.co/moonshotai/Kimi-K3) is Moonshot's 2.8-trillion-
parameter MoE: 93 layers (69 KDA linear-attention + 24 gated-NoPE MLA with
q-LoRA), 896 routed experts at top-16 plus 2 shared, and a **Stable LatentMoE**
where experts run in a 3584-d latent behind shared per-layer down/up projections
while the router reads the full 7168-d stream. `mlx-lm` has no native `kimi_k3`,
so the MLX port ships here — contributed by
[@anders94](https://github.com/anders94) in
[#71](https://github.com/manjunathshiva/turboquant-mlx/pull/71), verified at
≤ 2.7e-6 fp32 forward parity against `transformers`.

The published source is 1.56 TB of mxfp4 compressed-tensors, which TurboQuant
reads directly (bit-exact unpack, no fp16 round-trip):

```bash
pip install "turboquant-mlx-full[kimi]"   # tiktoken + blobfile for K3's tokenizer

python -m turboquant_mlx.convert --hf-path ./Kimi-K3 \
    --mlx-path ./Kimi-K3-tq3a-tqTe-down4-g64 \
    --bits 3 -g 64 --ternary-experts --expert-down-bits 4 --streaming
```

**This model does not fit resident on any Mac you can buy, at any bit width.**
2.8T parameters want roughly a 768 GB machine; the cheapest recipe here is
617 GB and 512 GB is the current ceiling. It runs by **streaming experts from
disk** — each token pages in only its router-selected experts:

```bash
sudo sysctl iogpu.wired_limit_mb=512000      # 512 GB box; resets on reboot

python -m turboquant_mlx.stream.stream_generate \
    --model ./Kimi-K3-tq3a-tqTe-down4-g64 \
    --prompt "Explain how Rayleigh scattering works." \
    --max-tokens 500 --cache-budget-gb auto \
    --max-active-experts 8 --prefetch-workers 16 --fanout
```

Measured on a **Mac Studio (512 GB)** with a 506 GB expert cache and the shipped
hot-expert list: **2.3 tok/s** at `--max-active-experts 8`, **1.2 tok/s** at
native top-16. Full numbers and the recipe breakdown are in
[`model_card_kimi_k3.md`](model_card_kimi_k3.md).

> `--fanout` matches how those numbers were measured — it parallelizes each
> layer's expert reads across the pool instead of reading them serially. It is
> off by default because it trades away the coalesced serial read path, which
> wins on bandwidth-bound external storage. It does not self-disable, so measure
> it on your own disk.
>
> `--max-active-experts 8` halves K3's native top-16. That is a 2× cut, the ratio
> validated elsewhere in this repo — but K-reduction changes which experts run
> and has broken agentic tool-calling on other models. Use
> `--max-active-experts 0` for native routing whenever quality matters.

### Sarvam MoE

[sarvam-30b](https://huggingface.co/sarvamai/sarvam-30b) and later models of that
type convert and run through the `sarvam_moe` port shipped here (mlx-lm has no
native module). The port is verified against the fp32 reference three ways: full
key/shape coverage of all 7122 source tensors, layer-1 parity at 5.4e-7, and
whole-model logit agreement.

> **No quantized sarvam build is published, deliberately.** At 4-bit this model
> degenerates on long generations — measured as a rate over repeated trials at
> temp 0.7 / top_p 0.9, **TurboQuant 4-bit degenerated 8/15 (53%) and mlx-lm
> affine 4-bit 15/15 (100%)**. TurboQuant is clearly the better of the two and
> neither is good enough to ship. Use a higher bit width and validate long-form
> output on your own prompts before relying on it.

### Muse Glimmer (30B dense VLM)

[Muse Glimmer](https://huggingface.co/meta-models/Muse-Glimmer-30B) is Meta
Superintelligence Lab's 29.8B-parameter dense multimodal model, built for
agentic work on consumer hardware. Architecturally it is Gemma2-style sandwich
norms plus **gated attention** (a sigmoid output gate alongside Q/K/V), a 3:1
sliding(2048)/full attention pattern with **NoPE on the full-attention layers**,
final logit softcapping, and a 1.9B ViT-G/14 perception encoder.

**This is the one case where TurboQuant beats MLX's affine quantizer outright
on quality**, at matched bit width — and does it while quantizing more of the
model:

| build | size | PPL ↓ | decode | notes |
|---|---|---|---|---|
| `mlx-community/Muse-Glimmer-30B-4bit` | 19.88 GiB | 4.3798 | 26.1 tok/s | embedding + vision tower stay bf16 |
| **TurboQuant `tq4-g64`** | **14.88 GiB** | **4.3315** | 9.4 tok/s | everything quantized |
| TurboQuant `tq3-g64` | 12.53 GiB | 5.0454 | 10.9 tok/s | smallest coherent build |

800-token fixed-passage perplexity, greedy, M4 Max. The affine build is 2.4–2.8×
faster to decode — the usual codebook-vs-affine trade. **Do not read the tq3 row
as a TurboQuant deficiency**: the gap there is the 3-bit step, which is exactly
what the matched-4-bit control isolates.

Most of the affine build's extra 5 GB is the embedding and the vision tower,
which mlx-vlm declines to quantize. `embed_tokens` is a `NormedEmbedding` whose
inherited `to_quantized` would silently drop its RMS normalization; TurboQuant
quantizes it through a norm-preserving subclass instead.

```bash
# mlx-vlm 0.6.12 shipped the Muse Glimmer classes, so this is a plain install.
pip install "turboquant-mlx-full[vlm]"

python -m turboquant_mlx.convert_vlm \
  --hf-path meta-models/Muse-Glimmer-30B \
  --mlx-path ./Muse-Glimmer-30B-tq4-g64 \
  --bits 4 --group-size 64 --quantize-extras --extras-bits 4

python -m turboquant_mlx.generate_vlm \
  --model ./Muse-Glimmer-30B-tq4-g64 --prompt "..." --max-tokens 256
```

`--extras-bits 4` matters here: the embedding and vision tower are 3.3B
parameters between them, ~3.9 GB at the 8-bit default versus ~1.8 GB at 4-bit.

> **16 GB Mac mini: yes — measured.** `tq3-g64` runs **resident** on a
> Mac16,10 (macOS 26.5.2) with `iogpu.wired_limit_mb=14336`, at every prompt
> length tried up to 5068 tokens:
>
> | prompt | peak | prefill | decode |
> |---|---|---|---|
> | 268 tok | 13.44 GiB | 19.9 tok/s | 3.70 tok/s |
> | 868 tok | 14.13 GiB | 20.2 tok/s | 3.44 tok/s |
> | 2068 tok | 13.91 GiB | 17.0 tok/s | 3.38 tok/s |
> | 5068 tok | 14.21 GiB | 6.0 tok/s | 3.56 tok/s |
>
> Meta's own smallest Apple Silicon artifact is 17.95 GB, text-only and without
> the drafter — larger than the machine's whole RAM. Two caveats: headroom at
> 5068 tokens is **0.20 GB**, and prefill collapses past ~2000 tokens (20 → 6
> tok/s) while decode stays flat, so long prompts are batch work. The smaller
> alternative does **not** work — a `tq3a-tq2e-g64` hybrid fits at 9.63 GiB but
> measures **PPL 7.5547** (vs 5.0454) and visibly corrupts text, the same way
> 2-bit experts failed on 32-expert GPT-OSS-20B. 2-bit on a *dense* MLP is not
> a usable tier.

Muse Glimmer also ships a [DFlash block-diffusion
drafter](https://huggingface.co/meta-models/Muse-Glimmer-30B-assistant) (5.11 GB,
5 layers, block 16, tapping target layers {1,13,25,37,49}). It is not wired up
here. Worth noting it is a *dense* target, which is the regime where DFlash
speculation paid off in earlier testing rather than the MoE regime where it
did not.

---

### Qwen3.8-27B (dense 27.8B hybrid Gated-DeltaNet VLM)

[Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) reports
`model_type: qwen3_5` despite the name — a **dense 27.8B multimodal** model whose
64 decoder layers are **48 Gated-DeltaNet (linear attention) + 16 full
attention**, one full layer in every four. Hidden 5120, intermediate 17408,
**vocabulary 248,320**, plus a 27-block vision tower. mlx-lm 0.31.3 and mlx-vlm
0.6.3 already carry the architecture, so nothing needed porting; conversion
takes about 18 seconds.

Two builds are published, and they target different machines.

| build | size | peak (1.3K prompt) | decode | Opencode agentic |
|---|---|---|---|---|
| **[`tq4-g64`](https://huggingface.co/manjunathshiva/Qwen3.8-27B-tq4-g64)** | **15.15 GiB** | 22.85 GB | 11.4 tok/s | **pass** (6m12s) |
| [`tq3-mini-g64`](https://huggingface.co/manjunathshiva/Qwen3.8-27B-tq3-mini-g64) | 11.55 GiB | 13.30 GiB | 12.2 tok/s | pass (6m32s)¹ |
| `tq3-g64`, 8-bit extras (unpublished) | 12.88 GiB | 20.60 GB | 9.2 tok/s | pass (6m32s) |

¹ the agentic run was done on the 8-bit-extras `tq3`; the published `tq3-mini`
differs only in the extras tier, at an identical 8.0052-vs-8.0551 perplexity.

Both fix a planted off-by-one end to end: run the given pytest command, read the
file, make the minimal edit, re-run to green, explain with `file:line`. Vision
works on both — shapes, colours, positions and grounded bounding boxes.

**Take `tq4` if it fits; take `tq3-mini` if it does not.** At 4-bit, `tq4` is
*faster as well as better* than the 8-bit-extras `tq3` for 2.3 GiB more — the
same result the [speed flip](#the-speed-flip) predicts, since the codebook path
only earns its keep at 2-bit and 3-bit's 10-indices-per-`uint32` packing costs
more to unpack than 4-bit's clean 8.

The exception is the 16 GB tier, where nothing else reaches.
[`tq3-mini-g64`](https://huggingface.co/manjunathshiva/Qwen3.8-27B-tq3-mini-g64)
is **verified on a real 16 GB M4 Mac mini** — not projected down from a bigger
machine — at `iogpu.wired_limit_mb=14336` and `--prefill-step-size 256`:

| prompt | peak, 16 GB mini | peak, 64 GB M4 Max\* | prefill | decode |
|---|---|---|---|---|
| 220 tok | **12.95 GiB** | 13.22 GiB | 20.7 tok/s | 3.9 tok/s |
| 818 tok | **12.96 GiB** | 13.05 GiB | 19.0 tok/s | 3.9 tok/s |
| 2,014 tok | **13.14 GiB** | 13.30 GiB | 21.0 tok/s | 3.6 tok/s |
| 5,017 tok | **13.61 GiB** | 13.81 GiB | 20.6 tok/s | 3.7 tok/s |

\* The M4 Max sweep realized 246 / 844 / 2,040 / 5,043 prompt tokens — a
constant 26 more than the mini's, worth 1.62 MiB of KV at 64 KiB/token. Read the
comparison as close, not as an exact A/B.

Vision passes there too — OCR correct at 350×100 (12.44 GiB) and 700×200
(12.68 GiB). Peaks reproduced identically across two independent sweeps, and
they run **0.09–0.27 GiB *below*** the comparable prompts on the M4 Max — 55–170×
more than that 26-token gap can account for — so the larger machine is the
pessimistic estimate. Reproduce with
[`benchmarks/bench_mini_qwen38.py`](benchmarks/bench_mini_qwen38.py); the full
record is in
[`bench_mini_qwen38_results.json`](benchmarks/bench_mini_qwen38_results.json).

Neither 4-bit build reaches that tier: affine 4-bit peaks at **16.16 GiB**, more
than a 16 GB machine has in total. Decode there is **3.7 tok/s** — fine for
document Q&A, summarization and vision calls, too slow for interactive chat.

The hybrid stack makes KV very cheap — only 16 of 64 layers cache anything, so
**64 KiB/token**, or 2.0 GiB at 32K context.

```bash
pip install "turboquant-mlx-full[vlm]"

python -m turboquant_mlx.convert_vlm \
  --hf-path Qwen/Qwen3.8-27B \
  --mlx-path ./Qwen3.8-27B-tq4-g64 \
  --bits 4 --group-size 64 --quantize-extras --extras-bits 8

python -m turboquant_mlx.generate_vlm \
  --model ./Qwen3.8-27B-tq4-g64 --prompt "..." --max-tokens 256
```

Measured on mlx-lm 0.31.3 with **both mlx-vlm 0.6.3 and 0.6.13** — identical
peaks and identical vision results on each, so the `[vlm]` floor of 0.6.12
reproduces these numbers.

`--prefill-step-size` is the memory knob for long prompts, and it is a large
one. mlx-vlm only chunks prefill *above* the step size, so a 1342-token prompt
runs unchunked at the 2048 default:

| `--prefill-step-size` | peak | prefill |
|---|---|---|
| default (2048 — unchunked here) | 21.28 GB | 162 tok/s |
| **256** | **16.77 GB** | 77 tok/s |

**−4.5 GB for 2.1× slower prefill.** 256 also keeps every TurboQuant layer on
the fused Metal kernel, which materializes nothing. This is what makes the
24 GB target work.

#### `lm_head` stays off the polar path

With a 248,320-token vocabulary `lm_head` is 248320×5120 = **1.271B
parameters**, the largest matrix in the model. `PolarQuantizedLinear` has a
fused Metal kernel only up to `_QMM_MAX_TOKENS` (256) tokens; past that it
dequantizes the weight through full-size intermediates. Measured on exactly that
shape at 3-bit/g64:

| tokens | transient peak |
|---|---|
| 256 | +0.116 GiB |
| **257** | **+13.141 GiB** |
| 2048 | +13.953 GiB |

The same shape as an 8-bit affine layer costs +1.895 GiB at 2048 tokens — that
is the logits array itself, with no weight-dequant term at all. So `qwen3_5`
joins `muse_glimmer` in `VLM_SKIP_PATTERNS`, and `lm_head` is left to the
`--quantize-extras` affine path. End to end on a 1342-token prompt:

| build | size | peak |
|---|---|---|
| `lm_head` affine | 15.15 GiB | **22.85 GB** |
| `lm_head` polar | 14.52 GiB | **33.34 GB** |

Polar buys 0.63 GB on disk and costs 10.5 GB at runtime — and that 33 GB peak is
the difference between fitting a 36 GB Mac and not.

This is easy to miss because mlx-vlm's chunked prefill *discards* the chunk
forward's return value, so MLX never evaluates the matmul at all. But chunking
only engages **above** `prefill_step_size` (default 2048); a prompt of 257–2048
tokens takes the single-shot branch in `generate/ar.py`, which slices
`logits[:, -1, :]` and therefore does evaluate `lm_head` across the whole
sequence. That is an ordinary chat turn, not an edge case.

#### Against MLX's own affine 4-bit

[`mlx-community/Qwen3.8-27B-4bit`](https://huggingface.co/mlx-community/Qwen3.8-27B-4bit)
is the same architecture at the same 4-bit / group-size-64 setting, with no MTP
weights on either side — so this is a like-for-like comparison, run through the
same harnesses and the same server.

The two TurboQuant columns differ only in what the *non-polar* remainder
(embeddings, `lm_head`, vision tower) is stored at — `--extras-bits 8` versus
`4`. They are separate builds with separate numbers, so they get separate
columns.

| | affine 4-bit | `tq4-g64` extras 8 | `tq4-g64` extras 4 |
|---|---|---|---|
| on disk | 14.95 GiB | 15.15 GiB | **13.81 GiB** |
| bits/weight (polar tier) | 4.500 | **4.251** | **4.251** |
| PPL, 16K tok | 7.2685 | **7.2496** | 7.3043 |
| PPL, 32K tok | 8.3593 | **8.3321** | 8.4061 |
| decode | **29.7 tok/s** | 11.3 tok/s | 11.5 tok/s |
| peak, same harness | **16.16 GiB** | 17.80 GiB | 16.47 GiB |
| vision battery | 3/4 | 4/4 | — |
| Opencode agentic | **pass** (6m42s) | **pass** (6m12s) | — |

Only the `extras 8` build is published, and the vision/agentic runs were done
on it; the `extras 4` column is the size-matched control.

**Take the affine build if you want 4-bit Qwen3.8-27B.** It is 2.6× faster to
decode at a lower peak, and the perplexity difference is 0.3% — real and stable
in ordering across both corpus sizes, but far too small to trade that speed for.

The one place TurboQuant is structurally ahead is storage. At identical bit
width and group size, on the same 17408×5120 projection:

| | weight | scales | biases | bits/weight |
|---|---|---|---|---|
| affine 4-bit | 42.50 MiB | 2.66 | **2.66** | **4.500** |
| polar 4-bit | 42.50 MiB | 2.66 | — | **4.251** |

The Hadamard rotation symmetrizes each group, so there is no zero-point to
store — 5.5% leaner for free. That margin is a rounding error at 4-bit and the
reason the codebook only earns its keep at 2-bit and below (see
[the speed flip](#the-speed-flip)).

Two cautions on reading the table. The vision split is **not** a capability
gap: affine's single miss is one string (`VOLTAGE 47` → `VOLTAGE`), and a
five-string follow-up probe put it at 4/5, so treat 3/4-vs-4/4 as a tie. And
the agentic wall-clock is **not** a speed metric — affine decodes 2.6× faster
yet finished *slower*, because it spent more tool turns.

Reproduce both halves:

```bash
# 16K-token column (--chunks 32) and 32K-token column (--chunks 64);
# both records are checked in under benchmarks/
python -m turboquant_mlx.benchmarks.eval_vlm_perplexity \
    ./Qwen3.8-27B-tq4-g64 ./Qwen3.8-27B-tq4-g64-x4 \
    mlx-community/Qwen3.8-27B-4bit --chunks 32
python -m turboquant_mlx.benchmarks.eval_vlm_perplexity \
    ./Qwen3.8-27B-tq4-g64 ./Qwen3.8-27B-tq4-g64-x4 \
    mlx-community/Qwen3.8-27B-4bit --chunks 64
python -m turboquant_mlx.benchmarks.eval_vlm_vision \
    ./Qwen3.8-27B-tq4-g64 mlx-community/Qwen3.8-27B-4bit
```

#### Will it fit?

`turboquant-plan --model <path>` at 16K context, per machine:

| Mac | verdict |
|---|---|
| 16 GB | ❌ **Will not run.** Peak 18.44 GB against 15.46 GB usable — over by 2.98 GB. A dense model has no expert tier to stream, so the [16 GB recipe](#recipe-a-coding-agent-on-a-16-gb-mac-mini) does not apply |
| 24 GB | ⚠️ Fits **after raising the Metal wired cap**, with `--prefill-step-size 256` — that knob measures 16.77 GB peak on a 1342-token prompt, against 21.28 GB at the default |
| 36 GB | ✅ Resident, 12.34 GB headroom |
| 64 GB | ✅ Resident, 39.40 GB headroom |

> **Speed is the caveat.** At ~11 tok/s decode and ~110–160 tok/s prefill an
> agentic turn takes about six minutes, against 33 s for Laguna-XS.2 at 3-bit.
> It passes, but this is a model to serve for quality rather than latency.

---

## How It Works

TurboQuant is a two-stage, **calibration-free** quantization pipeline:

1. **Hadamard Rotation** — Multiply weights by a randomized Hadamard matrix, transforming any weight distribution into a near-Gaussian shape. This is data-oblivious (no calibration data needed).

2. **Lloyd-Max Codebook** — Apply information-theoretically optimal quantization for Gaussian distributions. The codebook is a mathematical constant, precomputed once.

The result: near-zero quality loss at 3-bit, and usable 2-bit quantization where standard affine completely breaks down.

For MoE models, all experts within a layer share the same rotation signs and codebook, keeping storage efficient.

## CLI Options

```
python -m turboquant_mlx.convert --help

Options:
  --hf-path TEXT       HuggingFace model path or local path (required)
  --mlx-path TEXT      Output directory (default: mlx_model)
  --bits {2,3,4}       Quantization bit-width (default: 3)
  --group-size {32,64,128}  Elements per quantization group (default: 64)
  --rotation TEXT      Rotation method: hadamard, blockwise_hadamard, none
  --use-qjl           Enable 1-bit QJL residual correction (+1 bit overhead)
  --dtype TEXT         Model dtype before quantization: float16, bfloat16
```

## Supported Architectures

| Architecture | Model Type | MoE | Status |
|-------------|-----------|-----|--------|
| LLaMA / Llama 3 | `llama` | No | Tested |
| Qwen2 / Qwen2.5 | `qwen2` | No | Tested |
| Qwen3.5 / Qwen3.8 (text) | `qwen3_5` | No | Tested |
| Qwen3.5-family VLM (hybrid Gated-DeltaNet + full attention, via **mlx-vlm**) | `qwen3_5` | No | Tested (Qwen3.8-27B, 27.8B dense: `tq4-g64` = 15.15 GiB, `tq3-mini-g64` = 11.55 GiB **verified on a real 16 GB Mac mini**; both pass an Opencode agentic task and the vision path). `lm_head` is kept off the polar path — see [Qwen3.8-27B](#qwen38-27b-dense-278b-hybrid-gated-deltanet-vlm) |
| Mistral | `mistral` | No | Tested |
| Qwen1.5-MoE | `qwen2_moe` | Yes | Tested |
| GPT-OSS | `gpt_oss` | Yes | Tested |
| Qwen3.5-MoE / Qwen3.6-35B-A3B | `qwen3_5_moe` | Yes (256 experts) | Tested (122B, 35B-A3B); 35B streams on a 16 GB Mac mini |
| Qwen3-MoE | `qwen3_moe` | Yes (128 experts, top-8) | Tested — Qwen3-235B-A22B converted to a hybrid **tq3a-tq2e** build (70.5 GB) on a 16 GB Mac mini via `--streaming`; streams and passes 5/6 quality probes on a 64 GB Mac |
| Nemotron-H (Mamba/attention hybrid) | `nemotron_h` | Yes (512 experts w/ latent MoE on Super-120B) | Tested (Nano-4B, Super-120B) — requires mlx-lm ≥ 0.31.3 |
| Poolside Laguna (SWA + global attn, per-head gating) | `laguna` | Yes (256 experts, top-8/top-10) | Tested (XS.2 33B at 3-bit; S-2.1 118B at ternary 1.58-bit — both convert + generate + Opencode agentic pass; expert streaming wired for both). Custom arch — MLX port + loader shipped here (mlx-lm has no native `laguna`); logit-parity 2e-7 vs transformers |
| DeepSeek-V2 / V3 (MLA + MoE) | `deepseek_v2` / `deepseek_v3` / `deepseek_v32` | Yes (SwitchGLU experts) | Tested (V2-Lite: convert + resident + streaming, coherent at 3-bit); V3/V3.2 share the MLA+MoE layout and reuse the config (untested — need ~250 GB disk) |
| Kimi K3 (KDA linear-attn + gated-NoPE MLA, Stable LatentMoE) | `kimi_k3` | Yes (896 experts, top-16 + 2 shared) | Tested (2.8T: mxfp4 source → 931 GB `tq3a-tqTe-down4-g64`, streams from disk on a 512 GB Mac Studio; does **not** fit resident on any Mac). Custom arch — MLX port shipped here (mlx-lm has no native `kimi_k3`); fp32 forward parity ≤ 2.7e-6 vs transformers. Needs `[kimi]` extra |
| Sarvam MoE (Megatron-style fused QKV, DeepSeek-V3 routing) | `sarvam_moe` | Yes | Port tested against fp32 (7122/7122 tensors, layer-1 parity 5.4e-7). Custom arch — MLX port shipped here. **No quantized build published**: 4-bit degenerates on long generations (53% of trials; mlx-lm affine 4-bit 100%) |
| Muse Glimmer (dense VLM: gated attention, sandwich norms, sliding/NoPE mix, ViT-G/14) | `muse_glimmer` | No | Tested (29.8B: `tq4-g64` beats affine 4-bit on PPL — 4.3315 vs 4.3798 — in 25% less space; `tq3-g64` = 12.53 GiB). Structural match verified against the checkpoint, 1436/1436 tensors. Needs **mlx-vlm >= 0.6.12** ([#1838](https://github.com/Blaizzy/mlx-vlm/pull/1838), merged) — see [Muse Glimmer](#muse-glimmer-30b-dense-vlm) |
| DiffusionGemma (block-diffusion MoE, via **mlx-vlm**) | `diffusion_gemma` | Yes (128 experts, top-8) | Tested (26B-A4B: convert + block-diffusion sampler, coherent at 3-bit — [HF](https://huggingface.co/manjunathshiva/diffusiongemma-26B-A4B-it-tq3-g32)). **Experimental**: decode is much slower than native 4-bit until a batched codebook gather-GEMM kernel lands |

### mlx-vlm architectures (multimodal / diffusion)

Architectures that live in [mlx-vlm](https://github.com/Blaizzy/mlx-vlm) rather
than mlx-lm convert and run through dedicated entry points (v0.7.0+):

```bash
pip install "turboquant-mlx-full[vlm]"   # adds mlx-vlm >= 0.6.12

# Convert (vision towers, routers, and known quant-sensitive blocks stay full precision)
python -m turboquant_mlx.convert_vlm \
    --hf-path google/diffusiongemma-26B-A4B-it \
    --mlx-path ./diffusiongemma-26B-A4B-it-tq3-g32 --bits 3 -g 32

# Generate (runs mlx-vlm's sampler — block-diffusion denoising for DiffusionGemma)
python -m turboquant_mlx.generate_vlm \
    --model ./diffusiongemma-26B-A4B-it-tq3-g32 \
    --prompt "Write a short paragraph about the ocean." --max-tokens 256
```

## Project Structure

```
turboquant_mlx/
    config.py                 # TurboQuantConfig
    convert.py                # CLI: HF model -> TurboQuant MLX
    generate.py               # Text generation with TurboQuant models
    evaluate.py               # Perplexity evaluation
    quantize_model.py         # Model traversal & layer replacement
    demo_kv.py                # Streaming generation demo with KV cache compression
    test_kv_cache.py          # KV cache roundtrip + integration tests
    core/
        codebook.py           # Lloyd-Max codebooks for Gaussian
        rotation.py           # Randomized Hadamard rotation
        polar_quantize.py     # Rotate + codebook quantize
        packing.py            # Bit-packing into uint32
        qjl.py                # QJL residual correction
    layers/
        polar_linear.py       # PolarQuantizedLinear (dense)
        polar_switch_linear.py # PolarQuantizedSwitchLinear (MoE)
        polar_kv_cache.py     # TurboQuantKVCache (runtime KV compression)
    kernels/
        polar_qmv.py          # Fused Metal kernel (dense decode)
        polar_gather_qmv.py   # Fused Metal kernel (MoE shared input)
        polar_multi_gather_qmv.py  # Fused Metal kernel (MoE per-expert input)
    integration/
        rotation_configs.py   # Per-architecture rotation configs
    stream/                   # Expert streaming — run MoE models beyond RAM (v0.4.0)
        safetensors_reader.py # Per-expert disk slice reads (os.pread + F_NOCACHE; coalesced ranges)
        streaming_switch.py   # StreamingSwitchLinear + byte-budgeted LRU ExpertCache (+ prefetch/pin)
        loader.py             # load_streaming(): swap experts to streaming after lazy load
        stream_generate.py    # CLI: stream-generate (--cache-budget-gb, --prefetch-ahead, --pin-file)
        calibrate_experts.py  # Routing trace → pin.json (hot experts) + perm.json (co-activation)
        repack_experts.py     # Optional co-activation on-disk relayout (byte-identical)
```

## Citation

```bibtex
@misc{turboquant_mlx,
    title={TurboQuant-MLX: Extreme Weight and KV Cache Compression for Apple Silicon},
    year={2025},
    note={MLX implementation of TurboQuant (Zandieh et al., 2025) for both weight quantization and runtime KV cache compression}
}
```

## License

MIT

## Acknowledgments

- [TurboQuant](https://arxiv.org/abs/2504.19874) — Zandieh, Daliri, Hadian, Mirrokni (2025)
- [MLX](https://github.com/ml-explore/mlx) — Apple Machine Learning Research
- [mlx-lm](https://github.com/ml-explore/mlx-examples) — MLX language model utilities
