# mlx-optiq

**Run any LLM locally on your Mac. Quantize it, serve it, and code with it.**

**Website:** https://mlx-optiq.com &nbsp;|&nbsp; **Docs:** https://mlx-optiq.com/docs/ &nbsp;|&nbsp; **Models:** https://mlx-optiq.com/models &nbsp;|&nbsp; **Blog:** https://mlx-optiq.com/blog/ &nbsp;|&nbsp; **HF org:** https://huggingface.co/mlx-community

mlx-optiq is the local-LLM stack for Apple Silicon: an optimizing compiler and runtime for MLX that turns a full-precision model into the best version for a given memory and latency budget on your Mac, using per-layer sensitivity measurement instead of uniform 4-bit everywhere. The same signal drives weights, KV cache, LoRA fine-tuning, and runtime adapter swapping.

One `pip install` gives you three ways to work with a local model: the **CLI** (quantize, serve, fine-tune), **OptiQ Lab** (a local web workbench), and **OptiQ Code** (a terminal coding agent that drives your served model).

```bash
pip install mlx-optiq
```

## What it does

- **Mixed-precision weight quantization** that beats uniform 4-bit at the same size. `optiq convert` measures each layer's sensitivity and allocates bits per layer. A `static` method assigns bits by architecture for models too large to measure. [Methods](https://mlx-optiq.com/docs/sensitivity).
- **SSD expert streaming** runs large MoE quants that don't fit in RAM. A 2-bit Qwen3.5-122B-A10B runs on a 36 GB Mac at ~12 GB resident, experts streamed off disk. [How](https://mlx-optiq.com/blog/stream-122b-on-a-mac).
- **Mixed-precision KV cache** for longer context at lower memory. `optiq serve` runs a per-layer KV quant pipeline.
- **One server, two protocols.** `optiq serve` speaks both the OpenAI and Anthropic APIs from one process. Point Claude Code or either SDK at the same local URL.
- **Speculative decoding** via bundled MTP heads or paired drafters (`--mtp`, `--drafter`).
- **Distributed inference across Macs.** `optiq cluster serve` shards a model's layers across two or more Macs over Thunderbolt and exposes one OpenAI endpoint. A 2-bit Qwen3.5-122B-A10B (42.8 GiB) runs fully resident across a 36 GB + 24 GB pair at ~20 tok/s, against 4.9 tok/s streaming experts off SSD on one Mac. [How](https://mlx-optiq.com/docs/cluster).
- **Sensitivity-aware LoRA** (SFT + DPO) and runtime hot-swap adapters.
- **OptiQ Lab** (`pip install "mlx-optiq[lab]"` then `optiq lab`): a local web UI for chat, quantize, fine-tune, and dataset work. [Product](https://mlx-optiq.com/lab).
- **OptiQ Code** (`optiq code`): a terminal coding agent that drives whatever `optiq serve` is serving, offline, engineered for local models (never an empty patch, edit-resilient, stall-proof). [Product](https://mlx-optiq.com/code).

## Quickstart

Every [mlx-optiq quant](https://huggingface.co/mlx-community) loads with stock `mlx-lm`:

```python
from mlx_lm import load, generate
model, tok = load("mlx-community/Qwen3.5-9B-OptiQ-4bit")
print(generate(model, tok, prompt="Hello", max_tokens=50))
```

Installing mlx-optiq unlocks the rest. A few starting points:

```bash
# Serve with the OpenAI + Anthropic API and ~1.4x speculative decode
optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit --mtp

# Run a huge MoE that doesn't fit in RAM (experts stream off SSD)
optiq serve --model mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit --stream-experts

# Quantize a fresh model (exact sensitivity, or fast structural rules for big bases)
optiq convert Qwen/Qwen3.5-9B --target-bpw 5.0 --candidate-bits 4,8
optiq convert <large-moe> --method static --candidate-bits 2,4 --target-bpw 2.5

# Fine-tune with sensitivity-aware LoRA
optiq lora train mlx-community/Qwen3.5-9B-OptiQ-4bit --data ./jsonl_dir --rank 8

# Shard a model across two Macs over Thunderbolt (one OpenAI endpoint)
optiq cluster up                                    # on every Mac
optiq cluster serve --model mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit

# Code with your local model, in your terminal (offline)
optiq code                                          # interactive, in a repo
optiq code -p "Fix the failing test in parser.py"   # headless
```

Full guides for serving, KV-quant, LoRA, MTP and per-family setup are in the [docs](https://mlx-optiq.com/docs/). The [models page](https://mlx-optiq.com/models) lists every quant with its Capability Score, and the [blog](https://mlx-optiq.com/blog/) has the deeper write-ups.

## Requirements

- Apple Silicon (M1 or newer), macOS, Python 3.11+.
- The published quants load with stock `mlx-lm`. Converting and some MoE / multimodal runtime features track `mlx-lm` main; install it from git when a model card asks for it.

## License

MIT for the package. Quantized models follow their base model's license.
