Metadata-Version: 2.4
Name: qmlx-serve
Version: 0.11.0
Summary: Disk KV restore for hybrid MoE models on Apple Silicon. A Qwen-specialised fork of Rapid-MLX that keeps a 122B model warm on a Mac.
Author: Andryo Marzuki, Rapid-MLX contributors
License: MIT AND Apache-2.0
Project-URL: Homepage, https://qmlx.mrzk.io
Project-URL: Documentation, https://github.com/marzukia/qMLX#readme
Project-URL: Repository, https://github.com/marzukia/qMLX
Keywords: llm,mlx,apple-silicon,inference,transformers
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: LICENSE-APACHE
License-File: NOTICE
Requires-Dist: mlx>=0.31.2
Requires-Dist: mlx-lm>=0.31.3
Requires-Dist: transformers<5.13,>=5.0.0
Requires-Dist: tokenizers>=0.19.0
Requires-Dist: huggingface-hub>=0.23.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: tqdm>=4.66.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: requests>=2.28.0
Requires-Dist: tabulate>=0.9.0
Requires-Dist: psutil>=5.9.0
Requires-Dist: fastapi>=0.100.0
Requires-Dist: uvicorn>=0.23.0
Requires-Dist: mcp>=1.0.0
Requires-Dist: jsonschema>=4.0.0
Requires-Dist: argcomplete>=3.6
Requires-Dist: websockets>=12.0
Requires-Dist: openai-harmony>=0.0.8
Provides-Extra: vision
Requires-Dist: mlx-vlm>=0.6.3; extra == "vision"
Requires-Dist: opencv-python>=4.8.0; extra == "vision"
Requires-Dist: torch>=2.3.0; extra == "vision"
Requires-Dist: torchvision>=0.18.0; extra == "vision"
Requires-Dist: pillow>=10.0.0; extra == "vision"
Provides-Extra: dflash
Requires-Dist: mlx-vlm>=0.6.3; extra == "dflash"
Provides-Extra: mtp
Requires-Dist: pillow>=10.0.0; extra == "mtp"
Provides-Extra: embeddings
Requires-Dist: mlx-embeddings>=0.1.0; extra == "embeddings"
Provides-Extra: mirror
Requires-Dist: boto3>=1.34.0; extra == "mirror"
Provides-Extra: chat
Requires-Dist: gradio>=4.0.0; extra == "chat"
Requires-Dist: pytz>=2024.1; extra == "chat"
Provides-Extra: all
Requires-Dist: mlx-vlm>=0.6.3; extra == "all"
Requires-Dist: opencv-python>=4.8.0; extra == "all"
Requires-Dist: torch>=2.3.0; extra == "all"
Requires-Dist: torchvision>=0.18.0; extra == "all"
Requires-Dist: pillow>=10.0.0; extra == "all"
Requires-Dist: gradio>=4.0.0; extra == "all"
Requires-Dist: pytz>=2024.1; extra == "all"
Requires-Dist: mlx-embeddings>=0.1.0; extra == "all"
Provides-Extra: test
Requires-Dist: pytest>=7.0.0; extra == "test"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "test"
Requires-Dist: prometheus_client>=0.16.0; extra == "test"
Requires-Dist: tomli>=2.0.1; python_version < "3.11" and extra == "test"
Requires-Dist: aiohttp>=3.9.0; extra == "test"
Requires-Dist: pillow>=10.0.0; extra == "test"
Requires-Dist: mlx-vlm>=0.6.3; platform_system == "Darwin" and extra == "test"
Requires-Dist: mlx-audio<0.4.4,>=0.2.9; platform_system == "Darwin" and extra == "test"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: prometheus_client>=0.16.0; extra == "dev"
Requires-Dist: tomli>=2.0.1; python_version < "3.11" and extra == "dev"
Requires-Dist: aiohttp>=3.9.0; extra == "dev"
Requires-Dist: pillow>=10.0.0; extra == "dev"
Requires-Dist: mlx-vlm>=0.6.3; platform_system == "Darwin" and extra == "dev"
Requires-Dist: mlx-audio<0.4.4,>=0.2.9; platform_system == "Darwin" and extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Provides-Extra: vllm
Requires-Dist: vllm>=0.4.0; extra == "vllm"
Provides-Extra: guided
Requires-Dist: outlines[mlxlm]>=1.0.0; extra == "guided"
Provides-Extra: audio
Requires-Dist: mlx-audio<0.4.4,>=0.2.9; extra == "audio"
Requires-Dist: sounddevice>=0.4.0; extra == "audio"
Requires-Dist: soundfile>=0.12.0; extra == "audio"
Requires-Dist: scipy>=1.10.0; extra == "audio"
Requires-Dist: numba>=0.57.0; extra == "audio"
Requires-Dist: tiktoken>=0.5.0; extra == "audio"
Requires-Dist: misaki[ja,zh]>=0.5.0; extra == "audio"
Requires-Dist: spacy>=3.7.0; extra == "audio"
Requires-Dist: num2words>=0.5.0; extra == "audio"
Requires-Dist: loguru>=0.7.0; extra == "audio"
Requires-Dist: espeakng-loader>=0.2.0; extra == "audio"
Requires-Dist: phonemizer-fork>=3.3.0; extra == "audio"
Requires-Dist: ordered_set>=4.1.0; extra == "audio"
Requires-Dist: cn2an>=0.5.0; extra == "audio"
Requires-Dist: fugashi>=1.3.0; extra == "audio"
Requires-Dist: unidic-lite>=1.0.0; extra == "audio"
Requires-Dist: jieba>=0.42.0; extra == "audio"
Dynamic: license-file

<div align="center">

[![qMLX](assets/qmlx-light.png#gh-light-mode-only)](https://qmlx.mrzk.io)
[![qMLX](assets/qmlx-dark.png#gh-dark-mode-only)](https://qmlx.mrzk.io)

</div>

<p align="center">
  <strong>Keeping a hybrid 122B warm on a Mac.</strong>
  <br>
  <em>A Qwen-specialised fork of <a href="https://github.com/raullenchai/Rapid-MLX">Rapid-MLX</a> for long-context serving of hybrid MoE models on Apple Silicon.</em>
</p>

<p align="center">
  <a href="https://qmlx.mrzk.io">Website</a> &middot;
  <a href="https://github.com/marzukia/qMLX#readme">Docs</a> &middot;
  <a href="https://charted.mrzk.io">charted</a>
</p>

---

## Why this exists

Qwen3.5-122B-A10B is a hybrid: about 75% of its layers are DeltaNet (recurrent, linear-attention) and 25% are full attention. The recurrent state cannot be rewound to an earlier position, so the standard in-memory prefix cache drops every entry that contains those layers. On this model it misses 100% of the time. In a normal window we measured zero in-memory hits against 109 disk hits.

So the only thing that keeps the model warm is disk KV restore: checkpoint the attention KV to SSD, page it back on the next turn. It is not a fallback here, it is the entire cache. qMLX is that subsystem built properly, plus the fixes needed to make it hold on real agentic-coding traffic.

The result: a follow-up question on a 130,000-token conversation goes from a multi-minute cold prefill to a sub-second restore. Measured on an M3 Ultra, a repeated 32k prompt drops from 88 seconds of prefill to 0.64 seconds, 137x faster.

## What is in it

- **Disk KV checkpoint and restore** for hybrid recurrent + attention MoE caches, with int4 checkpoints dequantised on restore.
- **Matchable-aware disk-cap eviction** so the checkpoint the next turn needs never gets evicted by unmatchable interval writes.
- **Honest, phase-split metrics**: real decode tok/s (decode window only), real prefill throughput (excludes cached tokens), disk-restore hit rate, TTFT. No amortised (prompt+gen)/wall throughput lie.
- **Live divergence logging** that pinpoints the exact token where a prefix-cache match broke, so this class of bug is diagnosable in minutes.

## Status

Alpha. It runs one model (Qwen3.5-122B-A10B) on one class of machine (M3 Ultra, 96GB+ unified). Qwen-first, and honest about what is built and what is not. Decode slows gradually with context because the dense-attention layers re-read a growing KV each token, but there is no cliff: it stays usable well past 100k tokens on this hardware. Windowed attention to flatten that curve further is on the roadmap.

## Install

From source (this is a fork, not published to PyPI under its own name):

```sh
git clone https://github.com/marzukia/qMLX.git
cd qMLX
pip install -e .
```

## Serving

```sh
rapid-mlx serve mlx-community/Qwen3.5-122B-A10B-4bit \
  --text-only --host 0.0.0.0 --port 8095 --max-num-seqs 1 \
  --enable-prefix-cache --prefix-cache-index radix \
  --enable-disk-kv-restore --kv-disk-checkpoint-interval 256
```

Drop-in OpenAI / Anthropic API, same as upstream. `--text-only` is required: the vision path is incompatible with the hybrid continuous-batching that the cache work depends on.

## Credit

Forked from [raullenchai/Rapid-MLX](https://github.com/raullenchai/Rapid-MLX). The base engine, the OpenAI/Anthropic API surface, and the MLX serving path are theirs. qMLX adds the hybrid-aware disk restore, the eviction and metrics work, and the Qwen specialisation. We went a different direction on hybrid attention, too fundamental to reconcile in a PR, hence the fork.

## Notes

The package is still imported as `vllm_mlx` and the CLI is still `rapid-mlx`; those are kept as functional identifiers for compatibility. `rapid_mlx_*` metric names, `RAPID_MLX_*` env vars, and the `~/.cache/rapid-mlx/` cache path are unchanged for the same reason.
