Metadata-Version: 2.4
Name: dfly-mlx
Version: 0.1.0
Summary: Native DFly speculative decoding for Apple Silicon
License-Expression: Apache-2.0
Project-URL: Repository, https://github.com/will702/dfly-mlx
Project-URL: Model, https://huggingface.co/will702/Qwen3-8B-DFly-MLX
Project-URL: Paper, https://arxiv.org/abs/2607.25852
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: dflash-mlx<0.2,>=0.1.8
Requires-Dist: mlx>=0.31.0
Requires-Dist: mlx-lm>=0.31.0
Provides-Extra: convert
Requires-Dist: huggingface-hub>=0.34; extra == "convert"
Requires-Dist: safetensors>=0.4; extra == "convert"
Requires-Dist: torch>=2.6; extra == "convert"
Provides-Extra: test
Requires-Dist: pytest>=8; extra == "test"
Dynamic: license-file

# dfly-mlx

Native [DFly](https://arxiv.org/abs/2607.25852) speculative decoding for Apple
Silicon. One MLX target, one official drafter, one reproducible benchmark.

DFly runs a five-layer block-parallel draft backbone, fuses five intermediate
target representations differently at every draft layer, then samples the
seven proposals left-to-right through its lightweight hidden-correction head.
The Qwen3-8B target verifies every proposal before it is committed.

> Early implementation: conversion and generation pass on a 16 GB Apple M5,
> but no performance number is published yet. The 4-bit target can choose a
> different greedy token at near-tied logits when run as a verification block
> instead of one token at a time, so the strict parity benchmark currently
> fails closed on affected prompts.

## Requirements

- Apple Silicon Mac
- Python 3.10–3.13 and MLX/MLX-LM 0.31 or newer
- About 7 GB for the 4-bit target plus converted drafter weights; keep roughly
  10 GB free during conversion and leave unified-memory headroom for KV caches

## Install

```sh
python3.11 -m venv .venv
source .venv/bin/activate
pip install -e '.[convert,test]'
```

## Convert the official checkpoint

The upstream Hugging Face repository currently has a bad latest `config.json`:
it describes an 80-layer target instead of Qwen3-8B. The converter therefore
pins the last correct upstream revision, `5712926`, validates its architecture,
and writes MLX safetensors.

```sh
dfly-mlx convert --output ./models/qwen3-8b-dfly-mlx
```

An already-downloaded checkpoint directory can be supplied with `--draft`;
the same metadata and tensor-shape gates still apply.

The output remains an Apache-2.0 AngelSlim model derivative and includes its
model license.

Or download the ready-to-use MLX conversion:

```sh
hf download will702/Qwen3-8B-DFly-MLX \
  --local-dir ./models/qwen3-8b-dfly-mlx
```

## Generate

```sh
dfly-mlx generate \
  --draft ./models/qwen3-8b-dfly-mlx \
  --prompt 'Write a quicksort in Python.' \
  --max-tokens 256
```

The target defaults to `mlx-community/Qwen3-8B-4bit`. Prompts use Qwen3's
no-thinking chat template because the released DFly drafter was trained in
no-thinking mode.

## Reproduce the benchmark

```sh
dfly-mlx bench \
  --draft ./models/qwen3-8b-dfly-mlx \
  --trials 3 \
  --output benchmark.json
```

The benchmark warms both paths, records exact software and hardware metadata,
and refuses to publish a row unless target-only and DFly token IDs match.
Results are medians across trials, not best runs. This is deliberately stricter
than the usual speculative-decoding guarantee, where every emitted proposal is
accepted by the batched target verifier.

## v0.1 boundaries

Qwen3-8B, no-thinking, greedy decoding, and one request at a time. There is no
server or training stack yet. Add those only after the native path demonstrates
a repeatable speedup.

## Attribution

DFly was introduced by Tencent's AngelSpec team. The MLX target execution and
draft/verify machinery come from `dflash-mlx`. See [NOTICE](NOTICE) for precise
attribution and licenses.
