Metadata-Version: 2.5
Name: index-tts-2.5-mlx
Version: 0.1.1
Summary: IndexTTS-2.5 voice cloning on Apple Silicon: MLX int8, torch-free, one-click uvx
Project-URL: Homepage, https://huggingface.co/yunfengwang/IndexTTS-2.5-mlx
Project-URL: Source, https://huggingface.co/yunfengwang/IndexTTS-2.5-mlx
Author: yunfengwang
License: Bilibili-IndexTTS
Keywords: apple-silicon,indextts,mlx,tts,voice-cloning
Requires-Python: >=3.10
Requires-Dist: huggingface-hub>=0.24
Requires-Dist: kaldi-native-fbank>=1.18
Requires-Dist: librosa>=0.10
Requires-Dist: mlx>=0.24
Requires-Dist: numpy>=1.24
Requires-Dist: pyyaml>=6.0
Requires-Dist: soundfile>=0.12
Requires-Dist: tiktoken>=0.7
Requires-Dist: transformers>=4.45
Requires-Dist: wetext
Description-Content-Type: text/markdown

# index-tts-2.5-mlx

[IndexTTS-2.5](https://modelscope.cn/models/IndexTeam/IndexTTS-2.5) voice cloning on **Apple Silicon**, rebuilt from PyTorch onto **MLX** with an **int8-quantized GPT** decoder. Torch-free, runs entirely on the unified-memory GPU, and ships as a one-click `uvx` package that auto-downloads the weights from Hugging Face.

- **Voice cloning** from a short reference clip (≤15 s) — Chinese / English / Japanese / Cantonese, including mixed-language text with numbers and abbreviations.
- **Faster than real-time** (RTF ≈ 0.45) and **~2.4× faster than the official PyTorch MPS backend**.
- **Torch-free**: numpy + MLX only. No PyTorch, no ONNX Runtime, no MNN.
- **One self-contained wheel**: the inference core and all MLX model ports are vendored in — `pip install` gives you everything except the weights.

> Hardware: Apple Silicon Mac (M1 or newer), macOS 13+, Python 3.10+. MLX uses the unified-memory GPU.

---

## Features

- **Zero-shot voice cloning** — supply any ≤15 s clean reference; the timbre and speaking style are carried into the output. Build the speaker once and reuse it across unlimited lines.
- **Multilingual + code-switching** — `zh`, `en`, `ja`, `yue`, and mixed text in a single sentence (e.g. `Use the CPU or GPU, 都可以`).
- **Text normalization** — numbers, abbreviations and symbols are read out correctly via `wetext` (e.g. `2025 年` → “二零二五年”, `100 万` → “一百万”). Disable with `--no-normalization`.
- **Rich decoding controls** — greedy or sampling (`top_k` / `top_p` / `temperature` / `seed`), `repetition_penalty`, `duration_factor` (speech rate), and the flow-matching solver knobs (`n_timesteps`, `cfg_rate`).
- **Auto-download** — weights pull from Hugging Face on first run and are cached for reuse; CLI and Python API share the same cache.
- **Timing report** — every `synth` prints load / clone / synth time, RTF, and a per-stage breakdown.
- **Inline special tokens** (experimental) — see below.

### Inline tags / emotion

There are **no `[laugh]` / `[sigh]` / `[猪叫]`-style paralinguistic tags** — square brackets are rewritten by the text normalizer (`[`/`]` → `'`), so `[sigh]` reaches the model as `'sigh'`. The only inline markers that survive tokenization are the uppercase `<|…|>` special tokens inherited from the ASR annotation vocabulary: `<|Laughter|>`, `<|Applause|>`, `<|BGM|>`, `<|HAPPY|>`, `<|SAD|>`, `<|ANGRY|>`, `<|NEUTRAL|>`.

These are **not** documented generative controls — empirically `<|Laughter|>` injects an unpredictable non-speech vocalization rather than a clean laugh. In the upstream model, real emotion control uses separate inputs (`emo_vector`, `emo_audio_prompt`, `emo_text`); this port intentionally drops those and lets the **reference audio carry both timbre and emotion**. So the reliable way to get an expressive read (a sigh, a giggle, an excited tone) is to put that expression *in the reference clip*.

## Install / one-click run

No install needed with [uv](https://docs.astral.sh/uv/):

```bash
uvx index-tts-2.5-mlx synth \
    --ref /path/to/voice.wav \
    --text "大家好, this is IndexTTS running on MLX." \
    --out out.wav
```

The first run downloads the int8 model (~5 GB) from Hugging Face into the standard HF cache (`~/.cache/huggingface`); later runs reuse it. Each `synth` prints a timing report:

```
wrote out.wav
audio       2.83 s
load        2.12 s   (model download + weight load)
clone       1.69 s   (speaker embedding from --ref)
synth       1.32 s
RTF        0.467     (2.14x realtime; <1 = faster than real-time)
stages    gpt=0.35s  codec=0.00s  regulator=0.00s  cfm=0.46s  bigvgan=0.51s
```

(`load` is only slow the very first time, while it downloads. `synth` is the marginal cost per line once warm — reuse one `spk` across lines to skip repeated `clone` work.)

Or install into an environment:

```bash
pip install index-tts-2.5-mlx
```

Pre-download the weights ahead of time:

```bash
uvx index-tts-2.5-mlx download
```

## CLI usage

```
index-tts-2.5-mlx synth --ref REF.wav --text "..." --out out.wav [options]
```

| Option | Default | Description |
|---|---|---|
| `--ref` | (required) | Reference audio to clone (≤15 s, clear speech). |
| `--text` | (required) | Text to synthesize (zh/en/ja/yue, mixed OK). |
| `--out` | `output.wav` | Output WAV path (22050 Hz, int16). |
| `--lang` | `zh` | Language hint: `zh`, `en`, `ja`, `yue`. |
| `--greedy` | off | Greedy decoding (deterministic). |
| `--seed` | random | RNG seed for sampling. |
| `--top-k` / `--top-p` / `--temperature` | `30` / `0.8` / `0.8` | Sampling controls. |
| `--repetition-penalty` | `10.0` | Repetition penalty. |
| `--max-mel-tokens` | `1500` | Max acoustic tokens per segment. |
| `--duration-factor` | `1.0` | Speech-rate multiplier. |
| `--n-timesteps` / `--cfg-rate` | `25` / `0.7` | Flow-matching solver controls. |
| `--model-dir` | auto | Use a local weight dir instead of downloading. |
| `--no-normalization` | off | Disable text normalization. |

## Python API

```python
from index_tts_2_5_mlx import IndexTTS

tts = IndexTTS()                       # auto-downloads weights on first use
sr, pcm = tts.clone(
    "AI 模型在 2025 年处理了 100 万条数据。",
    ref_audio_path="voice.wav",
    out="clone.wav",                   # optional; also returns pcm
    lang="zh",
)

# Reuse one cloned voice across many lines (build the speaker once):
spk = tts.build_speaker("voice.wav")
for i, line in enumerate(["第一句。", "Second sentence.", "第三句。"]):
    sr, pcm = tts.clone(line, ref_audio_path=None, spk=spk, out=f"line{i}.wav")
```

`synthesize(...)` returns the raw int16 PCM array (`numpy`) at `tts.sample_rate` (22050 Hz); `clone(...)` additionally writes a WAV when `out` is given. Use your own reference audio only with permission — see **License**.

## Speed

End-to-end synthesis, warm, mean of 3 runs (Apple M5 Pro). **RTF** = synthesis time ÷ audio duration (lower is better; **<1 = faster than real-time**).

| Backend | fx0 RTF | fx1 RTF | vs PyTorch MPS |
|---|---|---|---|
| PyTorch MPS (official) | 1.17 | 1.11 | 1.0× |
| **MLX fp32** | 0.67 | 0.71 | ~1.7× |
| **MLX int8 (this package)** | **0.47** | **0.45** | **~2.4×** |

Stage breakdown (int8, ~3 s of audio): GPT decode ≈ 0.35 s, flow-matching CFM ≈ 0.48 s, BigVGAN vocoder ≈ 0.56 s. The int8 quantization fuses dequant into the Metal matmul kernels, which is where the GPT autoregressive decode speedup comes from; the other modules are compute-bound and stay fp32.

### MLX vs PyTorch MPS on Apple Silicon

Both MLX and PyTorch-MPS run on the same Metal GPU, so why is MLX ~2.4× faster here?

- **Unified memory, zero copies.** PyTorch-MPS keeps a host/device split and pays CPU↔GPU transfer and synchronization costs across the pipeline's many small ops. MLX targets Apple Silicon's unified memory directly — arrays live in one address space, so the numpy orchestration (frontend, samplers, solvers) and the GPU networks hand off without copies.
- **Fused Metal kernels.** MLX fuses common subgraphs and, crucially, fuses int8 **dequantization into the matmul kernel** (`nn.QuantizedLinear`). The GPT autoregressive decode is memory-bandwidth-bound, so reading 1-byte weights instead of 4-byte ones nearly doubles decode throughput (GPT: ~0.66 s fp32 → ~0.35 s int8).
- **Lighter runtime.** No PyTorch dispatcher / autograd overhead on the inference path.

A note on the baseline: the torch-MPS backend itself diverges from the torch-CPU reference under greedy decoding (MPS matmul rounding), so MPS numbers are a **speed** reference, not an accuracy reference. This MLX port is verified against the deterministic CPU fixtures, not against MPS.

For context, a CPU-only MNN build of the same pipeline lands at RTF ≈ 2.4 (fp32) — and its int8/int4 *weight-only* quantization actually runs *slower* than fp32, because MNN dequantizes weights back to fp32 at load time and computes in fp32. The bottleneck (the CFM DiT) is FLOP-bound, where the GPU is ~12× faster than CPU; no CPU quantization closes that. MLX on the GPU is the right target for Apple Silicon.

## Quality / effect

**Numeric fidelity** — each MLX module matches the PyTorch reference with cosine similarity ≥ 0.999; the full greedy pipeline reproduces the reference acoustic tokens exactly (fx0 73/73, fx1 83/83). Vocoder output matches the reference at mel-spectrogram SNR ≥ 25 dB (inaudible difference).

**Voice cloning** — measured with a CampPlus speaker-embedding cosine between each synthesized clip and its reference vs. an unrelated voice. Every clip scores higher against its own reference, confirming the timbre follows the given reference:

| Clip | sim(own ref) | sim(other voice) | follows ref |
|---|---|---|---|
| clone A ×3 | 0.61–0.73 | 0.29–0.39 | ✓ |
| clone B ×3 | 0.51–0.61 | 0.48–0.50 | ✓ |

**Intelligibility (ASR, Whisper)** — synthesized mixed-language clips with numbers and abbreviations transcribe correctly, e.g. `AI 模型在 2025 年处理了 100 万条数据。` → “AI…2025 年处理了 100 万条数据”, and `Use the CPU or GPU, 都可以` → “用 CPU 或 GPU 都可以”. Cloning quality tracks reference quality: use a clean, natural recording.

## How it works

Pipeline: text frontend (tiktoken + wetext normalization) → w2v-bert semantic features → semantic codec → **int8 GPT** autoregressive acoustic tokens → length regulator → flow-matching **CFM** (DiT) → **BigVGAN** vocoder → 22050 Hz WAV. The orchestration (samplers, solvers, DSP) is numpy; the heavy networks are MLX modules loading the shipped safetensors.

Model weights: [`yunfengwang/IndexTTS-2.5-mlx`](https://huggingface.co/yunfengwang/IndexTTS-2.5-mlx) (int8 GPT + fp32 feed-forward modules).

## PyTorch → MLX conversion notes

For anyone porting a similar model, this is what the conversion involved. Eight sub-networks were re-implemented as MLX `nn.Module`s and verified against PyTorch reference dumps.

**Weight extraction.** Each torch `state_dict` is dumped to safetensors, then remapped to MLX layout:

- Everything runs **channels-last** `[B, T, C]` (vs torch `[B, C, T]`).
- `Conv1d` weight `[out, in, K]` → MLX `[out, K, in]` via `transpose(0, 2, 1)`.
- `ConvTranspose1d` weight `[in, out, K]` → MLX `[out, K, in]` via `transpose(1, 2, 0)`.
- `Conv2d` weight `[out, in, kh, kw]` → MLX `[out, kh, kw, in]`.
- `nn.Linear` weight `[out, in]` copies as-is.
- `weight_norm` (`g`/`v`) is folded into a single dense weight at export time.
- `GroupNorm` needs `pytorch_compatible=True`; `BatchNorm` modules need `.eval()` (MLX defaults to training mode and would otherwise use batch statistics instead of the running stats).

**Gotchas hit during the port:**

- *GPU conv precision.* MLX's GPU `conv_transpose1d` (stride > 1) and `conv1d` with non-multiple-of-16 channels accumulate at reduced precision, which wrecked the BigVGAN vocoder phase. Fixed by rewriting the transpose-conv as a phase-decomposed stride-1 `conv1d` and padding channels to a multiple of 16 in fp32 — lifting vocoder SNR from ~12 dB to ~89 dB against the torch reference.
- *Relative-position attention.* The w2v-bert semantic encoder uses `relative_key` attention: a per-layer `distance_embedding` table (73 buckets = 64 left + 8 right + 1, dim 64), with an additive bias `einsum("bhld,lrd->bhlr", q, emb[clip(r-l,-64,8)+64]) / 8` fed into `mx.fast.scaled_dot_product_attention`. Only 17 of the 24 layers actually execute (`hidden_states[17]`).
- *State in modules.* An `mx.array` can't be a plain `nn.Module` instance attribute; lazy module-level globals are used instead.
- *Sample-vs-mel SNR.* A tiny GPU-vs-CPU CFM difference (mel cos ≈ 0.999991) is amplified by the vocoder into fine phase detail, so raw sample-SNR looks low (~12 dB) while the mel-domain SNR (≈33 dB) is inaudible. Fidelity is therefore gated in the mel domain.

**int8 quantization.** Only the GPT is quantized — it's the decode bottleneck and the only memory-bandwidth-bound module (the CFM/BigVGAN are FLOP-bound, so quant doesn't help them). `nn.quantize(group_size=64, bits=8, class_predicate=Linear)` quantizes the Linear mats while keeping embeddings in fp32. For distribution the quantized weights are **serialized** (packed `weight` + `scales` + `biases`) into `gpt_int8.safetensors` (909 MB vs 2.3 GB fp32); at load time a pre-quantized `Gpt` is built first, then the packed params load straight into it (`load_weights(..., strict=True)`) — no fp32 detour, and the round-trip is bit-identical.

**Verification.** Every module is gated at cosine ≥ 0.999 against a deterministic torch fixture before being wired in; the end-to-end check reproduces the reference greedy acoustic tokens exactly (fx0 73/73, fx1 83/83 — the MPS baseline manages only ~1.4% on fx1).

## License

Code in this package is provided under the same terms as the upstream project. IndexTTS-2.5 model weights are subject to the original **Bilibili IndexTTS** license — see the upstream model card. Use voice cloning responsibly and only with consent from the voice owner.
