# VoxCPMANE2

Install the VoxCPM2 package:

```bash
uv tool install --python '>=3.10,<3.13' voxcpmane2
```

VoxCPMANE2 is the VoxCPM2 version of [VoxCPMANE](../VoxCPMANE). It provides a
pure numpy/CoreML runtime and FastAPI HTTP server for running VoxCPM2 TTS on
Apple Silicon with Apple Neural Engine acceleration.

The package includes a browser-based playground at `http://localhost:8000/` for
trying voices, streaming, playback, and custom voice creation without writing
client code.

CoreML model assets are downloaded from
[seba/VoxCPM2ANE-Preview](https://huggingface.co/seba/VoxCPM2ANE-Preview)
by default.

- VoxCPM2 text-to-speech generation
- OpenAI-compatible `/v1/audio/speech` endpoint
- Streaming audio generation
- Server-side playback
- Web playground for generation, voice management, streaming, and playback
- Custom cached voices

## Requirements

- macOS on Apple Silicon
- Python `>=3.10,<3.13` is required
- `uv` or `pip`
- CoreML runtime support through `coremltools`
- Optional: `pydub` for `mp3`, `opus`, `ogg`, and `aac` responses

## Installation

Install as a `uv` tool:

```bash
uv tool install --python '>=3.10,<3.13' voxcpmane2
```

Or install into an environment with `uv pip`:

```bash
uv pip install --python '>=3.10,<3.13' -U voxcpmane2
```

Or install with `pip`:

```bash
pip install -U voxcpmane2
```

For editable development from a source checkout, run
`uv pip install --python '>=3.10,<3.13' -e .` from this directory.

## Run The Server

```bash
voxcpmane2-server
```

The server starts on `http://localhost:8000` by default. Open
`http://localhost:8000/` to use the included web playground. It exposes the main
workflows from the browser: generate speech, stream audio, play audio on the
server, create custom voices, and inspect available voices.

## Web Playground

Most users can start with the playground instead of writing API requests. After
starting the server, open `http://localhost:8000/` to generate speech with the
included voices, test streaming behavior, use server-side playback, create
custom voices, and switch between voice modes from the browser.

![VoxCPMANE2 web playground](assets/voxcpmane_frontend.png)

Common options:

```bash
voxcpmane2-server \
  --host 0.0.0.0 \
  --port 8000 \
```

If `--model-dir` is omitted, the server downloads the CoreML model directory
from `--repo-id`. If individual package paths are not supplied, components are
loaded from that downloaded directory. The default repo layout includes
`config.json`, `embed_tokens.npy`, a small `.mlpackage` marker for CoreML repo
recognition, and the runtime packages:
`base_lm_multifunction.mlmodelc`, `residual_lm_fused_multifunction.mlmodelc`,
and the compiled component packages at the repo root. Included voice caches live
under `caches/`.

## Working Modes

`--lm-mode` controls how multifunction LM prefill and decode handles are kept in
memory. The default mode is fixed-length `16`, exposed as `single-length` with
prefill/decode length `16`. Available prefill lengths are `1`, `8`, `16`, `32`,
`64`, and `128`; any of these can be used with `single-length` mode.

| Mode | Behavior | Tradeoff |
| --- | --- | --- |
| `hot-swap` | Warms length `1` decode at startup, unloads it, loads the prefill function while idle, then swaps to length `1` for decode and back after generation. | Lower idle memory, with function load/unload cost around generation. |
| `preload` | Preloads length `1` and the selected prefill size at startup, keeps decode resident, unloads prefill during decode, reloads prefill when idle. | Avoids cold decode load while still reducing decode-time memory. |
| `always-loaded` | Preloads length `1` and the selected prefill size and never unloads either function. | Fastest transitions, highest memory use. |
| `single-length` | Uses only the selected prefill length and restricts LM calls to that function. | Default at length `16`. Good TTFB/RTF tradeoff; decode also uses the selected length instead of length `1`. |

At startup the server also runs 5 synthetic predict calls through each CoreML
package/function that remains loaded. Use `--startup-warmup-repeats 0` to skip
this warmup, or set a different repeat count.

If memory use is not a concern and you want the best steady-state performance,
`preload` is usually the best option by RTF. Otherwise `single-length` tends to
provide the best latency/performance tradeoff. In informal power observations,
`single-length` 8 and 16 add very little power draw, under about 1 W; length 32
is around 1 W higher; larger lengths cost progressively more energy.

Benchmark results below use 2 warmup generations and 3 measured generations per
scenario. Lower TTFB and RTF are better. `Hot-swap` is only reported for
`hot-swap` mode; other modes do not perform a decode function swap.

| Scenario | Mode | TTFB | RTF | Prefill | Hot-swap |
| --- | --- | ---: | ---: | ---: | ---: |
| Text only | `preload` | 0.745s | 0.591 | 0.664s | - |
| Text only | `hot-swap` | 1.095s | 0.672 | 0.322s | 0.721s |
| Text only | `single-length-1` | 2.589s | 0.612 | 2.529s | - |
| Text only | `single-length-8` | 0.474s | 0.664 | 0.413s | - |
| Text only | `single-length-16` | 0.306s | 0.712 | 0.246s | - |
| Text only | `single-length-32` | 0.224s | 0.723 | 0.170s | - |
| Text only | `single-length-64` | 0.197s | 0.809 | 0.143s | - |
| Text only | `single-length-128` | 0.163s | 1.029 | 0.110s | - |
| Text + reference wav | `preload` | 0.745s | 0.593 | 0.652s | - |
| Text + reference wav | `hot-swap` | 1.419s | 0.718 | 0.547s | 0.815s |
| Text + reference wav | `single-length-1` | 3.681s | 0.646 | 3.605s | - |
| Text + reference wav | `single-length-8` | 0.763s | 0.668 | 0.689s | - |
| Text + reference wav | `single-length-16` | 0.538s | 0.686 | 0.473s | - |
| Text + reference wav | `single-length-32` | 0.408s | 0.726 | 0.351s | - |
| Text + reference wav | `single-length-64` | 0.382s | 0.812 | 0.329s | - |
| Text + reference wav | `single-length-128` | 0.338s | 1.022 | 0.285s | - |
| Text + prompt wav + transcript | `preload` | 0.394s | 0.590 | 0.306s | - |
| Text + prompt wav + transcript | `hot-swap` | 1.225s | 0.662 | 0.361s | 0.811s |
| Text + prompt wav + transcript | `single-length-1` | 4.100s | 0.632 | 4.023s | - |
| Text + prompt wav + transcript | `single-length-8` | 0.841s | 0.664 | 0.781s | - |
| Text + prompt wav + transcript | `single-length-16` | 0.583s | 0.699 | 0.527s | - |
| Text + prompt wav + transcript | `single-length-32` | 0.468s | 0.728 | 0.412s | - |
| Text + prompt wav + transcript | `single-length-64` | 0.382s | 0.813 | 0.328s | - |
| Text + prompt wav + transcript | `single-length-128` | 0.341s | 1.045 | 0.287s | - |
| Text + reference + prompt | `preload` | 0.520s | 0.596 | 0.445s | - |
| Text + reference + prompt | `hot-swap` | 1.628s | 0.721 | 0.779s | 0.796s |
| Text + reference + prompt | `single-length-1` | 5.000s | 0.637 | 4.934s | - |
| Text + reference + prompt | `single-length-8` | 1.308s | 0.685 | 1.226s | - |
| Text + reference + prompt | `single-length-16` | 0.749s | 0.688 | 0.686s | - |
| Text + reference + prompt | `single-length-32` | 0.572s | 0.729 | 0.518s | - |
| Text + reference + prompt | `single-length-64` | 0.486s | 0.812 | 0.434s | - |
| Text + reference + prompt | `single-length-128` | 0.484s | 1.083 | 0.431s | - |
| Preset voice reference | `preload` | 1.171s | 0.608 | 1.058s | - |
| Preset voice reference | `hot-swap` | 1.423s | 0.673 | 0.560s | 0.802s |
| Preset voice reference | `single-length-1` | 2.660s | 0.634 | 2.603s | - |
| Preset voice reference | `single-length-8` | 0.503s | 0.682 | 0.445s | - |
| Preset voice reference | `single-length-16` | 0.334s | 0.681 | 0.278s | - |
| Preset voice reference | `single-length-32` | 0.262s | 0.722 | 0.208s | - |
| Preset voice reference | `single-length-64` | 0.230s | 0.809 | 0.177s | - |
| Preset voice reference | `single-length-128` | 0.198s | 1.066 | 0.146s | - |
| Preset voice high similarity | `preload` | 1.182s | 0.592 | 1.045s | - |
| Preset voice high similarity | `hot-swap` | 1.659s | 0.677 | 0.766s | 0.810s |
| Preset voice high similarity | `single-length-1` | 10.845s | 0.622 | 10.728s | - |
| Preset voice high similarity | `single-length-8` | 1.739s | 0.681 | 1.610s | - |
| Preset voice high similarity | `single-length-16` | 0.969s | 0.678 | 0.863s | - |
| Preset voice high similarity | `single-length-32` | 0.648s | 0.724 | 0.552s | - |
| Preset voice high similarity | `single-length-64` | 0.494s | 0.806 | 0.390s | - |
| Preset voice high similarity | `single-length-128` | 0.457s | 1.056 | 0.359s | - |

Examples:

```bash
# Default single-length behavior with length 16.
voxcpmane2-server

# Keep both prefill and decode functions resident.
voxcpmane2-server --lm-mode always-loaded

# Preload decode and prefill, but unload prefill during decode.
voxcpmane2-server --lm-mode preload

# Use only one LM function length.
voxcpmane2-server --lm-mode single-length --lm-prefill-chunk-size 16
```

## Model Path Options


When running with local package paths, included voices are loaded from
`--included-voice-cache-dir` if provided, then from `<model-dir>/caches` if it
exists. If neither is available, the server downloads only `caches/*` from
`--repo-id`, so the bundled voices still appear without downloading the model
packages again.

```bash
voxcpmane2-server \
  --model-dir /path/to/local-models \
  --included-voice-cache-dir /path/to/local-models/caches
```

## API

### Generate Full Audio

```bash
curl http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "voxcpm2",
    "input": "Hello from VoxCPM2 on Apple Neural Engine.",
    "voice": "af_alloy",
    "voice_mode": "reference",
    "response_format": "wav",
    "max_length": 2048,
    "cfg_value": 2.0,
    "inference_timesteps": 10
  }' \
  --output speech.wav
```

When `voice` is set, `voice_mode` controls preset voice conditioning:
`reference` uses the cached reference audio only and has lower first-byte
latency; `reference_plus_prompt` uses the cached reference voice plus a supplied
`prompt_wav_path` and matching `prompt_text`; `high_similarity` uses cached
prompt embeddings, transcript, and VAE decoder warmup context when available.

Supported `response_format` values are `wav`, `flac`, `mp3`, `opus`, `ogg`,
and `aac`. Non-`wav`/`flac` formats require `pydub`.

`max_length` is bounded by the available LM KV cache after prompt prefill. If
the generated length exceeds the cache capacity, the server caps generation to
the remaining cache length.

### Stream Raw PCM16 Audio

```bash
curl http://localhost:8000/v1/audio/speech/stream \
  -H "Content-Type: application/json" \
  -d '{"model":"voxcpm2","input":"Streaming speech."}' \
  --output stream.pcm
```

The stream response is raw PCM16 at the sample rate exposed in the
`X-Sample-Rate` header.

### Other Endpoints

- `GET /`: browser-based web playground for generation and voice management
- `GET /health`: server status
- `GET /voices`: available cached voices
- `POST /v1/voices`: create a cached custom voice
- `DELETE /v1/voices/{voice_name}`: delete a cached custom voice
- `POST /v1/audio/speech/playback`: generate and play on the server audio device
- `POST /v1/audio/speech/cancel`: cancel the current job

## Custom Voices

Included voices are stored in the model `caches/` directory or the directory
provided with `--included-voice-cache-dir`. Custom voices created at runtime are
stored in `--cache-dir` (`~/.cache/ane_tts` by default).

Voice caches use feature-encoder outputs, not full VAE encoder latents:

- `name.embed.npy`: reference voice embeddings
- `name.prompt.embed.npy`: optional continuation prompt embeddings
- `name.prompt.cond.npy`: optional final prompt VAE patch used to seed
  high-similarity continuation decoding
- `name.prompt.decode_context.npy`: optional tail prompt VAE patches used to
  warm the streaming VAE decoder for high-similarity audio continuity

Included voices may also ship LM prefix KV caches as `caches/name.lm_prefix.npz`.
Matching caches restore the base/residual LM prefix on the first request. Missing
or custom voices build a local copy under `--cache-dir` on first use.

Old VAE-latent voice caches are not migrated at startup. A valid cache file is
`(T, hidden_size)` feature-encoder output; if an older cache shape is present,
delete and recreate that custom voice.

You can create a voice through the web UI or API:

```bash
curl http://localhost:8000/v1/voices \
  -H "Content-Type: application/json" \
  -d '{
    "voice_name": "myvoice",
    "reference_wav_path": "/path/to/reference.wav",
    "replace": false
  }'
```

For higher-similarity continuation cloning, include the exact transcript of the
same audio. The server then caches feature embeddings for both the VoxCPM2
reference and the prompt continuation:

```bash
curl http://localhost:8000/v1/voices \
  -H "Content-Type: application/json" \
  -d '{
    "voice_name": "myvoice",
    "reference_wav_path": "/path/to/reference.wav",
    "prompt_text": "The exact transcript of the reference audio.",
    "replace": true
  }'
```

A transcript is optional for VoxCPM2 reference-only cloning, but required for
prompt-continuation cloning.

## Metrics And Tuning

Use `--live-rtf` to print real-time-factor metrics:

```bash
voxcpmane2-server --live-rtf live
voxcpmane2-server --live-rtf final
```

VAE streaming latency can be tuned with:

- `--vae-early-decode-steps`: number of initial AR steps decoded immediately
- `--vae-batch-decode-steps`: number of AR steps to batch after the early phase

Defaults are `--vae-early-decode-steps 16` and `--vae-batch-decode-steps 4`.

## Acknowledgments

- [VoxCPM](https://github.com/OpenBMB/VoxCPM) for the original VoxCPM model family
- [VoxCPMANE](../VoxCPMANE) for the earlier Apple Neural Engine server/runtime
