Metadata-Version: 2.4
Name: voxweave
Version: 0.17.0
Summary: BGM-robust subtitles for anime / film / clips: vocal separation + song-skip so ASR doesn't hallucinate on background music, OP/ED, or insert songs. Local-first Qwen3 ASR + forced alignment + edit-and-resync, CJK-aware.
Author: Hao Li
License-Expression: MIT
Keywords: subtitles,asr,forced-alignment,whisper,qwen,vtt
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Environment :: GPU :: NVIDIA CUDA
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: silero-vad>=5
Requires-Dist: soundfile>=0.12
Requires-Dist: numpy<2.5,>=1.26
Requires-Dist: click>=8.1
Requires-Dist: rich>=13
Requires-Dist: rich-click>=1.8
Requires-Dist: huggingface-hub>=0.24
Requires-Dist: einops>=0.8
Requires-Dist: rotary-embedding-torch>=0.6
Requires-Dist: beartype>=0.18
Requires-Dist: librosa>=0.10
Requires-Dist: pyyaml>=6
Requires-Dist: torchaudio>=2.1
Requires-Dist: pyannote-audio<5,>=4
Requires-Dist: ctc-forced-aligner>=1.0.2
Requires-Dist: unidecode>=1.3
Requires-Dist: pysbd>=0.3.4
Requires-Dist: panns-inference>=0.1.0
Requires-Dist: budoux<0.9,>=0.8
Requires-Dist: jieba>=0.42
Requires-Dist: fugashi>=1.3
Requires-Dist: unidic-lite>=1.0.8
Requires-Dist: openai>=1.40
Provides-Extra: cuda
Requires-Dist: qwen-asr>=0.0.4; extra == "cuda"
Requires-Dist: onnxruntime-gpu>=1.20; sys_platform != "darwin" and extra == "cuda"
Requires-Dist: faster-whisper>=1.1; sys_platform != "darwin" and extra == "cuda"
Requires-Dist: huggingface-hub<1.0; extra == "cuda"
Provides-Extra: mps
Requires-Dist: onnxruntime>=1.20; sys_platform == "darwin" and extra == "mps"
Requires-Dist: mlx-audio>=0.4.4; (sys_platform == "darwin" and platform_machine == "arm64") and extra == "mps"
Requires-Dist: mlx-whisper>=0.4; (sys_platform == "darwin" and platform_machine == "arm64") and extra == "mps"
Provides-Extra: diarize
Dynamic: license-file

<div align="center">

<img src="resources/VoxWeave_icon.png" alt="VoxWeave" width="200"/>

# VoxWeave

**BGM-robust subtitles for anime, film, and clips.**

Vocal separation and song-skip so ASR never hallucinates on background music, OP/ED, or
insert songs. Local-first Qwen3 ASR, forced alignment, and edit-and-resync — CJK-aware.

[![CI](https://github.com/hali0515/VoxWeave/actions/workflows/ci.yml/badge.svg)](https://github.com/hali0515/VoxWeave/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/voxweave)](https://pypi.org/project/voxweave/)
![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)
![Python 3.11+](https://img.shields.io/badge/Python-3.11+-blue.svg)
![CUDA cu128](https://img.shields.io/badge/CUDA-cu128-76B900?logo=nvidia&logoColor=white)
![Apple Silicon MLX](https://img.shields.io/badge/Apple_Silicon-MLX-000000?logo=apple&logoColor=white)
[![Buy Me A Coffee](https://img.shields.io/badge/Buy_Me_A_Coffee-FFDD00?logo=buymeacoffee&logoColor=black)](https://buymeacoffee.com/hali0515)

<https://github.com/user-attachments/assets/e75b6dd3-fa37-4afe-89db-b6ee2c28f6bc>

<sub>Sliced clip under heavy BGM · <code>voxweave Test.mp4</code> · Qwen3-ASR-1.7B</sub>

</div>

> [!NOTE]
> **Local-first.** Separation, ASR, and forced alignment all run in-process on your GPU — no
> network endpoints, no audio leaves the machine. Runs on **NVIDIA CUDA** (PyTorch) and on
> **Apple Silicon**, where ASR + alignment use the native **MLX** Qwen3 models. Weights download
> once on first run. (Translation and ASR-correction are the only optional features that call an
> external LLM, and only when you invoke them.)

> [!NOTE]
> **Hardware.** The default pipeline (`Qwen3-ASR-0.6B`, `peak` load strategy) runs in **~8 GB of
> VRAM** — the separator is freed before ASR + alignment load, so peak ≈ `max(stage)`, not their
> sum. `--asr-model qwen3-asr-1.7B` adds roughly **+2 GB** and still fits 8 GB under the default `peak`
> strategy. `load_strategy = "sum"` (concurrent, faster on big cards) makes peak the **sum** of the
> resident models — plan for 12 GB+. On **Apple Silicon** the MLX 8-bit weights roughly halve the
> Qwen footprint (so 1.7B fits comfortably in 16 GB unified memory). `--hybrid` loads a Whisper
> engine alongside Qwen to trade VRAM for accuracy — it does **not** save memory; the only knob that
> lowers it is staying on `0.6B` + the `peak` strategy.

VoxWeave derives from the WhisperX "edit-and-resync" workflow: transcribe once, then edit
the text and re-align it against the original audio for frame-accurate timestamps. Where it
differs is the front end — vocal separation and song-skip keep background music out of the
ASR, and a CJK-aware layout/alignment stack (MMS-300m for Japanese, BudouX/jieba for line
breaks) handles Chinese/Japanese/English as first-class.

## Contents

- [Why VoxWeave](#why-voxweave)
- [Setup](#setup)
- [Quickstart](#quickstart)
- [Usage](#usage)
  - [Transcribe (`voxweave <media>`)](#transcribe)
  - [Label speakers (`speakers`)](#label-speakers)
  - [Re-align after editing (`align`)](#re-align-after-editing)
  - [Re-layout offline (`render`)](#re-layout-offline)
  - [ASR correction (`correct`)](#asr-correction)
  - [Translate (`translate`)](#translate)
  - [Export (`export`)](#export)
  - [Pack soft subtitles (`pack`)](#pack-soft-subtitles)
  - [Burn hard subtitles (`burn`)](#burn-hard-subtitles)
- [The edit-and-resync workflow](#the-edit-and-resync-workflow)
- [Migration notes](MIGRATING.md)
- [How it works](#how-it-works)
- [Configuration](#configuration)
  - [Performance knobs](#performance-knobs)
- [Data contract](#data-contract)
- [Testing](#testing)
- [Support](#support)
- [License](#license)
- [Acknowledgments](#acknowledgments)

## Why VoxWeave

- **BGM removal before ASR.** A Mel-Band Roformer vocal separator (pure torch, full-band
  44.1k) strips music first, so ASR doesn't transcribe lyrics or hallucinate on score.
- **Song-skip.** PANNs detects singing/music on the separated vocals and skips OP/ED and
  insert songs before ASR — on by default, `--no-skip-songs` to keep them.
- **Local Qwen3 ASR + forced alignment.** Text and word-level timestamps in one pass, fully
  on-device — in-process PyTorch on NVIDIA, or the native MLX Qwen3 models on Apple Silicon. A
  Whisper hybrid engine is also available for when you prefer Whisper text (faster-whisper on
  NVIDIA, the native MLX Whisper port on Apple Silicon).
- **Edit-and-resync.** Fix the transcript by hand, then `align` re-derives timestamps from
  the audio — timestamps are _never_ hand-written.
- **CJK-aware.** Japanese aligns with MMS-300m + uroman (zero-OOV, immune to the per-cue
  drift that breaks wav2vec2-xlsr on rare kanji); line breaks use BudouX phrase atoms + jieba.
- **Optional LLM steps.** `correct` cleans up ASR typos/garbled names before alignment;
  `translate` does whole-episode context-aware translation while preserving cue count
  (dual-speaker `-line`/`-line` cues are translated one speaker at a time and re-assembled).
- **Ship the result.** `pack` soft-muxes finished VTTs into the media as titled subtitle
  tracks (instant stream copy); `burn` hardcodes them at constant quality with NVENC /
  VideoToolbox acceleration, matching the source bit depth.

## Setup

Two install variants: **`voxweave[cuda]`** (NVIDIA GPU — Blackwell sm_120 / cu128 by default)
and **`voxweave[mps]`** (Apple Silicon / macOS). Both need `ffmpeg` on PATH.

<details>
<summary><b>Install ffmpeg</b></summary>

```bash
# Ubuntu / Debian
sudo apt update && sudo apt install ffmpeg
# Arch Linux
sudo pacman -S ffmpeg
# macOS (Homebrew)
brew install ffmpeg
```

</details>

<details>
<summary><b>CUDA / PyTorch notes</b></summary>

On the `[cuda]` variant the torch wheel is pinned to the **cu128** build (Blackwell sm_120) and
installed into an isolated `uv` tool venv. The CUDA toolkit does **not** need to be installed
separately — the cu128 wheel bundles the required runtime libraries; only an NVIDIA driver is
required on the host. The `[mps]` variant uses the default PyPI torch wheel (Metal/MPS built in).
Override the torch index per-invocation: `make install TORCH_BACKEND=cpu`.

</details>

**Install from PyPI** (puts the global `voxweave` command on PATH):

```bash
# NVIDIA / Linux:
uv tool install --torch-backend=cu128 "voxweave[cuda]"   # full pipeline + faster-whisper hybrid
# Apple Silicon / macOS:
uv tool install "voxweave[mps]"                          # full pipeline + MLX Whisper hybrid
```

The full local pipeline — vocal separation, ASR, forced alignment (incl. MMS-300m for
Japanese/CJK), layout, song-skip, and speaker-diarization support — plus CJK line-break and
translation — is baked into the **core dependencies**. The variant selects the compute platform
**and the ASR/alignment backend**:

- `[cuda]` (NVIDIA/Linux): the in-process PyTorch Qwen3-ASR + forced aligner (`qwen-asr`), GPU
  onnxruntime (CUDAExecutionProvider for MMS alignment), and the faster-whisper hybrid engine.
- `[mps]` (Apple Silicon/macOS): **ASR** runs on the native MLX Qwen3-ASR from
  [`mlx-audio`](https://github.com/Blaizzy/mlx-audio) (Metal kernels + quantization). **Alignment**
  keeps the same per-language stack as `[cuda]`: English on wav2vec2 CTC (torch, runs on MPS;
  the forced-align DP falls to CPU as torchaudio has no Metal kernel), Japanese/CJK on the ONNX
  MMS aligner (CoreML/CPU — onnxruntime has no Metal provider). Only the Qwen fallback (zh·yue,
  or any CTC failure) is served by the MLX Qwen3-ForcedAligner, since the torch `qwen-asr` aligner
  is absent here. The Whisper hybrid/fusion engines (`--asr-model large-v3`, `--hybrid`) run on the
  native [`mlx-whisper`](https://pypi.org/project/mlx-whisper/) Metal port instead of faster-whisper
  (ctranslate2 has no Metal backend). Vocal separation (MelBandRoformer) + PANNs song-skip stay on
  torch-MPS. `qwen-asr` is excluded because its `transformers==4.57.6` pin conflicts with mlx-audio,
  so `[cuda]` and `[mps]` are mutually exclusive — pick one per host.

Speaker diarization support ships by default on both variants but remains opt-in at runtime.
The default model is `pyannote/speaker-diarization-community-1`, which separates and counts
multiple speakers noticeably better than 3.1. It is gated separately on Hugging Face: before
using `--diarize`, accept the conditions for the model you select, then authenticate once with
`hf auth login` or set `VOXWEAVE_HF_TOKEN` / `HF_TOKEN`. Users who have accepted only the 3.1
gate can keep it with `--diarize-model 3.1`.

Phase-0 measurements used pyannote.audio 4.0.7, the same Japanese 16 kHz test waveform, and an
RTX PRO 4000 24 GB. Peak allocated CUDA memory was identical in that run:

| Short name | Resolved pipeline | Model license | Peak allocated VRAM |
| ---------- | ----------------- | ------------- | ------------------- |
| `community-1` (default) | `pyannote/speaker-diarization-community-1` | CC-BY-4.0 | 2,723.963 MiB |
| `3.1` | `pyannote/speaker-diarization-3.1` | MIT | 2,723.963 MiB |

Community-1 generally improves speaker counting and separation; 3.1 stays selectable per run
for users who have accepted only the existing 3.1 gate.

**From source** (for development or pulling new code):

```bash
make install       # auto-detects the platform: Apple Silicon -> [mps], anything else -> [cuda]
                   # (cu128 torch wheel with an NVIDIA driver, CPU wheel without)
make reinstall     # after pulling new code (same platform detection)
make uninstall
```

Override the detection per invocation: `make install VARIANT=mps` or
`make install TORCH_BACKEND=cpu`.

<details>
<summary><b>Dependencies & what each variant pulls</b></summary>

- The core pulls a pure-torch Mel-Band Roformer vendored in `voxweave.vendor`
  (**no onnx/onnxruntime** —
  `audio-separator` is intentionally avoided because it eagerly imports onnxruntime at the
  top level) + MMS-300m forced aligner (`ctc-forced-aligner`) + layout (`pysbd`) + song-skip
  (`panns-inference`) + CJK break (`budoux` + `jieba`) + translation (`openai`) +
  `pyannote-audio>=4,<5` for opt-in speaker diarization.
- **`[cuda]`** (NVIDIA/Linux): `qwen-asr` + `onnxruntime-gpu` + `faster-whisper`. **`[mps]`**
  (Apple Silicon/macOS): `mlx-audio` + plain `onnxruntime`. Declared **conflicting** in
  `[tool.uv]` (incompatible `transformers` pins), so `uv` resolves each in its own fork — pick one
  per host (`make dev VARIANT=mps` on Apple Silicon).
- **Diarization ships by default but runs only with `--diarize`.** Both supported pyannote
  pipelines are gated independently. Accept the
  [`speaker-diarization-community-1`](https://hf.co/pyannote/speaker-diarization-community-1)
  conditions for the default, or the
  [`speaker-diarization-3.1`](https://hf.co/pyannote/speaker-diarization-3.1) conditions (and
  segmentation-3.0's) before selecting `--diarize-model 3.1`. Then use `hf auth login`,
  `VOXWEAVE_HF_TOKEN`, or `HF_TOKEN`.
- The device is auto-detected at runtime (cuda → mps → cpu); override with `VOXWEAVE_DEVICE`. On
  mps the MLX backend is selected automatically; force it either way with `VOXWEAVE_BACKEND=mlx|torch`.
- **Development**: `make dev` (= `uv sync --extra cuda --dev`; on Apple Silicon use
  `make dev VARIANT=mps` — `[cuda]`/`[mps]` are conflicting extras and can't be synced together).

</details>

## Quickstart

```bash
# Transcribe a video to a timestamped VTT (+ a JSON source of truth)
voxweave episode.mkv

# Opt in to speaker diarization (requires gated-model access; see Setup)
voxweave interview.mkv --diarize

# Stay on the older 3.1 pipeline if that is the gate you have accepted
voxweave interview.mkv --diarize --diarize-model 3.1

# ...edit episode.vtt by hand (fix wording, line breaks)...

# Re-align the edited text against the original audio
voxweave align episode.vtt

# Optionally translate the aligned subtitles to Chinese
voxweave translate episode.vtt --target zh
```

## Usage

All processing commands share numbered workflow steps, a segmented dot bar, and
elapsed time. For example, while translating:

```text
[2/3] translate cues
translate -> zh       [⣿⣿⣿⣿⣿⣿⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀] 128/400 0:00:12
```

The denominator reflects enabled steps, not an estimate of elapsed time. Downloads
and retries stay inside their current step. Cyan means active, green complete,
yellow warning, and red failure; elapsed time is subdued. Unknown totals use a
pulse instead of a fabricated percentage. Burn reports actual encoded frames/time.
Non-terminal runs emit static step lines and occasional encoding updates to stderr,
with no animation; stdout remains reserved for result paths or the speaker-service URL.

A run that finishes cleanly closes with one muted stderr line of wall-clock time per
workflow step, so "where did the time go" is answerable from any run:

```text
timing: inspect source 0.4s | prepare audio 1m17s | detect songs 31.6s | find speech 9.1s | transcribe and align 1m41s | detect shot changes 0.1s | layout subtitles 1.2s | write outputs 0.3s | total 3m41s
```

Anything under a minute is printed as seconds, anything above as `NmSSs`. A step
entered more than once accumulates; `total` is the sum of the listed steps. Failed
runs end in the error panel instead, and a run without a declared plan prints no
summary. A transcription run with `--debug` additionally records the steps finished
by mid-run in `debug/meta.json` under a `timings` key — that file is written while
`transcribe and align` is still open, so it holds a prefix of the printed line, and
only the transcribe path writes it.

### Transcribe

`voxweave transcribe <media>` (or `voxweave <media>`) — separation → song-skip → VAD chunking → ASR + forced alignment →
smart_split → writes `<stem>.vtt` (editable) + `<stem>.json` (word-level timestamp source of
truth). Models load in-process (see `voxweave.backend`); the separator is released from VRAM
before ASR+alignment load, so peak usage is ≈ max(sep, asr) rather than their sum.

```bash
voxweave episode.mkv
voxweave transcribe episode.mkv             # explicit form of the same command
voxweave clip.mp4 --no-separate          # clean speech (podcast/lecture): skip separation
voxweave episode.mkv --asr-model qwen3-asr-1.7B   # larger, more accurate ASR
voxweave episode.mkv --context "Ryland Grace, Astrophage, Hail Mary"   # bias names/terms
```

Unknown command words produce a command error. Bare subtitle or JSON paths are not
transcribed: use `align` for edited VTT, `export` for subtitle format conversion,
or `render` for layout from the sibling JSON. These inputs are never automatically
routed to an in-place editing command. See the [migration notes](MIGRATING.md) for old names.

<details>
<summary><b>Options</b></summary>

| Option                         | Description                                                                                                                                                                                                                                                                                              |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--language`                   | Force language (ISO code or full name); default auto-detect.                                                                                                                                                                                                                                             |
| `--no-separate`                | Skip vocal separation (for clean speech) to save GPU time.                                                                                                                                                                                                                                               |
| `--no-skip-songs`              | Keep lyrics / transcribe purely musical content (song-skip is on by default).                                                                                                                                                                                                                            |
| `-m, --asr-model`               | Local ASR model (default `Qwen3-ASR-0.6B`; `qwen3-asr-1.7B` is more accurate).                                                                                                                                                                                                                           |
| `--context`                    | ASR bias prompt: names/terms likely to appear (comma or newline separated). Bare term lists are auto-framed as `Proper nouns: ...` for Qwen — a bare list actually _regresses_ accuracy ([details](https://github.com/TypeWhisper/typewhisper-mac/issues/321)); prose or pre-framed text passes through. |
| `--hybrid`                     | Dual-ASR fusion: Whisper text + Qwen punctuation. Whisper's error bias is the opposite of Qwen's (it hallucinates rather than omits), so use this when Qwen drops uncertain words.                                                                                                                       |
| `--normalize/--no-normalize`   | Apply loudness normalization (`loudnorm`) to the 16k ASR input — helps when quiet words get dropped; off by default since it also amplifies noise.                                                                                                                                                       |
| `--timestamps/--no-timestamps` | VTT carries word-level timestamps (default on); `--no-timestamps` writes a plain-text editing draft.                                                                                                                                                                                                     |
| `--keep-lyrics`                | Transcribe detected songs instead of skipping them; sung cues are wrapped `♪ ... ♪` (italic in ASS export).                                                                                                                                                                                              |
| `--sdh`                        | Also write `<stem>.sdh.vtt`: PANNs non-speech event tags (`[explosion]`, `[phone ringing]`, ...) in speech-free gaps.                                                                                                                                                                                    |
| `--diarize`                    | Opt in to the default-installed pyannote speaker diarizer: multi-speaker cues split at speaker boundaries; on two-line languages a short exchange becomes a Netflix dual-speaker event (`-line` per speaker). The gated checkpoint requires `VOXWEAVE_HF_TOKEN`, `HF_TOKEN`, config `hf_token`, or a prior `hf auth login`. Speaker turns persist to the sibling JSON, so `voxweave render` replays the formatting without re-running the model. |
| `--diarize-model`              | Select `community-1` (the default), `3.1`, or any full Hugging Face pipeline id. The same setting is available as `VOXWEAVE_DIARIZE_MODEL` or `[diarize].model`; precedence is CLI > env > config > default. |
| `--voiceprints/--no-voiceprints` | Opt in to a voice-biometric centroid sidecar for reviewed cross-episode speaker suggestions. Requires a fresh `--diarize` run and is off by default. Precedence: CLI, `VOXWEAVE_VOICEPRINTS`, `[defaults].voiceprints`, then off. |
| `--min-speakers` / `--max-speakers` | Bound the diarizer's speaker count when you know it (e.g. `--max-speakers 2` for an interview) — the single best lever against over-splitting on noisy material.                                                                                                       |
| `--no-shot-snap`               | Disable shot-change detection/snapping (cue boundaries otherwise land on cuts per the Netflix zone rules).                                                                                                                                                                                               |
| `--vad-mask/--no-vad-mask`     | Suppress CTC emissions outside speech spans during alignment so words cannot park in music/silence (recommended for sparse-dialogue movies with songs; keep off when VAD may misjudge sung/whispered speech). Same as `VOXWEAVE_VAD_EMISSION_MASK=1`.                                                    |
| `--debug`                      | Write intermediate artifacts (full-band / vocals / per-chunk VAD + ASR + alignment) under `<media directory>/cache/<stem>/debug/`.                                                                                                                                              |

The boolean flags (`--separate`, `--skip-songs`, `--normalize`, `--diarize`, `--voiceprints`, `--timestamps`,
`--shot-snap`, `--vad-mask`) can have their defaults set persistently via the `[defaults]`
section of `~/.config/voxweave.conf` — an explicit CLI flag always wins for that run.

</details>

### Label speakers

After a diarized transcription, `voxweave speakers serve <media>` prepares an audition page in
memory, serves it on `127.0.0.1`, and opens it in your browser. The page embeds up to three
clean, non-overlapping speech clips per diarizer id. Listen, enter names, then select **Save**
to write them directly to the episode's speaker mapping. The server runs until Ctrl+C; use
`--no-open` to print the URL without opening a browser, or `--port N` to choose its loopback
port. No audition HTML is written to disk. `voxweave speakers <media>` is the supported
shorthand for `serve`; `--manual` disables voice matching for that session.

```bash
voxweave episode.mkv --diarize
voxweave speakers serve episode.mkv
# Save names in the browser, stop the server with Ctrl+C, then render them
voxweave render episode.json
```

If pyannote merged two people under one diarizer id, select **Split this speaker** on that
card. VoxWeave clusters the id's individual turns and lets you audition both proposed groups
before applying the split. This action requires the episode to have been captured with
`--voiceprints`; it refuses the proposal if the original embedding and audio provenance cannot
be reproduced (for example, when a bound separated-vocals cache is missing or stale). A
confirmed split rewrites `speaker_turns` and the bound voiceprint centroids, keeps a one-level
undo snapshot, and asks you to restart `voxweave speakers serve` to audition and name the new id. Undo
is refused after any rewritten input changes.

Voice matching across episodes is a separate, opt-in layer. Capture centroids with
`--diarize --voiceprints`, review the ordinary empty mapping, then enroll only those
human-entered names into an explicitly selected show store:

```bash
voxweave episode.mkv --diarize --voiceprints
voxweave speakers serve episode.mkv
# Save reviewed names in the browser first
voxweave speakers enroll episode.mkv \
  --voices ./example-show.voices.json --show "Example Show"

# Later episodes: suggestions appear in the served page and a regenerable cache record.
voxweave speakers serve episode-02.mkv --voices ./example-show.voices.json
```

A missing store is created only by `speakers enroll` with both explicit `--voices` and `--show`.
Use `speakers enroll --replace` to replace this episode's prior contribution to that store.
For reuse-only discovery, name it `voxweave.voices.json` beside the media and pass an equal
normalized `--show`; discovery without `--show` reports the store but stays in manual mode.
The shipped matching policy is **suggest-only**: `VOXWEAVE_VOICES_ACCEPT` defaults to `off`,
so stored names appear as review buttons and never become authoritative mapping values.
Even when an operator configures a finite accept threshold, a machine prefill exists only in
the served input; the v1 mapping is still created with empty names and must be reviewed and
saved by the operator.

The versioned mapping is intentionally small:

```json
{"version": 1, "speakers": {"SPEAKER_00": "Aoi"}}
```

`render` renders mapped names as WebVTT voice tags while keeping transcript text and the
sibling JSON clean. Empty or missing names remain unlabeled. An existing mapping is the normal
edit flow: the server re-reads it for the page and replaces it only when you select **Save**.
When burning a mapped SRT, recovered speaker identity is retained in the temporary ASS `Name`
field while the visible dialogue remains prefix-free.

`voxweave speakers purge episode.mkv` removes that episode's voiceprint
artifact and suggestion record from both supported legacy and cache locations. It also removes
the cache-only one-level speaker-split undo snapshot and any legacy audition HTML while holding
the episode transaction lock. It works after the media has been removed and deliberately keeps
the human-edited mapping.

`voxweave speakers list episode.mkv` inspects that episode's speaker turns, reviewed
mapping, and voiceprint state without changing them. Add `--json` for machine-readable
output. It does not list or select a show-level voices store.

### Re-align after editing

`voxweave align <vtt>` — takes the edited VTT text and **re-runs forced alignment against the
original audio**, overwriting the timestamped VTT and updating the JSON. Does not re-run ASR
or touch smart_split. Aligns on separated 16k vocals by default (prevents BGM interference);
prefers the episode's cached `vocals.32k.flac` under the artifact cache, otherwise re-separates
and caches it there. Existing media-adjacent `cache/<stem>.vocals.32k.flac` entries remain a
legacy read/write-back lane. When a voiceprint pair requires source binding, both managed and
legacy vocals caches are accepted only when their matching integrity companion validates.

```bash
voxweave align episode.vtt                 # finds episode.<ext> in the same dir
voxweave align episode.vtt --media original.mkv
voxweave align episode.vtt --no-separate   # align on the original audio (clean sources)
```

<details>
<summary><b>Options</b></summary>

| Option          | Description                                                        |
| --------------- | ------------------------------------------------------------------ |
| `--media`       | Source media path (default: same-name file in the same directory). |
| `--language`    | Force language (ISO code or full name); default: read from JSON.   |
| `--no-separate` | Align on the original audio instead of separated vocals.           |
| `--normalize/--no-normalize` | Apply `loudnorm` to the 16k alignment input.          |
| `--vad-mask/--no-vad-mask`   | Suppress CTC emissions outside the JSON's `vad_speech` spans (see the transcribe option of the same name). |

`--separate`, `--normalize`, and `--vad-mask` also honor the `[defaults]` section of
`~/.config/voxweave.conf` when not passed explicitly.

</details>

### Re-layout offline

`voxweave render <episode>` — re-run deterministic layout from `<stem>.json` without any models
(adjust line width / sentence breaks instantly).

```bash
voxweave render episode.json --max-line-length 14 --max-lines 1
voxweave render episode.json --no-timestamps   # plain-text editing draft
```

Pass the JSON, its VTT sibling, or the media path; each resolves to the same sibling JSON.
This command rewrites the working VTT and JSON. Save or align manual VTT edits before
rendering: the CLI rename does not add a backup or overwrite guard.

### ASR correction

`voxweave correct <vtt>` — optional **pre-align** LLM pass that fixes obvious ASR typos, split
words, and garbled proper nouns, producing a reviewable diff. Conservative substitution only
(no completion/rewrite), gated by a code check that the matched text equals the original
line-for-line. By default writes only an adjacent sidecar `<stem>.asrfix.vtt` plus an audit
JSON in the episode artifact cache — the original VTT is untouched. Use `--apply` to overwrite,
**then run `align`** to reassign timing.

```bash
voxweave correct episode.vtt --glossary names.json   # review the sidecar
voxweave correct episode.vtt --glossary names.json --apply
voxweave align episode.vtt
```

<details>
<summary><b>Options</b></summary>

| Option                         | Description                                                                                                  |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------ |
| `--glossary`                   | Term/name glossary (`.json` → mapping; other → raw prompt). Strongly recommended for ambiguous proper nouns. |
| `--apply`                      | Overwrite the original VTT (default: sidecar only, for review).                                              |
| `--model`                      | Correction model (default `VOXWEAVE_FIX_MODEL` env, `[llm].model` in the config, or `gpt-5.5`; `auto` = the endpoint's only served model). |
| `--base-url` / `--api-key-env` | OpenAI-compatible endpoint + which env var holds the key (defaults from `[llm]` in the config; see [Configuration](#configuration)). |

</details>

### Translate

`voxweave translate <subtitle>` — **after align**, translate each cue with whole-episode
context, preserving cue count, into `<stem>.<target>.<ext>` (the original is left unchanged).
Accepts `.vtt`/`.srt`/`.ass`/`.ssa`; the output mirrors the input format
(`episode.srt` → `episode.zh.srt`).

```bash
voxweave translate episode.vtt --target zh
voxweave translate episode.vtt --target en --context "sci-fi, formal register" --glossary terms.json
voxweave translate downloaded.srt -t zh               # foreign SRT in, SRT out
voxweave translate episode.vtt --target zh --reasoning-effort low
voxweave translate episode.vtt --target zh --concurrency 1   # one whole-episode request
```

Translation accepts any OpenAI-compatible Chat Completions endpoint, including a
self-hosted Qwen server. Configure it once in `~/.config/voxweave.conf`:

```toml
[llm]
base_url = "http://127.0.0.1:8000/v1"
model = "auto"              # or the exact served model ID from /v1/models
api_key_env = ""            # keyless server; otherwise name an environment variable
reasoning_effort = "low"    # translate only; omit to keep the server default
concurrency = 8             # translate windows in flight; 1 = single whole-episode request
window_cues = 100           # cues per window when concurrency > 1
```

By default the episode is translated as bounded windows (`window_cues` cues each)
with several requests in flight (`concurrency`); each window sees the preceding
source cues as context. This trades some cross-window stylistic continuity for
throughput and resilience on a self-hosted server -- `--glossary` and `--context`
are the consistency tools there. `--concurrency 1` sends one whole-episode request
(sequential windows with translated-tail continuity only past 800 cues), the best
choice for a hosted API.

Every response is checked for completeness: an answer that does not finish with
`stop` (length cap, a server aborting its structured-output grammar) is retried,
and the last attempt for that window runs without `response_format` (plain JSON).
Cues still untranslated after the retry stage fail the command and keep the
progress file, so rerunning resumes; `--allow-partial` writes the file anyway with
those cues in source text.

CLI options override environment variables, which override this configuration.
`--model` accepts a served model name without a built-in model list. `auto` requires
the endpoint to advertise exactly one model. For authenticated endpoints, set
`api_key_env = "MY_LLM_KEY"` and put the secret in that environment variable.

`--reasoning-effort` sends the standard top-level `reasoning_effort` field in the
request. Accepted values depend on the served model: for example, Qwen3.8-27B
supports `low`, `medium`, and `xhigh`. Use `--reasoning-effort default` to override
a configured value and omit the field. VoxWeave does not inject thinking template
switches or silently replace an effort rejected by the endpoint. Reasoning output
is kept out of subtitle text. Interrupted translations resume only when the
input, endpoint, resolved model, effort, context, glossary, and target match.

<details>
<summary><b>Options</b></summary>

| Option                         | Description                                                                          |
| ------------------------------ | ------------------------------------------------------------------------------------ |
| `-t, --target`                  | Target language code, written to `<stem>.<target>.<ext>` (default `zh`); use `export --format` to change file formats. |
| `--context`                    | Show/tone context injected into the prompt.                                          |
| `--glossary`                   | Term/name glossary (`.json` → mapping; other → raw prompt).                          |
| `--model`                      | Translation model (default `VOXWEAVE_TRANSLATE_MODEL` env, `[llm].model` in the config, or `gpt-5.5`; `auto` = the endpoint's only served model). |
| `--base-url` / `--api-key-env` | OpenAI-compatible endpoint + which env var holds the key (defaults from `[llm]` in the config; see [Configuration](#configuration)). |
| `--reasoning-effort`           | Model-specific effort. `VOXWEAVE_TRANSLATE_REASONING_EFFORT` > `[llm].reasoning_effort` > endpoint default; `default` explicitly omits the field. |
| `--concurrency N`              | Windows in flight at once (`VOXWEAVE_TRANSLATE_CONCURRENCY` > `[llm].concurrency` > 8). `1` = one whole-episode request with translated-tail continuity. |
| `--window N`                   | Cues per window when `--concurrency` > 1 (`VOXWEAVE_TRANSLATE_WINDOW_CUES` > `[llm].window_cues` > 100). |
| `--allow-partial`              | Write the output even when cues stay untranslated after the retry (they keep their source text). Default: fail and keep the progress file for a resumed rerun. |

</details>

### Export

`voxweave export <subtitle>` — convert between subtitle formats: VTT/SRT/ASS/SSA in,
SRT/ASS/VTT out (written next to the input; the VTT + JSON pair stays the source of truth
for voxweave-produced subtitles). ASS output carries a Default style; lyric cues (`♪ ... ♪`)
render italic. Named VTT cues become `NAME: text` in SRT and use the ASS Dialogue `Name`
field. Foreign SRT/ASS files can be exported to VTT to enter the editing workflow.

```bash
voxweave export episode.vtt --format srt
voxweave export episode.vtt -f srt -f ass
voxweave export downloaded.ass --format vtt        # foreign ASS -> VTT for editing/translate
```

Repeat `-f, --format` to request multiple output formats in one run.

### Pack (soft subtitles)

`voxweave pack <subtitle>...` — remux the source media with the subtitle file(s)
(VTT/SRT/ASS) added as proper subtitle tracks. Pure stream copy (instant, lossless,
reversible); each track is titled `VoxWeave <Language>` with the container language tag
taken from the filename (`episode.zh.vtt` → `chi` / "VoxWeave Chinese"), and the first
packed track is flagged default so players select it. ASS inputs keep their styling in
mkv targets (mp4/webm store text-only codecs, so styling is dropped there).

```bash
voxweave pack episode.zh.vtt                    # finds episode.<ext>, keeps its container
voxweave pack episode.zh.vtt episode.ja.vtt     # several tracks at once
voxweave pack episode.zh.vtt --container mp4    # mov_text in mp4 (image subs are dropped)
voxweave pack episode.zh.vtt --media other.mkv -o out.mkv
```

mkv targets keep every source stream (including attachments); mp4/webm targets keep
video+audio and existing _text_ subtitle tracks only. HEVC video muxed into mp4 is tagged
`hvc1` for Apple players.

### Burn (hard subtitles)

`voxweave burn <subtitle>` — render the subtitles (VTT/SRT/ASS) into the pixels and write a
clean file with **all subtitle tracks removed**. For VTT/SRT input a styled ASS is generated
at the actual frame size (same look as `export`, lyric cues italic); ASS/SSA input goes to
libass as-is, keeping its own styling. The video is re-encoded at constant quality with
hardware acceleration when available: **NVENC** on NVIDIA, **VideoToolbox** on macOS,
libx264/libx265/libsvt-av1 software fallback. Audio is stream-copied (mp4 targets re-encode
mp4-incompatible codecs to AAC).

```bash
voxweave burn episode.zh.vtt                          # hevc, auto hw encoder, -> episode.mp4
voxweave burn episode.zh.vtt --codec h264             # legacy-device compatibility
voxweave burn episode.zh.vtt --codec av1 --container mkv   # max compression, recent hardware
voxweave burn episode.zh.vtt --quality 20 --font "Noto Sans CJK SC"
```

<details>
<summary>Burn options & encoding policy</summary>

| Option        | Meaning                                                                                                    |
| ------------- | ---------------------------------------------------------------------------------------------------------- |
| `--codec`     | `hevc` (default: 10-bit capable, ~40% smaller than h264, plays everywhere as `hvc1` mp4) / `h264` / `av1`. |
| `--encoder`   | Force a specific ffmpeg encoder (default: auto-probe with a test encode).                                  |
| `--quality`   | Constant quality: NVENC `-cq` / software `-crf` (lower = better); VideoToolbox `-q:v` (higher = better).   |
| `--container` | `mp4` (default, maximum compatibility) or `mkv`.                                                           |
| `--font`      | Subtitle font family (fontconfig resolves fallbacks; e.g. `Noto Sans CJK SC`).                             |
| `--font-size` | Override the default 72-at-1080p scaled size.                                                              |

Bitrate is never targeted: pure constant-quality (`-b:v 0` on NVENC) lets the encoder spend
bits where the content needs them, with no overshoot against the source rate. Output bit
depth follows the source dynamically (8-bit stays 8-bit, 10-bit stays 10-bit; 12-bit is kept
on libx265 and clamped to 10 on NVENC/VideoToolbox/SVT-AV1, which top out there) — except on
h264 paths, which are always 8-bit for player compatibility (NVENC h264 cannot encode 10-bit
at all).

</details>

Progress is rendered with rich: countable stages (demix windows / PANNs batches / per-chunk
ASR+alignment / align per-cue / translate streaming per-line) show a real `x/N` bar with
elapsed time; indeterminate stages (decode / file write) show a pulse bar. `-v/--verbose`
enables DEBUG logging.

## The edit-and-resync workflow

```
voxweave episode.mkv          # 1. transcribe  -> episode.vtt + episode.json
  └─ (optional) correct       # 2. LLM ASR fix -> episode.asrfix.vtt (--apply to commit)
edit episode.vtt by hand      # 3. fix wording / line breaks
voxweave align episode.vtt    # 4. re-derive timestamps from audio (overwrites VTT + JSON)
voxweave translate episode.vtt --target zh   # 5. context-aware translation
voxweave pack episode.zh.vtt             # 6. soft-mux into the media (or burn for hardsubs)
```

Timestamps are **always** derived from the audio by the forced aligner — you never hand-edit
them. Edit the text freely; `align` puts the timing back.

## How it works

| Stage           | What runs                                                                                                                                                        |
| --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Separation**  | Mel-Band Roformer (full-band 44.1k stereo, vendored pure-torch) isolates vocals; downsampled to 16k afterwards.                                                  |
| **Song-skip**   | PANNs (route ii) flags singing/music on the separated vocals before ASR; songs are excised mid-segment with cuts snapped into real silences, and PANNs clean-speech evidence rescues dialogue the waveform VAD under-scores. |
| **Chunking**    | Silero VAD splits speech into ≤120s chunks (longer risks ASR repetition-loop collapse).                                                                          |
| **ASR + align** | Qwen3-ASR (default, text + units in one pass) / Whisper hybrid (faster-whisper on cuda, mlx-whisper on mps) / dual-ASR fusion — the pipeline is engine-agnostic. |
| **Alignment**   | `ja` → MMS-300m + uroman, `en` → wav2vec2-LV60K CTC (both full-file single pass, WhisperX-gold); `zh`·`yue` → Qwen. During transcribe the pass is cropped to the transcribed envelope and excised songs are muted, so untranscribed music can never host stretched words. |
| **Layout**      | gap-aware `smart_split`: word-level gaps + BudouX phrase atoms + line-length, on a shared timeline forked per language.                                          |

## Configuration

Precedence: **CLI flag > env var > `~/.config/voxweave.conf` > built-in default.** A commented
default config is written on first run (migrated automatically from a pre-rename `qsub.conf`).

<details>
<summary><b>Environment variables</b></summary>

**Models**

- `VOXWEAVE_ASR_MODEL` (default `Qwen/Qwen3-ASR-0.6B`; same as `--asr-model`)
- `VOXWEAVE_ALIGNER_MODEL` (default `Qwen/Qwen3-ForcedAligner-0.6B`)
- `VOXWEAVE_DIARIZE_MODEL` (default `pyannote/speaker-diarization-community-1`; short names `3.1`
  and `community-1`, or any full Hugging Face pipeline id; same as `--diarize-model`)
- `VOXWEAVE_TRANSLATE_MODEL` / `VOXWEAVE_FIX_MODEL` (default `[llm].model` in the config, else
  `gpt-5.5`; same as `--model` on `translate` / `correct`; `auto` = the endpoint's only served model)
- `OPENAI_BASE_URL` (default `[llm].base_url` in the config, else api.openai.com; same as `--base-url`)
- `VOXWEAVE_TRANSLATE_REASONING_EFFORT` (default `[llm].reasoning_effort`, else the endpoint default;
  same as `translate --reasoning-effort`; `default` leaves the request field unset)
- `VOXWEAVE_TRANSLATE_CONCURRENCY` / `VOXWEAVE_TRANSLATE_WINDOW_CUES` (default `[llm].concurrency` /
  `[llm].window_cues`, else 8 / 100; same as `translate --concurrency` / `--window`)
- `VOXWEAVE_DEVICE` (default: auto-detect `cuda:0` → `mps` → `cpu`)
- `VOXWEAVE_BACKEND` (`mlx` | `torch`; default: `mlx` on mps, else `torch`) — picks the ASR/alignment backend
- `VOXWEAVE_HF_TOKEN` / `HF_TOKEN` — authentication for gated models, including both pyannote
  diarizers; alternatively authenticate once with `hf auth login`
- `VOXWEAVE_OFFLINE` (`1` to enable) — once all models are cached, sets `HF_HUB_OFFLINE`/`TRANSFORMERS_OFFLINE` so loading skips the per-file HEAD revalidation + optional-file probing huggingface_hub/transformers otherwise do on every run (no network on a cache hit). Leave off for the first download.
- `VOXWEAVE_MLX_ASR_REPO` / `VOXWEAVE_MLX_ALIGNER_REPO` / `VOXWEAVE_MLX_WHISPER_REPO` — MLX backend
  repos. By default the ASR repo tracks `--asr-model` size (`--asr-model 1.7b` → `mlx-community/Qwen3-ASR-1.7B-8bit`)
  and the Whisper repo tracks the Whisper size (`--asr-model large-v3` → `mlx-community/whisper-large-v3-mlx`);
  set the matching var to hard-pin a specific quant (e.g. a 4-bit build) regardless of `--asr-model`.

Model weights (torch + MLX) and private media snapshots live under `~/.cache/voxweave/`
(override the root with `VOXWEAVE_CACHE_ROOT`), so a container only needs to bind-mount that
one directory for models. Per-media machine artifacts live beside the media itself under
`<media directory>/cache/<stem>/` (see "Files on disk" below), so they travel with the media.
Model weights are auto-downloaded under the `asr`,
`align`, and `audio` subdirectories on first use. Each model exposes an env override to swap the
HF repo, or to point at an explicit local file (which, if it exists, skips the HF download):

- `VOXWEAVE_SEPARATOR_REPO` / `VOXWEAVE_SEPARATOR_REPO_FILE` (default `KimberleyJSN/melbandroformer` /
  `MelBandRoformer.ckpt`), or `VOXWEAVE_SEPARATOR_CKPT` / `VOXWEAVE_SEPARATOR_CONFIG` for explicit
  weights + matching yaml
- `VOXWEAVE_PANNS_REPO` / `VOXWEAVE_PANNS_REPO_FILE` (default `thelou1s/panns-inference` /
  `Cnn14_mAP=0.431.pth`), or `VOXWEAVE_PANNS_CKPT` for an explicit checkpoint (song-skip CNN)
- `VOXWEAVE_MMS_REPO` / `VOXWEAVE_MMS_REPO_FILE` (default `deskpai/ctc_forced_aligner` /
  `04ac86b67129634da93aea76e0147ef3.onnx`), or `VOXWEAVE_MMS_MODEL` for an explicit onnx path
  (Japanese/CJK MMS-300m aligner)

**Tuning**

- `VOXWEAVE_VOICEPRINTS` (`1/0`, `true/false`, `yes/no`, or `on/off`; opt-in capture,
  overridden by the explicit CLI flag)
- `VOXWEAVE_VOICES_ACCEPT` (default `off`; finite `[-1,1]` enables reviewed audition-page prefills)
- `VOXWEAVE_VOICES_SUGGEST` (default `0.45`; minimum similarity shown as a suggestion)
- `VOXWEAVE_VOICES_MARGIN` (default `0.05`; minimum top-two margin for a prefill)

- `VOXWEAVE_MAX_CHUNK_SEC` (default 120; shorter chunks reduce ASR repetition loops on long segments)
- `VOXWEAVE_LOUDNORM` (default `loudnorm=I=-16:TP=-1.5:LRA=11`; the `-af` filter for `--normalize`)
- `VOXWEAVE_MIN_CUE_SEC` (default 0.8; minimum cue display duration in `align`)
- `VOXWEAVE_SNAP_VAD_THRESHOLD` (default 0.25; sensitive VAD used when repositioning
  zero-duration units against the original audio)
- `VOXWEAVE_SONG_CORE_MERGE_SEC` (default 15; song spans within this gap of a long OP/ED
  cluster into one song "core" that stops the dialogue edge trim — an isolated brief sting
  farther away is trimmed through instead of anchoring dialogue into the excised song)
- `VOXWEAVE_SPEECH_RESCUE_MIN_S` (default 3; minimum length of a PANNs clean-dialogue
  stretch with no silero coverage to be rescued into the chunk stream — catches dialogue
  silero under-scores, e.g. theatrical delivery)
- `VOXWEAVE_CTC_ENVELOPE_PAD_SEC` (default 2; lead-in/out pad when the full-file alignment
  pass is cropped to the transcribed chunk envelope during transcribe, keeping a skipped
  leading/trailing song out of the aligner's waveform)

**Throughput (opt-in; see [Performance knobs](#performance-knobs))**

- `VOXWEAVE_SEP_BATCH` / `VOXWEAVE_CTC_BATCH` / `VOXWEAVE_MMS_BATCH` / `VOXWEAVE_ASR_BATCH`
  (defaults 1 / 1 / 4 / 1; same as `[batch].separate` / `.ctc` / `.mms` / `.asr`) — windows or
  chunks per GPU forward pass. Values below 1 are clamped to 1; a non-integer value is ignored
  and the next source in the precedence chain applies
- `VOXWEAVE_ASR_BATCH_MIN_CPS` (default 0.5) / `VOXWEAVE_ASR_BATCH_MIN_CHECK_SEC` (default 2.0)
  — the qwen-asr #207 guard on the batched ASR path: a batched result with fewer than `MIN_CPS`
  alphanumeric characters per second of audio, for a chunk of at least `MIN_CHECK_SEC`, is
  rejected and that chunk is re-run alone. Shorter chunks are exempt (a cough legitimately
  transcribes to nothing)
- `VOXWEAVE_SEP_AUTOCAST` (`off` (default) | `bf16` | `fp16`; same as `[separate].autocast`) —
  mixed precision for the vocal-separation forward pass. CUDA only; ignored on CPU/MPS. An
  unrecognized value warns once and falls back to `off` rather than to the config file

</details>

<details>
<summary><b>Config file (<code>~/.config/voxweave.conf</code>, TOML)</b></summary>

Every key below is optional — delete a line to fall back to its built-in default. The values
shown are a usable starting point, not the defaults (the auto-written template has everything
commented out).

```toml
# ~/.config/voxweave.conf  —  TOML
# Precedence: CLI flag > env var > this file > built-in default.

# Default ASR model (= --asr-model). Short name (qwen3-asr-0.6b | qwen3-asr-1.7b) or full HF id.
# Special value "hybrid" (= --hybrid) -> dual-ASR fusion (whisper text + Qwen punctuation).
asr_model = "Qwen/Qwen3-ASR-1.7B"        # built-in default: Qwen/Qwen3-ASR-0.6B

# Model load strategy:
#   "peak" (default) — serial peak-shaving: all-chunk ASR -> release -> all-chunk align;
#                      ASR and aligner never co-reside, peak VRAM = max(models). Works on 8 GB.
#   "sum"            — concurrent per-chunk ASR+align; peak VRAM = sum(models), but skips two
#                      model swap round-trips (faster on large-VRAM cards).
load_strategy = "sum"

# Inference batch sizes: windows per GPU forward (env: VOXWEAVE_SEP_BATCH / VOXWEAVE_CTC_BATCH /
# VOXWEAVE_MMS_BATCH / VOXWEAVE_ASR_BATCH). On an 8 GB-class card batch=1 already saturates
# compute — measured no speedup at 2/4, just ~+0.8 GiB VRAM per extra separation window — so the
# defaults stay at 1. Only worth raising on much wider GPUs, and only after measuring.
[batch]
separate = 1                             # vocal separation (MelBandRoformer) 8s windows
ctc      = 1                             # wav2vec2 CTC emission 30s windows (en aligner)
mms      = 4                             # MMS-300m emission batch (ja aligner)
asr      = 1                             # Qwen3-ASR chunks per decode call; 1 = the per-chunk call.
                                         # Opt-in: faster but its transcripts differ from batch 1 —
                                         # see "Performance knobs" below

# Vocal separation numerics (= env VOXWEAVE_SEP_AUTOCAST). autocast wraps only the model
# forward; the overlap-add accumulation stays fp32. CUDA only (ignored on CPU / MPS).
#   "off" (default) — fp32 forward, the reference output, byte-identical run to run.
#   "bf16" | "fp16" — mixed-precision forward: faster and slightly leaner, at the cost of tiny
#                     waveform differences in the stem. See "Performance knobs" below.
[separate]
autocast = "off"

# Diarization pipeline (= --diarize-model / env VOXWEAVE_DIARIZE_MODEL).
# Values: "community-1" (built-in default), "3.1", or any full Hugging Face pipeline id.
# Voiceprint stores are per-model: centroids captured under one pipeline do not match under
# the other (different embedding space), so switching models starts a fresh voiceprint store.
[diarize]
model = "community-1"

# Default on/off for the boolean pipeline flags. An explicit CLI flag always wins
# (e.g. separate = false here, --separate on the command line for one run).
[defaults]
separate   = true                        # vocal separation before ASR/alignment (--separate/--no-separate)
skip_songs = true                        # PANNs music detection + skip before ASR (--skip-songs/--no-skip-songs)
normalize  = false                       # loudnorm on the 16k input (--normalize/--no-normalize)
diarize    = false                       # pyannote speaker diarization (--diarize/--no-diarize; gated-model token required)
voiceprints = false                      # opt-in biometric centroid capture; requires diarize
timestamps = true                        # word-level timestamps in the VTT (--timestamps/--no-timestamps)
shot_snap  = true                        # snap cue boundaries onto shot changes (--shot-snap/--no-shot-snap)
vad_mask   = false                       # suppress CTC emissions outside speech (--vad-mask/--no-vad-mask)

# LLM for translate / correct: any OpenAI-compatible chat-completions endpoint.
# Precedence per key: CLI option > env (VOXWEAVE_TRANSLATE_MODEL / VOXWEAVE_FIX_MODEL,
# OPENAI_BASE_URL) > this section > built-in (gpt-5.5 on api.openai.com, key from OPENAI_API_KEY).
[llm]
model = "auto"                           # or a model name; "auto" = the endpoint's only served model
base_url = "http://127.0.0.1:8000/v1"    # e.g. a local vLLM; remove for api.openai.com
api_key_env = ""                         # "" = keyless endpoint; else the env var holding the key
# reasoning_effort = "low"               # translate only; accepted values depend on the served model
# concurrency = 8                        # translate windows in flight; 1 = single whole-episode request
# window_cues = 100                      # cues per window when concurrency > 1

# dual-ASR fusion sub-models — only consulted when running with --hybrid.
[fusion]
whisper = "large-v3-turbo"               # Whisper size: large-v3 (best) | large-v3-turbo (~5x faster); faster-whisper on cuda, mlx-whisper on mps
qwen    = "Qwen/Qwen3-ASR-1.7B"          # punctuation model; must emit punctuation -> 1.7B, not 0.6B

# Per-language forced-alignment model. Key = ISO-639-1 code; unlisted languages use Qwen3-ForcedAligner.
# Values:
#   "mms"   — MMS-300m + uroman, full-file single pass (immune to per-cue drift; the gold standard).
#   HF id   — wav2vec2 CTC via HF transformers; weights land in ~/.cache/voxweave/align (per-cue crop).
#   bundle  — torchaudio bundle name, e.g. "WAV2VEC2_ASR_LARGE_LV60K_960H" (same model, cached in ~/.cache/torch).
#   ""      — explicitly fall back to Qwen for that language.
[align]
en = "facebook/wav2vec2-large-960h-lv60-self"  # English: LV60K-self CTC, per-cue crop (HF hub)
ja = "mms"                                      # Japanese: MMS-300m + uroman full-file (= whisperx fork align_ctc)
# zh  = "mms"                                   # Chinese can also use MMS; default is Qwen (native CJK char-level)
# yue = ""                                      # force Qwen for Cantonese
```

</details>

### Performance knobs

Two GPU throughput settings are **opt-in and off by default**, because both change the
output. They are worth enabling only on a wide GPU, and only if you have re-checked the
result on your own material — there is no truth ruler in the pipeline that can tell you
whether the changed output is better or worse.

| Knob | Default | Measured on an RTX PRO 4000 (24 GB) | What changes |
| --- | --- | --- | --- |
| `[batch].asr` / `VOXWEAVE_ASR_BATCH` | `1` (per-chunk call) | 4 → 1.34x faster at 6.4 GiB peak; 8 → 1.49x at 8.9 GiB (Qwen3-ASR-1.7B, greedy, 24-min episode) | Transcripts drift ~1.5% CER from batch 1 — scattered small edits from bf16 batched kernels, no chunk lost |
| `[separate].autocast` / `VOXWEAVE_SEP_AUTOCAST` | `off` (fp32) | `bf16` → 1.35x faster separation, peak VRAM 1.69 → 1.57 GiB | The vocal stem differs slightly (~52 dB SNR against the fp32 stem), and the ASR run on it drifts ~2.3% CER |

Neither has a CLI flag; precedence for both is env var > config file > built-in default.
`[batch].asr` applies to the torch Qwen engine only — Whisper and the Apple Silicon MLX
adapter take one chunk per ASR call whatever it says — and autocast applies to CUDA only,
being ignored on CPU/MPS. Raising `[batch].separate` above 1 bought no speedup on the same
GPU (the separator is already compute-bound), so it stays at 1 as well.

Batched ASR is guarded against qwen-asr #207, where a mixed-length batch can corrupt its
shorter item into a lone `!`: chunks are grouped by duration to keep each batch's lengths
close, and any batched result that comes back implausibly empty for its chunk's duration is
re-run alone through the per-chunk call (thresholds: `VOXWEAVE_ASR_BATCH_MIN_CPS`,
`VOXWEAVE_ASR_BATCH_MIN_CHECK_SEC`). A batch whose call raises is likewise redone chunk by
chunk, so one poisoned chunk degrades alone.

Shot-change detection needs no setting: it is a CPU-only ffmpeg pass that now starts before
transcription and is joined at its workflow step, so it overlaps the GPU stages and the
`detect shot changes` entry in the timing line is normally ~0s.

## Data contract

Each input keeps its editable delivery set beside the media:

- **`<stem>.json`** — the source of truth: word/character-level segments, language, VAD speech,
  plus optional replay data (`shot_changes`, `sing_spans`, `speaker_turns`) so `render` can
  redo shot snapping, lyric flagging, and speaker formatting without re-running any model.
- **`<stem>.vtt`** — editable subtitles. By default cues carry word-level timestamps (same
  precision as `align` output, ready to use); `--no-timestamps` writes a plain-text editing
  draft for hand-correction, which `align` re-times.
- **Subtitle-family deliverables** — translated `.vtt`/`.srt`/`.ass` files and derived files
  such as `<stem>.sdh.vtt` and `<stem>.asrfix.vtt` also stay beside the media.

Other episode state is stored beside the media, under `<media directory>/cache/<stem>/`.
For `episode.mkv`, the layout is:

```text
cache/episode/
├── source.json                    # source file-name claim (never an absolute path)
├── episode.episode.lock           # episode transaction lock
├── .episode-domain.lock           # same-directory stem publication lock
├── speakers.json                  # reviewed diarizer-id-to-name mapping
├── speakers.suggest.json          # regenerable match suggestions
├── voiceprints.json               # optional biometric centroids
├── speaker-split.undo.json        # guarded one-level split undo
├── vocals.32k.flac                # separated-vocals cache
├── vocals.32k.flac.meta.json      # source-bound integrity companion, when required
├── vocals.32k.flac.lock           # vocals-cache lock
├── episode.zh.progress.json       # interrupted translation resume state
├── episode.align-evidence.json    # durable alignment evidence
├── episode.asrfix.json            # correction audit
└── debug/                         # optional --debug bundle
```

Because the claim records only the source file name, moving or renaming the media
*directory* (local disk to NAS, season folder reorganizations) keeps every artifact valid —
the cache travels with the media. The marker normally claims the media file; for a supported
standalone subtitle command with no discoverable sibling media, it claims that input file
instead. If another same-stem file in the same directory already owns the plain stem
(`episode.mkv` next to `episode.mp3`), VoxWeave uses `<stem>--<sha1-of-file-name-first-8>/`
for the second claim.

Existing adjacent machine sidecars remain compatible: when an adjacent
`<stem>.speakers.json`, `<stem>.speakers.suggest.json`, `<stem>.voiceprints.json`, translation
progress file, alignment-evidence file, correction audit, or old `cache/` vocals entry exists,
VoxWeave reads it first and writes updates back to that same legacy lane. New artifacts use the
cache. An existing adjacent `<stem>.episode.lock` is joined to the cache lock so old and new
VoxWeave processes still serialize the same episode. The adjacent `<stem>.json` transcript is
not a legacy exception: it remains the current source of truth by design. The
explicit/discovered show-level `voxweave.voices.json` is a user-managed database and also stays
where the user placed it.

### Sensitive and derived speaker data

Treat voiceprint artifacts, speaker-split undo snapshots, show-level voices stores, suggestion
records, in-memory auditions, calibration reports, crash-temporary files, and media snapshots
as sensitive or derived data.
The served audition contains embedded audio but is never saved by VoxWeave; a snapshot contains
private media bytes rather than biometrics. Snapshots live under
`${VOXWEAVE_CACHE_ROOT:-~/.cache/voxweave}/snapshots`, are mode
`0600` in an owner-only directory, and inactive crash residue older than one hour is cleaned on
a later snapshot creation. Active snapshots are lock-protected from that janitor.

The purge command removes only the episode voiceprints, suggestion record, split undo snapshot,
and any legacy audition HTML; it preserves the reviewed mapping, transcript, subtitles, and
unrelated cache state. Remove a show voices store, calibration report, inactive snapshot residue, or
crash-temporary file manually when it is no longer needed. Do not remove an active snapshot.
Backups, filesystem snapshots, synced folders, and manual copies are outside VoxWeave's control
and are not erased by purge; delete them separately according to their retention policy.

Both VTT forms are accepted by `align`. The aligner strips punctuation as a hard constraint;
ASR punctuation is re-injected by time so the final output has correct spacing and breaks
without stray marks.

## Testing

- Unit tests (models mocked, no network): `make test` (= `uv run --extra $(VARIANT) pytest tests/ -v`)
- Lint / format: `make lint`

## Support

If VoxWeave saves you time, you can support development here:

<a href="https://buymeacoffee.com/hali0515"><img src="https://img.shields.io/badge/Buy_Me_A_Coffee-FFDD00?logo=buymeacoffee&logoColor=black" alt="Buy Me A Coffee"/></a>

## License

MIT — see [LICENSE](LICENSE).

## Acknowledgments

- [WhisperX](https://github.com/m-bain/whisperX) — the forced-alignment + edit-and-resync
  workflow this project builds on; the Japanese MMS full-file alignment path is a faithful
  port of its `ctc` align backend.
- [stable-ts](https://github.com/jianfch/stable-ts) — inspiration for timestamp post-processing
  and documentation structure.
- [Qwen3-ASR / Qwen3-ForcedAligner](https://github.com/QwenLM) (Alibaba) — local ASR + aligner.
- [MMS-300m](https://github.com/facebookresearch/fairseq/tree/main/examples/mms) (Meta) via
  [ctc-forced-aligner](https://github.com/MahmoudAshraf97/ctc-forced-aligner) — zero-OOV CJK alignment.
- [Mel-Band Roformer](https://github.com/lucidrains/BS-RoFormer) (lucidrains) +
  [KimberleyJSN](https://huggingface.co/KimberleyJSN/melbandroformer) weights — vocal separation.
- [BudouX](https://github.com/google/budoux), [jieba](https://github.com/fxsjy/jieba),
  [PySBD](https://github.com/nipunsadvilkar/pySBD) — CJK/sentence line-break.
- [PANNs](https://github.com/qiuqiangkong/audioset_tagging_cnn) — song/music detection.
- [Silero VAD](https://github.com/snakers4/silero-vad) — voice activity detection.
