Metadata-Version: 2.4
Name: tts-trainer
Version: 0.0.2
Summary: Fine-tune Qwen3-TTS-12Hz-1.7B-Base into a multi-speaker Cantonese CustomVoice model (LoRA on Common Voice yue), merge it and export the GGUF the ggk engine runs — GUI, downloads, resumable pipeline
Author-email: gguf <info@gguf.us>
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
License-File: LICENSE
Requires-Dist: torch>=2.1
Requires-Dist: transformers>=4.57
Requires-Dist: safetensors>=0.4
Requires-Dist: huggingface_hub>=0.23
Requires-Dist: pyarrow>=14
Requires-Dist: numpy>=1.24
Requires-Dist: soundfile>=0.12
Requires-Dist: librosa>=0.10
Requires-Dist: gguf-connector>=3.7.5
Requires-Dist: tqdm>=4.60
Requires-Dist: psutil>=5.9
Project-URL: Repository, https://github.com/gguf-org/ggk

# tts-trainer

Train your own **Cantonese CustomVoice** model for the ggk engine: a LoRA fine-tune
of `Qwen/Qwen3-TTS-12Hz-1.7B-Base` on CC-0 Cantonese speech (Mozilla Common Voice
`yue`, through its Hugging Face parquet mirrors), merged back into the base weights
with the chosen speakers **baked in** as named voices, and exported as the GGUF that
`ggk` runs exactly like `hk-cantonese-v0.0.5` — with as many female and male voices
as the data supports instead of two.

```bash
pip install tts-trainer
tts-trainer                 # opens http://127.0.0.1:8656/ in the browser
```

The GUI runs in your browser against a local backend, in the style of `gguf-trainer`.
Nothing is uploaded: models, datasets and outputs are addressed by filesystem path.

## What it does

The recipe follows the official Qwen3-TTS fine-tune and the
[hobpond/qwen3-tts-cantonese-recipe](https://github.com/hobpond/qwen3-tts-cantonese-recipe)
behind `jmtl/hk-cantonese-cc0-qwen3-tts`, generalized to several voices:

1. **Select** — scan the dataset parquet files (`audio`, `sentence`, `client_id`,
   `gender`, `down_votes`), keep clips of 3–20 s with colloquial Cantonese
   transcripts and no down-votes, and choose the voices: the contributors with
   the most usable clips per self-reported gender (`n_female` + `n_male`, default
   3 + 3), named `yue_f1 … yue_m3`. Contributor ids never leave the project folder.
   Your own recordings can be added as a JSONL (`{"audio", "text", "speaker", "gender"}`);
   each local speaker becomes a voice under its own name.
2. **Encode** — decode, resample to 24 kHz, trim edge silence (speaker-set clips are
   padded back to length), run the 12 Hz speech tokenizer (16 codebooks) and the
   base model's ECAPA speaker encoder; shards go to `shards/`. Each voice's **mean**
   speaker embedding (fp32) is the row that gets baked.
3. **Train: language LoRA** — rank 32 / alpha 64 on the 196 q/k/v/o/gate/up/down
   projections of the talker (never the code predictor), one epoch over every usable
   clip, speaker slot = the clip's own embedding. Loss = codebook-0 cross-entropy with
   the recipe's label-shift fixes (sub-talker weight 0 by default).
4. **Train: speaker LoRA** — the language adapter merged in, a fresh LoRA over the
   voices' clips (3 epochs), speaker slot = the voice mean, i.e. exactly what the
   exported model is conditioned with.
5. **Export** — both adapters merged into the base weights, voice means written at
   `codec_embedding` rows 3000, 3001, …, `tts_model_type: custom_voice` with the
   `spk_id` map, speaker encoder dropped → `export/<name>/` (loads in `qwen_tts`).
   Plus `export/adapter/`: one rank-64 adapter in PEFT layout (the two phases
   block-concatenated, exact), `extra_tensors.safetensors` with the baked rows and
   `apply_adapter.py` — the hk-cantonese release layout, so the voice ships as a
   140 MB adapter if you want.
6. **Convert** — `<name>-f16.gguf` (and `q8_0`, `q4_k`) with qwen3-tts.cpp's converter,
   optionally the codec GGUF too.
7. **Evaluate** — a few Cantonese sentences per voice synthesized with the merged
   checkpoint (runaway / length statistics, playable in the Output tab; CER with an
   optional Qwen3-ASR checkpoint).

```bash
ggk -m yue_voices-f16.gguf --codec-model qwen-tokenizer-12hz-q8_0.gguf \
    --speaker yue_f1 --language chinese -t "今日天氣好好，我哋出去行下街啦。" -o out.wav
```

Keep `--language chinese` (the adapter carries the Cantonese) and split long text at
punctuation, about 45 characters a chunk.

## Materials

* `Qwen/Qwen3-TTS-12Hz-1.7B-Base` (~3.9 GB, Apache-2.0) — the Base model with the
  speaker encoder; CustomVoice / VoiceDesign checkpoints cannot be trained from.
* `Qwen/Qwen3-TTS-Tokenizer-12Hz` (~0.7 GB) — the speech tokenizer.
* Dataset parquet files, e.g. `JackyHoCL/common_voice_22_yue` (Common Voice yue
  22.0, CC-0; ~190k validated clips in 32 files of ~1.4 GB). `files/split` limits
  how many are downloaded; 4 files are enough for a first model, more clips give
  more voices to choose from.

Copies already on disk are linked instead of downloaded again (the directory the
GUI was started from, other projects, `TTS_TRAINER_MATERIALS`); downloads are
detached and resumable.

## Headless

```bash
tts-trainer new      --project ~/tts-trainer/projects/yue_voices
tts-trainer download --project ~/tts-trainer/projects/yue_voices
tts-trainer run      --project ~/tts-trainer/projects/yue_voices      # foreground
tts-trainer start    --project ~/tts-trainer/projects/yue_voices      # detached
tts-trainer stop     --project ~/tts-trainer/projects/yue_voices      # saves, then exits
tts-trainer status   --project ~/tts-trainer/projects/yue_voices
tts-trainer say      --project ~/tts-trainer/projects/yue_voices --speaker yue_m1 --text "唔好意思，請問附近有冇地鐵站？" --out m1.wav
tts-trainer start    --project ... --only export convert eval --force  # re-merge from the adapters
tts-trainer convert  --input export/yue_voices --output yue_voices-q8_0.gguf --type q8_0
tts-trainer convert  --input export/yue_voices --output yue_voices-q4_k.gguf --type q4_k
```

Every stage is idempotent and checkpointed: a reboot, a Stop or a crash resumes at
the next shard / training step. `project.json` holds every setting; epochs of a
running phase can be changed there (or in the GUI) and take effect within a log
interval.

## Hardware

The 1.7B talker trains in bf16 with fp32 LoRA weights: about 6 GB of VRAM at batch
4 (24 GB cards take batch 8–16), ~100 optimizer steps a minute on an RTX 5090 at
batch 4. The default run (30k language clips, one epoch; 6 voices × ≤800 clips,
three epochs) is roughly 7.5k + 3.5k steps. Encoding runs at ~40 clips/s on a GPU.
6 GB cards: batch 1–2 with gradient checkpointing on.

## Notes

* The Qwen3-TTS model code is vendored (Apache-2.0, `tts_trainer/qwen3_tts/`) and
  patched to import under transformers 5.x as well as 4.57; no `qwen-tts` / `peft`
  dependency. Synthesis uses its own decode loop (`synth.py`), not `generate()`.
* GGUF files are written with `gguf-connector` (writer, quant and reader in one
  package, as in `gguf-trainer`); the converters quantize to q8_0 themselves and
  the `gguf` package from llama.cpp is no longer needed.
* `q4_k` is encoded by `tts_trainer/kquant.py`, a numpy port of ggml's reference
  Q4_K quantizer that is byte-identical to `quantize_row_q4_K_ref` (gguf-connector
  up to 3.7.5 only *de*quantizes k-quants). The talker `q4_k` is a Q4_K_M-style
  mix: attention V / FFN down, the codec head and the small code predictor stay
  q8_0, the rest is Q4_K (~36 % of f16). Pure Q4_K made the fine-tuned talker run
  on past its sentence (2 of 8 seeds hit the 60 s cap, 3 more spoke 2-3x too long);
  the mix never hit the cap and stayed in the f16/q8_0 length range on 7 of 8 seeds.
  Expect q4_k to drift more than q8_0 — check lengths in the Output tab before you
  ship it. In the codec every linear/transformer weight is Q4_K and
  the conv kernels (row = kernel size) stay f16; rows that fit no block type are
  written as f16 and logged, never silently.
* Common Voice audio is used under its CC-0 release; the Mozilla Data Collective
  terms forbid re-hosting the clips or identifying contributors — the project keeps
  the mapping voice → contributor in `data/voices.json` only.

## Production repos
- [cantonese-v1-dev](https://huggingface.co/cantonese-dev/cantonese-v1-dev)
- [cantonese-gguf](https://huggingface.co/gguf-org/cantonese-gguf)
