Metadata-Version: 2.4
Name: streaming-vits
Version: 0.3.1
Summary: Streaming inference for VITS/piper TTS — a drop-in replacement for sherpa-onnx-offline-tts that starts playing before synthesis finishes
License: MIT
Keywords: tts,vits,piper,streaming,onnx,sherpa-onnx,speech
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.21
Requires-Dist: onnx>=1.14
Requires-Dist: onnxruntime>=1.15
Provides-Extra: play
Requires-Dist: sounddevice>=0.4; extra == "play"
Provides-Extra: fast-phonemes
Requires-Dist: piper-phonemize>=1.1; extra == "fast-phonemes"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Dynamic: license-file

# streaming-vits

Streaming inference for VITS / [piper](https://github.com/rhasspy/piper) TTS models.

Feed in a whole paragraph, get audio out incrementally — and the result is
**numerically identical** to a full-paragraph forward pass. No text splitting of
any kind: sentences stay whole, so the intra-sentence breaks that cost VITS its
intonation are never needed.

A drop-in replacement for `sherpa-onnx-offline-tts`: same model file, same flags,
same WAV. It just starts producing audio before synthesis finishes.

```diff
- sherpa-onnx-offline-tts \
+ streaming-vits-offline-tts \
      --vits-model=./en_US-libritts_r-medium.onnx \
      --vits-tokens=./tokens.txt \
      --vits-data-dir=./espeak-ng-data \
      --num-threads=4 --sid=0 \
      --output-filename=./test-0.wav \
      "The quick brown fox jumped skillfully over the lazy dog..."
```

```
Elapsed seconds: 1.012
Audio duration: 9.532 s
Real-time factor (RTF): 1.012/9.532 = 0.106
Time to first audio: 141 ms          <- 9.5 s of audio, first sound in 141 ms
```

Add `--play` to hear it live as it generates.

## Why this is possible

VITS is not autoregressive, so streaming looks like it shouldn't work. But
`SynthesizerTrn.infer()` splits cleanly at the monotonic alignment:

| half | what it does | context needed | cost |
|---|---|---|---|
| **frontend** | text encoder → duration predictor → alignment → prior expansion → `z_p` | global, over text | ~15-20% of compute, and **quadratic** in text length |
| **decoder** | flow → HiFi-GAN | **finite receptive field over frames** | the rest, linear in output length |

The frontend being quadratic matters: measured on `en_US-libritts_r-medium`, it
is 16% of total compute for a 7 s utterance, 37% at 59 s, and 68% at 178 s
(scaling exponent 1.89 against the decoder's 1.06). It is cheap at sentence and
paragraph scale and expensive at essay scale, so very long inputs should be cut
at sentence boundaries into segments — see `Speaker` below, which does that
automatically. Chunked decoding is still exact within each segment.

Everything that carries paragraph-level prosody lives in the cheap frontend,
which runs once on the whole input. The expensive half is a plain CNN over the
frame axis — no recurrence, no global attention — so it can be evaluated in
chunks, and with enough overlap context the chunks concatenate *exactly*.

This is the opposite trade-off from sentence chunking. Splitting text throws away
the global context that produces good prosody. Splitting in *frame* space costs
nothing, because the frame-space network cannot see beyond its receptive field
anyway.

One detail makes it exact rather than approximate: both `RandomNormalLike` nodes
(duration noise and prior noise) land in the frontend, so all sampling happens
once, for every frame, before any chunk is decoded. Two overlapping chunk decodes
necessarily see identical noise on the frames they share.

## Install

```bash
pip install streaming-vits

# phonemization backend (one of):
brew install espeak-ng          # macOS
apt install espeak-ng           # debian/ubuntu
pip install piper-phonemize     # faster, no subprocess

pip install 'streaming-vits[play]'   # optional: live playback
```

Point it at any piper voice — the same tarballs sherpa-onnx uses:

```bash
wget https://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models/vits-piper-en_US-libritts_r-medium.tar.bz2
tar xf vits-piper-en_US-libritts_r-medium.tar.bz2
```

The monolithic `.onnx` is split into a frontend and a decoder graph on first use
and cached next to the model. You don't have to do anything.

## Calibrate for your device

The chunk schedule is the whole game for latency, and the right one depends on
how fast the device is. Run this once per device:

```bash
streaming-vits calibrate \
    --vits-model=./en_US-libritts_r-medium.onnx \
    --vits-tokens=./tokens.txt \
    --vits-data-dir=./espeak-ng-data \
    --num-threads=4
```

It measures two different things:

**Margins** are a property of the model's weights — how much context does the
decoder actually need before its output stops changing? Swept empirically,
because it is not guessable: trained weights use far more of their nominal
receptive field than an untrained network suggests.

```
[1/3] receptive field (model property, device independent)
  seed 1234 -- deterministic across runs
  lookahead  rel RMS err
          4     32.835%
          8     11.672%
         12      4.197%
         16      1.047%
         20      0.174%
         24      0.006%
  -> left 34, right 28 (325 ms lookahead)
```

The frontend sampling is pinned to a fixed seed for this sweep (`--seed`), so
repeated calibrations of the same model give the same margins. Normal synthesis
is unaffected and stays stochastic.

**Schedule** is a property of the device. An affine cost model
`decode(n) = α + β·(n + margins)` is fitted, then the schedule that reaches the
speaker soonest without ever starving playback is solved for:

```
[2/3] decode cost on this device
  fit: 1.26 ms/call + 0.442 ms/frame (a frame is 11.61 ms of audio)

  predicted time to first audio  162 ms
  steady-state RTF per chunk     0.052
  survives a device              2.0x slower than this one
  steady-state decode overhead   1.21x (the ramp costs more early)
```

`--safety N` sets how much slower than measured to plan for; the prebuffer is
sized so headroom is a guarantee rather than an observation. Raise it on devices
with contended CPUs.

Margins are reproducible run to run, but the *schedule* is not quite: it depends
on wall-clock timings, which move with machine load. Calibrate on an otherwise
idle device, and treat `--safety` as the thing that absorbs the rest.

The profile is written next to the model and picked up automatically.

## Results

`en_US-libritts_r-medium`, 4 threads, M-series Mac, 15.2 s utterance:

```
non-streaming  0.701s  RTF 0.046   <- you wait this long before any sound

time to first audio  0.163s   (4.3x sooner than non-streaming)
total wall clock     1.301s  RTF 0.085  (1.86x the work)
playback             no underrun
```

Time-to-first-audio for the non-streaming path scales with the length of the
paragraph. For the streamed path it is constant, so the gap widens the more you
ask it to say.

Equivalence, on the real weights:

```
$ streaming-vits verify --vits-model=en_US-libritts_r-medium.onnx ...
  max |streamed - full| = 8.00e-05  OK

$ streaming-vits verify --vits-model=es_MX-claude-high.onnx ...
  max |streamed - full| = 1.79e-05  OK
```

## Commands

| command | what it does |
|---|---|
| `streaming-vits-offline-tts …` | drop-in for `sherpa-onnx-offline-tts` |
| `streaming-vits speak … --play` | synthesise, optionally play live |
| `streaming-vits calibrate …` | benchmark this device, write a profile |
| `streaming-vits bench …` | streamed vs non-streaming latency |
| `streaming-vits verify …` | check chunked output still equals a full decode |
| `streaming-vits info …` | show split graphs and active profile |
| `streaming-vits clear-cache …` | delete cached split graphs |

Accepted-and-ignored sherpa flags (`--vits-lexicon`, `--vits-dict-dir`,
`--tts-rule-fsts`, `--max-num-sentences`) print a note rather than failing, so
existing command lines keep working.

## Migrating from a sherpa-onnx streaming wrapper

If you already drive `sherpa_onnx.OfflineTts` with a per-sentence callback,
split text at a word cap for latency, and pipe chunks to `paplay --raw`, then
`streaming_vits.Speaker` is a one-class swap with the same constructor,
`say()`, `stop()`, `warmup()` and `sample_rate`:

```python
- from your_pipeline import Speaker           # sherpa-onnx based
+ from streaming_vits import Speaker

  spk = Speaker(model="en_US-libritts_r-medium.onnx",
                tokens="tokens.txt",
                data_dir="espeak-ng-data",
                num_threads=2,
                player="paplay",
-               max_words=12,      # accepted and ignored: no longer needed
                speed=1.0)
  spk.warmup()
  spk.say("...")
```

`max_words`/`first_words` are accepted and ignored. Word-cap splitting exists to
buy latency out of a text-space chunker; here the chunking is in frame space, so
you get the latency **and** keep whole sentences. Measured on one 7.5 s sentence,
2 threads, no text splitting either way:

```
sherpa-style (whole sentence, then play):  0.418 s to first audio
streaming-vits (same sentence, unsplit):   0.123 s to first audio   3.4x sooner
```

Text is normally **not** segmented. `max_ids` (default 1600 phonemes, roughly
35 s of speech) is a latency guard for very long input, not a quality setting;
set `max_ids=0` to disable it entirely. Sentences are never broken either way.

`stop()` cancels mid-utterance and kills the player. `Speaker` uses
`PipePlayer`, which needs no PortAudio — usually the difference between working
and not working on a Pi. The CLI exposes the same thing:

```bash
streaming-vits speak --player=paplay --vits-model=... "text"
```

## Python API

```python
from streaming_vits import StreamingTTS

tts = StreamingTTS("en_US-libritts_r-medium.onnx", tokens_path="tokens.txt")

for chunk in tts.stream("Hello there. This plays before it has finished."):
    speaker.write(chunk.audio)      # float32, mono, tts.sample_rate
    print(chunk.index, chunk.frames, chunk.ready_at)

audio = tts.synthesize("Same thing, one array.")
```

## Limits

- **Extra work.** Each chunk re-decodes its margin frames: ~1.2× in steady
  state, more during the ramp. Cached-state streaming convolutions would cut
  this to ~1.05× but are considerably more implementation.
- **TTFA is dominated by the frontend**, not by chunk size. On the numbers above
  the frontend is ~130 ms of a 162 ms budget; smaller first chunks will not help.
  Speeding up the stochastic duration predictor is the next lever.
- **Streaming text *in* is a separate problem.** The duration predictor needs the
  whole utterance before any audio exists. If text arrives from an LLM token
  stream you still chunk at clause level — but you can feed the previous clause
  as context and discard its audio to soften the seam.
- **The bundled phonemizer is pragmatic.** It shells out to `espeak-ng` per
  sentence and re-inserts punctuation, where piper drives libespeak-ng directly
  and gets clause terminators back. Install `piper-phonemize` for the faithful
  path. This does not affect the streaming claim — `verify` compares on identical
  phoneme ids.
- Verified on two piper voices: `en_US-libritts_r-medium` (904 speakers) and
  `es_MX-claude-high` (single speaker, different exporter version). Other VITS
  exports should work if the graph has the same seam; `streaming-vits info` will
  tell you before you rely on it.

## How the split is found

`streaming_vits/graph.py` locates the seam structurally rather than by hardcoded
node names: find the last `RandomNormalLike` (the prior noise), walk forward to
the `Add` that forms `z_p`, then collect whatever the downstream subgraph still
needs from upstream. It also reads the decoder config back off the weights —
piper's `config.json` does not record it, and it varies by voice
(`libritts_r-medium` uses 3 upsample stages, `[8,8,4]`, not the 4 in the
reference VITS config).

Two things are deliberately not read from node names, because those vary between
piper exporter versions even when the weights do not: the upsample config comes
from `dec.ups.N.weight`, and the seam comes from graph topology. Single-speaker
voices have no speaker embedding and no `sid` input, so the decoder takes two
inputs instead of three; that is detected rather than configured.

`research/` holds the original proof of concept, including `verify_streaming.py`,
which proves the same equivalence against the reference PyTorch VITS
implementation independently of ONNX.

## License

MIT
