Metadata-Version: 2.4
Name: streaming-vits
Version: 0.4.2
Summary: Streaming inference for VITS/piper TTS — a drop-in replacement for sherpa-onnx-offline-tts that starts playing before synthesis finishes
License: MIT
Keywords: tts,vits,piper,streaming,onnx,sherpa-onnx,speech
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.21
Requires-Dist: onnx>=1.14
Requires-Dist: onnxruntime>=1.15
Provides-Extra: play
Requires-Dist: sounddevice>=0.4; extra == "play"
Provides-Extra: export
Requires-Dist: torch>=2.0; extra == "export"
Provides-Extra: fast-phonemes
Requires-Dist: piper-phonemize>=1.1; extra == "fast-phonemes"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Dynamic: license-file

# streaming-vits

Streaming inference for VITS / [piper](https://github.com/rhasspy/piper) TTS models.

Feed in a whole paragraph, get audio out incrementally — and the result is
**numerically identical** to a full-paragraph forward pass. No text splitting of
any kind: sentences stay whole, so the intra-sentence breaks that cost VITS its
intonation are never needed.

A drop-in replacement for `sherpa-onnx-offline-tts`: same model file, same flags,
same WAV. It just starts producing audio before synthesis finishes.

```diff
- sherpa-onnx-offline-tts \
+ streaming-vits-offline-tts \
      --vits-model=./en_US-libritts_r-medium.onnx \
      --vits-tokens=./tokens.txt \
      --vits-data-dir=./espeak-ng-data \
      --num-threads=4 --sid=0 \
      --output-filename=./test-0.wav \
      "The quick brown fox jumped skillfully over the lazy dog..."
```

```
Elapsed seconds: 1.012
Audio duration: 9.532 s
Real-time factor (RTF): 1.012/9.532 = 0.106
Time to first audio: 141 ms          <- 9.5 s of audio, first sound in 141 ms
```

Add `--play` to hear it live as it generates.

## Why this is possible

VITS is not autoregressive, so streaming looks like it shouldn't work. But
`SynthesizerTrn.infer()` splits cleanly at the monotonic alignment:

| half | what it does | context needed | cost |
|---|---|---|---|
| **frontend** | text encoder → duration predictor → alignment → prior expansion → `z_p` | global, over text | ~15-20% of compute, and **quadratic** in text length |
| **decoder** | flow → HiFi-GAN | **finite receptive field over frames** | the rest, linear in output length |

The frontend being quadratic matters: measured on `en_US-libritts_r-medium`, it
is 16% of total compute for a 7 s utterance, 37% at 59 s, and 68% at 178 s
(scaling exponent 1.89 against the decoder's 1.06). It is cheap at sentence and
paragraph scale and expensive at essay scale, so very long inputs should be cut
at sentence boundaries into segments — see `Speaker` below, which does that
automatically. Chunked decoding is still exact within each segment.

Everything that carries paragraph-level prosody lives in the cheap frontend,
which runs once on the whole input. The expensive half is a plain CNN over the
frame axis — no recurrence, no global attention — so it can be evaluated in
chunks, and with enough overlap context the chunks concatenate *exactly*.

This is the opposite trade-off from sentence chunking. Splitting text throws away
the global context that produces good prosody. Splitting in *frame* space costs
nothing, because the frame-space network cannot see beyond its receptive field
anyway.

One detail makes it exact rather than approximate: both `RandomNormalLike` nodes
(duration noise and prior noise) land in the frontend, so all sampling happens
once, for every frame, before any chunk is decoded. Two overlapping chunk decodes
necessarily see identical noise on the frames they share.

## Install

```bash
pip install streaming-vits

# phonemization backend (one of):
brew install espeak-ng          # macOS
apt install espeak-ng           # debian/ubuntu
pip install piper-phonemize     # faster, no subprocess

pip install 'streaming-vits[play]'     # optional: live playback
pip install 'streaming-vits[export]'   # optional: build the cached-state decoder
```

Point it at any piper voice — the same tarballs sherpa-onnx uses:

```bash
wget https://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models/vits-piper-en_US-libritts_r-medium.tar.bz2
tar xf vits-piper-en_US-libritts_r-medium.tar.bz2
```

The monolithic `.onnx` is split into a frontend and a decoder graph on first use
and cached next to the model. You don't have to do anything.

## Calibrate for your device

The chunk schedule is the whole game for latency, and the right one depends on
how fast the device is. Run this once per device:

```bash
streaming-vits calibrate \
    --vits-model=./en_US-libritts_r-medium.onnx \
    --vits-tokens=./tokens.txt \
    --vits-data-dir=./espeak-ng-data \
    --num-threads=4
```

It measures two different things:

**Margins** are a property of the model's weights — how much context does the
decoder actually need before its output stops changing? Swept empirically,
because it is not guessable: trained weights use far more of their nominal
receptive field than an untrained network suggests.

```
[1/3] receptive field (model property, device independent)
  seed 1234 -- deterministic across runs
  lookahead  rel RMS err
          4     32.835%
          8     11.672%
         12      4.197%
         16      1.047%
         20      0.174%
         24      0.006%
  -> left 34, right 28 (325 ms lookahead)
```

The frontend sampling is pinned to a fixed seed for this sweep (`--seed`), so
repeated calibrations of the same model give the same margins. Normal synthesis
is unaffected and stays stochastic.

**Schedule** is a property of the device. An affine cost model
`decode(n) = α + β·(n + margins)` is fitted, then the schedule that reaches the
speaker soonest without ever starving playback is solved for:

```
[2/3] decode cost on this device
  fit: 1.26 ms/call + 0.442 ms/frame (a frame is 11.61 ms of audio)

  for a 300-phoneme reply (~547 frames, 6.4s of audio):
  predicted time to first audio  74 ms
  steady-state RTF per chunk     0.048
  survives a device              2.0x slower than this one
  steady-state decode overhead   1.00x

  without streaming, same utterance  286 ms to first audio
  streaming gets you                  74 ms  (3.8x)
```

The reported figure is for `--reply-ids` (default 300), not the long calibration
paragraph: the frontend is quadratic, so which utterance you measure changes the
answer substantially. Calibration also reports the non-streaming baseline and
says plainly when streaming loses or barely wins — on hardware that only just
outruns playback, it sometimes does.

`--safety N` sets how much slower than measured to plan for; the prebuffer is
sized so headroom is a guarantee rather than an observation. If the device
cannot hold the factor you ask for, calibration reports the ceiling it *can*
hold and backs off to the most headroom buyable with a prebuffer worth
tolerating, rather than refusing or burying it in seconds of latency.

Margins are reproducible run to run, but the *schedule* is not quite: it depends
on wall-clock timings, which move with machine load. Calibrate on an otherwise
idle device, and treat `--safety` as the thing that absorbs the rest.

The profile is written next to the model and picked up automatically.

## The cached-state decoder

By default each chunk re-decodes `left + right` margin frames. That is exact but
wasted, and on a device that barely outruns playback it decides everything. A
one-time export replaces it with a decoder that carries its convolution state
between chunks, so no frame is computed twice:

```bash
pip install 'streaming-vits[export]'
streaming-vits export-stream --vits-model=./en_US-libritts_r-medium.onnx
streaming-vits calibrate      --vits-model=./en_US-libritts_r-medium.onnx
```

Exporting needs torch; **the resulting graph runs on onnxruntime alone**, so
build it on a laptop and copy the file to the device. It is picked up
automatically once it sits in the model's `.streaming-vits/` directory, and the
windowed decoder stays as the fallback. `streaming-vits info` says which is live.

Measured on a Raspberry Pi 4 (4 threads, 327-phoneme reply):

| | time to first audio | vs no streaming | wasted work |
|---|---|---|---|
| no streaming | 5352 ms | — | — |
| windowed | 2964 ms | 1.8x | 1.41x |
| **cached-state** | **1336 ms** | **4.0x** | **1.00x** |

Two consequences worth knowing. The decoder now emits nothing until its whole
right-hand receptive field has been fed, so the first chunk is floored at that
(~43 frames) — smaller is wasted effort. And its per-frame cost is ~30% higher
than the windowed decoder's, which lowers the maximum safety factor the device
can hold; on the Pi above that ceiling is 1.14. Re-run `calibrate` after
exporting: the two decoders have different cost models and a profile fitted to
one produces the wrong schedule for the other (the CLI warns if you forget).

## Results

`en_US-libritts_r-medium`, 4 threads, M-series Mac, 15.2 s utterance:

```
non-streaming  0.701s  RTF 0.046   <- you wait this long before any sound

time to first audio  0.163s   (4.3x sooner than non-streaming)
total wall clock     1.301s  RTF 0.085  (1.86x the work)
playback             no underrun
```

Time-to-first-audio for the non-streaming path scales with the length of the
paragraph. For the streamed path it is constant, so the gap widens the more you
ask it to say.

Equivalence, on the real weights:

```
$ streaming-vits verify --vits-model=en_US-libritts_r-medium.onnx ...
  max |streamed - full| = 8.00e-05  OK

$ streaming-vits verify --vits-model=es_MX-claude-high.onnx ...
  max |streamed - full| = 1.79e-05  OK
```

## Commands

| command | what it does |
|---|---|
| `streaming-vits-offline-tts …` | drop-in for `sherpa-onnx-offline-tts` |
| `streaming-vits speak … --play` | synthesise, optionally play live |
| `streaming-vits calibrate …` | benchmark this device, write a profile |
| `streaming-vits bench …` | streamed vs non-streaming latency |
| `streaming-vits verify …` | check chunked output still equals a full decode |
| `streaming-vits export-stream …` | build the cached-state decoder (needs torch) |
| `streaming-vits info …` | show split graphs and active profile |
| `streaming-vits clear-cache …` | delete cached split graphs |

Accepted-and-ignored sherpa flags (`--vits-lexicon`, `--vits-dict-dir`,
`--tts-rule-fsts`, `--max-num-sentences`) print a note rather than failing, so
existing command lines keep working.

## Migrating from a sherpa-onnx streaming wrapper

If you already drive `sherpa_onnx.OfflineTts` with a per-sentence callback,
split text at a word cap for latency, and pipe chunks to `paplay --raw`, then
`streaming_vits.Speaker` is a one-class swap with the same constructor,
`say()`, `stop()`, `warmup()` and `sample_rate`:

```python
- from your_pipeline import Speaker           # sherpa-onnx based
+ from streaming_vits import Speaker

  spk = Speaker(model="en_US-libritts_r-medium.onnx",
                tokens="tokens.txt",
                data_dir="espeak-ng-data",
                num_threads=2,
                player="paplay",
-               max_words=12,      # accepted and ignored: no longer needed
                speed=1.0)
  spk.warmup()
  spk.say("...")
```

`max_words`/`first_words` are accepted and ignored. Word-cap splitting exists to
buy latency out of a text-space chunker; here the chunking is in frame space, so
you get the latency **and** keep whole sentences. Measured on one 7.5 s sentence,
2 threads, no text splitting either way:

```
sherpa-style (whole sentence, then play):  0.418 s to first audio
streaming-vits (same sentence, unsplit):   0.123 s to first audio   3.4x sooner
```

Text is normally **not** segmented. `max_ids` (default 1600 phonemes, roughly
35 s of speech) is a latency guard for very long input, not a quality setting;
set `max_ids=0` to disable it entirely. Sentences are never broken either way.

`stop()` cancels mid-utterance and kills the player. `Speaker` uses
`PipePlayer`, which needs no PortAudio — usually the difference between working
and not working on a Pi. The CLI exposes the same thing:

```bash
streaming-vits speak --player=paplay --vits-model=... "text"
```

## Python API

```python
from streaming_vits import StreamingTTS

tts = StreamingTTS("en_US-libritts_r-medium.onnx", tokens_path="tokens.txt")

for chunk in tts.stream("Hello there. This plays before it has finished."):
    speaker.write(chunk.audio)      # float32, mono, tts.sample_rate
    print(chunk.index, chunk.frames, chunk.ready_at)

audio = tts.synthesize("Same thing, one array.")
```

## Limits

- **Extra work, unless you export the cached-state decoder.** The default path
  re-decodes its margin frames every chunk: ~1.2-1.4× in steady state. The
  cached-state decoder removes it entirely (1.00×) — see above.
- **TTFA is dominated by the frontend**, not by chunk size, and the gap widens
  once the decoder is fast: on the Pi above, 743 ms of the 1336 ms is the text
  encoder running before any sample can exist. Smaller chunks will not help.
  The encoder is quadratic *and* not chunkable — its output converges only as
  ~1/context, still 2.7e-02 off with 512 tokens of context either side, against
  3e-06 for the decoder at 32 frames — so this is a floor, not an oversight.
- **Streaming text *in* is a separate problem.** The duration predictor needs the
  whole utterance before any audio exists. If text arrives from an LLM token
  stream you still chunk at clause level — but you can feed the previous clause
  as context and discard its audio to soften the seam.
- **The bundled phonemizer is pragmatic.** It shells out to `espeak-ng` per
  sentence and re-inserts punctuation, where piper drives libespeak-ng directly
  and gets clause terminators back. Install `piper-phonemize` for the faithful
  path. This does not affect the streaming claim — `verify` compares on identical
  phoneme ids.
- Verified on two piper voices: `en_US-libritts_r-medium` (904 speakers) and
  `es_MX-claude-high` (single speaker, different exporter version). Other VITS
  exports should work if the graph has the same seam; `streaming-vits info` will
  tell you before you rely on it.

## How the split is found

`streaming_vits/graph.py` locates the seam structurally rather than by hardcoded
node names: find the last `RandomNormalLike` (the prior noise), walk forward to
the `Add` that forms `z_p`, then collect whatever the downstream subgraph still
needs from upstream. It also reads the decoder config back off the weights —
piper's `config.json` does not record it, and it varies by voice
(`libritts_r-medium` uses 3 upsample stages, `[8,8,4]`, not the 4 in the
reference VITS config).

Two things are deliberately not read from node names, because those vary between
piper exporter versions even when the weights do not: the upsample config comes
from `dec.ups.N.weight`, and the seam comes from graph topology. Single-speaker
voices have no speaker embedding and no `sid` input, so the decoder takes two
inputs instead of three; that is detected rather than configured.

`research/` holds the original proof of concept, including `verify_streaming.py`,
which proves the same equivalence against the reference PyTorch VITS
implementation independently of ONNX.

## License

MIT
