Metadata-Version: 2.5
Name: livekit-wakeword
Version: 2.1.1
Summary: Type a wake word, get a streaming model: training and streaming detection for LiveKit wake word models
Project-URL: Homepage, https://github.com/livekit/livekit-wakeword
Project-URL: Issues, https://github.com/livekit/livekit-wakeword/issues
Author-email: LiveKit <hello@livekit.io>
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: keyword-spotting,livekit,onnx,speech,tflite,voice,wake-word
Requires-Python: >=3.11
Requires-Dist: numpy>=1.24
Requires-Dist: onnxruntime>=1.17
Provides-Extra: listener
Requires-Dist: pyaudio>=0.2.14; extra == 'listener'
Provides-Extra: tflite
Requires-Dist: ai-edge-litert>=1.0; extra == 'tflite'
Requires-Dist: huggingface-hub>=0.20; extra == 'tflite'
Requires-Dist: librosa>=0.10; extra == 'tflite'
Requires-Dist: livekit-wakeword-bench>=0.1.1; (python_version >= '3.12') and extra == 'tflite'
Requires-Dist: onnx-graphsurgeon>=0.3.27; extra == 'tflite'
Requires-Dist: onnx2tf>=1.22; extra == 'tflite'
Requires-Dist: onnx>=1.15; extra == 'tflite'
Requires-Dist: onnxsim>=0.4; extra == 'tflite'
Requires-Dist: pronouncing>=0.2; extra == 'tflite'
Requires-Dist: psutil>=5.9; extra == 'tflite'
Requires-Dist: pyarrow>=14; extra == 'tflite'
Requires-Dist: pydantic>=2.0; extra == 'tflite'
Requires-Dist: pyyaml>=6.0; extra == 'tflite'
Requires-Dist: rich>=13.0; extra == 'tflite'
Requires-Dist: safetensors>=0.4; extra == 'tflite'
Requires-Dist: setuptools<85,>=84.1.0; extra == 'tflite'
Requires-Dist: sng4onnx>=1.0.4; extra == 'tflite'
Requires-Dist: soundfile>=0.12; extra == 'tflite'
Requires-Dist: soxr>=0.3; extra == 'tflite'
Requires-Dist: tensorflow>=2.15; extra == 'tflite'
Requires-Dist: tf-keras>=2.15; extra == 'tflite'
Requires-Dist: torch>=2.5; extra == 'tflite'
Requires-Dist: torchaudio>=2.5; extra == 'tflite'
Requires-Dist: typer>=0.12; extra == 'tflite'
Provides-Extra: train
Requires-Dist: huggingface-hub>=0.20; extra == 'train'
Requires-Dist: librosa>=0.10; extra == 'train'
Requires-Dist: livekit-wakeword-bench>=0.1.1; (python_version >= '3.12') and extra == 'train'
Requires-Dist: onnx>=1.15; extra == 'train'
Requires-Dist: pronouncing>=0.2; extra == 'train'
Requires-Dist: pyarrow>=14; extra == 'train'
Requires-Dist: pydantic>=2.0; extra == 'train'
Requires-Dist: pyyaml>=6.0; extra == 'train'
Requires-Dist: rich>=13.0; extra == 'train'
Requires-Dist: safetensors>=0.4; extra == 'train'
Requires-Dist: setuptools<85,>=84.1.0; extra == 'train'
Requires-Dist: soundfile>=0.12; extra == 'train'
Requires-Dist: soxr>=0.3; extra == 'train'
Requires-Dist: torch>=2.5; extra == 'train'
Requires-Dist: torchaudio>=2.5; extra == 'train'
Requires-Dist: typer>=0.12; extra == 'train'
Description-Content-Type: text/markdown

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/livekit/livekit-wakeword/main/.github/banner_dark.png">
  <source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/livekit/livekit-wakeword/main/.github/banner_light.png">
  <img style="width:100%;" alt="The LiveKit icon, the name of the repository and some sample code in the background." src="https://raw.githubusercontent.com/livekit/livekit-wakeword/main/.github/banner_light.png">
</picture>

# LiveKit WakeWord

[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue)](https://github.com/livekit/livekit-wakeword/tree/main/client/python)
[![Swift](https://img.shields.io/badge/swift-iOS%20%7C%20macOS-orange)](https://github.com/livekit/livekit-wakeword/tree/main/client/swift)
[![JavaScript](https://img.shields.io/badge/js-browser%20%7C%20Node-yellow)](https://github.com/livekit/livekit-wakeword/tree/main/client/js)
[![ESP32](https://img.shields.io/badge/ESP32-TFLite%20Micro-red)](https://github.com/livekit/livekit-wakeword/tree/main/client/esp32)
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://github.com/livekit/livekit-wakeword/blob/main/LICENSE)

**[Why LiveKit WakeWord](#why-livekit-wakeword) · [Comparison](#comparison) ·
[Guides and examples](#guides-and-examples) · [Train your own](#train-a-wake-word) ·
[Models](#models) · [Development](#development)**

LiveKit WakeWord is an open-source toolkit for training and running wake word models. You provide
a phrase, and the training CLI generates speech with LiveKit Inference TTS and adapts a pretrained
encoder to recognize it. The resulting model runs locally in Python, Swift, JavaScript (browser and Node) or on an ESP32, without
sending audio to a server. You do not need to record your own training data.

> [!IMPORTANT]
> The original LiveKit WakeWord implementation, based on openWakeWord, is deprecated.
> LiveKit WakeWord now uses LiveKit's own pretrained encoders. In our “hey livekit” benchmark,
> the standard model detected 89% of wake words at 0.5 false alarms per hour, compared with 46%
> for openWakeWord 0.6. It used 16.5 ms of CPU time per second of audio, compared with 29.5 ms.
> Models from the deprecated implementation are not compatible with the new clients;
> [retrain your phrase](#train-a-wake-word) to upgrade.

## Why LiveKit WakeWord

Training a standard model for “hey livekit” took 11 minutes on one RTX PRO 6000 GPU: about
6 minutes to generate speech and 5 minutes to train, calibrate and export the model. Training
used 1,000 generated wake-word clips. The same workflow produces a micro model in about 18 minutes,
including the standard model it learns from.

The standard model is about 7 MB and runs in Python, Swift, the browser or Node through ONNX Runtime. The micro
model is 451 KB and runs on ESP32 through TensorFlow Lite Micro. You can start with a
[ready-made model](#models), or [train a model](#train-a-wake-word) for your own phrase. Running
detection requires neither a GPU nor LiveKit Cloud credentials.

## Comparison

We trained “hey livekit” separately in each engine using its own recipe, then evaluated the
models on the same benchmark. At 0.5 false alarms per hour, LiveKit WakeWord detected 89% of wake
words, nanowakeword 49%, and openWakeWord 46%. Their training times were 11, 34 and 94 minutes,
respectively. Streaming inference used 16.5, 30.5 and 29.5 ms of CPU time per second of audio.

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/livekit/livekit-wakeword/main/.github/assets/accuracy_dark.png">
  <img width="100%" alt="More accurate at every false-alarm budget. Wake words caught at 0.1, 0.5 and 1 false alarms per hour: LiveKit 87%, 89%, 90%; LiveKit micro 24%, 78%, 79%; nanowakeword 0%, 49%, 55%; openWakeWord 32%, 46%, 52%; microWakeWord 0%, 0%, 24%." src="https://raw.githubusercontent.com/livekit/livekit-wakeword/main/.github/assets/accuracy_light.png">
</picture>

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/livekit/livekit-wakeword/main/.github/assets/speed_dark.png">
  <img width="100%" alt="Faster to train, lighter to run. Train time: LiveKit 11 min, LiveKit micro 18 min, nanowakeword 34, openWakeWord 94, microWakeWord 39. Wake-word clips: 1k and 1k against 2.5k, 100k and 25k. CPU per second of audio: 16.5 and 4.4 ms against 30.5, 29.5 and 2.2 ms." src="https://raw.githubusercontent.com/livekit/livekit-wakeword/main/.github/assets/speed_light.png">
</picture>

The micro model detected 78% of wake words at the same false-alarm rate. microWakeWord scored
0% at that operating point, although it used less CPU: 2.2 ms per second of audio versus 4.4 ms
for LiveKit micro. These results compare the complete training workflows for one phrase,
including their different speech generators; they do not isolate encoder performance.

<details>
<summary>How we compared</summary>

We used [LiveKit WakeWord Bench](https://github.com/livekit/wakeword-bench) with party scenes:
7,873 clips covering unseen voices, noise, rooms, codecs, similar-sounding phrases and
conversational speech, plus 24 hours of TV, music and podcasts. Accuracy in the chart is recall
at 0.1, 0.5 and 1 false alarms per hour. Each model's threshold is selected for that rate; this is different
from setting every model's score threshold to 0.5.

The alternatives used Piper TTS for training speech, while LiveKit WakeWord used LiveKit
Inference TTS. Benchmark voices were held out of training, but the benchmark also uses LiveKit
Inference, which may favor our models.

We tested nanowakeword 3.0 and openWakeWord 0.6 (100,000 samples). For microWakeWord, training
time and clip counts use its scaled-up recipe (25,000 samples, 40,000 steps); accuracy is the
better result from that recipe and its notebook default.

Each pipeline ran alone on one RTX PRO 6000 GPU with the same 8 CPU cores. Training times exclude
one-time downloads: 3.8 GB for LiveKit WakeWord and 21–35 GB for the alternatives. The micro
training time includes its standard teacher model. CPU measurements use one core and one thread
with streaming audio.

</details>

## Guides and examples

Each client has a README with installation and usage instructions, along with examples you can
run. Use a [ready-made model](#models) or one you trained yourself.

| Platform | Installation and usage | Examples |
|---|---|---|
| Python | [Python README](https://github.com/livekit/livekit-wakeword/blob/main/client/python/README.md) | [Audio clip inference](https://github.com/livekit/livekit-wakeword/blob/main/client/python/examples/inference.py) · [Microphone listener](https://github.com/livekit/livekit-wakeword/blob/main/client/python/examples/listener.py) |
| Swift | [Swift README](https://github.com/livekit/livekit-wakeword/blob/main/client/swift/README.md) | [SwiftUI demo for iOS and macOS, with setup instructions](https://github.com/livekit/livekit-wakeword/blob/main/client/swift/Example/README.md) |
| JavaScript | [JavaScript README](https://github.com/livekit/livekit-wakeword/blob/main/client/js/README.md) | [Browser demo with a live score meter](https://github.com/livekit/livekit-wakeword/tree/main/client/js/example) |
| ESP32 | [ESP32 README](https://github.com/livekit/livekit-wakeword/blob/main/client/esp32/README.md) | [I2S microphone detection](https://github.com/livekit/livekit-wakeword/tree/main/client/esp32/examples/i2s_mic) · [Wake word to LiveKit agent](https://github.com/livekit/livekit-wakeword/tree/main/client/esp32/examples/livekit_agent) · [Build and flash instructions](https://github.com/livekit/livekit-wakeword/blob/main/client/esp32/README.md#examples) |

## Models

The [latest release](https://github.com/livekit/livekit-wakeword/releases/latest) has ready-made
models and the pretrained encoders that training starts from:

```bash
gh release download -R livekit/livekit-wakeword -p "*.onnx" -p "*.tflite"
```

| model | bench score |
|---|:---:|
| `hey_livekit.onnx` | 0.909 |
| `hey_livekit_micro.tflite` | 0.767 |
| `hey_jarvis.onnx` | 0.914 |
| `hey_jarvis_micro.tflite` | 0.716 |

hey livekit is scored on the bench with party scenes (the one in the comparison above), and hey
jarvis on the plain bench. **standard** is ONNX (about 7 MB, about 1 ms per 80 ms step on one CPU
thread). **micro** is int8 TFLite (451 KB, 11.5 million multiply-accumulates per second of
audio) for TensorFlow Lite Micro. At the default threshold of 0.5, these four models give 0.5 to 2.5
false alarms per hour on the bench's conversational speech; raise the threshold if you see too many.

## Train a wake word

You need a CUDA GPU or an Apple silicon Mac, [uv](https://docs.astral.sh/uv/) and a LiveKit Cloud project
(the free plan works) for the TTS.

**1. Get LiveKit credentials.** Sign up at [cloud.livekit.io](https://cloud.livekit.io) and create a
project, then install the [LiveKit CLI](https://docs.livekit.io/reference/developer-tools/livekit-cli/)
and log in:

```bash
brew install livekit-cli                         # macOS
curl -sSL https://get.livekit.io/cli | bash      # Linux
winget install LiveKit.LiveKitCLI                # Windows

lk cloud auth                                    # log in in the browser and pick your project
lk app env -w                                    # writes LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET to .env.local
set -a && source .env.local && set +a            # export them in this shell
```

You can also create an API key in the project settings of the LiveKit Cloud dashboard and export
the three variables yourself.

**2. Install the CLI:**

```bash
uv tool install --python 3.13 "livekit-wakeword[tflite]"   # or [train] if you only need standard models
```

This puts `livekit-wakeword` on your PATH; you do not need to clone this repo. uv downloads
Python 3.13 if you do not have it. `uv tool upgrade livekit-wakeword` updates it later. `setup`
downloads the pretrained encoders from the v2.0.0 release once.

**3. Train** (in an empty folder: it holds the config, the models and a `.cache` of noise sources):

```bash
livekit-wakeword setup --wakeword "hey robot" --eval  # training speech + held-out test set
livekit-wakeword train --wakeword "hey robot" --type standard
livekit-wakeword eval --wakeword "hey robot"          # bench score on a held-out test set
uvx --from "livekit-wakeword[listener]" livekit-wakeword listen hey_robot.onnx  # try it live
```

A new phrase takes about 11 minutes on one data-center GPU (6 minutes of speech generation, 5 of
training). On an M4 Pro Mac, speech generation took 22 minutes and training 37. The first run
downloads about 4 GB of noise, room and speech sources once.

`listen` runs on the detection client alone (no trainer), so `uvx` fetches just that and PyAudio,
on any machine. PyAudio builds against PortAudio: `brew install portaudio` on macOS,
`apt install portaudio19-dev` on Debian/Ubuntu.

**For ESP32**, also train a micro model (about 7 more minutes):

```bash
livekit-wakeword train --wakeword "hey robot" --type micro  # -> hey_robot_micro.tflite
```

**More speech, or a different output folder:**

```bash
livekit-wakeword setup --wakeword "hey robot" --clips 4     # 4x the default training speech
livekit-wakeword train --wakeword "hey robot" --out models  # writes models/hey_robot.onnx
```

**Multilingual wake words.** A phrase can be in any language and script. Pass its language with
`--language` (`-l`, an ISO 639-1 code such as `es`, `de`, `vi` or `ja`) so the speech is generated
by voices that speak it; `train` and `eval` read it from the saved config:

```bash
livekit-wakeword setup --wakeword "hola asistente" --language es --eval
livekit-wakeword setup --wakeword "xin chào robot" --language vi
livekit-wakeword setup --wakeword "ねえ ロボット" --language ja
livekit-wakeword train --wakeword "ねえ ロボット"  # -> ねえ_ロボット.onnx
```

The model file is named after the phrase; set `model_name` in the config (for example
`ne_robot`) for an ASCII file name, such as for ESP32 builds.

In English, words outside the dictionary (names, brands, made-up words) need a pronunciation in
[ARPAbet](http://www.speech.cs.cmu.edu/cgi-bin/cmudict), so every voice says them the same way;
`setup` names any word it is missing. Words made of dictionary words, such as "livekit", work
without one. Phrases in other languages need none: their speech is generated from the text, by
every TTS model that speaks the language.

```bash
livekit-wakeword setup --wakeword "hey zork" --pronounce "zork=Z AO1 R K"
```

The [training guide](https://github.com/livekit/livekit-wakeword/blob/main/client/python/TRAINING.md#flags) lists every flag, with a full example from setup to
evaluation.

## Repository layout

| | |
|---|---|
| [`client/python/`](https://github.com/livekit/livekit-wakeword/tree/main/client/python) | Python client (`livekit.wakeword`: `WakeWordModel`, `WakeWordListener`) and the trainer behind the `livekit-wakeword` CLI (`livekit.wakeword_train`, the `train` extra: generates speech with LiveKit Inference TTS, trains, calibrates and evaluates; [training guide](https://github.com/livekit/livekit-wakeword/blob/main/client/python/TRAINING.md)) |
| [`client/swift/`](https://github.com/livekit/livekit-wakeword/tree/main/client/swift) | Swift package for iOS and macOS, plus a SwiftUI demo |
| [`client/js/`](https://github.com/livekit/livekit-wakeword/tree/main/client/js) | JavaScript/TypeScript package `@livekit/wakeword` for the browser (onnxruntime-web) and Node (onnxruntime-node), plus a browser demo |
| [`client/esp32/`](https://github.com/livekit/livekit-wakeword/tree/main/client/esp32) | ESP-IDF component and examples for micro models |
| [`models/`](https://github.com/livekit/livekit-wakeword/tree/main/models) | The hey livekit models the tests and examples use |

## Development

```bash
uv sync --extra tflite
uv run pytest                                    # Python: client and trainer
uv run ruff check . && uv run mypy -p livekit.wakeword -p livekit.wakeword_train
swift test                                       # Swift client
(cd client/js && pnpm install && pnpm test)      # JavaScript client
```

## License

Apache 2.0, see [LICENSE](https://github.com/livekit/livekit-wakeword/blob/main/LICENSE).
