Metadata-Version: 2.4
Name: voxlock
Version: 1.1.0
Summary: The best performing voice isolator there is: keeps the caller's voice and removes the other voices in the room. Streaming, CPU, ONNX.
Author-email: Arbaz Siddiqui <arbaz00@gmail.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://huggingface.co/arbazsiddiqui/voxlock
Project-URL: Repository, https://github.com/arbazsiddiqui/voxlock
Project-URL: Demo, https://arbazsiddiqui.me/voxlock
Project-URL: Issues, https://github.com/arbazsiddiqui/voxlock/issues
Keywords: voice isolation,background voice cancellation,target speaker extraction,speech enhancement,voice agents,telephony,speech-to-text,streaming,real-time,onnx,int8,pipecat
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: onnxruntime>=1.17
Requires-Dist: huggingface_hub>=0.20
Requires-Dist: soundfile>=0.12
Requires-Dist: soxr>=0.3
Provides-Extra: pipecat
Requires-Dist: pipecat-ai>=1.12; extra == "pipecat"
Dynamic: license-file

# VoxLock

**The best performing voice isolator there is.** VoxLock is an open-source voice isolation model: it keeps the caller's voice and removes the other voices in the room before speech-to-text hears them. Made for voice agents and phone calls: one input, the call audio, and it picks out the caller by itself. It streams on a CPU with 24 ms of algorithmic latency.

**[Live demo](https://arbazsiddiqui.me/voxlock)** · **[Model on Hugging Face](https://huggingface.co/arbazsiddiqui/voxlock)**

<p>
  <a href="https://pypi.org/project/voxlock/"><img alt="PyPI" src="https://img.shields.io/pypi/v/voxlock?label=pypi&cacheSeconds=3600"></a>
  <a href="https://huggingface.co/arbazsiddiqui/voxlock"><img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20model-voxlock-yellow"></a>
  <a href="https://github.com/arbazsiddiqui/voxlock#results"><img alt="Krisp benchmark WER 4.95%" src="https://img.shields.io/badge/Krisp%20benchmark%20WER-4.95%25-brightgreen"></a>
  <a href="https://github.com/arbazsiddiqui/voxlock/blob/main/LICENSE"><img alt="License: Apache-2.0" src="https://img.shields.io/badge/license-Apache--2.0-blue"></a>
</p>

<a href="https://raw.githubusercontent.com/arbazsiddiqui/voxlock/main/assets/demo.mp4"><img src="https://raw.githubusercontent.com/arbazsiddiqui/voxlock/main/assets/demo-play.jpg" alt="The same 6.5 seconds of a call with VoxLock off and on: off, speech-to-text transcribes the other person; on, it transcribes the caller" width="100%"></a>

## Results

<img src="https://raw.githubusercontent.com/arbazsiddiqui/voxlock/main/assets/results.png" alt="VoxLock has the lowest word error rate and leaks the fewest background words on the Krisp Voice Isolation Benchmark" width="100%">

| Model | Access | Word error rate % | Background words leaked / min |
|---|---|---|---|
| **VoxLock** | **open** | **4.95** | **0.11** |
| Krisp VI 2.5 Default | paid | 5.30 | 1.52 |
| Krisp VI 2.5 Balanced | paid | 5.32 | 2.02 |
| Krisp VI 2.5 HD | paid | 5.79 | 2.74 |
| Krisp VI 2.5 Lite | paid | 6.21 | 4.52 |
| ai-coustics Voice Focus 2.2 L | paid | 6.84 | 0.66 |
| ai-coustics Voice Focus 2.2 S | paid | 6.91 | 1.00 |
| Sanas Voice Isolation SE 1.0 | paid | 9.38 | 13.76 |
| REAL-TSE causal (VoxLock's starting checkpoint) | open | 9.81 | 13.73 |
| Hush | open | 12.09 | 19.58 |
| DeepFilterNet3 | open | 13.30 | 22.77 |
| TIGER | open | 14.72 | 37.24 |
| No processing | | 23.21 | 66.48 |
| SAM-Audio small (Meta) | open | 33.77 | 21.15 |
| ElevenLabs Voice Isolator\*\* | paid | 43.79\*\* | 144.6\*\* |

265 real recordings from the [Krisp Voice Isolation Benchmark](https://huggingface.co/datasets/Krisp-AI/VoiceIsolation-Benchmark-Dataset), transcribed with Parakeet TDT 0.6B v2 and scored with the benchmark's published method. **Background words leaked** counts words transcribed while only the background was talking, per minute of that audio.

\*\* ElevenLabs Voice Isolator, run on the 7 of these recordings with the most background speech, not all 265. On the same 7, no processing scores 45.82% and 154.3 leaked words per minute, and VoxLock 2.98% and 0.0.

On real AMI meetings not used in training, with the headset wearer as the caller:

| Model | Access | Word error rate % | Background words leaked / min |
|---|---|---|---|
| **VoxLock** | **open** | **10.12** | 2.12 |
| ai-coustics Voice Focus 2.2 S | paid | 11.85 | **1.06** |
| ai-coustics Voice Focus 2.2 L | paid | 12.13 | 1.72 |
| Sanas Voice Isolation SE 1.0 | paid | 13.30 | 5.30 |
| TIGER | open | 22.84 | 36.85 |
| DeepFilterNet3 | open | 26.78 | 38.84 |
| Hush | open | 33.00 | 68.93 |
| No processing | | 33.03 | 78.34 |

Caller words missed, deletions, insertions and 95% intervals: [bench/results](https://github.com/arbazsiddiqui/voxlock/tree/main/bench/results).

## Use

```bash
pip install voxlock
voxlock call.wav -o caller.wav
```

Any sample rate, mono or stereo. The model downloads from Hugging Face on first use: 75 MB, the 8-bit version (see [Quantized models](https://github.com/arbazsiddiqui/voxlock#quantized-models)).

```python
import voxlock

vl = voxlock.VoxLock()                     # precision="fp32" for the benchmarked model
call = voxlock.load_audio("call.wav")      # 16 kHz mono float32
caller = vl.isolate(call)                  # same length as the input
```

VoxLock keeps the voice in a 3-second sample of the caller. `isolate` takes the loudest 3 seconds in the first 15 seconds of the call, which is how it was benchmarked. Pass `sample=` to choose it yourself.

### Streaming

Push the call in as it arrives, as 16 kHz float32 chunks of any size. Each `push` returns the isolated audio that is ready, about 24 ms behind the input.

```python
vl = voxlock.VoxLock(threads=4)
stream = vl.live()                         # finds the caller by itself
for chunk in call_audio:                   # e.g. 40 ms = 640 samples
    send(stream.push(chunk))
send(stream.flush())                       # at the end of the call
```

`live()` passes the audio through unchanged until it has heard 3 seconds of speech, takes that speech as the voice sample and isolates from then on. On our test calls it switched over a median of 3.6 seconds in.

If you already have a sample, from earlier in the call or from a previous call, isolation starts with the first chunk:

```python
stream = vl.stream(sample)                 # about 3 s of the caller, 16 kHz mono
```

Streaming gives the same output as `isolate` with the same sample. To keep up with a live call, push chunks of 40 ms or more: 40 ms chunks on 4 threads run at 0.69x real time on an Apple M2 Pro and 0.89x on an 8-vCPU Xeon server, and 80 ms chunks on 2 threads at 0.67x on the Xeon. Larger chunks use less CPU and add their length to the delay.

### Pipecat

```bash
pip install "voxlock[pipecat]"
```

```python
from pipecat.transports.base_transport import TransportParams
from voxlock.pipecat import VoxLockFilter

params = TransportParams(audio_in_enabled=True, audio_in_filter=VoxLockFilter())
```

The filter cleans the caller's audio before VAD and speech-to-text, at any transport sample rate (Twilio's 8 kHz included). Like `live()`, it passes audio through until it has heard 3 seconds of the caller; pass `voice_sample=` to isolate from the start. It runs the `int8` model in 80 ms chunks on 2 threads by default (`chunk_ms=`, `threads=`, `precision=`). With a single vCPU, such as Pipecat Cloud's `agent-2x`, use `VoxLockFilter(chunk_ms=160, threads=1)`: 0.73x real time on the Xeon server. Half a vCPU (`agent-1x`) is too little. A full voice agent: [examples/pipecat_bot.py](https://github.com/arbazsiddiqui/voxlock/blob/main/examples/pipecat_bot.py). Tested with Pipecat 1.12.0.

### PyTorch

```bash
git clone https://github.com/arbazsiddiqui/voxlock && cd voxlock
pip install -r requirements.txt
huggingface-cli download arbazsiddiqui/voxlock model.safetensors config.yaml --local-dir ckpt
python isolate.py --ckpt ckpt --in call.wav --out caller.wav
```

## Quantized models

The ONNX model comes in four precisions, all in the [Hugging Face repository](https://huggingface.co/arbazsiddiqui/voxlock/tree/main/onnx). Choose with `VoxLock(precision=...)`, `voxlock --precision ...` or `VoxLockFilter(precision=...)`.

| Precision | Files | Download | Word error rate % | Background words leaked / min | Real-time factor, 40 ms chunks, 2 / 4 threads |
|---|---|---|---|---|---|
| `int8` (default) | `enroll.onnx`, `chunk_int8.onnx` | 75 MB | 5.11 | 0.12 | 0.89 / 0.69 |
| `int8-full` | `enroll.onnx`, `chunk_int8_full.onnx` | 67 MB | 5.85 | 0.04 | 0.67 / 0.54 |
| `fp16` | `enroll_fp16.onnx`, `chunk_fp16.onnx` | 48 MB | 4.95\* | 0.11\* | 1.31 / 1.00 |
| `fp32` | `enroll.onnx`, `chunk.onnx` | 95 MB | 4.95 | 0.11 | 1.33 / 0.97 |

Scores are on all 265 recordings of the Krisp benchmark above, so even the `int8` model is ahead of every other model in the table. The real-time factor is streaming on an Apple M2 Pro; below 1 keeps up with a live call. The 8-bit files are post-training dynamic quantization of the same weights: `int8` stores the LSTM weights in 8 bits, `int8-full` the LSTMs and the matrix multiplies. `int8-full` is the fastest: on one thread it keeps up with 80 ms chunks (0.80x real time on the Xeon server), where `int8` needs 160 ms chunks (0.73x).

\* `fp16` is the float32 model with its weights rounded to float16. Its output matches float32 to 45 dB, so it carries the float32 scores. The browser demo uses it.

`enroll` runs once on the voice sample, `chunk` runs on each chunk of the call and carries the LSTM state.

## Training

A 27M causal band-split RNN target-speaker extractor, fine-tuned from the [REAL-TSE challenge baseline](https://github.com/REAL-TSE/wesep-real-tse) on a single NVIDIA L4:

```
REAL-TSE tfmap_context_causal_100
  -> simulated calls        4,000 steps   LibriSpeech callers + background talkers in simulated rooms
  -> real AMI crosstalk     4,000 steps   headset recordings from 40 meetings
  -> long calls (30 s)        800 steps   callers who talk throughout
  -> rooms and distances     400 steps   callers near or far from the mic, background voices dry or in the room, room noise
```

## Repository

| Path | |
|---|---|
| `voxlock/` | the `pip install voxlock` package: whole files, streaming, command line, Pipecat filter |
| `examples/` | a Pipecat voice agent with VoxLock |
| `tests/` | the Pipecat filter test |
| `isolate.py` | PyTorch inference |
| `export_onnx.py` | ONNX export |
| `demo/` | the browser demo (`python demo/serve.py`) |
| `bench/` | benchmark harness, vendor runners, per-file results |
| `train/` | training scripts and data preparation |

## Acknowledgements

Built on the REAL-TSE challenge baseline and [WeSep](https://github.com/wenet-e2e/wesep) by Shuai Wang, Ke Zhang and co-authors, with [WeSpeaker](https://github.com/wenet-e2e/wespeaker)'s ECAPA model. Trained on [LibriSpeech](https://www.openslr.org/12) and the [AMI Meeting Corpus](https://groups.inf.ed.ac.uk/ami/corpus/).

## License

Apache-2.0.
