Metadata-Version: 2.5
Name: livekit-plugins-speakerbeamss
Version: 0.2.0
Summary: LiveKit Agents STT plugin that runs OpenSpeakerBeam-SS target-speaker extraction in front of any inner STT.
Author-email: linsan <bin.zaq@foxmail.com>
License: Apache-2.0
Keywords: asr,audio,livekit,realtime,speaker-extraction,target-speaker,webrtc
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Multimedia :: Sound/Audio
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: livekit-agents>=1.5.13
Requires-Dist: numpy>=1.24
Requires-Dist: onnxruntime>=1.17.0
Requires-Dist: soundfile>=0.12
Provides-Extra: examples
Requires-Dist: livekit-plugins-volcengine>=1.3.26; extra == 'examples'
Provides-Extra: test
Requires-Dist: numpy>=1.24; extra == 'test'
Requires-Dist: onnx>=1.14; extra == 'test'
Requires-Dist: pytest-asyncio>=0.21; extra == 'test'
Requires-Dist: pytest>=7.0; extra == 'test'
Description-Content-Type: text/markdown

# livekit-plugins-speakerbeamss

> A [LiveKit Agents](https://github.com/livekit/agents) STT plugin that
> runs [OpenSpeakerBeam-SS](../OpenSpeakerBeam-SS) target-speaker
> extraction in front of any inner STT instance. The agent hears only
> the user — even with competing speakers, music, or noise in the room.

The plugin is **ONNX-only**. The PyTorch separator never ships here.
The **trained INT8 model is bundled inside the wheel** at
`livekit/plugins/speakerbeamss/models/speakerbeam_ss_trained_int8.onnx`
(≈13 MB), so a fresh `pip install livekit-plugins-speakerbeamss` is
enough — no separate download. Follow step 0 below only if you want to
swap in a different export (FP32, the base release, or your own
fine-tune).

The reference audio for the target speaker is taken from the
*agent's first user turn*: the first sentence the user speaks is
forwarded to the inner STT **as-is** so the agent can still
understand it, and is *also* kept in a buffer. When the inner STT
emits `END_OF_SPEECH`, the buffer is collapsed to a Resemblyzer
d-vector and the plugin flips to extraction mode for every
subsequent utterance.

You can also pass a pre-recorded enrollment audio via
`reference_audio="path.wav"` to skip the capture phase entirely.

## Installation

```bash
# 1. Install this plugin (from the plugin source root):
cd livekit/plugins/speakerbeamss
pip install -e .
```

That's it — `pip install` wires up `livekit.plugins.speakerbeamss`
*and* ships the trained ONNX bundle, which the loader resolves via
`importlib.resources`. No env vars, no separate download.

### Using a different exported variant

If you have your own export (FP32, base release, a re-trained
checkpoint, …) you'd rather use, point the loader at it explicitly:

```python
from livekit.plugins.speakerbeamss import STT

sb_stt = STT(
    inner=inner,
    onnx_dir="/Users/linsan/Public/projects/OpenSpeakerBeam-SS/exports",
    onnx_filename="speakerbeam_ss_int8.onnx",   # base INT8 release
)
```

The loader's lookup order is:

1. **Bundled** — `livekit.plugins.speakerbeamss.models/<​filename>`
   (the trained INT8 lives here by default).
2. **Disk** — `{onnx_dir}/{filename}` if set, otherwise the
   `~/Public/projects/OpenSpeakerBeam-SS/exports/` fallback.

To export a different variant yourself:

```bash
cd ~/Public/projects/OpenSpeakerBeam-SS
pip install -r requirements.txt
python export_onnx.py --ckpt checkpoints/best_model.pth
ls exports/   # -> speakerbeam_ss_{fp32,int8,trained_fp32,trained_int8}.onnx
```

## Usage — wrap any inner STT

```python
from livekit.plugins.speakerbeamss import STT
from livekit.plugins.volcengine import STT as VolcSTT  # any streaming inner STT

inner = VolcSTT()                          # read keys from env, etc.
stt = STT(inner=inner)
# ... pass `stt` to your VoicePipelineAgent as usual.
```

That's it. The agent will see exactly the transcripts it would have
seen without the wrapper, but with the first turn's enrollment
frozen into a d-vector and applied to every chunk that follows.

## Usage — offline

```python
import numpy as np
from livekit.plugins.speakerbeamss import OnnxSpeakerBeamExtractor

extractor = OnnxSpeakerBeamExtractor()                     # INT8 by default
extractor.enroll(np.random.randn(16000).astype(np.float32))  # reference audio

chunk = np.random.randn(8000).astype(np.float32)             # < 1 s
print(extractor.append(chunk))    # []  (no full chunk yet)
print(extractor.append(np.zeros(8000, dtype=np.float32)))    # [ndarray of len 16000]
```

## How the state machine works

| Phase | Inner STT input | Plugin behavior |
|------|-----------------|------------------|
| `CAPTURING` (first user turn) | Raw audio, downmixed to mono 16 kHz | Buffers audio; on `END_OF_SPEECH`, computes d-vector and locks |
| `LOCKED` (after first turn) | ONNX-extracted mono 16 kHz | Embedding reused for all subsequent user turns until stream ends |
| `EXTERNAL` (when `reference_audio` was provided) | Raw audio, downmixed to mono 16 kHz | No enrollment phase; extraction applied from the first frame |

## Configuration knobs

`STT(...)` arguments:

| Name | Default | Purpose |
|------|---------|---------|
| `inner` | *(required)* | Any `stt.STT` with `streaming=True`. |
| `reference_audio` | `None` | Path to an enrollment WAV/PCM file. Bypasses the capture phase. |
| `onnx_dir` | `~/Public/projects/OpenSpeakerBeam-SS/exports` | Fallback directory for `.onnx` files (used only when the bundled model is missing). |
| `onnx_filename` | `speakerbeam_ss_trained_int8.onnx` (bundled) | Override the exact model filename. Looked up in the bundled `models/` directory first, then on disk. |
| `chunk_seconds` | `1.0` | ONNX window length. Must match the time axis baked into the export. |
| `sample_rate` | `16_000` | Sample rate for both extractor and the framework resampler. |
| `min_enrollment_seconds` | `0.5` | Refuse to lock enrollment from < this much captured audio. |
| `pass_through_first_utterance` | `True` | When `True`, the first turn reaches the inner STT untouched so the agent's first reply still makes sense. |
| `interim_results` | *(inherit from inner)* | Capability flag for downstream `VoicePipelineAgent`. |

Environment variables:

| Var | Purpose |
|-----|---------|
| `SPEAKERBEAMSS_ONNX_DIR` | Override `onnx_dir` for one-off runs. |

## Verified against

- LiveKit Agents `>= 1.5.13`
- OpenSpeakerBeam-SS commit exporting
  `speakerbeam_ss_trained_int8.onnx` (~13 MB) at opset 20, chunk length
  16 000 samples / 16 kHz — the bundle that ships inside the wheel.

## License

Apache-2.0 (same as LiveKit Agents).
