Metadata-Version: 2.4
Name: livekit-plugins-voicefilter-lite
Version: 0.1.0
Summary: LiveKit Agents plugin: streaming target-speaker enhancement via open-voicefilter-lite, drop-in for room_io.AudioInputOptions(noise_cancellation=...).
Author-email: linsan <bin.zaq@foxmail.com>
License: Apache-2.0
Project-URL: Homepage, https://github.com/linsan/livekit-plugin-voicefilter-lite
Keywords: livekit,livekit-agents,voicefilter,speaker-enhancement,noise-cancellation
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: livekit-agents<2.0,>=1.8
Requires-Dist: open-voicefilter-lite>=0.2.0
Dynamic: license-file

# livekit-plugins-voicefilter-lite

LiveKit Agents plugin that streams **target-speaker enhancement** in front of
an arbitrary streaming STT by wrapping [open-voicefilter-lite](https://pypi.org/project/open-voicefilter-lite/).
It slots in as a `livekit.agents.stt.STT` so it can sit **right next to the
speech recogniser**, getting read access to the underlying STT's
`SpeechEvent.alternatives[0].start_time` (sentence-start times) for snapping
the d-vector enrolment window to a word boundary.

The aim is to **reduce false barge-in / phantom recognitions in
multi-talker environments** by isolating the target speaker's voice
*before* the STT sees the audio. A two-phase startup handles the
cold-start problem:

1. **Buffering** — no target-speaker reference yet, so audio is
   forwarded to the underlying STT unchanged, *and* accumulated in a
   rolling PCM ring the plugin maintains.
2. **Enhancing** — once enough enrolment audio is collected, it's
   passed to `open_voicefilter_lite.StreamingVoiceFilter.set_enrollment`;
   from then on every frame is filtered through the mask predictor
   before reaching the underlying STT.

The "enough audio" check has two paths because phone-call scenarios
rarely give you 5 s of *continuous* target speech up front:

- **Single continuous slice** — the user speaks without pause for
  ≥ `enrollment_seconds`. The last N seconds of the rolling ring are
  fed to the d-vector encoder, optionally snapped to the most recent
  sentence start from STT transcripts so the window starts on a word
  boundary.
- **Multi-turn stitcher** (phone-call / short-utterance case) — each
  turn's transcribed audio range (read out of `SpeechData.start_time` /
  `end_time`) is sliced out of the rolling ring and appended to a
  separate segments pool. When the cumulative pool size reaches
  `enrollment_seconds`, the chunks are concatenated and passed to
  `set_enrollment`. This is what kicks in when the user only says a
  few words per turn and there is no pre-recorded enrolment sample.

## Install

```bash
# uv
uv add livekit-plugins-voicefilter-lite

# pip
pip install livekit-plugins-voicefilter-lite
```

## Quick start

```python
from livekit.agents import AgentSession, Agent
from livekit.plugins import deepgram, voicefilter_lite as vfl

stt = vfl.STT(
    underlying_stt=deepgram.STT(model="nova-2"),
    enrollment_seconds=5.0,
)

session = AgentSession(
    vad=...,                     # any VAD works
    stt=stt,                     # ← the plugin sits here
    llm=...,
    tts=...,
)
await session.start(agent=MyAgent(), room=ctx.room)
```

The plugin automatically harvests sentence-start times from the wrapped
STT's `SpeechEvent`s (`FINAL_TRANSCRIPT`, `INTERIM_TRANSCRIPT`,
`PREFLIGHT_TRANSCRIPT`). When the buffer hits 5 s and a usable sentence
boundary exists, enrolment happens on the next frame and subsequent
frames are enhanced.

If your provider's transcripts don't carry aligned timestamps — or you
want to anchor enrolment differently — feed sentence starts manually:

```python
stream = stt.stream()
async for ev in stream:
    if ev.type in (stt.SpeechEventType.INTERIM_TRANSCRIPT, stt.SpeechEventType.FINAL_TRANSCRIPT):
        for alt in ev.alternatives:
            if alt.start_time:
                stream.mark_sentence_start_at_seconds(alt.start_time)
```

Or supply a pre-recorded enrolment:

```python
import soundfile as sf
pcm, sr = sf.read("target_speaker.wav", dtype="float32")
# Triggered from the same event loop driving the stream:
stream = stt.stream()
stream.enroll_with_audio(pcm, sample_rate=sr)
```

### Phone-call / no-preroll scenario

The default `enrollment_seconds=5.0` is calibrated for the d-vector
quality floor, but it does not require 5 s of *continuous* target
speech. In a phone call there is no pre-recorded sample, and the
target speaker typically only says a few words per turn ("yes",
"book it", "Tuesday at three"). The wrapper handles this by
**stitching the d-vector across multiple turns**:

- Every `FINAL_TRANSCRIPT` (and `PREFLIGHT_TRANSCRIPT`) event whose
  alternatives carry both `start_time` and `end_time` has its
  transcribed range sliced out of the rolling PCM ring and appended
  to a separate segments pool.
- When the cumulative pool size reaches `enrollment_seconds` (e.g.
  after 4–6 short turns), the chunks are concatenated and passed to
  `StreamingVoiceFilter.set_enrollment`.
- Once enrolled, the wrapper starts enhancing every frame and the
  continuous ring stops mattering — subsequent barge-in attempts
  from off-target speakers are filtered out before the STT sees
  them.

If the user instead happens to talk nonstop for ≥ `enrollment_seconds`,
the continuous-slice path kicks in instead and the d-vector is built
from a single aligned slice. The two paths coexist; whichever fires
first wins.

## API

### `vfl.STT(...)` (alias for `VoicefilterLiteSTT`)

| Parameter                 | Type                  | Default | Purpose |
|---------------------------|-----------------------|---------|---------|
| `underlying_stt`          | `agents.stt.STT`      | —       | The real STT to wrap. Must support streaming. |
| `enrollment_seconds`      | `float`               | `5.0`   | How much audio to collect before the auto-enrol. |
| `pre_roll_max_seconds`    | `float`               | `30.0`  | Hard cap on the buffering ring; oldest samples are dropped. |
| `min_enrollment_seconds`  | `float`               | `1.0`   | Refuse to enrol until at least this much buffered. |
| `sample_rate`             | `int`                 | `16000` | Rate the enhancer operates at. The base `RecognizeStream` installs an `AudioResampler` to bring inbound frames up to this rate automatically. |
| `mask_model`              | `str` \| `None`       | bundled | Custom mask ONNX. |
| `speaker_encoder_model`   | `str` \| `None`       | bundled | Custom embedder ONNX. |

The wrapper inherits the underlying STT's `STTCapabilities` (interim
results, diarization, keyterms, chat_context) so the agent's downstream
behaviour is unchanged.

### `VoicefilterLiteStream`

Each `vfl.STT.stream()` call returns one of these. Useful accessors:

| Method / property                                                                              | Purpose |
|------------------------------------------------------------------------------------------------|---------|
| `state` (`"buffering"` \| `"enhancing"`)                                                      | Current phase. |
| `buffered_seconds` (`float`)                                                                  | How much audio is in the pre-enrol ring right now. |
| `is_enrolled` (`bool`)                                                                        | `True` once a d-vector is registered. |
| `mark_sentence_start(at_samples: int)`                                                        | Record a sentence-start offset in the pre-enrol ring. |
| `mark_sentence_start_at_seconds(t_seconds: float)`                                            | Same, but takes a session-clock time (the same clock STT emits `start_time` in). |
| `enroll_with_audio(pcm, *, sample_rate: int)`                                                 | Force-enrol with an explicit recording right now. |
| `clear_enrollment()`                                                                          | Drop the d-vector; future frames pass through unchanged. |

### `EnrollmentAccumulator`

A small standalone helper — handy in tests, or when only the alignment
math is needed without an STT. Keeps a rolling PCM ring and a list of
sentence-start offsets.

```python
acc = vfl.EnrollmentAccumulator(sample_rate=16000, max_seconds=30.0)
acc.add_frame_samples(samples_1d_float32)
acc.mark_sentence_start_at_seconds(0.4)
# … many frames later …
if acc.has_enrollment_audio:
    audio_1d = acc.take_enrollment(seconds=5.0)  # 16 kHz mono float32
```

## How it works

```
              ┌─────────────────── buffering ─────────────────────┐
              │                                                   │
   mic ──────▶│ RoomInput ──▶ VoicefilterLiteStream               │
              │ (frames → _input_ch)                              │
              │                │                                  │
              │   ┌─── passthrough ──────────────────────────┐    │
              │   │                                          ▼    │
              │   │                            underlying_stream ──▶ STT (provider)
              │   │                                ▲                │
              │   │                                │                ▼
              │   │       ┌── EnrollmentAccumulator (continuous    final / interim transcripts
              │   │       │   ring)                                back unchanged to the agent
              │   │       │   ← frames from _input_ch              │
              │   │       │   ← sentence-start markers from        │
              │   │       │     FINAL/INTERIM/PREFLIGHT events     │
              │   │       │                                        │
              │   │       └── segments pool (FIFO)                 │
              │   │           ← transcribed ranges only            │
              │   │             ([start_time, end_time]            │
              │   │              sliced per FINAL/PREFLIGHT)       │
              │   │                                                │
              │   ├─ path 1 ─ stitched segments ≥ 5 s ──┐         │
              │   │   (phone-call / short utterances)   │         │
              │   │                                       ▼         │
              │   │                            take_segments_enrollment
              │   │                                       │         │
              │   ├─ path 2 ─ continuous ring ≥ 5 s ─┐    │         │
              │   │   (user talking nonstop)         │    │         │
              │   │                                  ▼    ▼         │
              │   │                            take_enrollment     │
              │   │                              (align to marker)  │
              │   │                                       │         │
              │   └───────────────────────────────────────┘         │
              │                                                   │
              │                            ▼                      │
              │                 StreamingRecognitionEnhancer      │
              │                 (StreamingVoiceFilter.set_enrollment)
              │                            │                      │
              └────────────────────────────┴──────────────────────┘
                                              ▼
                                       enhanced frames → underlying stream
                                       (only target-speaker audio reaches STT)
                                       ↓
                                       fewer false barge-ins
```

### Why wrap the STT rather than plug into `noise_cancellation`?

LiveKit's `room_io.AudioInputOptions(noise_cancellation=...)` slot is a
purely-audio frame hook — it sees audio in, audio out, nothing else.
By wrapping the STT, we sit at the closest possible position to the
recogniser *and* gain read access to the upstream provider's
`SpeechEvent.alternatives[0].start_time`. That timestamp is the linchpin
of the enrolment-window alignment: it lets us trim the 5 s d-vector
clip to start on a sentence boundary rather than mid-word.

If your use case only needs to enhance audio (no STT-driven
enrolment alignment) you can also wire the bundled `StreamingVoiceFilter`
directly at the audio front-end — the plugin's core modules
(`EnrollmentAccumulator`, `StreamingRecognitionEnhancer`) are split out
so the PCM path is reusable independently.

## Notes / caveats

- **Sample-rate**: the shipped `open-voicefilter-lite` ONNX locks the
  model to 16 kHz. The wrapper installs an automatic `AudioResampler`
  on the stream — pass `sample_rate` only if your upstream is also 16 kHz
  (skipping the resampler shaves a few ms of latency per frame).
- **Latency**: per-step latency is one hop (10 ms at 16 kHz). The OLA
  inside `StreamingVoiceFilter` keeps `(L-1)·hop + n_fft ≈ 85 ms` of
  context. The d-vector extraction at enrolment runs once.
- **Effect when target is silent**: voicefilter-lite's adaptive
  suppression smoother `w` collapses towards `0` in non-speech regions,
  so the model harmlessly returns to passthrough behaviour when nobody
  is talking.
- **Streaming-only**: the wrapper requires `underlying.capabilities.streaming`.
  For non-streaming STTs, wrap them first with
  `livekit.agents.stt.StreamAdapter(stt=…, vad=…)`.
- **One-shot recognition**: `recognize()` (single-shot) is delegated
  verbatim — the d-vector lives on the streaming path.
