Metadata-Version: 2.5
Name: smart-turn-livekit
Version: 0.3.0
Summary: LiveKit Agents plugin for Smart Turn end-of-turn detection
Project-URL: Homepage, https://github.com/santhosh-005/smart-turn-livekit
Project-URL: Documentation, https://github.com/santhosh-005/smart-turn-livekit#readme
Project-URL: Issues, https://github.com/santhosh-005/smart-turn-livekit/issues
Project-URL: Source, https://github.com/santhosh-005/smart-turn-livekit
Project-URL: Smart Turn (upstream), https://github.com/pipecat-ai/smart-turn
Project-URL: LiveKit Agents, https://github.com/livekit/agents
Author: Santhosh
License-Expression: BSD-2-Clause
License-File: LICENSE
Keywords: conversational-ai,end-of-turn,endpointing,eot,livekit,livekit-agents,livekit-plugin,onnx,pipecat,realtime,semantic-vad,smart-turn,speech,turn-detection,turn-detector,turn-taking,vad,voice-agent,voice-ai
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: AsyncIO
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: huggingface-hub>=0.23
Requires-Dist: numpy>=1.24
Requires-Dist: onnxruntime>=1.16
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: soundfile>=0.12; extra == 'dev'
Provides-Extra: livekit
Requires-Dist: livekit-agents>=1.0; extra == 'livekit'
Requires-Dist: soxr>=0.3; extra == 'livekit'
Provides-Extra: resample
Requires-Dist: soxr>=0.3; extra == 'resample'
Description-Content-Type: text/markdown

# smart-turn-livekit

**LiveKit Agents plugin for Smart Turn end-of-turn detection.**

[![PyPI](https://img.shields.io/pypi/v/smart-turn-livekit)](https://pypi.org/project/smart-turn-livekit/)
[![Python](https://img.shields.io/pypi/pyversions/smart-turn-livekit)](https://pypi.org/project/smart-turn-livekit/)
[![License](https://img.shields.io/badge/license-BSD--2--Clause-blue)](LICENSE)

A LiveKit Agents plugin that runs [Smart Turn](https://github.com/pipecat-ai/smart-turn) models to detect when a user has finished speaking based on audio prosody. It handles the mel front-end, the ONNX session, resampling, per-model language gating, and the threading, so any checkpoint plugs into an `AgentSession` in one line.

Works with upstream's `smart-turn-v3`, a fine-tune, or your own ONNX file.
**Fully local** — CPU-only ONNX inference, no PyTorch, no inference service, and
no network call on the hot path. See [Offline and air-gapped](#offline-and-air-gapped).

```bash
pip install 'smart-turn-livekit[livekit]'
```

## Quickstart

```python
from livekit.agents import AgentSession
from livekit.plugins import silero
from smart_turn_livekit import SmartTurnDetector

session = AgentSession(
    vad=silero.VAD.load(min_silence_duration=0.25),   # must be >= 0.25
    turn_handling={
        "turn_detection": SmartTurnDetector(),         # smart-turn-v3
        "endpointing": {"min_delay": 0.3, "max_delay": 2.5},
    },
)
```

Weights download from Hugging Face on first use and are cached. Nothing to
configure, nothing bundled in the wheel.

Inspect each decision with `on_prediction`:

```python
SmartTurnDetector(
    on_prediction=lambda p: print(
        f"end of turn {p.probability:.1%} -> "
        f"{'complete' if p.complete else 'incomplete'} ({p.inference_s * 1000:.0f} ms)"
    )
)
```

```
end of turn 1.2%  -> incomplete (55 ms)
end of turn 94.5% -> complete   (58 ms)
```

## Models

```python
SmartTurnDetector()                                  # smart-turn-v3, default
SmartTurnDetector(model="smart-turn-tamil-tiny")     # a fine-tune
SmartTurnDetector(model_path="/opt/my-model.onnx")   # your own
```

| key | languages | size | accuracy |
|---|---|---|---|
| **[`smart-turn-v3`](https://huggingface.co/pipecat-ai/smart-turn-v3)** *(default)* | [multi](https://github.com/pipecat-ai/smart-turn#features) | 8.7 MB | 92.63% on upstream's benchmark |
| [`smart-turn-v3-fp32`](https://huggingface.co/pipecat-ai/smart-turn-v3) | [multi](https://github.com/pipecat-ai/smart-turn#features) | 32 MB | unquantised reference graph |

#### Language-specific fine-tunes

| key | language | size | accuracy |
|---|---|---|---|
| [`smart-turn-tamil-tiny`](https://huggingface.co/santhosh-005/smart-turn-tamil) | ta | 8.7 MB | 83.71% on real Tamil telephone speech |
| [`smart-turn-tamil-base`](https://huggingface.co/santhosh-005/smart-turn-tamil) | ta | 21 MB | 86.13% on the same |

`MODEL_REGISTRY` is the only place that knows one model from another, so adding
a checkpoint is a dict entry. Anything not listed still runs via `model_path=`.

## Latency

| threads | p50 | p95 |
|---|---|---|
| 1 | 84.5 ms | 91.9 ms |
| 2 | 53.1 ms | 61.7 ms |

`smart-turn-v3`, batch 1, inference only, idle Intel i5-12450H. **Measure on
your own box** — `python -m smart_turn_livekit.bench`. This number lands
directly on the gap between the user finishing and the agent replying, so
budget it against `min_delay`. The bench refuses to print a table without
warning you when machine load is high, because a benchmark taken under load
once produced a wrong shipping recommendation here.

## Gotchas

- **`min_silence_duration >= 0.25`.** LiveKit will not request a prediction
  below `MIN_SILENCE_DURATION_MS + 50` ms and raises at session start if the
  VAD is faster. A floor, not a ceiling — it fails loudly.
- **The language must be one the model claims, or unknown.**
  `supports_language()` returns `False` otherwise, using the languages the
  model was *benchmarked* on. This one fails **silently** — the detector is
  skipped and the agent falls back to fixed timing. Override with
  `languages=(...)`.

## Threshold

Under LiveKit the threshold picks *how long to wait*, not whether to end the
turn: below it the session takes `max_delay` instead of `min_delay`. So a false
`complete` costs a shorter pause rather than an interruption — a softer knob
than the same number under Pipecat.

```python
SmartTurnDetector(threshold=0.72)                       # one value
SmartTurnDetector(threshold={"en": 0.6, "ta": 0.85})    # per language
SmartTurnDetector(operating_point="polite")             # if the model ships one
```

0.5 is the default threshold. Some language-specific fine-tunes offer predefined operating points (like `"polite"` or `"balanced"`) that are calibrated on held-out evaluation sets to provide a different tradeoff between interruption rate and latency.

A dict is an override table, resolved on exact code, then base language, then
the operating point — so `{"en-US": 0.6}` answers for `en-US` only, while
`{"en": 0.6}` answers for `en`, `en-US` and `en-GB` alike.

## Outside LiveKit

The detector is a thin adapter over a plain class you can call directly:

```python
from smart_turn_livekit import SmartTurn

st = SmartTurn()                  # smart-turn-v3
st.probability(wave_16k)          # P(speaker has finished), 0.0 .. 1.0
st.is_complete(wave_16k)          # bool, against st.threshold
```

16 kHz mono float, last 8 seconds only. Other sample rates need `soxr`.

**Pipecat** needs no adapter — its built-in analyzer takes any Smart Turn ONNX,
and this package can still resolve the file for it:

```python
from smart_turn_livekit import resolve_model
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3

analyzer = LocalSmartTurnAnalyzerV3(smart_turn_model_path=str(resolve_model()))
```

## Offline and air-gapped

Inference is entirely local: ONNX Runtime on CPU, in your own process. There is
no inference service, no API key, and no network call on the prediction path.

The one network access is fetching the weights, which happens once on first use
and is then cached. Any of these removes it:

```python
SmartTurnDetector(model_path="/opt/models/smart-turn.onnx")   # ship the file yourself
SmartTurnDetector(local_only=True)                            # cached weights only, fail otherwise
```

or `SMART_TURN_MODEL=/path/to.onnx`, or `HF_HUB_OFFLINE=1`. Pre-cache the
weights in your image build and the running agent never reaches the network —
which is what a strict data-residency or air-gapped deployment needs.

An explicit `model_path` claims **no** language until you pass `languages=(...)`.

## How it works

LiveKit exposes two turn-detector protocols — one reads text, one reads audio
— as plain `typing.Protocol` classes with no base class to import. This
package implements the audio one structurally, so it's accepted by duck typing
alone; there is nothing to register.

One ONNX code path runs every registered model, because every Smart Turn
checkpoint shares the same signature: `input_features (batch, 80, 800)` in, a
`logits` output that is already a sigmoid out. Applying a second sigmoid to
that output gives confident, plausible, wrong answers, so `SmartTurn.probability`
checks the output is in `[0, 1]` and raises rather than return a
silently-broken score.

Inference runs in a single-thread executor so a ~10–30 ms ONNX call never
stalls the event loop. Construction resolves LiveKit's event class and runs one
dummy inference up front (`warmup=True`, default) — skipping that would put a
~1.7 s import penalty on the first prediction of the first call, a cost
LiveKit's own CLI normally hides by paying it earlier.

The mel front-end matches `WhisperFeatureExtractor(chunk_length=8)` and is
checked against Pipecat's own extractor to <1e-3.

## Tests

```bash
pip install -e '.[dev,livekit]'
pytest                                  # 26 tests; 3 skip without the research corpus
python -m smart_turn_livekit.bench
```

Checks features against Pipecat's extractor, the LiveKit Protocols by
`isinstance`, and that **every registered model loads and returns a
probability** — including upstream's, which we do not control.

## Limitations

- **Not a VAD.** It answers "did they finish?", not "is anyone talking?" You
  still need Silero upstream.
- **Audio-only, no language token.** Nothing stops any of these models running
  on any language — `supports_language()` reflects what was *measured*, not
  what is possible.
- **Written against `livekit-agents` 1.7.0.** `TurnDetectionEvent` and both
  turn detector Protocols are private (`_`-prefixed) there, so treat a
  minor-version bump as something to re-check. The test suite fails loudly if
  the shape moves.

## Author

Built and maintained by **Santhosh** ([@santhosh-005](https://github.com/santhosh-005)).

## Licence

**This package** is BSD-2-Clause. It bundles Whisper's mel filterbank (MIT,
OpenAI).

**The weights** are downloaded, not distributed here, and licensed separately.
`smart-turn-v3` is BSD-2-Clause by Daily.
