Metadata-Version: 2.4
Name: hotato
Version: 0.2.3
Summary: Turn-taking regression tests for voice agents: score real call recordings for agents that talk over interruptions, stop for a mere 'mhm', or are slow to go quiet, and fail CI when the timing regresses. Offline, deterministic, MIT.
Author: Attention Labs, Inc.
License-Expression: MIT
Project-URL: Repository, https://github.com/attenlabs/hotato
Project-URL: Homepage, https://hotato.dev
Project-URL: Open engine (upstream), https://github.com/quantumCF/voice-agent-barge-in-tests
Keywords: hotato,voice-ai,voice-agents,evals,barge-in,turn-taking,mcp,agent-native,observability,reliability,livekit,pipecat,vapi
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Multimedia :: Sound/Audio :: Analysis
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: mcp
Requires-Dist: mcp>=1.2.0; extra == "mcp"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: jsonschema>=4; extra == "dev"
Requires-Dist: setuptools>=77; extra == "dev"
Provides-Extra: vapi
Provides-Extra: twilio
Provides-Extra: livekit
Requires-Dist: livekit; extra == "livekit"
Requires-Dist: livekit-agents; extra == "livekit"
Provides-Extra: pipecat
Requires-Dist: pipecat-ai; extra == "pipecat"
Provides-Extra: neural
Requires-Dist: silero-vad>=5; extra == "neural"
Requires-Dist: onnxruntime>=1.16; extra == "neural"
Requires-Dist: numpy>=1.21; extra == "neural"
Provides-Extra: all
Requires-Dist: mcp>=1.2.0; extra == "all"
Requires-Dist: livekit; extra == "all"
Requires-Dist: livekit-agents; extra == "all"
Requires-Dist: pipecat-ai; extra == "all"
Requires-Dist: silero-vad>=5; extra == "all"
Requires-Dist: onnxruntime>=1.16; extra == "all"
Requires-Dist: numpy>=1.21; extra == "all"
Dynamic: license-file

<p align="center">
  <a href="https://hotato.dev">
    <img src="https://raw.githubusercontent.com/attenlabs/hotato/main/.github/banner.png" alt="hotato: turn-taking regression tests for voice agents" width="840">
  </a>
</p>

<h1 align="center">
  <img src="https://raw.githubusercontent.com/attenlabs/hotato/main/docs/assets/mascot.svg" alt="" width="26" align="top"> hotato
</h1>

<p align="center"><b>Find interruption bugs in your voice agent before users do.</b></p>

<p align="center">
  <a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-blue.svg"></a>
  <img alt="Python 3.9 to 3.13" src="https://img.shields.io/badge/python-3.9%20to%203.13-blue.svg">
  <img alt="offline: yes" src="https://img.shields.io/badge/offline-yes-blue.svg">
  <img alt="runtime deps: zero" src="https://img.shields.io/badge/runtime%20deps-zero-blue.svg">
  <a href="https://github.com/attenlabs/hotato/actions/workflows/tests.yml"><img alt="tests" src="https://github.com/attenlabs/hotato/actions/workflows/tests.yml/badge.svg"></a>
</p>

Hotato is turn-taking regression tests for voice agents. It scores a call recording on your machine, so call audio never leaves it, and it catches the three failures callers feel most:

- **Talk-over**: the agent keeps talking while the caller is talking.
- **False stop**: the caller says a short acknowledgement like "mhm" (a backchannel, not a request to take over) and the agent stops mid-sentence.
- **Slow yield**: the caller starts talking and the agent takes too long to stop and let them speak.

Every failing event returns three measured signals (`did_yield`, `seconds_to_yield`, `talk_over_sec`) and a fix that names the exact setting to change in your stack.

## See it fail a bad agent

```bash
uvx hotato demo
```

![Hotato failing demo report](https://raw.githubusercontent.com/attenlabs/hotato/main/docs/assets/hotato-demo-report.png)

The demo scores a deliberately bad agent on synthetic audio, so you see the FAIL verdicts, the timelines, and the fix cards before touching your own calls.

## Score your own call

```bash
uvx hotato doctor --stereo your_call.wav   # two-channel WAV: caller ch0, agent ch1
```

`doctor` scores the call, writes the visual HTML report, and opens it. Or pull a recording straight from your stack:

- **Vapi**: `uvx hotato capture --stack vapi --call-id <id>` with `VAPI_API_KEY` set.
- **Twilio**: `uvx hotato capture --stack twilio --recording-sid RE...` with `TWILIO_ACCOUNT_SID` / `TWILIO_AUTH_TOKEN` set.
- **Retell**: `uvx hotato capture --stack retell --call-id <id>` with `RETELL_API_KEY` set; fetches the call's multichannel recording.
- **LiveKit / Pipecat**: `uvx hotato setup --stack livekit` (or `pipecat`) prints the recording config for your infra, then `hotato capture` scores the files it produces.

Input, stated once: Hotato needs the caller and the agent on separate audio tracks, as one two-channel WAV or two aligned mono WAVs. Per-stack details and verification dates: [`adapters/README.md`](adapters/README.md), [`docs/ADAPTER-STATUS.md`](docs/ADAPTER-STATUS.md).

## What Hotato measures

Three timing signals per event, measured frame by frame from the audio and re-derivable by hand with `--dump-frames`:

| Signal | What it answers |
| --- | --- |
| `did_yield` | did the agent stop talking after the caller started talking? |
| `seconds_to_yield` | how many seconds passed between the caller starting and the agent stopping? |
| `talk_over_sec` | for how many seconds were the caller and the agent talking at once? |

Every failing event carries exactly one fix, in one of two classes. Pass `--stack livekit`, `pipecat`, `vapi`, or `generic` and the fix uses your stack's own setting names:

| Failure | `fix_class` | The setting it names |
| --- | --- | --- |
| Missed a real interruption | `config` | interruption sensitivity: LiveKit `turn_handling` `interruption.min_duration` / `interruption.min_words`, Pipecat `VADParams(start_secs, stop_secs, confidence)`, Vapi `stopSpeakingPlan.numWords` |
| Slow yield | `config` | endpointing delay, how long the stack waits before deciding the caller is speaking or finished: LiveKit `endpointing.min_delay` / `endpointing.max_delay`, Pipecat `SpeechTimeoutUserTurnStopStrategy(user_speech_timeout=...)`, Vapi `stopSpeakingPlan.backoffSeconds` |
| Excess talk-over | `config` | overlap debounce: LiveKit `interruption.min_duration`, Pipecat `VADParams(stop_secs)`, Vapi `stopSpeakingPlan.voiceSeconds` |
| Yielded to its own echo | `config` | audio routing: echo cancellation and separate caller/agent channels |
| Stopped for a backchannel | `engagement-control` | no setting fixes this one: no timing threshold can tell "mhm" apart from a one-word "stop". If your stack has an interruption/backchannel classifier, use it; otherwise the fix is a learned layer that decides whether the caller is actually asking to take the turn (engagement control / addressee detection) |

Each `config` fix states which direction to move the setting and what that trades away. When one test set fails both ways at once (the agent missed a real interruption AND stopped for a backchannel), Hotato flags the pattern by name: a single sensitivity threshold trades those two failures against each other, so the fix is a classifier, not another threshold value.

## CI

```bash
uvx hotato run --suite barge-in --format json   # text output is for humans; json is for machines
```

The bundled `barge-in` suite scores recordings of callers barging in, that is, starting to talk while the agent is talking. Exit codes: `0` every event passed (or `--no-fail`), `1` a regression, `2` a usage or IO error, or a recording that is not scorable. Not scorable means the recording cannot answer the question (the caller channel is silent, or the agent was not talking when the caller started); the event reports `scorable: false` with the reason, never an invented verdict. Two ready-made gates: copy [`.github/workflows/hotato.yml`](.github/workflows/hotato.yml) for a PR check that posts one self-updating results comment ([`docs/CI.md`](docs/CI.md)), or add `--hotato-suite` to the pytest run you already have; the plugin registers itself on install and fails the session on a regression ([`docs/PYTEST.md`](docs/PYTEST.md)).

## What you get

- **`doctor`**: score a recording (or the bundled self-test), write the visual report, open it.
- **`report`**: self-contained HTML with per-event SVG timelines, a per-frame inspector, print CSS for PDF, and `--base base.json` regression deltas.
- **`team`**: aggregate a directory of runs into pass rate over time plus mean/median/p90 talk-over and time-to-yield.
- **`export`**: research-grade CSVs (`events.csv`, `frames.csv`, `envelope.json`), columns documented in-file.
- **`benchmark`**: score the battery you captured through each stack, then compare the result files side by side.
- **Pytest plugin**: a `hotato_score` fixture plus a session gate (`pytest --hotato-suite`).
- **MCP server**: one tool, `voice_eval_run`; pass `report_path` to also get the HTML report. `uvx --from "hotato[mcp]" hotato-mcp`
- **Tiered corpus suites**: 112 deterministic scenarios across silver and gold tiers, plus defect suites that fail on purpose to prove the scorer catches what it claims.

## Optional neural cross-check (verified, non-reference)

`pip install 'hotato[neural]'`, then `hotato run --stereo call.wav --backend neural` recomputes the same timing signals over speech regions found by Silero VAD, a small neural speech detector, instead of the default energy threshold. The ONNX weights ship inside the package and inference runs offline on CPU. Verified properties:

- **Same contract.** The neural run returns the identical result shape as the energy reference, on the same 10 ms frame grid, end to end.
- **Deterministic.** Repeated runs on the same audio are byte-identical.
- **The energy reference is untouched.** Installing the extra changes no energy number, and `--suite` always scores with energy (a note says so if you pass `--backend neural` there).
- **Built for real recordings.** The bundled fixtures are synthetic shaped noise rendered for the energy reference; a speech-trained model finds no speech in them, so run the cross-check on your own calls, where both tracks carry real speech.
- **Clear errors.** Without the extra, `--backend neural` exits with an explicit error, never a silent fallback to energy. Silero accepts 8000 Hz, 16000 Hz, and integer multiples of 16000 Hz; other rates get an actionable resample message.

Full method and the measured cross-check properties: [`METHODOLOGY.md`](METHODOLOGY.md).

## Install

`uvx hotato` runs every command with zero install. To add it to a project:

```bash
pip install hotato                 # core: stdlib-only, zero dependencies
pip install 'hotato[neural]'       # optional Silero VAD cross-check
pip install 'hotato[livekit]'      # LiveKit live capture
pip install 'hotato[pipecat]'      # Pipecat live capture
```

## More

- Reports and analytics: [`docs/REPORTS.md`](docs/REPORTS.md) · Pytest gate: [`docs/PYTEST.md`](docs/PYTEST.md) · Suites: [`docs/SUITES.md`](docs/SUITES.md) · CI recipes: [`docs/CI.md`](docs/CI.md)
- Turn a bad call into a regression test: [hotato.dev/docs/regression-loop.html](https://hotato.dev/docs/regression-loop.html)
- Python API: [`docs/API.md`](docs/API.md) · Stack benchmarks: [`docs/BENCHMARK-STACKS.md`](docs/BENCHMARK-STACKS.md)
- Adapters: [`adapters/README.md`](adapters/README.md) · Status and verification dates: [`docs/ADAPTER-STATUS.md`](docs/ADAPTER-STATUS.md)
- Why this exists: [`docs/WHY.md`](docs/WHY.md) · Method: [`METHODOLOGY.md`](METHODOLOGY.md) · Security: [`SECURITY.md`](SECURITY.md)
- Contributing: the highest-value PR is a real, labelled call fixture. Start at [`docs/SUBMITTING.md`](docs/SUBMITTING.md).

Why "hotato": good turn-taking is a game of hot potato. Literally: speak, then stop the moment the caller wants the turn. Hotato measures how fast and how cleanly your agent passes it. MIT licensed ([`LICENSE`](LICENSE)); the open core stays open.

mcp-name: io.github.attenlabs/hotato
