Metadata-Version: 2.4
Name: hotato
Version: 1.10.1
Summary: Regression testing for voice agents: turn timing scored from two-channel calls, exit 0/1 in CI; plus say-do verification from your traces and content-addressed failure records that reproduce byte-for-byte. Self-hosted, MIT.
Author: Attention Labs, Inc.
License-Expression: MIT
Project-URL: Homepage, https://hotato.dev
Project-URL: Documentation, https://github.com/attenlabs/hotato/tree/main/docs
Project-URL: Repository, https://github.com/attenlabs/hotato
Project-URL: Changelog, https://github.com/attenlabs/hotato/blob/main/CHANGELOG.md
Project-URL: Bug Tracker, https://github.com/attenlabs/hotato/issues
Project-URL: Open engine (upstream), https://github.com/quantumCF/voice-agent-barge-in-tests
Keywords: hotato,conversation-qa,voice-ai,voice-agents,ai-agents,voice-agent-testing,llm-eval,evals,regression-testing,ci,self-hosted,offline,barge-in,turn-taking,reliability,observability,mcp,agent-native,ollama,livekit,pipecat,vapi,retell,twilio
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS
Classifier: Environment :: Console
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Multimedia :: Sound/Audio :: Analysis
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: mcp
Requires-Dist: mcp>=1.2.0; extra == "mcp"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: jsonschema>=4; extra == "dev"
Requires-Dist: setuptools>=77; extra == "dev"
Provides-Extra: vapi
Provides-Extra: twilio
Provides-Extra: livekit
Requires-Dist: livekit; extra == "livekit"
Requires-Dist: livekit-agents; extra == "livekit"
Provides-Extra: pipecat
Requires-Dist: pipecat-ai; extra == "pipecat"
Provides-Extra: neural
Requires-Dist: onnxruntime>=1.16; extra == "neural"
Requires-Dist: numpy>=1.21; extra == "neural"
Provides-Extra: sign
Requires-Dist: cryptography>=41; extra == "sign"
Provides-Extra: diarize
Requires-Dist: pyannote.audio>=4.0; extra == "diarize"
Requires-Dist: torch>=2.8; extra == "diarize"
Requires-Dist: torchaudio>=2.8; extra == "diarize"
Requires-Dist: numpy>=1.21; extra == "diarize"
Provides-Extra: diarize-sortformer
Requires-Dist: nemo-toolkit[asr]>=2.7; extra == "diarize-sortformer"
Requires-Dist: torch>=2.8; extra == "diarize-sortformer"
Requires-Dist: numpy>=1.21; extra == "diarize-sortformer"
Provides-Extra: diarize-hosted
Requires-Dist: pyannoteai-sdk>=0.3; extra == "diarize-hosted"
Provides-Extra: transcribe
Requires-Dist: faster-whisper>=1.0; extra == "transcribe"
Provides-Extra: judge
Provides-Extra: all
Requires-Dist: mcp>=1.2.0; extra == "all"
Requires-Dist: livekit; extra == "all"
Requires-Dist: livekit-agents; extra == "all"
Requires-Dist: pipecat-ai; extra == "all"
Requires-Dist: onnxruntime>=1.16; extra == "all"
Requires-Dist: numpy>=1.21; extra == "all"
Requires-Dist: cryptography>=41; extra == "all"
Requires-Dist: pyannote.audio>=4.0; extra == "all"
Requires-Dist: torch>=2.8; extra == "all"
Requires-Dist: torchaudio>=2.8; extra == "all"
Requires-Dist: faster-whisper>=1.0; extra == "all"
Dynamic: license-file

<div align="center">

<img src="https://raw.githubusercontent.com/attenlabs/hotato/main/.github/assets/hotato-banner.svg" alt="hotato" width="442" style="max-width:100%;height:auto;">

<p>
<a href="https://pypi.org/project/hotato/"><img src="https://img.shields.io/pypi/v/hotato?style=flat-square&color=c23c07&label=pypi" alt="PyPI version"></a>
<a href="https://pypi.org/project/hotato/"><img src="https://img.shields.io/pypi/pyversions/hotato?style=flat-square&color=6f5d44" alt="Python versions"></a>
<a href="https://github.com/attenlabs/hotato/actions/workflows/tests.yml"><img src="https://github.com/attenlabs/hotato/actions/workflows/tests.yml/badge.svg?branch=main" alt="CI status"></a></p>

# hotato: regression testing for voice agents

**[hotato.dev](https://hotato.dev)**

</div>

**Your voice agent talked over the caller, and the transcript looks clean.** It also told the caller "you're all set" on a booking the backend never wrote. You find out when a customer complains, not in the pull request that shipped it, because the failure lives in the timing and the say-do gap, not in the words.

Hotato is self-hosted conversation QA for voice agents: it scores those failures from your own two-channel recording and turns each catch into a CI gate that runs offline. First catch in minutes, no account:

```console
$ uvx hotato start --demo
...
Conversation failed: Agent did not yield; measured talk-over was 0.25 s.
    talk-over     0.25s  seconds the agent kept talking while the caller held the floor
    response gap  2.18s  seconds of dead air from the caller's turn end to the reply
```

It measures turn timing and say-do, not intent.

## See the loop catch a regression

One recording in. The pinned failure becomes a CI gate:

```console
$ hotato investigate ./call.wav
  most likely failure: [1] t=7.63s agent_stop_no_caller
  next: hotato investigate label '.hotato/investigate-state.json#1' --expect yield

$ hotato investigate label '.hotato/investigate-state.json#1' --expect yield
created hotato contract: call-8s-yield  passed: False

$ hotato contract verify contracts
  [FAIL] call-8s-yield (expect yield): did_yield=False talk_over=0.00s
  0/1 contracts pass; exit_code=1
```

A committed contract is a pinned bad call: it is *meant* to stay exit `1` until you fix the agent and recapture, the same way a snapshot test stays red until you update the snapshot.

## Quickstart

The five commands are the whole path, first touch to a CI gate that guards every pull request:

```bash
# 1. see it catch a failure on two bundled calls (no account; this step exits 0)
uvx hotato start --demo
# 2. score your own two-channel recording
hotato investigate ./call.wav
# 3. commit the caught moment as a regression contract
hotato investigate label '.hotato/investigate-state.json#1' --expect yield
# 4. open the pull request that adds the CI gate
hotato pr create --fixtures contracts/<id>.hotato --repo OWNER/REPO --title 'Add hotato contract <id>'
# 5. the gate re-runs the stored evidence (exits 1 while the pinned call stays red)
hotato contract verify contracts/
```

Install with `pipx install hotato`, drive it over MCP with `uvx --from "hotato[mcp]" hotato-mcp`, or walk the path step by step in [`docs/GETTING-STARTED.md`](https://github.com/attenlabs/hotato/blob/main/docs/GETTING-STARTED.md).

## How it works

```text
two-channel recording
  ->  measure turn timing + verify say-do from the trace
  ->  content-addressed contract
  ->  CI verdict: exit 0 pass / exit 1 fail
```

A catch becomes a contract addressed by its own content, so the exact failure reproduces on any machine.

## What it scores

Each dimension scores on its own, then rolls up into one pass/fail verdict.

| Dimension | What it scores |
| :-- | :-- |
| ⏱️ **Speech** | Response latency and turn timing, measured from the two channels. |
| 💬 **Conversation** | Did the agent yield when the caller took the floor, and how fast. |
| 🎯 **Outcome** | Was the job done, judged on tool-call and state evidence. |
| 📋 **Policy** | Required disclosures and PII handling. |
| 📈 **Reliability** | `pass@1` / `pass@k` / `pass^k` with a Wilson interval. |

Timing comes straight from the two channels; say-do reads your `voice_trace.v1` spans, so what the agent told the caller is checked against what the backend did.

## Other ways to feed it

Every onramp feeds the same offline scoring and the same `0` / `1` / `2` exit contract.

**Traces you already log (no audio needed).** Wire the OTel spans you already emit into the same say-do check:

```bash
hotato trace ingest --otel traces.jsonl --out voice_trace.jsonl
hotato assert run --trace voice_trace.jsonl --transcript call.transcript.json --assertions assertions.yaml
```

Details: [`docs/ASSERTIONS.md`](https://github.com/attenlabs/hotato/blob/main/docs/ASSERTIONS.md) &#183; [`docs/TRACE.md`](https://github.com/attenlabs/hotato/blob/main/docs/TRACE.md) &#183; ground truth: [`examples/reference-agent`](https://github.com/attenlabs/hotato/blob/main/examples/reference-agent), a 375-run offline suite.

**Your stack's recorded calls.** Vapi, Twilio, and Retell fetch a separated two-channel file; everything scores offline afterwards.

```bash
hotato pull --stack vapi --limit 10
```

Details: [`docs/CONNECT.md`](https://github.com/attenlabs/hotato/blob/main/docs/CONNECT.md) &#183; drive a call against your live agent: [`docs/DRIVE-A-CALL.md`](https://github.com/attenlabs/hotato/blob/main/docs/DRIVE-A-CALL.md)

**Scripted fixtures (no production audio).** A deterministic scripted caller renders a `scenario.v1` labelled `origin=simulated`; a seeded replay is byte-identical.

```bash
hotato simulate --init demo.scenario.json && hotato simulate demo.scenario.json --out ./sim
```

Details: [`docs/SIMULATE.md`](https://github.com/attenlabs/hotato/blob/main/docs/SIMULATE.md)

## Point your agent at it

Point Claude Code, Cursor, or any coding agent at this repo: it reads [`AGENTS.md`](https://github.com/attenlabs/hotato/blob/main/AGENTS.md) and runs the spine itself, offline, no key.

```text
"Try hotato on the calls in ./recordings and add a CI gate that fails the build on a talk-over regression."
```

## Wire it into CI

The step's exit code **is** hotato's verdict:

```yaml
# .github/workflows/voice-qa.yml
name: voice qa
on: [pull_request]
permissions:
  contents: read          # read-only; runs fully offline
jobs:
  hotato:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: attenlabs/hotato@v1.10.1
        with:
          contracts: contracts/          # the catches you committed
          hotato-version: 1.10.1          # exact pin, never a range
```

<details>
<summary><b>Exit-code contract (gate on this, do not parse stdout)</b></summary>

| Exit | Meaning |
| :-: | :-- |
| `0` | every scorable event passed |
| `1` | a scorable event regressed |
| `2` | usage error or unusable input (bad flags, corrupt file, mono recording, or no scorable event) |

Copy-paste workflow with commit-SHA pin: [`docs/CI.md`](https://github.com/attenlabs/hotato/blob/main/docs/CI.md) &#183; [`docs/CONTRACTS.md`](https://github.com/attenlabs/hotato/blob/main/docs/CONTRACTS.md).

</details>

## Drive it over MCP

The MCP server from Quickstart exposes the `voice_eval_run` scorer plus read/verify/propose tools to any MCP client over local stdio. Setup: [`docs/MCP.md`](https://github.com/attenlabs/hotato/blob/main/docs/MCP.md).

## Specifications

| Property | Value |
| :-- | :-- |
| Footprint | ~10 MiB installed |
| Core dependencies | 0 (stdlib-only) |
| Reproducibility | byte-for-byte, content-addressed contract |
| Exit contract | `0` pass · `1` fail · `2` refuse |
| Release integrity | OIDC Trusted Publishing + build-provenance attested |
| Runtime | offline, off the production audio path |

## Verify the measurement yourself

<details>
<summary><b>Re-run the measurement benchmark</b></summary>

```bash
# re-run the measurement-error benchmark on the recorded AMI clips
PYTHONPATH=src python3 -m hotato.benchmark \
  --scenarios corpus/real/scenarios --audio corpus/real/audio
```

On 13 recorded AMI Meeting Corpus clips, the median error between measured caller-onset and the human word-alignment label is **20 ms**. Provenance and caveats: [`corpus/real/README.md`](https://github.com/attenlabs/hotato/blob/main/corpus/real) &#183; method: [`METHODOLOGY.md`](https://github.com/attenlabs/hotato/blob/main/METHODOLOGY.md) has the details.

</details>

<details>
<summary><b>Two channels, one party each</b></summary>

Timing between two voices is measurable only when they arrive on separate channels; a mono or mixed export is marked **NOT SCORABLE** and refused. It measures timing, not intent: a person labels each candidate moment yield or hold.

```bash
hotato trust --stereo call.wav        # per-channel activity, swap flag, scorability
```

</details>

## Contribute

Issues and PRs are welcome: [`CONTRIBUTING.md`](https://github.com/attenlabs/hotato/blob/main/CONTRIBUTING.md) &#183; [`SECURITY.md`](https://github.com/attenlabs/hotato/blob/main/SECURITY.md) &#183; [`CHANGELOG`](https://github.com/attenlabs/hotato/blob/main/CHANGELOG.md)

**Docs:** [`docs/GETTING-STARTED.md`](https://github.com/attenlabs/hotato/blob/main/docs/GETTING-STARTED.md) &#183; [`AGENTS.md`](https://github.com/attenlabs/hotato/blob/main/AGENTS.md) &#183; [`METHODOLOGY.md`](https://github.com/attenlabs/hotato/blob/main/METHODOLOGY.md) &#183; [`docs/START.md`](https://github.com/attenlabs/hotato/blob/main/docs/START.md) &#183; [`docs/CI.md`](https://github.com/attenlabs/hotato/blob/main/docs/CI.md) &#183; [`docs/CONTRACTS.md`](https://github.com/attenlabs/hotato/blob/main/docs/CONTRACTS.md) &#183; [`docs/MCP.md`](https://github.com/attenlabs/hotato/blob/main/docs/MCP.md)

## License

MIT ([`LICENSE`](https://github.com/attenlabs/hotato/blob/main/LICENSE))

mcp-name: io.github.attenlabs/hotato
