# hotato

> Find where your voice agent talks over callers, and keep it from coming back.
> Offline regression tests from your own call recordings, MIT and zero-install:
> score a real call recording, keep it as a fixture, fail CI when the timing
> regresses. Hotato measures whether the agent stopped talking when the caller
> started (a yield), how many seconds that took, and how many seconds both were
> talking at once (talk-over).
> Every failure points at a concrete fix. Runs entirely on your machine. Built
> to be called by an AI agent mid-task, as a CLI or a one-tool MCP server.
> Version 0.4.0. This file is the machine-readable index; `hotato describe
> --format json` emits the same information generated straight from the CLI's
> own argparse structure (every subcommand, its args, its exit codes, plus the
> schema URLs and the version), so it can never drift from the real flags.

## Run it (zero-install)

- The 5-minute path: `uvx hotato doctor --stereo call.wav` scores the recording,
  writes the visual HTML report, opens it. `uvx hotato doctor` runs the bundled
  self-test instead.
- See a catch: `uvx hotato demo` scores the packaged intentionally failing
  battery and opens its report (exit 0 by design; `--fail` returns the real
  regression exit code).
- Score the bundled battery: `uvx hotato run --suite barge-in` (barge-in: the
  caller starts talking while the agent is talking).
- Score one recording (two-channel WAV, caller and agent on separate channels):
  `uvx hotato run --stereo call.wav --stack livekit --format json`

## Every command (all offline; `--format json` on the scoring commands is for machines)

- `hotato run` -- score one dual-channel recording, or run the bundled
  synthetic self-test battery. `--stereo` (two-channel WAV) or `--caller` +
  `--agent` (aligned mono WAVs); `--suite barge-in` for the battery. Exit
  codes below.
- `hotato capture --stack {vapi,retell,twilio,livekit,pipecat} ...` -- pull a
  real dual-channel call from your stack and score it. Vapi/Retell/Twilio pull
  the recording for you (API key only, no SDK); LiveKit/Pipecat capture in
  your own infra (see `hotato setup`), then pass the file here. The only
  network call is the direct recording download; scoring itself is offline.
- `hotato setup --stack {livekit,pipecat}` -- print the copy-paste dual-channel
  recording scaffold for your stack, plus the command to score the result.
- `hotato connect <stack>` -- store a stack's credentials ONCE, locally
  (`~/.hotato/connections.json`, mode 0600, never sent anywhere but the
  vendor's own API), after a lightweight live auth check. Connectable stacks:
  vapi, retell, twilio, bland, elevenlabs, synthflow, millis, cartesia. After
  connecting, `--stack` and the key are optional for `pull`/`sweep` when one
  stack is connected.
- `hotato pull [--stack X] [--since 7d] [--limit 50] [--out DIR] [--allow-mono]`
  -- bulk-fetch your recent recordings by looping `capture`'s single-call fetch
  over the vendor's verified list endpoint, into a folder. Dual-channel stacks
  (vapi, twilio, retell) fetch stereo; mono/mixed stacks (bland, elevenlabs,
  synthflow, millis, cartesia) require `--allow-mono` and are indicative only.
  Retell has no verified list endpoint -- pull it from explicit `--call-id`
  values (never a fabricated endpoint). One bad call is a clean skip, never a
  crash. Platform payloads are treated strictly as untrusted data.
- `hotato sweep [--stack X] [--since 7d] ...` -- the flagship: `pull` recent
  recordings, then run the same zero-config `analyze` over them and write one
  self-contained offline dashboard of the ranked turn-taking moments across
  every real call. `--format json` for the ranked candidates plus a pull
  summary. Detail: `docs/CONNECT.md`; per-stack support: `docs/ADAPTER-STATUS.md`.
- `hotato doctor [--stereo call.wav]` -- the 5-minute path in one command:
  score (or self-test), render the self-contained HTML report, open it
  best-effort.
- `hotato demo [--fail]` -- run the packaged intentionally failing
  two-scenario battery (a talk-over and a false-stop-on-backchannel) and open
  its report; exits 0 by design unless `--fail` is passed.
- `hotato scan --stereo call.wav` -- list candidate turn-taking moments across
  a whole recording as timing facts only (overlap onsets, agent-starts-during
  caller-activity, long response gaps); you supply the yield/hold label. Long
  files are read in a windowed pass.
- `hotato analyze FOLDER` (or bare `hotato FOLDER`) -- zero-config drop-a-folder
  discovery: walk EVERY dual-channel recording label-free with `scan`, aggregate
  the candidate moments across all calls, and rank them by the scanner's own
  salience. Writes one self-contained offline HTML dashboard; for the top
  moments the real audio around each is embedded inline (base64, nothing
  uploaded) with a playhead that sweeps that moment's timeline in sync with
  playback, so you HEAR the overlap or gap land where the chart marks it.
  `--format json` for the ranked candidates. Measured candidate moments you
  review and label, never a pass/fail. Detail: `docs/ANALYZE.md`.
- `hotato fixture create --stereo bad-call.wav --id ID --onset SEC --expect
  {yield,hold} --out DIR` -- turn one bad call moment into a permanent
  regression fixture (scenario JSON + clipped two-channel WAV), validated by
  scoring it immediately.
- `hotato compare --before bad.wav --after fixed.wav --onset SEC --expect
  {yield,hold}` -- score a before/after take of the same moment with the
  identical expectation and bounds; one machine-stable result word: `fixed`,
  `regressed`, `improved`, `worse`, `unchanged`, `still_pass`, `not_scorable`.
- `hotato ingest` -- the composable passive on-ramp: wire a webhook to invoke
  it once and every completed call is scanned for candidate turn-taking
  moments automatically (parses the platform payload, reuses `capture`'s fetch,
  runs `scan`). Discovery only, never a pass/fail or an intent claim; never
  auto-labels, auto-fixtures, or auto-tunes. Not a daemon: you own the
  trigger. A webhook payload is treated strictly as untrusted data.
- `hotato report --stereo call.wav --out report.html [--format md] [--base
  base.json]` -- self-contained HTML (or Markdown) with per-event SVG
  timelines, analytics, a per-frame inspector, print CSS, and regression
  deltas against a saved baseline.
- `hotato team runs/ --html team.html [--max-response-gap SECONDS]` --
  aggregate a directory of run envelopes into pass rate over time plus
  mean/median/p90/p95 talk-over, time-to-yield, and response gap, and the
  most common failure class; `--max-response-gap` gates the pooled p95
  response gap as a latency SLA (exit 1 when exceeded).
- `hotato export --stereo call.wav --out research/ [--max-response-gap
  SECONDS]` -- write research CSVs (`events.csv`, `frames.csv`) plus
  `envelope.json`; columns documented in-file. Same `--max-response-gap` SLA
  gate as `team`.
- `hotato diagnose result.json` -- read-only: emit one diagnosis per failing
  event (finding, measured evidence, likely layer, `config_only_safe`, plain
  notes) plus a battery-level decision (for example
  `do_not_tune_single_threshold` when a battery misses a real interruption AND
  false-stops on a backchannel at once).
- `hotato inspect --stack {livekit,pipecat,vapi,retell} ...` -- read-only:
  fetch (Vapi, Retell) or statically parse (LiveKit, Pipecat) the turn-taking
  config a target is actually running, normalized into one model
  (`interrupt_min_words`, `interrupt_voice_seconds`, `resume_backoff_seconds`,
  `endpointing_wait_seconds`, `backchannel_aware`, raw fields, provenance).
  Only GETs; nothing is ever written back.
- `hotato plan result.json [target]` -- read-only: combines a diagnosis with
  the inspected config into a guarded fix-plan (schema
  `hotato/schema/fixplan.v1.json`, `hotato.fixplan.v1`). A change is proposed
  only when it maps cleanly to one setting, is one bounded step in an
  unambiguous direction, and the battery has a passing opposite-risk fixture;
  otherwise it downgrades honestly. `production_apply` is always `false`; no
  apply command exists.
- `hotato patch fixplan.json` -- render a fix plan into a LITERAL, paste-ready
  artifact per platform: a JSON merge-patch body plus a ready `curl` against the
  platform's real config-update endpoint (Vapi, Retell), or the exact source
  edit when the config lives in agent code (LiveKit, Pipecat). For the both-axes
  `do_not_tune_single_threshold` plan it emits NO config patch and prints the
  vendor-neutral, numbers-free engagement-control pointer instead. HONEST: patch
  PRODUCES the change; it never applies it and makes no network call.
- `hotato verify --before <old-run> --after <new-run>` -- battery-scale
  before/after proof after you applied a change and re-captured the failing
  fixtures: "N of M fixtures that used to fail now pass, and K of L hold fixtures
  still pass". Reuses the `compare` taxonomy per fixture and aggregate's pooled
  distributions. Reports COINCIDENCE, never causation; refuses the claim under
  low n (`--min-n`); an unjudgeable side is `not_scorable`. `--fail-on-regression`
  exits 1 on a regressed/worse fixture. Each side is a run envelope JSON or a
  directory of them.
- `hotato loop [FOLDER]` -- one-command orchestration of the closed loop with
  memory: first run discovers candidate moments (`analyze` -> `scan` -> rank) and
  records them in `.hotato/loop-state.json`; a second run, once you have labeled
  fixtures with `fixture create`, plans a guarded fix and points at `hotato patch`
  then `hotato verify`. Tracks state across runs; NEVER auto-labels (you supply
  every yield/hold intent) and NEVER auto-applies. Detail: `docs/FIX-LOOP.md`.
- `hotato benchmark --stack S --recordings DIR` -- score the battery you
  captured through your stack (see `setup` / `capture`), so result files are
  comparable across stacks and configs; scenarios without a matching
  recording are listed as not-captured, never scored as failures.
  `hotato benchmark compare a.json b.json` renders a side-by-side table of two
  or more result files.
- `hotato describe [--format json]` -- generated capability manifest of the
  whole CLI: every subcommand's name, purpose, argument list, and documented
  exit codes, plus the tool version and the schema URLs. Pure and
  deterministic; generated straight from the parser.
- Pytest plugin, auto-registered on install (`pytest11` entry point): a
  `hotato_score` fixture, plus `pytest --hotato-suite` to fail the session on
  a regression (`--hotato-suite-scenarios DIR --hotato-suite-audio DIR` for
  your own sets). Detail: `docs/PYTEST.md`.
- As a one-tool MCP server (stdio): `uvx --from "hotato[mcp]" hotato-mcp`
  exposes exactly one tool, `voice_eval_run`, returning the identical JSON
  envelope. Optional `report_path` also writes the HTML report; the envelope
  carries the path. NOTE (avoid the footgun): `uvx hotato-mcp` (no `--from`)
  FAILS -- uv then looks for a package literally named `hotato-mcp` on PyPI,
  which does not exist. Copy-paste client configs (Claude Desktop, Cursor,
  Codex CLI): `docs/MCP.md`.
- Tiered corpus suites (silver, silver-defects, gold, gold-defects; the defect
  suites fail on purpose and exit 1):
  `hotato run --suite barge-in --scenarios corpus/suites/gold/scenarios --audio corpus/suites/gold/audio`
- Additive scenario classes (`corpus/classes/`): mid-utterance-pause,
  backchannel-multilingual, noise-hold, telephony-degraded. Same deterministic
  build/`--check` contract as the suites above; detail in
  `corpus/classes/README.md`.
- PR check: copy `.github/workflows/hotato.yml`, unchanged, to
  `.github/workflows/hotato.yml` in your OWN repository, to score every pull
  request, post one self-updating results comment, and fail the job on a
  regression. `ci/github_action.yml` is hotato's own internal dev CI and is
  NOT a template -- do not copy it.

## What it returns

A single JSON envelope, the machine-readable result object (schema:
`https://hotato.dev/schema/envelope.v1.json`, `schema_version` "1",
additive-only). Per event, three timing signals: `did_yield` (did the agent
stop talking after the caller started), `seconds_to_yield` (seconds between
the caller starting and the agent stopping), `talk_over_sec` (seconds both
were talking at once), plus a namespaced `signals` block (`barge_in`,
`latency`, `echo`, `resume`) and a `verdict`. `signals.echo` (every event)
flags when the caller channel correlates with the agent's own audio (suspected
TTS bleed); `--echo-gate` on `hotato run` opts in to holding such a yield out
of the verdict instead of counting it as a clean pass. `signals.resume`
(yielded events only) reports whether, how fast, and with how long a run the
agent came back after it yielded, the timing fingerprint of a
restart-from-the-top.

Structured errors (any expected bad input: missing/mono/mismatched/not-found
file, unknown suite, ambiguous input mode, or a well-formed input with no
scorable event) share one shape across the CLI and the MCP tool: schema
`https://hotato.dev/schema/error.v1.json`, `ok: false`, a stable `error_code`,
and an actionable `message` -- never a raw uncaught exception.

Exit codes: `0` all pass, `1` a regression, `2` usage or IO error, or a single
recording that is not scorable. Not scorable means the recording cannot answer
the question, so no verdict is given: a silent caller channel, or an agent that
was not talking when the caller started, carries `scorable: false` and a plain
`not_scorable_reason`, never a fake pass or fail. Every failing event carries a
`fix`:

- `fix_class: "config"`: a named setting in your stack (livekit, pipecat, vapi,
  generic), which direction to move it, and what that trades away.
- `fix_class: "engagement-control"`: no timing threshold separates a
  backchannel (a short acknowledgement like "mhm") from a one-word command like
  "stop". The fix points, vendor-neutral, at a learned layer that decides
  whether the caller is asking to take the turn (engagement control /
  addressee detection), an open research problem.

Hotato does not infer intent. You label the expected behavior for the event:
yield means the agent should stop for the caller. hold means the agent should
keep speaking through a backchannel/noise/acknowledgement. Hotato then
measures whether the timing matched that label.

## Scope and method (each limit is a property of how it works)

- Reproducible timing against published thresholds. Every number is a
  measurement against a documented, overridable parameter, with an explicit
  ceiling; every frame is inspectable via `--dump-frames`.
- The detector reads audio energy over time, nothing else: it cannot tell who
  spoke or what was said. No speaker identification, diarization,
  transcription, or emotion detection.
- Two separate channels make overlap a fact of the recording. On a mono mix
  the scorer needs a caller-onset label and marks its overlap numbers as
  estimates.
- The bundled barge-in SUITE fixtures are synthetic: a deterministic floor and
  a regression guard. The `hotato demo` battery, by contrast, is two real
  operator-recorded calls (MIT-licensed). Real validity comes from your own
  labelled calls.

## Links

- README.md: quickstart and scope.
- docs/WHY.md: why timing bugs are invisible to text-level tests, and the
  "is this even a turn-taking bug" triage for conflated failure modes (STT
  hallucination, client-side buffering, LLM verbosity/tool-selection, safety
  false-refusal, wrong-language STT).
- METHODOLOGY.md: exactly how every number is computed, including the mean,
  median, p90, and p95 definitions (linear interpolation; rates are fractions,
  never a percentage), and the echo/resume cross-check methodology.
- docs/BAD-CALL-TO-CI.md: one bad call to a CI gate in five steps (fixture
  create, run, compare, the CI snippet, plan), with the use/do-not-use lists.
- docs/INGEST.md: the passive webhook-to-`ingest` on-ramp, per-stack payload
  fields, and why it composes `capture` + `scan` instead of adding new scope.
- docs/FIX-PLANS.md: the read-only diagnose/inspect/plan ladder.
- docs/FIX-LOOP.md: the closed loop -- patch (a plan into a paste-ready
  merge-patch/curl or source edit), verify (a battery-scale before/after proof),
  and loop (one-command orchestration that remembers where it left off).
- docs/ANALYZE.md: `hotato analyze FOLDER`, the zero-config drop-a-folder
  dashboard with the hear-the-bug audio playhead.
- docs/CONNECT.md: `hotato connect` / `pull` / `sweep`, the connect-once bulk
  pull-and-analyze recipe across your real calls, with per-stack credentials.
- docs/ADAPTER-STATUS.md: the honest per-stack map -- auto-pull dual-channel,
  mono-only (`--allow-mono`), capture-in-your-infra, not integrable, and
  unconfirmed -- each with the exact verified endpoint.
- docs/REPORTS.md: doctor, report, team, and export in depth.
- docs/PYTEST.md: the pytest fixture and the suite gate.
- docs/SUITES.md: the tiered corpus suites and the deterministic builder.
- docs/CI.md: the GitHub PR check and the sticky results comment.
- docs/MCP.md: the one-tool MCP server, copy-paste client configs (Claude
  Desktop, Cursor, Codex CLI), and the `uvx --from "hotato[mcp]" hotato-mcp`
  footgun to avoid.
- docs/API.md: the Python API and the shared success/error schema reference.
- docs/SUBMITTING.md: contribute a consented, labelled real call.
- CONTRIBUTING.md, docs/CORPUS-GOVERNANCE.md: contribute real labelled fixtures.
- CITATION.cff: how to cite hotato.
- llms-full.txt: this index plus the full text of every doc above,
  concatenated with file-boundary headers, for a single-fetch agent context
  dump. Built by `scripts/build_llms_full.py` (deterministic; run it again
  after editing any source doc).
- LICENSE: MIT. The open core is never relicensed.
