# hotato

> Find what broke in your agent calls. Pin it so it never ships again. Local
> call forensics and regression guards for AI agents, on your own machine.
> Point it at a recording and it prints the incidents it measured -- barge-in,
> talk-over, dead air, latency -- each with a timestamp, an id, and a
> self-contained HTML report. Pin one incident and it becomes a portable,
> content-addressed check; `hotato prove` re-runs the stored evidence in CI and
> fails closed.
>
> Timing comes from audio energy over time, one lane per party; completed
> actions come from your own trace and system of record. It measures turn
> timing and say-do, not intent. Runs offline, MIT, free at any scale, zero
> required dependencies, and nothing leaves the machine that invokes it. Built
> to be called by an AI agent mid-task, as a CLI or an MCP server: `--format
> json` on every scoring command, exit codes 0 pass / 1 regression / 2 refuse,
> one error shape.
>
> This file is the machine-readable index of the narrow story. For the full
> command surface, run `hotato describe --format json`: it regenerates straight
> from the CLI's argparse, so it never drifts from the real flags.
>
> Version 1.17.1

## Start here

- Install: `pip install hotato` (or `uvx hotato ...` for zero-install).
- One recording to an incident list: `hotato autopsy ./call.wav`.
- Agent wiring a repo: AGENTS.md (the safe, ordered recipe + the machine contract).
- Human quickstart and scope: README.md.
- The whole command surface, machine-readable: `hotato describe --format json`.

## The loop, five commands

- `hotato autopsy RECORDING` -- one call in, every measured incident out
  (barge-in, talk-over, dead air, latency), plus a self-contained HTML incident
  report. Deterministic on dual-channel audio; best-effort with a measured
  confidence on mono. docs/AUTOPSY.md
- `hotato scan DIRECTORY` -- the folder health report: the Voice Stability
  Score over dual-channel calls, per-category counts, worst calls, recurrence
  states, and per-call reports. docs/AUTOPSY.md
- `hotato <stack> health` -- `vapi`, `retell`, `bland`, `synthflow`, or
  `millis`: list your recent calls on the platform, pull each recording, check
  every one, print the health report. You never touch a WAV. Credentials
  resolve from `--api-key`, then the `hotato connect` store, then the stack's
  env var. docs/AUTOPSY.md, docs/CONNECT.md, docs/ADAPTER-STATUS.md
- `hotato pin apx-<id>#<rank>` -- one incident becomes a portable,
  content-addressed failure check. docs/CONTRACTS.md
- `hotato prove --contracts contracts/` -- composes every evidence lane you ran
  into one fail-closed proof: overall is pass only when every activated lane
  passed, and a lane that could not support its claim is inconclusive and exits
  non-zero, so CI never reads "could not tell" as green.

The deeper toolkit (run, trust, capture, connect, assert, contract, trace,
simulate, suite, rubric, fleet, serve) is listed in full by
`hotato describe --format json`.

## The evidence contract: four tiers, one policy

Every verdict states what evidence it stands on. Full text:
docs/EVIDENCE-CONTRACT.md.

- Tier 1, dual-channel audio (caller and agent separated at capture):
  deterministic. Overlap and timing are facts of the recording. The only
  verdict-eligible tier; contracts, `pin`, `prove`, and the CI gate stand here,
  and it alone enters the Voice Stability denominator.
- Tier 2, mono plus provider metadata (speaker-attributed transcript, pipeline
  trace, tool-call logs, diarizer turn labels): attributable. Each finding
  carries the declared authority of the source that attributed it and never
  merges into tier 1's counts.
- Tier 3, raw mixed mono: symptom detection. Dead air and latency gaps, each
  with a measured confidence and its derivation printed beside it. A mono gap
  says everything stopped, not who stopped.
- Tier 4, insufficient evidence: refused (exit 2) with the reason and the next
  step. A refusal leaves no artifact and never becomes a number.

Reproducible measurement error: on 13 recorded AMI Meeting Corpus clips the
median caller-onset error against the human word-alignment label is 20 ms, with
a 0 to 1430 ms range; median time-to-yield error is 190 ms over 6 clips. The
median, not the mean, is the summary, because the errors have a long tail. One
10 ms frame hop is the measurement floor. Re-run it:
`PYTHONPATH=src python3 -m hotato.benchmark --scenarios corpus/real/scenarios --audio corpus/real/audio`.
Method: docs/BENCHMARK.md. Provenance and caveats: corpus/real/README.md.

The model judge is advisory: it rides its own lane, is labelled as such in
every report, and cannot certify an outcome or change an exit code.

## What it returns

A single JSON envelope (schema `https://hotato.dev/schema/envelope.v1.json`,
`schema_version` "1", additive-only). Per event, three timing signals:
`did_yield` (did the agent stop talking after the caller started),
`seconds_to_yield` (seconds between the caller starting and the agent
stopping), `talk_over_sec` (seconds both were talking at once), plus a
namespaced `signals` block (`barge_in`, `latency`, `echo`, `resume`) and a
`verdict`.

Schemas ship in-package at `src/hotato/schema/*.v1.json`; validate `--format
json` output against them offline. The envelope carries `schema_version` and
`tool`, so a consumer maps those two fields to the right schema file.

Structured errors (missing, mono, mismatched, or not-found file; unknown suite;
ambiguous input mode; a well-formed input with no scorable event) share one
shape across the CLI and the MCP tool: schema
`https://hotato.dev/schema/error.v1.json`, `ok: false`, a stable `error_code`,
and an actionable `message` -- never a raw uncaught exception.

Exit codes: `0` all pass, `1` a regression, `2` usage or IO error or a
recording that is not scorable. Not scorable carries `scorable: false` and a
plain `not_scorable_reason`, never a fake pass or fail. Every failing event
carries a `fix`:

- `fix_class: "config"`: a named setting in your stack (livekit, pipecat, vapi,
  generic), which direction to move it, and what that trades away.
- `fix_class: "engagement-control"`: no timing threshold separates a
  backchannel ("mhm") from a one-word command ("stop"). The fix points,
  vendor-neutral, at a learned layer that decides whether the caller is asking
  to take the turn (engagement control / addressee detection).

You label the expected behavior for the event: `yield` means the agent should
stop for the caller, `hold` means it should keep speaking through a
backchannel. Hotato measures whether the agent's timing matched the label you
gave it.

## Scope and method

- Every number is a measurement against a documented, overridable parameter,
  with an explicit ceiling; every frame is inspectable via `--dump-frames`.
- The detector reads audio energy over time and nothing else: no speaker
  identification (a diarizer assigns anonymous SPEAKER_00/01, never a person),
  no transcription in the timing path, no emotion detection.
- Two separate channels are the gold reference and make overlap a fact of the
  recording. Mono runs through the tiers above.
- A timestamped transcript is also a full input: `hotato assert run
  --transcript call.transcript.json --assertions assertions.yaml` scores the
  words with no audio (phrase, PII, entity accuracy, latency). A PII hit
  reports the detector, the turn, and the role, and never echoes the value.

## The voice failure atlas (hotato.dev/atlas/)

A crawlable page per voice-agent failure mode that builders reported in
public, each one aggregating every independent report of the same mode with
the reporter's handle, the canonical link back to their post, a quote of at
most 30 words, and a status that moves only when a daily re-check stores the
evidence link, quote and date behind the move.

- https://hotato.dev/atlas/ -- the list of modes, filterable by status and stack.
- https://hotato.dev/atlas/feed.json -- every entry with its reports and status
  history (schema: https://hotato.dev/atlas/schema/feed.schema.json).
- https://hotato.dev/atlas/feed.atom -- the same entries as Atom 1.0.
- https://hotato.dev/atlas/atlas.llms.txt -- how to read an entry, plus terms.
- https://hotato.dev/atlas/remove/ -- the removal path: one email with the link,
  honored within 24 hours, tombstoned by id so nothing silently reappears.
- https://hotato.dev/atlas/sitemap.xml -- every atlas page with its lastmod.

Subscribe to feed.json rather than scraping the pages. Quote at most 30 words
and link the original. The handles are there to credit the reporter, not to
build an outreach list.

## Links

- README.md: quickstart and scope.
- AGENTS.md: the machine contract for an agent wiring a repo.
- docs/GETTING-STARTED.md: first touch to a CI gate, five steps.
- docs/AUTOPSY.md: `autopsy`, `scan`, the platform health commands, and `pin`.
- docs/EVIDENCE-CONTRACT.md: the four-tier evidence policy, stated once.
- docs/CONTRACTS.md: the portable, CI-enforced `<id>.hotato` failure contract.
- docs/CI.md: the GitHub PR check and the `hotato init ci` gate configs.
- docs/MCP.md: the hotato MCP server and copy-paste client configs. Start it
  with `uvx --from "hotato[mcp]" hotato-mcp`; a bare `uvx hotato-mcp` is a
  footgun that fails, because the entry point ships in the `[mcp]` extra.
- docs/CONNECT.md: connect-once bulk pull across your own calls, per-stack
  credentials.
- docs/ADAPTER-STATUS.md: the per-stack map, each with its verified endpoint.
- METHODOLOGY.md: exactly how every number is computed.
- docs/BENCHMARK.md: the AMI benchmark method; corpus/real/README.md: its
  provenance and caveats.
- docs/API.md and docs/SDK.md: the Python API and the typed SDK.
- llms-full.txt: this index plus the full text of every doc above, for a
  single-fetch agent context dump. Built by `scripts/build_llms_full.py`.
- CITATION.cff: how to cite hotato.
- LICENSE: MIT. The open core is never relicensed.
