# hotato

> Find what broke in your agent calls. Pin it so it never ships again. Local
> call forensics and regression guards for AI agents, on your own machine.
> Point it at a recording and it prints the incidents it measured -- barge-in,
> talk-over, dead air, latency -- each with a timestamp, an id, and a
> self-contained HTML report. Pin one incident and it becomes a portable
> `.hotato` bundle -- the audio, the frames, the check; `hotato prove` re-runs
> that evidence in CI and fails the build when it stops holding.
>
> Timing comes from audio energy over time, one channel per party; say-do reads
> the transcript for what the agent said it would do, your trace and system of
> record for what it did. It measures turn timing and say-do, not intent.
> Runs offline, MIT, free at any scale, zero required dependencies, and nothing
> leaves the machine that invokes it. Built to be called by an AI agent
> mid-task, as a CLI or MCP server: `--format json` on every scoring command,
> exit codes 0 pass / 1 regression / 2 refuse to guess, one error shape.
>
> This file is the machine-readable index: what hotato measures, what it
> returns, and where the docs are. For the full command surface, run `hotato
> describe --format json`, generated from your own install's argparse tree.
>
> Version 1.18.0

## Start here

- Install: `pip install hotato` (or `uvx hotato ...` for zero-install).
- One recording to an incident list: `hotato autopsy ./call.wav`.
- Agent wiring a repo: AGENTS.md (the safe, ordered recipe + the machine contract).
- Human quickstart and scope: README.md.
- The whole command surface, machine-readable: `hotato describe --format json`.

## The loop, five commands

- `hotato autopsy RECORDING` -- one call in, every measured incident out
  (barge-in, talk-over, dead air, latency), plus a self-contained HTML incident
  report. Deterministic on dual-channel audio; best-effort with a measured
  confidence on mono. docs/AUTOPSY.md
- `hotato scan DIRECTORY` -- the folder health report: the Voice Stability
  Score out of 100 (dual-channel calls with no critical incident), counts by
  category, worst calls, recurrence states, per-call reports. docs/AUTOPSY.md
- `hotato <stack> health` -- `vapi`, `retell`, `bland`, `synthflow`, or
  `millis`: list your recent calls on the platform, pull each recording, check
  every one, print the health report. You never touch a WAV. Credentials
  resolve from `--api-key`, then the `hotato connect` store, then the stack's
  env var. docs/AUTOPSY.md, docs/CONNECT.md, docs/ADAPTER-STATUS.md
- `hotato pin apx-<id>#<rank>` -- the incident id autopsy printed becomes a
  portable `.hotato` failure contract: audio, frames, check. docs/CONTRACTS.md
- `hotato prove --contracts contracts/` -- composes everything you already ran
  (pinned checks, suites, before/after runs) into one proof: it passes only
  when every one passed, and a check whose evidence cannot back its claim exits
  non-zero as inconclusive, so CI never reads "could not tell" as green.

The deep toolkit lives under `hotato lab`, by job: check the audio, then score
the call (trust, run), grade a transcript (assert, rubric), add trace evidence
(trace), run scripted tests (simulate, suite), run the control plane (fleet).

## The evidence contract: four tiers, one policy

Every verdict states what evidence it stands on. Full text:
docs/EVIDENCE-CONTRACT.md.

- Tier 1, dual-channel audio (caller and agent separated at capture):
  deterministic. Overlap and timing are facts of the recording. The only tier a
  verdict may stand on; failure contracts, `pin`, `prove`, and the CI gate rest
  here, and only these calls count toward the Voice Stability Score.
- Tier 2, mono plus provider metadata (speaker-attributed transcript, pipeline
  trace, tool-call logs, diarizer turn labels): attributable. Each finding
  names the source that attributed it and how far that source is trusted, and
  never merges into tier 1's counts.
- Tier 3, raw mixed mono: symptom detection. Dead air and latency gaps, each
  with a measured confidence and its derivation printed beside it. A mono gap
  says everything stopped, not who stopped.
- Tier 4, insufficient evidence: refused (exit 2) with the reason and the next
  step. A refusal leaves no artifact and never becomes a number.

Reproducible measurement error: on 13 recorded AMI Meeting Corpus clips the
median caller-onset error against the human word-alignment label is 20 ms, with
a 0 to 1430 ms range; median time-to-yield error is 190 ms over 6 clips. The
median, not the mean, is the summary, because the errors have a long tail. One
10 ms frame hop is the measurement floor. Re-run it:
`PYTHONPATH=src python3 -m hotato.benchmark --scenarios corpus/real/scenarios --audio corpus/real/audio`.
Method: docs/BENCHMARK.md. Provenance and caveats: corpus/real/README.md.

The model judge is advisory: it reports in its own section, labelled as such in
every report, and gates an exit code only when you ask (`rubric run --gate`).

## What it returns

A single JSON envelope (schema `https://hotato.dev/schema/envelope.v1.json`,
`schema_version` "1", additive-only). Per event, three timing signals:
`did_yield` (did the agent stop talking after the caller started),
`seconds_to_yield` (seconds between the caller starting and the agent
stopping), `talk_over_sec` (seconds both were talking at once), plus a
namespaced `signals` block (`barge_in`, `latency`, `echo`, `resume`) and a
`verdict`.

Schemas ship in-package at `src/hotato/schema/*.v1.json`; validate `--format
json` output against them offline. The envelope carries `schema_version` and
`tool`, so a consumer maps those two fields to the right schema file.

Structured errors (missing, mono, mismatched, or not-found file; unknown suite;
ambiguous input mode; a well-formed input with no scorable event) share one
shape across the CLI and the MCP tool: schema
`https://hotato.dev/schema/error.v1.json`, `ok: false`, a stable `error_code`,
and an actionable `message` -- never a raw uncaught exception.

Exit codes: `0` all pass, `1` a regression, `2` usage or IO error or a
recording that is not scorable. Not scorable carries `scorable: false` and a
plain `not_scorable_reason`, never a fake pass or fail. Every failing event
carries a `fix`:

- `fix_class: "config"`: a named setting in your stack (livekit, pipecat, vapi,
  generic), which direction to move it, and what that trades away.
- `fix_class: "engagement-control"`: no timing threshold separates a
  backchannel ("mhm", not a bid for the turn) from a one-word command ("stop").
  The fix names the layer to add, vendor-neutral: one that decides whether the
  caller is asking for the turn (engagement control, or addressee detection).

You label the expected behavior for the event: `yield` means the agent should
stop for the caller, `hold` means it should keep speaking through a
backchannel. Hotato measures whether the agent's timing matched the label you
gave it.

## Scope and method

- Every number is a measurement against a documented, overridable parameter,
  with an explicit ceiling; every frame is inspectable via `--dump-frames`.
- The detector reads audio energy over time and nothing else: no speaker
  identification (a diarizer assigns anonymous SPEAKER_00/01, never a person),
  no transcription in the timing path, no emotion detection.
- Two separate channels are the gold reference and make overlap a fact of the
  recording. Mono runs through the tiers above.
- A timestamped transcript is also a full input: `hotato assert run
  --transcript call.transcript.json --assertions assertions.yaml` scores the
  words with no audio (phrase, PII, entity accuracy, latency). A PII hit
  reports the detector, the turn, and the role, and never echoes the value.

## The voice failure atlas (hotato.dev/atlas/)

A crawlable page per voice-agent failure mode that builders reported in
public, each one aggregating every independent report of the same mode with
the reporter's handle, the canonical link back to their post, a quote of at
most 30 words, and a status that moves only when a daily re-check stores the
evidence link, quote and date behind the move.

- https://hotato.dev/atlas/ -- the list of modes, filterable by status and stack.
- https://hotato.dev/atlas/feed.json -- every entry with its reports and status
  history (schema: https://hotato.dev/atlas/schema/feed.schema.json).
- https://hotato.dev/atlas/feed.atom -- the same entries as Atom 1.0.
- https://hotato.dev/atlas/atlas.llms.txt -- how to read an entry, plus terms.
- https://hotato.dev/atlas/remove/ -- the removal path: one email with the link,
  honored within 24 hours, tombstoned by id so nothing silently reappears.
- https://hotato.dev/atlas/sitemap.xml -- every atlas page with its lastmod.

Subscribe to feed.json rather than scraping the pages. Quote at most 30 words
and link the original. The handles are there to credit the reporter, not to
build an outreach list.

## Links

- README.md: quickstart and scope.
- AGENTS.md: the machine contract for an agent wiring a repo.
- docs/GETTING-STARTED.md: first touch to a CI gate, five steps.
- docs/AUTOPSY.md: `autopsy`, `scan`, the platform health commands, and `pin`.
- docs/EVIDENCE-CONTRACT.md: the four-tier evidence policy, stated once.
- docs/CONTRACTS.md: the portable, CI-enforced `<id>.hotato` failure contract.
- docs/CI.md: the GitHub PR check and the `hotato init ci` gate configs.
- docs/MCP.md: the hotato MCP server and copy-paste client configs. Start it
  with `uvx --from "hotato[mcp]" hotato-mcp`: `--from` names the package that
  provides the command, and the `[mcp]` extra adds the MCP SDK it imports.
- docs/CONNECT.md: connect-once bulk pull across your own calls, per-stack
  credentials.
- docs/ADAPTER-STATUS.md: the per-stack map, each with its verified endpoint.
- METHODOLOGY.md: exactly how every number is computed.
- docs/BENCHMARK.md: the AMI benchmark method; corpus/real/README.md: its
  provenance and caveats.
- docs/API.md and docs/SDK.md: the Python API and the typed SDK.
- llms-full.txt: the full text of every doc above, concatenated, for a
  single-fetch agent context dump. Built by `scripts/build_llms_full.py`.
- CITATION.cff: how to cite hotato.
- LICENSE: MIT. The open core is never relicensed.
