================================================================================
FILE: README.md
================================================================================

<p align="center">
  <a href="https://hotato.dev">
    <img src="https://raw.githubusercontent.com/attenlabs/hotato/main/.github/banner.png" alt="hotato: find where your voice agent talks over callers, and keep it from coming back" width="840">
  </a>
</p>

<h1 align="center">
  <img src="https://raw.githubusercontent.com/attenlabs/hotato/main/docs/assets/mascot.svg" alt="" width="26" align="top"> hotato
</h1>

<p align="center"><b>The open-source flight recorder for production voice agents.</b></p>

<p align="center">Find where your voice agent talks over callers, and keep it from coming back: turn a failed call into a portable contract with audio, timing, traces, trust checks, human labels, CI gates, and verified fix trials. MIT.</p>

<p align="center">
  <a href="https://pypi.org/project/hotato/"><img alt="PyPI version" src="https://img.shields.io/pypi/v/hotato.svg"></a>
  <a href="https://pypi.org/project/hotato/"><img alt="PyPI monthly downloads" src="https://img.shields.io/pypi/dm/hotato.svg"></a>
  <a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-blue.svg"></a>
  <img alt="Python 3.9 to 3.13" src="https://img.shields.io/badge/python-3.9%20to%203.13-blue.svg">
  <img alt="offline: yes" src="https://img.shields.io/badge/offline-yes-blue.svg">
  <img alt="runtime deps: zero" src="https://img.shields.io/badge/runtime%20deps-zero-blue.svg">
  <a href="https://github.com/attenlabs/hotato/actions/workflows/tests.yml"><img alt="tests" src="https://github.com/attenlabs/hotato/actions/workflows/tests.yml/badge.svg"></a>
</p>

## Start here (no account, no keys, no network)

```bash
uvx hotato start --demo
```

That sweeps two bundled recorded calls a provider's default agent failed, writes the dashboard, and turns one real missed-interruption candidate into a demo failure contract it immediately verifies:

```
[start] demo: swept 2 bundled calls, 5 candidate moments;
        wrote hotato-sweep.json, hotato-sweep.html, hotato-no-single-threshold.svg
        wrote contracts/demo-missed-interruption.hotato; verified contract: FAIL as expected
```

Open `hotato-sweep.html` for the ranked candidate moments with a hear-the-bug playhead. One of them is the failure below: the agent missed a real interruption in one call and false-stopped on a backchannel in another, so no single sensitivity dial fixes both. That is the card `start --demo` renders for you:

<p align="center">
  <img src="https://raw.githubusercontent.com/attenlabs/hotato/main/docs/assets/cards/no-single-threshold-card.svg" alt="Hotato threshold-funnel card: no single threshold can fix this. Missed a real interruption; false-stopped on a backchannel. One sensitivity dial cannot satisfy both axes at once." width="760">
</p>

A real failure became a candidate, became a portable `.hotato` contract, and `contract verify` catches it. Hotato prints the exact next commands to promote it into a permanent fixture, run it in CI, and re-verify.

## Choose your path

| You want to | Run this |
| --- | --- |
| Try the full loop, no credentials | `uvx hotato start --demo` |
| Sweep the bundled demo calls | `uvx hotato sweep --demo` |
| Sweep recent calls from a real stack | `hotato connect vapi` then `hotato sweep --stack vapi --since 7d` |
| Add Hotato to an existing repo, CI gate included | `hotato init starter --stack vapi --out .` ([`docs/STARTER.md`](docs/STARTER.md)) |
| Turn a confirmed failure into a portable contract | `hotato contract create --from-candidate hotato-sweep.json#1 --expect yield --id refund-cutoff-001 --out contracts` ([`docs/CONTRACTS.md`](docs/CONTRACTS.md)) |
| Verify contracts in CI | `hotato contract verify contracts/ --junit contracts-junit.xml` |
| Attach observability traces to a contract | `hotato trace attach contracts/refund-cutoff-001.hotato --trace voice_trace.jsonl` ([`docs/TRACE.md`](docs/TRACE.md)) |
| Prove a candidate fix, before/after, fail-closed | `hotato fix trial patch.json --name staging-x --before before/ --after after/` ([`docs/FIX-TRIAL.md`](docs/FIX-TRIAL.md)) |
| Share a finding in a PR or slide | `hotato card hotato-sweep.json#1 --out finding.svg` |
| Drive it from a coding agent | `uvx --from "hotato[mcp]" hotato-mcp` (one tool, `voice_eval_run`; configs in [`docs/MCP.md`](docs/MCP.md)) |

Every command above takes a two-channel recording (caller on one channel, agent on the other). A mono file or a bad export is marked NOT SCORABLE, never turned into a confident but meaningless verdict.

## The loop

Catch it, confirm it, gate it, keep it gone:

1. **Sweep** surfaces candidate talk-over and false-stop moments across your recent calls, ranked by how far the timing missed.
2. **You label** one (`yield` = stop for the caller, `hold` = keep talking through a backchannel) and `fixture promote` saves it as a permanent regression test.
3. **CI runs** that fixture on every change and exits non-zero the moment the timing regresses, so the bug a caller already felt cannot come back unnoticed.

Hotato measures whether the agent stopped talking when the caller started, how many seconds that took, and how many seconds both were talking at once. It reports what it measured, never a guess at intent.

## Connect a production stack

The demo needs nothing. To point Hotato at real calls, connect once, then sweep on a schedule:

```bash
hotato connect vapi                                             # credentials stored 0600, local only
hotato sweep --stack vapi --since 7d --out hotato-sweep.html    # cron, CI, wherever
```

Run `sweep` on a timer and it becomes a passive monitor. Your audio stays on your machine unless you explicitly pull it from your stack. Full guide: [`docs/SET-AND-FORGET.md`](docs/SET-AND-FORGET.md) · runnable [`examples/set-and-forget/`](examples/set-and-forget/README.md).

## What Hotato is not

- **Not a full QA platform.** It does not grade the whole conversation, task
  success, or content. See [`docs/COMPARE.md`](docs/COMPARE.md) for where
  Hamming, Cekura, Coval, Bluejay, Roark, Vapi, and Retell fit instead.
- **Not transcript scoring.** It measures audio timing, not what was said.
- **Not speaker ID.** Channels are anonymous; nothing identifies who a person is.
- **Not semantic intent detection.** It produces candidate timing evidence.
  Humans label intent. CI enforces confirmed contracts.
- **Not a hand on production config.** It never sits in the live audio path
  and never changes a running agent.

## Install

`uvx hotato` runs any command with zero install. To add it to a project:

```bash
pip install hotato                 # core: stdlib-only, zero dependencies
pip install 'hotato[neural]'       # optional Silero VAD cross-check
pip install 'hotato[livekit]'      # LiveKit live capture
pip install 'hotato[pipecat]'      # Pipecat live capture
```

## Depth

- **Set-and-forget monitoring** (connect once, sweep on a schedule, promote confirmed bugs into fixtures): [`docs/SET-AND-FORGET.md`](docs/SET-AND-FORGET.md) · runnable [`examples/set-and-forget/`](examples/set-and-forget/README.md)
- **Bad call to CI regression test**, step by step: [`docs/BAD-CALL-TO-CI.md`](docs/BAD-CALL-TO-CI.md) · runnable [`examples/bad-call-to-ci/`](examples/bad-call-to-ci/README.md)
- **What it measures** (the three timing signals, re-derivable by hand): [`METHODOLOGY.md`](METHODOLOGY.md) · Python API [`docs/API.md`](docs/API.md)
- **The fix ladder** (each failure names a likely fix class; when the evidence maps cleanly to stack config, Hotato names the setting family and direction): [`docs/FIX-PLANS.md`](docs/FIX-PLANS.md)
- **Rule out the non-turn-taking bugs first** (STT, buffering, verbosity, refusals, wrong-language): [`docs/WHY.md`](docs/WHY.md)
- **Pull a call from your stack** (Vapi, Twilio, Retell, LiveKit, Pipecat): [`adapters/README.md`](adapters/README.md) · status [`docs/ADAPTER-STATUS.md`](docs/ADAPTER-STATUS.md)
- **CI gates**: GitHub Action [`docs/CI.md`](docs/CI.md) · pytest plugin [`docs/PYTEST.md`](docs/PYTEST.md)
- **Recorded-call battery**: 12 scripted calls against a live voice agent on its provider's default settings, where a missed interruption and a false stop on a backchannel fail in the same run, so `diagnose` refuses to name one threshold: [`corpus/vapi-defaults/README.md`](corpus/vapi-defaults/README.md)
- **Failure contracts and traces**: turn a labelled candidate into a portable, CI-verified bundle and attach observability evidence: [`docs/CONTRACTS.md`](docs/CONTRACTS.md) · [`docs/TRACE.md`](docs/TRACE.md) · [`docs/OTEL.md`](docs/OTEL.md)
- **Root cause and a proven fix**: `hotato explain` turns a failing result into root-cause-by-layer evidence, and `hotato fix trial` proves a candidate change before/after, fail-closed: [`docs/EXPLAIN.md`](docs/EXPLAIN.md) · [`docs/FIX-TRIAL.md`](docs/FIX-TRIAL.md) · [`docs/APPLY.md`](docs/APPLY.md) · [`docs/FIX-LOOP.md`](docs/FIX-LOOP.md)
- **Evidence**: what Hotato validates, the input-condition trust matrix, every card and CLI block reproducible, and where Hotato does and doesn't fit next to Hamming/Cekura/Coval/Bluejay/Roark/Vapi/Retell: [`docs/VALIDATION.md`](docs/VALIDATION.md) · [`docs/TRUST-MATRIX.md`](docs/TRUST-MATRIX.md) · [`docs/GALLERY.md`](docs/GALLERY.md) · [`docs/EVIDENCE-PACK.md`](docs/EVIDENCE-PACK.md) · [`docs/COMPARE.md`](docs/COMPARE.md)
- **For coding agents**: [`AGENTS.md`](AGENTS.md) · [`llms.txt`](llms.txt) · [`llms-full.txt`](llms-full.txt) · MCP server [`docs/MCP.md`](docs/MCP.md) · Security [`SECURITY.md`](SECURITY.md)
- **Contributing**: the highest-value PR is a labelled call fixture: [`docs/SUBMITTING.md`](docs/SUBMITTING.md)

Why "hotato": good turn-taking is a game of hot potato. Speak, then pass the turn the moment the caller wants it. MIT licensed ([`LICENSE`](LICENSE)); the open core stays open.

mcp-name: io.github.attenlabs/hotato


================================================================================
FILE: docs/WHY.md
================================================================================

# Why Hotato

## Voice agents fail in ways your tests do not catch

Your test suite checks what the agent says. Production calls fail on when it
speaks. Four timing failures show up again and again in real transcripts, and
all four are invisible to text-level tests. Three are interruption patterns
(missed interruption, false stop, slow yield); the fourth is an endpointing
gap:

1. **Missed interruption.** The caller says "stop, take that off" and the agent
   keeps talking. `did_yield` is false where it must be true. The caller
   repeats themselves, louder, then hangs up.
2. **False stop.** The caller says "mhm", a backchannel: a short acknowledgement,
   not a request to take over. The agent stops mid-sentence anyway. `did_yield`
   is true where it must be false. The call stalls, then restarts, then stalls
   again.
3. **Slow yield.** The caller interrupts and the agent does stop, eventually.
   Every second of `talk_over_sec` before it stops is the caller and the agent
   speaking at once, and the caller hears all of it.
4. **Endpointing misses.** Endpointing is detecting that the caller finished
   speaking. When it misses, the caller gets dead air after they finish
   (`response_gap_sec`), or the agent starts before they are done
   (`premature_start_sec`). Both read as a broken conversation partner. The
   silence ambiguity underneath (thinking, distracted, or gone quiet) is not
   something Hotato resolves either: it measures how long the silence lasted,
   never what it meant.

Each of these lives entirely in the audio timing of the call. A transcript
diff, an LLM judge on text, and a unit test on the agent's reply all score a
call with any of these failures as perfect. The transcript reads clean; the
caller called back anyway. That gap, between a passing text-level check and a
caller who came back unhappy, is exactly what these four patterns hide.

Hotato does not infer intent. You label the expected behavior for the event:
yield means the agent should stop for the caller. hold means the agent should
keep speaking through a backchannel/noise/acknowledgement. Hotato then
measures whether the timing matched that label.

## Is this even a turn-taking bug?

In our observed reports, many alleged barge-in bugs turn out not to be
turn-taking bugs at all. Each of these produces a symptom that reads exactly like a
missed interruption or a false stop, but the fix lives in a different layer,
and no VAD threshold will ever touch it:

- **STT hallucination.** The transcript has words the caller never said, or
  drops words that mattered, so the agent responds to something that was not
  actually said. That looks like an interruption the agent ignored. Reach
  for: an STT/ASR word-error-rate check against the raw audio, not a
  turn-taking scorer.
- **Client-side audio buffering.** The caller's own device or browser queues
  outgoing audio before it reaches the agent, so "the agent talked over me"
  is audio arriving late, not the agent failing to yield. Reach for:
  client/WebRTC jitter-buffer and network-latency instrumentation, not an
  interruption-sensitivity setting.
- **LLM verbosity or tool-selection.** The agent "kept talking through the
  interruption" because it was mid-tool-call or committed to finishing a long
  generation before it re-checked for a stop signal. Reach for: response-length
  and tool-call latency tracing in your agent framework, not a VAD setting.
- **Safety false-refusal.** The agent stops abruptly mid-sentence because a
  moderation or safety layer cut it off, not because it heard a barge-in or a
  backchannel. Timing-wise this is indistinguishable from a false stop on
  "mhm". Reach for: your safety/moderation logs, not an engagement-control
  layer.
- **Wrong-language STT.** The caller is speaking a language or accent the STT
  covers poorly, recognition comes back empty or garbled, and the agent's
  response looks unrelated or missing entirely. That reads as a missed
  interruption; it is a language-coverage gap. Reach for: per-locale STT
  accuracy tooling. (Hotato's own detector is energy-based and does not
  detect language either, by design: see `corpus/classes/README.md`.)

This is both an honesty guard and a shortcut. If your bug matches one of
these five, Hotato will not find it no matter how you tune it, because it is
not a timing bug; you will save the day you would have spent staring at
`turn_end_silence_sec`. If it matches none of them: two common complaints are
agent-talks-over-caller and false-stop-on-backchannel, and that is exactly what
Hotato measures, with the funnel (no single config value fixes both directions
at once) proven on real recorded calls, not synthetic fixtures
(`corpus/vapi-defaults/README.md`).

## What makes Hotato different

- **It scores recordings you already have.** A dual-channel WAV from your
  stack is the whole input. No test harness, no synthetic caller, no
  re-architecture.
- **It runs offline.** Scoring is local and deterministic; no audio,
  transcript, or result leaves your machine.
- **It emits machine-readable timing.** One JSON envelope per run, the same
  shape from the CLI, the MCP tool, and the pytest fixture, so an agent or a
  CI job consumes one schema.
- **It fails CI.** Exit code 1 on a regression, 0 on pass, 2 on a usage error
  or a recording that is not scorable: the recording cannot answer the
  question (the caller channel is silent, or the agent was not talking when
  the caller started), so no verdict is given. A turn-taking regression blocks
  a merge the same way a failing unit test does.
- **It routes every failure to a fix class.** When the failure maps cleanly to
  stack config, `config` names the setting family on your stack and the
  direction to investigate. `engagement-control` tells you
  no threshold value can fix this failure, because telling "mhm" apart from
  "stop" is a classification problem, so you stop burning days retuning a
  setting that cannot win.

## What it does not do

No transcription. No speaker identification (a diarizer assigns anonymous
SPEAKER_00/01; it never says who a person is). No emotion or intent detection.
Hotato measures energy over time on two channels; a single-channel (mono)
recording is scorable via the opt-in, quality-gated `[diarize]` front-end
(`hotato run --mono call.wav --diarize`), labeled indicative below the
confidence bar and never equivalent to a true dual-channel measurement. The
method and its ceiling are stated in [METHODOLOGY.md](../METHODOLOGY.md) and in
the `limits` block of every result.

## Why no accuracy score?

Because accuracy would hide the thing you need to debug. A single blended
percentage tells you nothing about which call failed, on which axis, or what
to change. Hotato reports three direct measurements per event instead:

- `did_yield`: did the agent stop talking after the caller started
- `seconds_to_yield`: seconds between the caller starting and the agent stopping
- `talk_over_sec`: seconds the caller and the agent spoke at the same time

Every number is reproducible from the frame dump by hand, every threshold is
exposed, and a failure points at its fix. That is what you debug with.

Ready to pin a real failure? The loop from one bad call to a CI gate, with
`hotato fixture create`, `compare`, and `plan`:
[BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md).


================================================================================
FILE: METHODOLOGY.md
================================================================================

# Methodology

How every number this tool reports is computed, why it is reproducible, and
where the method stops being trustworthy. There is no accuracy percentage in
this document and none is implied anywhere in the tool. What follows describes a
measurement, not a claim about any detector's internal quality.

Hotato scores the audio *timing* of turn-taking from a call recording: whether
the agent stopped talking once the caller started (a yield), how many seconds
that took, how many seconds it kept talking while the caller was talking
(talk-over), and, on the same two tracks, the endpointing timing: how the
agent's response sits around the moment the caller finished speaking. It
measures energy over time. It does not understand speech.

## Read this first: the honest ceiling

The limits come first because they bound everything below.

- **Energy is not intent.** The detector marks a frame "active" when its
 short-time energy crosses a threshold. A cough, a slammed door, or a burst of
 line noise reads as active exactly like a word does. Nothing here recovers
 *meaning*, it recovers *when there was speech-level energy*.
- **You supply the label.** Hotato does not infer intent. You label the
 expected behavior for the event: yield means the agent should stop for the
 caller. hold means the agent should keep speaking through a
 backchannel/noise/acknowledgement. Hotato then measures whether the timing
 matched that label. "mhm" and "stop" can carry identical speech energy; the
 verdict is only ever your label checked against measured timing.
- **Sub-second single-channel boundaries sit near the resolution limit.** At the
 default 10 ms hop a boundary is located to within a frame or two, but the
 hangover that keeps an utterance whole (default 150 ms) smears every trailing
 edge by up to that much, and on one mixed channel deciding *whose* energy
 crossed the threshold is not always possible.
- **Two channels is the gold reference; mono is scorable, quality-gated.** The
 reference input is separated caller and agent tracks: one two-channel WAV or
 two aligned mono WAVs, where each channel is one party and overlap is a fact of
 the recording (both tracks active at once, by construction). A single mixed
 mono call is scorable via the opt-in `[diarize]` front-end (`hotato run --mono
 call.wav --diarize`): a diarizer separates the mix into caller/agent activity,
 which is reconstructed and scored through the same path. It is quality-gated --
 above the confidence bar the verdict is labeled `diarized-mono`; below it, the
 verdict is labeled indicative only and no SLA gate fires; a non-separable file
 is refused. Diarized mono never equals a true dual-channel recording for
 sub-second talk-over attribution, and the gate is what keeps that honest per
 file.
- **Out of scope, permanently:** no speaker identification (a diarizer assigns
 anonymous SPEAKER_00/01; it never says who a person is), no speech-to-text, no
 emotion or intent detection, and no claim about any vendor's internal accuracy.
 Word-level semantics are not measured; only the timing of the yield is.

These same limits are emitted inline in the `limits` block of every JSON result
(`core.py`) and in the MCP tool description, so a consuming agent sees them.

## The pipeline, step by step

Everything runs offline, standard-library only (NumPy only accelerates the
per-frame RMS when importable; results are identical without it). Scoring is
deterministic: the same audio and `ScoreConfig` give the same output.

### Default parameters

Every value below is read directly from `ScoreConfig` (`score.py`) and
`VADParams` (`vad.py`); the same defaults apply to both channels unless overridden.

Framing and VAD (`VADParams`, applied per channel):

| Parameter | Default | Meaning |
|--------------------|-----------|---------|
| `frame_ms` | 20.0 ms | analysis window length for one RMS frame |
| `hop_ms` | 10.0 ms | step between frames (the time resolution) |
| `noise_percentile` | 0.10 | quietest 10% of frames estimates the noise floor |
| `rel_db` | 15.0 dB | a frame is active this many dB above the floor |
| `abs_gate_db` | -60.0 dBFS| nothing below this absolute level is ever active |
| `dyn_margin_db` | 22.0 dB | on a rarely-silent channel, hold the threshold at least this far below the loudest frames |
| `hangover_sec` | 0.15 s | stay "active" this long after energy drops |

Signal windows (`ScoreConfig`):

| Parameter | Default | Meaning |
|---------------------------|---------|---------|
| `yield_hangover_sec` | 0.20 s | agent must stay quiet this long to count as yielded |
| `max_search_sec` | 3.0 s | how long after onset a yield is searched for |
| `caller_proximity_sec` | 0.5 s | a yield only counts if the caller held the floor within this window of it |
| `turn_end_silence_sec` | 0.20 s | caller must stay quiet this long for the turn to count as ended |
| `premature_tolerance_sec` | 0.05 s | agent may lead the caller's turn end by up to this before it counts as premature |
| `onset_min_run_sec` | 0.05 s | minimum sustained active run for the caller onset to count |
| `agent_onset_lookback_sec`| 0.10 s | window before onset checked for whether the agent was already talking |

Both onset-detection windows are exposed on `ScoreConfig` as well:
`onset_min_run_sec` (the `0.05 s` sustained active run a caller onset must
clear, applied by `first_active_sec`) and `agent_onset_lookback_sec` (the
`0.10 s` window before onset checked for "was the agent talking at onset?").
Neither affects the per-frame active/threshold decision the frame dump records.
Every threshold the scorer uses is a named, overridable parameter -- there are
no hidden constants in the scoring path.

### Step 1, per-frame RMS

Each channel is cut into 20 ms windows stepped by 10 ms. For each window the
linear root-mean-square amplitude is computed (`frame_rms`, `audio.py`). At
16 kHz this is a 320-sample window with a 160-sample hop, so `hop_sec` is
exactly `160 / 16000 = 0.01 s`; it is derived from the integer sample hop over
the sample rate, so it matches `hop_ms` exactly at common rates.

### Step 2, RMS to dBFS

Each linear RMS value is converted to decibels relative to full scale:
`dBFS = 20 * log10(rms)`, with a floor of `-120.0 dBFS` substituted before the
log to avoid `log(0)` (`to_dbfs`, `audio.py`). This is the per-frame `*_dbfs`
column in the frame dump.

### Step 3, per-channel energy VAD

For each channel independently (`energy_vad`, `vad.py`):

1. Sort the frame dBFS values and take the `noise_percentile` (10th percentile)
 as the **noise floor**.
2. The base **threshold** is `max(noise_floor + rel_db, abs_gate_db)`, i.e.
 15 dB above the floor, but never below the -60 dBFS absolute gate.
3. **Dynamic-margin guard:** if a channel is almost never silent (e.g. an agent
 talking the whole clip), the 10th-percentile floor lands *inside* speech and
 would push the threshold above the speech itself. So compute
 `cap = max(dBFS) - dyn_margin_db`; if `cap` is above the absolute gate, lower
 the threshold to `min(threshold, cap)`. This cannot rescue a genuinely silent
 channel, because the guard only fires when loud content exists above the gate.
4. A frame is **raw-active** when its dBFS ≥ the threshold.
5. **Hangover:** after any raw-active frame, keep the channel active for
 `round(hangover_sec / hop_sec)` more frames, 15 frames (150 ms) at the
 defaults, so brief inter-word gaps do not fragment one utterance.

The resulting threshold and noise floor are constant across frames for a given
channel and are recorded verbatim in the frame dump.

### Step 4, caller onset

If a caller onset label is supplied (scenario `caller_onset_sec`, or `--onset`),
it is used directly. Otherwise the onset is the start of the caller's first
sustained active run, the first run of at least 0.05 s of active frames
(`first_active_sec`). The label is preferred because on mixed or noisy audio the
first energy is not always the caller. `onset_idx` is `round(onset / hop_sec)`,
clamped into range. No label and no detectable caller speech means there is no
caller event to score: the recording is reported not scorable (`scorable: false`
with a plain reason), never scored against frame 0, and a single-recording run
exits 2.

### Step 5, the three barge-in signals

Computed in `score_channels` (`score.py`):

- **`agent_talking_at_onset`**, was any agent frame active in the 0.10 s up to
 and including onset? If not, on a should-yield expectation there is nothing to
 yield: the event is reported not scorable (`scorable: false`, with the reason
 in `not_scorable_reason`) rather than passed or failed, and a single-recording
 run exits 2. The input is what is wrong (onset time, channel mapping, or
 expectation), and it is reported as such.
- **`did_yield`**, scanning from onset up to `onset + max_search_sec` (300
 frames), the first frame where the agent goes quiet and *stays* quiet for
 `yield_hangover_sec` (20 frames) **and** the caller held the floor within
 `caller_proximity_sec` (±50 frames) of that quiet point. The proximity
 condition is what stops an agent that merely finishes its own sentence seconds
 after an isolated backchannel from being scored as a barge-in response.
- **`time_to_yield_sec`**, `(yield_frame - onset_frame) * hop_sec`, floored at
 0; `None` if the agent never yielded within the search window.
- **`talk_over_sec`**, count of frames from onset to the yield point (or to the
 end of the search window if it never yielded) where the caller and the agent
 were *both* active, times `hop_sec`.

### Step 6, the two latency (endpointing) signals

Pure timing on the same two VAD tracks, no second model (`score.py`):

- **`caller turn end`**, the first frame at/after onset where the caller goes
 quiet and stays quiet for `turn_end_silence_sec` (20 frames). `None` if the
 caller never activates after onset or is still talking when the clip ends.
- **`agent response onset`**, skip any agent speech already in progress at
 onset (the pre-caller turn or the yield tail), then the first frame the agent
 becomes active again. `None` if the agent never starts a fresh run.
- With both defined, `lead = turn_end_frame - response_onset_frame` (positive
 when the agent starts *before* the caller finishes). If `lead` exceeds
 `premature_tolerance_sec` (5 frames), `premature_start_sec = lead * hop_sec`
 and `response_gap_sec` is null. Otherwise `premature_start_sec = 0.0` and
 `response_gap_sec = max(0, response_onset - turn_end) * hop_sec`. Both are
 null when not derivable from the tracks, never fabricated. Reported values
 are rounded to the millisecond (3 decimal places of seconds).

### Step 7, two additive, opt-in cross-checks: echo and resume

Both live entirely in hotato's own layer (`echo.py`, `resume.py`), never in
the vendored `_engine`, and both add an optional `signals` block without
changing `did_yield`, `seconds_to_yield`, or `talk_over_sec` for any existing
recording.

- **`signals.echo`** (`echo.py`), computed on every scored event. Leaked TTS
 (an agent that hears its own audio back on the caller channel) is, by
 construction, a delayed, scaled copy of the agent's own envelope. The check
 is a deterministic cross-channel coherence: at each lag from 0 up to
 `DEFAULT_MAX_LAG_SEC` (0.5 s), take the cosine similarity between the caller
 frame-RMS envelope and the agent envelope shifted by that lag; `coherence`
 is the best (highest) cosine found, `lag_sec` is the lag it occurred at, and
 `echo_suspected` is `coherence >= DEFAULT_COHERENCE_THRESHOLD` (0.7) with at
 least `_MIN_OVERLAP_FRAMES` (8) of overlap. Independent speech (real
 turn-taking, backchannels) does not correlate with the agent's own envelope
 at any lag, so it scores low. `--echo-gate` (opt-in, `hotato run`) holds an
 echo-suspected yield out of the verdict (`scorable: false`) instead of
 counting it as a clean pass; without the flag the primary verdict is
 unchanged and `signals.echo` is still reported. `hotato diagnose` and the
 single-run text output print a WARNING for every echo-suspected yield.
- **`signals.resume`** (`resume.py`), present only on events where the agent
 yielded. From the agent's own VAD track, look for a fresh onset within
 `DEFAULT_RESUME_WINDOW_SEC` (4.0 s) after the yield: `resumed` is whether one
 is found, `resume_gap_sec` is the seconds from yield to that onset (`null`
 if it did not resume). `restart_suspected` flags whether the longest
 contiguous agent run at or after the resume onset reaches
 `DEFAULT_RESTART_MIN_SEC` (2.0 s), the timing fingerprint of re-answering a
 whole paragraph from the top rather than finishing the interrupted clause.
 Whether the resumed words literally repeat the earlier ones is a transcript
 question, explicitly out of scope: `restart_suspected` is a run-length
 heuristic on timing, not a text diff.
- **`agent_stop_no_caller`** (`scan.py`, not part of the scored envelope: a
 `hotato scan` candidate) surfaces the companion timing fact: the agent went
 from active to quiet with zero caller energy anywhere in
 `caller_proximity_sec` on either side, so nothing on the caller channel
 explains the drop. It is a candidate for self-truncation, endpointing
 mis-fire, or an upstream (LLM/TTS) cutoff, not a verdict; label it with
 `hotato fixture create --expect hold` (or `yield`, if that silence was in
 fact correct) to turn it into a scored fixture.

## Why it is reproducible

- **Deterministic** given `(audio, ScoreConfig)`. No randomness, no network, no
 hidden state. The bundled fixtures are themselves rendered deterministically
 from a seed derived from `sha256(scenario_id)` (`render_examples.py`), and CI
 re-renders and diffs them to prove byte-stability.
- **Every threshold that drives the active/threshold decision is a named,
 overridable parameter** on `ScoreConfig` / `VADParams`, and the resolved
 values are echoed back in the frame dump's `config` block.
- **Every frame and every derived index is inspectable.** `--dump-frames PATH`
 writes, per frame: `t_sec`, `caller_dbfs`, `agent_dbfs`, `caller_active`,
 `agent_active`, and the constant `caller_threshold_db`,
 `caller_noise_floor_db`, `agent_threshold_db`, `agent_noise_floor_db`
 (`frame_dump`, `score.py`). Nothing the scorer decides is off the record.

### Worked example: re-deriving `did_yield` and `talk_over` by hand

The excerpt below is illustrative of the dump shape; run `--dump-frames` on your
own file for real values. Assume the defaults (`hop_sec = 0.01`), a caller onset
label at `2.40 s` (`onset_frame = 240`), and these constant thresholds from the
dump header: caller threshold `-55.0 dBFS`, agent threshold `-52.0 dBFS`.

```
 frame t_sec caller_dbfs caller_active agent_dbfs agent_active
 240 2.40 -18.4 True -21.1 True
 288 2.88 -19.0 True -20.7 True
 289 2.89 -20.2 True -48.9 True (hangover tail)
 290 2.90 -21.5 True -71.3 False
 ... (agent dBFS stays below -52 through frame 310) ...
 310 3.10 -20.8 True -69.4 False
```

Re-derive `did_yield`: agent was active at onset (frame 240), so the question is
meaningful. Scanning from 240, the agent first drops below its threshold at
frame 290 and stays below it for at least 20 consecutive frames (through 310), 
that satisfies `yield_hangover_sec`. The caller is active within ±50 frames of
290, so the proximity condition holds. Therefore `did_yield = True`,
`yield_frame = 290`, and `time_to_yield_sec = (290 - 240) * 0.01 = 0.50 s`.

Re-derive `talk_over`: count frames from 240 up to the yield frame 290 where
`caller_active AND agent_active`. In this excerpt that is frames 240 through 289
inclusive, 50 frames, so `talk_over_sec = 50 * 0.01 = 0.50 s`. Every one of
those booleans is just `dbfs >= threshold` (plus hangover), which you can
recompute yourself from the two dBFS columns and the header thresholds.

## Aggregate statistics: one definition each

The `report`, `team`, and `export` surfaces summarize many events with mean,
median, p90, and p95. All four come from one stdlib implementation
(`src/hotato/_stats.py`, `dist_summary`) so every published aggregate is
re-derivable by hand:

- **mean** is the arithmetic mean (`statistics.fmean`).
- **median** is `statistics.median` (p50).
- **p90** and **p95** are linear interpolation between closest ranks (the
 definition NumPy calls "linear"): sort the values ascending, let
 `pos = q * (n - 1)` for `q` in `{0.90, 0.95}`, then
 `p_q = v[floor(pos)] + frac * (v[floor(pos) + 1] - v[floor(pos)])`.
- **Rates are fractions**, never a percentage: a pass rate of 8 of 8 reads
 `1.00`. That keeps every aggregate in the same unit system as the
 measurements and leaves no door open to an accuracy-percentage reading.
- **Empty input returns null.** No aggregate is ever fabricated from zero
 measurements, and fewer than two runs is stated plainly rather than drawn as
 a trend.

`team` and `export` pool `response_gap_sec` (dead air before the agent
speaks, defined above) across every scored event into the same `dist_summary`
shape already used for talk-over and time-to-yield. `--max-response-gap`
turns the pooled p95 into a latency SLA gate: the run exits 1 exactly when the
pooled p95 exceeds the bound, the same pass/fail contract as
`--max-talk-over` and `--max-time-to-yield`, just pooled instead of
per-event. A plain `hotato export` (no `--max-response-gap`) still writes a
byte-identical `envelope.json` to a run with none of this stage's work
applied; the p95 and gate live only in the printed summary and the returned
manifest.

## Optional neural cross-check (non-reference)

The energy VAD above is the **reference**: every published, golden, and bundled
number comes from it, deterministically. Hotato also ships an **optional,
opt-in, explicitly non-reference** neural VAD backend so you can re-run the *same*
turn-taking timing math over a learned speech track and compare, a direct answer
to "this is just energy VAD, rebuildable in a weekend."

- **How to use it.** Install the extra and pass the flag on your *own* recording:
 `pip install 'hotato[neural]'` then `hotato run --stereo call.wav --backend neural`.
 The default is `--backend energy`. The `--suite` self-test **always** scores
 with energy (it is the reference), so `--backend neural` is ignored there.
- **What it is.** Silero VAD (MIT), run locally/offline. The engine itself keeps
 zero third-party dependencies and never imports a model; the backend is injected
 behind one shared interface, so an open-weight turn-detector (e.g. LiveKit's
 Smart-Turn) can be plugged in the same way later. It returns the **identical**
 `VADResult` shape as the energy backend, `active`, `hop_sec`, `threshold_db`,
 `noise_floor_db`, aligned to the same hop grid. A neural model has no dB
 threshold, so `threshold_db` / `noise_floor_db` are **synthesized**: they are the
 energy-domain description of the *same* audio, reported for inspection only; the
 neural `active` decision is a learned speech probability, not a dB crossing.
- **What it changes, honestly.** It **tightens onset precision** on clean speech
 (a learned boundary can beat a fixed dB threshold on some audio). It does **not**
 close the energy-vs-intent gap: a cough, a laugh, a door slam, or crosstalk still
 carries speech-band energy, and *any* single-channel VAD, energy or neural, can
 mark it active. Whether a sound is a genuine bid for the turn is not decidable
 from one channel's activity alone. **No accuracy percentage is claimed for either
 backend.** Neural is a flagged cross-check, not a new source of truth.
- **No silent fallback.** If the `[neural]` extra is absent, `--backend neural`
 raises a clean, explicit error and exits non-zero, it never quietly scores with
 energy and presents it as the neural result.
- **Verification status: verified against the real model.** Executed in this
 repo with silero-vad 6.2.1 (ONNX weights bundled inside the package, inference
 through onnxruntime on CPU, fully offline; note silero-vad itself depends on
 torch, so the extra installs it). Properties that hold, measured over the
 bundled 8 and the 40-scenario gold suite:
 - **Contract holds end to end.** The real model returns the identical
   `VADResult` shape as energy, on the same hop grid, through the public scorer.
 - **Deterministic.** Two full sweeps over all 48 fixtures produced
   byte-identical JSON, for both backends; single recordings are likewise
   byte-identical run to run.
 - **Reference untouched.** With the extra installed, energy outputs are
   byte-identical to energy outputs without it, and the frozen bundled 8
   verdicts are unchanged.
 - **On the synthetic fixtures the two tracks diverge, and that divergence is a
   property of the fixtures.** The renders are shaped noise built for the
   energy reference; a speech-trained model assigns them near-zero speech
   probability (identical through the ONNX and torch paths), so at Silero's
   default threshold the neural track is empty there and the timing math
   reports no yield on any of the 48. This measures the fixtures, not any
   agent. Point the neural cross-check at real recordings.
 - **On a real-speech recording** (a dual-channel barge-in fixture assembled
   from a recorded human utterance) both backends returned the same yield
   verdict; the measured yield timing differed (energy 0.65 s including its
   0.15 s hangover, neural 0.18 s, because a speech model segments the word
   gaps that the energy hangover bridges). One recording, reported as a timing
   observation, not an accuracy claim.
 - **Sample rates.** Silero accepts 8000 Hz, 16000 Hz, and integer multiples of
   16000 Hz (silero-vad decimates those itself and returns timestamps in the
   original sample coordinates). Any other rate fails with an actionable
   resample message from the seam. The energy backend measures at any rate.
 The real-model tests run automatically when the `[neural]` extra is installed
 and skip cleanly when it is absent (`tests/test_backend.py`).

## How validity is measured, not claimed

The bundled fixtures are synthetic: band-limited, syllable-modulated noise
rendered from exact segment boundaries declared in each scenario's
`reference_render` block (`caller_segments_sec`, `agent_segments_sec`, and an
optional `continuous` flag that renders one unbroken run per segment so the
active track equals the rendered boundaries to within one hop). Because the true
boundaries are known exactly, the scorer's **own measurement error** is
computable, and that is what to report, rather than an accuracy score:

- For a timing signal (onset, yield time, response gap), report
 `|measured - true|` in **milliseconds**, per scenario. The true value comes
 straight from the segment boundaries: true onset is the caller's first segment
 start; true yield time is the agent's segment end minus onset; true gap is the
 agent's next segment start minus the caller's turn end. Expect a residual on
 the order of the hop plus the hangover smear on trailing edges, read it, do
 not average it away.
- For `did_yield`, report a **confusion matrix** against the scenario `category`
 labels (`should_yield` vs `should_not_yield`), four cells: correct yields,
 missed yields, correct holds, phantom yields. Keep the cells separate.

**Never aggregate these into a single accuracy percentage.** A missed hard
interruption and a phantom yield on a backchannel are different failures with
different fixes; one blended number hides exactly the distinction the tool
exists to surface.

State plainly what the synthetic battery is for: it proves the plumbing works
end-to-end and catches regressions (a threshold change that moves a number shows
up immediately against the frozen golden output). It is **not** production audio
, no real accents, codecs, packet loss, or room acoustics. For validity on your
system, bring **10-15 of your own labelled calls** (two-channel where you can
get it) and run them through the same `run` / `--dump-frames` path; see
`CONTRIBUTING.md` and `docs/CORPUS-GOVERNANCE.md`.

## How to verify a score on your own recordings

1. Run `--dump-frames out.json` on the recording. Read the per-channel
 `*_threshold_db` and `*_noise_floor_db` from the header, and confirm
 `threshold = noise_floor + 15 dB` (or the dynamic-margin cap, or the -60 dBFS
 gate) matches your levels.
2. Scan the `*_dbfs` and `*_active` columns around the reported onset, yield,
 and gap indices and confirm each `active` flag is just `dbfs >= threshold`
 carried forward by the 150 ms hangover. Re-derive the signal by hand as above.
3. If the VAD is mislabeling your audio, tune and watch the numbers move
 predictably: raise `rel_db` or `abs_gate_db` if line noise is marked active;
 lower `rel_db` if quiet speech is missed; raise `hangover_sec` if one
 utterance fragments into several; adjust `yield_hangover_sec` /
 `caller_proximity_sec` to match how long *your* callers pause. Re-dump and
 confirm the thresholds and active flags moved as expected.

If you disagree with a number, the frames are on the table and every threshold
is yours to change. That is the point: a measurement you can audit, not a
verdict you have to trust.


================================================================================
FILE: docs/BAD-CALL-TO-CI.md
================================================================================

# From a bad call to a CI gate

One bad call moment becomes a permanent, offline regression test in five
steps. Everything below runs locally; no audio leaves your machine.

## The label comes from you

Hotato does not infer intent. You label the expected behavior for the event:
yield means the agent should stop for the caller. hold means the agent should
keep speaking through a backchannel/noise/acknowledgement. Hotato then
measures whether the timing matched that label.

That is the whole contract. "mhm" and "stop" can carry identical speech
energy; no timing measurement can tell them apart. What Hotato can measure,
reproducibly, is whether the agent did what your label says it should have
done, and how many seconds it took.

Input requirement, stated once: Hotato's main scorer requires separated
caller and agent tracks: either one two-channel WAV or two aligned mono WAVs.
A single mixed mono call is not enough to attribute talk-over reliably.

## Step 1: turn the moment into a fixture

You have a recording where the agent talked over a caller at 42.18 seconds.
Label it:

```bash
hotato fixture create --stereo bad-call.wav \
    --id refund-interruption-001 \
    --onset 42.18 --expect yield \
    --max-talk-over 0.6 --max-time-to-yield 1.0 \
    --out tests/hotato
```

This writes `tests/hotato/scenarios/refund-interruption-001.json` (the label,
with provenance) and `tests/hotato/audio/refund-interruption-001.example.wav`
(a two-channel clip around the event, onset re-based to the clip). The
fixture is scored immediately on creation; an input that cannot be judged is
refused with the reason and exit code 2, never written silently.

Do not know the onset? List the candidate moments first:

```bash
hotato scan --stereo bad-call.wav
```

`scan` reports timing facts only (overlap onsets, agent starts during caller
speech, long response gaps, an agent going quiet with no caller energy nearby,
a caller run that correlates with the agent's own audio, i.e. suspected TTS
echo). You pick the moment and supply the label.

## Step 2: run it

```bash
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio
```

The bad take fails, by design: that is the regression you are pinning. Exit
codes are the contract: 0 all pass, 1 a regression, 2 unusable input.

## Step 3: fix your agent, then compare

Change the setting, re-capture the same scenario, and let the numbers speak:

```bash
hotato compare --before bad-call.wav --after new-take.wav \
    --onset 42.18 --expect yield
```

```
hotato compare: bad-call.wav -> new-take.wav
  verdict:           FAIL -> PASS
  did_yield:         false -> true
  seconds_to_yield:  - -> 0.42s
  talk_over_sec:     2.65s -> 0.42s  improved -2.23s

result: fixed
```

If the moment shifted between takes, pass `--before-onset` and
`--after-onset` separately. `--out report.html` writes the shareable HTML
report with the before take as the base. A side that cannot be judged renders
NOT SCORABLE and exits 2; no verdict is invented.

## Step 4: gate CI on it

```yaml
# .github/workflows/turn-taking.yml
name: turn-taking
on: [pull_request]
jobs:
  hotato:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install hotato
      - run: hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio --format json
```

`hotato run` exits 1 on any regression, so the job fails when the timing
regresses. Already running pytest? `pytest --hotato-suite
--hotato-suite-scenarios tests/hotato/scenarios --hotato-suite-audio
tests/hotato/audio` adds the same gate to the run you have (see
[PYTEST.md](PYTEST.md)). The richer PR check with a sticky results comment is
in [CI.md](CI.md).

## Step 5: when it fails, plan the fix

```bash
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio \
    --format json > result.json
hotato plan result.json --stack vapi --assistant-id <id>
```

`plan` is read-only and guarded: it proposes at most one bounded step on one
setting, refuses to tune a single threshold when the battery fails on both
axes at once, and downgrades to a checklist when the evidence cannot isolate
the layer. It never applies anything (`platform_mutation.performed` is always
false). Details: [FIX-PLANS.md](FIX-PLANS.md).

## Use Hotato when

- You have (or can capture) separated caller/agent audio for the call.
- You can point at a moment and label what should have happened: yield or
  hold.
- You want a local, deterministic regression test that fails CI when the
  turn-taking timing regresses.

## When not to use Hotato

Hotato measures turn-taking timing from separated audio tracks. It is the
wrong tool for:

- Transcript quality or wording checks. Nothing here transcribes.
- Task success or goal completion. Hotato does not know what the call was
  for.
- Sentiment, tone, or emotion. Out of scope, permanently.
- Compliance or script adherence review.
- Tool-call or API-behavior testing.
- Mixed mono recordings. One summed channel cannot attribute talk-over to a
  speaker.
- Live, in-call decisions. Hotato scores recordings after the fact; it does
  not predict at runtime.
- Unlabeled whole-call analysis. `hotato scan` surfaces candidate moments,
  but every verdict needs your yield or hold label; Hotato never invents one.


================================================================================
FILE: docs/INGEST.md
================================================================================

# `hotato ingest` -- the composable passive on-ramp

You should not have to remember to run a CLI after every bad call. Wire a webhook
to invoke `hotato ingest` once, and every completed call is scanned for
**candidate** turn-taking moments automatically.

`ingest` is **discovery, not a verdict**. It surfaces timing candidates; it never
returns a pass/fail and never infers intent. You review the candidates and
promote one to a permanent regression test with `hotato fixture create`. The human
label step stays human: `ingest` never auto-labels, auto-fixtures, or auto-tunes.

It is built by **composition** and adds only a per-stack webhook parser:

```
parse the webhook payload   ->  extract the call id / recording locator
fetch the recording         ->  hotato.capture (the SAME fetch the adapters use)
scan for candidates         ->  hotato.scan (no labels, no verdict)
write a candidate report     ->  JSON always; --out an HTML report (optional)
```

## What it is not

- **Not a daemon.** Hotato ships the command; *you* own the trigger (a webhook
  handler, a serverless function, a cron over your call log). There is no
  long-running process and no hosted service, so the offline/self-host wedge stays
  intact. The only network is the same recording fetch `hotato capture` already
  does; everything else is offline.
- **Not a labeler.** A candidate is a timing event. Only you know whether a caller
  sound was "mhm" or "stop", so only you label it.

## Contract

```
hotato ingest --stack {vapi|retell|twilio|livekit|pipecat} \
    (--event PAYLOAD.json | --call-id ID | --recording-sid RE...) \
    [--out report.html] [--format text|json] [--allow-mono] [--top N] [--min-gap S]
```

- **Exit 0** = ran (candidates reported, possibly zero).
- **Exit 2** = parse / fetch / IO error, or not-scorable input (for example a mono
  recording, which cannot attribute overlap to caller vs agent). Never a pass/fail.

`--format` controls stdout (`text` listing or the `json` candidate list, capped by
`--top`). `--out` additionally writes an HTML candidate report containing every
candidate. A webhook payload is **untrusted DATA**: `ingest` reads only the named
locator fields and never executes, scaffolds, or acts on anything in it.

## Wire your webhook -> `hotato ingest`

Point your platform's call-completed webhook at a small handler that saves the
payload and shells out to `ingest`. The pattern is identical for every stack:

```python
# a minimal webhook handler (framework-agnostic pseudocode)
import json, subprocess, tempfile, os

def on_call_completed(request):
    payload = request.body                       # the platform's webhook payload
    with tempfile.NamedTemporaryFile(
        "w", suffix=".json", delete=False) as fh:
        fh.write(payload)
        event_path = fh.name
    # ingest is one process; run it out-of-band so the webhook returns fast.
    subprocess.Popen([
        "hotato", "ingest", "--stack", "vapi",
        "--event", event_path,
        "--out", f"candidates/{request_id}.html",
    ], env={**os.environ, "VAPI_API_KEY": os.environ["VAPI_API_KEY"]})
    return 200
```

A cron over your call log works equally well when you would rather batch:

```bash
# nightly: scan yesterday's calls for candidates
for id in $(your-call-log --since yesterday --ids); do
  hotato ingest --stack vapi --call-id "$id" \
      --format json --out "candidates/$id.html" >> candidates/$id.json
done
```

### Per-stack recipe

Webhook field paths verified against live vendor docs (2026-07-07). Where a field
could not be confirmed from the live docs, `ingest` parses it **defensively** (a
missing field is simply absent, never fabricated).

| Stack | Webhook | Field ingest reads | Credentials for the fetch |
|-------|---------|--------------------|---------------------------|
| **vapi** | end-of-call-report | `message.call.id` (confirmed) | `VAPI_API_KEY` |
| **retell** | call webhook | top-level `event` + `call.call_id` (confirmed) | `RETELL_API_KEY` |
| **twilio** | `recordingStatusCallback` | `RecordingSid` (confirmed; form-encoded body) | `TWILIO_ACCOUNT_SID` + `TWILIO_AUTH_TOKEN` |
| **livekit** | egress webhook | `egressInfo.fileResults[].location` / `.filename` (defensive) | none -- egress lands in your storage |
| **pipecat** | your own event | `recording_path` / `recording_url` (defensive) | none -- you produced the file |

For **vapi / retell / twilio**, `ingest` extracts the identifier from the payload
and delegates the dual-channel recording fetch to the same adapter `hotato capture`
uses (which already resolved the recording URLs; see
[ADAPTER-STATUS.md](ADAPTER-STATUS.md)). The recording URL is never read from the
untrusted payload.

For **livekit / pipecat**, the recording lands in *your* infra, so the event
carries the locator directly. LiveKit egress files land in your storage bucket;
supply a `recording_url` (downloaded) or a `recording_path` (read locally). A
Pipecat event is whatever you emit, for example:

```json
{ "recording_path": "captured.wav" }
```

You can also skip the payload entirely with a direct id:

```bash
hotato ingest --stack vapi   --call-id  <id>   --out candidates.html   # + VAPI_API_KEY
hotato ingest --stack twilio --recording-sid RE... --format json       # + TWILIO_*
hotato ingest --stack pipecat --event event.json                        # local file
```

## From a candidate to a regression test

`ingest` finds the moment; you decide what should have happened and freeze it:

```bash
# 1. ingest surfaces a candidate at t=42.18s in a call recording
# 2. you listen, decide the agent should have yielded, and promote it:
hotato fixture create --stereo call.wav --onset 42.18 \
    --expect yield --id refund-cutoff-001 --out tests/hotato
# 3. it is now a permanent test:
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio
```

See [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md) for the full bad-call-to-CI loop.

## Notes on `--allow-mono`

Discovery needs one party per channel to attribute overlap. `--allow-mono` (or
`HOTATO_ALLOW_MONO=1`) lets the *fetch* pull a mono-only recording on retell/twilio
(matching `hotato capture`), but a mono mix is still reported **not-scorable**
(exit 2) for discovery, because overlap cannot be split between caller and agent.
Record dual-channel to get candidates.


================================================================================
FILE: docs/FIX-PLANS.md
================================================================================

# Fix plans: the guarded ladder

Hotato measures turn-taking; this ladder turns a failing measurement into a
reviewable, bounded fix proposal. Every level in this phase is read-only: no
command mutates any platform config, and no apply command exists.

## The ladder

| Level | Command | What it does | Writes to your stack |
|---|---|---|---|
| 0 | `hotato diagnose result.json` | Per-failure diagnosis + a battery decision, with the advisory and the tradeoff stated | never |
| 1 | `hotato inspect --stack ...` | Reads the CURRENT turn-taking config (GET or static parse) and normalizes it | never |
| 2 | `hotato plan result.json [target]` | Combines diagnosis + inspected config into a fix-plan JSON (`hotato.fixplan.v1`) | never |
| 3 | apply / verify | NEXT PHASE, not shipped | see below |

Level 3 is deliberately absent from this release. When it ships it will be
PR-first and clone-first: apply to a cloned assistant or a branch config,
re-run the battery, and only then graduate to production behind an explicit
approval with a recorded rollback value. Every plan already pins
`"approval": {"default": "manual", "production_apply": false}` so nothing
built today can be replayed into an auto-apply.

## Level 0: diagnose

```
hotato run --suite barge-in --format json > result.json
hotato diagnose result.json
```

One diagnosis per failing event:

```
{"finding":          missed_real_interruption | false_stop_on_backchannel |
                     slow_yield | excess_talk_over | endpointing_miss |
                     not_scorable | threshold_funnel,
 "evidence":         the measured fields for that event,
 "likely_layer":     interruption_detection | endpointing | unknown_root_cause,
 "config_only_safe": bool,
 "notes":            plain language}
```

plus one battery-level decision. The text mode is the Level 0 advisory, for
example: "Missed real interruption. Likely config layer. Try lowering the
stop-speaking word threshold one step. Tradeoff: may increase false stops on
short acknowledgements." The tradeoff is always stated.

Exit codes: 0 no failing events, 1 failing events diagnosed, 2 unusable input.

## Level 1: inspect

```
hotato inspect --stack vapi --assistant-id <id>      # + VAPI_API_KEY
hotato inspect --stack retell --agent-id <id>        # + RETELL_API_KEY
hotato inspect --stack livekit --config agent.py     # static parse
hotato inspect --stack pipecat --config bot.py       # static parse
```

Output: one normalized model (`interrupt_min_words`,
`interrupt_voice_seconds`, `resume_backoff_seconds`,
`endpointing_wait_seconds`, `backchannel_aware`) plus the raw fields and
provenance (what was fetched, when, and which docs the field names were
verified against). Absent or unreadable options are null with a note; values
are never guessed. Suspicious values (an unusually high word threshold, a long
endpointing wait) are surfaced as observations, never judgments.

Read-only by construction: Vapi and Retell are one GET each; LiveKit and
Pipecat files are parsed with `ast` and never imported or executed. Missing
credentials exit 2 cleanly.

## Level 2: plan

```
hotato plan result.json --stack vapi --assistant-id <id> --out hotato-fixplan.json
hotato plan result.json                              # stack from the envelope, else generic
```

(`--run result.json` is the equivalent flag form.) The input must be a run
envelope; frame dumps, benchmark results, and compare results are rejected
with exit 2.

The plan is `hotato.fixplan.v1` (shipped schema:
`src/hotato/schema/fixplan.v1.json`). It carries the finding, a hypothesis,
zero or more changes, the verification gate, and the approval block. Every
plan also carries `kind: "fix-plan"`, the measured `evidence` behind it, the
stated `risks`, `next_commands` (apply the step manually, verify with
`hotato compare`, re-run the battery), not-scorable events as `input_issues`
(input problems, never fixed), and a `platform_mutation` block whose
`performed` is always false: hotato plan is read-only.

Twilio rule: `--stack twilio` (or a Twilio-stack envelope) never yields
agent-config advice. Twilio carries the audio; it does not decide when the
agent yields. The plan is a checklist: confirm the dual-channel caller/agent
assignment, then re-plan against the upstream voice-agent stack.

### Policy rules (verbatim, as implemented and tested)

A change is proposed ONLY when ALL of these hold:

  (a) the failure class maps cleanly to ONE setting;
  (b) the proposed value is ONE bounded step in an unambiguous direction
      within documented bounds - never an absolute magic value; `from` is the
      inspected current value, and when inspection was not run or not
      possible the plan carries direction and bounds only, with
      `current_unknown: true`;
  (c) the run's battery contains at least one passing OPPOSITE-RISK fixture,
      else the plan DOWNGRADES to insufficient_coverage ("add an
      opposite-risk fixture before tuning") and names the exact fixture
      family to add;
  (d) the diagnosis marked the failure config_only_safe.

And the standing refusals:

* threshold_funnel: when the battery contains BOTH a missed real interruption
  AND a false stop on a backchannel, the plan is a refusal:
  `"decision": "do_not_tune_single_threshold"` with a vendor-neutral
  engagement-control pointer (no product names, no digits in the pointer
  text). No single threshold satisfies both axes; raising it for one worsens
  the other -- the threshold treadmill teams describe from the inside: tuned
  endlessly, with no perfect setting, because both failures share one knob.
* slow_yield without a clear layer: TTS buffering, transport latency, and VAD
  smoothing are indistinguishable from one recording, so the plan is a
  diagnostic checklist (instrumentation steps), never a knob change. A slow
  yield becomes config-only-safe only when the battery contains a passing
  opposite-risk backchannel fixture that makes a one-step change verifiable.
* not_scorable events are input problems. They are never diagnosed as agent
  failures and never feed a plan.

Every plan, of every decision kind, carries the same verification gate:

```
"required_verification": ["real_interruption_fixture_must_pass",
                          "backchannel_fixture_must_not_regress",
                          "slow_yield_p95_must_not_worsen"]
```

### Worked example: a safe one-step plan

Battery: the agent missed a real interruption; the backchannel fixture
passes (the opposite-risk coverage the policy requires). Inspection read
`stopSpeakingPlan.numWords = 2` from the live assistant.

```json
{
  "schema": "hotato.fixplan.v1",
  "target": {"stack": "vapi", "inspected": true,
             "assistant_id": "asst_123", "current_unknown": false},
  "finding": "missed_real_interruption",
  "hypothesis": "The agent missed a real interruption: the caller took the floor and the agent kept talking. Inspected stopSpeakingPlan.numWords is 2; one bounded step decrease is the smallest verifiable change on this axis.",
  "config_only_safe": true,
  "decision": "propose_one_step",
  "changes": [
    {"field": "stopSpeakingPlan.numWords",
     "from": 2, "to": 1, "direction": "decrease", "bounds": [0, 10],
     "reason": "The caller took the floor and the agent never stopped within the search window. A one-step sensitivity increase is a config-layer candidate; the tradeoff is more false stops on short acknowledgements. One step decrease from the inspected value. Bounds basis: documented range 0-10 (docs.vapi.ai, 2026-07-06).",
     "risk": "more false stops on short acknowledgements; the backchannel fixture must not regress"}
  ],
  "required_verification": ["real_interruption_fixture_must_pass",
                            "backchannel_fixture_must_not_regress",
                            "slow_yield_p95_must_not_worsen"],
  "approval": {"default": "manual", "production_apply": false}
}
```

One field, one step, both endpoints of the move visible, the risk named, the
verification gate attached. Nothing in the plan is an absolute value pulled
from folklore.

### Worked example: the refusal

Battery: the packaged demo (`hotato demo --format json`), which misses a real
interruption AND yields to a bare backchannel.

```json
{
  "schema": "hotato.fixplan.v1",
  "target": {"stack": "generic", "inspected": false},
  "finding": "threshold_funnel",
  "hypothesis": "The battery missed a genuine interruption and also stopped for a backchannel. One sensitivity threshold cannot satisfy both: raising it to hold through backchannels drops real interruptions, lowering it to catch interruptions yields to backchannels. The failure class is discrimination, not calibration.",
  "config_only_safe": false,
  "decision": "do_not_tune_single_threshold",
  "changes": [],
  "recommended_fix": {
    "class": "engagement-control",
    "examples": [
      "enable adaptive interruption handling where available",
      "use a backchannel-aware interruption classifier",
      "add addressee/turn-intent discrimination before stopping TTS"
    ]
  },
  "required_verification": ["real_interruption_fixture_must_pass",
                            "backchannel_fixture_must_not_regress",
                            "slow_yield_p95_must_not_worsen"],
  "approval": {"default": "manual", "production_apply": false}
}
```

The refusal is the product working as designed: the strongest thing a tuning
tool can say about a funnel is that tuning will not fix it.

## Provenance

Vendor field names and documented ranges used by inspect and plan were
verified against docs.vapi.ai (assistant startSpeakingPlan/stopSpeakingPlan),
docs.retellai.com (get-agent), docs.livekit.io (turn handling options), and
docs.pipecat.ai (user turn strategies) on 2026-07-06, and each result records
its own `field_basis`. Where a platform documents no hard range, the plan says
so and uses a conservative working range with a note to verify against the
installed version.


================================================================================
FILE: docs/FIX-LOOP.md
================================================================================

# The closed loop: find -> fix -> prove it's fixed

`hotato diagnose` / `inspect` / `plan` (see [FIX-PLANS.md](FIX-PLANS.md)) end at
a **proposal**. The closed loop carries it the rest of the way, with the two
irreversible decisions kept firmly in human hands:

1. **`hotato patch`** turns a plan's abstract `{field, from, to}` into a
   **literal, paste-ready artifact** for your platform. It PRODUCES the change;
   it never applies it.
2. You apply it, in your own stack, and re-capture the failing moments.
3. **`hotato verify`** scores the old and new runs against each other and gives
   you a **battery-scale before/after proof**: *N of M fixtures that used to
   fail now pass, and K of L hold fixtures still pass.*
4. **`hotato loop`** orchestrates the whole thing and **remembers where it left
   off**, so you can walk away and come back.

Nothing here mutates a platform, auto-labels a moment, or auto-applies a change.

## `hotato patch <fixplan.json>` -- the paste-ready change

Reads a fix plan (schema `hotato.fixplan.v1`) and renders it per platform:

- **Vapi / Retell** (config behind a REST API): a JSON **merge-patch body** plus
  a ready **`curl`** against the platform's real config-update endpoint, using
  the exact field names the plan carries.

  ```
  hotato plan result.json --stack vapi --assistant-id <id> --out fixplan.json
  hotato patch fixplan.json
  ```
  ```
  curl -X PATCH https://api.vapi.ai/assistant/<assistant-id> \
    -H "Authorization: Bearer $VAPI_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"stopSpeakingPlan": {"numWords": 2}}'
  ```

- **LiveKit / Pipecat** (config lives in your agent source): there is no
  config-update REST call to hit, so patch emits the exact **source edit** --
  the constructor kwarg and the literal value to set -- never a fabricated
  endpoint. For example `InterruptionOptions(min_words=1)`.

- **generic / unknown stack**: patch names the knob **family** and asks for a
  concrete stack; it emits no literal body it cannot stand behind.

### The both-axes case: no patch, a pointer instead

`hotato patch` only handles the config-fixable classes. When the plan's decision
is `do_not_tune_single_threshold` -- the genuine **both-axes** case, where the
battery misses a real interruption AND false-stops on a backchannel at once -- no
single config value fixes both. patch emits **no config patch**. It prints the
vendor-neutral, numbers-free **engagement-control pointer** instead: it names the
problem class (discriminating a real bid for the floor from a backchannel) and
the KIND of fix it needs, names no product, and carries no digits. It is not an
upsell, and it fires ONLY on this case -- never on an ambiguous slow yield or a
coverage gap, which get a "no patch, here's why" instead.

Every other non-propose decision (diagnostic checklist, insufficient coverage,
already at a documented bound, no change) likewise emits no patch, with the
reason pointing back at the plan.

**Guardrail:** patch makes no network call and pins `applies_change` to false.
`--format json` emits the full artifact; `--out PATH` also writes it.

## `hotato verify --before <old> --after <new>` -- the proof

After you apply the change and re-capture the same fixtures, verify scores the
old and new run envelopes against each other. Each side is a single `hotato run`
envelope JSON (a whole battery), or a directory of them; fixtures pair by
`event_id` (then `scenario_id`).

```
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio \
    --format json > before.json          # the failing take
# ... apply the patch, re-capture the fixtures ...
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio-new \
    --format json > after.json
hotato verify --before before.json --after after.json
```

It **reuses** the `hotato compare` taxonomy per fixture (`fixed`, `regressed`,
`improved`, `worse`, `unchanged`, `still_pass`, `not_scorable`) and aggregate's
pooled-distribution definitions for the before/after talk-over and time-to-yield
shift. The rollup is the two axes that matter:

- **regression axis**: how many previously-failing fixtures now pass;
- **hold axis**: how many hold-labeled guard fixtures still pass (they must not
  regress).

**Guardrail:**

- verify reports **coincidence, never causation**: it says the improvement
  *coincides with* your change, never that it *caused* it. Hotato measures
  timing; it does not run a controlled experiment.
- it **refuses** the battery-scale claim when too few fixtures failed to
  characterize (`--min-n`, default 3): the per-fixture facts still print, but the
  headline proof is withheld and said so.
- an unjudgeable side is `not_scorable`, never an invented verdict; a fixture on
  only one side is reported unpaired, never silently folded into the rollup.

By default verify measures and exits 0; `--fail-on-regression` exits 1 if any
fixture regressed or got worse.

### `--policy hotato.verify.yaml` -- the anti-bandaid gate

`--policy` turns the measured rollup into a PASS/FAIL gate you can drop into CI.
A policy declares two things, and **both** must hold for verify to pass:

```yaml
target:
  improve:
    talk_over_sec_p95: -0.5   # pooled talk-over p95 must drop by >= 0.5s
    failed_count: decrease    # fewer fixtures may fail than before
guardrails:
  max_new_false_yields: 0     # no hold guard that passed before may newly yield
  max_not_scorable: 0         # every paired fixture must be judgeable
  require_hold_fixture: true  # the battery must contain a hold fixture...
  require_yield_fixture: true # ...and a yield fixture
```

```
hotato verify --before before.json --after after.json --policy hotato.verify.yaml
```

- **`target.improve`** is the success criteria. A signed number is a required
  delta (`after - before`), so `-0.5` means "must improve by at least 0.5"; a
  keyword (`decrease`, `increase`, `no_worse`, `no_better`, `unchanged`) states
  direction only. Metrics: `talk_over_sec_p95`, `seconds_to_yield_p95`,
  `failed_count`, `false_yield_count`.
- **`guardrails`** are hard fail conditions. `max_new_false_yields` and
  `max_not_scorable` cap what a naive threshold bandaid would silently trade in;
  `require_hold_fixture` / `require_yield_fixture` refuse to certify a battery
  that never even tests the opposite axis.

verify exits `1` unless **every guardrail holds AND every target is met**, so a
fix cannot pass by improving one axis while regressing (or never testing) the
other. A patch that cuts talk-over by making the agent yield to everything meets
the talk-over target but trips `max_new_false_yields` on the hold fixtures, and
the whole check fails. The guardrails and targets are shown in the `--out
verify.html` proof (a `Policy check: PASSED`/`FAILED` headline and an
ok/violated, met/unmet table) and in the text and JSON output.

The policy is parsed with the standard library only -- Hotato's core carries no
third-party runtime dependency -- over the small subset the shipped
`examples/verify-policy/hotato.verify.yaml` uses. An unknown key, a wrong-typed
value, an empty policy, a tab indent, or a list is a clean exit-2 usage error.

## `hotato loop [FOLDER]` -- one command, with memory

`hotato loop` drives the parts Hotato can drive and remembers where it left off
in a small local state file (`.hotato/loop-state.json` by default):

- **First run over a folder of calls**: it runs discovery (`analyze` -> `scan` ->
  rank) and records the candidate moments. Stage `awaiting_label`:

  > you have N candidate moment(s) awaiting your label.

- **You label the ones that matter** with `hotato fixture create` (loop never
  labels for you: only a human supplies the yield/hold intent).

- **Next run, with those fixtures present** (`--fixtures DIR`): it runs them,
  diagnoses the battery (including the threshold-funnel check), and plans a
  guarded fix. Stage `awaiting_verify`:

  > a fix plan is ready; apply it with hotato patch, then prove it with hotato
  > verify.

```
hotato loop ./recordings                          # run 1: discover
hotato fixture create --stereo rec.wav --onset 12.4 \
    --expect yield --id refund-001 --out tests/hotato
hotato loop ./recordings --fixtures tests/hotato   # run 2: plan
```

**Hard rules:** the loop NEVER auto-labels (you supply every yield/hold intent),
NEVER auto-applies (it produces a plan and points at `hotato patch`; applying and
verifying stay human steps), and mutates no platform. It orchestrates and tracks
state; you keep the two decisions that matter -- which moment is a real bug, and
whether to apply the fix.

## Exit codes

- `hotato patch`: `0` a patch (config artifact or the engagement-control
  pointer) was produced; `2` the input is not a fix plan or is unreadable.
- `hotato verify`: `0` the rollup was produced (a low-n claim is refused but
  still exits 0); `1` a gate you opted into failed -- with `--fail-on-regression`
  a fixture regressed or got worse, or with `--policy` a guardrail was violated
  or a `target.improve` criterion was not met; `2` usage error, unreadable
  input, an invalid `--policy` file, or no fixtures pair.
- `hotato loop`: `0` advanced or re-reported state; `2` no folder on the first
  run, an unreadable state file, or a path that is not a folder.


================================================================================
FILE: docs/APPLY.md
================================================================================

# hotato apply: the guarded, clone-only staged apply

`hotato apply` is the last rung of the fix ladder and the ONLY command in Hotato
that can mutate external platform state. Because of that, it is the most
conservative command in the codebase. It never touches your production/source
assistant under any flag combination. It reads a `hotato patch` artifact and
either PRINTS the fresh staging clone it would create (the default, fully
offline dry run) or, only with `--yes` and credentials, creates a NEW staging
assistant that is your source config with the patch applied.

```
hotato patch fixplan.json --format json --out patch.json
hotato apply patch.json --clone --name staging-refund-fix --battery tests/hotato
```

## The five hard rules (structural, not prose)

1. **Clone-only.** There is no production-apply path in this version. A
   non-`--clone` invocation is a clean usage error: `production apply is not
   supported; use --clone to apply to a fresh staging assistant`. Nothing here
   ever issues a `PUT`/`PATCH` against the source; the one writing call is a
   `POST` that creates a NEW assistant.

2. **Refusal-first.** If the patch is the both-axes threshold funnel (the plan
   decided `do_not_tune_single_threshold`), apply REFUSES before doing anything
   and prints the exact recommendation:

   ```
   No config patch will be applied
   Reason: both missed real interruption and false stop on backchannel, one threshold cannot safely fix both
   Recommended: enable or add engagement-control / backchannel-aware turn detection
   ```

   The refusal is a FEATURE. It exits with a distinct, documented code (`3`), so
   a script can tell "refused by design" apart from a usage error. One
   sensitivity threshold cannot both catch a real interruption and hold through
   a backchannel; that is a discrimination problem, not a calibration one, and
   no clone gets a patch that pretends otherwise.

3. **Opposite-risk required.** apply refuses unless `--battery` carries BOTH a
   yield fixture (a real interruption the agent must stop for) AND a hold
   fixture (a backchannel the agent must keep the floor through). A one-sided
   battery cannot catch the opposite risk a threshold move trades into, so
   applying would be blind. Point `--battery` at your fixtures directory (with a
   `scenarios/` folder, as `hotato fixture promote` writes) or at a folder of
   run-envelope / scenario JSONs.

4. **Gated side effect.** The default is a dry run that prints exactly the clone
   it would create and the patch it would apply, creating nothing and touching
   no network. Only `--yes` WITH credentials reaches the platform. The actual
   create is the only networked function: it reads the source config (`GET`),
   applies the patch to a copy, and creates a NEW assistant (`POST`). The source
   is only ever read.

5. **Name required.** The staging clone must be named explicitly (`--name`).
   apply never invents a name for something it creates.

## What gets cloned

Only the REST-config stacks have an assistant/agent to clone through an API:

| stack  | read (source, read-only)          | create (a NEW assistant)              | name field   |
| ------ | --------------------------------- | ------------------------------------- | ------------ |
| vapi   | `GET https://api.vapi.ai/assistant/{id}` | `POST https://api.vapi.ai/assistant`  | `name`       |
| retell | `GET https://api.retellai.com/get-agent/{id}` | `POST https://api.retellai.com/create-agent` | `agent_name` |

The clone config is your source config with the patch deep-merged on top, given
the new name, and stripped of the server-assigned ids so it is a fresh object,
never an overwrite of the source.

LiveKit and Pipecat keep turn-taking config in your agent SOURCE, not behind a
config API, so there is no assistant to clone. For those, `hotato patch` already
emits the exact source edit; apply points you at it rather than pretending a
platform clone exists. Twilio carries the audio but runs no turn-taking agent,
so apply points at the upstream stack.

## After the clone: prove it

The clone is a staging assistant so you can prove the fix before touching
production. Re-capture the same battery through the SOURCE (into `before/`) and
through the CLONE (into `after/`), then verify:

```
hotato verify --before before/ --after after/ --policy hotato.verify.yaml
```

`hotato verify` reports coincidence, never causation, and the policy is the
anti-bandaid gate: the fix passes only if every guardrail holds (including
`max_new_false_yields` on the hold fixtures) AND every target is met, so a patch
that just makes the agent yield to everything is caught. See `docs/FIX-LOOP.md`.

Or run the whole thing (this gate + verify + an optional contracts re-verify +
explain's attribution) as one fail-closed before/after report:
`hotato fix trial patch.json --name staging-refund-fix --before before/ --after
after/`. See [`docs/FIX-TRIAL.md`](FIX-TRIAL.md).

## Exit codes

- `0` the staging clone was rendered (dry run by default; with `--yes` and
  credentials the NEW staging assistant was created and patched, the source
  untouched).
- `2` usage error: no `--clone`, no `--name`, no opposite-risk battery (both a
  yield and a hold fixture), a stack with no assistant to clone, a patch that
  produced no config change, or unreadable input.
- `3` principled refusal: the plan is the both-axes threshold funnel, so no
  single-threshold patch is applied by design. The refusal is the feature.


================================================================================
FILE: docs/FIX-TRIAL.md
================================================================================

# fix trial: one before/after proof, composed, fail-closed

`hotato fix trial` is the last rung of the fix ladder: it composes the
already-shipped, already-guarded primitives into ONE before/after report that
says whether a candidate change actually holds. No new scoring engine, no new
networked path:

* **`hotato apply`'s exact offline gate** (`build_apply`, `clone=True`):
  refusal-first on the both-axes threshold funnel, opposite-risk-battery-
  required, clone-only. fix trial never creates a clone itself and never
  touches the network -- it never calls `apply.create_clone` /
  `apply._http_json`, so it carries the same clone-only, production-
  unmutatable guarantee apply's own dry run gives, by construction.
* **`hotato verify`'s battery-scale rollup**: the BEFORE run (the original
  failure evidence) against the AFTER run (re-captured through the clone you
  created separately with `hotato apply --clone --yes`), scoring EVERY paired
  fixture, not just the target failure -- the "neighbouring cases" check.
* **`hotato contract verify`**, when `--contracts DIR` is given: another
  neighbouring-cases check, on real labelled moments outside the battery.
* **`hotato explain`**, folded in as the report's attribution section: root
  cause of the ORIGINAL failure, reused exactly.

```
hotato patch fixplan.json --format json --out patch.json
hotato apply patch.json --clone --name staging-refund-fix --battery tests/hotato
# ... re-capture the battery through the source (before/) and the clone (after/) ...
hotato fix trial patch.json --name staging-refund-fix \
    --before before/ --after after/ --battery tests/hotato \
    --policy hotato.verify.yaml --out fix-trial.json --html fix-trial.html
```

## The verdict is fail-closed, never a soft pass

| Verdict | When | Exit |
| --- | --- | --- |
| `improved` | the verify claim is supported (>= `--min-n` previously-failing fixtures), at least one now passes, NOTHING regressed anywhere in the battery (including the hold/opposite-risk axis), no contract regressed, and `--policy` (if given) passed | `0` |
| `regressed` | any fixture regressed, a contract regressed, or the policy failed | `1` |
| `inconclusive` | too few previously-failing fixtures to characterize, or nothing that used to fail now passes | `1` |
| `refused` | the patch is the both-axes threshold funnel; apply's own refusal-first gate fires before any before/after evidence is even read | `3` |

**`inconclusive` is fail-closed, not a pass.** A low-n battery or a
zero-improvement battery exits the SAME non-zero code as a real regression,
so CI never treats "we could not tell" as green. A hold fixture that flips
from passing to failing (the opposite-risk axis) is `compare.classify_pair`'s
`"regressed"` result whichever axis it is on, so it is caught by the same
check that catches a talk-over regression -- there is no separate,
weaker gate for the opposite-risk axis.

## Refusal: correct output, not an error

If the patch is `do_not_tune_single_threshold` (the battery missed a real
interruption AND false-stopped on a backchannel in the same battery), fix
trial refuses before reading `--before` / `--after` / `--contracts` at all,
prints the exact canon recommendation, and exits `3` -- the SAME distinct
code `hotato apply` uses for the same refusal, so a script that already
branches on apply's refusal code recognizes fix trial's too.

```
No config patch will be applied
Reason: both missed real interruption and false stop on backchannel, one threshold cannot safely fix both
Recommended: enable or add engagement-control / backchannel-aware turn detection
```

## Flags

| Flag | Meaning |
| --- | --- |
| `PATCH_JSON` | a `hotato patch` artifact |
| `--name NAME` | name of the staging clone this trial is proving (required for a non-refused patch; the same `--name` `hotato apply` takes -- fix trial evaluates the SAME clone-only gate, it never creates a clone itself) |
| `--before RUN.json\|DIR` | the OLD run envelope(s): the original failure evidence. Also the default opposite-risk `--battery`, and the attribution source, when those are omitted |
| `--after RUN.json\|DIR` | the NEW run envelope(s), re-captured through the staging clone |
| `--battery DIR` | the opposite-risk battery apply's gate checks (BOTH a yield and a hold fixture); defaults to `--before` |
| `--contracts DIR` | also re-verify a directory of hotato contracts; any contract regression fails the trial |
| `--policy hotato.verify.yaml` | gate verify's rollup (see `docs/FIX-LOOP.md`); a violation fails the trial |
| `--min-n N` | minimum previously-failing fixtures needed to support the claim (default 3) |
| `--out PATH` | also write the full proof JSON |
| `--html PATH` | also write a self-contained before/after HTML report |

## Output

* **text** (default): the verdict, verify's own rendered proof, the contract
  verify rollup when `--contracts` was given, and the attribution section
  (one `hotato explain` render per file under `--before`).
* **`--format json`**: the full machine shape, schema `hotato.fix_trial.v1`.
  Every sub-result is the REAL nested result the underlying command already
  produces (`apply`, `verify`, `contract_verify`, `attribution`) -- nothing
  here is a re-derived summary.
* **`--html PATH`**: a self-contained report, reusing the same house style
  the other HTML reports use: the verdict chip, the verify proof table, the
  contract-verify rollup, and the attribution cards.

## Exit codes

| Code | Meaning |
| --- | --- |
| 0 | `improved` |
| 1 | fail-closed: `regressed` or `inconclusive` |
| 2 | usage error: the same gates `hotato apply` enforces (no `--name`, no opposite-risk battery, a stack with no clone target, a patch with no concrete change) or `hotato verify` / `contract verify` already enforce (no fixtures pair, an invalid `--policy`, a `--contracts` dir with no contracts, unreadable input) |
| 3 | principled refusal: the both-axes threshold funnel. Shared with `hotato apply`'s refusal code |

## What this is not

fix trial does not create, patch, or delete anything on your platform; it
never calls `apply.create_clone`. It does not prove authorization, identity,
compliance, or policy safety. It reports coincidence, never causation --
every underlying `verify` and `contract verify` number keeps that rule. See
[`FIX-LOOP.md`](FIX-LOOP.md) for the manual steps this composes, and
[`APPLY.md`](APPLY.md) / [`EXPLAIN.md`](EXPLAIN.md) for the primitives
themselves.


================================================================================
FILE: docs/ANALYZE.md
================================================================================

# `hotato analyze <folder>`: drop a folder, hear the bug

Zero-config discovery over a whole folder of real dual-channel call recordings.
No scenarios, no labels, no onset, no flags required: point it at the folder and
it does the rest.

```bash
hotato analyze ./recordings
```

That writes one self-contained, offline HTML dashboard (`hotato-analyze.html` by
default) and opens it. `hotato ./recordings`, a bare folder as the first
argument, routes to the same command.

## What it does

For every `.wav` under the folder (walked recursively, in sorted order):

1. runs the same whole-call scanner as [`hotato scan`](../src/hotato/scan.py):
   it walks the caller and agent VAD activity tracks across the ENTIRE call,
   label-free, and emits candidate timing moments
   (`overlap_while_agent_talking`, `agent_start_during_caller`,
   `long_response_gap`, `agent_stop_no_caller`, `echo_correlated_activity`),
   each with a timestamp and a measured number;
2. aggregates the candidates across ALL calls and ranks them by the scanner's
   own salience (overlap seconds / gap seconds / echo coherence) so the worst
   moments float to the top.

Then it emits three things.

### 1. A ranked dashboard

One card per top moment, in the [`hotato report`](REPORTS.md) house style: the
call file, the timestamp, the candidate kind, the measured number, and a
to-scale caller/agent timeline of that exact moment (the same SVG renderer the
report uses: activity spans, the shaded talk-over band, the onset marker, and
a yield marker where the scanner measured the agent going silent).

### 2. The hear-the-bug player

For the top `--audio-top` moments (default 8) the audio around the moment
is embedded inline as a base64 WAV data URI. Nothing is uploaded, the page has
zero external requests. Press play and a **playhead** sweeps that moment's
timeline in lockstep with `audio.currentTime` (via `requestAnimationFrame`), so
you HEAR the agent talk over the caller, or the dead-air gap, land exactly where
the chart marks it. Reduced-motion safe: with `prefers-reduced-motion: reduce`
the playhead still tracks playback (it rides `timeupdate` instead of the smooth
animation loop).

Only the top moments carry audio; the rest show the timeline only. That, plus
`--pre` / `--post` (the seconds of audio kept before/after each moment) and a
total-page audio budget, keeps the page a reasonable size.

### 3. JSON for agents

```bash
hotato analyze ./recordings --format json
```

prints the ranked candidates plus their metadata (source file, timestamp, kind,
salience, measured durations, and the audio/timeline window) to stdout, capped
by `--top`. Pass `--out FILE` to also write the full ranked JSON.

## Framing

These are **measured candidate timing moments**, not verdicts and not intent.
Energy is not intent: the scanner cannot know whether a caller sound was "mhm"
or "stop", so nothing here is a pass/fail, a failure count, or an accuracy
number. You decide the expected behavior and label the moments that matter:

```bash
hotato fixture create --stereo <call>.wav --onset <t> \
    --expect yield|hold --id found-moment-001 --out tests/hotato
```

Non-dual-channel or otherwise unreadable files are reported cleanly in a
"Skipped files" section with their reason (a mono mix cannot attribute
talk-over); a bad file never crashes the run.

## Flags

| flag | default | meaning |
| --- | --- | --- |
| `FOLDER` | (required) | directory of dual-channel WAVs, walked recursively |
| `--top` | 25 | ranked moments shown in the dashboard / stdout JSON (0 = all) |
| `--audio-top` | 8 | top moments that get the embedded hear-the-bug player |
| `--pre` | 2.0 | seconds of audio/timeline kept before each moment |
| `--post` | 4.0 | seconds of audio/timeline kept after each moment |
| `--min-gap` | 2.0 | minimum response gap (seconds) to surface as a candidate |
| `--caller-channel` / `--agent-channel` | 0 / 1 | channel assignment |
| `--format` | `html` | `html` (dashboard) or `json` (ranked candidates) |
| `--out` | `hotato-analyze.html` | where to write the dashboard, or the JSON |
| `--no-open` | off | do not launch a browser for the dashboard |

## Exit codes

- **0**: ran (candidate moments listed across the folder, possibly zero; never
  a pass/fail and never a verdict).
- **2**: usage error (the path is not a folder) or an IO error reading it.

Everything runs offline; no audio leaves the machine.


================================================================================
FILE: docs/TRUST.md
================================================================================

# `hotato trust --stereo call.wav`: is this recording even scorable?

The input-health check, or "trust doctor": inspect ONE recording and report
whether the audio is good enough to score, BEFORE you scan or run it. A bad
export (a mono file, a silent channel, a swapped channel map, a hot capture) is
caught up front, so it never turns into a confident-looking but meaningless
turn-taking verdict downstream.

```bash
hotato trust --stereo your-call.wav
```

```text
hotato trust: your-call.wav
  recording: 32.0s, 44100 Hz, 2 channels
  caller (ch0): 3.32s speech, first at 3.27s, peak -2.1 dBFS
  agent  (ch1): 23.30s speech, first at 0.45s, peak -0.9 dBFS
  leading silence: 0.45s
  crosstalk: coherence 0.057 (low) at 0.39s lag
  scorability: separated tracks yes, caller activity yes, agent activity yes
  => safe to scan
```

Exit code `0` means safe to scan; exit code `2` means NOT SCORABLE (or a usage
error / unreadable file), so `trust` composes straight into a shell gate:

```bash
hotato trust --stereo call.wav && hotato scan --stereo call.wav
```

## What it checks

By convention the caller is on channel 0 and the agent on channel 1 (override
with `--caller-channel` / `--agent-channel`). `trust` reports INPUT health only:

- **per-channel activity**: how much speech each channel carries and when each
  first speaks;
- **possible channel swap**: a heuristic flag: if the channel mapped as the
  caller holds the floor far longer than the channel mapped as the agent (the
  reverse of the usual pattern, where an assistant answers in paragraphs), the
  caller/agent channels may be reversed;
- **sample rate and duration**: the basic recording facts;
- **clipping**: per-channel peak level (dBFS) and the fraction of samples at
  full scale, so a too-hot capture is visible;
- **leading silence**: dead air before the first speech on either channel;
- **crosstalk risk**: cross-channel echo coherence: is the caller channel
  carrying a delayed copy of the agent's own audio (echo bleed / missing echo
  cancellation)?
- **scorability**: the three things a real score needs: separated tracks, enough
  caller activity, and enough agent activity;
- **recommendation**: `safe to scan`, or `NOT SCORABLE` with the specific reason
  AND the next step to fix it.

## What it is NOT

`trust` never labels intent and never emits a turn-taking verdict. There is no
`yield` / `hold`, no `pass` / `fail`, no `did_yield`, no talk-over number. It
answers exactly one question (is this audio good enough to score?) and stops.
A recording that is safe to scan may still contain agent bugs; finding those is
what [`hotato scan`](../src/hotato/scan.py) and `hotato run` are for.

## Not scorable

Three input defects make a recording unscorable. Each is reported with `scorable:
false`, a plain reason, and the next step, and exits `2`:

- **mono**: a single channel cannot separate the caller from the agent on its
  own. Next step: export a dual-channel recording (the gold reference), OR score
  it via the opt-in diarization front-end (see "Mono via `--diarize`" below).
- **identical channels**: a mono recording duplicated into two channels: two
  channels, but not separated. Same fix as mono.
- **a silent required channel**: for example `caller channel has no detected
  speech`. Next step: verify channel mapping or export dual-channel again.

Clipping, high leading silence, crosstalk risk, and a possible channel swap are
**warnings**: they are surfaced but do not, by themselves, make a recording
unscorable.

## Mono via `--diarize`: is this mono file confidently separable?

By default a mono file is not scorable (above). With the opt-in `[diarize]`
front-end, `hotato trust --stereo call.wav --diarize` reports whether the mono is
confidently SEPARABLE into caller/agent -- still WITHOUT emitting any turn-taking
verdict. It runs the selected diarizer (`--diarizer pyannote|sortformer|pyannoteai`,
default local `pyannote`), then reports a `scorability.separation` sub-block and a
confidence **tier**:

- **high**: confidently separable -- score it with `hotato run --mono call.wav
  --diarize` for a real (diarized-mono) verdict. Exit `0`.
- **low**: separable but only indicative -- e.g. voices close, overlap elevated,
  or the caller/agent mapping balanced. `hotato run --mono ... --diarize` will
  still score it, but the verdict is stamped `indicative_only` and no SLA gate
  fires. Exit `0`.
- **refuse**: not confidently separable (not two clean parties, a near-silent
  speaker, extreme overlap, voices too similar). Not scorable, exit `2`; next
  step: record a dual-channel call.

The six signals behind the tier (speaker count, per-speaker activity, mean
segmentation posterior, embedding cluster-separation margin, overlap ratio,
segment churn) are in the `separation.signals` block. A missing extra / token /
model raises a clean error (exit 2), never a raw-mono guess. The dual-channel
path stays the gold reference; a diarized-mono verdict is never equivalent to it.
See `docs/DIARIZE.md` for the full front-end.

## JSON for agents

`--format json` emits one machine-parseable report. Branch on `scorable` (and, on
a defect, read `not_scorable_reason` and `next_step`); `exit_code` mirrors the
process exit.

```bash
hotato trust --stereo call.wav --format json
```

```json
{
  "tool": "hotato",
  "kind": "input-health",
  "schema_version": "1",
  "source": "call.wav",
  "recording": {
    "sample_rate": 44100,
    "duration_sec": 32.0,
    "channels": 2,
    "clipping": {
      "caller": {"peak": 0.7906, "peak_dbfs": -2.1, "clipped_fraction": 0.0, "clipped": false},
      "agent": {"peak": 0.9016, "peak_dbfs": -0.9, "clipped_fraction": 0.0, "clipped": false}
    },
    "leading_silence_sec": 0.45
  },
  "channels": {
    "caller": {"channel": 0, "active_sec": 3.32, "first_speech_sec": 3.27, "has_speech": true, "enough_activity": true},
    "agent": {"channel": 1, "active_sec": 23.3, "first_speech_sec": 0.45, "has_speech": true, "enough_activity": true},
    "possible_swap": false,
    "swap_reason": null
  },
  "crosstalk_risk": {"coherence": 0.057, "lag_sec": 0.39, "suspected": false},
  "scorability": {"separated_tracks": true, "enough_caller_activity": true, "enough_agent_activity": true},
  "warnings": [],
  "scorable": true,
  "recommendation": "safe to scan",
  "not_scorable_reason": null,
  "next_step": null,
  "exit_code": 0
}
```

Everything runs offline and reuses hotato's existing primitives: the hardened
WAV reader, the reference framing, the energy VAD, and the cross-channel echo
coherence. No audio leaves the machine, and no accuracy percentage appears
anywhere.


================================================================================
FILE: docs/CONNECT.md
================================================================================

# Connect, pull, sweep: score every real call across your stack

`hotato sweep` is the "connect once, see every turn-taking problem across all
your real calls" flow. It has three steps you can also run on their own:

1. **`hotato connect <stack>`**: store a stack's credentials once, locally.
2. **`hotato pull`**: bulk-fetch your recent recordings into a folder.
3. **`hotato sweep`**: pull, then run the zero-config `analyze` over them and
   write one offline dashboard of the ranked turn-taking moments.

Everything scores offline. The only network is the direct recording download
from your vendor to your machine; your audio and your keys are never sent to
Hotato.

## 1. Connect once

```
hotato connect vapi --api-key <key>
```

This does a lightweight live auth check (it lists one recent call), then writes
the credentials to `~/.hotato/connections.json` with file mode `0600` (directory
`0700`). The key is never printed and never leaves your machine except to the
vendor's own API. Credentials also fall back to the stack's environment variable,
so this works too:

```
VAPI_API_KEY=<key> hotato connect vapi
```

Per-stack credentials:

| Stack | Flags (or env vars) |
| --- | --- |
| vapi | `--api-key` / `VAPI_API_KEY` |
| retell | `--api-key` / `RETELL_API_KEY` |
| twilio | `--account-sid` `--auth-token` / `TWILIO_ACCOUNT_SID` `TWILIO_AUTH_TOKEN` |
| bland | `--api-key` / `BLAND_API_KEY` |
| elevenlabs | `--api-key` / `ELEVENLABS_API_KEY` |
| synthflow | `--api-key` / `SYNTHFLOW_API_KEY` (+ `--model-id` / `SYNTHFLOW_MODEL_ID` to list) |
| millis | `--api-key` / `MILLIS_API_KEY` (+ `--base-url` for the EU region) |
| cartesia | `--api-key` / `CARTESIA_API_KEY` (+ `--agent-id` / `CARTESIA_AGENT_ID` to list) |

Retell has no list endpoint to verify against, so `connect retell` stores the
key and validates it on the first pull. Use `--no-verify` to skip the live check
for any stack. LiveKit and Pipecat are capture-in-your-infra. There is no
vendor recording to pull, so they are not connectable; use `hotato setup
--stack livekit|pipecat` instead.

After connecting, `--stack` and the credential flags are optional for `pull` and
`sweep`: when exactly one stack is connected, Hotato uses it.

## 2. Pull recent recordings

```
hotato pull --stack vapi --since 7d --limit 50
hotato pull                                   # the only connected stack
```

`pull` lists your recent recordings with the vendor's verified list endpoint and
downloads each one by looping the same single-call fetch `hotato capture` uses,
into `hotato-pull-<stack>/` (override with `--out DIR`). `--since` accepts `7d`,
`12h`, `30m`, `2w`. A recording that cannot be fetched (missing URL, HTTP error,
wrong channel count) is reported as a clean skip with its reason and the pull
continues. One bad call never crashes the run.

**Retell has no verified list endpoint.** Pull it from explicit ids instead
(Hotato never fabricates an endpoint):

```
hotato pull --stack retell --call-id c1 --call-id c2
```

For Twilio, `--call-id` values are Recording SIDs (`RE...`).

**Mono / mixed stacks** (bland, elevenlabs, synthflow, millis, cartesia) produce
a single combined recording with no per-party channel, so talk-over cannot be
attributed. They require `--allow-mono` and are indicative only:

```
hotato pull --stack bland --allow-mono --limit 20
```

## 3. Sweep = pull + analyze

```
hotato sweep --stack vapi --since 7d
hotato sweep                                  # the only connected stack
```

`sweep` pulls (as above) into `hotato-sweep-<stack>/` (override with `--dir`),
then runs the exact same zero-config `analyze` over the folder and writes one
self-contained, offline HTML dashboard (`hotato-sweep-<stack>.html`, override
with `--out`) of the ranked candidate turn-taking moments across every call,
with the hear-the-bug audio player on the top moments. `--format json` emits the
ranked candidates plus a pull summary instead.

Dual-channel stacks give separated scoring. Mono/mixed stacks can be swept with
`--allow-mono`, but their calls cannot be attributed per party and surface in the
dashboard's Skipped section.

Candidates are MEASURED timing moments you review and label with `hotato fixture
create`, never verdicts and never intent. There is no pass/fail, no failure
count, and no accuracy number anywhere.

## What is and isn't pullable

See [`docs/ADAPTER-STATUS.md`](ADAPTER-STATUS.md) for the full map:
which stacks auto-pull dual-channel, which are mono-only behind `--allow-mono`,
which are capture-in-your-infra, and which are not integrable, each with the
exact verified endpoint and the gaps.


================================================================================
FILE: docs/ADAPTER-STATUS.md
================================================================================

# Adapter status

What each capture adapter is built against, verified verbatim against the
vendor's live documentation on the date shown (integration research:
`hotato-launch/INTEGRATION-SPEC-2026-07-07.md`). This file is the map of
what Hotato can and cannot pull, and why.

Terms:

- **auto-pull**: Hotato fetches the recording itself with your API key
  (`hotato capture` / `hotato pull` / `hotato sweep`).
- **capture-in-your-infra**: there is no vendor recording API; Hotato prints
  the recording config (`hotato setup`) and scores the file your deployment
  writes.
- **mono / mixed**: a single combined channel. Overlap cannot be attributed to
  the caller or the agent, so Hotato scores it only behind an explicit
  `--allow-mono` / `HOTATO_ALLOW_MONO=1` opt-in and labels the result indicative
  only. Separated turn-taking analysis is not possible from mono.

Honesty rule (enforced): an adapter ships only with endpoints verified verbatim
in the spec. Where the spec marks a list-calls endpoint or a channel layout
**unconfirmed / none**, Hotato does not invent one; it supports the
fallback (explicit ids) and documents the gap here rather than guessing.

## Build now: real dual-channel (auto-pull, separated scoring)

| Stack | List recent calls | Fetch recording | Channel basis | Last verified |
| --- | --- | --- | --- | --- |
| Vapi | `GET https://api.vapi.ai/call` (params `limit`, `createdAtGt/Lt`) → JSON array of Call objects (`id`, `createdAt`) | `GET /call/{id}` → `artifact.recording.stereoUrl` (current); deprecated fallbacks `artifact.stereoRecordingUrl`, `call.stereoRecordingUrl` | `stereoUrl` is a distinct 2-channel file (customer ch0, assistant ch1) | 2026-07-07 |
| Twilio | `GET .../Accounts/{Sid}/Recordings.json` (params `PageSize`, `DateCreatedAfter/Before`, `callSid`) → `recordings[].sid` | `GET .../Recordings/{RE...}.wav?RequestedChannels=2` (HTTP Basic) | dual-channel only if the recording was created `RecordingChannels=dual`; 400 → clean stop or `--allow-mono` → `RequestedChannels=1`. Two-party order: ch0 = caller, ch1 = agent; conference: ch0 = first participant | 2026-07-07 |
| Retell | **none confirmed**: the spec marks list-calls unconfirmed; do not fabricate one. Pull from explicit `--call-id` | `GET https://api.retellai.com/v2/get-call/{id}` (Bearer) → `scrubbed_recording_multi_channel_url` preferred, then `recording_multi_channel_url`; plain mono `recording_url` rejected unless `--allow-mono` | per-party channels on the `*_multi_channel_url` fields | 2026-07-07 |
| LiveKit | **capture-in-your-infra**: `ListEgress` is an RPC method name only; no REST list. `hotato setup --stack livekit` | Two audio-only Track egresses (one per party); recording location in `egress_info.file_results[].location` | separated by running one Track egress per participant | 2026-07-07 |
| Pipecat | **capture-in-your-infra**: Pipecat Cloud session-list has no recording field; OSS has no list at all | `AudioBufferProcessor(num_channels=2)` in-pipeline (user left, bot right); you write the WAV | 2-channel in-pipeline | 2026-07-07 |

## Build now: mono / mixed only (auto-pull, `--allow-mono`, indicative only)

Each of these has a verified list + fetch, but the vendor produces a single
combined recording with **no documented per-party channel**, so scoring is
degraded and gated behind `--allow-mono`.

| Stack | List recent calls | Fetch recording | Mono basis (verbatim) | Last verified |
| --- | --- | --- | --- | --- |
| Bland AI | `GET https://api.bland.ai/v1/calls` → `calls[].call_id` | `GET /v1/calls/{id}` → `recording_url` | no stereo/channel field in the recording or call-details schema | 2026-07-07 |
| ElevenLabs Conversational AI | `GET https://api.elevenlabs.io/v1/convai/conversations` (params `page_size`, `call_start_after_unix`) → `conversations[].conversation_id` | `GET /v1/convai/conversations/{id}/audio` → combined audio | docs state the audio "does not include separate caller/agent channels, only a combined full conversation MP3" | 2026-07-07 |
| Synthflow | `GET https://api.synthflow.ai/v2/calls?model_id=…` (params `limit`, `from_date` epoch-ms) → `response.response.calls[].call_id` | `GET /v2/calls/{id}` → `response.response.calls[0].recording_url` (a Twilio Recordings URL) | Synthflow documents no dual-channel option; `recording_url` is a bare Twilio URL | 2026-07-07 |
| Millis AI | `GET {base}/call-logs` (US `api-west`, EU `api-eu-west`; param `limit`) → `histories[].session_id` | `GET /call-logs/{session_id}` → `recording.recording_url` | schema exposes only `recording_url`; `CallSettings` has only a boolean `enable_recording` | 2026-07-07 |
| Cartesia (Line) | `GET https://api.cartesia.ai/agents/calls?agent_id=…` (param `limit`; needs `Cartesia-Version` header) → `data[].id` | `GET /agents/calls/{id}/audio` → `audio/wav` | dual-vs-mono **unconfirmed** in the docs; treated as mono until a live channel-count check proves otherwise | 2026-07-07 |

`--stack synthflow` needs `--model-id` (its list endpoint requires `model_id`);
`--stack cartesia` needs `--agent-id`; `--stack millis` accepts `--base-url` for
the EU region. Single-call `capture`/`pull` by explicit id needs none of these.

## Not integrable (no vendor recording to pull)

| Stack | Why | Basis |
| --- | --- | --- |
| Deepgram Voice Agent API | Real-time WebSocket (`wss://agent.deepgram.com/v1/agent/converse`) with no REST list-calls, no fetch-recording endpoint, and no recording-ready webhook. Deepgram does not store the call. If you want a recording, capture it in your own infra (LiveKit/Pipecat pattern) | 2026-07-07, confirmed absent |
| PlayAI (formerly Play.ht) | The conversational-agent product is dead: `play.ai` DNS does not resolve and `docs.play.ai` returns `DEPLOYMENT_NOT_FOUND`. The only live domain, `docs.play.ht`, is TTS-only with no calls/recordings API | 2026-07-07, dead endpoints verified |

## Unconfirmed: needs credentials or a live probe before shipping

Documented weakly or behind a login/SPA; the spec could not verify a recording
field-path or a channel layout. Not shipped as adapters. Listed here so
nobody assumes support that was never confirmed.

- **Regal.ai**: no list-calls and no REST fetch-recording endpoint. The
  recording arrives only via the `call.recording.available` webhook's
  `properties.recording_link`, which is a literal Twilio Recordings URL. Feed
  that URL through Hotato's existing Twilio path; Hotato does not fabricate a
  Regal pull.
- **Thoughtly**: OpenAPI types responses as an untyped `GenericResponse.data`
  blob; no recording-URL field path could be confirmed live.
- **Sindarin**: no Sindarin-hosted recording endpoint found; `record` is
  submitted with the caller's own Twilio creds, implying the recording lands in
  the customer's own Twilio account (use the Twilio path).
- **xAI Grok Voice Agent Builder**: OpenAI-Realtime-compatible; no documented
  list-calls or fetch-recording endpoint (marketing mentions in-product
  recording, but no API contract was confirmable).
- **Cartesia dual-channel**: the `/agents/calls/{id}/audio` WAV's channel count
  was not confirmable in docs; treated as mono here pending a live check.
- **Twilio dual-channel edge behaviour**: the exact 400-on-unavailable and
  conference channel-ordering guarantees were carried forward from prior notes,
  not re-verified verbatim this pass.
- **Synthflow / Regal Twilio-URL dual-channel trick**: because their
  `recording_url` is a Twilio URL, `?RequestedChannels=2` *might* yield stereo,
  but neither vendor documents or guarantees it; unconfirmed until tested against
  a live account.

Other enterprise/partial platforms researched but not shipped (login-gated,
SPA-only docs, or transcript-only APIs): Daily, Ultravox, Hume EVI, Telnyx,
Infobip, OpenAI Realtime, Amazon Connect, Parloa, Sierra, Decagon, PolyAI,
Voiceflow, Cognigy, Dialogflow CX, Genesys Cloud, NICE CXone. See the
integration spec for the per-platform verified facts and gaps.

## Invariants

Every adapter validates that a dual-channel file it scores has one party per
channel (2 channels) before producing separated talk-over numbers; mono stacks
are scored degraded only behind `--allow-mono`. All live fetch/list paths are
stdlib-only (`urllib`); stack SDKs import lazily and only inside your own infra.
Credentials captured by `hotato connect` are stored locally at
`~/.hotato/connections.json` (mode 0600) and are sent only to the vendor's own
API, never to Hotato. Scoring is always offline; the only network is the direct
recording download.


================================================================================
FILE: docs/STARTER.md
================================================================================

# The starter kit: `hotato init starter`

The fastest way to add hotato to an existing voice-agent repository. It
scaffolds the CI gate, a stack-tuned config file, and the three directories
the rest of the docs assume already exist, so you can go straight to turning
your first bad call into a contract instead of wiring plumbing by hand.

```bash
hotato init starter --stack vapi --out .
```

`--stack` is one of `vapi`, `retell`, `twilio`, `livekit`, `pipecat` -- every
stack hotato has a real, shipped connector for today (see
[`ADAPTER-STATUS.md`](ADAPTER-STATUS.md)). `--out` is usually `.`, the root of
the repo you are adding hotato to. Offline: no network, no credentials needed
to generate.

## What it writes

```
HOTATO.md                                # what was added, next steps (read this first)
hotato.yaml                              # config skeleton, tuned for --stack
.gitignore                               # excludes local/pulled recordings;
                                          #   keeps pinned fixture/contract clips committed
.github/workflows/hotato-contracts.yml   # the CI gate
fixtures/
  README.md
  scenarios/.gitkeep                     # -> hotato fixture create --out fixtures
  audio/.gitkeep
contracts/
  README.md
  .gitkeep                               # -> hotato contract create --out contracts
reports/
  README.md
  .gitkeep                               # local/CI scratch: doctor/report/sweep output
```

Every file is refused if it already exists, unless `--force` is passed --
nothing is silently merged or overwritten, and nothing partial is left behind
if the scaffold refuses. The generated file names are deliberately namespaced
away from a real repo's own files (`HOTATO.md`, not `README.md`;
`hotato-contracts.yml`, not `hotato.yml`) so a first run does not collide with
files a voice-agent repo almost always already has.

## Two input paths, chosen for you by `--stack`

**Auto-pull** (`vapi`, `retell`, `twilio`): hotato fetches the recording
itself once you connect a key. `hotato.yaml`'s `credentials.env` names the
exact environment variable(s) (`VAPI_API_KEY`; `RETELL_API_KEY`;
`TWILIO_ACCOUNT_SID` + `TWILIO_AUTH_TOKEN`) `hotato connect <stack>` also
reads. `recording.access` is `auto-pull`.

**Capture-in-your-infra** (`livekit`, `pipecat`): there is no vendor
recording API to pull from, so no credentials are generated or needed.
`hotato.yaml`'s `credentials.env` is `[]` and `recording.access` is
`capture-in-your-infra`; `hotato setup --stack <stack>` prints the exact
two-track capture scaffold, and you point `hotato contract create --stereo`
at the WAV your own deployment writes.

## LiveKit and Pipecat runbook

LiveKit and Pipecat are a foundational orchestration tier a lot of the
market builds on, and they are the two stacks where capture and the
turn-taking config both live in your own code rather than behind a vendor
API. This is the operator runbook for both, capture through CI.

### LiveKit

1. **Capture.** Two audio-only Track egresses, one per participant --
   RoomComposite mixes both parties into one channel and cannot attribute
   overlap. `hotato setup --stack livekit` prints the copy-paste scaffold
   (Python `livekit-api`, `TrackEgressRequest` + `DirectFileOutput`); a
   ready-to-copy version also lives at `adapters/livekit_capture.py`.
2. **Find the turn-taking config.** It lives on
   `AgentSession(turn_handling=TurnHandlingOptions(...))`: `turn_detection`
   (`inference.TurnDetector()` / `"realtime_llm"` / `"vad"` / `"stt"` /
   `"manual"`), `endpointing` (`min_delay`, `max_delay`), and `interruption`
   (`enabled`, `mode`, `min_duration`, `min_words`,
   `false_interruption_timeout`, `resume_false_interruption`). Read what a
   given agent file is ACTUALLY running, statically, before you propose
   changing anything: `hotato inspect --stack livekit --config agent.py`.
3. **Score it.**
   `hotato capture --stack livekit --caller caller.wav --agent agent.wav --onset <sec> --expect yield`
   (convert the egress output to WAV first, e.g. `ffmpeg -i caller.ogg caller.wav`).
4. **Fixture, contract, CI.** Same as every stack from here -- see "Turn
   your first bad call into a contract" below.

### Pipecat

1. **Capture.** A 2-channel `AudioBufferProcessor` in-pipeline (channel 0 =
   user/caller, channel 1 = bot/agent) -- do not mix down to one channel.
   `hotato setup --stack pipecat` prints the copy-paste scaffold; a
   ready-to-copy version also lives at `adapters/pipecat_capture.py`.
2. **Find the turn-taking config.** It lives on `PipelineTask`'s user-turn
   strategies: start strategies (`VADUserTurnStartStrategy`,
   `TranscriptionUserTurnStartStrategy`,
   `MinWordsUserTurnStartStrategy(min_words=...)`,
   `KrispVivaIPUserTurnStartStrategy(...)`) and stop strategies
   (`SpeechTimeoutUserTurnStopStrategy(user_speech_timeout=...)`,
   `TurnAnalyzerUserTurnStopStrategy(turn_analyzer=...)`); note
   `MinWordsInterruptionStrategy` is deprecated since pipecat 0.0.99 in
   favor of `MinWordsUserTurnStartStrategy`. Read what a given bot file is
   ACTUALLY running, statically: `hotato inspect --stack pipecat --config bot.py`.
3. **Score it.**
   `hotato capture --stack pipecat --stereo captured.wav --expect yield`
   (write the WAV from the `AudioBufferProcessor`'s `on_audio_data` handler
   first).
4. **Fixture, contract, CI.** Same as every stack from here -- see "Turn
   your first bad call into a contract" below.

Both APIs move; `hotato setup` and `hotato inspect` state the verified-against
date. Full field-level detail and provenance: [`ADAPTER-STATUS.md`](ADAPTER-STATUS.md)
(capture) and [`FIX-PLANS.md`](FIX-PLANS.md) (inspect, Level 1 of the fix ladder).

## The CI gate

`.github/workflows/hotato-contracts.yml` runs on push, on pull request, and
weekly. It is two guarded steps, both a **no-op, never a failure**, until you
have added a first contract or fixture (a fresh scaffold's normal starting
state):

```bash
hotato contract verify contracts --junit hotato.xml --format json > contracts-verify.json
hotato run --scenarios fixtures/scenarios --audio fixtures/audio --format json > fixtures-run.json
```

The JUnit file is published as a build artifact on every run (`always()`),
whether the gate passed, failed, or had nothing to check yet.

For the three auto-pull stacks, the workflow also carries a `weekly-sweep`
job: a passive, candidate-only sweep of recent calls
(`hotato sweep --stack <stack>`), ranked by hotato's own salience -- never a
verdict, never auto-labeled. It ships **disabled** (`if: false`): flip it to
`true` once the stack's credential env var(s) are set as repo secrets
(Settings -> Secrets and variables -> Actions). Hotato never runs a live pull
against your account on its own initiative; enabling this job is an explicit
human decision, made once, in your own CI config. `livekit`/`pipecat` carry
no such job -- there is no vendor recording API to sweep.

## Turn your first bad call into a contract

```bash
# auto-pull stacks
hotato connect vapi --api-key <key>
hotato sweep --stack vapi --out hotato-sweep.html
# open hotato-sweep.html, pick a real candidate moment, then:
hotato contract create --from-candidate hotato-sweep.json#1 \
    --expect yield --id refund-cutoff-001 --out contracts

# capture-in-your-infra stacks
hotato setup --stack livekit
# once your deployment writes a two-channel WAV:
hotato contract create --stereo call.wav --onset 42.18 \
    --expect yield --id refund-cutoff-001 --out contracts
```

Commit the resulting `contracts/refund-cutoff-001.hotato/` directory. The
next push runs it through the CI gate above.

## Read more

- The bundle layout and the create/verify/inspect/pack/unpack commands:
  [`CONTRACTS.md`](CONTRACTS.md)
- The underlying fixture primitive, one bad call to a CI gate in five steps:
  [`BAD-CALL-TO-CI.md`](BAD-CALL-TO-CI.md)
- Per-stack connector support, verified against the vendor's live docs:
  [`ADAPTER-STATUS.md`](ADAPTER-STATUS.md)
- The connect-once bulk pull-and-analyze recipe: [`CONNECT.md`](CONNECT.md)
- An agent adding hotato to a repo end to end: [`../AGENTS.md`](../AGENTS.md)


================================================================================
FILE: docs/TRACE.md
================================================================================

# Voice traces

A voice trace is a timeline of discrete voice-pipeline events (caller/agent
audio activity, TTS cancel/stop, ASR partials, tool calls, ...) that
supplements a failure contract's frame-level timing evidence with the WHY
layer a caller/agent audio track alone cannot show: "the agent talked over
the caller" becomes "evidence suggests TTS cancellation lagged: cancel
requested at 42.40s, audio stopped at 43.60s".

Hotato does not prove root cause from a trace. It reports coincidence: a
pattern of events that lined up with the measured timing. See
[`docs/OTEL.md`](OTEL.md) for the ingest source formats.

Three commands:

```bash
hotato trace ingest --otel traces.jsonl --out voice_trace.jsonl
hotato trace attach contracts/refund-cutoff-001.hotato --trace voice_trace.jsonl
hotato trace export contracts/refund-cutoff-001.hotato --format otel --out otel.jsonl
```

## Schema

`voice_trace.jsonl` is JSONL (one meta line, then one line per span -- the
same convention `evidence/frames.jsonl` uses), validating against
`hotato.voice_trace.v1` (`src/hotato/schema/voice_trace.v1.json`) once
reassembled by `hotato.trace.load_voice_trace_jsonl`:

```json
{
  "schema": "hotato.voice_trace.v1",
  "call_id": null,
  "deployment": {"stack": "vapi", "agent_id": null, "git_sha": "deadbeef", "config_hash": "sha256:..."},
  "spans": [
    {"type": "agent_audio_active", "start_sec": 0.0, "end_sec": 2.9},
    {"type": "tts_cancel_requested", "time_sec": 2.6},
    {"type": "tts_audio_stopped", "time_sec": 2.9},
    {"type": "asr_partial", "start_sec": 2.4, "end_sec": 2.95, "text_redacted": true},
    {"type": "tool_call", "start_sec": 1.1, "end_sec": 1.42, "name": "lookup_order", "latency_ms": 320}
  ],
  "source": {"format": "otel-jsonl-bridge", "input_span_count": 6}
}
```

A span carries an open `type` string; the common ones are
`caller_audio_active`, `agent_audio_active`, `tts_cancel_requested`,
`tts_audio_stopped`, `asr_partial`, `tool_call`, `llm_first_token`,
`handoff`. An unrecognized type is passed through unchanged, never dropped
(additive, forward-compatible with a pipeline that emits a span type this
release does not name). Exactly one time shape applies per span: an
interval span carries `start_sec`/`end_sec`, a point event carries
`time_sec`.

## Redaction by default

`call_id` and `deployment.agent_id` are dropped (`null`) unless
`--include-identifiers` is passed at ingest time. An `asr_partial` span's
transcript text is dropped (`text_redacted: true`, no `text` key) unless
`--include-text` is passed. `deployment.stack` / `git_sha` / `config_hash`
are not treated as identifiers and are kept by default.

## Ingest

```bash
hotato trace ingest --otel traces.jsonl --out voice_trace.jsonl
hotato trace ingest --otel export.json --out voice_trace.jsonl \
    --stack vapi --include-identifiers --include-text
```

`--otel FILE` accepts either a standard OTel JSON export (a document with a
top-level `resourceSpans` array) or hotato's own documented OTel bridge
JSONL (see [`docs/OTEL.md`](OTEL.md)). `--stack` / `--call-id` /
`--agent-id` / `--git-sha` / `--config-hash` override or fill in whatever
the source's own resource attributes carried. Refused (exit 2, nothing
written) for an unreadable file, an empty file, or a source with zero
spans.

## Attach

```bash
hotato trace attach contracts/refund-cutoff-001.hotato --trace voice_trace.jsonl
```

Copies the trace into `<bundle>/traces/voice_trace.jsonl` and re-renders
`evidence/timeline.html` with the trace's events drawn as an additional row,
aligned to the SAME [0, duration] scale as the existing caller/agent
timeline. This reads the bundle's OWN `evidence/frames.jsonl` and
`contract.json` back in -- it never re-runs the VAD or the diarizer, so
attaching a trace never needs the diarization extra installed and never
re-scores the audio. On a diarized-mono bundle (no frame-level evidence),
the base timeline honestly states that instead of fabricating one, and the
trace row still renders on its own scale.

`contract.json` records the attachment (additive, schema-safe: `trace:
{attached, path, span_count, attached_at, source_format}`). Refused (exit 2)
for a missing bundle, a trace file that does not validate as a
`hotato.voice_trace.v1` JSONL, or an already-attached trace without
`--force`.

### Report wording

When a `tts_cancel_requested` / `tts_audio_stopped` pair is present, the
timeline states the measured delta plainly:

> Evidence suggests TTS cancellation delay: cancel requested at 2.60s, audio
> stopped at 2.90s (delta 0.30s).
> Hotato does not prove root cause.
> Unknowns: no client-side playout trace was attached.

The last line is always present in this release: a client-side audio
playout trace (the point where the CALLER'S device actually stopped
rendering audio, as opposed to when the server issued the stop) is not a
span type this release collects, so the honest gap is always named rather
than silently omitted.

## Export

```bash
hotato trace export contracts/refund-cutoff-001.hotato --format otel --out otel.jsonl
```

Writes the bundle's attached trace back out as hotato's OTel bridge JSONL --
the exact shape `trace ingest` reads, so `ingest -> attach -> export ->
ingest` round-trips the identical spans. `--format` is claimed by the export
format on this subcommand (only `otel` is supported today); pass `--json`
for the machine result summary instead of the default text line. Refused
(exit 2) when the bundle has no attached trace, or `--out` exists without
`--force`.

## What a voice trace does not prove

Hotato does not prove authorization, identity, compliance, or policy
safety. A voice trace adds timing correlation to a failure contract's
existing timing measurement; it never adds intent, never a root-cause
verdict, and never a claim that a client-side playout event happened at a
particular moment unless one was actually attached.

## Read more

- Source formats and the OTel bridge JSONL shape:
  [`docs/OTEL.md`](OTEL.md)
- Failure contracts a trace attaches to: [`docs/CONTRACTS.md`](CONTRACTS.md)


================================================================================
FILE: docs/OTEL.md
================================================================================

# OTel ingest: two source shapes

`hotato trace ingest --otel FILE` recognizes two input shapes. Both convert
into the same `hotato.voice_trace.v1` spans; which one it used is recorded
in `source.format` (`"otel-json"` or `"otel-jsonl-bridge"`).

This is a bridge, not an OTel collector: it reads a file you already have
(an exported trace, or a small script's own event log), offline, once. It
is not a running OTel receiver, does not speak the OTLP wire protocol, and
does not claim full OpenTelemetry semantic-convention coverage -- only
`name`, `startTimeUnixNano`/`endTimeUnixNano`, `attributes`, and span
`events` are read from a standard export.

## 1. Standard OTel JSON export (`otel-json`)

A single JSON document with a top-level `resourceSpans` array (the shape a
real OTel exporter/collector writes):

```json
{
  "resourceSpans": [{
    "resource": {"attributes": [
      {"key": "service.name", "value": {"stringValue": "vapi"}},
      {"key": "git_sha", "value": {"stringValue": "cafebabe"}}
    ]},
    "scopeSpans": [{"spans": [
      {
        "name": "agent_audio_active",
        "startTimeUnixNano": "1000000000",
        "endTimeUnixNano": "4400000000"
      },
      {
        "name": "tts.cancel_requested",
        "startTimeUnixNano": "2600000000",
        "events": []
      }
    ]}]
  }]
}
```

- Resource attributes are flattened into a plain dict; `service.name`
  (or a custom `stack` attribute) becomes `deployment.stack`, `git_sha` /
  `config_hash` attributes (if present) fill the matching deployment
  fields. Only the FIRST `resourceSpans` entry's resource is used for
  deployment metadata; every entry's spans are still walked.
- Timestamps convert to seconds relative to the EARLIEST timestamp anywhere
  in the file (matching the audio-relative-seconds convention every other
  hotato timestamp uses) -- not wall-clock time.
- A span's own `name` maps to a hotato span `type` via a small documented
  table (`caller_audio_active`, `agent_audio_active`,
  `tts.cancel_requested` -> `tts_cancel_requested`,
  `tts.audio_stopped` -> `tts_audio_stopped`, `asr.partial` -> `asr_partial`,
  `llm.first_token` -> `llm_first_token`); an unmapped name passes through
  unchanged.
- Span `events` (OTel's own point-in-time markers nested inside a span --
  the natural place a real pipeline would put a `tts.cancel_requested`
  marker inside a broader `tts_playback` span) are flattened into their own
  point events the same way top-level spans are.
- A `tool_call`-mapped span's `tool.name` / `gen_ai.tool.name` attribute
  becomes the span's `name` field; a `latency_ms` attribute is used if
  present, otherwise computed from the span's own start/end.
- An `asr_partial`-mapped span's `text` / `asr.transcript.partial` attribute
  is captured but immediately redacted (dropped, `text_redacted: true`)
  unless `--include-text` was passed.

## 2. Hotato's OTel bridge JSONL (`otel-jsonl-bridge`)

The simpler, documented shape for a script or test fixture that does not
have a real OTel exporter: one JSON object per line (or one bare JSON array
of the same objects). Each line is either a span or a meta/resource line.

A span line uses `type` directly (never `name` -- `name` on a span line is
reserved for `tool_call`'s own tool name, not the span kind):

```
{"type": "caller_audio_active", "start_sec": 2.40, "end_sec": 4.10}
{"type": "agent_audio_active", "start_sec": 0.00, "end_sec": 2.90}
{"type": "tts_cancel_requested", "time_sec": 2.60}
{"type": "tts_audio_stopped", "time_sec": 2.90}
{"type": "asr_partial", "start_sec": 2.40, "end_sec": 2.95, "text": "wait, I need a refund"}
{"type": "tool_call", "start_sec": 1.10, "end_sec": 1.42, "name": "lookup_order", "latency_ms": 320}
```

An optional meta/resource line (no `type` key) attaches deployment
metadata and a call id:

```
{"call_id": "demo-call-001", "deployment": {"stack": "vapi", "agent_id": "agent-demo-1", "git_sha": "deadbeefcafe", "config_hash": "sha256:0f1e2d3c"}}
```

`--call-id` / `--stack` / `--agent-id` / `--git-sha` / `--config-hash` on
`trace ingest` override whatever a meta line (or a standard export's
resource) provided. `hotato trace export` writes exactly this bridge
shape, so `ingest -> attach -> export -> ingest` round-trips the same
spans; the shipped test fixture (`tests/data/otel/demo-trace.otel.jsonl`)
uses this shape.

## Which one to use

If you already run an OTel collector or SDK and can dump a trace export,
point `--otel` at that file directly -- the standard-export path is
best-effort but reads it. If you are wiring a quick script (a webhook
handler, a log-line scraper) that does not go through OTel at all, write the
bridge JSONL directly; it is the same information with none of the nesting.


================================================================================
FILE: docs/REPORTS.md
================================================================================

# Reports: doctor, report, team, export

Four surfaces over the same scorer. Every number in every one of them is a
measurement from the envelope; nothing is recomputed, restyled into a
percentage, or fabricated. There is no accuracy percentage anywhere.

## `hotato doctor`: the 5-minute path

One command. If you pass a recording it scores that; otherwise it runs the
bundled self-test battery. Either way it writes the self-contained HTML report
and tries to open it in your browser (on a headless box it prints the path).

```bash
uvx hotato doctor --stereo call.wav     # score your call, open the report
uvx hotato doctor                       # self-test fallback, same flow
uvx hotato doctor --demo --no-open --out report.html
```

It is a convenience wrapper over the existing scorer and report. Nothing new is
claimed, and everything runs offline. Exit codes match `run`: `0` all pass,
`1` a regression (`--no-fail` forces `0`), `2` usage or IO error, or a
recording that is not scorable. Not scorable means the recording cannot answer
the question (the caller channel is silent, or the agent was not talking when
the caller started), so no verdict is given.

## `hotato report`: the visual report

One self-contained file: inline CSS, inline SVG, zero external requests. It
opens offline by double-click and survives being mailed around.

```bash
uvx hotato report --stereo call.wav --out report.html
uvx hotato report --suite barge-in --out selftest.html
uvx hotato report --stereo call.wav --format md --out report.md
```

Per event it draws a to-scale caller/agent activity timeline from the
frame data: the overlap shaded, the caller-onset and yield markers, the
measured talk-over seconds, expected vs actual, a PASS or FAIL chip, and the
exact `ScoreConfig` thresholds used.

Above the per-event cards sits an analytics block computed from the same
measurements:

- a **time-to-yield distribution** strip, one dot per measured yield, with
  mean, median, and p90 (definitions in `METHODOLOGY.md`);
- a **talk-over histogram**, per-event seconds bucketed on a fixed grid;
- **failure clustering by fix class**, so a batch of failures reads as "these
  five share one config setting" instead of five separate mysteries.

Every timeline carries a collapsible **frame inspector**: the full frame dump
behind that event as a table (`t_sec`, per-channel dBFS, active flags,
thresholds), so any pixel on the page can be re-derived by hand.

### Regression deltas with `--base`

Save an envelope, then compare a later run against it:

```bash
hotato run --suite barge-in --format json > base.json
hotato report --suite barge-in --base base.json --out report.html
```

The report renders per-scenario talk-over and time-to-yield deltas with clear
worse and better marks. The same `--base` flag works on
`scripts/pr_comment.py` for the CI comment (`docs/CI.md`).

### PDF

The page ships print CSS. Print it from any browser and the interactive parts
collapse into a clean paper layout, so print-to-PDF is the PDF export.

## `hotato team`: the trend view

Aggregates a directory of run envelopes into one trend.

```bash
hotato run --suite barge-in --format json > runs/001.json
# ... more runs over days or branches ...
hotato team runs/ --html team.html --out agg.json
```

It reports: number of runs, mean/median/p90 talk-over and time-to-yield pooled
across all events, mean/median/p90/p95 response gap (dead air before the agent
speaks) pooled the same way, pass rate per run over time, the most common
failure class, and a pass-rate trend line in the HTML page. `--order mtime`
(default) orders runs by file time; `--order name` uses the filename, so a
numeric prefix is an explicit index.

`--max-response-gap SECONDS` turns the pooled p95 response gap into a latency
SLA: the run exits `1` exactly when p95 exceeds the bound, the same
pass/fail contract as a talk-over or time-to-yield regression (`--no-fail`
always exits `0`). Percentile definitions: `METHODOLOGY.md`; the pooling
shape is `dist_summary` in `src/hotato/_stats.py`.

Fewer than two runs is stated plainly and exits `0`. It is never padded into a
trend, because a trend of one point is a fabrication.

## `hotato export`: research-grade CSVs

Scores a recording (or the bundled battery) exactly like `hotato run` and
writes three files into a directory:

```bash
uvx hotato export --stereo call.wav --out research/
uvx hotato export --suite barge-in --out research/
```

- `events.csv`: one row per scored event, every measured signal plus verdict.
- `frames.csv`: one row per VAD frame, the evidence behind every number.
- `envelope.json`: the standard machine envelope, unchanged.

Column meanings are documented in comment lines at the top of each CSV, so the
files are self-describing when they land in a notebook or a stats package
months later. An empty cell means "not derivable", never zero. Stdlib only,
offline.

`export` also prints mean/median/p90/p95 response gap pooled across the
exported events, and accepts the same `--max-response-gap SECONDS` latency
SLA gate as `team` (exit `1` when the pooled p95 exceeds the bound). A plain
export with no `--max-response-gap` writes a byte-identical `envelope.json`;
the pooled numbers and the gate live only in the printed summary and the
returned manifest, never in the CSVs or the envelope file.


================================================================================
FILE: docs/PYTEST.md
================================================================================

# Pytest: the fixture and the gate

Install hotato and the plugin registers itself (a standard `pytest11` entry
point). No `conftest.py` line, no import. It adds one fixture and one opt-in
session gate. Both score with the same engine as the CLI and return the same
envelope, the JSON result object every hotato surface emits.

```bash
pip install hotato
```

## The `hotato_score` fixture

Score a recording or a suite inside any test and assert on the envelope. Same
inputs as the CLI:

```python
def test_call_yields(hotato_score):
    env = hotato_score(stereo="call.wav", expect="yield")
    assert env["exit_code"] == 0
    assert all(e["verdict"]["passed"] for e in env["events"])

def test_split_channels(hotato_score):
    env = hotato_score(caller="caller.wav", agent="agent.wav", expect="yield")
    assert env["events"][0]["verdict"]["passed"]

def test_selftest_battery(hotato_score):
    env = hotato_score(suite="barge-in")
    assert env["exit_code"] == 0
```

The envelope is the standard machine shape (`schema_version` "1"): per event a
`verdict` (with `passed`, `did_yield`, `seconds_to_yield`, `talk_over_sec`), a
`signals` bus, a `fix` on every failure, and the `limits` block. Assert on
whichever measurement your test cares about.

## The `--hotato-suite` session gate

Opt in on the command line and the battery runs after your tests, printing a
summary and failing the whole session (exit 1) on a regression:

```bash
pytest --hotato-suite                    # default suite: barge-in
```

The bundled battery is a self-test of the harness. To gate on your own agent,
point the same flag at your own labelled scenario and audio directories:

```bash
pytest --hotato-suite \
  --hotato-suite-scenarios corpus/suites/gold/scenarios \
  --hotato-suite-audio corpus/suites/gold/audio
```

Any directory in the same scenario shape works, including your own labelled
calls (see `docs/SUITES.md` for the bundled tiers and `docs/SUBMITTING.md` for
building labelled fixtures from real recordings).

## Where it fits

- The gate is one flag on a test run you already have, so turn-taking rides
  along with every `pytest` invocation, locally and in CI.
- The GitHub workflow (`docs/CI.md`) is the other ready-made gate: it scores on
  every pull request and posts a sticky results comment. Use either, or both.
- Everything runs offline; the plugin makes no network call and the fixtures
  are deterministic, so the gate cannot flake on I/O it does not do.


================================================================================
FILE: docs/SUITES.md
================================================================================

# Corpus suites: tiered, deterministic, explicit about what they prove

`corpus/suites/` ships four labelled scenario suites, 112 scenarios total,
every one synthetic shaped noise rendered deterministically from its own
labelled timings (seed `sha256(scenario_id)`). The timings are the ground
truth. Synthetic audio is the floor: these suites prove the scorer runs end to
end and catch regressions; validity on your system comes from your own labelled
calls (`docs/SUBMITTING.md`).

## The four suites

| Suite | Scenarios | Conditions | Expected exit |
|---------------------|-----------|-----------------------------------------------------------------------|---------------|
| `silver` | 40 | clean, 16 kHz, default noise floor | `0` |
| `silver-defects` | 16 | clean conditions, deliberate defect renders | `1` |
| `gold` | 40 | hard conditions: noise floors, 8 kHz, gain extremes, echo, edge timings, endurance | `0` |
| `gold-defects` | 16 | hard-condition defect renders, plus two labelled capture-defect cases | `1` |

The scenario families span the behaviours that matter on a real call: hard
interruptions (onset, speed, duration, resume), backchannels, short
acknowledgements like "mhm" that the agent should talk through (varied in
position, density, repeats, and length), double-talk, one-word interruptions,
stutter onsets, multi-turn exchanges, resume-then-reinterrupt, and latency
prompts.
`corpus/suites/manifest.json` is the machine-readable inventory: per suite the
family and category breakdown, sample rates, and the expected exit code.

## Defect suites fail by design

Every scenario in `silver-defects` and `gold-defects` is rendered to fail on
its labelled axis: an agent that keeps talking through a real interruption, or
stops for a backchannel, or misses its latency budget. A defect suite that
exits `1` is the scorer catching what it claims to catch; a defect suite that
exits `0` would be a bug. This is the negative control the positive suites
need to mean anything.

## Run a suite

Any suite is just a scenarios directory plus an audio directory:

```bash
hotato run --suite barge-in \
  --scenarios corpus/suites/gold/scenarios \
  --audio corpus/suites/gold/audio
```

The same directories plug into every other surface: `hotato report ... --out
report.html` for the visual report, `hotato export ... --out research/` for
CSVs, and `pytest --hotato-suite --hotato-suite-scenarios ... --hotato-suite-audio ...`
for the session gate (`docs/PYTEST.md`).

## Deterministic builder

The suites are generated, not hand-edited, and the generator is the proof:

```bash
python3 corpus/suites/build_suites.py           # rebuild in place
python3 corpus/suites/build_suites.py --check   # regenerate to a temp dir, byte-compare
```

`--check` regenerating byte-identical output is the reproducibility guarantee:
the audio on disk is exactly what the labelled timings say it is, on any
machine. CI can run it as a drift gate.

## Additive scenario classes

`corpus/classes/` ships four small, deterministic classes on top of the four
suites above, built the same way (synthetic shaped noise, seed
`sha256(scenario_id)`, `--check` byte-compares a rebuild): `mid-utterance-pause`
(a multi-second thinking gap mid-turn, not a true turn end), `backchannel-multilingual`
(short non-English acknowledgement tokens; Hotato's VAD is energy-based and
does not detect language), `noise-hold` (sustained background presence, not a
brief backchannel, that the agent should hold through), and
`telephony-degraded` (an existing gold scenario re-rendered through G.711
mu-law plus a fixed packet-loss schedule, to prove the verdict is stable
across codec degradation). Kept separate from `corpus/suites/` because
`mid-utterance-pause` needs a non-default `turn_end_silence_sec` that the
generic suite tests do not apply. Full per-class detail: `corpus/classes/README.md`.

## Against a live stack

To run scenario audio through a real voice stack and score what comes back,
see [`docs/BENCHMARK-STACKS.md`](BENCHMARK-STACKS.md). Measurement-error
methodology for the scorer itself is in [`docs/BENCHMARK.md`](BENCHMARK.md).

## Contribute scenarios

New synthetic families are welcome through the normal PR path
(`CONTRIBUTING.md`). The highest-value contribution stays a real, consented,
labelled call: the walkthrough is [`docs/SUBMITTING.md`](SUBMITTING.md) and the
[corpus-submission issue form](https://github.com/attenlabs/hotato/issues/new?template=corpus_submission.yml)
opens the intake.


================================================================================
FILE: docs/CI.md
================================================================================

# CI: gate a PR on turn-taking

hotato scores turn-taking timing from call recordings, so a pull request can
carry a running score and fail when the agent gets slower to stop talking for
an interrupting caller. The workflow runs offline, needs no extra dependencies,
and posts one comment that updates itself on every push.

## Drop it in

Copy [`.github/workflows/hotato.yml`](../.github/workflows/hotato.yml) into your
repository at the same path. That is the whole setup. On the next pull request it
will:

- install the package with `pip install .`
- score the bundled barge-in suite to a JSON envelope
- render a Markdown summary with [`scripts/pr_comment.py`](../scripts/pr_comment.py)
- post or update one sticky comment (found by a hidden marker, so it never spams)
- fail the job on any regression

The sticky comment shows a pass/fail line, a per scenario table (expect, yielded,
time to yield, talk over, result), and a short regressions section.

The job needs `pull-requests: write` to post the comment. The workflow already
requests it; if your org restricts the default `GITHUB_TOKEN`, allow that scope.

## Point it at your own recordings

The bundled suite is a self-test: it scores frozen synthetic fixtures to prove
the harness works, not to judge your agent. The strongest gate is a suite of
your OWN bad moments, pinned as fixtures with `hotato fixture create` and run
with `hotato run --scenarios DIR --audio DIR`; the full loop from one bad call
to this gate is [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md). Alternatively, replace
one step, `Score turn-taking (head)`, with your own capture and score:

1. play each corpus `*.caller.wav` into your agent
2. record your agent's reply
3. score the pair, caller on channel 0 and your agent on channel 1:

```bash
hotato run --stereo your_call.wav --expect yield --format json --no-fail > head.json
```

The envelope shape and exit codes are identical, so the render, comment, and gate
steps stay exactly as they are. Keep `--no-fail` on the score step so the comment
still posts on a regression; the true pass/fail lives in the envelope's
`exit_code`, which the `Fail on regression` step reads.

## Deltas against the base branch

When the workflow can install and score the base branch, it runs the same suite
there in an isolated venv and uses it as the baseline. Any scenario where the
overlap grew (talk-over up) or the agent stopped later (time to yield up) is
listed under Regressions with the delta. This step is best effort: if it cannot
run, the comment falls back to the current pass/fail table and the gate still
holds.

## Render a comment yourself

`scripts/pr_comment.py` is stdlib only and reads the same JSON the CLI emits, so
you can preview the comment locally:

```bash
hotato run --suite barge-in --format json | python3 scripts/pr_comment.py
```

Add `--base base.json` to include deltas against a saved baseline run.

## The other ready-made gate: pytest

If your CI already runs pytest, one flag adds the same regression gate to that
run instead: `pytest --hotato-suite` scores the battery after your tests and
fails the session on a regression. Point it at your own labelled sets with
`--hotato-suite-scenarios` and `--hotato-suite-audio`. Details:
[`PYTEST.md`](PYTEST.md). For richer artifacts on a gate failure,
`hotato report` renders the visual report and `hotato team` tracks the trend
across runs ([`REPORTS.md`](REPORTS.md)).


================================================================================
FILE: docs/MCP.md
================================================================================

# MCP: the one-tool server

hotato ships a one-tool MCP server, `hotato-mcp`, that speaks MCP over stdio
and exposes exactly one tool, `voice_eval_run`, returning the identical JSON
envelope (`schema_version` "1") the CLI emits. Everything runs locally; no
audio leaves the machine.

## Run it (zero-install)

```bash
uvx --from "hotato[mcp]" hotato-mcp
```

**Common mistake:** `uvx hotato-mcp` (no `--from`) FAILS. uv then looks for a
package literally named `hotato-mcp` on PyPI, which does not exist; the
console script `hotato-mcp` lives inside the `hotato` distribution, installed
with its `mcp` extra. If you (or an agent) just tried the bare form and got a
"package not found" error, retry with `--from "hotato[mcp]"` exactly as
written above. If hotato is already installed in your environment,
`python -m hotato.mcp_server` also works and needs no extra invocation syntax.

## Add it to a client

Every block below is copy-paste exact; only the outer key (client-specific)
differs. All three use the same command and args.

### Claude Desktop

Edit `claude_desktop_config.json` (Settings -> Developer -> Edit Config):

```json
{
  "mcpServers": {
    "hotato": {
      "command": "uvx",
      "args": ["--from", "hotato[mcp]", "hotato-mcp"]
    }
  }
}
```

### Cursor

Project-scoped: `.cursor/mcp.json` in your repo root. User-scoped:
`~/.cursor/mcp.json`.

```json
{
  "mcpServers": {
    "hotato": {
      "command": "uvx",
      "args": ["--from", "hotato[mcp]", "hotato-mcp"]
    }
  }
}
```

### Codex CLI

Edit `~/.codex/config.toml`:

```toml
[mcp_servers.hotato]
command = "uvx"
args = ["--from", "hotato[mcp]", "hotato-mcp"]
```

## The one tool: `voice_eval_run`

Same two input modes as the CLI, all parameters optional beyond the mode you
pick:

| Parameter | Type | Default | Meaning |
| --- | --- | --- | --- |
| `stereo` | str | None | two-channel WAV path |
| `caller` | str | None | mono caller WAV (with `agent`) |
| `agent` | str | None | mono agent WAV (with `caller`) |
| `suite` | str | None | `"barge-in"` to run the bundled battery |
| `stack` | str | `"generic"` | livekit, pipecat, vapi, or generic |
| `expect` | str | `"yield"` | `"yield"` or `"hold"` |
| `onset_sec` | float | None | caller onset hint |
| `caller_channel` | int | 0 | caller channel index |
| `agent_channel` | int | 1 | agent channel index |
| `max_talk_over_sec` | float | None | pass threshold |
| `max_time_to_yield_sec` | float | None | pass threshold |
| `report_path` | str | None | also write the HTML report here; the envelope then carries `report_path` (absolute) |

Pass exactly one input mode: `stereo`, OR `caller` + `agent` together, OR
`suite`. Mixing modes (or passing none) is a structured error, never a raw
exception; see below.

## Output

On success, the identical envelope the CLI emits for `--format json`, schema
at [`https://hotato.dev/schema/envelope.v1.json`](https://hotato.dev/schema/envelope.v1.json)
(`schema_version` "1", additive-only: new keys may appear, the documented
core never changes).

Every expected failure (a missing / mono / mismatched / not-found file, an
unknown suite, an ambiguous input mode, or a well-formed input with no
scorable event) comes back as a structured error object, `ok: false` with a
stable `error_code` and an actionable `message`, schema at
[`https://hotato.dev/schema/error.v1.json`](https://hotato.dev/schema/error.v1.json)
-- never a raw uncaught exception. The MCP tool and the CLI share this one
error shape, so a caller (human or agent) parses one contract across both
surfaces.

## More

- Python API and the shared error contract: [`API.md`](API.md)
- What the envelope measures and the scope/ceiling: the top-level
  [`README.md`](../README.md) and [`METHODOLOGY.md`](../METHODOLOGY.md)
- Machine-readable index of every command (CLI and MCP): [`llms.txt`](../llms.txt)


================================================================================
FILE: docs/API.md
================================================================================

# Python API reference

Everything the CLI, the pytest plugin, and the MCP server do is a plain Python
call. The core is stdlib only, fully offline, and deterministic: the same audio
and config always produce the same envelope. Import and score:

```python
from hotato.core import run_single, run_suite

env = run_single(stereo="call.wav", expect="yield")
env = run_suite()  # bundled 8-scenario battery
```

The top-level package re-exports the essentials:
`from hotato import run_single, run_suite, LIMITS, SUITE_ID, __version__`.

All scoring functions take keyword arguments only.

## The envelope

`run_single` and `run_suite` return the same machine-readable dict
(JSON Schema: `src/hotato/schema/envelope.v1.json`):

```python
{
  "tool": "hotato",
  "schema_version": "1",
  "mode": "single" | "suite",
  "stack": "generic",            # normalized stack label
  "offline": True,
  "engine": {"name", "version", "upstream"},
  "limits": {...},               # scope and ceiling, hotato.core.LIMITS
  "summary": {"events", "passed", "failed", "regression"},
                                 # plus additive "not_scorable" (count) when
                                 # at least one event could not be judged
  "events": [...],               # one dict per scored event, below
  "fix_map": [...],              # one entry per failing event with a fix
  "funnel": {...} | None,        # systemic pointer, fires only when both axes fail
  "exit_code": 0 | 1,            # 1 when any scorable event failed; the CLI
                                 # process exits 2 for a single recording that
                                 # is not scorable (see Exit codes below)
  "suite": "barge-in",           # run_suite only
}
```

Each event:

```python
{
  "event_id": str,               # file basename or scenario id
  "scenario_id": str | None,
  "title": str | None,
  "category": str | None,        # e.g. "should_yield"
  "expected_yield": bool,
  "verdict": {
    "passed": bool,
    "did_yield": bool,
    "seconds_to_yield": float | None,
    "talk_over_sec": float,
    "reasons": [str],            # failure reasons, empty on pass
  },
  "measurements": {
    "caller_onset_sec": float,
    "agent_talking_at_onset": bool,
    "hop_sec": float,
    "notes": str,
  },
  "signals": {                   # namespaced signal bus, additive
    "barge_in": {"did_yield", "time_to_yield_sec", "talk_over_sec"},
    "latency": {"response_gap_sec", "premature_start_sec"},
    "echo": {                    # every event; cross-channel coherence,
                                 # deterministic, computed in hotato's own
                                 # layer (hotato/echo.py), never _engine
      "coherence", "lag_sec", "echo_suspected",
    },
    "resume": {                  # only on events where the agent yielded
      "resumed", "resume_gap_sec", "restart_suspected",
    },
  },
  "fix": None | {                # set on failing events
    "fix_class": "config" | "engagement-control",
    "title": str, "detail": str,
    "knob": str | None, "pointer": str | None,
  },
}
```

An event that cannot be judged (a silent caller channel with no onset label, or
a should-yield expectation with the agent silent at onset) additionally carries
`"scorable": False` and a plain `"not_scorable_reason"`. It counts in neither
`passed` nor `failed`, never routes a fix, and never fires the funnel: an input
problem is reported as one, not dressed up as an agent verdict. Envelopes for
valid recordings are byte-identical to before these keys existed.

## hotato.core

### run_single

```python
run_single(
    *,
    stereo: str | None = None,        # two-channel WAV path
    caller: str | None = None,        # mono WAV path (with agent=)
    agent: str | None = None,         # mono WAV path (with caller=)
    caller_channel: int = 0,
    agent_channel: int = 1,
    onset_sec: float | None = None,   # caller onset hint, seconds from start
    expect: str = "yield",            # "yield" or "hold" (backchannel)
    stack: str | None = None,         # livekit | pipecat | vapi | generic
    max_talk_over_sec: float | None = None,
    max_time_to_yield_sec: float | None = None,
    echo_gate: bool = False,          # hold an echo-suspected yield out of
                                      # the verdict (scorable: false) instead
                                      # of counting it as a clean pass
    cfg: ScoreConfig | None = None,
) -> dict
```

Scores one recording and returns the envelope. Provide either `stereo` or both
`caller` and `agent`. `expect="hold"` means the caller's speech is a
backchannel, a short acknowledgement like "mhm", and a correct agent keeps
talking through it. The two `max_*` thresholds tighten the pass criteria.
`echo_gate` is opt-in and off by default; `signals.echo` is always computed
and reported either way. Malformed or truncated WAVs raise a clean `ValueError` with the
ffmpeg export line to fix them.

### run_suite

```python
run_suite(
    *,
    suite: str = "barge-in",          # the only suite id (SUITE_ID)
    stack: str | None = None,
    scenarios_dir: str | None = None, # your scenario JSON labels
    audio_dir: str | None = None,     # your recordings, <scenario-id><suffix>
    suffix: str = ".example.wav",
    caller_channel: int = 0,
    agent_channel: int = 1,
    echo_gate: bool = False,          # see run_single
    cfg: ScoreConfig | None = None,
) -> dict
```

Runs a labelled battery and returns the envelope with a `suite` key. Defaults
to the bundled 8-scenario battery shipped inside the package, zero external
files. Point `scenarios_dir` and `audio_dir` at your own labelled set (for
example `corpus/suites/gold/scenarios` and `.../audio`). Suite audio must be
two-channel.

### dump_frames_for_input

```python
dump_frames_for_input(
    *,
    stereo: str | None = None,
    caller: str | None = None,
    agent: str | None = None,
    caller_channel: int = 0,
    agent_channel: int = 1,
    onset_sec: float | None = None,
    cfg: ScoreConfig | None = None,
) -> dict
```

The per-frame evidence behind every reported number: each channel's dBFS, VAD
activity, threshold, and noise floor, plus a self-describing `config` block.
Every reported signal is re-derivable by hand from this dump.

### LIMITS and SUITE_ID

`hotato.core.LIMITS` is the scope dict embedded in every envelope
(method, ceiling, best input, what it does not do). `SUITE_ID` is `"barge-in"`.

### ScoreConfig

Every threshold is an exposed parameter:

```python
from hotato._engine.score import ScoreConfig
from hotato._engine.vad import VADParams

cfg = ScoreConfig(
    frame_ms=20.0, hop_ms=10.0,
    yield_hangover_sec=0.20,       # agent quiet this long = yielded
    max_search_sec=3.0,            # yield search window after onset
    caller_proximity_sec=0.5,
    turn_end_silence_sec=0.20,
    premature_tolerance_sec=0.05,
    onset_min_run_sec=0.05,
    agent_onset_lookback_sec=0.10,
    caller_vad=VADParams(),        # rel_db=15.0, abs_gate_db=-60.0,
    agent_vad=VADParams(),         # hangover_sec=0.15, noise_percentile=0.10,
)                                  # dyn_margin_db=22.0, backend="energy"
```

`VADParams.backend` is `"energy"` (the deterministic reference behind every
published number) or `"neural"` (an optional Silero VAD cross-check via
`pip install 'hotato[neural]'`; without the extra it raises a clean
`BackendUnavailable`, never a silent fallback).

## hotato.report

Self-contained visual reports scored from the same measurements. All three
functions accept the full scoring parameter set (`stereo`, `caller`, `agent`,
`suite`, `scenarios_dir`, `audio_dir`, `suffix`, `caller_channel`,
`agent_channel`, `onset_sec`, `expect`, `stack`, `max_talk_over_sec`,
`max_time_to_yield_sec`, `cfg`) as keyword arguments.

```python
build_report_html(*, base: dict | None = None,
                  base_label: str | None = None, **kwargs) -> (str, dict)
build_report_md(*, base: dict | None = None,
                base_label: str | None = None, **kwargs) -> (str, dict)
write_report(path: str, fmt: str = "html", **kwargs) -> dict
```

`build_report_html` scores the input and returns `(html, envelope)`: one
self-contained file, inline CSS and SVG, per-event timelines, analytics, a
frame inspector, and print CSS for PDF. `build_report_md` mirrors it as
Markdown tables. `write_report` builds in `fmt` (`"html"` or `"md"`), writes to
`path`, and returns the envelope. Pass `base` (a previous envelope dict, for
example loaded from `hotato run --format json` output) to render per-scenario
regression deltas; `base_label` names it in the page.

```python
from hotato.report import write_report

env = write_report("report.html", suite="barge-in", stack="livekit")
```

## hotato.aggregate

Team mode: many run envelopes, one trend view.

```python
load_run_dir(dirpath: str, order: str = "mtime") -> dict
    # {"runs": [{"file", "path", "mtime", "env"}], "skipped": [...], "order"}
    # order: "mtime" (oldest first) or "name" (numeric prefix = explicit index)

aggregate_runs(runs: list, order: str = "mtime",
               skipped: list | None = None) -> dict
    # team envelope: kind "team-aggregate", runs, events_total,
    # talk_over_sec / seconds_to_yield distribution summaries (mean/median/p90),
    # pass_rate {latest, first, mean, direction}, pass_rate_over_time,
    # failure_classes, most_common_failure_class, skipped, exit_code 0.
    # Raises ValueError with fewer than 2 runs, never pads a trend.

build_team_section_html(agg: dict) -> str   # embeddable section
build_team_page_html(agg: dict) -> str      # full self-contained page
```

```python
from hotato.aggregate import load_run_dir, aggregate_runs, build_team_page_html

loaded = load_run_dir("runs/")
agg = aggregate_runs(loaded["runs"], order=loaded["order"],
                     skipped=loaded["skipped"])
html = build_team_page_html(agg)
```

## hotato.export

Research-grade flat files from the same scorer.

```python
run_export(
    *,
    out_dir: str,
    # plus the full scoring parameter set: stereo, caller, agent,
    # caller_channel, agent_channel, onset_sec, expect, stack, suite,
    # scenarios_dir, audio_dir, suffix, max_talk_over_sec,
    # max_time_to_yield_sec, cfg
) -> dict   # {"env", "events_rows", "frames_rows", "paths"}
```

Writes `events.csv` (one row per event, columns in
`hotato.export.EVENT_COLUMNS`), `frames.csv` (one row per VAD frame, columns in
`FRAME_COLUMNS`), and `envelope.json` into `out_dir` (created if missing).
Column meanings are documented in `#` comment lines at the top of each CSV.
Empty cell means not derivable, never fabricated.

## hotato.stackbench

Identical scenarios, your stack, comparable result files. No vendor numbers, no
leaderboard: every number is a measurement of recordings you provide.

### run_stackbench

```python
run_stackbench(
    *,
    stack: str,                       # one of BENCH_STACKS:
                                      # vapi | twilio | livekit | pipecat | generic
    recordings_dir: str,              # <scenario-id><suffix> WAVs
    scenarios_dir: str | None = None, # default: bundled battery
    suffix: str | None = None,        # default: auto-detected
                                      # (.wav, .stereo.wav, .example.wav)
    caller_channel: int = 0,
    agent_channel: int = 1,
    cfg: ScoreConfig | None = None,
) -> dict
```

Returns a result dict (`kind: "stack-benchmark"`) with the envelope fields plus
`config`, `scenarios {total, captured, not_captured}`, and `provenance`.
Scoring is `run_suite` unchanged. Scenarios with no matching recording are
listed under `not_captured`, never scored and never counted as failures. The
timestamp derives from input file mtimes, so the same inputs reproduce the same
result file.

### load_result, compare_results, render_comparison_md

```python
load_result(path: str) -> dict
    # loads and validates one result JSON; anything else is a clean ValueError

compare_results(inputs: Sequence[tuple[str, dict]]) -> dict
    # inputs: (path, loaded_result) pairs, at least two.
    # Compares the intersection of scenarios scored in EVERY input;
    # the rest is listed under "skipped". Deltas are signed differences
    # against the FIRST input. Returns kind "stack-benchmark-comparison"
    # with inputs, compared, skipped, per_scenario, medians.

render_comparison_md(cmp_env: dict) -> str
    # the comparison as Markdown tables
```

```python
from hotato.stackbench import load_result, compare_results, render_comparison_md

cmp = compare_results([(p, load_result(p)) for p in ("a.json", "b.json")])
print(render_comparison_md(cmp))
```

## Pytest fixture

Installed automatically via the `pytest11` entry point (or load explicitly with
`-p hotato.pytest_plugin`). Inert unless used.

```python
def test_call_yields(hotato_score):
    env = hotato_score(stereo="call.wav", expect="yield")
    assert env["summary"]["regression"] is False
    assert env["events"][0]["verdict"]["seconds_to_yield"] < 1.0
```

`hotato_score(**kwargs)` takes the same keyword arguments as `run_single`; pass
`suite="barge-in"` (plus `run_suite` keywords) to score a battery instead. It
returns the envelope and never asserts for you.

Session gate flags: `pytest --hotato-suite` runs the battery after your tests
and fails the session (exit 1) on a regression; `--hotato-suite-scenarios DIR`
and `--hotato-suite-audio DIR` point it at your own labelled set. Detail:
`docs/PYTEST.md`.

## MCP tool

`hotato-mcp` (or `python -m hotato.mcp_server`) speaks MCP over stdio and
exposes exactly one tool, `voice_eval_run`, returning the identical envelope.
Install: `uvx --from "hotato[mcp]" hotato-mcp`.

Parameters, all optional:

| Parameter | Type | Default | Meaning |
| --- | --- | --- | --- |
| `stereo` | str | None | two-channel WAV path |
| `caller` | str | None | mono caller WAV (with `agent`) |
| `agent` | str | None | mono agent WAV (with `caller`) |
| `suite` | str | None | `"barge-in"` to run the bundled battery |
| `stack` | str | `"generic"` | livekit, pipecat, vapi, or generic |
| `expect` | str | `"yield"` | `"yield"` or `"hold"` |
| `onset_sec` | float | None | caller onset hint |
| `caller_channel` | int | 0 | caller channel index |
| `agent_channel` | int | 1 | agent channel index |
| `max_talk_over_sec` | float | None | pass threshold |
| `max_time_to_yield_sec` | float | None | pass threshold |
| `report_path` | str | None | also write the HTML report here; the envelope then carries `report_path` (absolute) |

## Exit codes and errors

Envelopes carry `exit_code`: 0 all scorable events passed, 1 regression.
`hotato.core.process_exit_code(env)` maps the finished envelope to the CLI
process exit: a single-recording run whose event is not scorable (silent
caller with no onset label, or agent silent at onset; the reason is in
`not_scorable_reason`) exits 2, because that is an input problem, not an
agent verdict. Suite runs count such events in `summary.not_scorable` and
keep their 0/1 semantics. Malformed input (bad WAV, out-of-range channel,
negative onset, unknown suite or stack) raises `ValueError`, which the CLI
surfaces as exit code 2. Nothing is ever scored from a file the scorer could
not fully read.


================================================================================
FILE: docs/SUBMITTING.md
================================================================================

# Submitting a recording

One consented, de-identified, human-labeled real call makes this eval more
credible. This is the whole path: record, label, validate, submit, and what
happens at intake.

The scorer measures speech energy over time. Your human label supplies the
meaning: which onset was a real bid for the floor, and whether a well-behaved
agent yields there. Energy is not intent, so the label is the ground truth,
and the label is what you are contributing.

## 1. Record dual-channel

Caller on one channel, agent on the other, separated at capture: the two legs
of a SIP bridge, or two streams that never mix. Real separation makes overlap
a fact of the recording, exact to the sample. Save a WAV at the call's native
sample rate (8000 Hz telephony, 16000 Hz wideband).

Consent, PII, and PHI rules are normative in
[`CORPUS-GOVERNANCE.md`](CORPUS-GOVERNANCE.md): documented consent from every
audible party, identifiers redacted with same-duration tone or silence so the
timing survives, no PHI ever. Read it before you record; it includes a
reusable release paragraph.

## 2. Label it

The label is a JSON file next to your WAV: the bundled scenario shape plus
provenance and attestation. Spec of record:
[`corpus/label.schema.json`](../corpus/label.schema.json). Worked example:
[`corpus/examples/sample-contribution.json`](../corpus/examples/sample-contribution.json).

Minimal example:

```json
{
  "id": "call-021-address-change-interrupt",
  "title": "Caller interrupts the shipping recap to change the address",
  "category": "should_yield",
  "source_type": "real-call",
  "audio": "call-021-address-change-interrupt.wav",
  "channels": { "caller_channel": 0, "agent_channel": 1 },
  "sample_rate": 8000,
  "duration_sec": 9.2,
  "caller_onset_sec": 3.15,
  "expected": {
    "yield": true,
    "max_time_to_yield_sec": 0.7,
    "max_talk_over_sec": 0.8
  },
  "license": "MIT",
  "attestation": {
    "contributor": "Your Name <you@example.com>",
    "consent_on_file": true,
    "pii_removed": true,
    "no_phi": true,
    "right_to_release_mit": true
  }
}
```

For a hold case (the caller backchannels "mm-hm" and a good agent keeps
talking): set `category` to `should_not_yield`, `expected.yield` to `false`,
and both `max_*` bounds to `null`.

Label only what you can defend by hand. `caller_onset_sec` is required;
`reference_render` segment timings are welcome wherever you have them, and
the harness reports error for exactly the signals you supply.

## 3. Validate locally

```bash
python3 corpus/validate.py your_label.json
```

The WAV resolves next to the label via its `audio` field. Pass the audio path
as a second argument if it lives elsewhere:

```bash
python3 corpus/validate.py your_label.json your_recording.wav
```

PASS (exit code 0) means the pair conforms: fields well-typed, category and
expected bounds consistent, timings in range, attestation affirmed, and the
audio a readable WAV with two or more channels that matches the label. A
human still reviews consent and PII before anything merges.

## 4. Submit

Two doors, one intake:

- **Issue form.** Open
  ["Corpus submission: labeled recording"](https://github.com/attenlabs/hotato/issues/new?template=corpus_submission.yml).
  Attach the WAV as a .zip (GitHub accepts .zip attachments; a raw .wav
  upload is rejected) or link it, and paste your label JSON in the
  description.
- **Pull request.** Add the label and WAV under [`corpus/`](../corpus/), for
  example `corpus/call-021-address-change-interrupt.json` plus its `.wav`.
  State the source type in the PR body and confirm the attestation, per
  [`CONTRIBUTING.md`](../CONTRIBUTING.md).

## What maintainers do at intake

1. **Validate.** Re-run `corpus/validate.py` on the pair and check the label
   against [`corpus/label.schema.json`](../corpus/label.schema.json).
2. **Dedupe.** Hash the audio and compare call details against existing
   clips. The corpus stays small and high-integrity; every clip earns its
   place.
3. **Normalize.** Slug the `id`, align filenames, confirm the channel map and
   declared sample rate. Timing is preserved exactly; audio is never edited
   in a way that shifts a label.
4. **Add to a suite.** Register the clip so the benchmark harness scores it
   and reports millisecond error distributions and the yield confusion matrix
   ([`BENCHMARK.md`](BENCHMARK.md)).
5. **Credit.** `attestation.contributor` is the credit of record, and
   contributors are named in the changelog when their clip lands.

Removal requests are honored in the next release, per
[`CORPUS-GOVERNANCE.md`](CORPUS-GOVERNANCE.md).


================================================================================
FILE: docs/CORPUS-GOVERNANCE.md
================================================================================

# Corpus governance

How this project assembles, labels, and reports on a small corpus of **real**
labeled voice-agent calls that augments the bundled synthetic fixtures.

This document is normative. To contribute real audio, follow it. To understand
what the numbers in a report mean, read the "Validity metrics" section at the end.

---

## Why real recordings matter

The synthetic fixtures in `src/hotato/data/` are a deliberately conservative floor. They
are rendered from known segment timings, so the ground truth is exact and the eval
is runnable by anyone, offline, in seconds. That makes them ideal for regression
tests and for demonstrating that the scorer behaves as specified.

Production validity is a different question. Synthetic audio has clean onsets,
controlled overlap, and no room acoustics or codec artifacts beyond what we inject.
A scorer that looks flawless on synthetic fixtures still needs proving on a real
8 kHz call with echo bleed, cross-talk, and a caller who trails off mid-word.

A small, carefully labeled corpus of real calls closes that gap. It is the
difference between "the tool does what the spec says" and "the tool measures what
actually happens on a phone line." We keep the corpus small and high-integrity on
purpose: every clip is consented, de-identified, and human-labeled. Ten trustworthy
calls beat ten thousand scraped ones.

---

## Scope and boundaries

- The corpus measures the **audio timing of turn-taking**: whether the agent
  stopped talking when the caller interrupted (yield), how long both talked at
  once (talk-over), and whether the agent talked through a short acknowledgement
  like "mhm" (backchannel handling).
- Labels are about **events in time**: did the agent stop, and when did the
  caller start.
- Everything here is redistributed under the project's MIT license. A clip enters
  the corpus only if it can be released under MIT with consent.

---

## Consent / release template

Every party audible on a real recording must agree, in a form you can produce on
request, to its inclusion. The following paragraph is intentionally short and
reusable. Send it to a caller and/or agent-operator and keep the signed or written
affirmative reply on file. Fill the brackets before use.

> **Recording release, hotato open test corpus**
>
> I, [name or role], took part in the audio recording made on [date] and described
> as [short description]. I understand this recording will be used as a test
> fixture in *hotato*, an open-source project, and grant a perpetual, worldwide,
> royalty-free right to store, redistribute, and publish the recording and its
> derived timing labels as part of that project's MIT-licensed corpus. I confirm
> that the recording contains no confidential, personal, health, or account
> information, or that any such information has been removed or replaced before
> submission. I understand I can request removal of the recording from future
> releases at any time by contacting the maintainers.
>
> Signed / affirmed: [name], [date], [contact]

Notes:

- Get consent from **all** identifiable parties: the human caller, and whoever
  operates the agent side when that is a real person or a proprietary system whose
  owner must agree.
- Consent must be **affirmative and documented**. A generic call-center "this call
  may be recorded" notice is not sufficient for redistribution under an open
  license.
- Honor removal requests promptly in the next release.

---

## PII policy

Real audio must be de-identified before it is committed. In order of preference:
**do not capture it, then remove it, then reject the clip.**

Strip or redact:

- **Names** of any real individual (caller, agent, third parties mentioned).
- **Phone numbers**, extensions, and any dialed digits (DTMF tones included).
- **Postal and email addresses.**
- **Account identifiers**: customer numbers, order numbers, card numbers, policy
  numbers, ticket IDs, anything that keys back to a real person or account.
- Any other free-text detail that, alone or combined, re-identifies someone.

Strongly preferred practice:

- Use **synthetic or role-played content**. Actors reading a script hit the same
  turn-taking dynamics (interruption, correction, backchannel) with zero PII risk.
  This is the cleanest source of real acoustics.
- **No PHI.** Do not submit any recording containing protected health information,
  regardless of consent. It is out of scope for this project.

Redaction guidance:

- Replace spoken PII with a short tone or silence of the **same duration** so the
  turn-taking timing is preserved. Cutting samples out shifts every onset after the
  edit and corrupts the labels.
- Redact the audio in **all channels**. A name muted on the mixed channel but
  audible on the caller channel is still present.
- Keep a private note of what was redacted and when (timestamps). Never commit the
  removed content.

---

## Data-handling policy

- **The tool runs local-only.** `hotato` reads local audio and writes local
  reports, offline, with no telemetry. Contributing to the corpus is a deliberate,
  manual act of committing a de-identified, consented clip.
- **Storage.** Corpus audio lives in the repository alongside its scenario JSON,
  under the same MIT terms as the rest of the project. Keep working copies local
  until they are cleared for release; do not stage real audio in third-party cloud
  buckets, shared drives, or chat tools during preparation.
- **Raw originals stay out of the repo.** Only the final, redacted, consented clip
  is committed. The un-redacted original is the contributor's responsibility to
  secure or destroy per their own obligations.
- **Contributor attestation.** Each real-audio PR must include a short statement
  from the contributor affirming: (a) consent is on file for every party, (b) PII
  has been removed per the policy above, (c) the clip contains no PHI, and (d) the
  contributor has the right to release it under MIT. This attestation is part of
  the merge record.

---

## Validity metrics: what we publish

The corpus exists to quantify measurement quality. That shapes how results
are reported.

We publish:

- **Per-signal measurement error, in milliseconds.** For each labeled call we
  compare the scorer's measured event times to the human labels and report the
  error distribution per signal:
  - `time_to_yield`: signed and absolute error in ms between measured and labeled
    floor-yield.
  - `talk_over`: error in ms on measured overlap duration.
  - onset agreement: error in ms between measured caller onset and
    `caller_onset_sec`.
  We report these as distributions (median, spread, worst case), so a reader sees
  the real shape of the error.
- **A `did_yield` confusion matrix.** Against the human `category` (`should_yield`
  / `should_not_yield`) label, we report the four-cell matrix: correct yields,
  correct holds, false yields (yielded on a backchannel), and missed yields (kept
  talking over a real interruption). The two error cells are the ones operators
  feel, so we surface them directly.

The distribution and the four-cell matrix are the report. A single percentage would
collapse the two failure modes into one number and hide the trade-off between them,
so we keep them separate. The scorer works on speech energy over time, so validity
is timing agreement and decision agreement.

### Reading a corpus report

A healthy result looks like: tight millisecond error distributions on
`time_to_yield` and `talk_over`, onset agreement within a small ms band, and a
confusion matrix whose off-diagonal (false-yield / missed-yield) cells are small
and individually inspectable. A poor result looks like wide error spread or a heavy
off-diagonal, and it should point you toward a concrete fix in the call.

Where a real call fails, the fix map names candidate remedies. A learned
engagement-control layer is one of them, referenced at that level. Audio-only is
its weaker modality and a fully local offline model remains a known gap. The corpus
is here to measure reality.


================================================================================
FILE: docs/BENCHMARK-STACKS.md
================================================================================

# Benchmarking voice stacks with hotato

`hotato benchmark` scores the recordings you captured by running one fixed
scenario set through YOUR configured voice stack. Every stack is scored on the
same scenarios, the same labels, and the same thresholds, so the result files
are directly comparable: same battery, same measurements, same exposed config.

hotato measures timing on the recordings it is given. It ships no vendor
numbers, no leaderboard, and no accuracy percentage. Results depend on your
stack configuration and your captures; a benchmark result describes YOUR
setup, on the day you captured it.

## 1. Pick the scenario set

The bundled 8-scenario barge-in battery is the default. Any scenarios dir
works the same way, for example the larger tiered suites:

```
corpus/suites/gold/scenarios
corpus/suites/silver/scenarios
```

Each scenario JSON carries the caller transcript, the caller onset timing,
and the pass thresholds. The corpus suites also ship each scenario's caller
stimulus as `audio/<id>.caller.wav`.

## 2. Capture the battery through your stack

For every scenario, drive the caller side through your stack and record the
call dual-channel (caller on channel 0, agent on channel 1):

- Corpus suites: play the shipped `<id>.caller.wav` stimulus into your stack.
  For the bundled battery, reproduce the caller side from each scenario's
  transcript and `caller_onset_sec`.
- `hotato setup --stack livekit` or `--stack pipecat` prints the exact
  dual-channel recording scaffold for your infra.
- `hotato capture --stack vapi --call-id <id>` or
  `hotato capture --stack twilio --recording-sid RE...` pulls the finished
  dual-channel recording for you.

Name each recording by its scenario id, one directory per stack:

```
captures/livekit/01-hard-interruption.wav
captures/livekit/02-backchannel-mhm.wav
...
captures/vapi/01-hard-interruption.wav
```

## 3. Run the benchmark

```bash
hotato benchmark --stack livekit --recordings captures/livekit --out livekit.json
hotato benchmark --stack vapi    --recordings captures/vapi    --out vapi.json
```

Each result JSON records the stack, the scenario set, every measured signal
(the same events `hotato run` produces), the exact scoring thresholds, and a
provenance block: who ran it, on which files, with each file's mtime. The
result timestamp comes from the input files, not the wall clock, so the same
inputs reproduce the same result.

Scenarios without a matching recording are listed under `not_captured`. They
are never scored and never counted as failures. The exit code is 0 when the
run completes; add `--fail-on-regression` to exit 1 when a scored event fails
its scenario thresholds.

## 4. Compare stacks

```bash
hotato benchmark compare livekit.json vapi.json
```

Side by side, per scenario: yielded, talk-over seconds, and time-to-yield
seconds for each input, with signed deltas against the first file, plus
summary medians. If the files cover different scenario sets, the intersection
is compared and everything else is listed as skipped, with the files it was
missing from.

## What the numbers are

Reproducible timing measurements of the recordings you provided, scored by
the same engine as `hotato run`, with every threshold exposed in the result.
They tell you how your configuration of a stack handled the battery you
captured. They are not a vendor ranking, and no accuracy percentage appears
anywhere.


================================================================================
FILE: docs/BENCHMARK.md
================================================================================

# Benchmark methodology

`hotato` ships a reproducible measurement-error harness: `src/hotato/benchmark.py`.
Given a set of labelled dual-channel recordings, it reports how far the scorer's
measured event times land from the rendered or labelled ground truth, and how its
yield/hold decisions line up with the human labels. That is the whole output.

```bash
PYTHONPATH=src python3 -m hotato.benchmark
```

This runs on every synthetic fixture in the checkout (the bundled battery, the
`examples/` reference set, and the deliberately-bad `funnel-demo` set), prints a
markdown table, and writes `benchmark-report/measurement-error.json` and
`benchmark-report/measurement-error.md`.

The harness is a standalone measurement tool you run against fixtures. The `hotato`
CLI scores a single call or the battery.

---

## What it measures

For every `(recording, label)` pair it reports two things.

### 1. Per-signal measurement error, in milliseconds

For each timing signal it computes `|measured - rendered|` in ms, where `rendered`
is the exact value the synthetic fixture was rendered from (its `reference_render`
block) or the value a contributor labelled by hand:

| signal | measured | rendered/true reference |
|---|---|---|
| **caller onset** | onset the VAD detects (the scorer is given no onset label) | start of the first caller segment |
| **time to yield** | seconds from caller onset to the agent going quiet | the agent's in-progress turn end minus the caller onset |
| **response gap** | endpointing dead-air from the caller's turn end to the agent's next onset | the fixture's rendered response gap |

Errors are reported as a distribution: median, mean, worst case, best case, and n.
A signal is scored where a rendered or true reference exists and the scorer
produced a value; a missing reference yields a `-`. (The echo-of-agent fixture has
no independent caller speech, so it has no onset or yield error: a gap.)

Onset is measured in **detect mode** (the scorer gets no onset hint), so it is a
real test of the onset detector. Yield, talk-over, response gap, and `did_yield`
are measured in **label mode** (the scorer is given the human `caller_onset_sec`,
exactly as the shipped battery runs), so those are the numbers a user actually
sees.

### 2. A `did_yield` confusion matrix

Against the `should_yield` / `should_not_yield` label (each scenario's
`expected.yield`), it reports the four cells:

|  | measured **did_yield** | measured **held floor** |
|---|---|---|
| **should_yield** | correct yield | **missed yield** |
| **should_not_yield** | **false yield** | correct hold |

The two off-diagonal cells (missed yields, and false yields) are the failures an
operator feels, so the report surfaces them directly.

---

## Why milliseconds and a matrix

The report is a per-signal error distribution plus a four-cell confusion matrix.
The cells stay separate because a missed yield and a false yield are different
failures with different fixes; averaging them into one number would hide which
one you have. The distribution and the matrix are the report. This is the rule the corpus governance doc enforces
(`docs/CORPUS-GOVERNANCE.md`, "Validity metrics"), applied to the tooling.

The reported error is what the default shipped config measures, so it is the number
a real user gets. On the synthetic fixtures the yield error equals the exposed VAD
hangover and the onset/gap error is one frame hop, both documented `ScoreConfig`
parameters. Set the hangover to zero
(`caller_vad.hangover_sec = agent_vad.hangover_sec = 0`) and every signal collapses
to within one hop of the rendered ground truth; the test suite asserts exactly
this, so the claim is checkable.

The scorer works on speech energy over time, so the harness reports timing and
decisions.

---

## Reproducing it

```bash
# render the synthetic fixtures deterministically (sha256-seeded; byte-identical
# on any machine), then run the harness over them
python3 examples/render_examples.py
PYTHONPATH=src python3 -m hotato.benchmark
```

The report has no wall-clock timestamp and the render is deterministic, so two runs
on the same code produce byte-identical artifacts. The JSON carries a `config`
snapshot of every threshold that produced the numbers, so a reader can re-derive
any value. `tests/test_benchmark.py` pins the whole thing: it asserts the harness
runs, the confusion matrix matches the known rendered behaviour, and every ms-error
is within its known, config-derived tolerance.

The synthetic fixtures are a floor: deterministic rendered audio with exact known
timings. They prove the scorer behaves as specified and guard against regressions.
See `examples/README.md` and `docs/CORPUS-GOVERNANCE.md`.

---

## Extending to real recordings (bring your own labelled data)

The synthetic floor tells you the scorer does what the spec says. Real recordings
tell you it measures what happens on an actual phone line. The harness runs on your
own labelled data with no code change:

```bash
PYTHONPATH=src python3 -m hotato.benchmark \
  --scenarios path/to/your/scenarios \
  --audio     path/to/your/audio \
  --name      my-corpus \
  --out       my-benchmark-report
```

- `--scenarios` is a directory of scenario JSON labels, same shape as
  `src/hotato/data/scenarios/*.json`. Each needs an `id`, an `expected.yield`
  label, and whatever reference timings you can defend: a `reference_render` with
  segment timings (as the synthetic fixtures carry), or at minimum a hand-labelled
  `caller_onset_sec` and the true event times you want error scored against. Supply
  a reference only where you have ground truth; the harness reports error for those
  signals and leaves the rest blank.
- `--audio` is a directory of dual-channel `<id>.example.wav` recordings (caller on
  channel 0, agent on channel 1). Record two physically separated channels whenever
  you can: that is what makes overlap ground-truthable.

When `--scenarios` is given, only that set is scored, so you get a clean report for
your corpus alone.

The path to a real-model number is manual and consented: bring your own
labelled audio. Contributing real audio is governed by `docs/CORPUS-GOVERNANCE.md`
(consent, PII, data-handling) and the pipeline in `corpus/` (a labelling schema and
a validator). Synthetic fixtures keep their synthetic label.


================================================================================
FILE: docs/CARDS.md
================================================================================

# Cards: a shareable image from any hotato result

`hotato card` turns a machine result into a self-contained SVG you can drop into
a pull request, an issue, or a slide. One command, offline, and honest by
construction: the card names the measured timing moment and never a verdict about
intent, and it carries no accuracy number anywhere.

```bash
hotato card INPUT[#REF] --out card.svg
```

Everything runs locally. The SVG is a pure function of the input JSON (no
timestamp, no version, no randomness), so the same input renders the same bytes
forever, and it references no font, image, stylesheet, script, or link: every
color is inline. Drop it anywhere with no CDN and no asset host.

## The four cards, auto-detected

The input's kind decides the card; you do not pick.

| Input | Card |
|---|---|
| a `sweep`/`analyze` candidate ref, `FILE#N`, that is a talk-over moment | **talk-over candidate** |
| a `sweep`/`analyze` candidate ref, `FILE#N`, that is a false-stop moment | **false-stop candidate** |
| a fix plan whose `decision` is `do_not_tune_single_threshold` | **threshold funnel** (the hero) |
| a supported `hotato verify` rollup | **verify** |

`#N` is the same 1-based rank the sweep report and dashboard show, and it is the
same ref `hotato fixture promote` takes, so a card and a fixture speak of the
exact same moment.

### A. talk-over candidate

An `overlap_while_agent_talking` (or `agent_start_during_caller`) moment: the
agent kept the floor while the caller was speaking. The card leads with the
measured overlap in seconds and closes with "Hotato reports timing candidates,
not intent."

```bash
hotato sweep --demo --format json > hotato-sweep.json
hotato card hotato-sweep.json#3 --out talk-over.svg
```

### B. false-stop candidate

An `agent_stop_no_caller` moment: the agent went quiet with no caller nearby to
explain the drop. The card leads with the measured trailing silence.

```bash
hotato card hotato-sweep.json#1 --out false-stop.svg
```

### C. threshold funnel (the hero)

The plan the both-axes case produces: the battery missed a real interruption
**and** false-stopped on a backchannel, so no single sensitivity dial can satisfy
both axes at once. The card states that Hotato refused threshold tuning and names
the fix class (`engagement-control`). This is the card the project leads with.

```bash
hotato demo --format json > demo.json
hotato plan demo.json --out fix-plan.json
hotato card fix-plan.json --out no-single-threshold.svg
```

Only a `do_not_tune_single_threshold` plan renders this card; any other plan is
a clean exit-2 usage error (it is not one of the four kinds).

### D. verify

A supported `hotato verify` rollup: the previously-failing fixtures now pass and
no hold/backchannel fixture regressed. The card reads "FIX VERIFIED WITHOUT
BREAKING BACKCHANNELS" and closes with "Hotato reports coincidence, not
causation." A verify result that does not support that claim (too few
previously-failing fixtures, nothing now passing, or a regressed hold fixture)
is refused with exit 2 rather than stamped verified.

```bash
hotato card verify.json --out verified.svg
```

## Redaction: safe to share by default

A card is a public image, so identifiers are hidden by default. A call id, a
filesystem path (only a basename is ever a candidate for display), and a vendor
recording name are omitted. A pulled recording named `STACK__ID.wav` carries the
call id inside its name; that name is only ever shown under
`--include-identifiers`.

```bash
# shows the source recording's basename on a candidate card
hotato card hotato-sweep.json#1 --out card.svg --include-identifiers
```

## Output and exit codes

Without `--out`, the SVG is written to stdout, so you can pipe it. With `--out`
it is written there atomically.

- **0**: the SVG card was rendered (to `--out`, or to stdout).
- **2**: usage error, unreadable input, a bad candidate ref, or an input that is
  not a fix plan / verify result / sweep candidate.

## Regenerating the committed cards

Three commit-ready examples live under `docs/assets/cards/`
(`no-single-threshold-card.svg`, `talk-over-card.svg`, `false-stop-card.svg`),
rendered from the two bundled demo calls. Regenerate them with:

```bash
PYTHONPATH=src python3 scripts/render_card_assets.py
```


================================================================================
FILE: docs/COMPARE.md
================================================================================

# Compare: Hotato vs broad QA platforms

Hotato is not your full QA platform.

**Use Hamming, Cekura, Coval, Bluejay, Roark, Vapi, or Retell for:** broad
session QA, synthetic simulation, task success, transcript rubrics, compliance
workflows, production dashboards, load testing.

**Use Hotato for:** portable failure contracts from real calls, local/private
timing evidence, CI-enforced regression tests, trace-backed turn-taking proof,
refusing unsafe threshold bandaids.

**Hotato answers:** "Did this exact production timing failure come back?"
**Hotato does not answer:** "Was the whole call successful?"

These are complementary layers, not competing ones. A team running one of the
platforms above for broad conversation QA or as the agent runtime itself still
needs an answer to a narrower question: the specific talk-over or false-stop
moment a real caller hit last week, is it still fixed after today's prompt
change? That is the layer Hotato owns, and it is deliberately the only layer
Hotato owns.

## What each named platform is built for

Capability descriptions only, drawn from each platform's own public launch and
product material, not from running Hotato against them. No performance claims,
no ranking.

| Platform | What it is built for |
|---|---|
| **Hamming** | Automated voice-agent testing: prompt-to-test generation, simulated call batteries, a multi-layer QA framework, and production-replay CI/CD regression across a broad test suite. |
| **Cekura** | AI voice/chat agent QA: production-conversation ingestion, test-case extraction from real calls, simulated scenario testing, and monitoring across fleets of agents, including regulated verticals. |
| **Coval** | Voice and chat agent simulation and evaluation, with published comparisons of testing approaches across the category. |
| **Bluejay** | Synthetic stress-testing for voice agents at scale. |
| **Roark** | Production-call replay for QA that preserves what the caller said, how, and when, for building regression scenarios. |
| **Vapi** | A voice agent orchestration platform: the runtime stack a team builds and deploys its agent on. |
| **Retell** | A voice agent orchestration platform for building and deploying voice agents. |

## The honest job, and the better fit

| What you need | Best fit | Why |
|---|---|---|
| Broad conversation QA: did the call succeed, was the transcript right, did it follow the rubric, simulate hundreds of scenarios, dashboards for the team | **Hamming, Cekura, Coval, Bluejay, Roark, Vapi, or Retell** | These grade the whole conversation, task success, and content, or they are the agent platform itself. Hotato does not do content, task success, or simulation. |
| Prevent an interruption problem **in the moment**, at runtime: predict endpointing, suppress barge-in on noise, tune the live turn detector | **Pipecat, LiveKit** (and similar runtime layers) | These act during the live call. Hotato never runs at runtime and never touches a live call. |
| Prove a specific timing bug is fixed and **stays** fixed, from a real recorded call, portably, without sending audio anywhere | **Hotato** | A private, deterministic fixture: audio, a human label, and an explicit policy, scored the same way everywhere with `hotato verify`. Ships as a single portable contract bundle -- audio, timing evidence, trace evidence, label, policy, CI command -- in the release that adds the contract layer. |

If your open question is "is my agent good?", start with a QA platform. If it
is "is my agent interrupting well right now?", start with a runtime layer. If
it is "did the talk-over bug we fixed last month come back in this release?",
that is Hotato.

## Runtime layer vs regression layer

These two are often confused, so it is worth being exact.

- A **runtime layer** (Pipecat, LiveKit turn detection) makes a decision
  *during* the call: should the agent stop talking now? It optimizes the live
  experience. It does not keep a durable, re-runnable record of whether a
  specific past moment is handled correctly.
- A **regression layer** (Hotato) makes a decision *after* the call, from the
  recording: given this exact audio and this label, did the timing match, yes
  or no, byte-stable on every run? It exists to catch the day a fixed bug
  silently returns, and to hand that proof to CI as a portable artifact.

You want both. A runtime layer improves the median call. A regression layer
stops a good fix from quietly rotting three releases later. Hotato takes the
moment a runtime layer got wrong, freezes it as a fixture with `hotato
fixture promote`, and fails CI if it regresses.

## What Hotato is for, precisely

- **Private.** Scoring, scanning, reports, fixtures, and verification run
  offline. Audio stays on your machine unless you explicitly pull it from your
  own stack. Nothing is uploaded to Attention Labs. See
  [THREAT-MODEL.md](THREAT-MODEL.md).
- **Deterministic.** No learned score, no sampling. The same recording produces
  the same timing numbers every run, so a red build means the audio changed.
- **Portable.** A confirmed failure becomes a labelled fixture (audio, human
  label, explicit policy) that travels with the repository and verifies the
  same way on any machine with `hotato verify`. It ships as a self-contained
  contract bundle -- adding timing evidence, trace evidence, and a CI command
  in one artifact -- in the release that adds the contract layer.
- **Narrow on purpose.** Three timing signals: talking over the caller,
  false-stopping on a backchannel, yielding too slowly. It does not grade
  content, intent, or outcomes.

## On provider-default examples

When a Hotato example shows a call failing on a named stack, it is labelled a
**provider-default** run: one assistant, one configuration, one date, one
scripted caller, on that provider's out-of-the-box interruption settings. It
demonstrates the threshold funnel (how a default config can miss an
interruption and false-stop on a backchannel in the same battery). It is
**not** a vendor benchmark and **not** a ranking of one platform against
another. Any stack, tuned, can pass the same fixtures. Hotato never publishes a
scoreboard, of its own results or anyone else's.

## The short version

Use a QA platform for broad quality, or one of the named orchestration
platforms as your agent's runtime. Use a runtime layer for live interruption
handling. Use Hotato when you need to prove, privately and portably, that a
specific timing bug from a real call is fixed and does not come back. They
compose; none of them replaces the others.


================================================================================
FILE: docs/CONTRACTS.md
================================================================================

# Failure contracts

A failure contract turns ONE real call moment into a portable, private,
vendor-neutral bundle: the audio, frame-level timing evidence, an
input-health report, a shareable card, a CI pass/fail policy, and the exact
commands to replay and re-verify it. It is the CI object: `hotato contract
verify` re-scores a directory of contracts and exits non-zero when one
regresses.

Hotato does not infer intent. A contract's label is always a human call
(`label_source` is frozen to `"human"`); Hotato measures whether the recorded
timing matched that label, and `contract verify` re-measures the SAME
recording later and reports pass/fail.

Hotato does not prove authorization, identity, compliance, or policy safety.
Hotato proves timing behavior against this explicit contract.

## The bundle

`hotato contract create` writes one self-contained directory,
`<id>.hotato/`:

```
refund-cutoff-001.hotato/
  contract.json                      # the contract itself (schema hotato.contract.v1)
  audio/event.wav                    # the (clipped) two-channel recording, or the mono file
  evidence/
    frames.jsonl                     # per-frame timing evidence behind every measurement
    timeline.html                    # the to-scale caller/agent timeline, self-contained
    trust.json                       # the input-health (trust doctor) report
    card.svg                         # a shareable 1200x630 SVG card (redacted by default)
  traces/                            # empty until `hotato trace attach` (see docs/TRACE.md)
  source/
    call_metadata.json               # redacted-by-default: stack, category, expect
    stack_config_snapshot.json       # placeholder until populated by hand
  policy/verify.yaml                 # the SAME subset `hotato verify --policy` reads
  reports/
    initial.html                     # the full scored report at creation time
    after.html                       # placeholder until a fix is re-captured and verified
  provenance.json                    # who/when/how this contract was created
  ci/
    github-action.yml                # a weekly + on-push CI scaffold
    junit.xml                        # this ONE contract's JUnit result at creation time
```

Every path above is also recorded, machine-readable, in
`contract.json["bundle"]["paths"]`.

## Create

From a candidate a sweep or scan already surfaced:

```bash
hotato sweep --demo --format json > hotato-sweep.json
hotato contract create --from-candidate hotato-sweep.json#1 \
    --expect yield --id refund-cutoff-001 --out contracts
```

From a raw two-channel recording you already have:

```bash
hotato contract create --stereo bad-call.wav --onset 42.18 \
    --expect yield --id refund-cutoff-001 --out contracts \
    --max-talk-over 0.6 --max-time-to-yield 1.0
```

Both forms wrap the SAME round-trip guarantee `hotato fixture create` gives:
the moment is scored immediately, and a not-scorable input (the agent silent
at the onset, an unreadable file, a bad channel map) is refused with the
honest reason and exit code 2 -- no bundle is written. A single-channel
(mono) recording passed as `--stereo` is rejected the same way `fixture
create` rejects it: caller and agent cannot be told apart on one channel.

`--caller FILE --agent FILE` (two mono WAVs) is a third input form, scored
and clipped identically.

### Redaction

By default the bundle and the card hide a candidate ref and a source
recording's basename. Pass `--include-identifiers` to show them (in
`source/call_metadata.json`, `contract.json`, and `evidence/card.svg`).

### The opt-in diarized-mono path

A single-channel recording can still become a contract through the SAME
quality-gated diarizer front-end `hotato run --mono --diarize` uses:

```bash
hotato contract create --mono call.wav --diarize \
    --expect yield --id refund-cutoff-002 --out contracts
```

This NEVER silently upgrades an indicative-only verdict: a `low`-confidence
separation tier carries `measurement.indicative_only: true` all the way into
`contract.json` and every renderer, and a `refuse` tier (not two clean
parties, extreme overlap, unstable segmentation, voices too similar) is
refused exactly like a plain mono file, with the specific reason. Frame-level
evidence (`evidence/frames.jsonl`, the to-scale timeline) is not produced for
this path in this release; `evidence/timeline.html` says so plainly instead
of fabricating one, and points at `evidence/trust.json`'s separation
confidence tier.

## Verify

```bash
hotato contract verify contracts/
hotato contract verify contracts/ --format json --junit contracts-junit.xml
hotato contract verify contracts/refund-cutoff-001.hotato --html verify.html
```

`DIR` is a contracts directory (every `*.hotato` subdirectory that carries a
`contract.json`) or one bundle directly. For each contract, `verify`
re-scores the SAME bundled audio against the SAME policy recorded in its own
`contract.json` -- this is what changes after an engine upgrade, a threshold
change, or a re-captured recording swapped into `audio/event.wav` -- and
reports pass/fail per contract and overall.

Exit codes are the CI contract: `0` every contract passes, `1` at least one
regressed (or is no longer scorable), `2` a usage error, an empty directory,
or a corrupt `contract.json`. `--junit` writes one `<testcase>` per contract
for a CI dashboard; the shipped `ci/github-action.yml` scaffold runs this on
push, on PR, and weekly, and publishes the JUnit file as an artifact.

## Inspect

```bash
hotato contract inspect contracts/refund-cutoff-001.hotato
hotato contract inspect contracts/refund-cutoff-001.hotato/contract.json --format json
```

## Pack / unpack

A bundle directory travels as one file:

```bash
hotato contract pack contracts/refund-cutoff-001.hotato
# -> contracts/refund-cutoff-001.hotato.pack (deterministic; a MANIFEST.sha256.json
#    of every member travels inside it)

hotato contract unpack contracts/refund-cutoff-001.hotato.pack --out contracts/refund-cutoff-001.hotato
# every member is verified against the packed sha256 manifest; any mismatch
# (a corrupt or tampered archive) is refused (exit 2) and nothing partial is
# left behind
```

Packing the SAME bundle directory twice produces byte-identical archives
(sorted member order, fixed timestamps, and every other value written into
each member's ZipInfo, including its `create_system` byte): a contract's
pack is a pure function of its bundle contents, on any machine.

### Security: unpack treats an archive as hostile input

A `.hotato` archive is meant to be sent between teams, so `contract unpack`
never trusts it. Before or during extraction it refuses (exit 2), with
nothing written outside a scratch temp directory that is removed on any
failure:

* path traversal (`..`), absolute paths, and Windows-style backslash /
  drive-letter paths (`C:\...`) in a member name;
* symlink members and encrypted members;
* duplicate member names;
* any member the archive carries that its own `MANIFEST.sha256.json` does
  not declare;
* more members than a real bundle could plausibly need;
* a declared or ACTUAL decompressed size past the cap (default 512 MiB, set
  `HOTATO_CONTRACT_MAX_UNPACK_BYTES` or pass `--max-bytes` to raise it for a
  trusted archive) -- checked against the real bytes streamed out during
  extraction, not just the archive's own (untrusted) size metadata;
* a single member whose compression ratio is far beyond anything a real
  bundle member produces (a zip-bomb signal), even if it is well under the
  total-bytes cap.

The existing sha256-per-member check (above) still runs on top of all of
this: a member can pass every hardening check above and still be refused for
not matching the packed manifest.

## CI

The shipped `ci/github-action.yml` is the minimal wiring:

```bash
uvx hotato contract verify contracts/ --junit contracts-junit.xml --format json > contracts-verify.json
```

Gate a pull request or a scheduled job on the process exit code; do not parse
stdout to decide pass or fail.

## What a contract does not prove

Hotato does not prove authorization, identity, compliance, or policy safety.
Hotato proves timing behavior against this explicit contract. `verify`
reports coincidence, not causation: a passing re-verify after a config change
coincides with the change; it is not a controlled experiment. See
`docs/VALIDATION.md` and `docs/THREAT-MODEL.md`.

## Read more

- The underlying regression-fixture primitive `contract create` wraps:
  [`docs/BAD-CALL-TO-CI.md`](BAD-CALL-TO-CI.md)
- Battery-scale before/after proof: [`docs/FIX-LOOP.md`](FIX-LOOP.md)
- Input-health (trust doctor): [`docs/TRUST.md`](TRUST.md) ·
  [`docs/TRUST-MATRIX.md`](TRUST-MATRIX.md)
- Diarized-mono scoring: [`docs/DIARIZE.md`](DIARIZE.md)
- Shareable cards: [`docs/CARDS.md`](CARDS.md)
- Attaching a voice trace (observability bridge): [`docs/TRACE.md`](TRACE.md) ·
  [`docs/OTEL.md`](OTEL.md)


================================================================================
FILE: docs/DIARIZE.md
================================================================================

# `hotato run --mono call.wav --diarize`: score a single-channel recording

Hotato's gold reference is a **two-channel** recording -- the caller on one
channel, the agent on the other, each channel one party, no separation needed.
That path, and every published/golden number, is unchanged. A **mono** (single,
mixed channel) recording is the coverage wall: by default it is rejected as not
scorable, because with both voices summed into one waveform the scorer cannot
attribute energy to a speaker.

The opt-in `[diarize]` front-end widens that coverage. It runs an off-the-shelf
**speaker diarizer** over the mono to recover *who was active when*, reconstructs
two caller/agent tracks, and feeds the **existing** scorer -- so a mono call
becomes scorable. It is **quality-gated** and honestly labeled: above the
confidence bar the verdict is a real `diarized-mono` verdict; below it, the
verdict is labeled indicative only and no SLA gate fires; a non-separable file is
refused. A diarized-mono verdict is **never** equivalent to a true dual-channel
measurement for sub-second talk-over, and the gate is what keeps that honest per
file.

Diarization, not source separation: Hotato scores *timing* (who was active when,
and overlap), which a diarizer's turn timestamps reconstruct directly. It does
**not** reconstruct isolated waveforms, and does **not** do speaker
IDENTIFICATION -- a diarizer assigns anonymous `SPEAKER_00` / `SPEAKER_01`; it
never says who a person is.

## Quickstart

```bash
# Install the default (local, offline) diarizer extra, plus a Hugging Face token
# with the gated model conditions accepted (one-time download, then offline):
pip install 'hotato[diarize]'
export HUGGINGFACE_TOKEN=hf_...            # accept the model card conditions first

# Check first whether the mono is confidently separable (no scoring):
hotato trust --stereo call.wav --diarize            # -> high / low / refuse tier

# Score the mono:
hotato run --mono call.wav --diarize --format json  # diarized-mono verdict
```

With the extra (or the token/model) absent, `--diarize` errors cleanly and exits
`2`; it **never** falls back to scoring raw mono.

## Backends and extras

A pluggable backend seam (mirroring the neural-VAD seam). Pick with
`--diarizer`; install only the extra you select. The default backend is chosen by
the downstream benchmark, not pre-assumed -- `pyannote` is the accessible local
default but is **not** best on telephone, so a user who needs best-in-class picks
`sortformer` (local) or `pyannoteai` (hosted).

| `--diarizer` | extra | where it runs | notes |
|---|---|---|---|
| `pyannote` (default) | `[diarize]` | local, CPU-viable, offline | richest confidence signals (posterior + embedding margin); gated HF weights |
| `sortformer` | `[diarize-sortformer]` | local, GPU-leaning | best self-hostable on 2-speaker telephone; EEND (no embedding margin) |
| `pyannoteai` | `[diarize-hosted]` | HOSTED (audio leaves the machine) | best absolute accuracy; requires `--egress-opt-in` |

```toml
diarize            = ["pyannote.audio>=4.0", "torch>=2.8", "torchaudio>=2.8", "numpy>=1.21"]  # default; needs system ffmpeg
diarize-sortformer = ["nemo-toolkit[asr]>=2.7", "torch>=2.8", "numpy>=1.21"]                   # best self-hostable, GPU
diarize-hosted     = ["pyannoteai-sdk>=0.3"]                                                   # hosted, egress opt-in
```

The `[diarize]` path raises the effective Python floor to **>=3.10** (pyannote
4.x); the stdlib core stays >=3.9 -- this only constrains the optional path.

### Model licenses (log per FTO note)

Using off-the-shelf diarizers is integration, orthogonal to any Hotato IP claim,
but the dependency licenses are logged here and carried in the score envelope's
`diarization.licenses` block:

- `pyannote-audio` -- **MIT** (code)
- `speaker-diarization-community-1` weights -- **CC-BY-4.0** (attribution required)
- `segmentation-3.0` weights -- **MIT**; `wespeaker` embedding -- **CC-BY-4.0**
- `torch` -- **BSD-3-Clause**; `torchaudio` -- **BSD-2-Clause**
- `nemo-toolkit` -- **Apache-2.0**
- Sortformer **streaming v2** -- **CC-BY-4.0** (the offline v1 is **CC-BY-NC** ->
  non-commercial, and is never shipped)
- pyannoteAI hosted -- proprietary terms (verify before enabling egress)

If your licensing posture wants a permissive weights license, prefer
`speaker-diarization-3.1` (MIT weights) over community-1 (CC-BY-4.0) via
`HOTATO_DIARIZE_MODEL`.

## The confidence gate (the honesty core)

Aggregate diarization error is a corpus statistic; the gate is **per file, at
runtime, with no ground truth**. Six signals feed a `separation_confidence` in
`[0, 1]` and one of three tiers:

| signal | flags low quality when |
|---|---|
| speaker count == 2 | != 2: not two clean parties (1 = couldn't separate; 3+ = extra voices / mis-cluster) -> refuse |
| both speakers >= 0.30s activity | a near-silent "speaker" is a spurious split of one party -> refuse |
| mean segmentation posterior | near-chance: the model is unsure who is speaking (uncalibrated; a relative signal) |
| embedding cluster margin (pyannote only) | small: two voices too similar to attribute confidently |
| overlap ratio in a sane band | extreme: heavy crosstalk / collapsed turns -> talk-over unreliable |
| segment churn (short turns/sec) | high: a jittery, unstable timeline -> noisy timing |

**Tiers:**

- **high** -- score normally; the verdict is real but always tagged
  `source: "diarized-mono"` and `confidence_tier: "high"` (never presented as
  dual-channel).
- **low** -- score, but the envelope carries `indicative_only: true`: the verdict
  is "indicative only, reconstructed from single-channel diarization." **No
  pass/fail SLA gate** (`--max-talk-over` / `--max-time-to-yield`) fires on a low
  tier.
- **refuse** -- `scorable: false`, a reason naming the failed signal, exit `2`
  (exactly like today's mono rejection).

The thresholds are provisional and **uncalibrated** -- they are pinned by the
downstream verdict-agreement benchmark, not asserted as accuracy -- and are
exposed as constants in `hotato/diarize.py`.

## Caller vs agent assignment (never a silent guess)

A diarizer returns anonymous `SPEAKER_00` / `SPEAKER_01`; Hotato needs
caller/agent. The mapping is proposed, stated as an assumption, and overridable:

- **default proposal** -- reuse the floor-dominance heuristic (`trust`'s
  possible-swap band): an agent usually holds the floor longer, so the
  higher-talk-time speaker is proposed as the agent. Ambiguous (balanced floor
  time) mappings are broken by who-speaks-first and flagged `balanced: true`,
  which downgrades the verdict to indicative rather than a coin-flip.
- **override** -- `--caller-speaker SPEAKER_00 --agent-speaker SPEAKER_01`; when
  both are given no heuristic runs.

The chosen mapping and its basis are emitted in
`diarization.speaker_map: {caller, agent, basis, balanced, confidence}`.

## Echo / crosstalk is N/A on this path

The two reconstructed tracks are slices of **one physical microphone**, so
`signals.echo` / crosstalk coherence carries no echo information (it is trivially
high in overlap). On the diarized-mono path the echo block is marked
`applicable: false` and the `--echo-gate` can never fire -- it is meaningful only
for two physically separate channels.

## Honest limits (what stays indicative or refused)

- **very similar voices** (same gender/pitch, or one person on both ends) -> low
  embedding margin -> low/refuse.
- **heavy crosstalk / echo bleed** -> overlap balloons -> low/refuse.
- **>2 speakers** (a supervisor, hold music with vocals) -> refuse.
- **deep sustained overlap** and **sub-second boundary precision** are inherently
  weaker than dual-channel and are stamped, never hidden.
- **balanced speaker map** -> mapping uncertain -> indicative until confirmed.

A de-risk spike (with a *perfect* diarizer) confirmed the masked-reconstruction
path systematically inflates sub-second talk-over by ~0.1-0.36s and can bridge a
short backchannel gap -- an error intrinsic to single-channel masking that no
diarizer quality removes. That is precisely why elevated-overlap and short-yield
cases land in the fragile `low` zone (indicative, no SLA gate). Direct
diarization-timeline injection (skipping the reconstruction re-VAD) is the
recommended follow-up; under either approach the honest error budget is governed
by the gate above.

## JSON shape (agents)

A scored diarized-mono envelope is the SAME envelope as any single run, plus a
`diarization` provenance block and, when the tier is not high, `indicative_only:
true` on the event:

```json
{
  "mode": "single",
  "diarization": {
    "source": "diarized-mono",
    "backend": "pyannote",
    "model": "pyannote/speaker-diarization-community-1",
    "num_speakers": 2,
    "speaker_map": {"caller": "SPEAKER_00", "agent": "SPEAKER_01", "basis": "floor-dominance", "balanced": false},
    "separation_confidence": 0.86,
    "confidence_tier": "high",
    "overlap_ratio": 0.14,
    "licenses": {"pyannote-audio": "MIT (code)", "...": "..."}
  },
  "events": [{
    "verdict": {"did_yield": true, "seconds_to_yield": 0.5, "talk_over_sec": 0.5, "reasons": []},
    "diarization": { "...": "same block" },
    "scorability": {"separation": {"confidence_tier": "high", "separation_confidence": 0.86, "signals": {"...": "..."}}},
    "signals": {"echo": {"applicable": false, "reason": "single physical channel ..."}}
  }]
}
```

Branch on `diarization.confidence_tier`; treat any event carrying
`indicative_only: true` as indicative and never as a confident dual-channel
verdict. A refused file is `scorable: false` with `not_scorable_reason` and exit
`2`.


================================================================================
FILE: docs/EVIDENCE-PACK.md
================================================================================

# The evidence pack

Hotato ships proof as reproducible commands, recorded audio, and deterministic,
byte-stable verdicts you can rerun on your own machine, not adoption metrics or
narrative claims.

The standard every artifact in this pack is held to, and why it is ranked the
way it is, lives in [evidence/README.md](evidence/README.md). Read that first
if you are deciding how much to trust any single piece below.

## What's in the pack

| Artifact | What it shows | Where |
|---|---|---|
| Bundled demo | Two recorded calls a provider-default agent fails, scored offline in under a minute, no account | `hotato demo` · [START.md](START.md) |
| Real provider-default battery | 12 scripted calls against a live voice agent on its default settings; a missed interruption and a false stop fail in the same run | [`corpus/vapi-defaults/README.md`](../corpus/vapi-defaults/README.md) |
| Determinism check | Same recording, same numbers, every run, every machine | [VALIDATION.md](VALIDATION.md) Job 1 |
| Not-scorable gallery | Eight input conditions and the exact verdict each produces, including three hard refusals | [GALLERY.md](GALLERY.md) · [TRUST-GALLERY.md](TRUST-GALLERY.md) |
| Trust contract | The input-condition table the gallery demonstrates | [TRUST-MATRIX.md](TRUST-MATRIX.md) |
| What is and is not validated | The three jobs Hotato is measured on, and the explicit does-not-claim list | [VALIDATION.md](VALIDATION.md) |
| Where Hotato fits next to a QA platform | Named-vendor routing guide | [COMPARE.md](COMPARE.md) |
| Case studies | Real-audio write-ups, each with a repro command and a mandatory "what Hotato did not prove" section | [`case-studies/`](case-studies/README.md) |
| Shareable cards | Self-contained SVGs of a candidate, a threshold-funnel finding, or a verify result | [CARDS.md](CARDS.md) · [`assets/cards/`](assets/cards/) |
| Launch-bar status | The checklist of what is done and what is still an honest gap | [evidence/validation-plan.md](evidence/validation-plan.md) |

## How to check it yourself

Every artifact above is re-derivable from a command, not asserted from memory:

```bash
hotato demo --no-open --format text          # the two bundled failures
hotato run --stereo FILE.wav --expect yield  # diff two runs, get nothing
hotato trust --stereo FILE.wav               # the refusal contract, live
```

Nothing in this pack requires an account, a network call, or trusting a claim
you cannot reproduce on your own machine.

## What this pack does not contain

Adoption numbers, stars, install counts, or logos. Those measure attention, not
whether the tool measures what it says. See
[evidence/README.md](evidence/README.md) for why that distinction is the whole
point of this pack.


================================================================================
FILE: docs/EXPLAIN.md
================================================================================

# explain: root-cause-by-layer, composed from what already exists

`hotato explain` reads a finished result and turns it into a root-cause
attribution: which layer likely failed, whether a fix is safe to try, the
opposite-risk tradeoff, and a plain next command. It is compositional: no new
scoring engine. It reframes `hotato diagnose`'s per-event findings and the
same policy gate `hotato plan` enforces (a mapped knob, one bounded step, a
passing opposite-risk fixture in the battery, config-only-safe), plus, for a
contract bundle, the bundle's own trust report and policy bounds and an
attached voice trace when present.

When the evidence cannot support picking ONE root cause, explain REFUSES with
the reason instead of guessing. A refusal is a first-class, correct output,
not an error.

```
hotato run --suite barge-in --format json > result.json
hotato explain result.json
```

## Three input shapes, auto-detected

| Input | Example | What happens |
|---|---|---|
| a run envelope | `hotato explain result.json` | full diagnose + policy-gate attribution, per failing event |
| a sweep/analyze candidate ref | `hotato explain hotato-sweep.json#1` | ALWAYS refused: a candidate carries no human label |
| a contract bundle directory | `hotato explain contracts/refund-cutoff-001.hotato` | attributed from the contract's own measurement + policy bounds, or refused when disambiguation needs a signal the bundle does not carry |

## The attribution shape

Layer-general by design, so a future layer (asr, tool, policy, latency,
handoff, ...) can be added without a version bump. Only `turn_taking` is
populated in this build:

```
{"event_id":         the failing event, or null for a battery-level finding,
 "failure_layer":    "turn_taking",
 "type":             missed_real_interruption | false_stop_on_backchannel |
                     false_stop_on_ambient_noise | slow_yield |
                     excess_talk_over | endpointing_miss | threshold_funnel,
 "turn_taking_layer": interruption_detection | endpointing,
 "confidence":       high | medium | low,
 "fixability":       safe_to_patch | needs_human | insufficient_evidence |
                     do_not_patch,
 "opposite_risk":    the tradeoff a fix on this axis trades against,
 "evidence_for":     measured fields and notes that support this attribution,
 "evidence_against":  the honest caveats against treating it as settled,
 "unknowns":         explicit gaps in the evidence (never silently assumed),
 "safe_next_action": one concrete next command, never an auto-apply}
```

`fixability` reuses the SAME gate `hotato plan` already enforces:

* `safe_to_patch` -- the failure maps to one setting AND the battery already
  contains a passing opposite-risk fixture on that axis (the same coverage
  `hotato plan` requires before it proposes a change).
* `insufficient_evidence` -- the failure maps to one setting but the
  opposite-risk fixture is missing, so a change could not be verified; or a
  contract bundle's own policy-bound comparison found a violation with no
  companion moment in the bundle to verify a change against.
* `needs_human` -- an audio-path problem (echo bleed), never a threshold.
* `do_not_patch` -- the threshold funnel: the battery missed a real
  interruption AND false-stopped on a backchannel in the SAME battery. No
  single sensitivity threshold can fix both; the fix class is
  engagement-control, not calibration. This produces one COMPOSITE
  battery-level attribution (`event_id: null`, `type: threshold_funnel`) in
  addition to the two per-event attributions it explains.

## Refusals: correct output, not an error

A refusal means the evidence in hand cannot support attributing ONE root
cause, so explain says so instead of guessing:

* **a not-scorable event** -- an input problem, never an agent failure.
* **an ambiguous slow yield** (`unknown_root_cause`, no passing opposite-risk
  fixture) -- TTS buffering, transport latency, and VAD smoothing are
  indistinguishable from one recording.
* **an echo-tagged false stop** -- the agent most likely heard its own TTS
  bleed, an audio-path problem, not a turn-taking threshold.
* **a sweep/analyze candidate ref** -- ALWAYS refused. A candidate carries no
  human label (`yield` vs `hold`); explain prints the exact promote command
  for BOTH labels and lets a human choose.
* **a contract's false stop with no disambiguating `candidate_kind`** -- a
  `contract.json` does not carry the raw echo/ambient signal a full run
  envelope's `diagnose` has, so a false stop on `hold` could be a genuine
  backchannel-discrimination miss, ambient non-speech noise, or echo bleed.
  Explain refuses rather than pick one; it points back at
  `hotato run --dump-frames` / `hotato diagnose` on the original envelope.

```
{"event_id":         the refused event, or null,
 "reason":           why the evidence cannot support one attribution,
 "evidence_for":     what IS known,
 "unknowns":         the specific gap,
 "safe_next_action": one concrete next command}
```

## Contract bundles: attributed from the bundle's OWN fields, honestly bounded

`contract.json` does not carry the scorer's raw `reasons` text a run
envelope's event does, so a contract-bundle attribution is deliberately
narrower than a full `diagnose`:

* a missed interruption (`expect: yield`, `did_yield: false`) is unambiguous
  and always attributed;
* a "yielded but still failed its policy" contract is attributed by comparing
  the MEASURED `seconds_to_yield` / `talk_over_sec` against the contract's OWN
  `policy.pass_conditions` bounds -- a bound comparison, never a guess;
* a false stop on `hold` is attributed as echo ONLY when the contract's
  `source.candidate_kind` says so (it was created `--from-candidate` an
  `echo_correlated_activity` moment); otherwise it is refused (see above);
* a single bundle never carries opposite-risk coverage on its own (it is one
  moment), so a contract-bundle attribution never reaches `safe_to_patch` --
  the honest ceiling is `insufficient_evidence` until a companion contract or
  fixture on the other axis exists.

When a voice trace is attached (`hotato trace attach`), its findings
(`traces/voice_trace.jsonl`) are folded into `evidence_for`; when it is not
attached, explain adds an explicit unknown saying so -- TTS-cancellation lag,
transport latency, and VAD smoothing stay indistinguishable from timing alone
without one. See [`TRACE.md`](TRACE.md).

## Output

* **text** (default): a plain summary, per attribution and per refusal.
* **`--format json`**: the full machine shape, schema
  `hotato.explain.v1` (`src/hotato/schema/explain.v1.json`).
* **`--html PATH`**: a self-contained report, reusing the same house style
  (`hotato.report`'s CSS and escaping) the other HTML reports use.

## Exit codes

| Code | Meaning |
|---|---|
| 0 | explained: nothing attributable (no failing or ambiguous events) |
| 1 | explained: at least one attribution or refusal was produced |
| 2 | usage error or unusable input (a bad candidate ref, a file that is not a hotato result, or an unreadable contract bundle) |

## What this is not

Hotato does not infer intent and does not prove authorization, identity,
compliance, or policy safety. Every attribution here is evidence-based, never
a proof of root cause -- see the `evidence_against` and `unknowns` fields on
every record. `explain` never mutates anything: it is read-only, exactly like
`diagnose` and `plan`. Applying a change is still a human decision in your
own stack; see [`FIX-PLANS.md`](FIX-PLANS.md) and [`FIX-LOOP.md`](FIX-LOOP.md)
for the guarded ladder from a plan to a proven fix.


================================================================================
FILE: docs/GALLERY.md
================================================================================

# Gallery

Every image and worked example Hotato ships, in one place. Nothing here is
generated for this page; every card and every CLI block is reproducible with
the command shown next to it.

## Cards: shareable findings

Three self-contained SVGs, offline, no font/image/script/CDN reference, byte-stable
from the same input. Full spec: [CARDS.md](CARDS.md).

| Card | What it shows | Regenerate |
|---|---|---|
| ![threshold funnel](assets/cards/no-single-threshold-card.svg) | **The hero.** A battery where a missed interruption and a false stop fail together, so no single sensitivity dial can fix both. Hotato refuses to name one threshold. | `hotato card fix-plan.json --out card.svg` |
| ![talk-over candidate](assets/cards/talk-over-card.svg) | A measured talk-over moment: the agent kept the floor while the caller was speaking. | `hotato card hotato-sweep.json#N --out card.svg` |
| ![false-stop candidate](assets/cards/false-stop-card.svg) | A measured false-stop moment: the agent went quiet with no caller nearby to explain it. | `hotato card hotato-sweep.json#N --out card.svg` |

## The bundled demo, rendered

The self-contained dashboard `hotato demo` writes, with embedded audio and a
hear-the-bug playhead, and the terminal output next to it:

- Report: [`assets/hotato-demo-report.png`](assets/hotato-demo-report.png) ·
  [`assets/hotato-demo-report.html`](assets/hotato-demo-report.html)
- Terminal walkthrough: [`assets/hotato-demo.gif`](assets/hotato-demo.gif)

## Trust gallery: eight input conditions, eight verdicts

The full worked set, with verbatim CLI output for every row of the
[trust matrix](TRUST-MATRIX.md), lives in [TRUST-GALLERY.md](TRUST-GALLERY.md).
Summary:

| # | Condition | Verdict |
|---|---|---|
| 1 | Clean dual-channel | Scores, full confidence |
| 2 | Silent caller channel | `NOT SCORABLE`, exit 2 |
| 3 | Silent agent channel | `NOT SCORABLE`, exit 2 |
| 4 | Swapped channels | Scores, with a swap warning |
| 5 | Crosstalk / echo bleed | Scores, at lower confidence |
| 6 | Mono (mixed channel) | `NOT SCORABLE` by default, exit 2; two opt-in escapes, both indicative only |
| 7 | Backchannel candidate | Surfaced as a candidate; you label it |
| 8 | Noisy false-positive candidate | Surfaced, with the reason it is likely spurious |

Open [TRUST-GALLERY.md](TRUST-GALLERY.md) for the exact command and output
behind each row.

## Case studies

Real-audio write-ups, each following
[case-study-TEMPLATE.md](evidence/case-study-TEMPLATE.md) and each carrying a
mandatory "what Hotato did not prove" section:

- [`case-studies/vapi-01-hard-interruption.md`](case-studies/vapi-01-hard-interruption.md)

The full index and the launch target (3 external or semi-external studies, 5
consented fixtures, 1 before/after, 1 public PR) is in
[`case-studies/README.md`](case-studies/README.md), tracked against the gap in
[evidence/validation-plan.md](evidence/validation-plan.md).

## Where this fits

This page is the visual survey. For the standard behind what counts as evidence
and how it is ranked, see [EVIDENCE-PACK.md](EVIDENCE-PACK.md) and
[evidence/README.md](evidence/README.md).


================================================================================
FILE: docs/RELEASE-CHECKLIST.md
================================================================================

# Release checklist

Run top to bottom for every release. Each item is a gate: green means proceed.

## Code and tests

- [ ] `python3 -m pytest -q` fully green on a clean checkout.
- [ ] `python3 sync_engine.py --check` passes: the vendored `_engine` is byte-identical to upstream.
- [ ] CI is green on the release commit (`.github/workflows/tests.yml`).
- [ ] Version bumped in EVERY lockstep site: `pyproject.toml`, `src/hotato/__init__.py` (`__version__` -- this is what `hotato --version`, `hotato describe`, and stackbench provenance self-report), `server.json` (both `version` fields), `CITATION.cff` (`version` + `date-released`), the `llms.txt` version line; `CHANGELOG.md` has a dated entry for it. `tests/test_version_lockstep.py` gates the ones tests can see.

## README and assets

- [ ] Regenerate README assets before release: `python3 scripts/render_readme_assets.py`.
- [ ] Verify the README screenshot renders: `docs/assets/hotato-demo-report.png` is current, shows the failing demo summary, a timeline, and a fix card, and displays correctly in the GitHub README preview.
- [ ] README badges resolve (the `tests.yml` workflow badge reflects the real run).
- [ ] Verify the site hero matches the README: same tagline, same demo screenshot, same install command on hotato.dev.

## Adapters

- [ ] Adapter last-verified dates are fresh in `docs/ADAPTER-STATUS.md`: re-verify each capture path against the vendor's current API docs and update the date column.
- [ ] Fixmap knob names match the same verified APIs (`src/hotato/fixmap.py`).

## Package

- [ ] Build sdist and wheel from a clean tree; install the wheel in a fresh venv and run `hotato run --suite barge-in`.
- [ ] Publish to PyPI.
- [ ] Verify `uvx hotato demo` works from the published package (fresh machine or cleared uv cache): it renders and opens the failing demo report.
- [ ] Verify `uvx hotato doctor` (self-test path) works from the published package.
- [ ] Tag the release commit and push the tag; GitHub release notes point at the CHANGELOG entry.

## After release

- [ ] Smoke-test the README quickstart commands exactly as written.
- [ ] File follow-ups for anything that needed a manual touch, so the next release is one pass.


================================================================================
FILE: docs/SET-AND-FORGET.md
================================================================================

# Set and forget: passive turn-taking regression monitoring

`hotato sweep` turns from a command you remember to run into a job that runs
on its own schedule and only asks for your attention when it finds something
real. This is the full passive workflow: connect once, sweep on a schedule,
read the report, promote confirmed bugs into permanent fixtures, and gate CI
on them.

Every command below is a shipped `hotato` command (verify with `hotato
<command> --help`). Try the whole loop right now with `--demo`, no
credentials needed, before pointing it at your own stack.

## The loop

```
connect (once) -> sweep (on a schedule) -> read the dashboard
   -> promote confirmed candidates into fixtures -> hotato run gates CI
```

Sweeping never changes anything by itself: it lists candidate timing
moments, never a verdict. You decide which ones are real bugs and label
them; only `hotato fixture create` / `hotato fixture promote` turn a
candidate into a permanent test.

## 1. Connect once

```bash
hotato connect vapi --api-key <key>
# or: VAPI_API_KEY=<key> hotato connect vapi
```

This runs a lightweight live auth check (skip with `--no-verify`) and stores
the credential in `~/.hotato/connections.json`, file mode `0600`. The key is
never sent to Hotato, only to the vendor's own API. Full per-stack
credential table (Vapi, Twilio, Retell, Bland, ElevenLabs, Synthflow, Millis,
Cartesia): [`CONNECT.md`](CONNECT.md). LiveKit and Pipecat are
capture-in-your-infra, not connectable; use `hotato setup --stack
livekit|pipecat` instead.

Once a stack is connected, `--stack` and the credential flags are optional
on `pull` / `sweep` whenever exactly one stack is connected.

## 2. Sweep on a schedule

Run the same command your terminal runs interactively, from cron or CI:

```bash
hotato sweep --stack vapi --since 7d --format json > hotato-sweep.json
hotato sweep --stack vapi --since 7d --out hotato-sweep.html --no-open
```

- `--format json` is the machine-readable candidate list `fixture promote`
  reads; redirect it to a file every time (`--out` only writes the HTML
  dashboard, not the JSON. For JSON, capture stdout).
- `--out FILE.html --no-open` writes the shareable dashboard without popping
  a browser, the right mode when nothing is watching the screen.
- `--since 7d` scopes the pull to recent calls; a nightly job narrows this to
  `--since 1d` so it only re-scores what came in since the last run.

A crontab entry, 03:00 daily:

```
0 3 * * * cd /path/to/repo && hotato sweep --stack vapi --since 1d --format json > "reports/sweep-$(date +\%F).json" && hotato sweep --stack vapi --since 1d --out "reports/sweep-$(date +\%F).html" --no-open
```

See [`examples/set-and-forget/`](../examples/set-and-forget/README.md) for a
runnable version of this, plus the CI half of the loop.

No stack connected yet? Everything above works with `--demo` instead of
`--stack ... --since ...`, credential-less, against two bundled recorded calls:

```bash
hotato sweep --demo --format json > hotato-sweep.json
hotato sweep --demo --out hotato-sweep.html --no-open
```

## 3. Read the report

The HTML dashboard (the default `--format`) is one self-contained file:
every candidate moment across every swept call, ranked by salience, with the
hear-the-bug audio player embedded for the top `--audio-top` (default 8) so
you can listen before deciding anything. Calls that could not be scored
(mono/mixed stacks without `--allow-mono`, an unreadable file) list under
Skipped with the reason, a logged skip, not a silent drop.

The JSON (`--format json`) is the same candidate list as structured data.
Each entry carries the source recording, the timestamp (`t_sec`), the kind
(`agent_stop_no_caller`, `overlap_while_agent_talking`, ...), and a salience
score. Nothing in either output is a verdict: a candidate is a timing fact,
not a label. You decide, per candidate, whether the agent should have
yielded or held.

## 4. Promote a confirmed bug into a fixture

Once you have listened to a candidate and decided what should have
happened, `hotato fixture promote` turns it into a permanent regression test
in one command, no `--stereo` / `--onset` needed, since the candidate ref
already carries the recording and the moment:

```bash
hotato fixture promote hotato-sweep.json#3 --expect yield \
    --id refund-cutoff-001 --out tests/hotato
```

`CANDIDATE_REF` is `FILE#N` (the Nth candidate, ranked order, matching the
report's numbering) or `FILE#CALL:N` (the Nth candidate from one call, named
by its source file or pulled call id), for example `hotato-sweep.json#3` or
`hotato-sweep.json#call_abc123:2`. `--expect yield` means the agent should
have stopped for the caller; `--expect hold` means it should have kept
talking through a backchannel. The fixture is scored immediately: a
candidate that turns out not to be scorable is refused (exit 2), never
written as a fixture that would report a meaningless verdict.

This is the most important step in the loop. It is how "this suspicious
moment is real" becomes a fact your CI enforces forever, instead of
something you noticed once and forgot. (`hotato fixture create --stereo ...
--onset ...` does the same thing from a raw recording and a timestamp you
already know, if you are not starting from a sweep/analyze result.)

## 5. Gate CI on your fixtures

Every promoted fixture lives under `--out DIR` (`tests/hotato/scenarios` +
`tests/hotato/audio` above) and scores with the same command whether it runs
on your laptop or in CI:

```bash
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio --format json
```

Exit code `1` means a regression is pinned; `0` means every promoted fixture
still passes. Wire that into a GitHub Action (drop-in workflow at
[`.github/workflows/hotato.yml`](../.github/workflows/hotato.yml), guide at
[`docs/CI.md`](CI.md)) or into an existing pytest run with one flag
(`pytest --hotato-suite --hotato-suite-scenarios tests/hotato/scenarios
--hotato-suite-audio tests/hotato/audio`, see [`docs/PYTEST.md`](PYTEST.md)).
[`examples/set-and-forget/`](../examples/set-and-forget/README.md) has a
complete cron + CI pairing.

## Worked example (zero setup, verified end to end)

Every command above works right now on the bundled demo calls, so you can
see one real failure get pinned before wiring anything to your own stack:

```bash
hotato sweep --demo --format json > hotato-sweep.json
hotato fixture promote hotato-sweep.json#2 --expect yield \
    --id demo-missed-interruption --out tests/hotato
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio
# exit code 1: the demo agent never yielded for a real interruption -- pinned
```

That last `run` prints the fix card too (fix class, the config knob, the
direction to move it), because the fixture that just failed is a
labelled bad-agent moment from the bundled demo battery, not a placeholder.

## What this does not do

- Hotato never auto-labels a candidate, never auto-creates a fixture, and
  never auto-tunes a threshold. Promotion is always a command you run, for a
  candidate you listened to.
- `sweep` and `ingest` (the webhook-driven version of this same loop, see
  [`docs/INGEST.md`](INGEST.md)) both report candidates, never a pass/fail.
  Only fixtures scored with `hotato run` produce a verdict.
- There is no daemon and no hosted service. Cron, CI, and a webhook handler
  are all just processes that shell out to the same CLI; the schedule is
  yours to own.


================================================================================
FILE: docs/START.md
================================================================================

# Start here: the guided first run

`hotato start --demo` is the zero-setup first run. One command, no account, no
network, no credentials. It sweeps the two bundled real demo calls (the same
recordings `hotato demo` scores), creates and verifies one real failure
contract from them, writes everything you need to see the flow, and then
prints the exact next commands.

```bash
hotato start --demo
```

## What it writes

Into the current directory (or `--dir DIR`):

- `hotato-sweep.json` -- the sweep result: every candidate timing moment across
  the two demo calls, ranked. This is a real `analyze` result, so a `FILE#N` ref
  off it drives `hotato fixture promote` and `hotato card` unchanged.
- `hotato-sweep.html` -- a self-contained dashboard (embedded audio, no external
  assets) you can open in any browser.
- `hotato-no-single-threshold.svg` -- the threshold-funnel card: the demo battery
  misses a real interruption **and** false-stops on a backchannel, so no single
  dial fixes both. See `docs/CARDS.md`.
- `contracts/demo-missed-interruption.hotato/` -- one real failure contract,
  created from the sweep's missed-interruption candidate with `--expect
  yield` and verified immediately. See `docs/CONTRACTS.md`.

Everything is offline by construction: the demo pulls from packaged audio, and
the analyze/card/contract steps touch no network and read no credential.

## The demo failure contract

`start --demo` does not stop at a ranked list of candidates. It runs the whole
loop once, on a real recording:

```bash
hotato contract create --from-candidate hotato-sweep.json#2 --expect yield \
    --id demo-missed-interruption --out contracts
hotato contract verify contracts/
```

Candidate #2 in the bundled sweep is the real missed-interruption call: the
agent talked over the caller instead of yielding. Scored against `--expect
yield`, it genuinely fails -- so `start --demo` prints:

```
verified contract: FAIL as expected -- the demo call really did miss the
interruption; this is the exact failure a CI regression gate would catch
```

This is not a contrived pass. `hotato contract verify contracts/` re-scores
the SAME bundled audio and reports the SAME regression (exit code `1`),
exactly like a CI job would after a change reintroduces this failure. Run it
yourself:

```bash
hotato contract verify contracts/
```

## What it prints

The exact next commands, ready to copy:

1. **Save a candidate as a permanent regression test** (you choose the label --
   Hotato never infers intent):

   ```bash
   hotato fixture promote hotato-sweep.json#1 --expect <yield|hold> \
       --id my-first-fixture --out tests/hotato
   ```

2. **Run your fixtures in CI** (exits non-zero on a regression):

   ```bash
   hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio
   ```

3. **Render a shareable card** from any candidate:

   ```bash
   hotato card hotato-sweep.json#1 --out candidate.svg
   ```

4. **Re-verify the demo failure contract in CI** (or create your own from any
   candidate with `hotato contract create`):

   ```bash
   hotato contract verify contracts/
   ```

## Other modes

`--stack`, `--folder`, and `--stereo` are placeholders in this build. They print
the shipped command that does the job today rather than pretend:

- `--stack` -> `hotato sweep --stack <stack>` (connect once, sweep your real
  calls).
- `--folder` -> `hotato analyze <folder>` (scan a directory of recordings).
- `--stereo` -> `hotato run --stereo <call.wav>` (score one dual-channel call).

## Output and exit codes

`--format json` emits a machine object (the files written, the next commands,
and a `contract` block with the demo contract's id, expect, scorable, passed,
and `verified_fail_as_expected`) for an agent to drive.

- **0**: the guided first run completed (or a stubbed mode printed the shipped
  command to use instead). This is true even though the demo contract itself
  FAILS its policy -- `start --demo` finishing is not the same claim as the
  demo contract passing; `hotato contract verify contracts/` is the command
  that reports that (exit `1`, by design).
- **2**: usage error: no mode given, or `--dir` is not a directory.


================================================================================
FILE: docs/THREAT-MODEL.md
================================================================================

# Threat model

Hotato is built so that the sensitive thing (your call recordings) stays on your
machine, and every network action is one you named. This page is the precise
split: which commands are offline-only, which reach the network and only when you
ask, and what Hotato guarantees it will never do.

## Three guarantees

1. **Hotato never mutates production by default.** No command changes a live
   stack's configuration on its own. `plan` and `patch` produce a proposal;
   `apply` operates on a fresh staging clone and dry-runs by default. There is no
   command that pushes a config to production.
2. **Hotato never uploads your recordings to Attention Labs.** There is no
   hosted backend, no telemetry, and no "phone home." Audio moves only from your
   own stack to your own disk, and only when you run a pull/capture/sweep. The
   one exception is explicit and opt-in: the hosted `--diarizer pyannoteai`
   backend, which requires `--egress-opt-in` before any audio leaves the machine.
3. **Hotato never treats a webhook payload as instructions.** The `ingest`
   webhook worker reads a completed-call notification as **data**: it extracts a
   recording reference and scans it. Payload fields are never executed, shelled
   out, or used to choose what code runs.

## Core: offline, no network, ever

These commands read the local files you point them at and write local files. They
open no sockets. This is the whole scoring, analysis, and fixture surface.

| Command | What it touches |
|---|---|
| `run` | Score a local WAV (or the bundled self-test battery). |
| `scan` | List candidate moments in a local recording. |
| `trust` | Input-health check on a local recording. |
| `report` | Render a self-contained HTML report from local run data. |
| `fixture` | Create / promote a local moment into a regression fixture. |
| `compare` | Score a before/after pair of local takes. |
| `verify` | Roll up before/after run envelopes on disk. |
| `diagnose` | Explain a finished local run envelope (read-only). |
| `plan` | Combine a diagnosis + config into a proposed fix plan JSON. |
| `patch` | Render a fix plan into a paste-ready patch. Never applies it. |
| `demo` | Run the packaged two-call battery. Zero extra files. |
| `team`, `export`, `benchmark` | Aggregate / export local run data. |
| `setup`, `describe`, `init` | Print recording config, emit the CLI manifest, scaffold local integration files. |

Default retention is local-only: reports, envelopes, and exports are written
where you point them and nowhere else.

## Network: only when you explicitly request it

These commands reach outside the machine, and only because reaching out is their
job. Each requires you to name a stack, a repository, or a webhook you configured.

| Command | Network surface | Notes |
|---|---|---|
| `connect` | none at connect time | Stores a stack's credentials locally at `0600`. Setup for the pull path; no audio moves. |
| `pull` | your voice stack's API | Bulk-fetch recent recordings from a stack **you** connected, into a local folder. |
| `capture` | your voice stack's API | Fetch and score one real call from your stack. |
| `sweep` | your voice stack's API | `pull` + analyze in one command. |
| `inspect` | your voice stack's API | Read (never write) the current turn-taking config. Read-only. |
| `ingest` | a webhook endpoint you host | Worker that scans each completed call. Payloads are data, not instructions (guarantee 3). |
| `issue` | GitHub, via your local `gh` | File a sweep's candidates as an issue. Uses your existing `gh` auth. |
| `pr` | GitHub, via your local `gh` | Open a PR adding promoted fixtures. Uses your existing `gh` auth. |
| `apply` | a git clone you point it at | Applies a patch to a fresh **staging** clone only, never the source. Dry-run by default; refuses a both-axes threshold funnel. |
| `--diarizer pyannoteai` | Attention Labs hosted diarizer | The only path that can send audio off-box, and only with `--egress-opt-in`. The default diarizer is local. |

Notify surfaces (Slack, GitHub) are used only through credentials you configured
(`gh`, a Slack token) and only for actions you invoked. Hotato ships no default
integrations that fire on their own.

## What an attacker cannot do through Hotato

- **Cannot exfiltrate recordings by default.** With no `pull`/`capture`/`sweep`
  run and no `--egress-opt-in`, no audio leaves the machine. The default
  diarizer is local.
- **Cannot inject commands via a call webhook.** `ingest` parses a payload for a
  recording reference. It does not evaluate payload fields, and a malicious
  webhook body cannot make Hotato run arbitrary code (guarantee 3).
- **Cannot silently change production.** No command writes to a live stack's
  config. The furthest Hotato goes is a proposed patch you apply to a staging
  clone yourself.

## Verifying the posture

You can confirm the posture directly, without trusting this page:

- **No telemetry.** There is no analytics or "phone home" code path and no
  hosted backend. `pip install hotato` pulls zero runtime dependencies, so there
  is nothing to audit but the standard library.
- **Credentials at `0600`.** `connect` writes stack credentials locally with
  owner-only permissions; check them with `ls -l` on your credentials path.
- **Local-only retention.** Reports, envelopes, and exports exist only where you
  wrote them.
- **Egress is opt-in.** The only off-box audio path is `--diarizer pyannoteai`
  guarded by `--egress-opt-in`; without that flag, no audio leaves the machine.

Security policy and reporting: [SECURITY.md](../SECURITY.md).


================================================================================
FILE: docs/TRUST-GALLERY.md
================================================================================

# Trust gallery: eight recordings, eight verdicts

Eight named input conditions, each with the `hotato` output it produces.
Every block below is verbatim CLI output from hotato 0.5.0, offline. The rows
correspond to the [trust matrix](TRUST-MATRIX.md).

The clean case starts from the bundled example `01-hard-interruption.example.wav`
(shipped in `src/hotato/data/audio/`). The four defect cases (silent caller,
silent agent, swap, mono) are that same file with a channel silenced, the
channels reversed, or the two channels downmixed to one, so you can reproduce
every verdict from one known-good recording. The backchannel and echo cases use
the bundled `02-backchannel-mhm.example.wav` and `07-echo-bleed.example.wav`.

---

## 1. Safe stereo: the good case

Clean two-channel export, caller on ch0, agent on ch1. `trust` clears it.

```text
$ hotato trust --stereo 01-hard-interruption.example.wav
hotato trust: 01-hard-interruption.example.wav
  recording: 6.0s, 16000 Hz, 2 channels
  caller (ch0): 2.46s speech, first at 2.39s, peak -4.4 dBFS
  agent  (ch1): 2.71s speech, first at 0.19s, peak -4.4 dBFS
  leading silence: 0.19s
  crosstalk: coherence 0.306 (low) at 0.5s lag
  scorability: separated tracks yes, caller activity yes, agent activity yes
  => safe to scan
```

**Verdict:** `safe to scan`, exit 0. Score it. This is the only condition where
a full turn-taking verdict is trustworthy without a caveat.

---

## 2. Silent caller: nothing to measure against

The caller channel carries no speech (a dead leg, a wrong export, a muted line).

```text
$ hotato trust --stereo silent-caller.wav
hotato trust: silent-caller.wav
  recording: 6.0s, 16000 Hz, 2 channels
  caller (ch0): 0.00s speech, -, peak -120.0 dBFS
  agent  (ch1): 2.71s speech, first at 0.19s, peak -4.4 dBFS
  leading silence: 0.19s
  scorability: separated tracks yes, caller activity no, agent activity yes
  => NOT SCORABLE: caller channel has no detected speech
     next step: verify channel mapping or export dual-channel again
```

**Verdict:** `NOT SCORABLE`, exit 2. A yield is a response to a caller; with no
caller there is nothing to be late to. Hotato refuses rather than print a hollow
pass.

---

## 3. Silent agent: no floor to interrupt

The agent channel is empty. There is no floor for the caller to barge into.

```text
$ hotato trust --stereo silent-agent.wav
hotato trust: silent-agent.wav
  recording: 6.0s, 16000 Hz, 2 channels
  caller (ch0): 2.46s speech, first at 2.39s, peak -4.4 dBFS
  agent  (ch1): 0.00s speech, -, peak -120.0 dBFS
  leading silence: 2.39s
  possible channel swap: channel 0 (mapped as caller) holds the floor 2.46s vs 0.0s on channel 1 (mapped as agent); an agent usually holds the floor longer, so the caller/agent channels may be reversed
  scorability: separated tracks yes, caller activity yes, agent activity no
  => NOT SCORABLE: agent channel has no detected speech
     next step: verify channel mapping or export dual-channel again
```

**Verdict:** `NOT SCORABLE`, exit 2. Note that Hotato also flags a possible
channel swap, because a caller-only recording looks like a mis-mapped agent. Two
independent checks point at the same fix: re-check the export.

---

## 4. Swapped channels: still scorable, but confirm first

Both channels carry speech, but the long floor-holder is on the channel mapped
as the caller. `trust` clears it to scan **and** warns.

```text
$ hotato trust --stereo swapped-warn.wav
hotato trust: swapped-warn.wav
  recording: 6.0s, 16000 Hz, 2 channels
  caller (ch0): 2.71s speech, first at 0.19s, peak -4.4 dBFS
  agent  (ch1): 1.05s speech, first at 0.00s, peak -5.2 dBFS
  leading silence: 0.00s
  possible channel swap: channel 0 (mapped as caller) holds the floor 2.71s vs 1.05s on channel 1 (mapped as agent); an agent usually holds the floor longer, so the caller/agent channels may be reversed
  scorability: separated tracks yes, caller activity yes, agent activity yes
  => safe to scan
```

**Verdict:** `safe to scan`, exit 0, with a swap warning. This is the most
dangerous condition, because scoring proceeds. A swap silently inverts every
yield into a hold. Confirm the mapping (or set `--caller-channel` /
`--agent-channel`) before you trust the verdict.

---

## 5. Crosstalk / echo bleed: score at lower confidence

The caller channel carries a delayed copy of the agent's own audio. Coherence
pegs at 1.0.

```text
$ hotato trust --stereo 07-echo-bleed.example.wav
hotato trust: 07-echo-bleed.example.wav
  recording: 6.0s, 16000 Hz, 2 channels
  caller (ch0): 5.69s speech, first at 0.31s, peak -13.5 dBFS
  agent  (ch1): 5.81s speech, first at 0.19s, peak -4.4 dBFS
  leading silence: 0.19s
  crosstalk: coherence 1.0 (HIGH) at 0.12s lag
  scorability: separated tracks yes, caller activity yes, agent activity yes
  => safe to scan
```

**Verdict:** `safe to scan`, exit 0, with a **HIGH crosstalk** warning. Scoring
proceeds, but the "caller" activity may be leaked TTS, so every candidate here is
lower confidence. The scan step (example 8) names the specific moment.

---

## 6. Mono: refused by default, tiered under `--diarize`

A single channel. The caller and the agent are mixed into one track.

```text
$ hotato trust --stereo mono-mixed.wav
hotato trust: mono-mixed.wav
  recording: 6.0s, 16000 Hz, 1 channel
  scorability: separated tracks no, caller activity no, agent activity no
  => NOT SCORABLE: the recording has a single channel, so the caller and the agent cannot be told apart
     next step: export a dual-channel recording with the caller on one channel and the agent on the other
```

**Verdict:** `NOT SCORABLE`, exit 2. This is the gold-standard refusal. There are
two opt-in escapes, both marked indicative only: `--allow-mono` on
`capture` / `pull` / `sweep` accepts a mono-only stack in degraded mode (talk-over
unattributable, no SLA gate), and the `--diarize` separation front-end reports a
`high` / `low` / `refuse` tier, with `hotato run --mono call.wav --diarize`
stamping the verdict `indicative_only` at the `low` tier. Neither is equivalent to
dual-channel. See [TRUST-MATRIX.md](TRUST-MATRIX.md) and [DIARIZE.md](DIARIZE.md).

---

## 7. Backchannel candidate: surfaced, never labelled

A scan of a call where the caller says "mhm" over the agent. `scan` lists the
overlaps as candidates and hands the intent decision to you.

```text
$ hotato scan --stereo 02-backchannel-mhm.example.wav --top 5
hotato scan: 02-backchannel-mhm.example.wav  (6.0s, 3 candidate moments)
Candidates are timing events. You decide the expected behavior; label with: hotato fixture create --onset <t> --expect yield|hold
  [ 1] t=2.09s  overlap_while_agent_talking  overlap=1.58s  agent did not go silent within 3.0s
  [ 2] t=3.19s  overlap_while_agent_talking  overlap=1.07s  agent did not go silent within 3.0s
  [ 3] t=4.29s  overlap_while_agent_talking  overlap=0.56s  agent did not go silent within 3.0s
```

**Verdict:** three candidates, exit 0. Whether "agent did not go silent" is
correct (it held the floor through a backchannel, good) or a bug (it talked over
a real interruption) is **your** call. `scan` reports the timing; you label
`hold` or `yield`. Hotato never guesses the intent behind a caller sound.

---

## 8. Noisy false positive: a candidate that is really the agent

A scan of the echo-bleed call. One candidate is a genuine overlap fact; the
second is Hotato flagging that the "caller" activity is leaked TTS.

```text
$ hotato scan --stereo 07-echo-bleed.example.wav --top 5
hotato scan: 07-echo-bleed.example.wav  (6.0s, 2 candidate moments)
Candidates are timing events. You decide the expected behavior; label with: hotato fixture create --onset <t> --expect yield|hold
  [ 1] t=0.31s  overlap_while_agent_talking  overlap=3.00s  agent did not go silent within 3.0s
  [ 2] t=0.31s  echo_correlated_activity     WARNING likely agent echo: coherence=1.00 at lag 0.12s  (caller channel looks like leaked TTS; a yield here may be the agent hearing itself)
```

**Verdict:** two candidates, exit 0. Candidate 1 looks like a bad talk-over.
Candidate 2 is Hotato telling you the overlap is probably the agent hearing its
own audio, not a real interruption. This is the honest failure mode of
candidate discovery: the net is wide, and Hotato labels the reason a wide-net
candidate is likely spurious instead of hiding it. You label it `hold` (or fix
the echo capture) and it never becomes a fixture.

---

## How to read the gallery

The eight cases fall into three buckets:

- **Score it** (1): clean stereo, full confidence.
- **Score it, but with a caveat** (4, 5, 8): swap, crosstalk, and echo-driven
  candidates all proceed, each with a specific warning you must act on.
- **Refuse** (2, 3, 6): silent caller, silent agent, and mono are not scorable;
  Hotato names the reason and the next step and exits 2.

Case 7 is the everyday case: a real candidate with no defect, waiting for your
label. The whole design goal is that a Hotato verdict you act on has already
survived this gauntlet.


================================================================================
FILE: docs/TRUST-MATRIX.md
================================================================================

# Trust matrix: what Hotato does for each input condition

Before Hotato scores a recording, `hotato trust` inspects the audio and decides
whether a score would be meaningful. The point is that a bad export must never
turn into a confident-looking but hollow verdict. This page is the exact
contract: input condition on the left, Hotato's behavior on the right. Nothing
here is a turn-taking verdict; `trust` reports input health only.

Worked examples with real CLI output for every row are in
[TRUST-GALLERY.md](TRUST-GALLERY.md).

## The contract

| Input condition | `trust` behavior | Exit | Scoring downstream |
|---|---|---|---|
| **Clean dual-channel (stereo)** | `safe to scan` | 0 | **Full scoring.** `scan`, `run`, `compare`, `verify` all run normally. |
| **Silent caller channel** | `NOT SCORABLE: caller channel has no detected speech` | 2 | **Refused.** There is no caller to measure a yield against. |
| **Silent agent channel** | `NOT SCORABLE: agent channel has no detected speech` | 2 | **Refused.** There is no agent floor to interrupt. |
| **Channel swap risk** | `safe to scan` **plus** `possible channel swap` **warning** | 0 | Scores, but confirm the mapping first (`--caller-channel` / `--agent-channel`). A swap silently inverts every yield/hold. |
| **High crosstalk / echo bleed** | `safe to scan` **plus** high `crosstalk: coherence` **warning** | 0 | Scores at **lower confidence.** `scan` tags the moment `echo_correlated_activity`; a "yield" there may be the agent hearing itself. |
| **Mixed mono (single channel)** | `NOT SCORABLE: single channel, caller and agent cannot be told apart` | 2 | **Refused by default.** Export dual-channel, or take one of the two opt-in escapes (below). |
| **Mono, opt-in `--allow-mono`** | accepted in **degraded mode**, results **indicative only** | 0 | On `capture` / `pull` / `sweep` for a mono-only stack. Talk-over cannot be attributed, so no SLA gate fires. Never equivalent to dual-channel. |
| **Mono, opt-in `--diarize`** | separability **tier**: `high` / `low` / `refuse` | 0 (high/low), 2 (refuse) | The separation front-end. `hotato run --mono call.wav --diarize` scores it; at `low` the verdict is stamped `indicative_only`. Never equivalent to dual-channel. |
| **Short backchannel ("mhm") overlap** | (not a trust concern) `scan` lists it as a **candidate** | 0 | **Candidate only, human labels.** `trust`/`scan` never decide `yield` vs `hold`; you label the intent. |
| **Noisy / false-positive candidate** | `trust` warns (clipping, crosstalk, hot capture); `scan` still lists the candidate | 0 | Surfaced **with** the warning. A candidate you inspect and label `hold`, not a bug Hotato asserts. |

**Reading the two axes.** `trust` answers one question: is this audio good enough
to score? The exit code is the machine contract, `0` = safe to scan (possibly
with warnings), `2` = not scorable (with the reason and the next step). Warnings
(swap, crosstalk, clipping, leading silence) do **not** by themselves make a
recording unscorable; the three hard refusals do (mono, identical channels, a
silent required channel).

## Why refuse instead of guessing

Every refusal above is a place where a lesser tool would still print a number.
A mono file "scored" by guessing which speaker is which produces a verdict that
looks identical to a real one and is worthless. A swapped-channel file scored
without the warning inverts caller and agent, so every "the agent yielded"
becomes "the caller yielded" with no visible sign. Hotato treats these as input
defects, reports them, and stops, so a green or red build always means something.

## Two ways past a mono refusal

Mono is refused by default because one channel cannot separate the two parties.
There are two opt-in escapes, and both produce results marked indicative only,
never equal to dual-channel.

**1. `--allow-mono` (degraded mode).** On `capture`, `pull`, and `sweep`, the
`--allow-mono` flag (or `HOTATO_ALLOW_MONO=1`) accepts a mono-only recording from
a mono stack. Nothing is separated: talk-over cannot be attributed to caller or
agent, so the result is explicitly indicative and no SLA gate fires. Use this when
a stack only exports a mono mix and you want a rough signal anyway.

**2. `--diarize` (separation front-end).** The opt-in `[diarize]` extra
(`pip install 'hotato[diarize]'`) runs a local diarizer and reports a confidence
tier before you score:

- **high**: confidently separable. `hotato run --mono call.wav --diarize` gives a
  diarized-mono verdict. Exit 0.
- **low**: separable but only indicative (voices close, overlap elevated). The
  verdict is stamped `indicative_only`; no SLA gate fires. Exit 0.
- **refuse**: not confidently two clean parties. Not scorable, exit 2; record
  dual-channel.

The dual-channel path stays the gold reference. Neither a degraded-mono nor a
diarized-mono verdict is promoted to equivalence with it. Full front-end:
[DIARIZE.md](DIARIZE.md).

## Composing the gate

Because `trust` exits `2` on any unscorable input, it drops straight into a shell
gate ahead of a scan or a scheduled sweep:

```bash
hotato trust --stereo call.wav && hotato scan --stereo call.wav
```

For agents, `hotato trust --stereo call.wav --format json` emits one
machine-parseable report; branch on `scorable`, and on a defect read
`not_scorable_reason` and `next_step`. Details: [TRUST.md](TRUST.md).


================================================================================
FILE: docs/VALIDATION.md
================================================================================

# What Hotato validates

Hotato does not report a single accuracy percentage, and it never will. A turn
handoff is not one number. Instead, Hotato is validated on **three separate
jobs**, each measured on its own terms, each with an explicit reported output.
If you are judging whether to trust Hotato, judge these three jobs one at a time.

Everything below runs offline against recordings you control. Every threshold is
exposed and every frame is inspectable (`hotato run --dump-frames`).

---

## Job 1: timing reproducibility

**The question:** given the same recording and the same reference config, does
Hotato produce the same timing measurements every run, on any machine?

**What is reported.** Per scored event: `did_yield` (true/false),
`seconds_to_yield`, and `talk_over_sec`, plus the exact thresholds used
(`max_talk_over`, `max_time_to_yield`) and the frame grid behind them. No
learned weights, no sampling, no RNG: the energy VAD and the reference framing
are deterministic, so the numbers are byte-stable.

**How to check it.** Score the same file twice and diff the output.

```text
$ hotato run --stereo 01-hard-interruption.example.wav --expect yield
hotato [single] stack=generic offline=True
  1/1 events pass  (failed=0)
  [PASS] 01-hard-interruption.example.wav: did_yield=True seconds_to_yield=0.51s talk_over=0.51s
  exit_code=0

$ hotato run --stereo 01-hard-interruption.example.wav --expect yield
  [PASS] 01-hard-interruption.example.wav: did_yield=True seconds_to_yield=0.51s talk_over=0.51s
```

`seconds_to_yield=0.51s` and `talk_over=0.51s` are identical across runs. This
is the property a regression test needs: a red build means the audio changed,
not that the scorer drifted.

**What this job does NOT establish.** That 0.51s is the "true" yield latency in
some absolute sense, or that the reference thresholds are right for your product.
It establishes only that the measurement is stable and re-derivable by hand from
[`METHODOLOGY.md`](../METHODOLOGY.md).

---

## Job 2: candidate-discovery usefulness

**The question:** when Hotato scans a whole recording, does it surface the
moments a human reviewer would actually want to look at, ranked by salience?

**What is reported.** A ranked list of candidate turn-taking moments as **timing
facts only**: overlap onsets (caller became active while the agent was talking,
with the overlap length and whether the agent went silent), agent starts during
caller activity, and long response gaps. Each candidate is a timestamp and a
measurement. It is **not** a verdict, and it does not label intent.

```text
$ hotato scan --stereo 02-backchannel-mhm.example.wav --top 5
hotato scan: 02-backchannel-mhm.example.wav  (6.0s, 3 candidate moments)
Candidates are timing events. You decide the expected behavior; label with: hotato fixture create --onset <t> --expect yield|hold
  [ 1] t=2.09s  overlap_while_agent_talking  overlap=1.58s  agent did not go silent within 3.0s
  [ 2] t=3.19s  overlap_while_agent_talking  overlap=1.07s  agent did not go silent within 3.0s
  [ 3] t=4.29s  overlap_while_agent_talking  overlap=0.56s  agent did not go silent within 3.0s
```

The usefulness bar is **recall of human-notable moments at a workable candidate
count**, not precision against a ground-truth intent label (there is no such
label at scan time, by design). A candidate that turns out to be a harmless
backchannel is not a Hotato error: it is a candidate you label `hold` and move
on. The validation artifact is the [trust gallery](TRUST-GALLERY.md), which
includes a deliberate false positive so you can see what an unhelpful candidate
looks like and why Hotato still surfaces it.

**What this job does NOT establish.** That every candidate is a real bug, or that
a quiet region is guaranteed clean. Scan widens the net; you make the call.

---

## Job 3: contract verification

**The question:** once you have labelled a moment's expected behavior
(`yield` = stop for the caller, `hold` = keep the floor through a backchannel),
does Hotato's PASS/FAIL verdict agree with that label on the audio, against an
explicit, portable, CI-enforced policy?

Today this job runs on a fixture (`hotato fixture create` / `hotato run`): a
labelled recording plus an explicit threshold policy, scored the same way on
every machine. The portable contract bundle (`hotato contract create` /
`hotato contract verify`, audio plus timing evidence plus trace evidence plus
label plus policy plus a CI command in one artifact) carries this exact job
forward into a single self-contained object once it ships; the verdict this
job validates does not change shape, only the artifact it travels in.

**What is reported.** Per fixture: the verdict (`PASS`/`FAIL`), the measured
signals behind it, and the named fix class when the failure maps cleanly to a
config family. Agreement is checked against **your** label, not against an
opaque key.

```text
$ hotato demo --no-open --format text
hotato demo: recorded calls a provider's default agent fails
hotato [suite] stack=generic offline=True
  0/2 events pass  (failed=2)
  [FAIL] fd-01-missed-interruption: did_yield=False seconds_to_yield=- talk_over=0.25s
         fix[config]: Missed interruption: the agent kept talking over the caller
  [FAIL] fd-02-backchannel-yielded: did_yield=True seconds_to_yield=0.34s talk_over=0.32s
         fix[engagement-control]: False barge-in: a backchannel was treated as a bid for the floor
  note: no single sensitivity threshold satisfies this battery
  exit_code=1
```

Both labelled failures are caught, and the battery-level note refuses to name one
threshold when a missed interruption and a false stop fail in the same run. That
refusal is part of the validated behavior: Hotato reports the disagreement
instead of inventing a fix.

**What this job does NOT establish.** That the label was correct (you own the
label), or that a passing fixture means the agent is good in general. It
establishes that the verdict follows the audio and the label consistently.

---

## What we do not claim

Read this as a hard boundary, not a disclaimer.

- **No semantic intent.** Hotato measures timing. It does not know whether a
  caller sound meant "stop" or "mhm, go on." You supply that as a label.
- **No root-cause certainty.** A slow yield can be TTS buffering, transport, or
  VAD. `diagnose` names a likely layer and stays `unknown_root_cause` when one
  recording cannot separate them. A voice trace (once the trace layer ships)
  will narrow the candidates further; it will not convert a candidate into a
  proof.
- **No task success.** Whether the call booked the appointment, resolved the
  ticket, or satisfied the caller is out of scope. Use a QA platform for that
  (see [COMPARE.md](COMPARE.md)).
- **No vendor ranking.** Hotato never scores one platform against another. A
  provider-default example demonstrates the threshold funnel on one assistant,
  one config, one date, one scripted caller. It is not a benchmark of the vendor.
- **No human satisfaction or sentiment.** Hotato has no opinion on tone or CSAT.
- **No single accuracy score.** There is no headline percentage anywhere in
  Hotato, on purpose. The three jobs above are the whole claim.

The validation plan for the launch battery (external testers, consented
fixtures, before/after) lives in
[docs/evidence/validation-plan.md](evidence/validation-plan.md).


================================================================================
FILE: src/hotato/schema/envelope.v1.json
================================================================================

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://hotato.dev/schema/envelope.v1.json",
  "title": "hotato result envelope, schema_version 1",
  "description": "The single machine-readable shape emitted by the CLI, the one MCP tool, and the suite runner. Additive-only: new signal dimensions and sub-verdicts may appear as extra keys without a version bump, so consumers must ignore unknown fields. The stable core below is frozen for schema_version 1.",
  "type": "object",
  "required": ["tool", "schema_version", "mode", "stack", "offline", "engine", "limits", "summary", "events", "fix_map", "funnel", "exit_code"],
  "additionalProperties": true,
  "properties": {
    "tool": { "const": "hotato" },
    "schema_version": { "const": "1" },
    "mode": { "enum": ["single", "suite"] },
    "suite": { "type": "string" },
    "stack": { "type": "string" },
    "offline": { "const": true },
    "exit_code": { "enum": [0, 1] },
    "engine": {
      "type": "object",
      "required": ["name", "version", "upstream"],
      "additionalProperties": true,
      "properties": {
        "name": { "type": "string" },
        "version": { "type": "string" },
        "upstream": { "type": "string" }
      }
    },
    "limits": {
      "type": "object",
      "required": ["method", "accuracy_claim", "ceiling", "best_input", "does_not_do", "scope", "offline"],
      "additionalProperties": true,
      "properties": {
        "method": { "type": "string" },
        "accuracy_claim": { "type": "null", "description": "HONESTY INVARIANT: always null. This tool never emits an accuracy percentage." },
        "ceiling": { "type": "string" },
        "best_input": { "type": "string" },
        "does_not_do": { "type": "array", "items": { "type": "string" } },
        "scope": { "type": "string" }
      }
    },
    "summary": {
      "type": "object",
      "required": ["events", "passed", "failed", "regression"],
      "additionalProperties": true,
      "properties": {
        "events": { "type": "integer", "minimum": 0 },
        "passed": { "type": "integer", "minimum": 0 },
        "failed": { "type": "integer", "minimum": 0 },
        "regression": { "type": "boolean" },
        "not_scorable": { "type": "integer", "minimum": 1, "description": "Additive: count of events that could not be judged (malformed input, see each event's not_scorable_reason). Present ONLY when at least one event is not scorable; a run of valid recordings emits the exact pre-existing summary shape. Not-scorable events count in events but in neither passed nor failed." }
      }
    },
    "events": {
      "type": "array",
      "items": { "$ref": "#/definitions/event" }
    },
    "fix_map": {
      "type": "array",
      "items": { "$ref": "#/definitions/fix_map_entry" }
    },
    "funnel": {
      "oneOf": [
        { "type": "null" },
        {
          "type": "object",
          "required": ["reason", "pointer"],
          "additionalProperties": true,
          "properties": {
            "reason": { "type": "string" },
            "pointer": { "$ref": "#/definitions/engagement_pointer" }
          }
        }
      ]
    }
  },
  "definitions": {
    "event": {
      "type": "object",
      "required": ["event_id", "expected_yield", "verdict", "measurements"],
      "additionalProperties": true,
      "dependencies": {
        "scorable": ["not_scorable_reason"],
        "not_scorable_reason": ["scorable"]
      },
      "properties": {
        "event_id": { "type": "string" },
        "scorable": { "const": false, "description": "Additive: present ONLY on events that cannot be judged (and then always false); scorable events omit the key entirely, so every envelope for a valid recording is byte-identical to schema_version 1 before this key existed. A not-scorable event is excluded from passed/failed counting and can never carry a normal pass or fail." },
        "not_scorable_reason": { "type": "string", "description": "Plain-language reason the event cannot be judged (for example no detectable caller speech with no onset provided, or the agent silent at the caller onset under a should-yield expectation). Present exactly when scorable is present." },
        "scenario_id": { "type": ["string", "null"] },
        "title": { "type": ["string", "null"] },
        "category": { "type": ["string", "null"] },
        "expected_yield": { "type": "boolean" },
        "verdict": {
          "type": "object",
          "required": ["passed", "did_yield", "seconds_to_yield", "talk_over_sec", "reasons"],
          "additionalProperties": true,
          "properties": {
            "passed": { "type": "boolean" },
            "did_yield": { "type": "boolean" },
            "seconds_to_yield": { "type": ["number", "null"] },
            "talk_over_sec": { "type": "number" },
            "reasons": { "type": "array", "items": { "type": "string" } }
          }
        },
        "measurements": { "type": "object", "additionalProperties": true },
        "signals": {
          "type": "object",
          "additionalProperties": true,
          "description": "Namespaced signal bus. barge_in is present whenever scoring ran; other dimensions (latency, and future overlap/resume/backchannel) are additive.",
          "properties": {
            "barge_in": {
              "type": "object",
              "required": ["did_yield", "time_to_yield_sec", "talk_over_sec"],
              "additionalProperties": true
            },
            "latency": {
              "type": "object",
              "additionalProperties": true,
              "properties": {
                "response_gap_sec": { "type": ["number", "null"] },
                "premature_start_sec": { "type": ["number", "null"] }
              }
            },
            "resume": {
              "type": "object",
              "description": "Post-yield behaviour on the agent VAD track; present only when the agent yielded. resumed: did the agent start speaking again within the window after the yield. resume_gap_sec: seconds from the yield to that fresh onset (null when it did not resume). restart_suspected: the longest post-resume run is unusually long, a timing proxy for re-answering from the top (a transcript repeat is out of scope).",
              "additionalProperties": true,
              "properties": {
                "resumed": { "type": "boolean" },
                "resume_gap_sec": { "type": ["number", "null"] },
                "restart_suspected": { "type": "boolean" }
              }
            }
          }
        },
        "fix": {
          "oneOf": [ { "type": "null" }, { "$ref": "#/definitions/fix" } ]
        }
      }
    },
    "fix": {
      "type": "object",
      "required": ["fix_class", "title", "detail", "knob", "pointer"],
      "additionalProperties": true,
      "properties": {
        "fix_class": { "enum": ["config", "engagement-control"] },
        "title": { "type": "string" },
        "detail": { "type": "string" },
        "knob": {
          "oneOf": [
            { "type": "null" },
            {
              "type": "object",
              "required": ["stack", "parameter", "direction", "trade_off"],
              "additionalProperties": true
            }
          ]
        },
        "pointer": {
          "oneOf": [ { "type": "null" }, { "$ref": "#/definitions/engagement_pointer" } ]
        }
      }
    },
    "fix_map_entry": {
      "type": "object",
      "required": ["event_id", "fix_class", "title", "detail", "knob", "pointer"],
      "additionalProperties": true,
      "properties": {
        "event_id": { "type": "string" },
        "scenario_id": { "type": ["string", "null"] },
        "fix_class": { "enum": ["config", "engagement-control"] },
        "title": { "type": "string" },
        "detail": { "type": "string" }
      }
    },
    "engagement_pointer": {
      "type": "object",
      "description": "VENDOR-NEUTRAL pointer for the engagement-control fix class. Names the problem class and the KIND of fix (a learned engagement-control / addressee-detection layer); names no vendor, no product, and nothing you can adopt/license/buy, and carries no numbers.",
      "required": ["layer", "what", "honest_scope"],
      "additionalProperties": true,
      "properties": {
        "layer": { "type": "string" },
        "what": { "type": "string" },
        "honest_scope": { "type": "string" }
      }
    }
  }
}
