================================================================================
FILE: README.md
================================================================================

<p align="center">
  <a href="https://hotato.dev">
    <img src="https://raw.githubusercontent.com/attenlabs/hotato/main/.github/banner.png" alt="hotato: find where your voice agent talks over callers, and keep it from coming back" width="840">
  </a>
</p>

<h1 align="center">
  <img src="https://raw.githubusercontent.com/attenlabs/hotato/main/docs/assets/mascot.svg" alt="" width="26" align="top"> hotato
</h1>

<p align="center"><b>Find where your voice agent talks over callers, and keep it from coming back.</b></p>

<p align="center">Offline regression tests from your own call recordings. MIT.</p>

<p align="center">
  <a href="https://pypi.org/project/hotato/"><img alt="PyPI version" src="https://img.shields.io/pypi/v/hotato.svg"></a>
  <a href="https://pypi.org/project/hotato/"><img alt="PyPI monthly downloads" src="https://img.shields.io/pypi/dm/hotato.svg"></a>
  <a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-blue.svg"></a>
  <img alt="Python 3.9 to 3.13" src="https://img.shields.io/badge/python-3.9%20to%203.13-blue.svg">
  <img alt="offline: yes" src="https://img.shields.io/badge/offline-yes-blue.svg">
  <img alt="runtime deps: zero" src="https://img.shields.io/badge/runtime%20deps-zero-blue.svg">
  <a href="https://github.com/attenlabs/hotato/actions/workflows/tests.yml"><img alt="tests" src="https://github.com/attenlabs/hotato/actions/workflows/tests.yml/badge.svg"></a>
</p>

<p align="center">
  <img src="https://raw.githubusercontent.com/attenlabs/hotato/main/docs/assets/hotato-demo.gif" alt="Terminal recording of hotato scan, run, and fixture create on a recorded call" width="760">
</p>

One command finds every moment your agent talks over the caller:

```bash
uvx hotato scan --stereo your-call.wav   # two-channel WAV: caller ch0, agent ch1
```

From there: `run` scores a call to a PASS/FAIL verdict and an HTML report, `compare` measures a before/after delta for one fixed moment (fixed/regressed/improved/worse), `verify` proves it at battery scale across every fixture (reporting coincidence, not causation), and `fixture create` saves any moment as a permanent regression test that fails CI if it comes back. Everything runs on your machine; the audio never leaves it.

Hotato catches the three talk-over failures callers feel: the agent talking over the caller, false-stopping on a backchannel ("mhm"), or yielding too slowly. You label the expected behavior (`yield` = stop for the caller, `hold` = keep talking through a backchannel); Hotato measures whether the timing matched. It reports what it measured, never a guess at intent.

Try it with no audio of your own:

```bash
uvx hotato demo   # scores two real recorded calls a provider's default agent failed, so you see the FAILs, timelines, and fix cards
```

## MCP

```bash
uvx --from "hotato[mcp]" hotato-mcp   # one tool, voice_eval_run; client configs in docs/MCP.md
```

## Install

`uvx hotato` runs any command with zero install. To add it to a project:

```bash
pip install hotato                 # core: stdlib-only, zero dependencies
pip install 'hotato[neural]'       # optional Silero VAD cross-check
pip install 'hotato[livekit]'      # LiveKit live capture
pip install 'hotato[pipecat]'      # Pipecat live capture
```

## Depth

- **Bad call to CI regression test**, step by step: [`docs/BAD-CALL-TO-CI.md`](docs/BAD-CALL-TO-CI.md) · runnable [`examples/bad-call-to-ci/`](examples/bad-call-to-ci/README.md)
- **What it measures** (the three timing signals, re-derivable by hand): [`METHODOLOGY.md`](METHODOLOGY.md) · Python API [`docs/API.md`](docs/API.md)
- **The fix ladder** (each failure names the setting to move, in your stack's own terms): [`docs/FIX-PLANS.md`](docs/FIX-PLANS.md)
- **Rule out the non-turn-taking bugs first** (STT, buffering, verbosity, refusals, wrong-language): [`docs/WHY.md`](docs/WHY.md)
- **Pull a call from your stack** (Vapi, Twilio, Retell, LiveKit, Pipecat): [`adapters/README.md`](adapters/README.md) · status [`docs/ADAPTER-STATUS.md`](docs/ADAPTER-STATUS.md)
- **CI gates**: GitHub Action [`docs/CI.md`](docs/CI.md) · pytest plugin [`docs/PYTEST.md`](docs/PYTEST.md)
- **Reports and analytics**: [`docs/REPORTS.md`](docs/REPORTS.md) · Suites [`docs/SUITES.md`](docs/SUITES.md) · Stack benchmarks [`docs/BENCHMARK-STACKS.md`](docs/BENCHMARK-STACKS.md)
- **Real battery**: 12 scripted calls against a live voice agent on its provider's default settings, where a missed interruption and a false stop on a backchannel fail in the same run, so `diagnose` refuses to name one threshold: [`corpus/vapi-defaults/README.md`](corpus/vapi-defaults/README.md)
- **For agents**: [`llms.txt`](llms.txt) · [`llms-full.txt`](llms-full.txt) · MCP server [`docs/MCP.md`](docs/MCP.md) · Security [`SECURITY.md`](SECURITY.md)
- **Contributing**: the highest-value PR is a real, labelled call fixture: [`docs/SUBMITTING.md`](docs/SUBMITTING.md)

Why "hotato": good turn-taking is a game of hot potato. Speak, then pass the turn the moment the caller wants it. MIT licensed ([`LICENSE`](LICENSE)); the open core stays open.

mcp-name: io.github.attenlabs/hotato


================================================================================
FILE: docs/WHY.md
================================================================================

# Why Hotato

## Voice agents fail in ways your tests do not catch

Your test suite checks what the agent says. Production calls fail on when it
speaks. Four timing failures show up again and again in real transcripts, and
all four are invisible to text-level tests:

1. **Missed interruption.** The caller says "stop, take that off" and the agent
   keeps talking. `did_yield` is false where it must be true. The caller
   repeats themselves, louder, then hangs up.
2. **False stop.** The caller says "mhm", a backchannel: a short acknowledgement,
   not a request to take over. The agent stops mid-sentence anyway. `did_yield`
   is true where it must be false. The call stalls, then restarts, then stalls
   again.
3. **Slow yield.** The caller interrupts and the agent does stop, eventually.
   Every second of `talk_over_sec` before it stops is the caller and the agent
   speaking at once, and the caller hears all of it.
4. **Endpointing misses.** Endpointing is detecting that the caller finished
   speaking. When it misses, the caller gets dead air after they finish
   (`response_gap_sec`), or the agent starts before they are done
   (`premature_start_sec`). Both read as a broken conversation partner.

Each of these lives entirely in the audio timing of the call. A transcript
diff, an LLM judge on text, and a unit test on the agent's reply all score a
call with any of these failures as perfect.

Hotato does not infer intent. You label the expected behavior for the event:
yield means the agent should stop for the caller. hold means the agent should
keep speaking through a backchannel/noise/acknowledgement. Hotato then
measures whether the timing matched that label.

## Is this even a turn-taking bug?

In our observed reports, many alleged barge-in bugs turn out not to be
turn-taking bugs at all. Each of these produces a symptom that reads exactly like a
missed interruption or a false stop, but the fix lives in a different layer,
and no VAD threshold will ever touch it:

- **STT hallucination.** The transcript has words the caller never said, or
  drops words that mattered, so the agent responds to something that was not
  actually said. That looks like an interruption the agent ignored. Reach
  for: an STT/ASR word-error-rate check against the raw audio, not a
  turn-taking scorer.
- **Client-side audio buffering.** The caller's own device or browser queues
  outgoing audio before it reaches the agent, so "the agent talked over me"
  is audio arriving late, not the agent failing to yield. Reach for:
  client/WebRTC jitter-buffer and network-latency instrumentation, not an
  interruption-sensitivity setting.
- **LLM verbosity or tool-selection.** The agent "kept talking through the
  interruption" because it was mid-tool-call or committed to finishing a long
  generation before it re-checked for a stop signal. Reach for: response-length
  and tool-call latency tracing in your agent framework, not a VAD setting.
- **Safety false-refusal.** The agent stops abruptly mid-sentence because a
  moderation or safety layer cut it off, not because it heard a barge-in or a
  backchannel. Timing-wise this is indistinguishable from a false stop on
  "mhm". Reach for: your safety/moderation logs, not an engagement-control
  layer.
- **Wrong-language STT.** The caller is speaking a language or accent the STT
  covers poorly, recognition comes back empty or garbled, and the agent's
  response looks unrelated or missing entirely. That reads as a missed
  interruption; it is a language-coverage gap. Reach for: per-locale STT
  accuracy tooling. (Hotato's own detector is energy-based and does not
  detect language either, by design: see `corpus/classes/README.md`.)

This is both an honesty guard and a shortcut. If your bug matches one of
these five, Hotato will not find it no matter how you tune it, because it is
not a timing bug; you will save the day you would have spent staring at
`turn_end_silence_sec`. If it matches none of them: two common complaints are
agent-talks-over-caller and false-stop-on-backchannel, and that is exactly what
Hotato measures, with the funnel (no single config value fixes both directions
at once) proven on real recorded calls, not synthetic fixtures
(`corpus/vapi-defaults/README.md`).

## What makes Hotato different

- **It scores recordings you already have.** A dual-channel WAV from your
  stack is the whole input. No test harness, no synthetic caller, no
  re-architecture.
- **It runs offline.** Scoring is local and deterministic; no audio,
  transcript, or result leaves your machine.
- **It emits machine-readable timing.** One JSON envelope per run, the same
  shape from the CLI, the MCP tool, and the pytest fixture, so an agent or a
  CI job consumes one schema.
- **It fails CI.** Exit code 1 on a regression, 0 on pass, 2 on a usage error
  or a recording that is not scorable: the recording cannot answer the
  question (the caller channel is silent, or the agent was not talking when
  the caller started), so no verdict is given. A turn-taking regression blocks
  a merge the same way a failing unit test does.
- **It routes every failure to a fix class.** When the failure maps cleanly to
  stack config, `config` names the setting family on your stack and the
  direction to investigate. `engagement-control` tells you
  no threshold value can fix this failure, because telling "mhm" apart from
  "stop" is a classification problem, so you stop burning days retuning a
  setting that cannot win.

## What it does not do

No transcription. No speaker identification or diarization. No emotion or
intent detection. Hotato measures energy over time on two channels; the method
and its ceiling are stated in [METHODOLOGY.md](../METHODOLOGY.md) and in the
`limits` block of every result.

## Why no accuracy score?

Because accuracy would hide the thing you need to debug. A single blended
percentage tells you nothing about which call failed, on which axis, or what
to change. Hotato reports three direct measurements per event instead:

- `did_yield`: did the agent stop talking after the caller started
- `seconds_to_yield`: seconds between the caller starting and the agent stopping
- `talk_over_sec`: seconds the caller and the agent spoke at the same time

Every number is reproducible from the frame dump by hand, every threshold is
exposed, and a failure points at its fix. That is what you debug with.

Ready to pin a real failure? The loop from one bad call to a CI gate, with
`hotato fixture create`, `compare`, and `plan`:
[BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md).


================================================================================
FILE: METHODOLOGY.md
================================================================================

# Methodology

How every number this tool reports is computed, why it is reproducible, and
where the method stops being trustworthy. There is no accuracy percentage in
this document and none is implied anywhere in the tool. What follows describes a
measurement, not a claim about any detector's internal quality.

Hotato scores the audio *timing* of turn-taking from a call recording: whether
the agent stopped talking once the caller started (a yield), how many seconds
that took, how many seconds it kept talking while the caller was talking
(talk-over), and, on the same two tracks, the endpointing timing: how the
agent's response sits around the moment the caller finished speaking. It
measures energy over time. It does not understand speech.

## Read this first: the honest ceiling

The limits come first because they bound everything below.

- **Energy is not intent.** The detector marks a frame "active" when its
 short-time energy crosses a threshold. A cough, a slammed door, or a burst of
 line noise reads as active exactly like a word does. Nothing here recovers
 *meaning*, it recovers *when there was speech-level energy*.
- **You supply the label.** Hotato does not infer intent. You label the
 expected behavior for the event: yield means the agent should stop for the
 caller. hold means the agent should keep speaking through a
 backchannel/noise/acknowledgement. Hotato then measures whether the timing
 matched that label. "mhm" and "stop" can carry identical speech energy; the
 verdict is only ever your label checked against measured timing.
- **Sub-second single-channel boundaries sit near the resolution limit.** At the
 default 10 ms hop a boundary is located to within a frame or two, but the
 hangover that keeps an utterance whole (default 150 ms) smears every trailing
 edge by up to that much, and on one mixed channel deciding *whose* energy
 crossed the threshold is not always possible.
- **Two channels is the required input.** Hotato's main scorer requires
 separated caller and agent tracks: either one two-channel WAV or two aligned
 mono WAVs. A single mixed mono call is not enough to attribute talk-over
 reliably: with both voices summed into one waveform, the scorer cannot
 attribute energy to a speaker. With physically separate channels, overlap is
 a fact of the recording: both tracks are active at once, by construction.
- **Out of scope, permanently:** no speaker identification, no diarization, no
 speech-to-text, no emotion or intent detection, and no claim about any
 vendor's internal accuracy. Word-level semantics are not measured; only the
 timing of the yield is.

These same limits are emitted inline in the `limits` block of every JSON result
(`core.py`) and in the MCP tool description, so a consuming agent sees them.

## The pipeline, step by step

Everything runs offline, standard-library only (NumPy only accelerates the
per-frame RMS when importable; results are identical without it). Scoring is
deterministic: the same audio and `ScoreConfig` give the same output.

### Default parameters

Every value below is read directly from `ScoreConfig` (`score.py`) and
`VADParams` (`vad.py`); the same defaults apply to both channels unless overridden.

Framing and VAD (`VADParams`, applied per channel):

| Parameter | Default | Meaning |
|--------------------|-----------|---------|
| `frame_ms` | 20.0 ms | analysis window length for one RMS frame |
| `hop_ms` | 10.0 ms | step between frames (the time resolution) |
| `noise_percentile` | 0.10 | quietest 10% of frames estimates the noise floor |
| `rel_db` | 15.0 dB | a frame is active this many dB above the floor |
| `abs_gate_db` | -60.0 dBFS| nothing below this absolute level is ever active |
| `dyn_margin_db` | 22.0 dB | on a rarely-silent channel, hold the threshold at least this far below the loudest frames |
| `hangover_sec` | 0.15 s | stay "active" this long after energy drops |

Signal windows (`ScoreConfig`):

| Parameter | Default | Meaning |
|---------------------------|---------|---------|
| `yield_hangover_sec` | 0.20 s | agent must stay quiet this long to count as yielded |
| `max_search_sec` | 3.0 s | how long after onset a yield is searched for |
| `caller_proximity_sec` | 0.5 s | a yield only counts if the caller held the floor within this window of it |
| `turn_end_silence_sec` | 0.20 s | caller must stay quiet this long for the turn to count as ended |
| `premature_tolerance_sec` | 0.05 s | agent may lead the caller's turn end by up to this before it counts as premature |
| `onset_min_run_sec` | 0.05 s | minimum sustained active run for the caller onset to count |
| `agent_onset_lookback_sec`| 0.10 s | window before onset checked for whether the agent was already talking |

Both onset-detection windows are exposed on `ScoreConfig` as well:
`onset_min_run_sec` (the `0.05 s` sustained active run a caller onset must
clear, applied by `first_active_sec`) and `agent_onset_lookback_sec` (the
`0.10 s` window before onset checked for "was the agent talking at onset?").
Neither affects the per-frame active/threshold decision the frame dump records.
Every threshold the scorer uses is a named, overridable parameter -- there are
no hidden constants in the scoring path.

### Step 1, per-frame RMS

Each channel is cut into 20 ms windows stepped by 10 ms. For each window the
linear root-mean-square amplitude is computed (`frame_rms`, `audio.py`). At
16 kHz this is a 320-sample window with a 160-sample hop, so `hop_sec` is
exactly `160 / 16000 = 0.01 s`; it is derived from the integer sample hop over
the sample rate, so it matches `hop_ms` exactly at common rates.

### Step 2, RMS to dBFS

Each linear RMS value is converted to decibels relative to full scale:
`dBFS = 20 * log10(rms)`, with a floor of `-120.0 dBFS` substituted before the
log to avoid `log(0)` (`to_dbfs`, `audio.py`). This is the per-frame `*_dbfs`
column in the frame dump.

### Step 3, per-channel energy VAD

For each channel independently (`energy_vad`, `vad.py`):

1. Sort the frame dBFS values and take the `noise_percentile` (10th percentile)
 as the **noise floor**.
2. The base **threshold** is `max(noise_floor + rel_db, abs_gate_db)`, i.e.
 15 dB above the floor, but never below the -60 dBFS absolute gate.
3. **Dynamic-margin guard:** if a channel is almost never silent (e.g. an agent
 talking the whole clip), the 10th-percentile floor lands *inside* speech and
 would push the threshold above the speech itself. So compute
 `cap = max(dBFS) - dyn_margin_db`; if `cap` is above the absolute gate, lower
 the threshold to `min(threshold, cap)`. This cannot rescue a genuinely silent
 channel, because the guard only fires when loud content exists above the gate.
4. A frame is **raw-active** when its dBFS ≥ the threshold.
5. **Hangover:** after any raw-active frame, keep the channel active for
 `round(hangover_sec / hop_sec)` more frames, 15 frames (150 ms) at the
 defaults, so brief inter-word gaps do not fragment one utterance.

The resulting threshold and noise floor are constant across frames for a given
channel and are recorded verbatim in the frame dump.

### Step 4, caller onset

If a caller onset label is supplied (scenario `caller_onset_sec`, or `--onset`),
it is used directly. Otherwise the onset is the start of the caller's first
sustained active run, the first run of at least 0.05 s of active frames
(`first_active_sec`). The label is preferred because on mixed or noisy audio the
first energy is not always the caller. `onset_idx` is `round(onset / hop_sec)`,
clamped into range. No label and no detectable caller speech means there is no
caller event to score: the recording is reported not scorable (`scorable: false`
with a plain reason), never scored against frame 0, and a single-recording run
exits 2.

### Step 5, the three barge-in signals

Computed in `score_channels` (`score.py`):

- **`agent_talking_at_onset`**, was any agent frame active in the 0.10 s up to
 and including onset? If not, on a should-yield expectation there is nothing to
 yield: the event is reported not scorable (`scorable: false`, with the reason
 in `not_scorable_reason`) rather than passed or failed, and a single-recording
 run exits 2. The input is what is wrong (onset time, channel mapping, or
 expectation), and it is reported as such.
- **`did_yield`**, scanning from onset up to `onset + max_search_sec` (300
 frames), the first frame where the agent goes quiet and *stays* quiet for
 `yield_hangover_sec` (20 frames) **and** the caller held the floor within
 `caller_proximity_sec` (±50 frames) of that quiet point. The proximity
 condition is what stops an agent that merely finishes its own sentence seconds
 after an isolated backchannel from being scored as a barge-in response.
- **`time_to_yield_sec`**, `(yield_frame - onset_frame) * hop_sec`, floored at
 0; `None` if the agent never yielded within the search window.
- **`talk_over_sec`**, count of frames from onset to the yield point (or to the
 end of the search window if it never yielded) where the caller and the agent
 were *both* active, times `hop_sec`.

### Step 6, the two latency (endpointing) signals

Pure timing on the same two VAD tracks, no second model (`score.py`):

- **`caller turn end`**, the first frame at/after onset where the caller goes
 quiet and stays quiet for `turn_end_silence_sec` (20 frames). `None` if the
 caller never activates after onset or is still talking when the clip ends.
- **`agent response onset`**, skip any agent speech already in progress at
 onset (the pre-caller turn or the yield tail), then the first frame the agent
 becomes active again. `None` if the agent never starts a fresh run.
- With both defined, `lead = turn_end_frame - response_onset_frame` (positive
 when the agent starts *before* the caller finishes). If `lead` exceeds
 `premature_tolerance_sec` (5 frames), `premature_start_sec = lead * hop_sec`
 and `response_gap_sec` is null. Otherwise `premature_start_sec = 0.0` and
 `response_gap_sec = max(0, response_onset - turn_end) * hop_sec`. Both are
 null when not derivable from the tracks, never fabricated. Reported values
 are rounded to the millisecond (3 decimal places of seconds).

### Step 7, two additive, opt-in cross-checks: echo and resume

Both live entirely in hotato's own layer (`echo.py`, `resume.py`), never in
the vendored `_engine`, and both add an optional `signals` block without
changing `did_yield`, `seconds_to_yield`, or `talk_over_sec` for any existing
recording.

- **`signals.echo`** (`echo.py`), computed on every scored event. Leaked TTS
 (an agent that hears its own audio back on the caller channel) is, by
 construction, a delayed, scaled copy of the agent's own envelope. The check
 is a deterministic cross-channel coherence: at each lag from 0 up to
 `DEFAULT_MAX_LAG_SEC` (0.5 s), take the cosine similarity between the caller
 frame-RMS envelope and the agent envelope shifted by that lag; `coherence`
 is the best (highest) cosine found, `lag_sec` is the lag it occurred at, and
 `echo_suspected` is `coherence >= DEFAULT_COHERENCE_THRESHOLD` (0.7) with at
 least `_MIN_OVERLAP_FRAMES` (8) of overlap. Independent speech (real
 turn-taking, backchannels) does not correlate with the agent's own envelope
 at any lag, so it scores low. `--echo-gate` (opt-in, `hotato run`) holds an
 echo-suspected yield out of the verdict (`scorable: false`) instead of
 counting it as a clean pass; without the flag the primary verdict is
 unchanged and `signals.echo` is still reported. `hotato diagnose` and the
 single-run text output print a WARNING for every echo-suspected yield.
- **`signals.resume`** (`resume.py`), present only on events where the agent
 yielded. From the agent's own VAD track, look for a fresh onset within
 `DEFAULT_RESUME_WINDOW_SEC` (4.0 s) after the yield: `resumed` is whether one
 is found, `resume_gap_sec` is the seconds from yield to that onset (`null`
 if it did not resume). `restart_suspected` flags whether the longest
 contiguous agent run at or after the resume onset reaches
 `DEFAULT_RESTART_MIN_SEC` (2.0 s), the timing fingerprint of re-answering a
 whole paragraph from the top rather than finishing the interrupted clause.
 Whether the resumed words literally repeat the earlier ones is a transcript
 question, explicitly out of scope: `restart_suspected` is a run-length
 heuristic on timing, not a text diff.
- **`agent_stop_no_caller`** (`scan.py`, not part of the scored envelope: a
 `hotato scan` candidate) surfaces the companion timing fact: the agent went
 from active to quiet with zero caller energy anywhere in
 `caller_proximity_sec` on either side, so nothing on the caller channel
 explains the drop. It is a candidate for self-truncation, endpointing
 mis-fire, or an upstream (LLM/TTS) cutoff, not a verdict; label it with
 `hotato fixture create --expect hold` (or `yield`, if that silence was in
 fact correct) to turn it into a scored fixture.

## Why it is reproducible

- **Deterministic** given `(audio, ScoreConfig)`. No randomness, no network, no
 hidden state. The bundled fixtures are themselves rendered deterministically
 from a seed derived from `sha256(scenario_id)` (`render_examples.py`), and CI
 re-renders and diffs them to prove byte-stability.
- **Every threshold that drives the active/threshold decision is a named,
 overridable parameter** on `ScoreConfig` / `VADParams`, and the resolved
 values are echoed back in the frame dump's `config` block.
- **Every frame and every derived index is inspectable.** `--dump-frames PATH`
 writes, per frame: `t_sec`, `caller_dbfs`, `agent_dbfs`, `caller_active`,
 `agent_active`, and the constant `caller_threshold_db`,
 `caller_noise_floor_db`, `agent_threshold_db`, `agent_noise_floor_db`
 (`frame_dump`, `score.py`). Nothing the scorer decides is off the record.

### Worked example: re-deriving `did_yield` and `talk_over` by hand

The excerpt below is illustrative of the dump shape; run `--dump-frames` on your
own file for real values. Assume the defaults (`hop_sec = 0.01`), a caller onset
label at `2.40 s` (`onset_frame = 240`), and these constant thresholds from the
dump header: caller threshold `-55.0 dBFS`, agent threshold `-52.0 dBFS`.

```
 frame t_sec caller_dbfs caller_active agent_dbfs agent_active
 240 2.40 -18.4 True -21.1 True
 288 2.88 -19.0 True -20.7 True
 289 2.89 -20.2 True -48.9 True (hangover tail)
 290 2.90 -21.5 True -71.3 False
 ... (agent dBFS stays below -52 through frame 310) ...
 310 3.10 -20.8 True -69.4 False
```

Re-derive `did_yield`: agent was active at onset (frame 240), so the question is
meaningful. Scanning from 240, the agent first drops below its threshold at
frame 290 and stays below it for at least 20 consecutive frames (through 310), 
that satisfies `yield_hangover_sec`. The caller is active within ±50 frames of
290, so the proximity condition holds. Therefore `did_yield = True`,
`yield_frame = 290`, and `time_to_yield_sec = (290 - 240) * 0.01 = 0.50 s`.

Re-derive `talk_over`: count frames from 240 up to the yield frame 290 where
`caller_active AND agent_active`. In this excerpt that is frames 240 through 289
inclusive, 50 frames, so `talk_over_sec = 50 * 0.01 = 0.50 s`. Every one of
those booleans is just `dbfs >= threshold` (plus hangover), which you can
recompute yourself from the two dBFS columns and the header thresholds.

## Aggregate statistics: one definition each

The `report`, `team`, and `export` surfaces summarize many events with mean,
median, p90, and p95. All four come from one stdlib implementation
(`src/hotato/_stats.py`, `dist_summary`) so every published aggregate is
re-derivable by hand:

- **mean** is the arithmetic mean (`statistics.fmean`).
- **median** is `statistics.median` (p50).
- **p90** and **p95** are linear interpolation between closest ranks (the
 definition NumPy calls "linear"): sort the values ascending, let
 `pos = q * (n - 1)` for `q` in `{0.90, 0.95}`, then
 `p_q = v[floor(pos)] + frac * (v[floor(pos) + 1] - v[floor(pos)])`.
- **Rates are fractions**, never a percentage: a pass rate of 8 of 8 reads
 `1.00`. That keeps every aggregate in the same unit system as the
 measurements and leaves no door open to an accuracy-percentage reading.
- **Empty input returns null.** No aggregate is ever fabricated from zero
 measurements, and fewer than two runs is stated plainly rather than drawn as
 a trend.

`team` and `export` pool `response_gap_sec` (dead air before the agent
speaks, defined above) across every scored event into the same `dist_summary`
shape already used for talk-over and time-to-yield. `--max-response-gap`
turns the pooled p95 into a latency SLA gate: the run exits 1 exactly when the
pooled p95 exceeds the bound, the same pass/fail contract as
`--max-talk-over` and `--max-time-to-yield`, just pooled instead of
per-event. A plain `hotato export` (no `--max-response-gap`) still writes a
byte-identical `envelope.json` to a run with none of this stage's work
applied; the p95 and gate live only in the printed summary and the returned
manifest.

## Optional neural cross-check (non-reference)

The energy VAD above is the **reference**: every published, golden, and bundled
number comes from it, deterministically. Hotato also ships an **optional,
opt-in, explicitly non-reference** neural VAD backend so you can re-run the *same*
turn-taking timing math over a learned speech track and compare, a direct answer
to "this is just energy VAD, rebuildable in a weekend."

- **How to use it.** Install the extra and pass the flag on your *own* recording:
 `pip install 'hotato[neural]'` then `hotato run --stereo call.wav --backend neural`.
 The default is `--backend energy`. The `--suite` self-test **always** scores
 with energy (it is the reference), so `--backend neural` is ignored there.
- **What it is.** Silero VAD (MIT), run locally/offline. The engine itself keeps
 zero third-party dependencies and never imports a model; the backend is injected
 behind one shared interface, so an open-weight turn-detector (e.g. LiveKit's
 Smart-Turn) can be plugged in the same way later. It returns the **identical**
 `VADResult` shape as the energy backend, `active`, `hop_sec`, `threshold_db`,
 `noise_floor_db`, aligned to the same hop grid. A neural model has no dB
 threshold, so `threshold_db` / `noise_floor_db` are **synthesized**: they are the
 energy-domain description of the *same* audio, reported for inspection only; the
 neural `active` decision is a learned speech probability, not a dB crossing.
- **What it changes, honestly.** It **tightens onset precision** on clean speech
 (a learned boundary can beat a fixed dB threshold on some audio). It does **not**
 close the energy-vs-intent gap: a cough, a laugh, a door slam, or crosstalk still
 carries speech-band energy, and *any* single-channel VAD, energy or neural, can
 mark it active. Whether a sound is a genuine bid for the turn is not decidable
 from one channel's activity alone. **No accuracy percentage is claimed for either
 backend.** Neural is a flagged cross-check, not a new source of truth.
- **No silent fallback.** If the `[neural]` extra is absent, `--backend neural`
 raises a clean, explicit error and exits non-zero, it never quietly scores with
 energy and presents it as the neural result.
- **Verification status: verified against the real model.** Executed in this
 repo with silero-vad 6.2.1 (ONNX weights bundled inside the package, inference
 through onnxruntime on CPU, fully offline; note silero-vad itself depends on
 torch, so the extra installs it). Properties that hold, measured over the
 bundled 8 and the 40-scenario gold suite:
 - **Contract holds end to end.** The real model returns the identical
   `VADResult` shape as energy, on the same hop grid, through the public scorer.
 - **Deterministic.** Two full sweeps over all 48 fixtures produced
   byte-identical JSON, for both backends; single recordings are likewise
   byte-identical run to run.
 - **Reference untouched.** With the extra installed, energy outputs are
   byte-identical to energy outputs without it, and the frozen bundled 8
   verdicts are unchanged.
 - **On the synthetic fixtures the two tracks diverge, and that divergence is a
   property of the fixtures.** The renders are shaped noise built for the
   energy reference; a speech-trained model assigns them near-zero speech
   probability (identical through the ONNX and torch paths), so at Silero's
   default threshold the neural track is empty there and the timing math
   reports no yield on any of the 48. This measures the fixtures, not any
   agent. Point the neural cross-check at real recordings.
 - **On a real-speech recording** (a dual-channel barge-in fixture assembled
   from a recorded human utterance) both backends returned the same yield
   verdict; the measured yield timing differed (energy 0.65 s including its
   0.15 s hangover, neural 0.18 s, because a speech model segments the word
   gaps that the energy hangover bridges). One recording, reported as a timing
   observation, not an accuracy claim.
 - **Sample rates.** Silero accepts 8000 Hz, 16000 Hz, and integer multiples of
   16000 Hz (silero-vad decimates those itself and returns timestamps in the
   original sample coordinates). Any other rate fails with an actionable
   resample message from the seam. The energy backend measures at any rate.
 The real-model tests run automatically when the `[neural]` extra is installed
 and skip cleanly when it is absent (`tests/test_backend.py`).

## How validity is measured, not claimed

The bundled fixtures are synthetic: band-limited, syllable-modulated noise
rendered from exact segment boundaries declared in each scenario's
`reference_render` block (`caller_segments_sec`, `agent_segments_sec`, and an
optional `continuous` flag that renders one unbroken run per segment so the
active track equals the rendered boundaries to within one hop). Because the true
boundaries are known exactly, the scorer's **own measurement error** is
computable, and that is what to report, rather than an accuracy score:

- For a timing signal (onset, yield time, response gap), report
 `|measured - true|` in **milliseconds**, per scenario. The true value comes
 straight from the segment boundaries: true onset is the caller's first segment
 start; true yield time is the agent's segment end minus onset; true gap is the
 agent's next segment start minus the caller's turn end. Expect a residual on
 the order of the hop plus the hangover smear on trailing edges, read it, do
 not average it away.
- For `did_yield`, report a **confusion matrix** against the scenario `category`
 labels (`should_yield` vs `should_not_yield`), four cells: correct yields,
 missed yields, correct holds, phantom yields. Keep the cells separate.

**Never aggregate these into a single accuracy percentage.** A missed hard
interruption and a phantom yield on a backchannel are different failures with
different fixes; one blended number hides exactly the distinction the tool
exists to surface.

State plainly what the synthetic battery is for: it proves the plumbing works
end-to-end and catches regressions (a threshold change that moves a number shows
up immediately against the frozen golden output). It is **not** production audio
, no real accents, codecs, packet loss, or room acoustics. For validity on your
system, bring **10-15 of your own labelled calls** (two-channel where you can
get it) and run them through the same `run` / `--dump-frames` path; see
`CONTRIBUTING.md` and `docs/CORPUS-GOVERNANCE.md`.

## How to verify a score on your own recordings

1. Run `--dump-frames out.json` on the recording. Read the per-channel
 `*_threshold_db` and `*_noise_floor_db` from the header, and confirm
 `threshold = noise_floor + 15 dB` (or the dynamic-margin cap, or the -60 dBFS
 gate) matches your levels.
2. Scan the `*_dbfs` and `*_active` columns around the reported onset, yield,
 and gap indices and confirm each `active` flag is just `dbfs >= threshold`
 carried forward by the 150 ms hangover. Re-derive the signal by hand as above.
3. If the VAD is mislabeling your audio, tune and watch the numbers move
 predictably: raise `rel_db` or `abs_gate_db` if line noise is marked active;
 lower `rel_db` if quiet speech is missed; raise `hangover_sec` if one
 utterance fragments into several; adjust `yield_hangover_sec` /
 `caller_proximity_sec` to match how long *your* callers pause. Re-dump and
 confirm the thresholds and active flags moved as expected.

If you disagree with a number, the frames are on the table and every threshold
is yours to change. That is the point: a measurement you can audit, not a
verdict you have to trust.


================================================================================
FILE: docs/BAD-CALL-TO-CI.md
================================================================================

# From a bad call to a CI gate

One bad call moment becomes a permanent, offline regression test in five
steps. Everything below runs locally; no audio leaves your machine.

## The label comes from you

Hotato does not infer intent. You label the expected behavior for the event:
yield means the agent should stop for the caller. hold means the agent should
keep speaking through a backchannel/noise/acknowledgement. Hotato then
measures whether the timing matched that label.

That is the whole contract. "mhm" and "stop" can carry identical speech
energy; no timing measurement can tell them apart. What Hotato can measure,
reproducibly, is whether the agent did what your label says it should have
done, and how many seconds it took.

Input requirement, stated once: Hotato's main scorer requires separated
caller and agent tracks: either one two-channel WAV or two aligned mono WAVs.
A single mixed mono call is not enough to attribute talk-over reliably.

## Step 1: turn the moment into a fixture

You have a recording where the agent talked over a caller at 42.18 seconds.
Label it:

```bash
hotato fixture create --stereo bad-call.wav \
    --id refund-interruption-001 \
    --onset 42.18 --expect yield \
    --max-talk-over 0.6 --max-time-to-yield 1.0 \
    --out tests/hotato
```

This writes `tests/hotato/scenarios/refund-interruption-001.json` (the label,
with provenance) and `tests/hotato/audio/refund-interruption-001.example.wav`
(a two-channel clip around the event, onset re-based to the clip). The
fixture is scored immediately on creation; an input that cannot be judged is
refused with the reason and exit code 2, never written silently.

Do not know the onset? List the candidate moments first:

```bash
hotato scan --stereo bad-call.wav
```

`scan` reports timing facts only (overlap onsets, agent starts during caller
speech, long response gaps, an agent going quiet with no caller energy nearby,
a caller run that correlates with the agent's own audio, i.e. suspected TTS
echo). You pick the moment and supply the label.

## Step 2: run it

```bash
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio
```

The bad take fails, by design: that is the regression you are pinning. Exit
codes are the contract: 0 all pass, 1 a regression, 2 unusable input.

## Step 3: fix your agent, then compare

Change the setting, re-capture the same scenario, and let the numbers speak:

```bash
hotato compare --before bad-call.wav --after new-take.wav \
    --onset 42.18 --expect yield
```

```
hotato compare: bad-call.wav -> new-take.wav
  verdict:           FAIL -> PASS
  did_yield:         false -> true
  seconds_to_yield:  - -> 0.42s
  talk_over_sec:     2.65s -> 0.42s  improved -2.23s

result: fixed
```

If the moment shifted between takes, pass `--before-onset` and
`--after-onset` separately. `--out report.html` writes the shareable HTML
report with the before take as the base. A side that cannot be judged renders
NOT SCORABLE and exits 2; no verdict is invented.

## Step 4: gate CI on it

```yaml
# .github/workflows/turn-taking.yml
name: turn-taking
on: [pull_request]
jobs:
  hotato:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install hotato
      - run: hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio --format json
```

`hotato run` exits 1 on any regression, so the job fails when the timing
regresses. Already running pytest? `pytest --hotato-suite
--hotato-suite-scenarios tests/hotato/scenarios --hotato-suite-audio
tests/hotato/audio` adds the same gate to the run you have (see
[PYTEST.md](PYTEST.md)). The richer PR check with a sticky results comment is
in [CI.md](CI.md).

## Step 5: when it fails, plan the fix

```bash
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio \
    --format json > result.json
hotato plan result.json --stack vapi --assistant-id <id>
```

`plan` is read-only and guarded: it proposes at most one bounded step on one
setting, refuses to tune a single threshold when the battery fails on both
axes at once, and downgrades to a checklist when the evidence cannot isolate
the layer. It never applies anything (`platform_mutation.performed` is always
false). Details: [FIX-PLANS.md](FIX-PLANS.md).

## Use Hotato when

- You have (or can capture) separated caller/agent audio for the call.
- You can point at a moment and label what should have happened: yield or
  hold.
- You want a local, deterministic regression test that fails CI when the
  turn-taking timing regresses.

## When not to use Hotato

Hotato measures turn-taking timing from separated audio tracks. It is the
wrong tool for:

- Transcript quality or wording checks. Nothing here transcribes.
- Task success or goal completion. Hotato does not know what the call was
  for.
- Sentiment, tone, or emotion. Out of scope, permanently.
- Compliance or script adherence review.
- Tool-call or API-behavior testing.
- Mixed mono recordings. One summed channel cannot attribute talk-over to a
  speaker.
- Live, in-call decisions. Hotato scores recordings after the fact; it does
  not predict at runtime.
- Unlabeled whole-call analysis. `hotato scan` surfaces candidate moments,
  but every verdict needs your yield or hold label; Hotato never invents one.


================================================================================
FILE: docs/INGEST.md
================================================================================

# `hotato ingest` -- the composable passive on-ramp

You should not have to remember to run a CLI after every bad call. Wire a webhook
to invoke `hotato ingest` once, and every completed call is scanned for
**candidate** turn-taking moments automatically.

`ingest` is **discovery, not a verdict**. It surfaces timing candidates; it never
returns a pass/fail and never infers intent. You review the candidates and
promote one to a permanent regression test with `hotato fixture create`. The human
label step stays human: `ingest` never auto-labels, auto-fixtures, or auto-tunes.

It is built by **composition** and adds only a per-stack webhook parser:

```
parse the webhook payload   ->  extract the call id / recording locator
fetch the recording         ->  hotato.capture (the SAME fetch the adapters use)
scan for candidates         ->  hotato.scan (no labels, no verdict)
write a candidate report     ->  JSON always; --out an HTML report (optional)
```

## What it is not

- **Not a daemon.** Hotato ships the command; *you* own the trigger (a webhook
  handler, a serverless function, a cron over your call log). There is no
  long-running process and no hosted service, so the offline/self-host wedge stays
  intact. The only network is the same recording fetch `hotato capture` already
  does; everything else is offline.
- **Not a labeler.** A candidate is a timing event. Only you know whether a caller
  sound was "mhm" or "stop", so only you label it.

## Contract

```
hotato ingest --stack {vapi|retell|twilio|livekit|pipecat} \
    (--event PAYLOAD.json | --call-id ID | --recording-sid RE...) \
    [--out report.html] [--format text|json] [--allow-mono] [--top N] [--min-gap S]
```

- **Exit 0** = ran (candidates reported, possibly zero).
- **Exit 2** = parse / fetch / IO error, or not-scorable input (for example a mono
  recording, which cannot attribute overlap to caller vs agent). Never a pass/fail.

`--format` controls stdout (`text` listing or the `json` candidate list, capped by
`--top`). `--out` additionally writes an HTML candidate report containing every
candidate. A webhook payload is **untrusted DATA**: `ingest` reads only the named
locator fields and never executes, scaffolds, or acts on anything in it.

## Wire your webhook -> `hotato ingest`

Point your platform's call-completed webhook at a small handler that saves the
payload and shells out to `ingest`. The pattern is identical for every stack:

```python
# a minimal webhook handler (framework-agnostic pseudocode)
import json, subprocess, tempfile, os

def on_call_completed(request):
    payload = request.body                       # the platform's webhook payload
    with tempfile.NamedTemporaryFile(
        "w", suffix=".json", delete=False) as fh:
        fh.write(payload)
        event_path = fh.name
    # ingest is one process; run it out-of-band so the webhook returns fast.
    subprocess.Popen([
        "hotato", "ingest", "--stack", "vapi",
        "--event", event_path,
        "--out", f"candidates/{request_id}.html",
    ], env={**os.environ, "VAPI_API_KEY": os.environ["VAPI_API_KEY"]})
    return 200
```

A cron over your call log works equally well when you would rather batch:

```bash
# nightly: scan yesterday's calls for candidates
for id in $(your-call-log --since yesterday --ids); do
  hotato ingest --stack vapi --call-id "$id" \
      --format json --out "candidates/$id.html" >> candidates/$id.json
done
```

### Per-stack recipe

Webhook field paths verified against live vendor docs (2026-07-07). Where a field
could not be confirmed from the live docs, `ingest` parses it **defensively** (a
missing field is simply absent, never fabricated).

| Stack | Webhook | Field ingest reads | Credentials for the fetch |
|-------|---------|--------------------|---------------------------|
| **vapi** | end-of-call-report | `message.call.id` (confirmed) | `VAPI_API_KEY` |
| **retell** | call webhook | top-level `event` + `call.call_id` (confirmed) | `RETELL_API_KEY` |
| **twilio** | `recordingStatusCallback` | `RecordingSid` (confirmed; form-encoded body) | `TWILIO_ACCOUNT_SID` + `TWILIO_AUTH_TOKEN` |
| **livekit** | egress webhook | `egressInfo.fileResults[].location` / `.filename` (defensive) | none -- egress lands in your storage |
| **pipecat** | your own event | `recording_path` / `recording_url` (defensive) | none -- you produced the file |

For **vapi / retell / twilio**, `ingest` extracts the identifier from the payload
and delegates the dual-channel recording fetch to the same adapter `hotato capture`
uses (which already resolved the recording URLs; see
[ADAPTER-STATUS.md](ADAPTER-STATUS.md)). The recording URL is never read from the
untrusted payload.

For **livekit / pipecat**, the recording lands in *your* infra, so the event
carries the locator directly. LiveKit egress files land in your storage bucket;
supply a `recording_url` (downloaded) or a `recording_path` (read locally). A
Pipecat event is whatever you emit, for example:

```json
{ "recording_path": "captured.wav" }
```

You can also skip the payload entirely with a direct id:

```bash
hotato ingest --stack vapi   --call-id  <id>   --out candidates.html   # + VAPI_API_KEY
hotato ingest --stack twilio --recording-sid RE... --format json       # + TWILIO_*
hotato ingest --stack pipecat --event event.json                        # local file
```

## From a candidate to a regression test

`ingest` finds the moment; you decide what should have happened and freeze it:

```bash
# 1. ingest surfaces a candidate at t=42.18s in a call recording
# 2. you listen, decide the agent should have yielded, and promote it:
hotato fixture create --stereo call.wav --onset 42.18 \
    --expect yield --id refund-cutoff-001 --out tests/hotato
# 3. it is now a permanent test:
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio
```

See [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md) for the full bad-call-to-CI loop.

## Notes on `--allow-mono`

Discovery needs one party per channel to attribute overlap. `--allow-mono` (or
`HOTATO_ALLOW_MONO=1`) lets the *fetch* pull a mono-only recording on retell/twilio
(matching `hotato capture`), but a mono mix is still reported **not-scorable**
(exit 2) for discovery, because overlap cannot be split between caller and agent.
Record dual-channel to get candidates.


================================================================================
FILE: docs/FIX-PLANS.md
================================================================================

# Fix plans: the guarded ladder

Hotato measures turn-taking; this ladder turns a failing measurement into a
reviewable, bounded fix proposal. Every level in this phase is read-only: no
command mutates any platform config, and no apply command exists.

## The ladder

| Level | Command | What it does | Writes to your stack |
|---|---|---|---|
| 0 | `hotato diagnose result.json` | Per-failure diagnosis + a battery decision, with the honest advisory and the tradeoff stated | never |
| 1 | `hotato inspect --stack ...` | Reads the CURRENT turn-taking config (GET or static parse) and normalizes it | never |
| 2 | `hotato plan result.json [target]` | Combines diagnosis + inspected config into a fix-plan JSON (`hotato.fixplan.v1`) | never |
| 3 | apply / verify | NEXT PHASE, not shipped | see below |

Level 3 is deliberately absent from this release. When it ships it will be
PR-first and clone-first: apply to a cloned assistant or a branch config,
re-run the battery, and only then graduate to production behind an explicit
approval with a recorded rollback value. Every plan already pins
`"approval": {"default": "manual", "production_apply": false}` so nothing
built today can be replayed into an auto-apply.

## Level 0: diagnose

```
hotato run --suite barge-in --format json > result.json
hotato diagnose result.json
```

One diagnosis per failing event:

```
{"finding":          missed_real_interruption | false_stop_on_backchannel |
                     slow_yield | excess_talk_over | endpointing_miss |
                     not_scorable | threshold_funnel,
 "evidence":         the measured fields for that event,
 "likely_layer":     interruption_detection | endpointing | unknown_root_cause,
 "config_only_safe": bool,
 "notes":            plain language}
```

plus one battery-level decision. The text mode is the Level 0 advisory, for
example: "Missed real interruption. Likely config layer. Try lowering the
stop-speaking word threshold one step. Tradeoff: may increase false stops on
short acknowledgements." The tradeoff is always stated.

Exit codes: 0 no failing events, 1 failing events diagnosed, 2 unusable input.

## Level 1: inspect

```
hotato inspect --stack vapi --assistant-id <id>      # + VAPI_API_KEY
hotato inspect --stack retell --agent-id <id>        # + RETELL_API_KEY
hotato inspect --stack livekit --config agent.py     # static parse
hotato inspect --stack pipecat --config bot.py       # static parse
```

Output: one normalized model (`interrupt_min_words`,
`interrupt_voice_seconds`, `resume_backoff_seconds`,
`endpointing_wait_seconds`, `backchannel_aware`) plus the raw fields and
provenance (what was fetched, when, and which docs the field names were
verified against). Absent or unreadable options are null with a note; values
are never guessed. Suspicious values (an unusually high word threshold, a long
endpointing wait) are surfaced as observations, never judgments.

Read-only by construction: Vapi and Retell are one GET each; LiveKit and
Pipecat files are parsed with `ast` and never imported or executed. Missing
credentials exit 2 cleanly.

## Level 2: plan

```
hotato plan result.json --stack vapi --assistant-id <id> --out hotato-fixplan.json
hotato plan result.json                              # stack from the envelope, else generic
```

(`--run result.json` is the equivalent flag form.) The input must be a run
envelope; frame dumps, benchmark results, and compare results are rejected
with exit 2.

The plan is `hotato.fixplan.v1` (shipped schema:
`src/hotato/schema/fixplan.v1.json`). It carries the finding, a hypothesis,
zero or more changes, the verification gate, and the approval block. Every
plan also carries `kind: "fix-plan"`, the measured `evidence` behind it, the
stated `risks`, `next_commands` (apply the step manually, verify with
`hotato compare`, re-run the battery), not-scorable events as `input_issues`
(input problems, never fixed), and a `platform_mutation` block whose
`performed` is always false: hotato plan is read-only.

Twilio rule: `--stack twilio` (or a Twilio-stack envelope) never yields
agent-config advice. Twilio carries the audio; it does not decide when the
agent yields. The plan is a checklist: confirm the dual-channel caller/agent
assignment, then re-plan against the upstream voice-agent stack.

### Policy rules (verbatim, as implemented and tested)

A change is proposed ONLY when ALL of these hold:

  (a) the failure class maps cleanly to ONE setting;
  (b) the proposed value is ONE bounded step in an unambiguous direction
      within documented bounds - never an absolute magic value; `from` is the
      inspected current value, and when inspection was not run or not
      possible the plan carries direction and bounds only, with
      `current_unknown: true`;
  (c) the run's battery contains at least one passing OPPOSITE-RISK fixture,
      else the plan DOWNGRADES to insufficient_coverage ("add an
      opposite-risk fixture before tuning") and names the exact fixture
      family to add;
  (d) the diagnosis marked the failure config_only_safe.

And the standing refusals:

* threshold_funnel: when the battery contains BOTH a missed real interruption
  AND a false stop on a backchannel, the plan is a refusal:
  `"decision": "do_not_tune_single_threshold"` with a vendor-neutral
  engagement-control pointer (no product names, no digits in the pointer
  text). No single threshold satisfies both axes; raising it for one worsens
  the other.
* slow_yield without a clear layer: TTS buffering, transport latency, and VAD
  smoothing are indistinguishable from one recording, so the plan is a
  diagnostic checklist (instrumentation steps), never a knob change. A slow
  yield becomes config-only-safe only when the battery contains a passing
  opposite-risk backchannel fixture that makes a one-step change verifiable.
* not_scorable events are input problems. They are never diagnosed as agent
  failures and never feed a plan.

Every plan, of every decision kind, carries the same verification gate:

```
"required_verification": ["real_interruption_fixture_must_pass",
                          "backchannel_fixture_must_not_regress",
                          "slow_yield_p95_must_not_worsen"]
```

### Worked example: a safe one-step plan

Battery: the agent missed a real interruption; the backchannel fixture
passes (the opposite-risk coverage the policy requires). Inspection read
`stopSpeakingPlan.numWords = 2` from the live assistant.

```json
{
  "schema": "hotato.fixplan.v1",
  "target": {"stack": "vapi", "inspected": true,
             "assistant_id": "asst_123", "current_unknown": false},
  "finding": "missed_real_interruption",
  "hypothesis": "The agent missed a real interruption: the caller took the floor and the agent kept talking. Inspected stopSpeakingPlan.numWords is 2; one bounded step decrease is the smallest verifiable change on this axis.",
  "config_only_safe": true,
  "decision": "propose_one_step",
  "changes": [
    {"field": "stopSpeakingPlan.numWords",
     "from": 2, "to": 1, "direction": "decrease", "bounds": [0, 10],
     "reason": "The caller took the floor and the agent never stopped within the search window. A one-step sensitivity increase is a config-layer candidate; the tradeoff is more false stops on short acknowledgements. One step decrease from the inspected value. Bounds basis: documented range 0-10 (docs.vapi.ai, 2026-07-06).",
     "risk": "more false stops on short acknowledgements; the backchannel fixture must not regress"}
  ],
  "required_verification": ["real_interruption_fixture_must_pass",
                            "backchannel_fixture_must_not_regress",
                            "slow_yield_p95_must_not_worsen"],
  "approval": {"default": "manual", "production_apply": false}
}
```

One field, one step, both endpoints of the move visible, the risk named, the
verification gate attached. Nothing in the plan is an absolute value pulled
from folklore.

### Worked example: the refusal

Battery: the packaged demo (`hotato demo --format json`), which misses a real
interruption AND yields to a bare backchannel.

```json
{
  "schema": "hotato.fixplan.v1",
  "target": {"stack": "generic", "inspected": false},
  "finding": "threshold_funnel",
  "hypothesis": "The battery missed a genuine interruption and also stopped for a backchannel. One sensitivity threshold cannot satisfy both: raising it to hold through backchannels drops real interruptions, lowering it to catch interruptions yields to backchannels. The failure class is discrimination, not calibration.",
  "config_only_safe": false,
  "decision": "do_not_tune_single_threshold",
  "changes": [],
  "recommended_fix": {
    "class": "engagement-control",
    "examples": [
      "enable adaptive interruption handling where available",
      "use a backchannel-aware interruption classifier",
      "add addressee/turn-intent discrimination before stopping TTS"
    ]
  },
  "required_verification": ["real_interruption_fixture_must_pass",
                            "backchannel_fixture_must_not_regress",
                            "slow_yield_p95_must_not_worsen"],
  "approval": {"default": "manual", "production_apply": false}
}
```

The refusal is the product working as designed: the strongest thing a tuning
tool can say about a funnel is that tuning will not fix it.

## Provenance

Vendor field names and documented ranges used by inspect and plan were
verified against docs.vapi.ai (assistant startSpeakingPlan/stopSpeakingPlan),
docs.retellai.com (get-agent), docs.livekit.io (turn handling options), and
docs.pipecat.ai (user turn strategies) on 2026-07-06, and each result records
its own `field_basis`. Where a platform documents no hard range, the plan says
so and uses a conservative working range with a note to verify against the
installed version.


================================================================================
FILE: docs/FIX-LOOP.md
================================================================================

# The closed loop: find -> fix -> prove it's fixed

`hotato diagnose` / `inspect` / `plan` (see [FIX-PLANS.md](FIX-PLANS.md)) end at
a **proposal**. The closed loop carries it the rest of the way, with the two
irreversible decisions kept firmly in human hands:

1. **`hotato patch`** turns a plan's abstract `{field, from, to}` into a
   **literal, paste-ready artifact** for your platform. It PRODUCES the change;
   it never applies it.
2. You apply it, in your own stack, and re-capture the failing moments.
3. **`hotato verify`** scores the old and new runs against each other and gives
   you a **battery-scale before/after proof**: *N of M fixtures that used to
   fail now pass, and K of L hold fixtures still pass.*
4. **`hotato loop`** orchestrates the whole thing and **remembers where it left
   off**, so you can walk away and come back.

Nothing here mutates a platform, auto-labels a moment, or auto-applies a change.

## `hotato patch <fixplan.json>` -- the paste-ready change

Reads a fix plan (schema `hotato.fixplan.v1`) and renders it per platform:

- **Vapi / Retell** (config behind a REST API): a JSON **merge-patch body** plus
  a ready **`curl`** against the platform's real config-update endpoint, using
  the exact field names the plan carries.

  ```
  hotato plan result.json --stack vapi --assistant-id <id> --out fixplan.json
  hotato patch fixplan.json
  ```
  ```
  curl -X PATCH https://api.vapi.ai/assistant/<assistant-id> \
    -H "Authorization: Bearer $VAPI_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"stopSpeakingPlan": {"numWords": 2}}'
  ```

- **LiveKit / Pipecat** (config lives in your agent source): there is no
  config-update REST call to hit, so patch emits the exact **source edit** --
  the constructor kwarg and the literal value to set -- never a fabricated
  endpoint. For example `InterruptionOptions(min_words=1)`.

- **generic / unknown stack**: patch names the knob **family** and asks for a
  concrete stack; it emits no literal body it cannot stand behind.

### The both-axes case: no patch, a pointer instead

`hotato patch` only handles the config-fixable classes. When the plan's decision
is `do_not_tune_single_threshold` -- the genuine **both-axes** case, where the
battery misses a real interruption AND false-stops on a backchannel at once -- no
single config value fixes both. patch emits **no config patch**. It prints the
vendor-neutral, numbers-free **engagement-control pointer** instead: it names the
problem class (discriminating a real bid for the floor from a backchannel) and
the KIND of fix it needs, names no product, and carries no digits. It is not an
upsell, and it fires ONLY on this case -- never on an ambiguous slow yield or a
coverage gap, which get their honest "no patch, here's why" instead.

Every other non-propose decision (diagnostic checklist, insufficient coverage,
already at a documented bound, no change) likewise emits no patch, with the
reason pointing back at the plan.

**Honesty:** patch makes no network call and pins `applies_change` to false.
`--format json` emits the full artifact; `--out PATH` also writes it.

## `hotato verify --before <old> --after <new>` -- the proof

After you apply the change and re-capture the same fixtures, verify scores the
old and new run envelopes against each other. Each side is a single `hotato run`
envelope JSON (a whole battery), or a directory of them; fixtures pair by
`event_id` (then `scenario_id`).

```
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio \
    --format json > before.json          # the failing take
# ... apply the patch, re-capture the fixtures ...
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio-new \
    --format json > after.json
hotato verify --before before.json --after after.json
```

It **reuses** the `hotato compare` taxonomy per fixture (`fixed`, `regressed`,
`improved`, `worse`, `unchanged`, `still_pass`, `not_scorable`) and aggregate's
pooled-distribution definitions for the before/after talk-over and time-to-yield
shift. The rollup is the two axes that matter:

- **regression axis**: how many previously-failing fixtures now pass;
- **hold axis**: how many hold-labeled guard fixtures still pass (they must not
  regress).

**Honesty:**

- verify reports **coincidence, never causation**: it says the improvement
  *coincides with* your change, never that it *caused* it. Hotato measures
  timing; it does not run a controlled experiment.
- it **refuses** the battery-scale claim when too few fixtures failed to
  characterize (`--min-n`, default 3): the per-fixture facts still print, but the
  headline proof is withheld and said so.
- an unjudgeable side is `not_scorable`, never an invented verdict; a fixture on
  only one side is reported unpaired, never silently folded into the rollup.

By default verify measures and exits 0; `--fail-on-regression` exits 1 if any
fixture regressed or got worse.

## `hotato loop [FOLDER]` -- one command, with memory

`hotato loop` drives the parts Hotato can drive and remembers where it left off
in a small local state file (`.hotato/loop-state.json` by default):

- **First run over a folder of calls**: it runs discovery (`analyze` -> `scan` ->
  rank) and records the candidate moments. Stage `awaiting_label`:

  > you have N candidate moment(s) awaiting your label.

- **You label the ones that matter** with `hotato fixture create` (loop never
  labels for you: only a human supplies the yield/hold intent).

- **Next run, with those fixtures present** (`--fixtures DIR`): it runs them,
  diagnoses the battery (including the threshold-funnel check), and plans a
  guarded fix. Stage `awaiting_verify`:

  > a fix plan is ready; apply it with hotato patch, then prove it with hotato
  > verify.

```
hotato loop ./recordings                          # run 1: discover
hotato fixture create --stereo rec.wav --onset 12.4 \
    --expect yield --id refund-001 --out tests/hotato
hotato loop ./recordings --fixtures tests/hotato   # run 2: plan
```

**Hard rules:** the loop NEVER auto-labels (you supply every yield/hold intent),
NEVER auto-applies (it produces a plan and points at `hotato patch`; applying and
verifying stay human steps), and mutates no platform. It orchestrates and tracks
state; you keep the two decisions that matter -- which moment is a real bug, and
whether to apply the fix.

## Exit codes

- `hotato patch`: `0` a patch (config artifact or the engagement-control
  pointer) was produced; `2` the input is not a fix plan or is unreadable.
- `hotato verify`: `0` the rollup was produced (a low-n claim is refused but
  still exits 0); `1` with `--fail-on-regression`, a fixture regressed or got
  worse; `2` usage error, unreadable input, or no fixtures pair.
- `hotato loop`: `0` advanced or re-reported state; `2` no folder on the first
  run, an unreadable state file, or a path that is not a folder.


================================================================================
FILE: docs/ANALYZE.md
================================================================================

# `hotato analyze <folder>` — drop a folder, hear the bug

Zero-config discovery over a whole folder of real dual-channel call recordings.
No scenarios, no labels, no onset, no flags required: point it at the folder and
it does the rest.

```bash
hotato analyze ./recordings
```

That writes one self-contained, offline HTML dashboard (`hotato-analyze.html` by
default) and opens it. `hotato ./recordings` — a bare folder as the first
argument — routes to the same command.

## What it does

For every `.wav` under the folder (walked recursively, in sorted order):

1. runs the same whole-call scanner as [`hotato scan`](../src/hotato/scan.py):
   it walks the caller and agent VAD activity tracks across the ENTIRE call,
   label-free, and emits candidate timing moments —
   `overlap_while_agent_talking`, `agent_start_during_caller`,
   `long_response_gap`, `agent_stop_no_caller`, `echo_correlated_activity` —
   each with a timestamp and a measured number;
2. aggregates the candidates across ALL calls and ranks them by the scanner's
   own salience (overlap seconds / gap seconds / echo coherence) so the worst
   moments float to the top.

Then it emits three things.

### 1. A ranked dashboard

One card per top moment, in the [`hotato report`](REPORTS.md) house style: the
call file, the timestamp, the candidate kind, the measured number, and a
to-scale caller/agent timeline of that exact moment (the same SVG renderer the
report uses — activity spans, the shaded talk-over band, the onset marker, and
a yield marker where the scanner measured the agent going silent).

### 2. The hear-the-bug player

For the top `--audio-top` moments (default 8) the REAL audio around the moment
is embedded inline as a base64 WAV data URI — nothing is uploaded, the page has
zero external requests. Press play and a **playhead** sweeps that moment's
timeline in lockstep with `audio.currentTime` (via `requestAnimationFrame`), so
you HEAR the agent talk over the caller, or the dead-air gap, land exactly where
the chart marks it. Reduced-motion safe: with `prefers-reduced-motion: reduce`
the playhead still tracks playback (it rides `timeupdate` instead of the smooth
animation loop).

Only the top moments carry audio; the rest show the timeline only. That, plus
`--pre` / `--post` (the seconds of audio kept before/after each moment) and a
total-page audio budget, keeps the page a reasonable size.

### 3. JSON for agents

```bash
hotato analyze ./recordings --format json
```

prints the ranked candidates plus their metadata (source file, timestamp, kind,
salience, measured durations, and the audio/timeline window) to stdout, capped
by `--top`. Pass `--out FILE` to also write the full ranked JSON.

## Honest framing

These are **measured candidate timing moments**, not verdicts and not intent.
Energy is not intent: the scanner cannot know whether a caller sound was "mhm"
or "stop", so nothing here is a pass/fail, a failure count, or an accuracy
number. You decide the expected behavior and label the moments that matter:

```bash
hotato fixture create --stereo <call>.wav --onset <t> \
    --expect yield|hold --id found-moment-001 --out tests/hotato
```

Non-dual-channel or otherwise unreadable files are reported cleanly in a
"Skipped files" section with their reason (a mono mix cannot attribute
talk-over); a bad file never crashes the run.

## Flags

| flag | default | meaning |
| --- | --- | --- |
| `FOLDER` | (required) | directory of dual-channel WAVs, walked recursively |
| `--top` | 25 | ranked moments shown in the dashboard / stdout JSON (0 = all) |
| `--audio-top` | 8 | top moments that get the embedded hear-the-bug player |
| `--pre` | 2.0 | seconds of audio/timeline kept before each moment |
| `--post` | 4.0 | seconds of audio/timeline kept after each moment |
| `--min-gap` | 2.0 | minimum response gap (seconds) to surface as a candidate |
| `--caller-channel` / `--agent-channel` | 0 / 1 | channel assignment |
| `--format` | `html` | `html` (dashboard) or `json` (ranked candidates) |
| `--out` | `hotato-analyze.html` | where to write the dashboard, or the JSON |
| `--no-open` | off | do not launch a browser for the dashboard |

## Exit codes

- **0** — ran (candidate moments listed across the folder, possibly zero; never
  a pass/fail and never a verdict).
- **2** — usage error (the path is not a folder) or an IO error reading it.

Everything runs offline; no audio leaves the machine.


================================================================================
FILE: docs/CONNECT.md
================================================================================

# Connect, pull, sweep: score every real call across your stack

`hotato sweep` is the "connect once, see every turn-taking problem across all
your real calls" flow. It has three steps you can also run on their own:

1. **`hotato connect <stack>`** — store a stack's credentials once, locally.
2. **`hotato pull`** — bulk-fetch your recent recordings into a folder.
3. **`hotato sweep`** — pull, then run the zero-config `analyze` over them and
   write one offline dashboard of the ranked turn-taking moments.

Everything scores offline. The only network is the direct recording download
from your vendor to your machine; your audio and your keys are never sent to
Hotato.

## 1. Connect once

```
hotato connect vapi --api-key <key>
```

This does a lightweight live auth check (it lists one recent call), then writes
the credentials to `~/.hotato/connections.json` with file mode `0600` (directory
`0700`). The key is never printed and never leaves your machine except to the
vendor's own API. Credentials also fall back to the stack's environment variable,
so this works too:

```
VAPI_API_KEY=<key> hotato connect vapi
```

Per-stack credentials:

| Stack | Flags (or env vars) |
| --- | --- |
| vapi | `--api-key` / `VAPI_API_KEY` |
| retell | `--api-key` / `RETELL_API_KEY` |
| twilio | `--account-sid` `--auth-token` / `TWILIO_ACCOUNT_SID` `TWILIO_AUTH_TOKEN` |
| bland | `--api-key` / `BLAND_API_KEY` |
| elevenlabs | `--api-key` / `ELEVENLABS_API_KEY` |
| synthflow | `--api-key` / `SYNTHFLOW_API_KEY` (+ `--model-id` / `SYNTHFLOW_MODEL_ID` to list) |
| millis | `--api-key` / `MILLIS_API_KEY` (+ `--base-url` for the EU region) |
| cartesia | `--api-key` / `CARTESIA_API_KEY` (+ `--agent-id` / `CARTESIA_AGENT_ID` to list) |

Retell has no list endpoint to verify against, so `connect retell` stores the
key and validates it on the first pull. Use `--no-verify` to skip the live check
for any stack. LiveKit and Pipecat are capture-in-your-infra — there is no
vendor recording to pull, so they are not connectable; use `hotato setup
--stack livekit|pipecat` instead.

After connecting, `--stack` and the credential flags are optional for `pull` and
`sweep`: when exactly one stack is connected, Hotato uses it.

## 2. Pull recent recordings

```
hotato pull --stack vapi --since 7d --limit 50
hotato pull                                   # the only connected stack
```

`pull` lists your recent recordings with the vendor's verified list endpoint and
downloads each one by looping the same single-call fetch `hotato capture` uses,
into `hotato-pull-<stack>/` (override with `--out DIR`). `--since` accepts `7d`,
`12h`, `30m`, `2w`. A recording that cannot be fetched (missing URL, HTTP error,
wrong channel count) is reported as a clean skip with its reason and the pull
continues — one bad call never crashes the run.

**Retell has no verified list endpoint.** Pull it from explicit ids instead
(Hotato never fabricates an endpoint):

```
hotato pull --stack retell --call-id c1 --call-id c2
```

For Twilio, `--call-id` values are Recording SIDs (`RE...`).

**Mono / mixed stacks** (bland, elevenlabs, synthflow, millis, cartesia) produce
a single combined recording with no per-party channel, so talk-over cannot be
attributed. They require `--allow-mono` and are indicative only:

```
hotato pull --stack bland --allow-mono --limit 20
```

## 3. Sweep = pull + analyze

```
hotato sweep --stack vapi --since 7d
hotato sweep                                  # the only connected stack
```

`sweep` pulls (as above) into `hotato-sweep-<stack>/` (override with `--dir`),
then runs the exact same zero-config `analyze` over the folder and writes one
self-contained, offline HTML dashboard (`hotato-sweep-<stack>.html`, override
with `--out`) of the ranked candidate turn-taking moments across every call,
with the hear-the-bug audio player on the top moments. `--format json` emits the
ranked candidates plus a pull summary instead.

Dual-channel stacks give separated scoring. Mono/mixed stacks can be swept with
`--allow-mono`, but their calls cannot be attributed per party and surface in the
dashboard's Skipped section — honestly, not faked.

Candidates are MEASURED timing moments you review and label with `hotato fixture
create`, never verdicts and never intent. There is no pass/fail, no failure
count, and no accuracy number anywhere.

## What is and isn't pullable

See [`docs/ADAPTER-STATUS.md`](ADAPTER-STATUS.md) for the full, honest map:
which stacks auto-pull dual-channel, which are mono-only behind `--allow-mono`,
which are capture-in-your-infra, and which are not integrable — each with the
exact verified endpoint and the honest gaps.


================================================================================
FILE: docs/ADAPTER-STATUS.md
================================================================================

# Adapter status

What each capture adapter is built against, verified verbatim against the
vendor's live documentation on the date shown (integration research:
`hotato-launch/INTEGRATION-SPEC-2026-07-07.md`). This file is the honest map of
what Hotato can and cannot pull, and why.

Terms:

- **auto-pull** — Hotato fetches the recording itself with your API key
  (`hotato capture` / `hotato pull` / `hotato sweep`).
- **capture-in-your-infra** — there is no vendor recording API; Hotato prints
  the recording config (`hotato setup`) and scores the file your deployment
  writes.
- **mono / mixed** — a single combined channel. Overlap cannot be attributed to
  the caller or the agent, so Hotato scores it only behind an explicit
  `--allow-mono` / `HOTATO_ALLOW_MONO=1` opt-in and labels the result indicative
  only. Separated turn-taking analysis is not possible from mono.

Honesty rule (enforced): an adapter ships only with endpoints verified verbatim
in the spec. Where the spec marks a list-calls endpoint or a channel layout
**unconfirmed / none**, Hotato does not invent one — it supports the honest
fallback (explicit ids) and documents the gap here rather than guessing.

## Build now — real dual-channel (auto-pull, separated scoring)

| Stack | List recent calls | Fetch recording | Channel basis | Last verified |
| --- | --- | --- | --- | --- |
| Vapi | `GET https://api.vapi.ai/call` (params `limit`, `createdAtGt/Lt`) → JSON array of Call objects (`id`, `createdAt`) | `GET /call/{id}` → `artifact.recording.stereoUrl` (current); deprecated fallbacks `artifact.stereoRecordingUrl`, `call.stereoRecordingUrl` | `stereoUrl` is a distinct 2-channel file (customer ch0, assistant ch1) | 2026-07-07 |
| Twilio | `GET .../Accounts/{Sid}/Recordings.json` (params `PageSize`, `DateCreatedAfter/Before`, `callSid`) → `recordings[].sid` | `GET .../Recordings/{RE...}.wav?RequestedChannels=2` (HTTP Basic) | dual-channel only if the recording was created `RecordingChannels=dual`; 400 → clean stop or `--allow-mono` → `RequestedChannels=1`. Two-party order: ch0 = caller, ch1 = agent; conference: ch0 = first participant | 2026-07-07 |
| Retell | **none confirmed** — the spec marks list-calls unconfirmed; do not fabricate one. Pull from explicit `--call-id` | `GET https://api.retellai.com/v2/get-call/{id}` (Bearer) → `scrubbed_recording_multi_channel_url` preferred, then `recording_multi_channel_url`; plain mono `recording_url` rejected unless `--allow-mono` | per-party channels on the `*_multi_channel_url` fields | 2026-07-07 |
| LiveKit | **capture-in-your-infra** — `ListEgress` is an RPC method name only; no REST list. `hotato setup --stack livekit` | Two audio-only Track egresses (one per party); recording location in `egress_info.file_results[].location` | separated by running one Track egress per participant | 2026-07-07 |
| Pipecat | **capture-in-your-infra** — Pipecat Cloud session-list has no recording field; OSS has no list at all | `AudioBufferProcessor(num_channels=2)` in-pipeline (user left, bot right); you write the WAV | 2-channel in-pipeline | 2026-07-07 |

## Build now — mono / mixed only (auto-pull, `--allow-mono`, indicative only)

Each of these has a verified list + fetch, but the vendor produces a single
combined recording with **no documented per-party channel**, so scoring is
degraded and gated behind `--allow-mono`.

| Stack | List recent calls | Fetch recording | Mono basis (verbatim) | Last verified |
| --- | --- | --- | --- | --- |
| Bland AI | `GET https://api.bland.ai/v1/calls` → `calls[].call_id` | `GET /v1/calls/{id}` → `recording_url` | no stereo/channel field in the recording or call-details schema | 2026-07-07 |
| ElevenLabs Conversational AI | `GET https://api.elevenlabs.io/v1/convai/conversations` (params `page_size`, `call_start_after_unix`) → `conversations[].conversation_id` | `GET /v1/convai/conversations/{id}/audio` → combined audio | docs state the audio "does not include separate caller/agent channels — only a combined full conversation MP3" | 2026-07-07 |
| Synthflow | `GET https://api.synthflow.ai/v2/calls?model_id=…` (params `limit`, `from_date` epoch-ms) → `response.response.calls[].call_id` | `GET /v2/calls/{id}` → `response.response.calls[0].recording_url` (a Twilio Recordings URL) | Synthflow documents no dual-channel option; `recording_url` is a bare Twilio URL | 2026-07-07 |
| Millis AI | `GET {base}/call-logs` (US `api-west`, EU `api-eu-west`; param `limit`) → `histories[].session_id` | `GET /call-logs/{session_id}` → `recording.recording_url` | schema exposes only `recording_url`; `CallSettings` has only a boolean `enable_recording` | 2026-07-07 |
| Cartesia (Line) | `GET https://api.cartesia.ai/agents/calls?agent_id=…` (param `limit`; needs `Cartesia-Version` header) → `data[].id` | `GET /agents/calls/{id}/audio` → `audio/wav` | dual-vs-mono **unconfirmed** in the docs; treated as mono until a live channel-count check proves otherwise | 2026-07-07 |

`--stack synthflow` needs `--model-id` (its list endpoint requires `model_id`);
`--stack cartesia` needs `--agent-id`; `--stack millis` accepts `--base-url` for
the EU region. Single-call `capture`/`pull` by explicit id needs none of these.

## Not integrable (no vendor recording to pull)

| Stack | Why | Basis |
| --- | --- | --- |
| Deepgram Voice Agent API | Real-time WebSocket (`wss://agent.deepgram.com/v1/agent/converse`) with no REST list-calls, no fetch-recording endpoint, and no recording-ready webhook — Deepgram does not store the call. If you want a recording, capture it in your own infra (LiveKit/Pipecat pattern) | 2026-07-07, confirmed absent |
| PlayAI (formerly Play.ht) | The conversational-agent product is dead: `play.ai` DNS does not resolve and `docs.play.ai` returns `DEPLOYMENT_NOT_FOUND`. The only live domain, `docs.play.ht`, is TTS-only with no calls/recordings API | 2026-07-07, dead endpoints verified |

## Unconfirmed — needs credentials or a live probe before shipping

Documented weakly or behind a login/SPA; the spec could not verify a recording
field-path or a channel layout. Not shipped as adapters. Listed here honestly so
nobody assumes support that was never confirmed.

- **Regal.ai** — no list-calls and no REST fetch-recording endpoint. The
  recording arrives only via the `call.recording.available` webhook's
  `properties.recording_link`, which is a literal Twilio Recordings URL. Feed
  that URL through Hotato's existing Twilio path; Hotato does not fabricate a
  Regal pull.
- **Thoughtly** — OpenAPI types responses as an untyped `GenericResponse.data`
  blob; no recording-URL field path could be confirmed live.
- **Sindarin** — no Sindarin-hosted recording endpoint found; `record` is
  submitted with the caller's own Twilio creds, implying the recording lands in
  the customer's own Twilio account (use the Twilio path).
- **xAI Grok Voice Agent Builder** — OpenAI-Realtime-compatible; no documented
  list-calls or fetch-recording endpoint (marketing mentions in-product
  recording, but no API contract was confirmable).
- **Cartesia dual-channel** — the `/agents/calls/{id}/audio` WAV's channel count
  was not confirmable in docs; treated as mono here pending a live check.
- **Twilio dual-channel edge behaviour** — the exact 400-on-unavailable and
  conference channel-ordering guarantees were carried forward from prior notes,
  not re-verified verbatim this pass.
- **Synthflow / Regal Twilio-URL dual-channel trick** — because their
  `recording_url` is a Twilio URL, `?RequestedChannels=2` *might* yield stereo,
  but neither vendor documents or guarantees it; unconfirmed until tested against
  a live account.

Other enterprise/partial platforms researched but not shipped (login-gated,
SPA-only docs, or transcript-only APIs): Daily, Ultravox, Hume EVI, Telnyx,
Infobip, OpenAI Realtime, Amazon Connect, Parloa, Sierra, Decagon, PolyAI,
Voiceflow, Cognigy, Dialogflow CX, Genesys Cloud, NICE CXone. See the
integration spec for the per-platform verified facts and honest gaps.

## Invariants

Every adapter validates that a dual-channel file it scores has one party per
channel (2 channels) before producing separated talk-over numbers; mono stacks
are scored degraded only behind `--allow-mono`. All live fetch/list paths are
stdlib-only (`urllib`); stack SDKs import lazily and only inside your own infra.
Credentials captured by `hotato connect` are stored locally at
`~/.hotato/connections.json` (mode 0600) and are sent only to the vendor's own
API — never to Hotato. Scoring is always offline; the only network is the direct
recording download.


================================================================================
FILE: docs/REPORTS.md
================================================================================

# Reports: doctor, report, team, export

Four surfaces over the same scorer. Every number in every one of them is a real
measurement from the envelope; nothing is recomputed, restyled into a
percentage, or fabricated. There is no accuracy percentage anywhere.

## `hotato doctor`: the 5-minute path

One command. If you pass a recording it scores that; otherwise it runs the
bundled self-test battery. Either way it writes the self-contained HTML report
and tries to open it in your browser (on a headless box it prints the path).

```bash
uvx hotato doctor --stereo call.wav     # score your call, open the report
uvx hotato doctor                       # self-test fallback, same flow
uvx hotato doctor --demo --no-open --out report.html
```

It is a convenience wrapper over the existing scorer and report. Nothing new is
claimed, and everything runs offline. Exit codes match `run`: `0` all pass,
`1` a regression (`--no-fail` forces `0`), `2` usage or IO error, or a
recording that is not scorable. Not scorable means the recording cannot answer
the question (the caller channel is silent, or the agent was not talking when
the caller started), so no verdict is given.

## `hotato report`: the visual report

One self-contained file: inline CSS, inline SVG, zero external requests. It
opens offline by double-click and survives being mailed around.

```bash
uvx hotato report --stereo call.wav --out report.html
uvx hotato report --suite barge-in --out selftest.html
uvx hotato report --stereo call.wav --format md --out report.md
```

Per event it draws a to-scale caller/agent activity timeline from the real
frame data: the overlap shaded, the caller-onset and yield markers, the
measured talk-over seconds, expected vs actual, a PASS or FAIL chip, and the
exact `ScoreConfig` thresholds used.

Above the per-event cards sits an analytics block computed from the same
measurements:

- a **time-to-yield distribution** strip, one dot per measured yield, with
  mean, median, and p90 (definitions in `METHODOLOGY.md`);
- a **talk-over histogram**, real per-event seconds bucketed on a fixed grid;
- **failure clustering by fix class**, so a batch of failures reads as "these
  five share one config setting" instead of five separate mysteries.

Every timeline carries a collapsible **frame inspector**: the full frame dump
behind that event as a table (`t_sec`, per-channel dBFS, active flags,
thresholds), so any pixel on the page can be re-derived by hand.

### Regression deltas with `--base`

Save an envelope, then compare a later run against it:

```bash
hotato run --suite barge-in --format json > base.json
hotato report --suite barge-in --base base.json --out report.html
```

The report renders per-scenario talk-over and time-to-yield deltas with clear
worse and better marks. The same `--base` flag works on
`scripts/pr_comment.py` for the CI comment (`docs/CI.md`).

### PDF

The page ships print CSS. Print it from any browser and the interactive parts
collapse into a clean paper layout, so print-to-PDF is the PDF export.

## `hotato team`: the trend view

Aggregates a directory of run envelopes into one honest trend.

```bash
hotato run --suite barge-in --format json > runs/001.json
# ... more runs over days or branches ...
hotato team runs/ --html team.html --out agg.json
```

It reports: number of runs, mean/median/p90 talk-over and time-to-yield pooled
across all events, mean/median/p90/p95 response gap (dead air before the agent
speaks) pooled the same way, pass rate per run over time, the most common
failure class, and a pass-rate trend line in the HTML page. `--order mtime`
(default) orders runs by file time; `--order name` uses the filename, so a
numeric prefix is an explicit index.

`--max-response-gap SECONDS` turns the pooled p95 response gap into a latency
SLA: the run exits `1` exactly when p95 exceeds the bound, the same
pass/fail contract as a talk-over or time-to-yield regression (`--no-fail`
always exits `0`). Percentile definitions: `METHODOLOGY.md`; the pooling
shape is `dist_summary` in `src/hotato/_stats.py`.

Fewer than two runs is stated plainly and exits `0`. It is never padded into a
trend, because a trend of one point is a fabrication.

## `hotato export`: research-grade CSVs

Scores a recording (or the bundled battery) exactly like `hotato run` and
writes three files into a directory:

```bash
uvx hotato export --stereo call.wav --out research/
uvx hotato export --suite barge-in --out research/
```

- `events.csv`: one row per scored event, every measured signal plus verdict.
- `frames.csv`: one row per VAD frame, the evidence behind every number.
- `envelope.json`: the standard machine envelope, unchanged.

Column meanings are documented in comment lines at the top of each CSV, so the
files are self-describing when they land in a notebook or a stats package
months later. An empty cell means "not derivable", never zero. Stdlib only,
offline.

`export` also prints mean/median/p90/p95 response gap pooled across the
exported events, and accepts the same `--max-response-gap SECONDS` latency
SLA gate as `team` (exit `1` when the pooled p95 exceeds the bound). A plain
export with no `--max-response-gap` writes a byte-identical `envelope.json`;
the pooled numbers and the gate live only in the printed summary and the
returned manifest, never in the CSVs or the envelope file.


================================================================================
FILE: docs/PYTEST.md
================================================================================

# Pytest: the fixture and the gate

Install hotato and the plugin registers itself (a standard `pytest11` entry
point). No `conftest.py` line, no import. It adds one fixture and one opt-in
session gate. Both score with the same engine as the CLI and return the same
envelope, the JSON result object every hotato surface emits.

```bash
pip install hotato
```

## The `hotato_score` fixture

Score a recording or a suite inside any test and assert on the envelope. Same
inputs as the CLI:

```python
def test_call_yields(hotato_score):
    env = hotato_score(stereo="call.wav", expect="yield")
    assert env["exit_code"] == 0
    assert all(e["verdict"]["passed"] for e in env["events"])

def test_split_channels(hotato_score):
    env = hotato_score(caller="caller.wav", agent="agent.wav", expect="yield")
    assert env["events"][0]["verdict"]["passed"]

def test_selftest_battery(hotato_score):
    env = hotato_score(suite="barge-in")
    assert env["exit_code"] == 0
```

The envelope is the standard machine shape (`schema_version` "1"): per event a
`verdict` (with `passed`, `did_yield`, `seconds_to_yield`, `talk_over_sec`), a
`signals` bus, a `fix` on every failure, and the `limits` block. Assert on
whichever measurement your test cares about.

## The `--hotato-suite` session gate

Opt in on the command line and the battery runs after your tests, printing a
summary and failing the whole session (exit 1) on a regression:

```bash
pytest --hotato-suite                    # default suite: barge-in
```

The bundled battery is a self-test of the harness. To gate on your own agent,
point the same flag at your own labelled scenario and audio directories:

```bash
pytest --hotato-suite \
  --hotato-suite-scenarios corpus/suites/gold/scenarios \
  --hotato-suite-audio corpus/suites/gold/audio
```

Any directory in the same scenario shape works, including your own labelled
calls (see `docs/SUITES.md` for the bundled tiers and `docs/SUBMITTING.md` for
building labelled fixtures from real recordings).

## Where it fits

- The gate is one flag on a test run you already have, so turn-taking rides
  along with every `pytest` invocation, locally and in CI.
- The GitHub workflow (`docs/CI.md`) is the other ready-made gate: it scores on
  every pull request and posts a sticky results comment. Use either, or both.
- Everything runs offline; the plugin makes no network call and the fixtures
  are deterministic, so the gate cannot flake on I/O it does not do.


================================================================================
FILE: docs/SUITES.md
================================================================================

# Corpus suites: tiered, deterministic, honest about what they prove

`corpus/suites/` ships four labelled scenario suites, 112 scenarios total,
every one synthetic shaped noise rendered deterministically from its own
labelled timings (seed `sha256(scenario_id)`). The timings are the ground
truth. Synthetic audio is the floor: these suites prove the scorer runs end to
end and catch regressions; validity on your system comes from your own labelled
calls (`docs/SUBMITTING.md`).

## The four suites

| Suite | Scenarios | Conditions | Expected exit |
|---------------------|-----------|-----------------------------------------------------------------------|---------------|
| `silver` | 40 | clean, 16 kHz, default noise floor | `0` |
| `silver-defects` | 16 | clean conditions, deliberate defect renders | `1` |
| `gold` | 40 | hard conditions: noise floors, 8 kHz, gain extremes, echo, edge timings, endurance | `0` |
| `gold-defects` | 16 | hard-condition defect renders, plus two labelled capture-defect cases | `1` |

The scenario families span the behaviours that matter on a real call: hard
interruptions (onset, speed, duration, resume), backchannels, short
acknowledgements like "mhm" that the agent should talk through (varied in
position, density, repeats, and length), double-talk, one-word interruptions,
stutter onsets, multi-turn exchanges, resume-then-reinterrupt, and latency
prompts.
`corpus/suites/manifest.json` is the machine-readable inventory: per suite the
family and category breakdown, sample rates, and the expected exit code.

## Defect suites fail by design

Every scenario in `silver-defects` and `gold-defects` is rendered to fail on
its labelled axis: an agent that keeps talking through a real interruption, or
stops for a backchannel, or misses its latency budget. A defect suite that
exits `1` is the scorer catching what it claims to catch; a defect suite that
exits `0` would be a bug. This is the negative control the positive suites
need to mean anything.

## Run a suite

Any suite is just a scenarios directory plus an audio directory:

```bash
hotato run --suite barge-in \
  --scenarios corpus/suites/gold/scenarios \
  --audio corpus/suites/gold/audio
```

The same directories plug into every other surface: `hotato report ... --out
report.html` for the visual report, `hotato export ... --out research/` for
CSVs, and `pytest --hotato-suite --hotato-suite-scenarios ... --hotato-suite-audio ...`
for the session gate (`docs/PYTEST.md`).

## Deterministic builder

The suites are generated, not hand-edited, and the generator is the proof:

```bash
python3 corpus/suites/build_suites.py           # rebuild in place
python3 corpus/suites/build_suites.py --check   # regenerate to a temp dir, byte-compare
```

`--check` regenerating byte-identical output is the reproducibility guarantee:
the audio on disk is exactly what the labelled timings say it is, on any
machine. CI can run it as a drift gate.

## Additive scenario classes

`corpus/classes/` ships four small, deterministic classes on top of the four
suites above, built the same way (synthetic shaped noise, seed
`sha256(scenario_id)`, `--check` byte-compares a rebuild): `mid-utterance-pause`
(a multi-second thinking gap mid-turn, not a true turn end), `backchannel-multilingual`
(short non-English acknowledgement tokens; Hotato's VAD is energy-based and
does not detect language), `noise-hold` (sustained background presence, not a
brief backchannel, that the agent should hold through), and
`telephony-degraded` (an existing gold scenario re-rendered through G.711
mu-law plus a fixed packet-loss schedule, to prove the verdict is stable
across codec degradation). Kept separate from `corpus/suites/` because
`mid-utterance-pause` needs a non-default `turn_end_silence_sec` that the
generic suite tests do not apply. Full per-class detail: `corpus/classes/README.md`.

## Against a live stack

To run scenario audio through a real voice stack and score what comes back,
see [`docs/BENCHMARK-STACKS.md`](BENCHMARK-STACKS.md). Measurement-error
methodology for the scorer itself is in [`docs/BENCHMARK.md`](BENCHMARK.md).

## Contribute scenarios

New synthetic families are welcome through the normal PR path
(`CONTRIBUTING.md`). The highest-value contribution stays a real, consented,
labelled call: the walkthrough is [`docs/SUBMITTING.md`](SUBMITTING.md) and the
[corpus-submission issue form](https://github.com/attenlabs/hotato/issues/new?template=corpus_submission.yml)
opens the intake.


================================================================================
FILE: docs/CI.md
================================================================================

# CI: gate a PR on turn-taking

hotato scores turn-taking timing from call recordings, so a pull request can
carry a running score and fail when the agent gets slower to stop talking for
an interrupting caller. The workflow runs offline, needs no extra dependencies,
and posts one comment that updates itself on every push.

## Drop it in

Copy [`.github/workflows/hotato.yml`](../.github/workflows/hotato.yml) into your
repository at the same path. That is the whole setup. On the next pull request it
will:

- install the package with `pip install .`
- score the bundled barge-in suite to a JSON envelope
- render a Markdown summary with [`scripts/pr_comment.py`](../scripts/pr_comment.py)
- post or update one sticky comment (found by a hidden marker, so it never spams)
- fail the job on any regression

The sticky comment shows a pass/fail line, a per scenario table (expect, yielded,
time to yield, talk over, result), and a short regressions section.

The job needs `pull-requests: write` to post the comment. The workflow already
requests it; if your org restricts the default `GITHUB_TOKEN`, allow that scope.

## Point it at your own recordings

The bundled suite is a self-test: it scores frozen synthetic fixtures to prove
the harness works, not to judge your agent. The strongest gate is a suite of
your OWN bad moments, pinned as fixtures with `hotato fixture create` and run
with `hotato run --scenarios DIR --audio DIR`; the full loop from one bad call
to this gate is [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md). Alternatively, replace
one step, `Score turn-taking (head)`, with your own capture and score:

1. play each corpus `*.caller.wav` into your agent
2. record your agent's reply
3. score the pair, caller on channel 0 and your agent on channel 1:

```bash
hotato run --stereo your_call.wav --expect yield --format json --no-fail > head.json
```

The envelope shape and exit codes are identical, so the render, comment, and gate
steps stay exactly as they are. Keep `--no-fail` on the score step so the comment
still posts on a regression; the true pass/fail lives in the envelope's
`exit_code`, which the `Fail on regression` step reads.

## Deltas against the base branch

When the workflow can install and score the base branch, it runs the same suite
there in an isolated venv and uses it as the baseline. Any scenario where the
overlap grew (talk-over up) or the agent stopped later (time to yield up) is
listed under Regressions with the delta. This step is best effort: if it cannot
run, the comment falls back to the current pass/fail table and the gate still
holds.

## Render a comment yourself

`scripts/pr_comment.py` is stdlib only and reads the same JSON the CLI emits, so
you can preview the comment locally:

```bash
hotato run --suite barge-in --format json | python3 scripts/pr_comment.py
```

Add `--base base.json` to include deltas against a saved baseline run.

## The other ready-made gate: pytest

If your CI already runs pytest, one flag adds the same regression gate to that
run instead: `pytest --hotato-suite` scores the battery after your tests and
fails the session on a regression. Point it at your own labelled sets with
`--hotato-suite-scenarios` and `--hotato-suite-audio`. Details:
[`PYTEST.md`](PYTEST.md). For richer artifacts on a gate failure,
`hotato report` renders the visual report and `hotato team` tracks the trend
across runs ([`REPORTS.md`](REPORTS.md)).


================================================================================
FILE: docs/MCP.md
================================================================================

# MCP: the one-tool server

hotato ships a one-tool MCP server, `hotato-mcp`, that speaks MCP over stdio
and exposes exactly one tool, `voice_eval_run`, returning the identical JSON
envelope (`schema_version` "1") the CLI emits. Everything runs locally; no
audio leaves the machine.

## Run it (zero-install)

```bash
uvx --from "hotato[mcp]" hotato-mcp
```

**Common mistake:** `uvx hotato-mcp` (no `--from`) FAILS. uv then looks for a
package literally named `hotato-mcp` on PyPI, which does not exist; the
console script `hotato-mcp` lives inside the `hotato` distribution, installed
with its `mcp` extra. If you (or an agent) just tried the bare form and got a
"package not found" error, retry with `--from "hotato[mcp]"` exactly as
written above. If hotato is already installed in your environment,
`python -m hotato.mcp_server` also works and needs no extra invocation syntax.

## Add it to a client

Every block below is copy-paste exact; only the outer key (client-specific)
differs. All three use the same command and args.

### Claude Desktop

Edit `claude_desktop_config.json` (Settings -> Developer -> Edit Config):

```json
{
  "mcpServers": {
    "hotato": {
      "command": "uvx",
      "args": ["--from", "hotato[mcp]", "hotato-mcp"]
    }
  }
}
```

### Cursor

Project-scoped: `.cursor/mcp.json` in your repo root. User-scoped:
`~/.cursor/mcp.json`.

```json
{
  "mcpServers": {
    "hotato": {
      "command": "uvx",
      "args": ["--from", "hotato[mcp]", "hotato-mcp"]
    }
  }
}
```

### Codex CLI

Edit `~/.codex/config.toml`:

```toml
[mcp_servers.hotato]
command = "uvx"
args = ["--from", "hotato[mcp]", "hotato-mcp"]
```

## The one tool: `voice_eval_run`

Same two input modes as the CLI, all parameters optional beyond the mode you
pick:

| Parameter | Type | Default | Meaning |
| --- | --- | --- | --- |
| `stereo` | str | None | two-channel WAV path |
| `caller` | str | None | mono caller WAV (with `agent`) |
| `agent` | str | None | mono agent WAV (with `caller`) |
| `suite` | str | None | `"barge-in"` to run the bundled battery |
| `stack` | str | `"generic"` | livekit, pipecat, vapi, or generic |
| `expect` | str | `"yield"` | `"yield"` or `"hold"` |
| `onset_sec` | float | None | caller onset hint |
| `caller_channel` | int | 0 | caller channel index |
| `agent_channel` | int | 1 | agent channel index |
| `max_talk_over_sec` | float | None | pass threshold |
| `max_time_to_yield_sec` | float | None | pass threshold |
| `report_path` | str | None | also write the HTML report here; the envelope then carries `report_path` (absolute) |

Pass exactly one input mode: `stereo`, OR `caller` + `agent` together, OR
`suite`. Mixing modes (or passing none) is a structured error, never a raw
exception; see below.

## Output

On success, the identical envelope the CLI emits for `--format json`, schema
at [`https://hotato.dev/schema/envelope.v1.json`](https://hotato.dev/schema/envelope.v1.json)
(`schema_version` "1", additive-only: new keys may appear, the documented
core never changes).

Every expected failure (a missing / mono / mismatched / not-found file, an
unknown suite, an ambiguous input mode, or a well-formed input with no
scorable event) comes back as a structured error object, `ok: false` with a
stable `error_code` and an actionable `message`, schema at
[`https://hotato.dev/schema/error.v1.json`](https://hotato.dev/schema/error.v1.json)
-- never a raw uncaught exception. The MCP tool and the CLI share this one
error shape, so a caller (human or agent) parses one contract across both
surfaces.

## More

- Python API and the shared error contract: [`API.md`](API.md)
- What the envelope measures and the honest scope/ceiling: the top-level
  [`README.md`](../README.md) and [`METHODOLOGY.md`](../METHODOLOGY.md)
- Machine-readable index of every command (CLI and MCP): [`llms.txt`](../llms.txt)


================================================================================
FILE: docs/API.md
================================================================================

# Python API reference

Everything the CLI, the pytest plugin, and the MCP server do is a plain Python
call. The core is stdlib only, fully offline, and deterministic: the same audio
and config always produce the same envelope. Import and score:

```python
from hotato.core import run_single, run_suite

env = run_single(stereo="call.wav", expect="yield")
env = run_suite()  # bundled 8-scenario battery
```

The top-level package re-exports the essentials:
`from hotato import run_single, run_suite, LIMITS, SUITE_ID, __version__`.

All scoring functions take keyword arguments only.

## The envelope

`run_single` and `run_suite` return the same machine-readable dict
(JSON Schema: `src/hotato/schema/envelope.v1.json`):

```python
{
  "tool": "hotato",
  "schema_version": "1",
  "mode": "single" | "suite",
  "stack": "generic",            # normalized stack label
  "offline": True,
  "engine": {"name", "version", "upstream"},
  "limits": {...},               # honest scope and ceiling, hotato.core.LIMITS
  "summary": {"events", "passed", "failed", "regression"},
                                 # plus additive "not_scorable" (count) when
                                 # at least one event could not be judged
  "events": [...],               # one dict per scored event, below
  "fix_map": [...],              # one entry per failing event with a fix
  "funnel": {...} | None,        # systemic pointer, fires only when both axes fail
  "exit_code": 0 | 1,            # 1 when any scorable event failed; the CLI
                                 # process exits 2 for a single recording that
                                 # is not scorable (see Exit codes below)
  "suite": "barge-in",           # run_suite only
}
```

Each event:

```python
{
  "event_id": str,               # file basename or scenario id
  "scenario_id": str | None,
  "title": str | None,
  "category": str | None,        # e.g. "should_yield"
  "expected_yield": bool,
  "verdict": {
    "passed": bool,
    "did_yield": bool,
    "seconds_to_yield": float | None,
    "talk_over_sec": float,
    "reasons": [str],            # failure reasons, empty on pass
  },
  "measurements": {
    "caller_onset_sec": float,
    "agent_talking_at_onset": bool,
    "hop_sec": float,
    "notes": str,
  },
  "signals": {                   # namespaced signal bus, additive
    "barge_in": {"did_yield", "time_to_yield_sec", "talk_over_sec"},
    "latency": {"response_gap_sec", "premature_start_sec"},
    "echo": {                    # every event; cross-channel coherence,
                                 # deterministic, computed in hotato's own
                                 # layer (hotato/echo.py), never _engine
      "coherence", "lag_sec", "echo_suspected",
    },
    "resume": {                  # only on events where the agent yielded
      "resumed", "resume_gap_sec", "restart_suspected",
    },
  },
  "fix": None | {                # set on failing events
    "fix_class": "config" | "engagement-control",
    "title": str, "detail": str,
    "knob": str | None, "pointer": str | None,
  },
}
```

An event that cannot be judged (a silent caller channel with no onset label, or
a should-yield expectation with the agent silent at onset) additionally carries
`"scorable": False` and a plain `"not_scorable_reason"`. It counts in neither
`passed` nor `failed`, never routes a fix, and never fires the funnel: an input
problem is reported as one, not dressed up as an agent verdict. Envelopes for
valid recordings are byte-identical to before these keys existed.

## hotato.core

### run_single

```python
run_single(
    *,
    stereo: str | None = None,        # two-channel WAV path
    caller: str | None = None,        # mono WAV path (with agent=)
    agent: str | None = None,         # mono WAV path (with caller=)
    caller_channel: int = 0,
    agent_channel: int = 1,
    onset_sec: float | None = None,   # caller onset hint, seconds from start
    expect: str = "yield",            # "yield" or "hold" (backchannel)
    stack: str | None = None,         # livekit | pipecat | vapi | generic
    max_talk_over_sec: float | None = None,
    max_time_to_yield_sec: float | None = None,
    echo_gate: bool = False,          # hold an echo-suspected yield out of
                                      # the verdict (scorable: false) instead
                                      # of counting it as a clean pass
    cfg: ScoreConfig | None = None,
) -> dict
```

Scores one recording and returns the envelope. Provide either `stereo` or both
`caller` and `agent`. `expect="hold"` means the caller's speech is a
backchannel, a short acknowledgement like "mhm", and a correct agent keeps
talking through it. The two `max_*` thresholds tighten the pass criteria.
`echo_gate` is opt-in and off by default; `signals.echo` is always computed
and reported either way. Malformed or truncated WAVs raise a clean `ValueError` with the
ffmpeg export line to fix them.

### run_suite

```python
run_suite(
    *,
    suite: str = "barge-in",          # the only suite id (SUITE_ID)
    stack: str | None = None,
    scenarios_dir: str | None = None, # your scenario JSON labels
    audio_dir: str | None = None,     # your recordings, <scenario-id><suffix>
    suffix: str = ".example.wav",
    caller_channel: int = 0,
    agent_channel: int = 1,
    echo_gate: bool = False,          # see run_single
    cfg: ScoreConfig | None = None,
) -> dict
```

Runs a labelled battery and returns the envelope with a `suite` key. Defaults
to the bundled 8-scenario battery shipped inside the package, zero external
files. Point `scenarios_dir` and `audio_dir` at your own labelled set (for
example `corpus/suites/gold/scenarios` and `.../audio`). Suite audio must be
two-channel.

### dump_frames_for_input

```python
dump_frames_for_input(
    *,
    stereo: str | None = None,
    caller: str | None = None,
    agent: str | None = None,
    caller_channel: int = 0,
    agent_channel: int = 1,
    onset_sec: float | None = None,
    cfg: ScoreConfig | None = None,
) -> dict
```

The per-frame evidence behind every reported number: each channel's dBFS, VAD
activity, threshold, and noise floor, plus a self-describing `config` block.
Every reported signal is re-derivable by hand from this dump.

### LIMITS and SUITE_ID

`hotato.core.LIMITS` is the honest scope dict embedded in every envelope
(method, ceiling, best input, what it does not do). `SUITE_ID` is `"barge-in"`.

### ScoreConfig

Every threshold is an exposed parameter:

```python
from hotato._engine.score import ScoreConfig
from hotato._engine.vad import VADParams

cfg = ScoreConfig(
    frame_ms=20.0, hop_ms=10.0,
    yield_hangover_sec=0.20,       # agent quiet this long = yielded
    max_search_sec=3.0,            # yield search window after onset
    caller_proximity_sec=0.5,
    turn_end_silence_sec=0.20,
    premature_tolerance_sec=0.05,
    onset_min_run_sec=0.05,
    agent_onset_lookback_sec=0.10,
    caller_vad=VADParams(),        # rel_db=15.0, abs_gate_db=-60.0,
    agent_vad=VADParams(),         # hangover_sec=0.15, noise_percentile=0.10,
)                                  # dyn_margin_db=22.0, backend="energy"
```

`VADParams.backend` is `"energy"` (the deterministic reference behind every
published number) or `"neural"` (an optional Silero VAD cross-check via
`pip install 'hotato[neural]'`; without the extra it raises a clean
`BackendUnavailable`, never a silent fallback).

## hotato.report

Self-contained visual reports scored from the same measurements. All three
functions accept the full scoring parameter set (`stereo`, `caller`, `agent`,
`suite`, `scenarios_dir`, `audio_dir`, `suffix`, `caller_channel`,
`agent_channel`, `onset_sec`, `expect`, `stack`, `max_talk_over_sec`,
`max_time_to_yield_sec`, `cfg`) as keyword arguments.

```python
build_report_html(*, base: dict | None = None,
                  base_label: str | None = None, **kwargs) -> (str, dict)
build_report_md(*, base: dict | None = None,
                base_label: str | None = None, **kwargs) -> (str, dict)
write_report(path: str, fmt: str = "html", **kwargs) -> dict
```

`build_report_html` scores the input and returns `(html, envelope)`: one
self-contained file, inline CSS and SVG, per-event timelines, analytics, a
frame inspector, and print CSS for PDF. `build_report_md` mirrors it as
Markdown tables. `write_report` builds in `fmt` (`"html"` or `"md"`), writes to
`path`, and returns the envelope. Pass `base` (a previous envelope dict, for
example loaded from `hotato run --format json` output) to render per-scenario
regression deltas; `base_label` names it in the page.

```python
from hotato.report import write_report

env = write_report("report.html", suite="barge-in", stack="livekit")
```

## hotato.aggregate

Team mode: many run envelopes, one honest trend view.

```python
load_run_dir(dirpath: str, order: str = "mtime") -> dict
    # {"runs": [{"file", "path", "mtime", "env"}], "skipped": [...], "order"}
    # order: "mtime" (oldest first) or "name" (numeric prefix = explicit index)

aggregate_runs(runs: list, order: str = "mtime",
               skipped: list | None = None) -> dict
    # team envelope: kind "team-aggregate", runs, events_total,
    # talk_over_sec / seconds_to_yield distribution summaries (mean/median/p90),
    # pass_rate {latest, first, mean, direction}, pass_rate_over_time,
    # failure_classes, most_common_failure_class, skipped, exit_code 0.
    # Raises ValueError with fewer than 2 runs, never pads a trend.

build_team_section_html(agg: dict) -> str   # embeddable section
build_team_page_html(agg: dict) -> str      # full self-contained page
```

```python
from hotato.aggregate import load_run_dir, aggregate_runs, build_team_page_html

loaded = load_run_dir("runs/")
agg = aggregate_runs(loaded["runs"], order=loaded["order"],
                     skipped=loaded["skipped"])
html = build_team_page_html(agg)
```

## hotato.export

Research-grade flat files from the same scorer.

```python
run_export(
    *,
    out_dir: str,
    # plus the full scoring parameter set: stereo, caller, agent,
    # caller_channel, agent_channel, onset_sec, expect, stack, suite,
    # scenarios_dir, audio_dir, suffix, max_talk_over_sec,
    # max_time_to_yield_sec, cfg
) -> dict   # {"env", "events_rows", "frames_rows", "paths"}
```

Writes `events.csv` (one row per event, columns in
`hotato.export.EVENT_COLUMNS`), `frames.csv` (one row per VAD frame, columns in
`FRAME_COLUMNS`), and `envelope.json` into `out_dir` (created if missing).
Column meanings are documented in `#` comment lines at the top of each CSV.
Empty cell means not derivable, never fabricated.

## hotato.stackbench

Identical scenarios, your stack, comparable result files. No vendor numbers, no
leaderboard: every number is a measurement of recordings you provide.

### run_stackbench

```python
run_stackbench(
    *,
    stack: str,                       # one of BENCH_STACKS:
                                      # vapi | twilio | livekit | pipecat | generic
    recordings_dir: str,              # <scenario-id><suffix> WAVs
    scenarios_dir: str | None = None, # default: bundled battery
    suffix: str | None = None,        # default: auto-detected
                                      # (.wav, .stereo.wav, .example.wav)
    caller_channel: int = 0,
    agent_channel: int = 1,
    cfg: ScoreConfig | None = None,
) -> dict
```

Returns a result dict (`kind: "stack-benchmark"`) with the envelope fields plus
`config`, `scenarios {total, captured, not_captured}`, and `provenance`.
Scoring is `run_suite` unchanged. Scenarios with no matching recording are
listed under `not_captured`, never scored and never counted as failures. The
timestamp derives from input file mtimes, so the same inputs reproduce the same
result file.

### load_result, compare_results, render_comparison_md

```python
load_result(path: str) -> dict
    # loads and validates one result JSON; anything else is a clean ValueError

compare_results(inputs: Sequence[tuple[str, dict]]) -> dict
    # inputs: (path, loaded_result) pairs, at least two.
    # Compares the intersection of scenarios scored in EVERY input;
    # the rest is listed under "skipped". Deltas are signed differences
    # against the FIRST input. Returns kind "stack-benchmark-comparison"
    # with inputs, compared, skipped, per_scenario, medians.

render_comparison_md(cmp_env: dict) -> str
    # the comparison as Markdown tables
```

```python
from hotato.stackbench import load_result, compare_results, render_comparison_md

cmp = compare_results([(p, load_result(p)) for p in ("a.json", "b.json")])
print(render_comparison_md(cmp))
```

## Pytest fixture

Installed automatically via the `pytest11` entry point (or load explicitly with
`-p hotato.pytest_plugin`). Inert unless used.

```python
def test_call_yields(hotato_score):
    env = hotato_score(stereo="call.wav", expect="yield")
    assert env["summary"]["regression"] is False
    assert env["events"][0]["verdict"]["seconds_to_yield"] < 1.0
```

`hotato_score(**kwargs)` takes the same keyword arguments as `run_single`; pass
`suite="barge-in"` (plus `run_suite` keywords) to score a battery instead. It
returns the envelope and never asserts for you.

Session gate flags: `pytest --hotato-suite` runs the battery after your tests
and fails the session (exit 1) on a regression; `--hotato-suite-scenarios DIR`
and `--hotato-suite-audio DIR` point it at your own labelled set. Detail:
`docs/PYTEST.md`.

## MCP tool

`hotato-mcp` (or `python -m hotato.mcp_server`) speaks MCP over stdio and
exposes exactly one tool, `voice_eval_run`, returning the identical envelope.
Install: `uvx --from "hotato[mcp]" hotato-mcp`.

Parameters, all optional:

| Parameter | Type | Default | Meaning |
| --- | --- | --- | --- |
| `stereo` | str | None | two-channel WAV path |
| `caller` | str | None | mono caller WAV (with `agent`) |
| `agent` | str | None | mono agent WAV (with `caller`) |
| `suite` | str | None | `"barge-in"` to run the bundled battery |
| `stack` | str | `"generic"` | livekit, pipecat, vapi, or generic |
| `expect` | str | `"yield"` | `"yield"` or `"hold"` |
| `onset_sec` | float | None | caller onset hint |
| `caller_channel` | int | 0 | caller channel index |
| `agent_channel` | int | 1 | agent channel index |
| `max_talk_over_sec` | float | None | pass threshold |
| `max_time_to_yield_sec` | float | None | pass threshold |
| `report_path` | str | None | also write the HTML report here; the envelope then carries `report_path` (absolute) |

## Exit codes and errors

Envelopes carry `exit_code`: 0 all scorable events passed, 1 regression.
`hotato.core.process_exit_code(env)` maps the finished envelope to the CLI
process exit: a single-recording run whose event is not scorable (silent
caller with no onset label, or agent silent at onset; the reason is in
`not_scorable_reason`) exits 2, because that is an input problem, not an
agent verdict. Suite runs count such events in `summary.not_scorable` and
keep their 0/1 semantics. Malformed input (bad WAV, out-of-range channel,
negative onset, unknown suite or stack) raises `ValueError`, which the CLI
surfaces as exit code 2. Nothing is ever scored from a file the scorer could
not fully read.


================================================================================
FILE: docs/SUBMITTING.md
================================================================================

# Submitting a recording

One consented, de-identified, human-labeled real call makes this eval more
credible. This is the whole path: record, label, validate, submit, and what
happens at intake.

The scorer measures speech energy over time. Your human label supplies the
meaning: which onset was a real bid for the floor, and whether a well-behaved
agent yields there. Energy is not intent, so the label is the ground truth,
and the label is what you are contributing.

## 1. Record dual-channel

Caller on one channel, agent on the other, separated at capture: the two legs
of a SIP bridge, or two streams that never mix. Real separation makes overlap
a fact of the recording, exact to the sample. Save a WAV at the call's native
sample rate (8000 Hz telephony, 16000 Hz wideband).

Consent, PII, and PHI rules are normative in
[`CORPUS-GOVERNANCE.md`](CORPUS-GOVERNANCE.md): documented consent from every
audible party, identifiers redacted with same-duration tone or silence so the
timing survives, no PHI ever. Read it before you record; it includes a
reusable release paragraph.

## 2. Label it

The label is a JSON file next to your WAV: the bundled scenario shape plus
provenance and attestation. Spec of record:
[`corpus/label.schema.json`](../corpus/label.schema.json). Worked example:
[`corpus/examples/sample-contribution.json`](../corpus/examples/sample-contribution.json).

Minimal example:

```json
{
  "id": "call-021-address-change-interrupt",
  "title": "Caller interrupts the shipping recap to change the address",
  "category": "should_yield",
  "source_type": "real-call",
  "audio": "call-021-address-change-interrupt.wav",
  "channels": { "caller_channel": 0, "agent_channel": 1 },
  "sample_rate": 8000,
  "duration_sec": 9.2,
  "caller_onset_sec": 3.15,
  "expected": {
    "yield": true,
    "max_time_to_yield_sec": 0.7,
    "max_talk_over_sec": 0.8
  },
  "license": "MIT",
  "attestation": {
    "contributor": "Your Name <you@example.com>",
    "consent_on_file": true,
    "pii_removed": true,
    "no_phi": true,
    "right_to_release_mit": true
  }
}
```

For a hold case (the caller backchannels "mm-hm" and a good agent keeps
talking): set `category` to `should_not_yield`, `expected.yield` to `false`,
and both `max_*` bounds to `null`.

Label only what you can defend by hand. `caller_onset_sec` is required;
`reference_render` segment timings are welcome wherever you have them, and
the harness reports error for exactly the signals you supply.

## 3. Validate locally

```bash
python3 corpus/validate.py your_label.json
```

The WAV resolves next to the label via its `audio` field. Pass the audio path
as a second argument if it lives elsewhere:

```bash
python3 corpus/validate.py your_label.json your_recording.wav
```

PASS (exit code 0) means the pair conforms: fields well-typed, category and
expected bounds consistent, timings in range, attestation affirmed, and the
audio a readable WAV with two or more channels that matches the label. A
human still reviews consent and PII before anything merges.

## 4. Submit

Two doors, one intake:

- **Issue form.** Open
  ["Corpus submission: labeled recording"](https://github.com/attenlabs/hotato/issues/new?template=corpus_submission.yml).
  Attach the WAV as a .zip (GitHub accepts .zip attachments; a raw .wav
  upload is rejected) or link it, and paste your label JSON in the
  description.
- **Pull request.** Add the label and WAV under [`corpus/`](../corpus/), for
  example `corpus/call-021-address-change-interrupt.json` plus its `.wav`.
  State the source type in the PR body and confirm the attestation, per
  [`CONTRIBUTING.md`](../CONTRIBUTING.md).

## What maintainers do at intake

1. **Validate.** Re-run `corpus/validate.py` on the pair and check the label
   against [`corpus/label.schema.json`](../corpus/label.schema.json).
2. **Dedupe.** Hash the audio and compare call details against existing
   clips. The corpus stays small and high-integrity; every clip earns its
   place.
3. **Normalize.** Slug the `id`, align filenames, confirm the channel map and
   declared sample rate. Timing is preserved exactly; audio is never edited
   in a way that shifts a label.
4. **Add to a suite.** Register the clip so the benchmark harness scores it
   and reports millisecond error distributions and the yield confusion matrix
   ([`BENCHMARK.md`](BENCHMARK.md)).
5. **Credit.** `attestation.contributor` is the credit of record, and
   contributors are named in the changelog when their clip lands.

Removal requests are honored in the next release, per
[`CORPUS-GOVERNANCE.md`](CORPUS-GOVERNANCE.md).


================================================================================
FILE: docs/CORPUS-GOVERNANCE.md
================================================================================

# Corpus governance

How this project assembles, labels, and reports on a small corpus of **real**
labeled voice-agent calls that augments the bundled synthetic fixtures.

This document is normative. To contribute real audio, follow it. To understand
what the numbers in a report mean, read the "Validity metrics" section at the end.

---

## Why real recordings matter

The synthetic fixtures in `src/hotato/data/` are a deliberately honest floor. They
are rendered from known segment timings, so the ground truth is exact and the eval
is runnable by anyone, offline, in seconds. That makes them ideal for regression
tests and for demonstrating that the scorer behaves as specified.

Production validity is a different question. Synthetic audio has clean onsets,
controlled overlap, and no room acoustics or codec artifacts beyond what we inject.
A scorer that looks flawless on synthetic fixtures still needs proving on a real
8 kHz call with echo bleed, cross-talk, and a caller who trails off mid-word.

A small, carefully labeled corpus of real calls closes that gap. It is the
difference between "the tool does what the spec says" and "the tool measures what
actually happens on a phone line." We keep the corpus small and high-integrity on
purpose: every clip is consented, de-identified, and human-labeled. Ten trustworthy
calls beat ten thousand scraped ones.

---

## Scope and boundaries

- The corpus measures the **audio timing of turn-taking**: whether the agent
  stopped talking when the caller interrupted (yield), how long both talked at
  once (talk-over), and whether the agent talked through a short acknowledgement
  like "mhm" (backchannel handling).
- Labels are about **events in time**: did the agent stop, and when did the
  caller start.
- Everything here is redistributed under the project's MIT license. A clip enters
  the corpus only if it can be released under MIT with consent.

---

## Consent / release template

Every party audible on a real recording must agree, in a form you can produce on
request, to its inclusion. The following paragraph is intentionally short and
reusable. Send it to a caller and/or agent-operator and keep the signed or written
affirmative reply on file. Fill the brackets before use.

> **Recording release, hotato open test corpus**
>
> I, [name or role], took part in the audio recording made on [date] and described
> as [short description]. I understand this recording will be used as a test
> fixture in *hotato*, an open-source project, and grant a perpetual, worldwide,
> royalty-free right to store, redistribute, and publish the recording and its
> derived timing labels as part of that project's MIT-licensed corpus. I confirm
> that the recording contains no confidential, personal, health, or account
> information, or that any such information has been removed or replaced before
> submission. I understand I can request removal of the recording from future
> releases at any time by contacting the maintainers.
>
> Signed / affirmed: [name], [date], [contact]

Notes:

- Get consent from **all** identifiable parties: the human caller, and whoever
  operates the agent side when that is a real person or a proprietary system whose
  owner must agree.
- Consent must be **affirmative and documented**. A generic call-center "this call
  may be recorded" notice is not sufficient for redistribution under an open
  license.
- Honor removal requests promptly in the next release.

---

## PII policy

Real audio must be de-identified before it is committed. In order of preference:
**do not capture it, then remove it, then reject the clip.**

Strip or redact:

- **Names** of any real individual (caller, agent, third parties mentioned).
- **Phone numbers**, extensions, and any dialed digits (DTMF tones included).
- **Postal and email addresses.**
- **Account identifiers**: customer numbers, order numbers, card numbers, policy
  numbers, ticket IDs, anything that keys back to a real person or account.
- Any other free-text detail that, alone or combined, re-identifies someone.

Strongly preferred practice:

- Use **synthetic or role-played content**. Actors reading a script hit the same
  turn-taking dynamics (interruption, correction, backchannel) with zero PII risk.
  This is the cleanest source of real acoustics.
- **No PHI.** Do not submit any recording containing protected health information,
  regardless of consent. It is out of scope for this project.

Redaction guidance:

- Replace spoken PII with a short tone or silence of the **same duration** so the
  turn-taking timing is preserved. Cutting samples out shifts every onset after the
  edit and corrupts the labels.
- Redact the audio in **all channels**. A name muted on the mixed channel but
  audible on the caller channel is still present.
- Keep a private note of what was redacted and when (timestamps). Never commit the
  removed content.

---

## Data-handling policy

- **The tool runs local-only.** `hotato` reads local audio and writes local
  reports, offline, with no telemetry. Contributing to the corpus is a deliberate,
  manual act of committing a de-identified, consented clip.
- **Storage.** Corpus audio lives in the repository alongside its scenario JSON,
  under the same MIT terms as the rest of the project. Keep working copies local
  until they are cleared for release; do not stage real audio in third-party cloud
  buckets, shared drives, or chat tools during preparation.
- **Raw originals stay out of the repo.** Only the final, redacted, consented clip
  is committed. The un-redacted original is the contributor's responsibility to
  secure or destroy per their own obligations.
- **Contributor attestation.** Each real-audio PR must include a short statement
  from the contributor affirming: (a) consent is on file for every party, (b) PII
  has been removed per the policy above, (c) the clip contains no PHI, and (d) the
  contributor has the right to release it under MIT. This attestation is part of
  the merge record.

---

## Validity metrics: what we publish

The corpus exists to quantify measurement quality honestly. That shapes how results
are reported.

We publish:

- **Per-signal measurement error, in milliseconds.** For each labeled call we
  compare the scorer's measured event times to the human labels and report the
  error distribution per signal:
  - `time_to_yield`: signed and absolute error in ms between measured and labeled
    floor-yield.
  - `talk_over`: error in ms on measured overlap duration.
  - onset agreement: error in ms between measured caller onset and
    `caller_onset_sec`.
  We report these as distributions (median, spread, worst case), so a reader sees
  the real shape of the error.
- **A `did_yield` confusion matrix.** Against the human `category` (`should_yield`
  / `should_not_yield`) label, we report the four-cell matrix: correct yields,
  correct holds, false yields (yielded on a backchannel), and missed yields (kept
  talking over a real interruption). The two error cells are the ones operators
  feel, so we surface them directly.

The distribution and the four-cell matrix are the report. A single percentage would
collapse the two failure modes into one number and hide the trade-off between them,
so we keep them separate. The scorer works on speech energy over time, so validity
is timing agreement and decision agreement.

### Reading a corpus report

A healthy result looks like: tight millisecond error distributions on
`time_to_yield` and `talk_over`, onset agreement within a small ms band, and a
confusion matrix whose off-diagonal (false-yield / missed-yield) cells are small
and individually inspectable. A poor result looks like wide error spread or a heavy
off-diagonal, and it should point you toward a concrete fix in the call.

Where a real call fails, the fix map names candidate remedies. A learned
engagement-control layer is one of them, referenced at that level. Audio-only is
its weaker modality and a fully local offline model remains a known gap. The corpus
is here to measure reality.


================================================================================
FILE: docs/BENCHMARK-STACKS.md
================================================================================

# Benchmarking voice stacks with hotato

`hotato benchmark` scores the recordings you captured by running one fixed
scenario set through YOUR configured voice stack. Every stack is scored on the
same scenarios, the same labels, and the same thresholds, so the result files
are directly comparable: same battery, same measurements, same exposed config.

hotato measures timing on the recordings it is given. It ships no vendor
numbers, no leaderboard, and no accuracy percentage. Results depend on your
stack configuration and your captures; a benchmark result describes YOUR
setup, on the day you captured it.

## 1. Pick the scenario set

The bundled 8-scenario barge-in battery is the default. Any scenarios dir
works the same way, for example the larger tiered suites:

```
corpus/suites/gold/scenarios
corpus/suites/silver/scenarios
```

Each scenario JSON carries the caller transcript, the caller onset timing,
and the pass thresholds. The corpus suites also ship each scenario's caller
stimulus as `audio/<id>.caller.wav`.

## 2. Capture the battery through your stack

For every scenario, drive the caller side through your stack and record the
call dual-channel (caller on channel 0, agent on channel 1):

- Corpus suites: play the shipped `<id>.caller.wav` stimulus into your stack.
  For the bundled battery, reproduce the caller side from each scenario's
  transcript and `caller_onset_sec`.
- `hotato setup --stack livekit` or `--stack pipecat` prints the exact
  dual-channel recording scaffold for your infra.
- `hotato capture --stack vapi --call-id <id>` or
  `hotato capture --stack twilio --recording-sid RE...` pulls the finished
  dual-channel recording for you.

Name each recording by its scenario id, one directory per stack:

```
captures/livekit/01-hard-interruption.wav
captures/livekit/02-backchannel-mhm.wav
...
captures/vapi/01-hard-interruption.wav
```

## 3. Run the benchmark

```bash
hotato benchmark --stack livekit --recordings captures/livekit --out livekit.json
hotato benchmark --stack vapi    --recordings captures/vapi    --out vapi.json
```

Each result JSON records the stack, the scenario set, every measured signal
(the same events `hotato run` produces), the exact scoring thresholds, and a
provenance block: who ran it, on which files, with each file's mtime. The
result timestamp comes from the input files, not the wall clock, so the same
inputs reproduce the same result.

Scenarios without a matching recording are listed under `not_captured`. They
are never scored and never counted as failures. The exit code is 0 when the
run completes; add `--fail-on-regression` to exit 1 when a scored event fails
its scenario thresholds.

## 4. Compare stacks

```bash
hotato benchmark compare livekit.json vapi.json
```

Side by side, per scenario: yielded, talk-over seconds, and time-to-yield
seconds for each input, with signed deltas against the first file, plus
summary medians. If the files cover different scenario sets, the intersection
is compared and everything else is listed as skipped, with the files it was
missing from.

## What the numbers are

Reproducible timing measurements of the recordings you provided, scored by
the same engine as `hotato run`, with every threshold exposed in the result.
They tell you how your configuration of a stack handled the battery you
captured. They are not a vendor ranking, and no accuracy percentage appears
anywhere.


================================================================================
FILE: docs/BENCHMARK.md
================================================================================

# Benchmark methodology

`hotato` ships a reproducible measurement-error harness: `src/hotato/benchmark.py`.
Given a set of labelled dual-channel recordings, it reports how far the scorer's
measured event times land from the rendered or labelled ground truth, and how its
yield/hold decisions line up with the human labels. That is the whole output.

```bash
PYTHONPATH=src python3 -m hotato.benchmark
```

This runs on every synthetic fixture in the checkout (the bundled battery, the
`examples/` reference set, and the deliberately-bad `funnel-demo` set), prints a
markdown table, and writes `benchmark-report/measurement-error.json` and
`benchmark-report/measurement-error.md`.

The harness is a standalone measurement tool you run against fixtures. The `hotato`
CLI scores a single call or the battery.

---

## What it measures

For every `(recording, label)` pair it reports two things.

### 1. Per-signal measurement error, in milliseconds

For each timing signal it computes `|measured - rendered|` in ms, where `rendered`
is the exact value the synthetic fixture was rendered from (its `reference_render`
block) or the value a contributor labelled by hand:

| signal | measured | rendered/true reference |
|---|---|---|
| **caller onset** | onset the VAD detects (the scorer is given no onset label) | start of the first caller segment |
| **time to yield** | seconds from caller onset to the agent going quiet | the agent's in-progress turn end minus the caller onset |
| **response gap** | endpointing dead-air from the caller's turn end to the agent's next onset | the fixture's rendered response gap |

Errors are reported as a distribution: median, mean, worst case, best case, and n.
A signal is scored where a rendered or true reference exists and the scorer
produced a value; a missing reference yields a `-`. (The echo-of-agent fixture has
no independent caller speech, so it has no onset or yield error: an honest gap.)

Onset is measured in **detect mode** (the scorer gets no onset hint), so it is a
real test of the onset detector. Yield, talk-over, response gap, and `did_yield`
are measured in **label mode** (the scorer is given the human `caller_onset_sec`,
exactly as the shipped battery runs), so those are the numbers a user actually
sees.

### 2. A `did_yield` confusion matrix

Against the `should_yield` / `should_not_yield` label (each scenario's
`expected.yield`), it reports the four cells:

|  | measured **did_yield** | measured **held floor** |
|---|---|---|
| **should_yield** | correct yield | **missed yield** |
| **should_not_yield** | **false yield** | correct hold |

The two off-diagonal cells (missed yields, and false yields) are the failures an
operator feels, so the report surfaces them directly.

---

## Why milliseconds and a matrix

The report is a per-signal error distribution plus a four-cell confusion matrix.
The cells stay separate because a missed yield and a false yield are different
failures with different fixes; averaging them into one number would hide which
one you have. The distribution and the matrix are the report. This is the rule the corpus governance doc enforces
(`docs/CORPUS-GOVERNANCE.md`, "Validity metrics"), applied to the tooling.

The reported error is what the default shipped config measures, so it is the number
a real user gets. On the synthetic fixtures the yield error equals the exposed VAD
hangover and the onset/gap error is one frame hop, both documented `ScoreConfig`
parameters. Set the hangover to zero
(`caller_vad.hangover_sec = agent_vad.hangover_sec = 0`) and every signal collapses
to within one hop of the rendered ground truth; the test suite asserts exactly
this, so the claim is checkable.

The scorer works on speech energy over time, so the harness reports timing and
decisions.

---

## Reproducing it

```bash
# render the synthetic fixtures deterministically (sha256-seeded; byte-identical
# on any machine), then run the harness over them
python3 examples/render_examples.py
PYTHONPATH=src python3 -m hotato.benchmark
```

The report has no wall-clock timestamp and the render is deterministic, so two runs
on the same code produce byte-identical artifacts. The JSON carries a `config`
snapshot of every threshold that produced the numbers, so a reader can re-derive
any value. `tests/test_benchmark.py` pins the whole thing: it asserts the harness
runs, the confusion matrix matches the known rendered behaviour, and every ms-error
is within its known, config-derived tolerance.

The synthetic fixtures are a floor: deterministic rendered audio with exact known
timings. They prove the scorer behaves as specified and guard against regressions.
See `examples/README.md` and `docs/CORPUS-GOVERNANCE.md`.

---

## Extending to real recordings (bring your own labelled data)

The synthetic floor tells you the scorer does what the spec says. Real recordings
tell you it measures what happens on an actual phone line. The harness runs on your
own labelled data with no code change:

```bash
PYTHONPATH=src python3 -m hotato.benchmark \
  --scenarios path/to/your/scenarios \
  --audio     path/to/your/audio \
  --name      my-corpus \
  --out       my-benchmark-report
```

- `--scenarios` is a directory of scenario JSON labels, same shape as
  `src/hotato/data/scenarios/*.json`. Each needs an `id`, an `expected.yield`
  label, and whatever reference timings you can defend: a `reference_render` with
  segment timings (as the synthetic fixtures carry), or at minimum a hand-labelled
  `caller_onset_sec` and the true event times you want error scored against. Supply
  a reference only where you have ground truth; the harness reports error for those
  signals and leaves the rest blank.
- `--audio` is a directory of dual-channel `<id>.example.wav` recordings (caller on
  channel 0, agent on channel 1). Record two physically separated channels whenever
  you can: that is what makes overlap ground-truthable.

When `--scenarios` is given, only that set is scored, so you get a clean report for
your corpus alone.

The honest path to a real-model number is manual and consented: bring your own
labelled audio. Contributing real audio is governed by `docs/CORPUS-GOVERNANCE.md`
(consent, PII, data-handling) and the pipeline in `corpus/` (a labelling schema and
a validator). Synthetic fixtures keep their synthetic label.


================================================================================
FILE: docs/RELEASE-CHECKLIST.md
================================================================================

# Release checklist

Run top to bottom for every release. Each item is a gate: green means proceed.

## Code and tests

- [ ] `python3 -m pytest -q` fully green on a clean checkout.
- [ ] `python3 sync_engine.py --check` passes: the vendored `_engine` is byte-identical to upstream.
- [ ] CI is green on the release commit (`.github/workflows/tests.yml`).
- [ ] Version bumped in EVERY lockstep site: `pyproject.toml`, `src/hotato/__init__.py` (`__version__` -- this is what `hotato --version`, `hotato describe`, and stackbench provenance self-report), `server.json` (both `version` fields), `CITATION.cff` (`version` + `date-released`), the `llms.txt` version line; `CHANGELOG.md` has a dated entry for it. `tests/test_version_lockstep.py` gates the ones tests can see.

## README and assets

- [ ] Regenerate README assets before release: `python3 scripts/render_readme_assets.py`.
- [ ] Verify the README screenshot renders: `docs/assets/hotato-demo-report.png` is current, shows the failing demo summary, a timeline, and a fix card, and displays correctly in the GitHub README preview.
- [ ] README badges resolve (the `tests.yml` workflow badge reflects the real run).
- [ ] Verify the site hero matches the README: same tagline, same demo screenshot, same install command on hotato.dev.

## Adapters

- [ ] Adapter last-verified dates are fresh in `docs/ADAPTER-STATUS.md`: re-verify each capture path against the vendor's current API docs and update the date column.
- [ ] Fixmap knob names match the same verified APIs (`src/hotato/fixmap.py`).

## Package

- [ ] Build sdist and wheel from a clean tree; install the wheel in a fresh venv and run `hotato run --suite barge-in`.
- [ ] Publish to PyPI.
- [ ] Verify `uvx hotato demo` works from the published package (fresh machine or cleared uv cache): it renders and opens the failing demo report.
- [ ] Verify `uvx hotato doctor` (self-test path) works from the published package.
- [ ] Tag the release commit and push the tag; GitHub release notes point at the CHANGELOG entry.

## After release

- [ ] Smoke-test the README quickstart commands exactly as written.
- [ ] File follow-ups for anything that needed a manual touch, so the next release is one pass.


================================================================================
FILE: src/hotato/schema/envelope.v1.json
================================================================================

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://hotato.dev/schema/envelope.v1.json",
  "title": "hotato result envelope, schema_version 1",
  "description": "The single machine-readable shape emitted by the CLI, the one MCP tool, and the suite runner. Additive-only: new signal dimensions and sub-verdicts may appear as extra keys without a version bump, so consumers must ignore unknown fields. The stable core below is frozen for schema_version 1.",
  "type": "object",
  "required": ["tool", "schema_version", "mode", "stack", "offline", "engine", "limits", "summary", "events", "fix_map", "funnel", "exit_code"],
  "additionalProperties": true,
  "properties": {
    "tool": { "const": "hotato" },
    "schema_version": { "const": "1" },
    "mode": { "enum": ["single", "suite"] },
    "suite": { "type": "string" },
    "stack": { "type": "string" },
    "offline": { "const": true },
    "exit_code": { "enum": [0, 1] },
    "engine": {
      "type": "object",
      "required": ["name", "version", "upstream"],
      "additionalProperties": true,
      "properties": {
        "name": { "type": "string" },
        "version": { "type": "string" },
        "upstream": { "type": "string" }
      }
    },
    "limits": {
      "type": "object",
      "required": ["method", "accuracy_claim", "ceiling", "best_input", "does_not_do", "scope", "offline"],
      "additionalProperties": true,
      "properties": {
        "method": { "type": "string" },
        "accuracy_claim": { "type": "null", "description": "HONESTY INVARIANT: always null. This tool never emits an accuracy percentage." },
        "ceiling": { "type": "string" },
        "best_input": { "type": "string" },
        "does_not_do": { "type": "array", "items": { "type": "string" } },
        "scope": { "type": "string" }
      }
    },
    "summary": {
      "type": "object",
      "required": ["events", "passed", "failed", "regression"],
      "additionalProperties": true,
      "properties": {
        "events": { "type": "integer", "minimum": 0 },
        "passed": { "type": "integer", "minimum": 0 },
        "failed": { "type": "integer", "minimum": 0 },
        "regression": { "type": "boolean" },
        "not_scorable": { "type": "integer", "minimum": 1, "description": "Additive: count of events that could not be judged (malformed input, see each event's not_scorable_reason). Present ONLY when at least one event is not scorable; a run of valid recordings emits the exact pre-existing summary shape. Not-scorable events count in events but in neither passed nor failed." }
      }
    },
    "events": {
      "type": "array",
      "items": { "$ref": "#/definitions/event" }
    },
    "fix_map": {
      "type": "array",
      "items": { "$ref": "#/definitions/fix_map_entry" }
    },
    "funnel": {
      "oneOf": [
        { "type": "null" },
        {
          "type": "object",
          "required": ["reason", "pointer"],
          "additionalProperties": true,
          "properties": {
            "reason": { "type": "string" },
            "pointer": { "$ref": "#/definitions/engagement_pointer" }
          }
        }
      ]
    }
  },
  "definitions": {
    "event": {
      "type": "object",
      "required": ["event_id", "expected_yield", "verdict", "measurements"],
      "additionalProperties": true,
      "dependencies": {
        "scorable": ["not_scorable_reason"],
        "not_scorable_reason": ["scorable"]
      },
      "properties": {
        "event_id": { "type": "string" },
        "scorable": { "const": false, "description": "Additive: present ONLY on events that cannot be judged (and then always false); scorable events omit the key entirely, so every envelope for a valid recording is byte-identical to schema_version 1 before this key existed. A not-scorable event is excluded from passed/failed counting and can never carry a normal pass or fail." },
        "not_scorable_reason": { "type": "string", "description": "Plain-language reason the event cannot be judged (for example no detectable caller speech with no onset provided, or the agent silent at the caller onset under a should-yield expectation). Present exactly when scorable is present." },
        "scenario_id": { "type": ["string", "null"] },
        "title": { "type": ["string", "null"] },
        "category": { "type": ["string", "null"] },
        "expected_yield": { "type": "boolean" },
        "verdict": {
          "type": "object",
          "required": ["passed", "did_yield", "seconds_to_yield", "talk_over_sec", "reasons"],
          "additionalProperties": true,
          "properties": {
            "passed": { "type": "boolean" },
            "did_yield": { "type": "boolean" },
            "seconds_to_yield": { "type": ["number", "null"] },
            "talk_over_sec": { "type": "number" },
            "reasons": { "type": "array", "items": { "type": "string" } }
          }
        },
        "measurements": { "type": "object", "additionalProperties": true },
        "signals": {
          "type": "object",
          "additionalProperties": true,
          "description": "Namespaced signal bus. barge_in is present whenever scoring ran; other dimensions (latency, and future overlap/resume/backchannel) are additive.",
          "properties": {
            "barge_in": {
              "type": "object",
              "required": ["did_yield", "time_to_yield_sec", "talk_over_sec"],
              "additionalProperties": true
            },
            "latency": {
              "type": "object",
              "additionalProperties": true,
              "properties": {
                "response_gap_sec": { "type": ["number", "null"] },
                "premature_start_sec": { "type": ["number", "null"] }
              }
            },
            "resume": {
              "type": "object",
              "description": "Post-yield behaviour on the agent VAD track; present only when the agent yielded. resumed: did the agent start speaking again within the window after the yield. resume_gap_sec: seconds from the yield to that fresh onset (null when it did not resume). restart_suspected: the longest post-resume run is unusually long, a timing proxy for re-answering from the top (a transcript repeat is out of scope).",
              "additionalProperties": true,
              "properties": {
                "resumed": { "type": "boolean" },
                "resume_gap_sec": { "type": ["number", "null"] },
                "restart_suspected": { "type": "boolean" }
              }
            }
          }
        },
        "fix": {
          "oneOf": [ { "type": "null" }, { "$ref": "#/definitions/fix" } ]
        }
      }
    },
    "fix": {
      "type": "object",
      "required": ["fix_class", "title", "detail", "knob", "pointer"],
      "additionalProperties": true,
      "properties": {
        "fix_class": { "enum": ["config", "engagement-control"] },
        "title": { "type": "string" },
        "detail": { "type": "string" },
        "knob": {
          "oneOf": [
            { "type": "null" },
            {
              "type": "object",
              "required": ["stack", "parameter", "direction", "trade_off"],
              "additionalProperties": true
            }
          ]
        },
        "pointer": {
          "oneOf": [ { "type": "null" }, { "$ref": "#/definitions/engagement_pointer" } ]
        }
      }
    },
    "fix_map_entry": {
      "type": "object",
      "required": ["event_id", "fix_class", "title", "detail", "knob", "pointer"],
      "additionalProperties": true,
      "properties": {
        "event_id": { "type": "string" },
        "scenario_id": { "type": ["string", "null"] },
        "fix_class": { "enum": ["config", "engagement-control"] },
        "title": { "type": "string" },
        "detail": { "type": "string" }
      }
    },
    "engagement_pointer": {
      "type": "object",
      "description": "VENDOR-NEUTRAL pointer for the engagement-control fix class. Names the problem class and the KIND of fix (a learned engagement-control / addressee-detection layer); names no vendor, no product, and nothing you can adopt/license/buy, and carries no numbers.",
      "required": ["layer", "what", "honest_scope"],
      "additionalProperties": true,
      "properties": {
        "layer": { "type": "string" },
        "what": { "type": "string" },
        "honest_scope": { "type": "string" }
      }
    }
  }
}
