================================================================================
FILE: README.md
================================================================================

<div align="center">

<pre>
        ) )   ( (
       ( (     ) )
██╗  ██╗ ██████╗ ████████╗ █████╗ ████████╗ ██████╗ 
██║  ██║██╔═══██╗╚══██╔══╝██╔══██╗╚══██╔══╝██╔═══██╗
███████║██║   ██║   ██║   ███████║   ██║   ██║   ██║
██╔══██║██║   ██║   ██║   ██╔══██║   ██║   ██║   ██║
██║  ██║╚██████╔╝   ██║   ██║  ██║   ██║   ╚██████╔╝
╚═╝  ╚═╝ ╚═════╝    ╚═╝   ╚═╝  ╚═╝   ╚═╝    ╚═════╝ 
</pre>

<h1>hotato</h1>

</div>

<p align="center"><b>Open-source, self-hosted conversation QA for voice agents.</b></p>

<p align="center">
  Score every call across five dimensions reported on their own (outcome, policy, conversation, speech, reliability), with the evidence behind every result. Deterministic checks and the model-judged rubric each score on their own lane.<br>
  Runs offline &middot; MIT &middot; zero dependencies.
</p>

<p align="center">
  <a href="https://pypi.org/project/hotato/"><img alt="PyPI version" src="https://img.shields.io/pypi/v/hotato.svg"></a>
  <a href="https://pypi.org/project/hotato/"><img alt="PyPI monthly downloads" src="https://img.shields.io/pypi/dm/hotato.svg"></a>
  <a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-blue.svg"></a>
  <img alt="Python 3.9 to 3.13" src="https://img.shields.io/badge/python-3.9%20to%203.13-blue.svg">
  <img alt="offline: yes" src="https://img.shields.io/badge/offline-yes-blue.svg">
  <img alt="runtime deps: zero" src="https://img.shields.io/badge/runtime%20deps-zero-blue.svg">
  <a href="https://github.com/attenlabs/hotato/actions/workflows/tests.yml"><img alt="tests" src="https://github.com/attenlabs/hotato/actions/workflows/tests.yml/badge.svg"></a>
</p>

Your voice agent passes every text assertion and still loses the call. It talks over the caller, skips a required disclosure, or confirms a refund that never posted. hotato scores the conversation and the outcome, with the timing evidence behind every flag, then turns a caught bug into a CI contract so it stays caught.

## Quickstart: one command, on your machine

```bash
pip install hotato && hotato start --demo
```

Already have [uv](https://docs.astral.sh/uv/)? Zero-install, same command:

```bash
uvx hotato start --demo
```

One command sweeps two bundled calls a provider's default agent failed, writes the candidate dashboard, and turns one missed-interruption moment into a demo failure contract it verifies on the spot, all offline:


```
[start] demo: swept 2 bundled calls, 5 candidate moments; wrote hotato-sweep.json, hotato-sweep.html, hotato-no-single-threshold.svg, contracts/demo-missed-interruption.hotato/contract.json
hotato start: swept the 2 bundled demo calls offline.
  sweep dashboard: hotato-sweep.html
  demo contract:   contracts/demo-missed-interruption.hotato
  verified contract: FAIL as expected -- the demo call missed the interruption
  [ ... then the exact next commands: promote a candidate, gate it in CI, re-check it ... ]
```

Open `hotato-sweep.html`. The candidate moments come ranked by how far the timing missed, each with a hear-the-bug playhead. This is the artifact you screenshot into a PR:

<p align="center">
  <img src="https://raw.githubusercontent.com/attenlabs/hotato/main/docs/assets/sweep-dashboard.png" alt="The hotato candidate dashboard: 5 candidate moments ranked by salience across 2 calls, each card showing the caller and agent timeline, an embedded hear-the-bug audio playhead, and promote-as-yield / promote-as-hold / ignore buttons." width="820">
</p>
<p align="center"><sub><code>hotato-sweep.html</code> &middot; candidate moments ranked by salience, each with a playhead that sweeps the timeline in sync with the embedded audio. Candidates you review and label into your verdict.</sub></p>

Point it at your own recording to walk the same loop end to end:

```bash
hotato start --stereo my-call.wav          # trust -> scan -> review -> label -> contract
```

Every command takes a two-channel recording (caller on one channel, agent on the other). A mono file or a bad export is marked NOT SCORABLE, so every verdict rests on inputs that carry the timing evidence.

## What hotato catches: five dimensions, reported on their own

<p align="center">
  <a href="https://hotato.dev"><img src="https://raw.githubusercontent.com/attenlabs/hotato/main/.github/banner.png" alt="A hotato scorecard: one call graded across outcome, policy, conversation, speech, and reliability, deterministic checks kept separate from the model-judged rubric, each dimension scored on its own." width="840"></a>
</p>


A voice agent can pass every text assertion and still lose the call: the refund never fires, the recording disclosure gets skipped, the caller gets talked over. `hotato test run` grades one call against a conversation-test file across five dimensions, each kept apart in its own count:

- **Outcome** &middot; did the job get done, graded on tool-call and state evidence, not the transcript's say-so.
- **Policy** &middot; required disclosures, PII handling, and the compliance phrases your team owns.
- **Conversation** &middot; the deterministic turn-taking core: did the agent yield when the caller took the floor, and how fast.
- **Speech** &middot; response latency and the timing around each turn.
- **Reliability** &middot; pass@1 / pass@k / pass^k across repeated runs with a Wilson interval, so a flaky check reads as flaky.

Two lanes stay structurally separate. Deterministic checks (phrase, PII, policy, tool-call, sequence, latency, outcome, all pure regex, checksum, and span-lookup) live behind a wall from the model-judged rubric lane. A rubric verdict is `deterministic: false` and advisory, kept in its own count; the deterministic lane sets the gate. Each dimension keeps its own line in the output, including `--format json`, and the scored-output schemas reject an `overall_score` key outright.

```bash
hotato scenario init refund-check --out conversation-test.yaml   # a starter you edit for your own call
hotato test run conversation-test.yaml --agent support-bot
```

```
success: FAIL  (required: all_deterministic_assertions_pass, no_rubric_failure)
per-dimension (grouped view; never blended):
  outcome       0 pass / 0 fail / 1 inconclusive
  policy        0 pass / 0 fail / 1 inconclusive
  conversation  0 pass / 0 fail / 1 inconclusive
  speech        0 pass / 0 fail / 1 inconclusive
  reliability   0 pass / 0 fail / 0 inconclusive
```

Supply the call as `--transcript`, `--trace`, `--state`, and/or `--audio` and each check turns to pass or fail; a check whose evidence is absent stays INCONCLUSIVE until the evidence arrives. Full walkthrough: [`docs/CONVERSATION-TEST.md`](docs/CONVERSATION-TEST.md).

## The receipt: a per-dimension scorecard with the evidence attached

The scored HTML report is the thing that ends an argument in review: a verdict per event, a time-to-yield and talk-over histogram, failure clusters by fix class, and the per-event timeline with both channels and the measured numbers. The same audio and config reproduce every figure on the page.

<p align="center">
  <img src="https://raw.githubusercontent.com/attenlabs/hotato/main/docs/assets/report-scored.png" alt="The scored hotato HTML report: 0 of 1 events pass with a REGRESSION verdict, an analytics panel (time to yield, talk-over histogram, failure clusters by fix class), and the per-event timeline with caller and agent channels, the measured metrics, and the fix note." width="820">
</p>
<p align="center"><sub>One recording, the pinned scorer, a FAIL against the labeled yield expectation. Share it in a PR with <code>hotato card hotato-sweep.json#1 --out finding.svg</code>.</sub></p>

## The loop: catch, confirm, gate, prove

Surface a moment, confirm what it should have done, pin it to a CI contract, then re-check today's agent.

### 1. Catch: surface the candidate moments

`sweep` ranks the talk-over and false-stop moments across your recent calls by how far the timing missed. Two candidate types, straight from the open scorer, each a reproducible timing measurement:

<table align="center">
<tr>
<td width="412"><img src="https://raw.githubusercontent.com/attenlabs/hotato/main/docs/assets/cards/talk-over-card.svg" alt="Level 1 candidate card: 0.32s of overlap while the agent was talking, at t=2s in the recording. Hotato reports the measured timing." width="400"></td>
<td width="412"><img src="https://raw.githubusercontent.com/attenlabs/hotato/main/docs/assets/cards/false-stop-card.svg" alt="Level 1 candidate card: 0.46s of silence after the agent stopped with no caller nearby, at t=1.28s in the recording. Hotato reports the measured timing." width="400"></td>
</tr>
<tr>
<td colspan="2" align="center"><sub><b>Level 1 -- candidate.</b> Hotato reports what it measured (0.32s of overlap; 0.46s of trailing silence). Each is a candidate timing moment worth review.</sub></td>
</tr>
</table>

### 2. Confirm: you label yield or hold

You decide the expected behavior for a candidate: `yield` (stop for the caller) or `hold` (keep talking through a backchannel). That judgment stays with you: a threshold that stops missing interruptions starts false-stopping on backchannels, so when both fail in one run `diagnose` surfaces the tradeoff instead of naming one threshold:

<p align="center">
  <img src="https://raw.githubusercontent.com/attenlabs/hotato/main/docs/assets/cards/no-single-threshold-card.svg" alt="One sensitivity dial trades a missed interruption against a false stop on a backchannel, so hotato surfaces the tradeoff and leaves threshold tuning aside; fix class engagement-control." width="760">
</p>
<p align="center"><sub><b>Level 2 -- human-labeled failure.</b> A reviewer confirms the recording broke an explicit yield-or-hold expectation. When a missed interruption and a false stop collide, the fix lives in the engagement-control class.</sub></p>

### 3. Gate: pin it to a CI contract

`fixture promote` saves your labeled call as a permanent regression test. On every push, CI re-scores that recording under the pinned thresholds and scorer, and exits non-zero if the stored evidence stops matching your policy.

### 4. Prove: re-check the current agent

The frozen recording catches the evidence, thresholds, or scorer drifting. To check that the CURRENT agent still behaves, recapture the same scenario and score a NEW contract under the same policy:

```bash
# place the same call against today's agent, capture dual-channel, then:
hotato contract create --stereo fresh-call.wav --onset 41.90 --expect yield \
    --id refund-cutoff-001-recapture --out contracts
hotato contract verify contracts/refund-cutoff-001-recapture.hotato
```

<p align="center"><sub><b>Level 4 -- fresh-recapture comparison.</b> A newly captured call meets the same labeled policy and every submitted paired guard held. This is the claim only a fresh recapture can make. Walkthrough: <a href="docs/RECAPTURE.md"><code>docs/RECAPTURE.md</code></a>.</sub></p>

### Five levels of evidence, each scored on its own lane

Every card, report, and CLI result names its evidence level, and the public tier is the weakest one its inputs support.

| Level | Name | What it means |
| --- | --- | --- |
| 1 | Candidate | A candidate timing moment worth human review. |
| 2 | Human-labeled failure | A reviewer confirmed this recording broke an explicit yield-or-hold expectation. |
| 3 | Stored-evidence check | The historical audio still produces the expected result under the pinned policy and scorer. |
| 4 | Fresh-recapture comparison | A newly captured call passed the same contract, and no submitted paired guard regressed. |
| 5 | External proof | An independent team confirmed a caught regression or a fresh recapture. Not yet published. |

A before/after experiment (`hotato fix trial`, and the fleet loop) re-derives every verdict from the on-disk audio under one pinned trial manifest. A proof stands on the original recordings, intact fixtures, and that manifest, so the number is re-scored from the audio on every run.

## Gate it in CI

hotato speaks CI natively.

- Every scoring command exits non-zero when a deterministic check fails and zero when they pass, so a red build is a caught regression.
- `hotato contract verify contracts/ --junit contracts-junit.xml` writes JUnit XML your runner already renders.
- `--format json` carries an `exit_code` field, so an agent or a workflow reads the verdict without parsing prose.
- The model-judged rubric is advisory by default and blocks a build only when you pass `--gate`.

Drop-in GitHub Action and a pytest plugin: [`docs/CI.md`](docs/CI.md) &middot; [`docs/PYTEST.md`](docs/PYTEST.md). One bad call to a CI gate, step by step: [`docs/BAD-CALL-TO-CI.md`](docs/BAD-CALL-TO-CI.md) &middot; runnable [`examples/bad-call-to-ci/`](examples/bad-call-to-ci/README.md).

## Scale one call into a release gate

- `hotato suite run suite.yaml --agent support-bot` -- run a whole `suite.v1`; scenario-driven tests execute offline through the deterministic scripted-caller simulator, and every run records into the local registry.
- `hotato simulate --matrix scenario.yaml --out ./conv` -- render a `scenario.v1` with a deterministic scripted caller into `origin=simulated` conversation artifacts; the variation matrix expands into hundreds of runs, seeded and byte-identical on replay.
- `hotato rubric run --rubrics rubrics.yaml --transcript call.json` -- the model-judged lane on a pinned local model; zero egress, advisory unless `--gate`. [`docs/RUBRIC.md`](docs/RUBRIC.md).
- `hotato release compare BASELINE CANDIDATE` -- diff two recorded releases per dimension and per scenario, digest-exact, surfacing new failures and fixed-since.
- `hotato serve` -- a self-hosted, read-only web app over the local registry (release readiness, scenario matrix, conversation inspector, failure clusters, production health) on `127.0.0.1`, bearer-token authenticated. [`docs/WORKSPACE.md`](docs/WORKSPACE.md).

The bundled reference agent shows the shape at scale: 25 voice-agent jobs x 5 caller behaviours x 3 audio environments = 375 offline simulated runs, scored across the dimensions and byte-reproducible on replay. Run it with `make reference` in [`examples/reference-agent/`](examples/reference-agent/README.md) ([`docs/SUITE-RUN.md`](docs/SUITE-RUN.md)); the measurement-error harness is [`docs/BENCHMARK.md`](docs/BENCHMARK.md).

## Connect a production stack

The demo runs on the bundled calls alone. To point Hotato at your live calls, connect once, then sweep on a schedule:

```bash
hotato connect vapi                                             # credentials stored 0600, local only
hotato sweep --stack vapi --since 7d --out hotato-sweep.html    # cron, CI, wherever
```

Run `sweep` on a timer and it becomes a scheduled batch scanner. Your audio stays on your machine unless you explicitly pull it from your stack. Full guide: [`docs/SET-AND-FORGET.md`](docs/SET-AND-FORGET.md) &middot; runnable [`examples/set-and-forget/`](examples/set-and-forget/README.md).

`sweep` and `hotato fleet run` can POST a one-line JSON summary to a webhook when they finish, off by default, opt in with `--notify` (repeatable for more than one URL):

```bash
hotato sweep --stack vapi --since 7d --notify https://hooks.slack.com/services/...
```

The payload stays metadata-only: counts, the top candidate moments (id, kind, timing numbers only), local artifact paths, and a `text` field a Slack incoming webhook renders directly. A down or slow webhook leaves the run intact: a delivery failure is one warning line on stderr. Egress details: [`docs/EGRESS.md`](docs/EGRESS.md).

## Fleet: private, self-hosted

`hotato fleet` runs the loop across every agent from one local workspace: ingest calls, surface candidates, label them, and run a before/after experiment that recomputes both sides from audio under a pinned manifest. It recommends a change and leaves the deploy to you. Local mode is stdlib-only (SQLite plus a content-addressed store), runs entirely on your machine, and registers as many agents as you want.

```bash
hotato fleet init    -w acme
hotato fleet agent add -w acme --name support-bot --stack vapi --assistant-id asst_123
hotato fleet ingest  -w acme --agent support-bot call.wav
hotato fleet discover -w acme --agent support-bot call.wav
hotato fleet review   -w acme
```

Once a workspace has some history, `hotato fleet trend -w acme` reads the same local SQLite registry and writes one self-contained HTML page: per-agent talk-over and time-to-yield trend lines (p50/p95 per day), candidate moments discovered over time, and experiment outcomes (improved/inconclusive/refused). A day with no measurements gets no point, and a series with fewer than two days of history reports plainly as "not enough history to trend" instead of an interpolated line. Full guide: [`docs/GUARDIAN-FLEET.md`](docs/GUARDIAN-FLEET.md).

## Self-host in your own cloud or VPC

The whole team workspace ships as a container. One command stands up the read-only, token-authenticated workspace (`hotato serve`) on host loopback over a private volume; the default stack keeps all traffic on your own infrastructure at run time.

```bash
docker compose up -d                    # workspace on 127.0.0.1:8321
docker compose run --rm hotato-init     # optional: seed example data
docker compose --profile judge up -d    # optional: a local Ollama model judge
```

Build, connect your own calls, the local judge, air-gap, backup, and the zero-migration promise (self-host and cloud share one set of schemas): [`docs/SELF-HOST.md`](docs/SELF-HOST.md). Confirm the offline posture on your own machine with [`deploy/verify-zero-egress.sh`](deploy/verify-zero-egress.sh).

## Built for coding agents

Hotato is a first-class tool for an LLM or an agent to drive: machine JSON on every command, meaningful exit codes, a described capability manifest, [`llms.txt`](llms.txt), JSON-LD, and an MCP server.

```bash
uvx --from "hotato[mcp]" hotato-mcp      # the voice_eval_run scorer + eleven fleet tools
```

Configs and the tool contract: [`docs/MCP.md`](docs/MCP.md) &middot; [`AGENTS.md`](AGENTS.md) &middot; [`llms-full.txt`](llms-full.txt).

## Choose your path

| You want to | Run this |
| --- | --- |
| Try the full loop, no credentials | `hotato start --demo` |
| Sweep the bundled demo calls | `hotato sweep --demo` |
| Sweep recent calls from your stack | `hotato connect vapi` then `hotato sweep --stack vapi --since 7d` |
| Add Hotato to an existing repo, CI gate included | `hotato init starter --stack vapi --out .` ([`docs/STARTER.md`](docs/STARTER.md)) |
| Turn a confirmed failure into a portable contract | `hotato contract create --from-candidate hotato-sweep.json#1 --expect yield --id refund-cutoff-001 --out contracts` ([`docs/CONTRACTS.md`](docs/CONTRACTS.md)) |
| Verify contracts in CI | `hotato contract verify contracts/ --junit contracts-junit.xml` |
| Attach observability traces to a contract | `hotato trace attach contracts/refund-cutoff-001.hotato --trace voice_trace.jsonl` ([`docs/TRACE.md`](docs/TRACE.md)) |
| Test a candidate fix, before/after, fail-closed | `hotato fix trial patch.json --name staging-x --before before/ --after after/` ([`docs/FIX-TRIAL.md`](docs/FIX-TRIAL.md)) |
| Share a finding in a PR or slide | `hotato card hotato-sweep.json#1 --out finding.svg` |
| Drive it from a coding agent | `uvx --from "hotato[mcp]" hotato-mcp` ([`docs/MCP.md`](docs/MCP.md)) |

`contract verify` and a promoted fixture in CI are two different guarantees, depending on which recording goes in:

| | On the frozen recording (every push) | On a fresh recapture (by hand, see [`docs/RECAPTURE.md`](docs/RECAPTURE.md)) |
| --- | --- | --- |
| Proves | The evidence, policy, and scorer are intact | The CURRENT agent's behavior still matches the label |
| Does not prove | That the deployed agent has not changed | -- |

A contract bundle contains call audio. Do not commit a raw customer contract to a public repository; use sanitized fixtures for anything public. See [`docs/CONTRACTS.md`](docs/CONTRACTS.md).

## Install

Add hotato to a project with `pip`; or run any command zero-install with [`uvx`](https://docs.astral.sh/uv/):

```bash
pip install hotato                 # core: stdlib-only, zero dependencies
pip install 'hotato[neural]'       # optional Silero VAD cross-check
pip install 'hotato[transcribe]'   # optional ASR transcript, attached as context beside the score
pip install 'hotato[livekit]'      # LiveKit live capture
pip install 'hotato[pipecat]'      # Pipecat live capture
```

## Contribute a labeled call

The highest-value PR is one labeled dual-channel call. A small, high-integrity corpus of recorded calls with human-labeled turn-taking ground truth is the part of hotato that compounds: the more moments the community labels, the sharper every scorer gets. Add a clip: [`docs/SUBMITTING.md`](docs/SUBMITTING.md) &middot; the corpus and its schema live in [`corpus/`](corpus/README.md), with a recorded battery in [`corpus/vapi-defaults/README.md`](corpus/vapi-defaults/README.md). Contributor guide: [`CONTRIBUTING.md`](CONTRIBUTING.md).

## Where Hotato fits

- **Conversation QA that shows its work.** See [`docs/COMPARE.md`](docs/COMPARE.md) for how the per-dimension scorecard fits alongside runtime voice layers.
- **Audio-timing scoring.** It scores how the conversation is timed. The opt-in `--transcribe` flag attaches an ASR transcript as context to read next to the verdict, with the timing score grounded in the audio. See [`docs/TRANSCRIBE.md`](docs/TRANSCRIBE.md).
- **Channel-based timing.** Each channel is one party (caller on one, agent on the other); Hotato measures the timing between them, and the channels stay anonymous.
- **Timing evidence you label.** It produces candidate timing evidence; humans label intent, and CI enforces the confirmed contracts.
- **Offline and out-of-band.** It reads recordings after the call, so your live audio path and running agent stay in your hands.

## Docs

- **Set-and-forget monitoring** (connect once, sweep on a schedule, promote confirmed bugs into fixtures): [`docs/SET-AND-FORGET.md`](docs/SET-AND-FORGET.md) &middot; runnable [`examples/set-and-forget/`](examples/set-and-forget/README.md)
- **Bad call to CI regression test**, step by step: [`docs/BAD-CALL-TO-CI.md`](docs/BAD-CALL-TO-CI.md) &middot; runnable [`examples/bad-call-to-ci/`](examples/bad-call-to-ci/README.md)
- **What it measures** (the three timing signals, re-derivable by hand): [`METHODOLOGY.md`](METHODOLOGY.md) &middot; Python API [`docs/API.md`](docs/API.md)
- **The fix ladder** (each failure names a likely fix class; when the evidence maps cleanly to stack config, Hotato names the setting family and direction): [`docs/FIX-PLANS.md`](docs/FIX-PLANS.md)
- **Rule out the non-turn-taking bugs first** (STT, buffering, verbosity, refusals, wrong-language): [`docs/WHY.md`](docs/WHY.md)
- **Pull a call from your stack** (Vapi, Twilio, Retell, LiveKit, Pipecat): [`adapters/README.md`](adapters/README.md) &middot; status [`docs/ADAPTER-STATUS.md`](docs/ADAPTER-STATUS.md)
- **CI gates**: GitHub Action [`docs/CI.md`](docs/CI.md) &middot; pytest plugin [`docs/PYTEST.md`](docs/PYTEST.md)
- **Recorded-call battery**: 12 scripted calls against a live voice agent on its provider's default settings, where a missed interruption and a false stop on a backchannel fail in the same run, so `diagnose` surfaces the tradeoff instead of naming one threshold: [`corpus/vapi-defaults/README.md`](corpus/vapi-defaults/README.md)
- **Failure contracts and traces**: turn a labeled candidate into a portable, CI-verified bundle and attach observability evidence: [`docs/CONTRACTS.md`](docs/CONTRACTS.md) &middot; [`docs/TRACE.md`](docs/TRACE.md) &middot; [`docs/OTEL.md`](docs/OTEL.md)
- **Deterministic assertions on the transcript/trace** (phrase, PII, policy, tool-call, outcome; each scored on its own lane): [`docs/ASSERTIONS.md`](docs/ASSERTIONS.md)
- **Checking the CURRENT agent, not just the frozen recording**: the recapture walkthrough: [`docs/RECAPTURE.md`](docs/RECAPTURE.md)
- **Egress**: a per-command network table derived from the code, what is local, what reaches your vendor, what optional extras add a hosted call: [`docs/EGRESS.md`](docs/EGRESS.md)
- **Layered failure evidence and a before/after fix trial**: `hotato explain` breaks a failing result down by layer, and `hotato fix trial` tests a candidate change before/after, fail-closed: [`docs/EXPLAIN.md`](docs/EXPLAIN.md) &middot; [`docs/FIX-TRIAL.md`](docs/FIX-TRIAL.md) &middot; [`docs/APPLY.md`](docs/APPLY.md) &middot; [`docs/FIX-LOOP.md`](docs/FIX-LOOP.md)
- **Evidence**: what Hotato validates, the input-condition trust matrix, every card and CLI block reproducible, and where Hotato fits alongside broader voice-agent testing tools: [`docs/VALIDATION.md`](docs/VALIDATION.md) &middot; [`docs/TRUST-MATRIX.md`](docs/TRUST-MATRIX.md) &middot; [`docs/GALLERY.md`](docs/GALLERY.md) &middot; [`docs/EVIDENCE-PACK.md`](docs/EVIDENCE-PACK.md) &middot; [`docs/COMPARE.md`](docs/COMPARE.md)
- **For coding agents**: [`AGENTS.md`](AGENTS.md) &middot; [`llms.txt`](llms.txt) &middot; [`llms-full.txt`](llms-full.txt) &middot; MCP server [`docs/MCP.md`](docs/MCP.md) &middot; Security [`SECURITY.md`](SECURITY.md)
- **Contributing**: the highest-value PR is a labeled call fixture: [`docs/SUBMITTING.md`](docs/SUBMITTING.md)

Why "hotato": good turn-taking is a game of hot potato. Speak, then pass the turn the moment the caller wants it. MIT licensed ([`LICENSE`](LICENSE)); the open core stays open.

mcp-name: io.github.attenlabs/hotato


================================================================================
FILE: docs/WHY.md
================================================================================

# Why Hotato

## Voice agents fail in ways your tests do not catch

Your test suite checks what the agent says. Production calls fail on when it
speaks. Four timing failures show up again and again in real transcripts, and
all four are invisible to text-level tests. Three are interruption patterns
(missed interruption, false stop, slow yield); the fourth is an endpointing
gap:

1. **Missed interruption.** The caller says "stop, take that off" and the agent
   keeps talking. `did_yield` is false where it must be true. The caller
   repeats themselves, louder, then hangs up.
2. **False stop.** The caller says "mhm", a backchannel: a short acknowledgement,
   not a request to take over. The agent stops mid-sentence anyway. `did_yield`
   is true where it must be false. The call stalls, then restarts, then stalls
   again.
3. **Slow yield.** The caller interrupts and the agent does stop, eventually.
   Every second of `talk_over_sec` before it stops is the caller and the agent
   speaking at once, and the caller hears all of it.
4. **Endpointing misses.** Endpointing is detecting that the caller finished
   speaking. When it misses, the caller gets dead air after they finish
   (`response_gap_sec`), or the agent starts before they are done
   (`premature_start_sec`). Both read as a broken conversation partner. The
   silence ambiguity underneath (thinking, distracted, or gone quiet) is not
   something Hotato resolves either: it measures how long the silence lasted,
   never what it meant.

Each of these lives entirely in the audio timing of the call. A transcript
diff, an LLM judge on text, and a unit test on the agent's reply all score a
call with any of these failures as perfect: the transcript reads clean, the
caller called back anyway. That gap is exactly what these four patterns hide.

You label the expected behavior for the event: yield means the agent should
stop for the caller. hold means the agent should keep speaking through a
backchannel/noise/acknowledgement. Hotato measures whether the timing matched
that label -- intent is always yours to call.

## Is this even a turn-taking bug?

In our observed reports, many alleged barge-in bugs turn out not to be
turn-taking bugs at all. Each of these produces a symptom that reads exactly like a
missed interruption or a false stop, but the fix lives in a different layer,
and no VAD threshold will ever touch it:

- **STT hallucination.** The transcript has words the caller never said, or
  drops words that mattered, so the agent responds to something the caller
  did not say. That looks like an interruption the agent ignored. Reach for:
  an STT/ASR word-error-rate check against the raw audio.
- **Client-side audio buffering.** The caller's own device or browser queues
  outgoing audio before it reaches the agent, so "the agent talked over me"
  is audio arriving late at the agent, an ordering artifact of transport.
  Reach for: client/WebRTC jitter-buffer and network-latency instrumentation.
- **LLM verbosity or tool-selection.** The agent "kept talking through the
  interruption" because it was mid-tool-call or committed to finishing a long
  generation before it re-checked for a stop signal. Reach for:
  response-length and tool-call latency tracing in your agent framework.
- **Safety false-refusal.** The agent stops abruptly mid-sentence because a
  moderation or safety layer cut it off, a decision made upstream of any
  barge-in or backchannel detection. Timing-wise this is indistinguishable
  from a false stop on "mhm". Reach for: your safety/moderation logs.
- **Wrong-language STT.** The caller is speaking a language or accent the STT
  covers poorly, recognition comes back empty or garbled, and the agent's
  response looks unrelated or missing entirely. That reads as a missed
  interruption; it is a language-coverage gap. Reach for: per-locale STT
  accuracy tooling. (Hotato's own detector works from energy alone, so it
  measures the same way regardless of language: see
  `corpus/classes/README.md`.)

This is both a scope guard and a shortcut. If your bug matches one of
these five, tuning Hotato will not surface it, because the fix lives in a
different layer entirely; you will save the day you would have spent staring
at `turn_end_silence_sec`. If it matches none of them: two common complaints are
agent-talks-over-caller and false-stop-on-backchannel, and that is exactly what
Hotato measures, with the funnel (no single config value fixes both directions
at once) proven on recorded calls, not synthetic fixtures
(`corpus/vapi-defaults/README.md`).

## What makes Hotato different

- **It scores recordings you already have.** A dual-channel WAV from your
  stack is the whole input -- plug it in directly, with no test harness,
  synthetic caller, or re-architecture required.
- **It runs offline.** Scoring is local and deterministic, with audio,
  transcript, and result all staying on your machine.
- **It emits machine-readable timing.** One JSON envelope per run, the same
  shape from the CLI, the MCP tool, and the pytest fixture, so an agent or a
  CI job consumes one schema.
- **It fails CI.** Exit code 1 on a regression, 0 on pass, 2 when a recording
  isn't scorable yet -- the caller channel is silent, or the agent wasn't
  talking when the caller started -- so the verdict waits for a recording
  that can answer the question. A turn-taking regression blocks a merge the
  same way a failing unit test does.
- **It routes every failure to a fix class.** When the failure maps cleanly to
  stack config, `config` names the setting family on your stack and the
  direction to investigate. `engagement-control` names the failure as a
  classification problem -- telling "mhm" apart from "stop" -- so you spend
  your time on the fix that works instead of retuning a setting that cannot
  win.

## What Hotato measures (and what it leaves to other tools)

Hotato measures energy over time on two channels -- that's the whole method.
Transcription, emotion, and intent detection sit outside it; a diarizer,
where used, assigns anonymous SPEAKER_00/01 labels and never identifies who a
person is. A single-channel (mono) recording is scorable via the opt-in,
quality-gated `[diarize]` front-end (`hotato run --mono call.wav --diarize`),
labeled indicative below the confidence bar, with dual-channel as the
reference standard. The method and its ceiling are stated in
[METHODOLOGY.md](../METHODOLOGY.md) and in the `limits` block of every result.

## Reproducible timing measurements, with the method exposed

A single blended percentage tells you nothing about which call failed, on
which axis, or what to change, so Hotato reports three direct measurements
per event:

- `did_yield`: did the agent stop talking after the caller started
- `seconds_to_yield`: seconds between the caller starting and the agent stopping
- `talk_over_sec`: seconds the caller and the agent spoke at the same time

Every number is reproducible from the frame dump by hand, every threshold is
exposed, and a failure points at its fix. That is what you debug with.

Ready to pin a failure? The loop from one bad call to a CI gate, with
`hotato fixture create`, `compare`, and `plan`:
[BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md).


================================================================================
FILE: METHODOLOGY.md
================================================================================

# Methodology

How every number this tool reports is computed, why it is reproducible, and
where the method stops being trustworthy. There is no accuracy percentage in
this document and none is implied anywhere in the tool. What follows describes a
measurement, not a claim about any detector's internal quality.

Hotato scores the audio *timing* of turn-taking from a call recording: whether
the agent stopped talking once the caller started (a yield), how many seconds
that took, how many seconds it kept talking while the caller was talking
(talk-over), and, on the same two tracks, the endpointing timing: how the
agent's response sits around the moment the caller finished speaking. It
measures energy over time. It does not understand speech.

## Read this first: the honest ceiling

The limits come first because they bound everything below.

- **Energy is not intent.** The detector marks a frame "active" when its
 short-time energy crosses a threshold. A cough, a slammed door, or a burst of
 line noise reads as active exactly like a word does. Nothing here recovers
 *meaning*, it recovers *when there was speech-level energy*.
- **You supply the label.** Hotato does not infer intent. You label the
 expected behavior for the event: yield means the agent should stop for the
 caller. hold means the agent should keep speaking through a
 backchannel/noise/acknowledgement. Hotato then measures whether the timing
 matched that label. "mhm" and "stop" can carry identical speech energy; the
 verdict is only ever your label checked against measured timing.
- **Sub-second single-channel boundaries sit near the resolution limit.** At the
 default 10 ms hop a boundary is located to within a frame or two, but the
 hangover that keeps an utterance whole (default 150 ms) smears every trailing
 edge by up to that much, and on one mixed channel deciding *whose* energy
 crossed the threshold is not always possible.
- **Two channels is the gold reference; mono is scorable, quality-gated.** The
 reference input is separated caller and agent tracks: one two-channel WAV or
 two aligned mono WAVs, where each channel is one party and overlap is a fact of
 the recording (both tracks active at once, by construction). A single mixed
 mono call is scorable via the opt-in `[diarize]` front-end (`hotato run --mono
 call.wav --diarize`): a diarizer separates the mix into caller/agent activity,
 which is reconstructed and scored through the same path. It is quality-gated --
 above the confidence bar the verdict is labeled `diarized-mono`; below it, the
 verdict is labeled indicative only and no SLA gate fires; a non-separable file
 is refused. Diarized mono never equals a true dual-channel recording for
 sub-second talk-over attribution, and the gate is what keeps that honest per
 file.
- **Out of scope, permanently:** no speaker identification (a diarizer assigns
 anonymous SPEAKER_00/01; it never says who a person is), no speech-to-text, no
 emotion or intent detection, and no claim about any vendor's internal accuracy.
 Word-level semantics are not measured; only the timing of the yield is.

These same limits are emitted inline in the `limits` block of every JSON result
(`core.py`) and in the MCP tool description, so a consuming agent sees them.

## The pipeline, step by step

Everything runs offline, standard-library only (NumPy only accelerates the
per-frame RMS when importable; results are identical without it). Scoring is
deterministic: the same audio and `ScoreConfig` give the same output.

### Default parameters

Every value below is read directly from `ScoreConfig` (`score.py`) and
`VADParams` (`vad.py`); the same defaults apply to both channels unless overridden.

Framing and VAD (`VADParams`, applied per channel):

| Parameter | Default | Meaning |
|--------------------|-----------|---------|
| `frame_ms` | 20.0 ms | analysis window length for one RMS frame |
| `hop_ms` | 10.0 ms | step between frames (the time resolution) |
| `noise_percentile` | 0.10 | quietest 10% of frames estimates the noise floor |
| `rel_db` | 15.0 dB | a frame is active this many dB above the floor |
| `abs_gate_db` | -60.0 dBFS| nothing below this absolute level is ever active |
| `dyn_margin_db` | 22.0 dB | on a rarely-silent channel, hold the threshold at least this far below the loudest frames |
| `hangover_sec` | 0.15 s | stay "active" this long after energy drops |

Signal windows (`ScoreConfig`):

| Parameter | Default | Meaning |
|---------------------------|---------|---------|
| `yield_hangover_sec` | 0.20 s | agent must stay quiet this long to count as yielded |
| `max_search_sec` | 3.0 s | how long after onset a yield is searched for |
| `caller_proximity_sec` | 0.5 s | a yield only counts if the caller held the floor within this window of it |
| `turn_end_silence_sec` | 0.20 s | caller must stay quiet this long for the turn to count as ended |
| `premature_tolerance_sec` | 0.05 s | agent may lead the caller's turn end by up to this before it counts as premature |
| `onset_min_run_sec` | 0.05 s | minimum sustained active run for the caller onset to count |
| `agent_onset_lookback_sec`| 0.10 s | window before onset checked for whether the agent was already talking |

Both onset-detection windows are exposed on `ScoreConfig` as well:
`onset_min_run_sec` (the `0.05 s` sustained active run a caller onset must
clear, applied by `first_active_sec`) and `agent_onset_lookback_sec` (the
`0.10 s` window before onset checked for "was the agent talking at onset?").
Neither affects the per-frame active/threshold decision the frame dump records.
Every threshold the scorer uses is a named, overridable parameter -- there are
no hidden constants in the scoring path.

### Step 1, per-frame RMS

Each channel is cut into 20 ms windows stepped by 10 ms. For each window the
linear root-mean-square amplitude is computed (`frame_rms`, `audio.py`). At
16 kHz this is a 320-sample window with a 160-sample hop, so `hop_sec` is
exactly `160 / 16000 = 0.01 s`; it is derived from the integer sample hop over
the sample rate, so it matches `hop_ms` exactly at common rates.

### Step 2, RMS to dBFS

Each linear RMS value is converted to decibels relative to full scale:
`dBFS = 20 * log10(rms)`, with a floor of `-120.0 dBFS` substituted before the
log to avoid `log(0)` (`to_dbfs`, `audio.py`). This is the per-frame `*_dbfs`
column in the frame dump.

### Step 3, per-channel energy VAD

For each channel independently (`energy_vad`, `vad.py`):

1. Sort the frame dBFS values and take the `noise_percentile` (10th percentile)
 as the **noise floor**.
2. The base **threshold** is `max(noise_floor + rel_db, abs_gate_db)`, i.e.
 15 dB above the floor, but never below the -60 dBFS absolute gate.
3. **Dynamic-margin guard:** if a channel is almost never silent (e.g. an agent
 talking the whole clip), the 10th-percentile floor lands *inside* speech and
 would push the threshold above the speech itself. So compute
 `cap = max(dBFS) - dyn_margin_db`; if `cap` is above the absolute gate, lower
 the threshold to `min(threshold, cap)`. This cannot rescue a genuinely silent
 channel, because the guard only fires when loud content exists above the gate.
4. A frame is **raw-active** when its dBFS ≥ the threshold.
5. **Hangover:** after any raw-active frame, keep the channel active for
 `round(hangover_sec / hop_sec)` more frames, 15 frames (150 ms) at the
 defaults, so brief inter-word gaps do not fragment one utterance.

The resulting threshold and noise floor are constant across frames for a given
channel and are recorded verbatim in the frame dump.

### Step 4, caller onset

If a caller onset label is supplied (scenario `caller_onset_sec`, or `--onset`),
it is used directly. Otherwise the onset is the start of the caller's first
sustained active run, the first run of at least 0.05 s of active frames
(`first_active_sec`). The label is preferred because on mixed or noisy audio the
first energy is not always the caller. `onset_idx` is `round(onset / hop_sec)`,
clamped into range. No label and no detectable caller speech means there is no
caller event to score: the recording is reported not scorable (`scorable: false`
with a plain reason), never scored against frame 0, and a single-recording run
exits 2.

### Step 5, the three barge-in signals

Computed in `score_channels` (`score.py`):

- **`agent_talking_at_onset`**, was any agent frame active in the 0.10 s up to
 and including onset? If not, on a should-yield expectation there is nothing to
 yield: the event is reported not scorable (`scorable: false`, with the reason
 in `not_scorable_reason`) rather than passed or failed, and a single-recording
 run exits 2. The input is what is wrong (onset time, channel mapping, or
 expectation), and it is reported as such.
- **`did_yield`**, scanning from onset up to `onset + max_search_sec` (300
 frames), the first frame where the agent goes quiet and *stays* quiet for
 `yield_hangover_sec` (20 frames) **and** the caller held the floor within
 `caller_proximity_sec` (±50 frames) of that quiet point. The proximity
 condition is what stops an agent that merely finishes its own sentence seconds
 after an isolated backchannel from being scored as a barge-in response.
- **`time_to_yield_sec`**, `(yield_frame - onset_frame) * hop_sec`, floored at
 0; `None` if the agent never yielded within the search window.
- **`talk_over_sec`**, count of frames from onset to the yield point (or to the
 end of the search window if it never yielded) where the caller and the agent
 were *both* active, times `hop_sec`.

### Step 6, the two latency (endpointing) signals

Pure timing on the same two VAD tracks, no second model (`score.py`):

- **`caller turn end`**, the first frame at/after onset where the caller goes
 quiet and stays quiet for `turn_end_silence_sec` (20 frames). `None` if the
 caller never activates after onset or is still talking when the clip ends.
- **`agent response onset`**, skip any agent speech already in progress at
 onset (the pre-caller turn or the yield tail), then the first frame the agent
 becomes active again. `None` if the agent never starts a fresh run.
- With both defined, `lead = turn_end_frame - response_onset_frame` (positive
 when the agent starts *before* the caller finishes). If `lead` exceeds
 `premature_tolerance_sec` (5 frames), `premature_start_sec = lead * hop_sec`
 and `response_gap_sec` is null. Otherwise `premature_start_sec = 0.0` and
 `response_gap_sec = max(0, response_onset - turn_end) * hop_sec`. Both are
 null when not derivable from the tracks, never fabricated. Reported values
 are rounded to the millisecond (3 decimal places of seconds).

### Step 7, two additive, opt-in cross-checks: echo and resume

Both live entirely in hotato's own layer (`echo.py`, `resume.py`), never in
the vendored `_engine`, and both add an optional `signals` block without
changing `did_yield`, `seconds_to_yield`, or `talk_over_sec` for any existing
recording.

- **`signals.echo`** (`echo.py`), computed on every scored event. Leaked TTS
 (an agent that hears its own audio back on the caller channel) is, by
 construction, a delayed, scaled copy of the agent's own envelope. The check
 is a deterministic cross-channel coherence: at each lag from 0 up to
 `DEFAULT_MAX_LAG_SEC` (0.5 s), take the cosine similarity between the caller
 frame-RMS envelope and the agent envelope shifted by that lag; `coherence`
 is the best (highest) cosine found, `lag_sec` is the lag it occurred at, and
 `echo_suspected` is `coherence >= DEFAULT_COHERENCE_THRESHOLD` (0.7) with at
 least `_MIN_OVERLAP_FRAMES` (8) of overlap. Independent speech (real
 turn-taking, backchannels) does not correlate with the agent's own envelope
 at any lag, so it scores low. `--echo-gate` (opt-in, `hotato run`) holds an
 echo-suspected yield out of the verdict (`scorable: false`) instead of
 counting it as a clean pass; without the flag the primary verdict is
 unchanged and `signals.echo` is still reported. `hotato diagnose` and the
 single-run text output print a WARNING for every echo-suspected yield.
- **`signals.resume`** (`resume.py`), present only on events where the agent
 yielded. From the agent's own VAD track, look for a fresh onset within
 `DEFAULT_RESUME_WINDOW_SEC` (4.0 s) after the yield: `resumed` is whether one
 is found, `resume_gap_sec` is the seconds from yield to that onset (`null`
 if it did not resume). `restart_suspected` flags whether the longest
 contiguous agent run at or after the resume onset reaches
 `DEFAULT_RESTART_MIN_SEC` (2.0 s), the timing fingerprint of re-answering a
 whole paragraph from the top rather than finishing the interrupted clause.
 Whether the resumed words literally repeat the earlier ones is a transcript
 question, explicitly out of scope: `restart_suspected` is a run-length
 heuristic on timing, not a text diff.
- **`agent_stop_no_caller`** (`scan.py`, not part of the scored envelope: a
 `hotato scan` candidate) surfaces the companion timing fact: the agent went
 from active to quiet with zero caller energy anywhere in
 `caller_proximity_sec` on either side, so nothing on the caller channel
 explains the drop. It is a candidate for self-truncation, endpointing
 mis-fire, or an upstream (LLM/TTS) cutoff, not a verdict; label it with
 `hotato fixture create --expect hold` (or `yield`, if that silence was in
 fact correct) to turn it into a scored fixture.

## Why it is reproducible

- **Deterministic** given `(audio, ScoreConfig)`. No randomness, no network, no
 hidden state. The bundled fixtures are themselves rendered deterministically
 from a seed derived from `sha256(scenario_id)` (`render_examples.py`), and CI
 re-renders and diffs them to prove byte-stability.
- **Every threshold that drives the active/threshold decision is a named,
 overridable parameter** on `ScoreConfig` / `VADParams`, and the resolved
 values are echoed back in the frame dump's `config` block.
- **Every frame and every derived index is inspectable.** `--dump-frames PATH`
 writes, per frame: `t_sec`, `caller_dbfs`, `agent_dbfs`, `caller_active`,
 `agent_active`, and the constant `caller_threshold_db`,
 `caller_noise_floor_db`, `agent_threshold_db`, `agent_noise_floor_db`
 (`frame_dump`, `score.py`). Nothing the scorer decides is off the record.

### Worked example: re-deriving `did_yield` and `talk_over` by hand

The excerpt below is illustrative of the dump shape; run `--dump-frames` on your
own file for real values. Assume the defaults (`hop_sec = 0.01`), a caller onset
label at `2.40 s` (`onset_frame = 240`), and these constant thresholds from the
dump header: caller threshold `-55.0 dBFS`, agent threshold `-52.0 dBFS`.

```
 frame t_sec caller_dbfs caller_active agent_dbfs agent_active
 240 2.40 -18.4 True -21.1 True
 288 2.88 -19.0 True -20.7 True
 289 2.89 -20.2 True -48.9 True (hangover tail)
 290 2.90 -21.5 True -71.3 False
 ... (agent dBFS stays below -52 through frame 310) ...
 310 3.10 -20.8 True -69.4 False
```

Re-derive `did_yield`: agent was active at onset (frame 240), so the question is
meaningful. Scanning from 240, the agent first drops below its threshold at
frame 290 and stays below it for at least 20 consecutive frames (through 310), 
that satisfies `yield_hangover_sec`. The caller is active within ±50 frames of
290, so the proximity condition holds. Therefore `did_yield = True`,
`yield_frame = 290`, and `time_to_yield_sec = (290 - 240) * 0.01 = 0.50 s`.

Re-derive `talk_over`: count frames from 240 up to the yield frame 290 where
`caller_active AND agent_active`. In this excerpt that is frames 240 through 289
inclusive, 50 frames, so `talk_over_sec = 50 * 0.01 = 0.50 s`. Every one of
those booleans is just `dbfs >= threshold` (plus hangover), which you can
recompute yourself from the two dBFS columns and the header thresholds.

## Aggregate statistics: one definition each

The `report`, `team`, and `export` surfaces summarize many events with mean,
median, p90, and p95. All four come from one stdlib implementation
(`src/hotato/_stats.py`, `dist_summary`) so every published aggregate is
re-derivable by hand:

- **mean** is the arithmetic mean (`statistics.fmean`).
- **median** is `statistics.median` (p50).
- **p90** and **p95** are linear interpolation between closest ranks (the
 definition NumPy calls "linear"): sort the values ascending, let
 `pos = q * (n - 1)` for `q` in `{0.90, 0.95}`, then
 `p_q = v[floor(pos)] + frac * (v[floor(pos) + 1] - v[floor(pos)])`.
- **Rates are fractions**, never a percentage: a pass rate of 8 of 8 reads
 `1.00`. That keeps every aggregate in the same unit system as the
 measurements and leaves no door open to an accuracy-percentage reading.
- **Empty input returns null.** No aggregate is ever fabricated from zero
 measurements, and fewer than two runs is stated plainly rather than drawn as
 a trend.

`team` and `export` pool `response_gap_sec` (dead air before the agent
speaks, defined above) across every scored event into the same `dist_summary`
shape already used for talk-over and time-to-yield. `--max-response-gap`
turns the pooled p95 into a latency SLA gate: the run exits 1 exactly when the
pooled p95 exceeds the bound, the same pass/fail contract as
`--max-talk-over` and `--max-time-to-yield`, just pooled instead of
per-event. A plain `hotato export` (no `--max-response-gap`) still writes a
byte-identical `envelope.json` to a run with none of this stage's work
applied; the p95 and gate live only in the printed summary and the returned
manifest.

## Optional neural cross-check (non-reference)

The energy VAD above is the **reference**: every published, golden, and bundled
number comes from it, deterministically. Hotato also ships an **optional,
opt-in, explicitly non-reference** neural VAD backend so you can re-run the *same*
turn-taking timing math over a learned speech track and compare, a direct answer
to "this is just energy VAD, rebuildable in a weekend."

- **How to use it.** Install the extra and pass the flag on your *own* recording:
 `pip install 'hotato[neural]'` then `hotato run --stereo call.wav --backend neural`.
 The default is `--backend energy`. The `--suite` self-test **always** scores
 with energy (it is the reference), so `--backend neural` is ignored there.
- **What it is.** Silero VAD (MIT), run locally/offline. The engine itself keeps
 zero third-party dependencies and never imports a model; the backend is injected
 behind one shared interface, so an open-weight turn-detector (e.g. LiveKit's
 Smart-Turn) can be plugged in the same way later. It returns the **identical**
 `VADResult` shape as the energy backend, `active`, `hop_sec`, `threshold_db`,
 `noise_floor_db`, aligned to the same hop grid. A neural model has no dB
 threshold, so `threshold_db` / `noise_floor_db` are **synthesized**: they are the
 energy-domain description of the *same* audio, reported for inspection only; the
 neural `active` decision is a learned speech probability, not a dB crossing.
- **What it changes, honestly.** It **tightens onset precision** on clean speech
 (a learned boundary can beat a fixed dB threshold on some audio). It does **not**
 close the energy-vs-intent gap: a cough, a laugh, a door slam, or crosstalk still
 carries speech-band energy, and *any* single-channel VAD, energy or neural, can
 mark it active. Whether a sound is a genuine bid for the turn is not decidable
 from one channel's activity alone. **No accuracy percentage is claimed for either
 backend.** Neural is a flagged cross-check, not a new source of truth.
- **No silent fallback.** If the `[neural]` extra is absent, `--backend neural`
 raises a clean, explicit error and exits non-zero, it never quietly scores with
 energy and presents it as the neural result.
- **Verification status: verified against the real model.** Executed in this
 repo with silero-vad 6.2.1 (ONNX weights bundled inside the package, inference
 through onnxruntime on CPU, fully offline; note silero-vad itself depends on
 torch, so the extra installs it). Properties that hold, measured over the
 bundled 8 and the 40-scenario gold suite:
 - **Contract holds end to end.** The real model returns the identical
   `VADResult` shape as energy, on the same hop grid, through the public scorer.
 - **Deterministic.** Two full sweeps over all 48 fixtures produced
   byte-identical JSON, for both backends; single recordings are likewise
   byte-identical run to run.
 - **Reference untouched.** With the extra installed, energy outputs are
   byte-identical to energy outputs without it, and the frozen bundled 8
   verdicts are unchanged.
 - **On the synthetic fixtures the two tracks diverge, and that divergence is a
   property of the fixtures.** The renders are shaped noise built for the
   energy reference; a speech-trained model assigns them near-zero speech
   probability (identical through the ONNX and torch paths), so at Silero's
   default threshold the neural track is empty there and the timing math
   reports no yield on any of the 48. This measures the fixtures, not any
   agent. Point the neural cross-check at real recordings.
 - **On a real-speech recording** (a dual-channel barge-in fixture assembled
   from a recorded human utterance) both backends returned the same yield
   verdict; the measured yield timing differed (energy 0.65 s including its
   0.15 s hangover, neural 0.18 s, because a speech model segments the word
   gaps that the energy hangover bridges). One recording, reported as a timing
   observation, not an accuracy claim.
 - **Sample rates.** Silero accepts 8000 Hz, 16000 Hz, and integer multiples of
   16000 Hz (silero-vad decimates those itself and returns timestamps in the
   original sample coordinates). Any other rate fails with an actionable
   resample message from the seam. The energy backend measures at any rate.
 The real-model tests run automatically when the `[neural]` extra is installed
 and skip cleanly when it is absent (`tests/test_backend.py`).

## How validity is measured, not claimed

The bundled fixtures are synthetic: band-limited, syllable-modulated noise
rendered from exact segment boundaries declared in each scenario's
`reference_render` block (`caller_segments_sec`, `agent_segments_sec`, and an
optional `continuous` flag that renders one unbroken run per segment so the
active track equals the rendered boundaries to within one hop). Because the true
boundaries are known exactly, the scorer's **own measurement error** is
computable, and that is what to report, rather than an accuracy score:

- For a timing signal (onset, yield time, response gap), report
 `|measured - true|` in **milliseconds**, per scenario. The true value comes
 straight from the segment boundaries: true onset is the caller's first segment
 start; true yield time is the agent's segment end minus onset; true gap is the
 agent's next segment start minus the caller's turn end. Expect a residual on
 the order of the hop plus the hangover smear on trailing edges, read it, do
 not average it away.
- For `did_yield`, report a **confusion matrix** against the scenario `category`
 labels (`should_yield` vs `should_not_yield`), four cells: correct yields,
 missed yields, correct holds, phantom yields. Keep the cells separate.

**Never aggregate these into a single accuracy percentage.** A missed hard
interruption and a phantom yield on a backchannel are different failures with
different fixes; one blended number hides exactly the distinction the tool
exists to surface.

State plainly what the synthetic battery is for: it proves the plumbing works
end-to-end and catches regressions (a threshold change that moves a number shows
up immediately against the frozen golden output). It is **not** production audio
, no real accents, codecs, packet loss, or room acoustics. For validity on your
system, bring **10-15 of your own labelled calls** (two-channel where you can
get it) and run them through the same `run` / `--dump-frames` path; see
`CONTRIBUTING.md` and `docs/CORPUS-GOVERNANCE.md`.

## How to verify a score on your own recordings

1. Run `--dump-frames out.json` on the recording. Read the per-channel
 `*_threshold_db` and `*_noise_floor_db` from the header, and confirm
 `threshold = noise_floor + 15 dB` (or the dynamic-margin cap, or the -60 dBFS
 gate) matches your levels.
2. Scan the `*_dbfs` and `*_active` columns around the reported onset, yield,
 and gap indices and confirm each `active` flag is just `dbfs >= threshold`
 carried forward by the 150 ms hangover. Re-derive the signal by hand as above.
3. If the VAD is mislabeling your audio, tune and watch the numbers move
 predictably: raise `rel_db` or `abs_gate_db` if line noise is marked active;
 lower `rel_db` if quiet speech is missed; raise `hangover_sec` if one
 utterance fragments into several; adjust `yield_hangover_sec` /
 `caller_proximity_sec` to match how long *your* callers pause. Re-dump and
 confirm the thresholds and active flags moved as expected.

If you disagree with a number, the frames are on the table and every threshold
is yours to change. That is the point: a measurement you can audit, not a
verdict you have to trust.


================================================================================
FILE: docs/BAD-CALL-TO-CI.md
================================================================================

# From a bad call to a CI gate

One bad call moment becomes a permanent, offline regression test in five
steps. Everything, audio included, runs locally on your machine.

## The label comes from you

You label the expected behavior for the event: yield means the agent should
stop for the caller, hold means the agent should keep speaking through a
backchannel/noise/acknowledgement. Hotato measures whether the timing matched
that label.

That is the whole contract. "mhm" and "stop" can carry identical speech
energy, indistinguishable by timing alone. What Hotato measures, reproducibly,
is whether the agent did what your label says it should have done, and how
many seconds it took.

Input requirement: Hotato's main scorer requires separated caller and agent
tracks -- either one two-channel WAV or two aligned mono WAVs -- the level of
separation needed to attribute talk-over reliably.

## Step 1: turn the moment into a fixture

You have a recording where the agent talked over a caller at 42.18 seconds.
Label it:

```bash
hotato fixture create --stereo bad-call.wav \
    --id refund-interruption-001 \
    --onset 42.18 --expect yield \
    --max-talk-over 0.6 --max-time-to-yield 1.0 \
    --out tests/hotato
```

This writes `tests/hotato/scenarios/refund-interruption-001.json` (the label,
with provenance) and `tests/hotato/audio/refund-interruption-001.example.wav`
(a two-channel clip around the event, onset re-based to the clip). The
fixture is scored immediately on creation; an input that cannot be judged is
refused with the reason and exit code 2.

Do not know the onset? List the candidate moments first:

```bash
hotato scan --stereo bad-call.wav
```

`scan` reports timing facts only (overlap onsets, agent starts during caller
speech, long response gaps, an agent going quiet with no caller energy nearby,
a caller run that correlates with the agent's own audio, i.e. suspected TTS
echo). You pick the moment and supply the label.

## Step 2: run it

```bash
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio
```

The bad take fails, by design: that is the regression you are pinning. Exit
codes are the contract: 0 all pass, 1 a regression, 2 unusable input.

## Step 3: fix your agent, then compare

Change the setting, re-capture the same scenario, and let the numbers speak:

```bash
hotato compare --before bad-call.wav --after new-take.wav \
    --onset 42.18 --expect yield
```

```
hotato compare: bad-call.wav -> new-take.wav
  verdict:           FAIL -> PASS
  did_yield:         false -> true
  seconds_to_yield:  - -> 0.42s
  talk_over_sec:     2.65s -> 0.42s  improved -2.23s

result: fixed
```

If the moment shifted between takes, pass `--before-onset` and
`--after-onset` separately. `--out report.html` writes the shareable HTML
report with the before take as the base. A side that cannot be judged renders
NOT SCORABLE and exits 2.

## Step 4: gate CI on it

```yaml
# .github/workflows/turn-taking.yml
name: turn-taking
on: [pull_request]
jobs:
  hotato:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install hotato
      - run: hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio --format json
```

`hotato run` exits 1 on any regression, so the job fails when the timing
regresses. Already running pytest? `pytest --hotato-suite
--hotato-suite-scenarios tests/hotato/scenarios --hotato-suite-audio
tests/hotato/audio` adds the same gate to the run you have (see
[PYTEST.md](PYTEST.md)). The richer PR check with a sticky results comment is
in [CI.md](CI.md).

## Step 5: when it fails, plan the fix

```bash
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio \
    --format json > result.json
hotato plan result.json --stack vapi --assistant-id <id>
```

`plan` is read-only and guarded: it proposes at most one bounded step on one
setting, holds off tuning a single threshold when the battery fails on both
axes at once, and downgrades to a checklist when the evidence cannot isolate
the layer. It only proposes -- `platform_mutation.performed` is always false.
Details: [FIX-PLANS.md](FIX-PLANS.md).

## Use Hotato when

- You have (or can capture) separated caller/agent audio for the call.
- You can point at a moment and label what should have happened: yield or
  hold.
- You want a local, deterministic regression test that fails CI when the
  turn-taking timing regresses.

## When not to use Hotato

Hotato measures turn-taking timing from separated audio tracks. Outside that
lane:

- Transcript quality or wording checks -- Hotato doesn't transcribe.
- Task success or goal completion -- Hotato doesn't know what the call was
  for.
- Sentiment, tone, or emotion -- permanently out of scope.
- Compliance or script adherence review.
- Tool-call or API-behavior testing.
- Mixed mono recordings -- a summed channel can't attribute talk-over to a
  speaker.
- Live, in-call decisions -- Hotato scores recordings after the fact, not at
  runtime.
- Unlabeled whole-call analysis -- `hotato scan` surfaces candidate moments,
  and every verdict needs your yield or hold label.


================================================================================
FILE: docs/INGEST.md
================================================================================

# `hotato ingest` -- the composable passive on-ramp

Wire a webhook to invoke `hotato ingest` once, and every completed call is
scanned for **candidate** turn-taking moments automatically.

`ingest` surfaces timing candidates for you to review. You decide which ones
matter and promote them to a permanent regression test with
`hotato fixture create` -- the yield/hold label always comes from you.

It is built by **composition** and adds only a per-stack webhook parser:

```
parse the webhook payload   ->  extract the call id / recording locator
fetch the recording         ->  hotato.capture (the SAME fetch the adapters use)
scan for candidates         ->  hotato.scan (no labels, no verdict)
write a candidate report     ->  JSON always; --out an HTML report (optional)
```

## How it runs

- **You own the trigger.** Hotato ships the command; wire it to a webhook
  handler, a serverless function, or a cron over your call log, and it runs
  offline and self-hosted, on your infrastructure. The only network call is
  the same recording fetch `hotato capture` already makes.
- **You own the label.** A candidate is a timing event. Only you know whether
  a caller sound was "mhm" or "stop", so the label is yours.

## Contract

```
hotato ingest --stack {vapi|retell|twilio|livekit|pipecat} \
    (--event PAYLOAD.json | --call-id ID | --recording-sid RE...) \
    [--out report.html] [--format text|json] [--allow-mono] [--top N] [--min-gap S]
```

- **Exit 0** = ran (candidates reported, possibly zero).
- **Exit 2** = parse / fetch / IO error, or not-scorable input (for example a mono
  recording, which cannot attribute overlap to caller vs agent). Never a pass/fail.

`--format` controls stdout (`text` listing or the `json` candidate list, capped by
`--top`). `--out` additionally writes an HTML candidate report containing every
candidate. A webhook payload is **untrusted DATA**: `ingest` reads only the named
locator fields out of it.

## Wire your webhook -> `hotato ingest`

Point your platform's call-completed webhook at a small handler that saves the
payload and shells out to `ingest`. The pattern is identical for every stack:

```python
# a minimal webhook handler (framework-agnostic pseudocode)
import json, subprocess, tempfile, os

def on_call_completed(request):
    payload = request.body                       # the platform's webhook payload
    with tempfile.NamedTemporaryFile(
        "w", suffix=".json", delete=False) as fh:
        fh.write(payload)
        event_path = fh.name
    # ingest is one process; run it out-of-band so the webhook returns fast.
    subprocess.Popen([
        "hotato", "ingest", "--stack", "vapi",
        "--event", event_path,
        "--out", f"candidates/{request_id}.html",
    ], env={**os.environ, "VAPI_API_KEY": os.environ["VAPI_API_KEY"]})
    return 200
```

A cron over your call log works equally well when you would rather batch:

```bash
# nightly: scan yesterday's calls for candidates
for id in $(your-call-log --since yesterday --ids); do
  hotato ingest --stack vapi --call-id "$id" \
      --format json --out "candidates/$id.html" >> candidates/$id.json
done
```

### Per-stack recipe

Webhook field paths verified against live vendor docs (2026-07-07). Where a field
could not be confirmed from the live docs, `ingest` parses it **defensively** (a
missing field is simply absent, never fabricated).

| Stack | Webhook | Field ingest reads | Credentials for the fetch |
|-------|---------|--------------------|---------------------------|
| **vapi** | end-of-call-report | `message.call.id` (confirmed) | `VAPI_API_KEY` |
| **retell** | call webhook | top-level `event` + `call.call_id` (confirmed) | `RETELL_API_KEY` |
| **twilio** | `recordingStatusCallback` | `RecordingSid` (confirmed; form-encoded body) | `TWILIO_ACCOUNT_SID` + `TWILIO_AUTH_TOKEN` |
| **livekit** | egress webhook | `egressInfo.fileResults[].location` / `.filename` (defensive) | none -- egress lands in your storage |
| **pipecat** | your own event | `recording_path` / `recording_url` (defensive) | none -- you produced the file |

For **vapi / retell / twilio**, `ingest` extracts the identifier from the payload
and delegates the dual-channel recording fetch to the same adapter `hotato capture`
uses (which already resolved the recording URLs; see
[ADAPTER-STATUS.md](ADAPTER-STATUS.md)) -- the recording URL always comes from
that validated fetch, not the raw payload.

For **livekit / pipecat**, the recording lands in *your* infra, so the event
carries the locator directly. LiveKit egress files land in your storage bucket;
supply a `recording_url` (downloaded) or a `recording_path` (read locally). A
Pipecat event is whatever you emit, for example:

```json
{ "recording_path": "captured.wav" }
```

You can also skip the payload entirely with a direct id:

```bash
hotato ingest --stack vapi   --call-id  <id>   --out candidates.html   # + VAPI_API_KEY
hotato ingest --stack twilio --recording-sid RE... --format json       # + TWILIO_*
hotato ingest --stack pipecat --event event.json                        # local file
```

## From a candidate to a regression test

`ingest` finds the moment; you decide what should have happened and freeze it:

```bash
# 1. ingest surfaces a candidate at t=42.18s in a call recording
# 2. you listen, decide the agent should have yielded, and promote it:
hotato fixture create --stereo call.wav --onset 42.18 \
    --expect yield --id refund-cutoff-001 --out tests/hotato
# 3. it is now a permanent test:
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio
```

See [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md) for the full bad-call-to-CI loop.

## Notes on `--allow-mono`

Discovery needs one party per channel to attribute overlap. `--allow-mono` (or
`HOTATO_ALLOW_MONO=1`) lets the *fetch* pull a mono-only recording on retell/twilio
(matching `hotato capture`), but a mono mix is still reported **not-scorable**
(exit 2) for discovery, because overlap cannot be split between caller and agent.
Record dual-channel to get candidates.


================================================================================
FILE: docs/FIX-PLANS.md
================================================================================

# Fix plans: the guarded ladder

Hotato measures turn-taking; this ladder turns a failing measurement into a
reviewable, bounded fix proposal. Levels 0-2 are read-only. Level 3
(apply / verify) has shipped as a guarded, clone-only staged step.

## The ladder

| Level | Command | What it does | Writes to your stack |
|---|---|---|---|
| 0 | `hotato diagnose result.json` | Per-failure diagnosis + a battery decision, with the advisory and the tradeoff stated | never |
| 1 | `hotato inspect --stack ...` | Reads the CURRENT turn-taking config (GET or static parse) and normalizes it | never |
| 2 | `hotato plan result.json [target]` | Combines diagnosis + inspected config into a fix-plan JSON (`hotato.fixplan.v1`) | never |
| 3 | `hotato apply` / `hotato fix trial` / `hotato verify` | Applies a plan to a CLONE and re-scores the battery on it under a pinned manifest | cloned assistant / branch only, never production |

Level 3 kept the guard it was designed with, PR-first and clone-first:
`hotato apply` applies a plan to a CLONED assistant
(or a branch config), `hotato fix trial` re-scores the battery on that clone
under a pinned manifest, and `hotato verify` gates the before/after -- it
graduates to production only behind an explicit human approval with a recorded
rollback value. Every plan still pins
`"approval": {"default": "manual", "production_apply": false}`, so every plan
built here requires a human's manual go-ahead before it reaches production. See
[APPLY.md](APPLY.md), [FIX-TRIAL.md](FIX-TRIAL.md), and
[FIX-LOOP.md](FIX-LOOP.md).

## Level 0: diagnose

```
hotato run --suite barge-in --format json > result.json
hotato diagnose result.json
```

One diagnosis per failing event:

```
{"finding":          missed_real_interruption | false_stop_on_backchannel |
                     slow_yield | excess_talk_over | endpointing_miss |
                     not_scorable | threshold_funnel,
 "evidence":         the measured fields for that event,
 "likely_layer":     interruption_detection | endpointing | unknown_root_cause,
 "config_only_safe": bool,
 "notes":            plain language}
```

plus one battery-level decision. The text mode is the Level 0 advisory, for
example: "Missed real interruption. Likely config layer. Try lowering the
stop-speaking word threshold one step. Tradeoff: may increase false stops on
short acknowledgements." The tradeoff is always stated.

Exit codes: 0 no failing events, 1 failing events diagnosed, 2 unusable input.

## Level 1: inspect

```
hotato inspect --stack vapi --assistant-id <id>      # + VAPI_API_KEY
hotato inspect --stack retell --agent-id <id>        # + RETELL_API_KEY
hotato inspect --stack livekit --config agent.py     # static parse
hotato inspect --stack pipecat --config bot.py       # static parse
```

Output: one normalized model (`interrupt_min_words`,
`interrupt_voice_seconds`, `resume_backoff_seconds`,
`endpointing_wait_seconds`, `backchannel_aware`) plus the raw fields and
provenance (what was fetched, when, and which docs the field names were
verified against). Absent or unreadable options are null with a note.
Suspicious values (an unusually high word threshold, a long endpointing wait)
are surfaced as observations, for you to judge.

Read-only by construction: Vapi and Retell are one GET each; LiveKit and
Pipecat files are parsed with `ast`, statically, as text. Missing credentials
exit 2 cleanly.

## Level 2: plan

```
hotato plan result.json --stack vapi --assistant-id <id> --out hotato-fixplan.json
hotato plan result.json                              # stack from the envelope, else generic
```

(`--run result.json` is the equivalent flag form.) The input must be a run
envelope; frame dumps, benchmark results, and compare results are rejected
with exit 2.

The plan is `hotato.fixplan.v1` (shipped schema:
`src/hotato/schema/fixplan.v1.json`). It carries the finding, a hypothesis,
zero or more changes, the verification gate, and the approval block. Every
plan also carries `kind: "fix-plan"`, the measured `evidence` behind it, the
stated `risks`, `next_commands` (apply the step manually, verify with
`hotato compare`, re-run the battery), not-scorable events as `input_issues`
(input problems, kept separate from fixes), and a `platform_mutation` block
whose `performed` is always false: hotato plan is read-only.

Twilio rule: Twilio carries the audio; the upstream voice-agent stack decides
when the agent yields. So `--stack twilio` (or a Twilio-stack envelope) gets a
checklist instead of agent-config advice: confirm the dual-channel
caller/agent assignment, then re-plan against that upstream stack.

### Policy rules (verbatim, as implemented and tested)

A change is proposed ONLY when ALL of these hold:

  (a) the failure class maps cleanly to ONE setting;
  (b) the proposed value is ONE bounded step in an unambiguous direction
      within documented bounds, anchored to the inspected current value as
      `from`; when inspection was not run or not possible the plan carries
      direction and bounds only, with `current_unknown: true`;
  (c) the run's battery contains at least one passing OPPOSITE-RISK fixture,
      else the plan DOWNGRADES to insufficient_coverage ("add an
      opposite-risk fixture before tuning") and names the exact fixture
      family to add;
  (d) the diagnosis marked the failure config_only_safe.

And the standing refusals:

* threshold_funnel: when the battery contains BOTH a missed real interruption
  AND a false stop on a backchannel, the plan is a refusal:
  `"decision": "do_not_tune_single_threshold"` with a vendor-neutral
  engagement-control pointer (no product names, no digits in the pointer
  text). No single threshold satisfies both axes; raising it for one worsens
  the other -- the threshold treadmill teams describe from the inside: tuned
  endlessly, with no perfect setting, because both failures share one knob.
* slow_yield without a clear layer: TTS buffering, transport latency, and VAD
  smoothing are indistinguishable from one recording, so the plan proposes a
  diagnostic checklist (instrumentation steps) first. A slow yield becomes
  config-only-safe only when the battery contains a passing opposite-risk
  backchannel fixture that makes a one-step change verifiable.
* not_scorable events are input problems, tracked separately from agent
  failures and kept out of the plan.

Every plan, of every decision kind, carries the same verification gate:

```
"required_verification": ["real_interruption_fixture_must_pass",
                          "backchannel_fixture_must_not_regress",
                          "slow_yield_p95_must_not_worsen"]
```

### Worked example: a safe one-step plan

Battery: the agent missed a real interruption; the backchannel fixture
passes (the opposite-risk coverage the policy requires). Inspection read
`stopSpeakingPlan.numWords = 2` from the live assistant.

```json
{
  "schema": "hotato.fixplan.v1",
  "target": {"stack": "vapi", "inspected": true,
             "assistant_id": "asst_123", "current_unknown": false},
  "finding": "missed_real_interruption",
  "hypothesis": "The agent missed a real interruption: the caller took the floor and the agent kept talking. Inspected stopSpeakingPlan.numWords is 2; one bounded step decrease is the smallest verifiable change on this axis.",
  "config_only_safe": true,
  "decision": "propose_one_step",
  "changes": [
    {"field": "stopSpeakingPlan.numWords",
     "from": 2, "to": 1, "direction": "decrease", "bounds": [0, 10],
     "reason": "The caller took the floor and the agent never stopped within the search window. A one-step sensitivity increase is a config-layer candidate; the tradeoff is more false stops on short acknowledgements. One step decrease from the inspected value. Bounds basis: documented range 0-10 (docs.vapi.ai, 2026-07-06).",
     "risk": "more false stops on short acknowledgements; the backchannel fixture must not regress"}
  ],
  "required_verification": ["real_interruption_fixture_must_pass",
                            "backchannel_fixture_must_not_regress",
                            "slow_yield_p95_must_not_worsen"],
  "approval": {"default": "manual", "production_apply": false}
}
```

One field, one step, both endpoints of the move visible, the risk named, the
verification gate attached: every value in the plan traces back to the
inspected config or a documented bound.

### Worked example: the refusal

Battery: the packaged demo (`hotato demo --format json`), which misses a real
interruption AND yields to a bare backchannel.

```json
{
  "schema": "hotato.fixplan.v1",
  "target": {"stack": "generic", "inspected": false},
  "finding": "threshold_funnel",
  "hypothesis": "The battery missed a genuine interruption and also stopped for a backchannel. One sensitivity threshold cannot satisfy both: raising it to hold through backchannels drops real interruptions, lowering it to catch interruptions yields to backchannels. The failure class is discrimination, not calibration.",
  "config_only_safe": false,
  "decision": "do_not_tune_single_threshold",
  "changes": [],
  "recommended_fix": {
    "class": "engagement-control",
    "examples": [
      "enable adaptive interruption handling where available",
      "use a backchannel-aware interruption classifier",
      "add addressee/turn-intent discrimination before stopping TTS"
    ]
  },
  "required_verification": ["real_interruption_fixture_must_pass",
                            "backchannel_fixture_must_not_regress",
                            "slow_yield_p95_must_not_worsen"],
  "approval": {"default": "manual", "production_apply": false}
}
```

The refusal is the product working as designed: the strongest thing a tuning
tool can say about a funnel is that tuning will not fix it.

## Provenance

Vendor field names and documented ranges used by inspect and plan were
verified against docs.vapi.ai (assistant startSpeakingPlan/stopSpeakingPlan),
docs.retellai.com (get-agent), docs.livekit.io (turn handling options), and
docs.pipecat.ai (user turn strategies) on 2026-07-06, and each result records
its own `field_basis`. Where a platform documents no hard range, the plan says
so and uses a conservative working range with a note to verify against the
installed version.


================================================================================
FILE: docs/FIX-LOOP.md
================================================================================

# The closed loop: find -> fix -> prove it's fixed

`hotato diagnose` / `inspect` / `plan` (see [FIX-PLANS.md](FIX-PLANS.md)) end at
a **proposal**. The closed loop carries it the rest of the way, with the two
irreversible decisions kept firmly in human hands:

1. **`hotato patch`** turns a plan's abstract `{field, from, to}` into a
   **literal, paste-ready artifact** for your platform. It PRODUCES the
   artifact; applying it is your call.
2. You apply it, in your own stack, and re-capture the failing moments.
3. **`hotato verify`** scores the old and new runs against each other and gives
   you a **battery-scale before/after proof**: *N of M fixtures that used to
   fail now pass, and K of L hold fixtures still pass.*
4. **`hotato loop`** orchestrates the whole thing and **remembers where it left
   off**, so you can walk away and come back.

Every step here produces a proposal, a label, or a reviewable artifact -- you
decide what ships to your platform.

## `hotato patch <fixplan.json>` -- the paste-ready change

Reads a fix plan (schema `hotato.fixplan.v1`) and renders it per platform:

- **Vapi / Retell** (config behind a REST API): a JSON **merge-patch body** plus
  a ready **`curl`** against the platform's real config-update endpoint, using
  the exact field names the plan carries.

  ```
  hotato plan result.json --stack vapi --assistant-id <id> --out fixplan.json
  hotato patch fixplan.json
  ```
  ```
  curl -X PATCH https://api.vapi.ai/assistant/<assistant-id> \
    -H "Authorization: Bearer $VAPI_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"stopSpeakingPlan": {"numWords": 2}}'
  ```

- **LiveKit / Pipecat** (config lives in your agent source): there is no
  config-update REST call to hit, so patch emits the exact **source edit** --
  the constructor kwarg and the literal value to set -- never a fabricated
  endpoint. For example `InterruptionOptions(min_words=1)`.

- **generic / unknown stack**: patch names the knob **family** and asks for a
  concrete stack; it emits no literal body it cannot stand behind.

### The both-axes case: an engagement-control pointer

`hotato patch` only handles the config-fixable classes. When the plan's decision
is `do_not_tune_single_threshold` -- the **both-axes** case, where the
battery misses a real interruption AND false-stops on a backchannel at once -- no
single config value fixes both. patch prints the vendor-neutral, numbers-free
**engagement-control pointer** instead: it names the problem class
(discriminating a real bid for the floor from a backchannel) and the KIND of
fix it needs, naming no product and carrying no digits. It fires ONLY on this
case; an ambiguous slow yield or a coverage gap gets a "no patch, here's why"
explanation instead.

Every other non-propose decision (diagnostic checklist, insufficient coverage,
already at a documented bound, no change) likewise emits no patch, with the
reason pointing back at the plan.

**Guardrail:** patch runs entirely offline and pins `applies_change` to false.
`--format json` emits the full artifact; `--out PATH` also writes it.

## `hotato verify --before <old> --after <new>` -- the proof

After you apply the change and re-capture the same fixtures, verify scores the
old and new run envelopes against each other. Each side is a single `hotato run`
envelope JSON (a whole battery), or a directory of them; fixtures pair by
`event_id` (then `scenario_id`).

```
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio \
    --format json > before.json          # the failing take
# ... apply the patch, re-capture the fixtures ...
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio-new \
    --format json > after.json
hotato verify --before before.json --after after.json
```

It **reuses** the `hotato compare` taxonomy per fixture (`fixed`, `regressed`,
`improved`, `worse`, `unchanged`, `still_pass`, `not_scorable`) and aggregate's
pooled-distribution definitions for the before/after talk-over and time-to-yield
shift. The rollup is the two axes that matter:

- **regression axis**: how many previously-failing fixtures now pass;
- **hold axis**: how many hold-labeled guard fixtures still pass (they must not
  regress).

**Guardrail:**

- verify reports **coincidence**: it says the improvement *coincides with*
  your change. Hotato measures timing, not a controlled experiment, so a
  coincidence is exactly the strength of claim the evidence supports.
- it **refuses** the battery-scale claim when too few fixtures failed to
  characterize (`--min-n`, default 3): the per-fixture facts still print, but the
  headline proof is withheld and said so.
- an unjudgeable side reports `not_scorable`; a fixture on only one side is
  reported unpaired, kept out of the rollup.

By default verify measures and exits 0; `--fail-on-regression` exits 1 if any
fixture regressed or got worse.

### `--policy hotato.verify.yaml` -- the anti-bandaid gate

`--policy` turns the measured rollup into a PASS/FAIL gate you can drop into CI.
A policy declares two things, and **both** must hold for verify to pass:

```yaml
target:
  improve:
    talk_over_sec_p95: -0.5   # pooled talk-over p95 must drop by >= 0.5s
    failed_count: decrease    # fewer fixtures may fail than before
guardrails:
  max_new_false_yields: 0     # no hold guard that passed before may newly yield
  max_not_scorable: 0         # every paired fixture must be judgeable
  require_hold_fixture: true  # the battery must contain a hold fixture...
  require_yield_fixture: true # ...and a yield fixture
```

```
hotato verify --before before.json --after after.json --policy hotato.verify.yaml
```

- **`target.improve`** is the success criteria. A signed number is a required
  delta (`after - before`), so `-0.5` means "must improve by at least 0.5"; a
  keyword (`decrease`, `increase`, `no_worse`, `no_better`, `unchanged`) states
  direction only. Metrics: `talk_over_sec_p95`, `seconds_to_yield_p95`,
  `failed_count`, `false_yield_count`.
- **`guardrails`** are hard fail conditions. `max_new_false_yields` and
  `max_not_scorable` cap what a naive threshold bandaid would silently trade in;
  `require_hold_fixture` / `require_yield_fixture` make sure a battery tests
  the opposite axis before it can be certified.

verify exits `1` unless **every guardrail holds AND every target is met**, so a
fix only passes when it improves one axis without regressing (or skipping) the
other. A patch that cuts talk-over by making the agent yield to everything meets
the talk-over target but trips `max_new_false_yields` on the hold fixtures, and
the whole check fails. The guardrails and targets are shown in the `--out
verify.html` proof (a `Policy check: PASSED`/`FAILED` headline and an
ok/violated, met/unmet table) and in the text and JSON output.

The policy is parsed with the standard library only -- Hotato's core carries no
third-party runtime dependency -- over the small subset the shipped
`examples/verify-policy/hotato.verify.yaml` uses. An unknown key, a wrong-typed
value, an empty policy, a tab indent, or a list is a clean exit-2 usage error.

## `hotato loop [FOLDER]` -- one command, with memory

`hotato loop` drives the parts Hotato can drive and remembers where it left off
in a small local state file (`.hotato/loop-state.json` by default):

- **First run over a folder of calls**: it runs discovery (`analyze` -> `scan` ->
  rank) and records the candidate moments. Stage `awaiting_label`:

  > you have N candidate moment(s) awaiting your label.

- **You label the ones that matter** with `hotato fixture create` -- the
  yield/hold intent always comes from a human.

- **Next run, with those fixtures present** (`--fixtures DIR`): it runs them,
  diagnoses the battery (including the threshold-funnel check), and plans a
  guarded fix. Stage `awaiting_verify`:

  > a fix plan is ready; apply it with hotato patch, then prove it with hotato
  > verify.

```
hotato loop ./recordings                          # run 1: discover
hotato fixture create --stereo rec.wav --onset 12.4 \
    --expect yield --id refund-001 --out tests/hotato
hotato loop ./recordings --fixtures tests/hotato   # run 2: plan
```

**What stays yours:** every yield/hold intent (you supply it), and applying the
fix (loop produces a plan and points at `hotato patch`; applying and verifying
stay your steps). loop orchestrates and tracks state across runs, leaving your
platform untouched -- you keep the two decisions that matter: which moment is
a real bug, and whether to apply the fix.

## Exit codes

- `hotato patch`: `0` a patch (config artifact or the engagement-control
  pointer) was produced; `2` the input is not a fix plan or is unreadable.
- `hotato verify`: `0` the rollup was produced (a low-n claim is refused but
  still exits 0); `1` a gate you opted into failed -- with `--fail-on-regression`
  a fixture regressed or got worse, or with `--policy` a guardrail was violated
  or a `target.improve` criterion was not met; `2` usage error, unreadable
  input, an invalid `--policy` file, or no fixtures pair.
- `hotato loop`: `0` advanced or re-reported state; `2` no folder on the first
  run, an unreadable state file, or a path that is not a folder.


================================================================================
FILE: docs/APPLY.md
================================================================================

# hotato apply: the guarded, clone-only staged apply

`hotato apply` is the last rung of the fix ladder and the ONLY command in Hotato
that can mutate external platform state. Because of that, it is the most
conservative command in the codebase: every flag combination leaves your
production/source assistant untouched. It reads a `hotato patch` artifact and
either PRINTS the fresh staging clone it would create (the default, fully
offline dry run) or, only with `--yes` and credentials, creates a NEW staging
assistant that is your source config with the patch applied.

```
hotato patch fixplan.json --format json --out patch.json
hotato apply patch.json --clone --name staging-refund-fix --battery tests/hotato
```

## The five hard rules, enforced in code

1. **Clone-only.** This version ships one apply path: staging-clone only. A
   non-`--clone` invocation is a clean usage error: `production apply is not
   supported; use --clone to apply to a fresh staging assistant`. The one
   writing call is a `POST` that creates a NEW assistant, never a `PUT`/`PATCH`
   against the source.

2. **Refusal-first.** If the patch is the both-axes threshold funnel (the plan
   decided `do_not_tune_single_threshold`), apply REFUSES before doing anything
   and prints the exact recommendation:

   ```
   No config patch will be applied
   Reason: both missed real interruption and false stop on backchannel, one threshold cannot safely fix both
   Recommended: enable or add engagement-control / backchannel-aware turn detection
   ```

   The refusal is a FEATURE. It exits with a distinct, documented code (`3`), so
   a script can tell "refused by design" apart from a usage error. A single
   sensitivity threshold trades against itself between catching an
   interruption and holding through a backchannel -- solving it takes a
   discrimination fix, and every clone's patch reflects that.

3. **Opposite-risk required.** apply requires `--battery` to carry BOTH a
   yield fixture (an interruption the agent must stop for) AND a hold
   fixture (a backchannel the agent must keep the floor through), so it can see
   the opposite risk a threshold move trades into before applying. Point
   `--battery` at your fixtures directory (with a `scenarios/` folder, as
   `hotato fixture promote` writes) or at a folder of run-envelope / scenario
   JSONs.

4. **Gated side effect.** The default is a dry run that prints exactly the clone
   it would create and the patch it would apply, fully offline. Only `--yes`
   WITH credentials reaches the platform. The create
   call is the only networked function: it reads the source config (`GET`),
   applies the patch to a copy, and creates a NEW assistant (`POST`).

5. **Name required.** The staging clone must be named explicitly (`--name`);
   apply always uses the name you give it.

## What gets cloned

Only the REST-config stacks have an assistant/agent to clone through an API:

| stack  | read (source, read-only)          | create (a NEW assistant)              | name field   |
| ------ | --------------------------------- | ------------------------------------- | ------------ |
| vapi   | `GET https://api.vapi.ai/assistant/{id}` | `POST https://api.vapi.ai/assistant`  | `name`       |
| retell | `GET https://api.retellai.com/get-agent/{id}` | `POST https://api.retellai.com/create-agent` | `agent_name` |

The clone config is your source config with the patch deep-merged on top, given
the new name, and stripped of the server-assigned ids so it is a fresh object,
never an overwrite of the source.

LiveKit and Pipecat keep turn-taking config directly in your agent SOURCE, so
there is no assistant to clone through an API; `hotato patch` already emits the
exact source edit for those stacks, and apply points you straight at it.
Twilio carries the audio but runs no turn-taking agent, so apply points at the
upstream stack.

## After the clone: prove it

The clone is a staging assistant so you can prove the fix before touching
production. Re-capture the same battery through the SOURCE (into `before/`) and
through the CLONE (into `after/`), then verify:

```
hotato verify --before before/ --after after/ --policy hotato.verify.yaml
```

`hotato verify` reports coincidence, never causation, and the policy is the
anti-bandaid gate: the fix passes only if every guardrail holds (including
`max_new_false_yields` on the hold fixtures) AND every target is met, so a patch
that just makes the agent yield to everything is caught. See `docs/FIX-LOOP.md`.

Or run the whole thing (this gate + verify + an optional contracts re-verify +
explain's attribution) as one fail-closed before/after report:
`hotato fix trial patch.json --name staging-refund-fix --before before/ --after
after/`. See [`docs/FIX-TRIAL.md`](FIX-TRIAL.md).

## Exit codes

- `0` the staging clone was rendered (dry run by default; with `--yes` and
  credentials the NEW staging assistant was created and patched, the source
  untouched).
- `2` usage error: no `--clone`, no `--name`, no opposite-risk battery (both a
  yield and a hold fixture), a stack with no assistant to clone, a patch that
  produced no config change, or unreadable input.
- `3` principled refusal: the plan is the both-axes threshold funnel, so apply
  refuses a single-threshold patch by design. The refusal is the feature.


================================================================================
FILE: docs/FIX-TRIAL.md
================================================================================

# fix trial: one before/after proof, composed, fail-closed

`hotato fix trial` is the last rung of the fix ladder: it composes the
already-shipped, already-guarded primitives into ONE before/after report that
says whether a candidate change holds. No new scoring engine, no new
networked path:

* **`hotato apply`'s exact offline gate** (`build_apply`, `clone=True`):
  refusal-first on the both-axes threshold funnel, opposite-risk-battery-
  required, clone-only. fix trial never creates a clone itself and never
  touches the network -- it never calls `apply.create_clone` /
  `apply._http_json`, so it carries the same clone-only, production-
  unmutatable guarantee apply's own dry run gives, by construction.
* **`hotato verify`'s battery-scale rollup**: the BEFORE run (the original
  failure evidence) against the AFTER run (re-captured through the clone you
  created separately with `hotato apply --clone --yes`), scoring EVERY paired
  fixture, not just the target failure -- the "neighbouring cases" check.
* **`hotato contract verify`**, when `--contracts DIR` is given: another
  neighbouring-cases check, on real labelled moments outside the battery.
* **`hotato explain`**, folded in as the report's attribution section: root
  cause of the ORIGINAL failure, reused exactly.

```
hotato patch fixplan.json --format json --out patch.json
hotato apply patch.json --clone --name staging-refund-fix --battery tests/hotato
# ... re-capture the battery through the source (before/) and the clone (after/) ...
hotato fix trial patch.json --name staging-refund-fix \
    --before before/ --after after/ --battery tests/hotato \
    --policy hotato.verify.yaml --out fix-trial.json --html fix-trial.html
```

## The verdict is fail-closed, never a soft pass

| Verdict | When | Exit |
| --- | --- | --- |
| `improved` | the verify claim is supported (>= `--min-n` previously-failing fixtures), at least one now passes, NOTHING regressed anywhere in the battery (including the hold/opposite-risk axis), no contract regressed, `--policy` (if given) passed, every before target and hold has an after counterpart, AND every guarded fixture (target and hold) carries VERIFIABLE before/after audio identity (well-formed, freshly distinct decoded PCM, recomputed from the audio on disk -- the provenance guard, below) | `0` |
| `regressed` | any fixture regressed, a contract regressed, or the policy failed | `1` |
| `inconclusive` | too few previously-failing fixtures to characterize, nothing that used to fail now passes, OR the audio identity is present-but-unverifiable for a guarded fixture (a malformed provenance block, a missing block, or a well-formed assertion hotato could not recompute at trial time because the audio was not present) | `1` |
| `refused` | the patch is the both-axes threshold funnel (apply's refusal-first gate fires before any before/after evidence is even read); OR the after set drops a required before fixture (an incomplete, cherry-picked comparison); OR a guarded fixture's recorded provenance does not match the audio on disk; OR a guarded fixture's before/after audio is the SAME conversation (identical decoded PCM -- a re-score, not a recapture) | `3` |

**`inconclusive` is fail-closed, not a pass.** A low-n battery or a
zero-improvement battery exits the SAME non-zero code as a real regression,
so CI never treats "we could not tell" as green. A hold fixture that flips
from passing to failing (the opposite-risk axis) is `compare.classify_pair`'s
`"regressed"` result whichever axis it is on, so it is caught by the same
check that catches a talk-over regression -- there is no separate,
weaker gate for the opposite-risk axis.

## Refusal: correct output, not an error

If the patch is `do_not_tune_single_threshold` (the battery missed a real
interruption AND false-stopped on a backchannel in the same battery), fix
trial refuses before reading `--before` / `--after` / `--contracts` at all,
prints the exact canon recommendation, and exits `3` -- the SAME distinct
code `hotato apply` uses for the same refusal, so a script that already
branches on apply's refusal code recognizes fix trial's too.

```
No config patch will be applied
Reason: both missed real interruption and false stop on backchannel, one threshold cannot safely fix both
Recommended: enable or add engagement-control / backchannel-aware turn detection
```

## Fresh-capture provenance guard: a re-score is never a fix

`hotato apply`'s clone-only gate and `hotato verify`'s battery-scale rollup
answer "did the numbers move." Neither one asks whether the AFTER evidence
was RE-CAPTURED, or is just the SAME recording the BEFORE run
scored, re-scored under a looser threshold. That gap is exploitable: run the
same fixture twice with different `--max-time-to-yield` / `--max-talk-over`
bounds, and you get a convincing "improved" verdict with no code,
config, or model change behind it at all.

Every run envelope records an `audio_provenance` block per event: a streamed
sha256 of the raw file bytes AND a streamed sha256 of the decoded PCM samples
(plus sample rate and frame count), computed at capture time by `hotato run`
/ `hotato capture`. `hotato fix trial` does not trust the string; it VERIFIES
the identity for every GUARDED fixture -- the fail->pass targets AND the
still-passing holds (a frozen hold is a re-score too, so holds get the same
guard). The guard's job is narrower: make the
motivated failure modes impossible or loud, recompute what can be recomputed
from the actual files, and state exactly what was and was NOT verified:

| Situation | What it is | Effect on verdict |
| --- | --- | --- |
| Well-formed, freshly distinct (decoded PCM) identity, recomputed from the audio present on disk and matching | a verified fresh recapture | proceeds -- eligible for `improved` |
| Identical decoded PCM before vs. after on any guarded fixture | the after run re-scored the SAME conversation (a header-only edit or trailing-byte append cannot hide it -- the comparison is on samples, not container bytes) | `refused` (exit `3`) |
| Recorded digest does NOT match the audio present on disk | the provenance was hand-edited or the audio was swapped after capture | `refused` (exit `3`) |
| A required before fixture (target or hold) missing from the after set | a cherry-picked, incomplete comparison | `refused` (exit `3`) |
| Malformed block (non-hex digest, absurd sample rate / frame count, or a top-level digest inconsistent with the per-side digests) | an unvalidated assertion, not a distinct recording | `inconclusive` (exit `1`) |
| A provenance block missing on either side | an older envelope, or one hand-built without `audio_provenance` -- identity is UNKNOWN | `inconclusive` (exit `1`) |
| Well-formed identity that hotato could NOT recompute (the audio was not present) | asserted, not proven | `inconclusive` (exit `1`) |

A provenance-guard refusal is NOT the apply-gate refusal: it fires AFTER
`verify` / `contract verify` / `explain` have already run, so (unlike the
both-axes refusal, which reads no evidence at all) the full report --
verify's proof, the contract rollup, the provenance identities, the
attribution -- still renders below the refusal banner. Every refusal path
exits the SAME code `3`; `refusal_kind` in the JSON output
(`"threshold_funnel"`, `"incomplete_after"`, `"recompute_mismatch"`,
`"same_audio_recapture"`) tells them apart for a script that wants to.

```
No fix will be certified from re-scored audio
Reason: 1 fixture(s) this verdict rests on (f1) have identical before/after decoded PCM: the after run re-scored the SAME conversation the before run scored, just against a different threshold or scorer config
Recommended: recapture the fixture(s) through the applied clone (hotato apply --clone --yes) and re-run hotato fix trial against the new after evidence
```

Every rendered report (text, `--format json`, `--html`) surfaces the short
digest and the verified status for every guarded fixture, before and after --
so a reader never has to take "fresh capture" on faith, and can see exactly
what was verified versus merely asserted. The effective `--min-n` is echoed
in every surface too, so a lowered floor is always visible. The report's own
conclusion states plainly what a passing check proves: that the fresh take
passed the same human-labeled contract, not that the change caused it (hotato
reports coincidence, never causation, throughout).

The text and HTML renders of this section also print, verbatim, wherever it
appears (an `improved` verdict, or a `refused`/`inconclusive` one the guard
itself downgraded): *"Provenance caution: this proves the specific fresh
capture scored above, at the revision it was captured from. It does not
certify a later deploy or every future call, and it does not re-run itself;
recapture again after the next change."* See
[`docs/RECAPTURE.md`](RECAPTURE.md#claim-language-what-each-kind-of-evidence-lets-you-honestly-say)
for the fuller claim-language table this line is drawn from.

## What this does not stop

This is an offline tool: a user who controls every input can always lie to
themselves. Recomputing identity from the actual audio (above) makes the
specific forgeries an external red-team demonstrated against a prior build --
hand-written envelopes, a flipped header byte, a re-scored recording, a
cherry-picked after set -- impossible or loud. It does not close everything,
and none of the following is a bug the guard failed to catch; each is outside
what an offline recompute over supplied files can ever establish:

* **Fabricated inputs are still yours to fabricate.** Hand fix trial a
  fresh recording of a call that never happened, or one that does
  not match the bug you are claiming to fix, and the guard verifies
  the audio identity and still reaches `improved`. It checks that
  the bytes it scored are what they claim to be, never that the stimulus
  itself is real.
* **A contract's `MANIFEST.sha256.json` is integrity, not authenticity.** It
  proves the archive still agrees with itself after packing; it does not
  prove who approved the policy inside it. Loosen a `.hotato` bundle's
  policy (raise `max_talk_over_sec`, say) before `contract pack`, and
  `contract verify` on the repacked bundle still passes -- it is re-checking
  the archive against itself, not against an external record of what the
  policy was supposed to be. Only a trusted signature over the manifest
  closes this; none is implemented today.
* **A resample, re-encode, or gain change of the SAME call still changes the
  decoded PCM.** The freshness check above is exactly "is the decoded PCM
  different." A deliberately transcoded copy of the identical recording
  (resampled, gain-adjusted, round-tripped through a lossy codec) decodes to
  different samples, so it reads as a distinct capture -- it is not; it is
  the same call in a different container. This is a known, undetected
  residual of a PCM-identity check, not a claim the guard makes and breaks.
* **Signatures are not implemented.** Nothing here is cryptographically
  signed; a sha256 digest is a checksum, not an attestation of who produced
  it.

A green fix trial does not prove the audio was freshly captured for the
scenario claimed, that the same policy and labels were used throughout, that
any omitted fixture was safe to omit, that the named revision or clone
existed, that the patch was applied to it, or that the deployed agent
improved. The verdict states exactly what it recomputed and verified from
the files it was given; everything else is a boundary of the offline model,
not a promise this tool makes and quietly fails to keep.

## Flags

| Flag | Meaning |
| --- | --- |
| `PATCH_JSON` | a `hotato patch` artifact |
| `--name NAME` | name of the staging clone this trial is proving (required for a non-refused patch; the same `--name` `hotato apply` takes -- fix trial evaluates the SAME clone-only gate, it never creates a clone itself) |
| `--before RUN.json\|DIR` | the OLD run envelope(s): the original failure evidence. Also the default opposite-risk `--battery`, and the attribution source, when those are omitted |
| `--after RUN.json\|DIR` | the NEW run envelope(s), re-captured through the staging clone |
| `--battery DIR` | the opposite-risk battery apply's gate checks (BOTH a yield and a hold fixture); defaults to `--before` |
| `--contracts DIR` | also re-verify a directory of hotato contracts; any contract regression fails the trial |
| `--policy hotato.verify.yaml` | gate verify's rollup (see `docs/FIX-LOOP.md`); a violation fails the trial |
| `--min-n N` | minimum previously-failing fixtures needed to support the claim (default 3) |
| `--out PATH` | also write the full proof JSON |
| `--html PATH` | also write a self-contained before/after HTML report |

## Output

Every surface renders the apply receipt right beside the verdict, not buried
in the raw `apply` sub-object: fix trial calls `apply.build_apply` and never
`apply.create_clone`, so `apply_dry_run` is `True` and `apply_created` /
`apply_applies_change` are `False` on EVERY run, including an `improved` one.
A green verdict never means "and the change was applied" -- the report says
so in the same breath as the verdict, in text (`apply: dry_run=True
created=False applies_change=False`, plus the plain-English sentence right
under it), JSON (top-level `apply_dry_run` / `apply_created` /
`apply_applies_change` / `apply_receipt_note`, alongside `verdict`), and HTML
(pills and a header line, in the `<header>` block itself). Proving the change
reached the clone or agent is `hotato apply --clone --yes`'s job,
recorded in its own receipt -- fix trial only proves what the before/after
evidence shows once that has already happened.

* **text** (default): the apply receipt, the verdict, verify's own rendered
  proof, the contract verify rollup when `--contracts` was given, and the
  attribution section (one `hotato explain` render per file under
  `--before`). When the trial's own verdict is not `improved` (a provenance,
  completeness, contract, or policy issue downgraded it), verify's nested
  `CLAIM` line is tagged `CLAIM (SUPERSEDED BY {VERDICT})` with a one-line
  restatement, whenever that sub-claim would otherwise read "supported" --
  because a fix-trial verdict of `regressed` / `refused` / `inconclusive`
  can still contain a verify claim that, read alone, looks like a pass; the
  parent verdict controls, and the render says so rather than leaving a
  clean-looking claim floating under a red result.
* **`--format json`**: the full machine shape, schema `hotato.fix_trial.v1`.
  Every sub-result is the REAL nested result the underlying command already
  produces (`apply`, `verify`, `contract_verify`, `attribution`) -- nothing
  here is a re-derived summary. `apply_dry_run` / `apply_created` /
  `apply_applies_change` / `apply_receipt_note` sit at the top level next to
  `verdict`, not only inside the nested `apply` object.
* **`--html PATH`**: a self-contained report, reusing the same house style
  the other HTML reports use: the apply-receipt pills and note in the
  header, the verdict chip, the verify proof table (with the same
  superseded-claim label as text, when it applies), the contract-verify
  rollup, and the attribution cards.

## Exit codes

| Code | Meaning |
| --- | --- |
| 0 | `improved` |
| 1 | fail-closed: `regressed` or `inconclusive` |
| 2 | usage error: the same gates `hotato apply` enforces (no `--name`, no opposite-risk battery, a stack with no clone target, a patch with no concrete change) or `hotato verify` / `contract verify` already enforce (no fixtures pair, an invalid `--policy`, a `--contracts` dir with no contracts, unreadable input) |
| 3 | principled refusal: the both-axes threshold funnel. Shared with `hotato apply`'s refusal code |

## What this is not

fix trial does not create, patch, or delete anything on your platform; it
never calls `apply.create_clone`. It does not prove authorization, identity,
compliance, or policy safety. It reports coincidence, never causation --
every underlying `verify` and `contract verify` number keeps that rule. See
[`FIX-LOOP.md`](FIX-LOOP.md) for the manual steps this composes, and
[`APPLY.md`](APPLY.md) / [`EXPLAIN.md`](EXPLAIN.md) for the primitives
themselves.


================================================================================
FILE: docs/ANALYZE.md
================================================================================

# `hotato analyze <folder>`: drop a folder, hear the bug

Zero-config discovery over a whole folder of dual-channel call recordings:
point it at the folder and it does the rest.

```bash
hotato analyze ./recordings
```

That writes one self-contained, offline HTML dashboard (`hotato-analyze.html` by
default) and opens it. `hotato ./recordings`, a bare folder as the first
argument, routes to the same command.

## What it does

For every `.wav` under the folder (walked recursively, in sorted order):

1. runs the same whole-call scanner as [`hotato scan`](../src/hotato/scan.py):
   it walks the caller and agent VAD activity tracks across the ENTIRE call,
   label-free, and emits candidate timing moments
   (`overlap_while_agent_talking`, `agent_start_during_caller`,
   `long_response_gap`, `agent_stop_no_caller`, `echo_correlated_activity`),
   each with a timestamp and a measured number;
2. aggregates the candidates across ALL calls and ranks them by the scanner's
   own salience (overlap seconds / gap seconds / echo coherence) so the worst
   moments float to the top.

Then it emits three things.

### 1. A ranked dashboard

One card per top moment, in the [`hotato report`](REPORTS.md) house style: the
call file, the timestamp, the candidate kind, the measured number, and a
to-scale caller/agent timeline of that exact moment (the same SVG renderer the
report uses: activity spans, the shaded talk-over band, the onset marker, and
a yield marker where the scanner measured the agent going silent).

### 2. The hear-the-bug player

For the top `--audio-top` moments (default 8) the audio around the moment
is embedded inline as a base64 WAV data URI, so the page is fully
self-contained with zero external requests. Press play and a **playhead**
sweeps that moment's
timeline in lockstep with `audio.currentTime` (via `requestAnimationFrame`), so
you HEAR the agent talk over the caller, or the dead-air gap, land exactly where
the chart marks it. Reduced-motion safe: with `prefers-reduced-motion: reduce`
the playhead still tracks playback (it rides `timeupdate` instead of the smooth
animation loop).

Only the top moments carry audio; the rest show the timeline only. That, plus
`--pre` / `--post` (the seconds of audio kept before/after each moment) and a
total-page audio budget, keeps the page a reasonable size.

### 3. JSON for agents

```bash
hotato analyze ./recordings --format json
```

prints the ranked candidates plus their metadata (source file, timestamp, kind,
salience, measured durations, and the audio/timeline window) to stdout, capped
by `--top`. Pass `--out FILE` to also write the full ranked JSON.

## Framing

These are **measured candidate timing moments**: a timestamp and a number,
not a verdict on intent. Energy sounds the same whether a caller said "mhm"
or "stop", so you decide the expected behavior and label the moments that
matter:

```bash
hotato fixture create --stereo <call>.wav --onset <t> \
    --expect yield|hold --id found-moment-001 --out tests/hotato
```

Non-dual-channel or otherwise unreadable files are reported cleanly in a
"Skipped files" section with their reason (a mono mix cannot attribute
talk-over), and the run keeps going through the rest of the folder.

## Flags

| flag | default | meaning |
| --- | --- | --- |
| `FOLDER` | (required) | directory of dual-channel WAVs, walked recursively |
| `--top` | 25 | ranked moments shown in the dashboard / stdout JSON (0 = all) |
| `--audio-top` | 8 | top moments that get the embedded hear-the-bug player |
| `--pre` | 2.0 | seconds of audio/timeline kept before each moment |
| `--post` | 4.0 | seconds of audio/timeline kept after each moment |
| `--min-gap` | 2.0 | minimum response gap (seconds) to surface as a candidate |
| `--caller-channel` / `--agent-channel` | 0 / 1 | channel assignment |
| `--format` | `html` | `html` (dashboard) or `json` (ranked candidates) |
| `--out` | `hotato-analyze.html` | where to write the dashboard, or the JSON |
| `--no-open` | off | do not launch a browser for the dashboard |

## Exit codes

- **0**: ran -- candidate moments listed across the folder, possibly zero.
- **2**: usage error (the path is not a folder) or an IO error reading it.

Everything, audio included, stays on your machine.


================================================================================
FILE: docs/TRUST.md
================================================================================

# `hotato trust --stereo call.wav`: is this recording even scorable?

The input-health check, or "trust doctor": inspect ONE recording and report
whether the audio is good enough to score, BEFORE you scan or run it. A bad
export (a mono file, a silent channel, a swapped channel map, a hot capture) is
caught up front, so it never turns into a confident-looking but meaningless
turn-taking verdict downstream.

```bash
hotato trust --stereo your-call.wav
```

```text
hotato trust: your-call.wav
  recording: 32.0s, 44100 Hz, 2 channels
  caller (ch0): 3.32s speech, first at 3.27s, peak -2.1 dBFS
  agent  (ch1): 23.30s speech, first at 0.45s, peak -0.9 dBFS
  leading silence: 0.45s
  crosstalk: coherence 0.057 (low) at 0.39s lag
  scorability: separated tracks yes, caller activity yes, agent activity yes
  => eligible for scan
```

Exit code `0` means eligible for scan; exit code `2` means NOT SCORABLE (or a usage
error / unreadable file), so `trust` composes straight into a shell gate:

```bash
hotato trust --stereo call.wav && hotato scan --stereo call.wav
```

## What it checks

By convention the caller is on channel 0 and the agent on channel 1 (override
with `--caller-channel` / `--agent-channel`). `trust` reports INPUT health only:

- **per-channel activity**: how much speech each channel carries and when each
  first speaks;
- **possible channel swap**: a heuristic flag: if the channel mapped as the
  caller holds the floor far longer than the channel mapped as the agent (the
  reverse of the usual pattern, where an assistant answers in paragraphs), the
  caller/agent channels may be reversed;
- **sample rate and duration**: the basic recording facts;
- **clipping**: per-channel peak level (dBFS) and the fraction of samples at
  full scale, so a too-hot capture is visible;
- **leading silence**: dead air before the first speech on either channel;
- **crosstalk risk**: cross-channel echo coherence: is the caller channel
  carrying a delayed copy of the agent's own audio (echo bleed / missing echo
  cancellation)?
- **cross-channel leakage** (`crosstalk_risk.leakage_db`): the level, in dB below
  the source, of a consistent attenuated delayed COPY of one channel found on the
  other. The whole-clip coherence above is a single best-lag cosine over the
  entire envelope, so unrelated activity elsewhere in the call dilutes it: bleed
  loud enough to corrupt a downstream timing verdict can sit under the coherence
  bar and go unflagged. This number is measured differently -- from the per-frame
  level ratio of the copy, which a real leak holds constant across every frame the
  source speaks -- so it catches that regime. At or above `-40 dB` (calibrated to
  the level at which symmetric bleed was red-teamed into flipping a verdict) the
  copy is loud enough to be counted as the other party's activity, so it is
  flagged and the recommendation is downgraded off `eligible for scan`;
- **low signal level**: when even the loudest channel peaks below `-30 dBFS`, the
  capture is quiet enough that turn timing can be under-measured downstream; a
  warning, never a not-scorable condition;
- **scorability**: the three things a real score needs: separated tracks, enough
  caller activity, and enough agent activity;
- **recommendation**: `eligible for scan`; `scan with caution` (scorable, but a loud
  cross-channel leak may corrupt the timing a scan produces); or `NOT SCORABLE`
  with the specific reason AND the next step to fix it.

## Scope: one question, and it stops there

`trust` answers exactly one question -- is this audio good enough to score? --
and stops: no `yield` / `hold`, no `pass` / `fail`, no `did_yield`, no
talk-over number, no intent label, no turn-taking verdict.
A recording that is eligible for scan may still contain agent bugs; finding those is
what [`hotato scan`](../src/hotato/scan.py) and `hotato run` are for.

## Not scorable

Three input defects make a recording unscorable. Each is reported with `scorable:
false`, a plain reason, and the next step, and exits `2`:

- **mono**: a single channel cannot separate the caller from the agent on its
  own. Next step: export a dual-channel recording (the gold reference), OR score
  it via the opt-in diarization front-end (see "Mono via `--diarize`" below).
- **identical channels**: a mono recording duplicated into two channels: two
  channels, but not separated. Same fix as mono.
- **a silent required channel**: for example `caller channel has no detected
  speech`. Next step: verify channel mapping or export dual-channel again.

Clipping, high leading silence, crosstalk risk, cross-channel leakage, a very low
signal level, and a possible channel swap are **warnings**: surfaced as signal,
without changing scorability by themselves. A loud cross-channel leak
additionally downgrades the recommendation to `scan with caution` -- the tracks
stay separated and scorable, so the scorability gate and exit code hold
steady; only the human-facing recommendation records that a scan's timing may
be wrong. `trust` adds signals and discloses limits -- the not-scorable
boundary, and what passes, stay fixed.

## Mono via `--diarize`: is this mono file confidently separable?

By default a mono file is not scorable (above). With the opt-in `[diarize]`
front-end, `hotato trust --stereo call.wav --diarize` reports whether the mono is
confidently SEPARABLE into caller/agent -- still WITHOUT emitting any turn-taking
verdict. It runs the selected diarizer (`--diarizer pyannote|sortformer|pyannoteai`,
default local `pyannote`), then reports a `scorability.separation` sub-block and a
confidence **tier**:

- **high**: confidently separable -- score it with `hotato run --mono call.wav
  --diarize` for a real (diarized-mono) verdict. Exit `0`.
- **low**: separable but only indicative -- e.g. voices close, overlap elevated,
  or the caller/agent mapping balanced. `hotato run --mono ... --diarize` will
  still score it, but the verdict is stamped `indicative_only` and no SLA gate
  fires. Exit `0`.
- **refuse**: not confidently separable (not two clean parties, a near-silent
  speaker, extreme overlap, voices too similar). Not scorable, exit `2`; next
  step: record a dual-channel call.

The six signals behind the tier (speaker count, per-speaker activity, mean
segmentation posterior, embedding cluster-separation margin, overlap ratio,
segment churn) are in the `separation.signals` block. A missing extra / token /
model raises a clean error (exit 2) instead of a raw-mono guess. The
dual-channel path stays the gold reference; a diarized-mono verdict stays
distinct from it. See `docs/DIARIZE.md` for the full front-end.

## JSON for agents

`--format json` emits one machine-parseable report. Branch on `scorable` (and, on
a defect, read `not_scorable_reason` and `next_step`); `exit_code` mirrors the
process exit.

```bash
hotato trust --stereo call.wav --format json
```

```json
{
  "tool": "hotato",
  "kind": "input-health",
  "schema_version": "1",
  "source": "call.wav",
  "recording": {
    "sample_rate": 44100,
    "duration_sec": 32.0,
    "channels": 2,
    "clipping": {
      "caller": {"peak": 0.7906, "peak_dbfs": -2.1, "clipped_fraction": 0.0, "clipped": false},
      "agent": {"peak": 0.9016, "peak_dbfs": -0.9, "clipped_fraction": 0.0, "clipped": false}
    },
    "leading_silence_sec": 0.45
  },
  "channels": {
    "caller": {"channel": 0, "active_sec": 3.32, "first_speech_sec": 3.27, "has_speech": true, "enough_activity": true},
    "agent": {"channel": 1, "active_sec": 23.3, "first_speech_sec": 0.45, "has_speech": true, "enough_activity": true},
    "possible_swap": false,
    "swap_reason": null
  },
  "crosstalk_risk": {"coherence": 0.057, "lag_sec": 0.39, "suspected": false, "leakage_db": null, "leakage_direction": null},
  "scorability": {"separated_tracks": true, "enough_caller_activity": true, "enough_agent_activity": true},
  "warnings": [],
  "scorable": true,
  "recommendation": "eligible for scan",
  "not_scorable_reason": null,
  "next_step": null,
  "exit_code": 0
}
```

Everything runs offline and reuses hotato's existing primitives: the hardened
WAV reader, the reference framing, the energy VAD, and the cross-channel echo
coherence, all on your machine, reporting input-health signals only -- never
an accuracy percentage.


================================================================================
FILE: docs/CONNECT.md
================================================================================

# Connect, pull, sweep: score every call across your stack

`hotato sweep` is the "connect once, see every turn-taking problem across all
your calls" flow. It has three steps you can also run on their own:

1. **`hotato connect <stack>`**: store a stack's credentials once, locally.
2. **`hotato pull`**: bulk-fetch your recent recordings into a folder.
3. **`hotato sweep`**: pull, then run the zero-config `analyze` over them and
   write one offline dashboard of the ranked turn-taking moments.

Everything scores offline. The only network is the direct recording download
from your vendor to your machine; your audio and your keys stay between you
and the vendor.

## 1. Connect once

```
hotato connect vapi --api-key <key>
```

This does a lightweight live auth check (it lists one recent call), then writes
the credentials to `~/.hotato/connections.json` with file mode `0600` (directory
`0700`). The key stays out of any printed output and travels only to the
vendor's own API. Credentials also fall back to the stack's environment variable,
so this works too:

```
VAPI_API_KEY=<key> hotato connect vapi
```

Per-stack credentials:

| Stack | Flags (or env vars) |
| --- | --- |
| vapi | `--api-key` / `VAPI_API_KEY` |
| retell | `--api-key` / `RETELL_API_KEY` |
| twilio | `--account-sid` `--auth-token` / `TWILIO_ACCOUNT_SID` `TWILIO_AUTH_TOKEN` |
| bland | `--api-key` / `BLAND_API_KEY` |
| elevenlabs | `--api-key` / `ELEVENLABS_API_KEY` |
| synthflow | `--api-key` / `SYNTHFLOW_API_KEY` (+ `--model-id` / `SYNTHFLOW_MODEL_ID` to list) |
| millis | `--api-key` / `MILLIS_API_KEY` (+ `--base-url` for the EU region) |
| cartesia | `--api-key` / `CARTESIA_API_KEY` (+ `--agent-id` / `CARTESIA_AGENT_ID` to list) |

Retell has no list endpoint to verify against, so `connect retell` stores the
key and validates it on the first pull. Use `--no-verify` to skip the live check
for any stack. LiveKit and Pipecat are capture-in-your-infra: there's no
vendor recording to pull, so connect them via `hotato setup
--stack livekit|pipecat` instead.

After connecting, `--stack` and the credential flags are optional for `pull` and
`sweep`: when exactly one stack is connected, Hotato uses it.

## 2. Pull recent recordings

```
hotato pull --stack vapi --since 7d --limit 50
hotato pull                                   # the only connected stack
```

`pull` lists your recent recordings with the vendor's verified list endpoint and
downloads each one by looping the same single-call fetch `hotato capture` uses,
into `hotato-pull-<stack>/` (override with `--out DIR`). `--since` accepts `7d`,
`12h`, `30m`, `2w`. A recording that cannot be fetched (missing URL, HTTP error,
wrong channel count) is reported as a clean skip with its reason, and the pull
keeps going through the rest.

**Retell has no verified list endpoint.** Pull it from explicit ids instead
(Hotato ships only verified endpoints):

```
hotato pull --stack retell --call-id c1 --call-id c2
```

For Twilio, `--call-id` values are Recording SIDs (`RE...`).

**Mono / mixed stacks** (bland, elevenlabs, synthflow, millis, cartesia) produce
a single combined recording with no per-party channel; attributing talk-over
needs the `--allow-mono` opt-in, and results are indicative only:

```
hotato pull --stack bland --allow-mono --limit 20
```

## 3. Sweep = pull + analyze

```
hotato sweep --stack vapi --since 7d
hotato sweep                                  # the only connected stack
```

`sweep` pulls (as above) into `hotato-sweep-<stack>/` (override with `--dir`),
then runs the exact same zero-config `analyze` over the folder and writes one
self-contained, offline HTML dashboard (`hotato-sweep-<stack>.html`, override
with `--out`) of the ranked candidate turn-taking moments across every call,
with the hear-the-bug audio player on the top moments. `--format json` emits the
ranked candidates plus a pull summary instead.

Dual-channel stacks give separated scoring. Mono/mixed stacks can be swept with
`--allow-mono`; lacking per-party separation, their calls surface in the
dashboard's Skipped section.

Candidates are MEASURED timing moments you review and label with `hotato fixture
create`: a timestamp and a number, not a verdict on intent.

## What is and isn't pullable

See [`docs/ADAPTER-STATUS.md`](ADAPTER-STATUS.md) for the full map:
which stacks auto-pull dual-channel, which are mono-only behind `--allow-mono`,
which are capture-in-your-infra, and which are not integrable, each with the
exact verified endpoint and the gaps.


================================================================================
FILE: docs/ADAPTER-STATUS.md
================================================================================

# Adapter status

What each capture adapter is built against, verified verbatim against the
vendor's live documentation on the date shown (integration research:
`hotato-launch/INTEGRATION-SPEC-2026-07-07.md`). This file is the map of
what Hotato can pull from each stack, and why.

Terms:

- **auto-pull**: Hotato fetches the recording itself with your API key
  (`hotato capture` / `hotato pull` / `hotato sweep`).
- **capture-in-your-infra**: there is no vendor recording API; Hotato prints
  the recording config (`hotato setup`) and scores the file your deployment
  writes.
- **mono / mixed**: a single combined channel. Attributing overlap to the
  caller or the agent needs separated tracks, so Hotato scores a combined
  channel only behind an explicit `--allow-mono` / `HOTATO_ALLOW_MONO=1`
  opt-in and labels the result indicative only. Separated turn-taking
  analysis runs on a dual-channel file.

Honesty rule (enforced): an adapter ships only with endpoints verified verbatim
in the spec. Where the spec marks a list-calls endpoint or a channel layout
**unconfirmed / none**, Hotato supports the fallback (explicit ids) and
documents the gap here.

## Build now: dual-channel (auto-pull, separated scoring)

| Stack | List recent calls | Fetch recording | Channel basis | Last verified |
| --- | --- | --- | --- | --- |
| Vapi | `GET https://api.vapi.ai/call` (params `limit`, `createdAtGt/Lt`) → JSON array of Call objects (`id`, `createdAt`) | `GET /call/{id}` → `artifact.recording.stereoUrl` (current); deprecated fallbacks `artifact.stereoRecordingUrl`, `call.stereoRecordingUrl` | `stereoUrl` is a distinct 2-channel file (customer ch0, assistant ch1) | 2026-07-07 |
| Twilio | `GET .../Accounts/{Sid}/Recordings.json` (params `PageSize`, `DateCreatedAfter/Before`, `callSid`) → `recordings[].sid` | `GET .../Recordings/{RE...}.wav?RequestedChannels=2` (HTTP Basic) | dual-channel only if the recording was created `RecordingChannels=dual`; 400 → clean stop or `--allow-mono` → `RequestedChannels=1`. Two-party order: ch0 = caller, ch1 = agent; conference: ch0 = first participant | 2026-07-07 |
| Retell | **none confirmed**: the spec marks list-calls unconfirmed; do not fabricate one. Pull from explicit `--call-id` | `GET https://api.retellai.com/v2/get-call/{id}` (Bearer) → `scrubbed_recording_multi_channel_url` preferred, then `recording_multi_channel_url`; plain mono `recording_url` rejected unless `--allow-mono` | per-party channels on the `*_multi_channel_url` fields | 2026-07-07 |
| LiveKit | **capture-in-your-infra**: `ListEgress` is an RPC method name only; no REST list. `hotato setup --stack livekit` | Two audio-only Track egresses (one per party); recording location in `egress_info.file_results[].location` | separated by running one Track egress per participant | 2026-07-07 |
| Pipecat | **capture-in-your-infra**: Pipecat Cloud session-list has no recording field; OSS has no list at all | `AudioBufferProcessor(num_channels=2)` in-pipeline (user left, bot right); you write the WAV | 2-channel in-pipeline | 2026-07-07 |

## Build now: mono / mixed only (auto-pull, `--allow-mono`, indicative only)

Each of these has a verified list + fetch, but the vendor produces a single
combined recording with **no documented per-party channel**, so scoring is
degraded and gated behind `--allow-mono`.

| Stack | List recent calls | Fetch recording | Mono basis (verbatim) | Last verified |
| --- | --- | --- | --- | --- |
| Bland AI | `GET https://api.bland.ai/v1/calls` → `calls[].call_id` | `GET /v1/calls/{id}` → `recording_url` | no stereo/channel field in the recording or call-details schema | 2026-07-07 |
| ElevenLabs Conversational AI | `GET https://api.elevenlabs.io/v1/convai/conversations` (params `page_size`, `call_start_after_unix`) → `conversations[].conversation_id` | `GET /v1/convai/conversations/{id}/audio` → combined audio | docs state the audio "does not include separate caller/agent channels, only a combined full conversation MP3" | 2026-07-07 |
| Synthflow | `GET https://api.synthflow.ai/v2/calls?model_id=…` (params `limit`, `from_date` epoch-ms) → `response.response.calls[].call_id` | `GET /v2/calls/{id}` → `response.response.calls[0].recording_url` (a Twilio Recordings URL) | Synthflow documents no dual-channel option; `recording_url` is a bare Twilio URL | 2026-07-07 |
| Millis AI | `GET {base}/call-logs` (US `api-west`, EU `api-eu-west`; param `limit`) → `histories[].session_id` | `GET /call-logs/{session_id}` → `recording.recording_url` | schema exposes only `recording_url`; `CallSettings` has only a boolean `enable_recording` | 2026-07-07 |
| Cartesia (Line) | `GET https://api.cartesia.ai/agents/calls?agent_id=…` (param `limit`; needs `Cartesia-Version` header) → `data[].id` | `GET /agents/calls/{id}/audio` → `audio/wav` | dual-vs-mono **unconfirmed** in the docs; treated as mono until a live channel-count check proves otherwise | 2026-07-07 |

`--stack synthflow` needs `--model-id` (its list endpoint requires `model_id`);
`--stack cartesia` needs `--agent-id`; `--stack millis` accepts `--base-url` for
the EU region. Single-call `capture`/`pull` by explicit id needs none of these.

## Not integrable (no vendor recording to pull)

| Stack | Why | Basis |
| --- | --- | --- |
| Deepgram Voice Agent API | Real-time WebSocket (`wss://agent.deepgram.com/v1/agent/converse`) with no REST list-calls, no fetch-recording endpoint, and no recording-ready webhook. Deepgram does not store the call. If you want a recording, capture it in your own infra (LiveKit/Pipecat pattern) | 2026-07-07, confirmed absent |
| PlayAI (formerly Play.ht) | The conversational-agent product is dead: `play.ai` DNS does not resolve and `docs.play.ai` returns `DEPLOYMENT_NOT_FOUND`. The only live domain, `docs.play.ht`, is TTS-only with no calls/recordings API | 2026-07-07, dead endpoints verified |

## Unconfirmed: needs credentials or a live probe before shipping

Documented weakly or behind a login/SPA; the spec could not verify a recording
field-path or a channel layout. Not shipped as adapters. Listed here so
nobody assumes support that was never confirmed.

- **Regal.ai**: no list-calls and no REST fetch-recording endpoint. The
  recording arrives only via the `call.recording.available` webhook's
  `properties.recording_link`, which is a literal Twilio Recordings URL. Feed
  that URL through Hotato's existing Twilio path -- the one shipped route for
  Regal-sourced recordings.
- **Thoughtly**: OpenAPI types responses as an untyped `GenericResponse.data`
  blob; no recording-URL field path could be confirmed live.
- **Sindarin**: no Sindarin-hosted recording endpoint found; `record` is
  submitted with the caller's own Twilio creds, implying the recording lands in
  the customer's own Twilio account (use the Twilio path).
- **xAI Grok Voice Agent Builder**: OpenAI-Realtime-compatible; no documented
  list-calls or fetch-recording endpoint (marketing mentions in-product
  recording, but no API contract was confirmable).
- **Cartesia dual-channel**: the `/agents/calls/{id}/audio` WAV's channel count
  was not confirmable in docs; treated as mono here pending a live check.
- **Twilio dual-channel edge behaviour**: the exact 400-on-unavailable and
  conference channel-ordering guarantees were carried forward from prior notes,
  not re-verified verbatim this pass.
- **Synthflow / Regal Twilio-URL dual-channel trick**: because their
  `recording_url` is a Twilio URL, `?RequestedChannels=2` *might* yield stereo,
  but neither vendor documents or guarantees it; unconfirmed until tested against
  a live account.

Other enterprise/partial platforms researched but not shipped (login-gated,
SPA-only docs, or transcript-only APIs): Daily, Ultravox, Hume EVI, Telnyx,
Infobip, OpenAI Realtime, Amazon Connect, Parloa, Sierra, Decagon, PolyAI,
Voiceflow, Cognigy, Dialogflow CX, Genesys Cloud, NICE CXone. See the
integration spec for the per-platform verified facts and gaps.

## Invariants

Every adapter validates that a dual-channel file it scores has one party per
channel (2 channels) before producing separated talk-over numbers; mono stacks
are scored degraded only behind `--allow-mono`. All live fetch/list paths are
stdlib-only (`urllib`); stack SDKs import lazily and only inside your own infra.
Credentials captured by `hotato connect` are stored locally at
`~/.hotato/connections.json` (mode 0600) and go straight to the vendor's own
API -- Hotato stays out of that exchange. Scoring is always offline; the only
network is the direct recording download.


================================================================================
FILE: docs/STARTER.md
================================================================================

# The starter kit: `hotato init starter`

The fastest way to add hotato to an existing voice-agent repository. It
scaffolds the CI gate, a stack-tuned config file, and the three directories
the rest of the docs assume already exist, so you can go straight to turning
your first bad call into a contract instead of wiring plumbing by hand.

```bash
hotato init starter --stack vapi --out .
```

`--stack` is one of `vapi`, `retell`, `twilio`, `livekit`, `pipecat` -- every
stack hotato has a shipped connector for today (see
[`ADAPTER-STATUS.md`](ADAPTER-STATUS.md)). `--out` is usually `.`, the root of
the repo you are adding hotato to. Generation runs offline, in one command,
nothing to connect.

## What it writes

```
HOTATO.md                                # what was added, next steps (read this first)
hotato.yaml                              # config skeleton, tuned for --stack
.gitignore                               # excludes local/pulled recordings;
                                          #   keeps pinned fixture/contract clips committed
.github/workflows/hotato-contracts.yml   # the CI gate
fixtures/
  README.md
  scenarios/.gitkeep                     # -> hotato fixture create --out fixtures
  audio/.gitkeep
contracts/
  README.md
  .gitkeep                               # -> hotato contract create --out contracts
reports/
  README.md
  .gitkeep                               # local/CI scratch: doctor/report/sweep output
```

Every file writes cleanly only when it doesn't already exist (pass `--force`
to overwrite); each write lands whole or not at all. The generated file names
are deliberately namespaced away from an existing repo's own files (`HOTATO.md`,
distinct from `README.md`; `hotato-contracts.yml`, distinct from
`hotato.yml`) so a first run drops in cleanly alongside files a voice-agent
repo almost always already has.

## Two input paths, chosen for you by `--stack`

**Auto-pull** (`vapi`, `retell`, `twilio`): hotato fetches the recording
itself once you connect a key. `hotato.yaml`'s `credentials.env` names the
exact environment variable(s) (`VAPI_API_KEY`; `RETELL_API_KEY`;
`TWILIO_ACCOUNT_SID` + `TWILIO_AUTH_TOKEN`) `hotato connect <stack>` also
reads. `recording.access` is `auto-pull`.

**Capture-in-your-infra** (`livekit`, `pipecat`): capture happens inside your
own deployment, so no credentials are needed.
`hotato.yaml`'s `credentials.env` is `[]` and `recording.access` is
`capture-in-your-infra`; `hotato setup --stack <stack>` prints the exact
two-track capture scaffold, and you point `hotato contract create --stereo`
at the WAV your own deployment writes.

## LiveKit and Pipecat runbook

LiveKit and Pipecat are the two stacks where capture and the turn-taking
config both live in your own code, ahead of any vendor API. This is the
operator runbook for both, capture through CI.

### LiveKit

1. **Capture.** Two audio-only Track egresses, one per participant --
   RoomComposite mixes both parties into one channel and cannot attribute
   overlap. `hotato setup --stack livekit` prints the copy-paste scaffold
   (Python `livekit-api`, `TrackEgressRequest` + `DirectFileOutput`); a
   ready-to-copy version also lives at `adapters/livekit_capture.py`.
2. **Find the turn-taking config.** It lives on
   `AgentSession(turn_handling=TurnHandlingOptions(...))`: `turn_detection`
   (`inference.TurnDetector()` / `"realtime_llm"` / `"vad"` / `"stt"` /
   `"manual"`), `endpointing` (`min_delay`, `max_delay`), and `interruption`
   (`enabled`, `mode`, `min_duration`, `min_words`,
   `false_interruption_timeout`, `resume_false_interruption`). Read what a
   given agent file is running, statically, before you propose
   changing anything: `hotato inspect --stack livekit --config agent.py`.
3. **Score it.**
   `hotato capture --stack livekit --caller caller.wav --agent agent.wav --onset <sec> --expect yield`
   (convert the egress output to WAV first, e.g. `ffmpeg -i caller.ogg caller.wav`).
4. **Fixture, contract, CI.** Same as every stack from here -- see "Turn
   your first bad call into a contract" below.

### Pipecat

1. **Capture.** A 2-channel `AudioBufferProcessor` in-pipeline (channel 0 =
   user/caller, channel 1 = bot/agent) -- do not mix down to one channel.
   `hotato setup --stack pipecat` prints the copy-paste scaffold; a
   ready-to-copy version also lives at `adapters/pipecat_capture.py`.
2. **Find the turn-taking config.** It lives on `PipelineTask`'s user-turn
   strategies: start strategies (`VADUserTurnStartStrategy`,
   `TranscriptionUserTurnStartStrategy`,
   `MinWordsUserTurnStartStrategy(min_words=...)`,
   `KrispVivaIPUserTurnStartStrategy(...)`) and stop strategies
   (`SpeechTimeoutUserTurnStopStrategy(user_speech_timeout=...)`,
   `TurnAnalyzerUserTurnStopStrategy(turn_analyzer=...)`); note
   `MinWordsInterruptionStrategy` is deprecated since pipecat 0.0.99 in
   favor of `MinWordsUserTurnStartStrategy`. Read what a given bot file is
   running, statically: `hotato inspect --stack pipecat --config bot.py`.
3. **Score it.**
   `hotato capture --stack pipecat --stereo captured.wav --expect yield`
   (write the WAV from the `AudioBufferProcessor`'s `on_audio_data` handler
   first).
4. **Fixture, contract, CI.** Same as every stack from here -- see "Turn
   your first bad call into a contract" below.

Both APIs move; `hotato setup` and `hotato inspect` state the verified-against
date. Full field-level detail and provenance: [`ADAPTER-STATUS.md`](ADAPTER-STATUS.md)
(capture) and [`FIX-PLANS.md`](FIX-PLANS.md) (inspect, Level 1 of the fix ladder).

## The CI gate

`.github/workflows/hotato-contracts.yml` runs on push, on pull request, and
weekly. It is two guarded steps that pass clean, as a **no-op**, until you
have added a first contract or fixture (a fresh scaffold's normal starting
state):

```bash
hotato contract verify contracts --junit hotato.xml --format json > contracts-verify.json
hotato run --scenarios fixtures/scenarios --audio fixtures/audio --format json > fixtures-run.json
```

The JUnit file is published as a build artifact on every run (`always()`),
whether the gate passed, failed, or had nothing to check yet.

For the three auto-pull stacks, the workflow also carries a `weekly-sweep`
job: a passive, candidate-only sweep of recent calls
(`hotato sweep --stack <stack>`), ranked by hotato's own salience -- a
candidate list for you to review and label. It ships **disabled** (`if:
false`): flip it to `true` once the stack's credential env var(s) are set as
repo secrets (Settings -> Secrets and variables -> Actions). A live pull
against your account only runs on your explicit say-so: enabling this job is
a human decision, made once, in your own CI config. `livekit`/`pipecat` skip
this job entirely, since capture for those stacks already happens inside
your own deployment.

## Turn your first bad call into a contract

```bash
# auto-pull stacks
hotato connect vapi --api-key <key>
hotato sweep --stack vapi --out hotato-sweep.html
# open hotato-sweep.html, pick a candidate moment, then:
hotato contract create --from-candidate hotato-sweep.json#1 \
    --expect yield --id refund-cutoff-001 --out contracts

# capture-in-your-infra stacks
hotato setup --stack livekit
# once your deployment writes a two-channel WAV:
hotato contract create --stereo call.wav --onset 42.18 \
    --expect yield --id refund-cutoff-001 --out contracts
```

Commit the resulting `contracts/refund-cutoff-001.hotato/` directory. The
next push runs it through the CI gate above.

**A contract bundle contains call audio** (`audio/event.wav`). If this repo
is or could become public, commit a sanitized fixture (synthetic or
consent-cleared), and keep customer contracts in a private repository
or controlled artifact storage. See [`CONTRACTS.md`](CONTRACTS.md).

## Read more

- The bundle layout and the create/verify/inspect/pack/unpack commands:
  [`CONTRACTS.md`](CONTRACTS.md)
- The underlying fixture primitive, one bad call to a CI gate in five steps:
  [`BAD-CALL-TO-CI.md`](BAD-CALL-TO-CI.md)
- Per-stack connector support, verified against the vendor's live docs:
  [`ADAPTER-STATUS.md`](ADAPTER-STATUS.md)
- The connect-once bulk pull-and-analyze recipe: [`CONNECT.md`](CONNECT.md)
- An agent adding hotato to a repo end to end: [`../AGENTS.md`](../AGENTS.md)


================================================================================
FILE: docs/TRACE.md
================================================================================

# Voice traces

A voice trace is a timeline of discrete voice-pipeline events (caller/agent
audio activity, TTS cancel/stop, ASR partials, tool calls, ...) that
supplements a failure contract's frame-level timing evidence with a WHY
layer beyond what a caller/agent audio track alone can show: "the agent
talked over the caller" becomes "evidence suggests TTS cancellation lagged:
cancel requested at 42.40s, audio stopped at 43.60s".

A voice trace reports coincidence: a pattern of events that lined up with the
measured timing, not a proof of root cause. See [`docs/OTEL.md`](OTEL.md) for
the ingest source formats.

Three commands:

```bash
hotato trace ingest --otel traces.jsonl --out voice_trace.jsonl
hotato trace attach contracts/refund-cutoff-001.hotato --trace voice_trace.jsonl
hotato trace export contracts/refund-cutoff-001.hotato --format otel --out otel.jsonl
```

## Schema

`voice_trace.jsonl` is JSONL (one meta line, then one line per span -- the
same convention `evidence/frames.jsonl` uses), validating against
`hotato.voice_trace.v1` (`src/hotato/schema/voice_trace.v1.json`) once
reassembled by `hotato.trace.load_voice_trace_jsonl`:

```json
{
  "schema": "hotato.voice_trace.v1",
  "call_id": null,
  "deployment": {"stack": "vapi", "agent_id": null, "git_sha": "deadbeef", "config_hash": "sha256:..."},
  "spans": [
    {"type": "agent_audio_active", "start_sec": 0.0, "end_sec": 2.9},
    {"type": "tts_cancel_requested", "time_sec": 2.6},
    {"type": "tts_audio_stopped", "time_sec": 2.9},
    {"type": "asr_partial", "start_sec": 2.4, "end_sec": 2.95, "text_redacted": true},
    {"type": "tool_call", "start_sec": 1.1, "end_sec": 1.42, "name": "lookup_order", "latency_ms": 320}
  ],
  "source": {"format": "otel-jsonl-bridge", "input_span_count": 6}
}
```

A span carries an open `type` string; the common ones are
`caller_audio_active`, `agent_audio_active`, `tts_cancel_requested`,
`tts_audio_stopped`, `asr_partial`, `tool_call`, `llm_first_token`,
`handoff`. An unrecognized type passes through unchanged (additive,
forward-compatible with a pipeline that emits a span type this release does
not name). Exactly one time shape applies per span: an
interval span carries `start_sec`/`end_sec`, a point event carries
`time_sec`.

## Redaction by default

`call_id` and `deployment.agent_id` are dropped (`null`) unless
`--include-identifiers` is passed at ingest time. An `asr_partial` span's
transcript text is dropped (`text_redacted: true`, no `text` key) unless
`--include-text` is passed. `deployment.stack` / `git_sha` / `config_hash`
are not treated as identifiers and are kept by default.

## Ingest

```bash
hotato trace ingest --otel traces.jsonl --out voice_trace.jsonl
hotato trace ingest --otel export.json --out voice_trace.jsonl \
    --stack vapi --include-identifiers --include-text
```

`--otel FILE` accepts either a standard OTel JSON export (a document with a
top-level `resourceSpans` array) or hotato's own documented OTel bridge
JSONL (see [`docs/OTEL.md`](OTEL.md)). `--stack` / `--call-id` /
`--agent-id` / `--git-sha` / `--config-hash` override or fill in whatever
the source's own resource attributes carried. Refused (exit 2, nothing
written) for an unreadable file, an empty file, or a source with zero
spans.

## Attach

```bash
hotato trace attach contracts/refund-cutoff-001.hotato --trace voice_trace.jsonl
```

Copies the trace into `<bundle>/traces/voice_trace.jsonl` and re-renders
`evidence/timeline.html` with the trace's events drawn as an additional row,
aligned to the SAME [0, duration] scale as the existing caller/agent
timeline. This reads the bundle's OWN `evidence/frames.jsonl` and
`contract.json` back in, reusing the existing scored evidence as-is:
attaching a trace needs no diarization extra installed and no re-scoring of
the audio. On a diarized-mono bundle (no frame-level evidence), the base
timeline states that plainly, and the trace row still renders on its own
scale.

`contract.json` records the attachment (additive, schema-safe: `trace:
{attached, path, span_count, attached_at, source_format}`). Refused (exit 2)
for a missing bundle, a trace file that does not validate as a
`hotato.voice_trace.v1` JSONL, or an already-attached trace without
`--force`.

### Report wording

When a `tts_cancel_requested` / `tts_audio_stopped` pair is present, the
timeline states the measured delta plainly:

> Evidence suggests TTS cancellation delay: cancel requested at 2.60s, audio
> stopped at 2.90s (delta 0.30s).
> Hotato does not prove root cause.
> Unknowns: no client-side playout trace was attached.

The last line is always present in this release: a client-side audio
playout trace (the point where the CALLER'S device stopped
rendering audio, as opposed to when the server issued the stop) sits outside
what this release's span types collect, so the gap is always named plainly.

## Export

```bash
hotato trace export contracts/refund-cutoff-001.hotato --format otel --out otel.jsonl
```

Writes the bundle's attached trace back out as hotato's OTel bridge JSONL --
the exact shape `trace ingest` reads, so `ingest -> attach -> export ->
ingest` round-trips the identical spans. `--format` is claimed by the export
format on this subcommand (only `otel` is supported today); pass `--json`
for the machine result summary instead of the default text line. Refused
(exit 2) when the bundle has no attached trace, or `--out` exists without
`--force`.

## What a voice trace adds

A voice trace adds timing correlation to a failure contract's existing
timing measurement: a pattern of events lined up with it, named plainly.
Authorization, identity, compliance, policy safety, intent, and root cause
stay outside its claims, and a client-side playout moment is named only when
one was attached.

## Read more

- Source formats and the OTel bridge JSONL shape:
  [`docs/OTEL.md`](OTEL.md)
- Failure contracts a trace attaches to: [`docs/CONTRACTS.md`](CONTRACTS.md)


================================================================================
FILE: docs/OTEL.md
================================================================================

# OTel ingest: two source shapes

`hotato trace ingest --otel FILE` recognizes two input shapes. Both convert
into the same `hotato.voice_trace.v1` spans; which one it used is recorded
in `source.format` (`"otel-json"` or `"otel-jsonl-bridge"`).

This is a bridge: it reads a file you already have (an exported trace, or a
small script's own event log), offline, once, and translates `name`,
`startTimeUnixNano`/`endTimeUnixNano`, `attributes`, and span `events` from a
standard export into hotato's own span shape.

## 1. Standard OTel JSON export (`otel-json`)

A single JSON document with a top-level `resourceSpans` array (the shape a
standard OTel exporter/collector writes):

```json
{
  "resourceSpans": [{
    "resource": {"attributes": [
      {"key": "service.name", "value": {"stringValue": "vapi"}},
      {"key": "git_sha", "value": {"stringValue": "cafebabe"}}
    ]},
    "scopeSpans": [{"spans": [
      {
        "name": "agent_audio_active",
        "startTimeUnixNano": "1000000000",
        "endTimeUnixNano": "4400000000"
      },
      {
        "name": "tts.cancel_requested",
        "startTimeUnixNano": "2600000000",
        "events": []
      }
    ]}]
  }]
}
```

- Resource attributes are flattened into a plain dict; `service.name`
  (or a custom `stack` attribute) becomes `deployment.stack`, `git_sha` /
  `config_hash` attributes (if present) fill the matching deployment
  fields. Only the FIRST `resourceSpans` entry's resource is used for
  deployment metadata; every entry's spans are still walked.
- Timestamps convert to seconds relative to the EARLIEST timestamp anywhere
  in the file, matching the audio-relative-seconds convention every other
  hotato timestamp uses.
- A span's own `name` maps to a hotato span `type` via a small documented
  table (`caller_audio_active`, `agent_audio_active`,
  `tts.cancel_requested` -> `tts_cancel_requested`,
  `tts.audio_stopped` -> `tts_audio_stopped`, `asr.partial` -> `asr_partial`,
  `llm.first_token` -> `llm_first_token`); an unmapped name passes through
  unchanged.
- Span `events` (OTel's own point-in-time markers nested inside a span --
  the natural place a production pipeline would put a `tts.cancel_requested`
  marker inside a broader `tts_playback` span) are flattened into their own
  point events the same way top-level spans are.
- A `tool_call`-mapped span's `tool.name` / `gen_ai.tool.name` attribute
  becomes the span's `name` field; a `latency_ms` attribute is used if
  present, otherwise computed from the span's own start/end.
- An `asr_partial`-mapped span's `text` / `asr.transcript.partial` attribute
  is captured but immediately redacted (dropped, `text_redacted: true`)
  unless `--include-text` was passed.

## 2. Hotato's OTel bridge JSONL (`otel-jsonl-bridge`)

The simpler, documented shape for a script or test fixture that does not
have a full OTel exporter: one JSON object per line (or one bare JSON array
of the same objects). Each line is either a span or a meta/resource line.

A span line uses `type` directly; `name` on a span line is reserved for
`tool_call`'s own tool name, distinct from the span kind:

```
{"type": "caller_audio_active", "start_sec": 2.40, "end_sec": 4.10}
{"type": "agent_audio_active", "start_sec": 0.00, "end_sec": 2.90}
{"type": "tts_cancel_requested", "time_sec": 2.60}
{"type": "tts_audio_stopped", "time_sec": 2.90}
{"type": "asr_partial", "start_sec": 2.40, "end_sec": 2.95, "text": "wait, I need a refund"}
{"type": "tool_call", "start_sec": 1.10, "end_sec": 1.42, "name": "lookup_order", "latency_ms": 320}
```

An optional meta/resource line (no `type` key) attaches deployment
metadata and a call id:

```
{"call_id": "demo-call-001", "deployment": {"stack": "vapi", "agent_id": "agent-demo-1", "git_sha": "deadbeefcafe", "config_hash": "sha256:0f1e2d3c"}}
```

`--call-id` / `--stack` / `--agent-id` / `--git-sha` / `--config-hash` on
`trace ingest` override whatever a meta line (or a standard export's
resource) provided. `hotato trace export` writes exactly this bridge
shape, so `ingest -> attach -> export -> ingest` round-trips the same
spans; the shipped test fixture (`tests/data/otel/demo-trace.otel.jsonl`)
uses this shape.

## Which one to use

If you already run an OTel collector or SDK and can dump a trace export,
point `--otel` at that file directly -- the standard-export path reads it,
best-effort. If you are wiring a quick script (a webhook handler, a log-line
scraper) outside a full OTel pipeline, write the bridge JSONL directly --
the same information, already flat.


================================================================================
FILE: docs/REPORTS.md
================================================================================

# Reports: doctor, report, team, export

Four surfaces over the same scorer: reproducible timing measurements read
straight from the envelope, with the method exposed at every layer.

## `hotato doctor`: the 5-minute path

One command. If you pass a recording it scores that; otherwise it runs the
bundled self-test battery. Either way it writes the self-contained HTML report
and tries to open it in your browser (on a headless box it prints the path).

```bash
uvx hotato doctor --stereo call.wav     # score your call, open the report
uvx hotato doctor                       # self-test fallback, same flow
uvx hotato doctor --demo --no-open --out report.html
```

It wraps the existing scorer and report, and everything runs offline. Exit
codes match `run`: `0` all pass, `1` a regression (`--no-fail` forces `0`),
`2` usage or IO error, or a recording that is not scorable. Not scorable
means the recording lacks a moment to measure (the caller channel is silent,
or the agent was not talking when the caller started), reported plainly
instead of a verdict.

**Your own recording gets its audio embedded in the report by default.**
Scoring a call with `--stereo` / `--caller`+`--agent` writes the exact scored
audio into the HTML file as a base64 data URI, so hearing the moment next to
its timeline works offline. The bundled self-test fallback stays unembedded.
If you plan to share, mail, or post the resulting `report.html`, treat it the
same as sharing the raw recording -- because it contains the raw recording.

## `hotato report`: the visual report

One self-contained file: inline CSS, inline SVG, zero external requests. It
opens offline by double-click and survives being mailed around.

```bash
uvx hotato report --stereo call.wav --out report.html
uvx hotato report --suite barge-in --out selftest.html
uvx hotato report --stereo call.wav --format md --out report.md
uvx hotato report --stereo call.wav --embed-audio --out report.html  # opt-in: embed the audio too
```

`--embed-audio` embeds the exact scored audio (base64, under a size cap) so
the report is a fully self-contained, hearable artifact -- and, for the same
reason, a shareable-HTML caution applies: a report built with `--embed-audio`
(or `hotato doctor` on your own recording, which sets it by default) carries
the call audio inside the HTML file. Give it the same care as the raw
recording before posting it somewhere public or attaching it to a public
issue/PR.

Per event it draws a to-scale caller/agent activity timeline from the
frame data: the overlap shaded, the caller-onset and yield markers, the
measured talk-over seconds, expected vs measured, and a PASS or FAIL chip.

After the per-event cards, once the page has at least three of them, sits an
analytics rollup computed from the same measurements (a page with fewer
events skips it):

- a **time-to-yield distribution** strip, one dot per measured yield, with
  mean, median, and p90 (definitions in `METHODOLOGY.md`);
- a **talk-over histogram**, per-event seconds bucketed on a fixed grid;
- **failure clustering by fix class**, so a batch of failures reads as "these
  five share one config setting" instead of five separate mysteries.

Every timeline carries a collapsible **frame inspector**: the full frame dump
behind that event as a table (`t_sec`, per-channel dBFS, active flags,
thresholds), so any pixel on the page can be re-derived by hand. The exact
`ScoreConfig` thresholds used sit in one collapsed "Thresholds used" panel at
the end of the page, so the run is reproducible without stamping the
parameter table above every render.

### Voice-trace context with `--trace`

Pass `--trace voice_trace.jsonl` (a `hotato.voice_trace.v1` file, written by
`hotato trace ingest`) to attach a voice trace as **context** next to the
timing:

```bash
hotato trace ingest --otel spans.otel.jsonl --out voice_trace.jsonl
hotato report --stereo call.wav --trace voice_trace.jsonl --out report.html
```

The report grows one collapsed, clearly-labelled "Trace (context, not a
score)" section: the trace's discrete voice-pipeline events -- TTS
cancel/stop, ASR partials, tool calls -- as a mono span table (type, name,
start, end, detail). Exactly like `--base` and an attached `assert.v1`
envelope, the report renders the already-produced trace artifact as data,
context alongside the score. The section stays scoped to context: `did_yield`,
`talk_over_sec`, `seconds_to_yield`, and the PASS/FAIL verdict come from the
scorer alone, and the trace is folded into the machine envelope as an
additive top-level `trace_context` key. A report built without `--trace` is
byte-identical to one built before the flag existed.

Redaction is respected: a span carrying `text_redacted: true` (e.g. an
`asr_partial` ingested without `--include-text`) shows a `[redacted]`
placeholder in place of its text, so a report shared outside the fleet
carries only what the trace itself already chose to keep. The same `trace=`
parameter
is on `build_report_html` / `build_report_md`; see `docs/TRACE.md` for the
trace format and `hotato trace ingest`.

### Reliability in the scorecard (pass@1 / pass@k / pass^k)

When a report carries an `assert.v1` envelope whose results are dimension-tagged,
the "Deterministic" shelf renders as a per-dimension **scorecard** (outcome /
policy / conversation / speech / reliability). The **Reliability** dimension is
pass^k's home, and it renders the repetition data you thread in:

```python
from hotato import report, simulate

summary = simulate.run_matrix(scenario, conversation_test=ct)   # a matrix aggregate
html, _ = report.build_report_html(stereo="call.wav",
                                   assertions=env, reliability=summary)
```

`reliability=` accepts a `simulate.run_matrix` summary, a bare
`simulate.reliability()` dict, or a `{"aggregate": <reliability dict>, "origin":
...}` wrapper. The dimension then shows each number **labeled**, tabular mono:

- **pass@1** (single-run pass rate), **pass@k** (>=1 of k passed), and **pass^k**
  (ALL k passed), with `n`, `k`, `passes`, and a **Wilson 95% CI** on pass@1;
- a **per-variation-cell** breakdown (each cell its own pass^k) when the summary
  carries one;
- a **SIMULATOR_INVALID** bucket -- broken fixtures, shown separately and
  **excluded from n**, kept distinct from an agent PASS/FAIL.

pass^k is its OWN number, kept on its own lane -- there's no `overall_score`
field for it to blend into. When the runs were simulated the section is
labeled **origin=simulated**: a simulator's replay reliability, scoped apart
from production reliability.

`hotato test run --repetitions N` (with `N > 1`) computes this aggregate over the
N deterministic runs and threads it into `report.{html,md}` automatically. With
no repetition data (`reliability=None`, or `--repetitions 1`) the dimension shows
the empty-state -- "not measured: no repeated runs in this report" -- and
the report is byte-identical to one built without the parameter.

### Regression deltas with `--base`

Save an envelope, then compare a later run against it:

```bash
hotato run --suite barge-in --format json > base.json
hotato report --suite barge-in --base base.json --out report.html
```

The report renders per-scenario talk-over and time-to-yield deltas with clear
worse and better marks. The same `--base` flag works on
`scripts/pr_comment.py` for the CI comment (`docs/CI.md`).

### PDF

The page ships print CSS. Print it from any browser and the interactive parts
collapse into a clean paper layout, so print-to-PDF is the PDF export.

## `hotato team`: the trend view

Aggregates a directory of run envelopes into one trend.

```bash
hotato run --suite barge-in --format json > runs/001.json
# ... more runs over days or branches ...
hotato team runs/ --html team.html --out agg.json
```

It reports: number of runs, mean/median/p90 talk-over and time-to-yield pooled
across all events, mean/median/p90/p95 response gap (dead air before the agent
speaks) pooled the same way, pass rate per run over time, the most common
failure class, and a pass-rate trend line in the HTML page. `--order mtime`
(default) orders runs by file time; `--order name` uses the filename, so a
numeric prefix is an explicit index.

`--max-response-gap SECONDS` turns the pooled p95 response gap into a latency
SLA: the run exits `1` exactly when p95 exceeds the bound, the same
pass/fail contract as a talk-over or time-to-yield regression (`--no-fail`
always exits `0`). Percentile definitions: `METHODOLOGY.md`; the pooling
shape is `dist_summary` in `src/hotato/_stats.py`.

Fewer than two runs is stated plainly and exits `0`; a trend line renders
once there are enough points for it to mean something.

## `hotato export`: research-grade CSVs

Scores a recording (or the bundled battery) exactly like `hotato run` and
writes three files into a directory:

```bash
uvx hotato export --stereo call.wav --out research/
uvx hotato export --suite barge-in --out research/
```

- `events.csv`: one row per scored event, every measured signal plus verdict.
- `frames.csv`: one row per VAD frame, the evidence behind every number.
- `envelope.json`: the standard machine envelope, unchanged.

Column meanings are documented in comment lines at the top of each CSV, so the
files are self-describing when they land in a notebook or a stats package
months later. An empty cell means "not derivable", distinct from a zero.
Stdlib only, offline.

`export` also prints mean/median/p90/p95 response gap pooled across the
exported events, and accepts the same `--max-response-gap SECONDS` latency
SLA gate as `team` (exit `1` when the pooled p95 exceeds the bound). A plain
export with no `--max-response-gap` writes a byte-identical `envelope.json`;
the pooled numbers and the gate live only in the printed summary and the
returned manifest, never in the CSVs or the envelope file.


================================================================================
FILE: docs/ASSERTIONS.md
================================================================================

# Assertions (`assert.v1`): deterministic, typed, each scored on its own lane

`hotato.assert_` evaluates a small, fixed set of assertions against a call's
transcript, ingested trace, and timing -- each kind scored on its own lane, by
construction. Every kind here is regex, checksum, or span/dict lookup. None of
them is a model call.

```python
from hotato import assert_ as A

ctx = A.build_context(
    transcript_path="transcript.json",   # or transcript=[...] turns already in hand
    trace_path="voice_trace.jsonl",      # or spans=[...] already in hand
    timing=envelope,                     # optional: a run's envelope.v1 events
)
env = A.run_assertions_from_file("assertions.yaml", ctx)
print(env["exit_code"])   # 0 pass, 1 fail
```

Python API: [`docs/API.md`](API.md) covers the scoring envelope this
complements; the schema is `src/hotato/schema/assert.v1.json`.

## The core deterministic kinds

Every kind is deterministic, which is why every result carries
`deterministic: true` -- including a result whose status is `INCONCLUSIVE`.
The five below are the original core; the full vocabulary is listed under
[the whole `kind` vocabulary](#the-whole-kind-vocabulary).

| kind | checks | reads |
| --- | --- | --- |
| `phrase` | a regex is present (or, in `absent` mode, never present), with an optional `role` filter and `position` (`first`/`last`/`any`) | transcript text |
| `pii` | deterministic detectors (`ssn`, `card_luhn` with full Luhn validation, `email`, `phone`) find nothing, `mode: must_not_leak` | transcript text |
| `policy` | a named, versioned, offline rule pack's banned-language and required-disclosure rules | transcript text |
| `tool_call` | a tool was (or was not) called, with an optional argument subset, a count bound, a required order across tools, or a "never before" ordering constraint | ingested `voice_trace.v1` spans only |
| `outcome` | task success as `all_of`/`any_of` a list of the sub-predicates above (`tool_called`, `phrase`, `field_present`), reported as a `met`/`of` fraction | whichever context each sub-predicate needs |

`tool_call` checks the ingested trace (`hotato trace ingest`, `docs/TRACE.md`)
alone -- that's the evidence a tool ran; an agent's own words claiming it ran
don't move the needle. `pii` surfaces only a `[REDACTED]` transcript artifact
plus hit metadata (detector name, transcript turn index, role) -- never the
matched text itself.

### The whole `kind` vocabulary

The full
`assert.v1` `kind` vocabulary is **17 deterministic kinds** -- all
`deterministic: true`, all model-free. Alongside the five core kinds:
`tool_result` and `tool_error` (a tool's returned value or raised error, read
from the trace), `state` and `state_change` (a state adapter's snapshot or
transition -- see [STATE-ADAPTERS.md](STATE-ADAPTERS.md)), `handoff`, `dtmf`,
`termination`, `latency`, `timing_contract`, `entity_accuracy`, `sequence`,
and `count`. Two further NAMED kinds, `human_rubric` and `judge_rubric`, belong
to the SEPARATE model-judged rubric lane ([RUBRIC.md](RUBRIC.md)); inside a raw
`assert.v1` document they resolve to a deterministic `INCONCLUSIVE` that points
at that lane, so no model runs here and the deterministic guarantee holds.

## Context: transcript, trace, timing

`build_context` assembles the three inputs an assertion run is evaluated
against, using hotato's existing primitives to produce each one:

- **transcript**: `hotato.transcribe` (the opt-in `[transcribe]` extra,
  faster-whisper) produces one, or pass `--transcript FILE` / `transcript_path=`
  with a JSON file you already have -- a plain array of `{role, text, start,
  end}` turns, or the `{"segments": [...]}` shape `hotato.transcribe` and the
  MCP surface already write. Either way, `assert` works out of the box.
- **trace**: `hotato trace ingest --otel FILE --out voice_trace.jsonl`
  (`docs/TRACE.md`, `docs/OTEL.md`) produces the `hotato.voice_trace.v1` spans
  `tool_call` reads (`name`, `arguments`, and -- when the source trace carries
  them -- `result`/`error`).
- **timing**: a scoring run's own envelope (`hotato run --format json`,
  `docs/API.md`) passed straight through as read-only context for `outcome`'s
  `field_present` sub-predicate. Nothing here recomputes it.

A piece of context that was never supplied stays `None` -- distinct from `[]`
or `{}`, a value that WAS supplied and happens to be empty. An assertion whose
required input is simply absent reports `INCONCLUSIVE`. `tool_call` with
`spans=[]` (a trace WAS ingested, it just has zero spans) is a `FAIL` --
distinct from `tool_call` with no trace at all, which is `INCONCLUSIVE`.

## `assertions.yaml`

A small, dependency-free YAML subset (block mappings/sequences, flow
`[...]`/`{...}`, quoted or bare scalars, `#` comments) -- or a document that is
already valid JSON, accepted directly. Hotato parses this subset itself, so
the core stays zero third-party dependency either way.

```yaml
version: 1
assertions:
  - id: refund-confirmed
    kind: outcome
    all_of: [{tool_called: issue_refund}, {phrase: "confirmation number", role: agent}]
  - id: tool-order
    kind: tool_call
    require_order: [verify_identity, lookup_account, issue_refund]
    never_before: {tool: issue_refund, until: verify_identity}
  - id: disclosure
    kind: phrase
    regex: "recorded for quality"
    role: agent
    position: first
  - id: no-ssn-leak
    kind: pii
    detectors: [ssn, card_luhn]
    mode: must_not_leak
```

Every assertion needs a unique `id` and a recognized `kind` (the core table
above, or any of the additional deterministic kinds listed with it); the
kind-specific fields are validated (bad regex, unknown detector, missing
required field, an unsupported `version`) up front, before anything is
evaluated -- a malformed file is caught whole, before partial results exist.

## The deterministic/judge split

This is the entire point of the module, and it is structural, not a
convention someone can quietly break:

- Every result carries `kind` and `deterministic: true` -- true on every
  deterministic kind and every status, including `INCONCLUSIVE`, which is
  itself a deterministic read of missing required input.
- The envelope's `summary` **splits** `deterministic` (`{pass, fail,
  inconclusive}`) from `judge` (`{pass, fail}`), each kept in its own count --
  the schema (`src/hotato/schema/assert.v1.json`) enforces this with
  `"overall_score": false` and a `not: {required: [overall_score]}` on the
  summary object.
- `judge` -- an LLM-scored rubric kind -- is a separate, quarantined
  capability reserved for a future release. `summary.judge` is always
  `{"pass": 0, "fail": 0}`, and `summary.note` states plainly how many
  judge-scored assertions ran (zero, in this build).
- Same inputs, same assertions file, same result, every time: `run_assertions`
  is byte-stable across repeated calls on identical input -- every field in
  the envelope is reproducible, with no wall-clock timestamp or random id in
  the mix.

The report (below) renders this as two visually separate shelves
instead of one number, so it is visible on the page, not just in the JSON.

## The report: two shelves, each counted on its own

`hotato.report.build_report_html` / `build_report_md` accept an optional
`assertions=` parameter: an already-evaluated `assert.v1` envelope (build one
with `run_assertions` / `run_assertions_from_file` / `run_assertions_from_yaml`
above). Exactly like `base` (a previous run envelope) and `transcript` (an
already-produced ASR artifact), the report purely renders an assertion result
that was handed to it.

```python
from hotato import assert_ as A, report

env = A.run_assertions_from_file("assertions.yaml", ctx)
html, _ = report.build_report_html(suite="barge-in", assertions=env)
```

When present, it adds one "Assertions" section with:

- **The headline.** Two counts side by side, each scored on its own:
  `N deterministic pass / M fail  K judge-scored (advisory)`.
- **Deterministic** (audio / timing / transcript / trace derived): one
  PER-DIMENSION TYPED card per result -- a kind tag, the `deterministic` flag
  stated plainly, the PASS/FAIL/INCONCLUSIVE chip, and whatever kind-specific
  fields that particular result carries (a `pii` card's hit detail
  and redacted transcript, a `policy` card's matched rules and pack name, a
  `tool_call` card's grounding span ids, an `outcome` card's met/of fraction).
- **Model-assisted (advisory, quarantined)**: reserved for a future release;
  empty in this build, with a note explaining why.

`assertions=None` (the default) is byte-identical to a report built before
this parameter existed.

## `inconclusive_policy`: making missing input gate CI

By default an `INCONCLUSIVE` result -- a check whose required input was simply
absent -- leaves the run's exit code unaffected. That is the default for an
exploratory run, but it means a suite whose transcript or trace never arrived
stays silently green. `inconclusive_policy` lets a suite opt into gating on
that:

| value | how `INCONCLUSIVE` gates | `exit_code` |
| --- | --- | --- |
| `report` (default) | reports missing input and leaves the gate unchanged | `1` if any `FAIL`, else `0` |
| `fail` | gates exactly like a `FAIL` | `1` if any `FAIL` **or** `INCONCLUSIVE`, else `0` |
| `refuse` | refuses to return a verdict at all | `2` if any `INCONCLUSIVE`, else `1` if any `FAIL`, else `0` |

**`refuse` precedence.** Under `refuse`, an `INCONCLUSIVE` result exits `2`
*even if another assertion also `FAIL`ed* -- the exit-2 refusal, withholding a
verdict when required input is missing, takes precedence over the FAIL. A run
that cannot fully see its inputs withholds its verdict.

The **default is `report`**, so a suite that sets nothing gates exactly as it
did before this field existed -- fully backward-compatible. **CI and
compliance suites should set `fail` or `refuse`**, so a missing transcript or
trace fails loudly.

Set it as an optional top-level key in the assertions document:

```yaml
version: 1
inconclusive_policy: fail   # or: refuse | report (default)
assertions:
  - id: disclosure
    kind: phrase
    regex: "recorded for quality"
    role: agent
```

or override the document's key from the CLI (the flag wins):

```bash
hotato assert run --assertions assertions.yaml --transcript call.json \
    --inconclusive-policy refuse
```

or from Python (an explicit argument overrides the document's key; absent
both, the default `report` applies):

```python
env = A.run_assertions_from_file("assertions.yaml", ctx, inconclusive_policy="fail")
```

A bad value (anything but `report`/`fail`/`refuse`), whether in the document
or passed explicitly, is a usage error (`ValueError`, exit `2`) raised during
validation, before any assertion is evaluated. The envelope always carries the
`inconclusive_policy` applied and states it, with the counts, in
`summary.note`.

## Mapping to a CI gate

Same exit-code convention as every other hotato command: `0` every assertion
passed (under the default `report` policy an `INCONCLUSIVE` result reports
missing input and leaves the exit code at `0`; under `fail` it gates like a
`FAIL`; under `refuse` it exits `2`), `1` at least one assertion's
deterministic status is `FAIL` (or, under `fail`, `INCONCLUSIVE`), `2` a
refusal under `refuse` (an `INCONCLUSIVE` result, taking precedence over a
`FAIL`) or a malformed assertions file / bad input, raised before any
assertion is evaluated.

```bash
python3 - <<'PY'
from hotato import assert_ as A
import sys

ctx = A.build_context(transcript_path="transcript.json", trace_path="voice_trace.jsonl")
env = A.run_assertions_from_file("assertions.yaml", ctx)
print(env["summary"]["note"])
sys.exit(env["exit_code"])
PY
```

A gate on `assert.v1` and a gate on the timing scorer's own `exit_code`
(`hotato run` / `hotato verify`, `docs/CI.md`) are two different, composable
guarantees: one gates turn-taking timing, the other gates transcript/trace
content. Run both; neither one's exit code substitutes for the other's.


================================================================================
FILE: docs/CONVERSATION-TEST.md
================================================================================

# Conversation tests (`hotato test run`): one file, one call, a per-dimension scorecard

A **conversation test** (`hotato.conversation-test.v1`) is one file that defines
one testable conversation: the agent under test, the simulated caller, the
environment, two SEPARATE assertion lanes, and an explicit success condition.
`hotato test run` evaluates a supplied call against that file and produces a
**conversation artifact** (`hotato.conversation.v1`) whose evidence is bound by
sha256, plus a **per-dimension scorecard**. Success is a boolean over a small
closed vocabulary of named conditions, with every dimension counted on its
own, including in `--format json`.

This is the single documented end-to-end conversation-QA workflow:

```
hotato scenario init   ->   hotato test run   ->   hotato conversation verify
   (author a file)          (evaluate a call)        (digest-check the artifact)
```

## 1. Author a conversation test

```bash
hotato scenario init refund-flow --agent support-v3 --out refund.yaml
```

The starter carries the two lanes, tagged checks across the five report
dimensions, and a boolean `success`:

```yaml
kind: hotato.conversation-test
version: 1
id: refund-flow
agent: support-v3
caller:
  persona: a customer whose order arrived damaged and wants a refund
  goal: get a refund for order A-1001
  facts: {order_id: A-1001}
assertions:
  # DETERMINISTIC lane: pure, offline, no model. Each result is TAGGED with one
  # of the five report dimensions (a grouping key, never a weight).
  deterministic:
    - id: disclosure-said
      kind: phrase
      regex: "recorded for quality"
      role: agent
      dimension: policy
    - id: refund-tool-called
      kind: tool_call
      name: issue_refund
      dimension: outcome
    - id: lookup-then-refund
      kind: sequence
      steps: [{tool: lookup_order}, {tool: issue_refund}]
      dimension: conversation
    - id: refund-latency
      kind: latency
      tool: issue_refund
      max_ms: 1500
      dimension: speech
  # RUBRIC (model-judged) lane lands in Phase 3. In Phase 1 each is INCONCLUSIVE
  # (no model runs); it never counts toward the deterministic result or exit code.
  rubric:
    - id: was-empathetic
      kind: judge_rubric
      dimension: conversation
# inconclusive_policy: fail  # CI/compliance suites should set fail or refuse
success:
  # Success is the conjunction of these named conditions -- never a score.
  required: [all_deterministic_assertions_pass, no_rubric_failure]
  report_dimensions: [outcome, policy, conversation, speech, reliability]
```

Validate one file or a whole directory (exit 2 on any malformed file):

```bash
hotato scenario validate refund.yaml
hotato scenario validate ./scenarios --format json
```

The closed vocabularies are enforced structurally (`hotato.conversation_test`):
`success.required` is drawn from `all_deterministic_assertions_pass`,
`no_deterministic_fail`, `no_rubric_failure`, `no_inconclusive`; `dimension` is
one of `outcome`, `policy`, `conversation`, `speech`, `reliability`; an
`overall_score` key anywhere is rejected.

## 2. Evaluate a call

`hotato test run` takes the file plus whatever evidence you have for the call --
a scored recording (`--audio`), an ingested trace (`--trace`, see
[`docs/TRACE.md`](TRACE.md)), a transcript (`--transcript`), and a post-call
state sandbox (`--state`, Authority 2). Each supplied piece feeds the
assertions; each ABSENT piece leaves the checks that need it `INCONCLUSIVE`.

```bash
hotato test run refund.yaml --agent support-v3 \
    --audio call.wav \
    --trace voice_trace.jsonl \
    --transcript call.transcript.json \
    --out ./conv-artifact --format html
```

Flow: load + validate the file -> build the evaluation context from the supplied
transcript / trace / state / (scored) timing -> evaluate the DETERMINISTIC lane
(the rubric lane stays quarantined -> `INCONCLUSIVE`) -> evaluate
`success.required` over the results -> bind the evidence into a
`hotato.conversation.v1` artifact -> render the unified report + per-dimension
scorecard. The per-dimension summary printed to stdout:

```
hotato test run: refund-flow (agent support-v3) -- exit_code=0
inconclusive_policy: report
success: PASS  (required: all_deterministic_assertions_pass, no_rubric_failure)
  [ok] all_deterministic_assertions_pass
  [ok] no_rubric_failure
per-dimension (grouped view; never blended):
  outcome       1 pass / 0 fail / 0 inconclusive
  policy        1 pass / 0 fail / 0 inconclusive
  conversation  1 pass / 0 fail / 0 inconclusive
  speech        1 pass / 0 fail / 0 inconclusive
  reliability   0 pass / 0 fail / 0 inconclusive
reliability over 1 repeated run(s) of the deterministic lane on the same supplied real recording; the lane has zero variance, so pass^k == pass@1 -- reported honestly, not fabricated variance
rubric lane: 1 assertion(s) INCONCLUSIVE (quarantined, Phase 3 -- no model ran)
deterministic: 4 pass, 0 fail, 0 inconclusive
judge: 0 pass, 0 fail (no judge/rubric kind is built in this release)
```

`--format`:

| format | stdout | writes into `--out` |
| --- | --- | --- |
| `text` (default) | the per-dimension summary | the conversation artifact |
| `html` / `md` | the summary (+ where files landed) | the artifact **and** `report.{html,md}` (needs `--audio`) |
| `json` | the full machine result | the conversation artifact |

### Exit code

The exit code honors the file's `inconclusive_policy` exactly as `assert run`
does, then is raised to non-zero when a `success.required` condition fails:

| `inconclusive_policy` | an `INCONCLUSIVE` (missing-input) result | a `FAIL` |
| --- | --- | --- |
| `report` (default) | does not gate (exit 0) | exit 1 |
| `fail` | gates like a FAIL (exit 1) | exit 1 |
| `refuse` | withholds the verdict (exit 2, precedence) | exit 1 |

A `success.required` failure makes an otherwise-passing run non-zero; a refuse
(exit 2) is never downgraded.

### Reliability and repetitions

`--repetitions N` runs the deterministic lane N times and reports the per-run
results, the run count, and a reliability aggregate: **pass@1** (single-run
pass rate), **pass@k** (>=1 of k passed), **pass^k** (all k passed), plus a
Wilson 95% CI. Every run scores the same recording, so the deterministic lane
has zero variance and `pass^k == pass@1`. With `N > 1` the aggregate is
threaded into the report's Reliability dimension (`--format html/md`); with no
repetition data that dimension shows the empty-state ("not measured: no
repeated runs in this report"). pass^k stays its OWN number, kept separate
from every other dimension and from `overall_score`.

## 3. Verify the artifact

The conversation artifact directory binds each child by sha256:

```
conv-artifact/
  conversation.json      # the manifest (origin real|simulated + bound digests)
  audio/call.wav         # the recording (single dual-channel form)
  transcript.json
  trace.jsonl
  timing.json            # the scored envelope
  assertions.json        # the evaluated assert.v1 envelope
  report.html            # the unified report + per-dimension scorecard
```

`hotato conversation verify` re-hashes every bound child and REFUSES (exit 2) on
any digest mismatch or missing file -- a tampered or absent artifact is refused
outright:

```bash
hotato conversation verify ./conv-artifact
# conversation refund-flow: VERIFIED
#   verified: assertions, audio, timing, trace, transcript
#   all 5 bound artifact(s) re-hashed to their recorded digest
```

`origin.kind` is `real` by default (a supplied recording is evaluated as-is) and
`simulated` only when the test file carries a `simulator` block, which must
declare its `model_id` / `scenario_id` / `seed` -- synthetic and live origins
are never conflated.

## The deterministic/judge split (structural)

* **Each dimension scored on its own.** Success is a boolean conjunction of
  named conditions; the scorecard groups the same results by dimension, each
  with its own counts.
* **Two separate lanes.** Deterministic checks (regex / checksum / span / state
  lookup -- no model) are evaluated; the model-judged rubric lane is quarantined
  until Phase 3, reported `INCONCLUSIVE` and kept out of the deterministic
  summary and the exit code.
* **Grounded in authority.** `tool_result` / `tool_error` (Authority
  1) read the ingested trace spans; `state` / `state_change` (Authority 2) query
  a post-call state adapter -- the trace and the adapter are the evidence a
  tool ran or a state changed, deterministic and model-free.
* **Missing input is `INCONCLUSIVE`.** A check whose evidence was not supplied
  reports `INCONCLUSIVE`.

## The bundled end-to-end example

`tests/data/conversation/` ships one worked call -- a conversation-test file, a
transcript, and a `voice_trace.v1` trace -- evaluated against the bundled
recording `01-hard-interruption.example.wav`. `tests/test_test_run_cli.py`
(`test_bundled_call_one_file_end_to_end_scorecard_and_artifact`) drives the
whole workflow: one call evaluated for outcome / policy / timing /
transcript-facts / tool-behaviour from one file, producing an artifact that
`conversation verify` passes and a five-dimension scorecard.

## See also

* [`docs/ASSERTIONS.md`](ASSERTIONS.md) -- the deterministic assertion kinds.
* [`docs/TRACE.md`](TRACE.md) -- ingesting a `voice_trace.v1` trace.
* [`docs/REPORTS.md`](REPORTS.md) -- the unified report and the scorecard.


================================================================================
FILE: docs/RUBRIC.md
================================================================================

# Rubric evaluation: the model-judged lane (`rubric.v1`)

`hotato rubric` scores a call against a **user-authored rubric** with a pinned
**local** model, in a lane kept structurally separate from the deterministic
`assert.v1` wall. Every rubric result carries `deterministic: false` with full
provenance and lives on its own report shelf, kept in its own count, apart
from any overall number.

- `assert.v1`'s kinds stay `deterministic: {const: true}` forever. A model
  verdict lives in its own `rubric.v1` lane, structurally apart from
  `assert.v1`.
- The default judge is a **local Ollama** model (`http://localhost:11434`) reached
  with only the stdlib. Zero egress. A hosted judge is opt-in (see below).
- **Advisory by default**: the deterministic checks set the gate; a rubric
  FAIL is reported and the model's read stays advisory. `--gate` (or
  `--gate-judge` on `test run`) opts a team into gating on a rubric FAIL too.
- Missing/insufficient evidence -> `INCONCLUSIVE`, always labeled plainly. A
  `human_rubric` stays `INCONCLUSIVE` and human-required, by design.

## The rubric object

```yaml
version: 1
rubrics:
  - id: acknowledged-frustration
    kind: judge_rubric            # or human_rubric (human-scored only)
    dimension: conversation       # optional report-dimension tag
    criterion: "Did the agent acknowledge the caller's frustration before proposing a fix?"
    evidence: [transcript]        # transcript and/or tool_trace; absent -> INCONCLUSIVE
    examples: {pass: "...", fail: "..."}      # optional
    evaluation:
      model: qwen2.5vl:3b         # optional; else --judge-model / the default
      repetitions: 1              # N model calls, aggregated
      aggregation: unanimous_or_inconclusive
      confidence_required: 0.85
    review:
      human_required_on: [fail, disagreement, confidence_below_threshold]
```

## Run it

```bash
# advisory (exit 0 regardless of verdicts)
hotato rubric run --rubrics rubrics.yaml --transcript call.json --trace trace.jsonl

# opt into CI gating on a rubric FAIL
hotato rubric run --rubrics rubrics.yaml --transcript call.json --gate

# re-query the model and DIFF against the cached verdict (surface drift)
hotato rubric run --rubrics rubrics.yaml --transcript call.json --no-cache
```

Inside a conversation-test, author the rubric in the `assertions.rubric` lane;
`hotato test run` scores it inline (advisory; `--gate-judge` to gate) and the
unified report shows a populated **Model-assisted (advisory)** shelf beside the
deterministic one -- two counts side by side, each on its own lane.

## Result provenance (`rubric.v1`)

Every result records: the pinned `model` + its content `model_digest`, `provider`,
`prompt_id` / `prompt_version` / `prompt_sha256`, `temperature: 0`, the
`input_sha256`, the `cache_key`, `cached`, the raw `votes`, `disagreement`,
`confidence`, and `citations` to the exact transcript turns / trace events --
its own number, scored on its own lane; the `rubric.v1` schema rejects an `overall_score` key.

## Reproducibility (stated precisely)

**What's guaranteed is replay.** Every verdict is content-addressed by
`sha256(provider:model + prompt_sha256 + input_sha256)` and cached (reusing the
content-addressed artifact store), so a cache hit reproduces the same
`verdict_sha256` forever. A fresh model call is a separate question, checked
explicitly with `--no-cache`, which re-queries the model and **diffs** the
result against the cached verdict, surfacing any drift directly. `--sign`
optionally signs a cached verdict as a "judge-record" (Ed25519 `human` tier via
the `[sign]` extra, else HMAC `human-shared`), so a stored verdict is provably
unmutated.

## Egress

The default local judge stays on the box. A **hosted** judge
(`--judge-provider hosted --judge-endpoint URL`) or a **non-local**
`--judge-endpoint` sends the transcript off-box, and requires
`--judge-egress-opt-in` to proceed (exit 2 without it) -- the same posture as
`--diarizer pyannoteai --egress-opt-in`. See [`docs/EGRESS.md`](EGRESS.md) and
[`docs/THREAT-MODEL.md`](THREAT-MODEL.md).

## Calibration: measured credibility

```bash
hotato rubric calibrate --labeled ./labeled --out agreement.json
```

Each `*.json` item is `{rubric, transcript, trace?, label: pass|fail|inconclusive,
split?: train|held_out}`. Human labels are **mandatory** here: humans author the
labels, and the model is scored *against* them. The command computes
**agreement** (held-out items where the model verdict equals the human label) and
**selective accuracy** (agreement restricted to items where the model committed
to a verdict) and writes a reproducible artifact of raw counts + method + provenance.
Re-running on the same corpus with the same model reproduces the same split and
verdicts -- the numbers travel with their method and provenance built in.


================================================================================
FILE: docs/SIMULATE.md
================================================================================

# Simulate (`hotato simulate`): a scenario to a deterministic, labelled conversation

`hotato simulate` renders a **scenario** (`hotato.scenario.v1`) through a
DETERMINISTIC scripted caller into one or more **conversation artifacts**
(`hotato.conversation.v1`), each labelled `origin=simulated`. It runs fully
offline, scripted-caller only: the deterministic INPUT side of a simulation --
the ground-truth a caller holds, the caller's scripted turns, the environment,
and an optional variation matrix.

It is the trustworthy regression foundation you build BEFORE any generative
caller: a fixed `(scenario, seed)` reproduces the transcript **byte-for-byte**
(the produced transcript is content-hashed), so a run is a stable, reusable
fixture.

## Quickstart (works from a bare `pip install`)

```bash
pip install hotato

# 1. Write a minimal scenario simulate accepts as-is:
hotato simulate --init demo.scenario.json

# 2. Render it into a labelled origin=simulated conversation:
hotato simulate demo.scenario.json --out ./sim

# 3. (optional) digest-verify the produced artifact:
hotato conversation verify ./sim
```

Prefer zero files? The package ships a minimal scenario you can run directly --
no file on disk:

```bash
hotato simulate --example --out ./sim
```

Step 2 prints:

```
wrote 1 simulated artifact(s) under ./sim/
hotato simulate: demo -- 1 run(s), origin=simulated (never real)
  run 1: seed=0 77f1eb9e6994 sim=ok  -> ./sim
reliability: pass@1=1.000 pass@k=1.000 pass^k=1.000 (n=1)
```

The `./sim` directory holds a `conversation.json` plus its bound transcript and
`voice_trace.jsonl`, all `origin.kind=simulated`.

> **Two different `scenario` concepts, one name.** `hotato simulate` consumes a
> `hotato.scenario.v1` doc -- author one with `hotato simulate --init`.
> `hotato scenario init` writes a separate file, a `hotato.conversation-test.v1`
> that `hotato test run` consumes -- see
> [CONVERSATION-TEST.md](CONVERSATION-TEST.md). Feed a conversation-test file
> to `simulate` and you get an actionable error that points you back to
> `--init`.

## What `--init` writes

`hotato simulate --init demo.scenario.json` writes a MINIMAL, valid
`hotato.scenario.v1` doc. It is a starter you edit for your own agent -- shape
the caller turns to match your own call. The scenario id is derived from the
filename stem.

```json
{
  "kind": "hotato.scenario",
  "version": 1,
  "id": "demo",
  "goal": { "type": "get_refund", "target": "order A-1001" },
  "facts": { "order_id": "A-1001" },
  "caller": {
    "script": [
      { "say": "Hi, my order A-1001 arrived damaged and I would like a refund." },
      { "say": "Yes, please refund it to my card." }
    ],
    "behavior": { "backchannels": { "probability": 0.0 } }
  },
  "environment": { "locale": "en-US", "route": "phone" },
  "seed": 0
}
```

The caller's script holds only the caller's own turns -- a `say` is the caller
speaking. The schema enforces this structurally: there's no field for the
agent's words, so a scenario stays scoped to the caller's side by construction,
at the schema level.

Required fields: `kind`, `version`, `id`, `goal` (`type` + `target`), and
`caller.script` (at least one `say`). The full schema, including `variation_matrix`
and the optional deterministic `agent_mock`, is
[`schema/scenario.v1.json`](../src/hotato/schema/scenario.v1.json).

## The invariants (enforced structurally)

- **`origin=simulated` on every produced conversation**, kept in its own
  bucket, apart from real calls. `write_artifact` only writes artifacts whose
  origin is simulated.
- **A bad rendering is `SIMULATOR_INVALID`, kept distinct from an agent
  PASS/FAIL.** The simulator only decides whether the produced conversation is
  a FAITHFUL rendering of its scenario. Scoring is the SEPARATE assert layer's
  job, over the produced artifact.
- **Reproducibility means a SEEDED REPLAY is byte-identical** -- there's no
  model in this path. A fixed `(scenario, seed)` produces the same transcript
  bytes every time; different seeds differ ONLY where the scenario allows it
  (probabilistic backchannels).
- **Each dimension scored on its own lane** -- enforced structurally in both
  the schema, which rejects an `overall_score` key, and the code path.

## Reliability: pass@1 / pass@k / pass^k

`simulate` reports **Reliability** as its own dimension, scored on its own lane:

- `pass@1` -- the fraction of runs that rendered faithfully.
- `pass@k` -- at-least-one-passes across `k` runs.
- `pass^k` -- all-`k`-pass.

For the scripted deterministic caller `pass^k == pass@1`: a seeded replay is
byte-identical, so every run has the same outcome. That is the correct,
plainly reported outcome; variance shows up only where the scenario
introduces it. `--repetitions N` expands the variation matrix so Reliability
is measured over `N` runs.

## Simulate many scenarios in parallel (`--matrix`)

```bash
# expand the scenario's FULL variation matrix (locale x speaking_rate x noise x
# behavior x repetitions), render + validate each in a bounded pool:
hotato simulate --matrix demo.scenario.json --out ./matrix

# score each produced conversation against a conversation-test's DETERMINISTIC
# assertions; SIMULATOR_INVALID runs are bucketed separately, kept distinct
# from a PASS/FAIL:
hotato simulate --matrix demo.scenario.json \
    --conversation-test refund.test.yaml --parallel 8 --format json
```

The summary stays byte-identical no matter the worker count. Every result is
attributed to its own variation cell, each scored on its own lane.

## Exit codes

| Code | Meaning |
| ---- | ------- |
| `0`  | every produced conversation is `origin=simulated` and validated as a faithful rendering (and, under `--matrix --conversation-test`, every scored aggregate passed). |
| `1`  | at least one produced simulation was `SIMULATOR_INVALID` -- a broken fixture, kept distinct from an agent PASS/FAIL -- or, under `--matrix --conversation-test`, a scored aggregate FAILed. |
| `2`  | usage error / unusable input: a malformed or unreadable scenario / conversation-test file (or, under `--matrix --conversation-test` with `inconclusive_policy refuse`, a withheld verdict). |

## Related

- [CONVERSATION-TEST.md](CONVERSATION-TEST.md) -- `hotato scenario init` /
  `hotato test run`: score a real call against a conversation-test.
- [`schema/scenario.v1.json`](../src/hotato/schema/scenario.v1.json) -- the full
  scenario schema.


================================================================================
FILE: docs/SUITE-RUN.md
================================================================================

# `hotato suite run`: execute a suite of conversation-tests

`hotato suite run` loads a `hotato.suite` file (a NAMED SET of
conversation-test refs, each scored on its own), resolves each ref relative to the suite
file, executes every test, and emits a per-dimension + reliability report. It
also records the run into the fleet registry so `hotato serve` can browse it.

A conversation-test is one file that defines one testable conversation; see
[CONVERSATION-TEST.md](CONVERSATION-TEST.md). A suite groups those files under a
`suite_id`, a `purpose`, an `inconclusive_policy`, and a
`required_for_release` flag. The suite file is JSON or YAML.

## How a test runs

- A test that declares a `scenario` runs **offline** through the deterministic
  scripted-caller simulator. The scenario's variation matrix expands into
  concrete runs, each rendered, validated, and scored against the test's
  DETERMINISTIC lane. Authority-1 tool spans and the Authority-2 mock-state
  sandbox come from the scenario's `agent_mock`, fully simulated and offline.
  (Authority 2 is grounded state; see [STATE-ADAPTERS.md](STATE-ADAPTERS.md).)
- A test with no scenario is evaluated once against an empty context: every
  input-dependent check is INCONCLUSIVE, never a guessed pass/fail.

## Every dimension scored on its own

The report carries per-dimension counts (`outcome`, `policy`, `conversation`,
`speech`, `reliability`) plus a reliability aggregate -- `pass@1`, `pass@k`,
`pass^k`. Per-dimension counts group results across tests; reliability stands
as its own dimension over every valid simulated run. Each dimension keeps its
own count and its own verdict, exposed as-is, including in `--format json`;
the schemas reject an `overall_score` key.

A `SIMULATOR_INVALID` run flags a broken fixture. It is bucketed separately
and reported by test id and run id, kept out of every dimension and the
reliability aggregate.

## The suite's policy is authoritative

The SUITE's `inconclusive_policy` is the effective policy for every test it
names, keeping a required CI/compliance suite's gate consistent regardless of
a test's own policy. So a suite with `inconclusive_policy: fail` makes an
INCONCLUSIVE (absent required input) FAIL the gate; `refuse` withholds the
verdict (exit `2`, precedence).

The suite exit code is the **worst** test outcome under that policy
(`refuse` > `fail` > `pass`), raised to at least `1` by any `SIMULATOR_INVALID`
run.

## Run it

```bash
hotato suite run examples/reference-agent/suite.json \
    --agent reference-agent-v1 \
    --release reference-agent-v1 \
    --out ./suite-out
```

`--agent ID` (required) is recorded on every conversation and the release.
`--release ID` names the release the runs are recorded under (default
`<agent>@<suite_id>`); use a stable id per release so
[`hotato release compare`](RELEASE-COMPARE.md) can diff two of them. `--out DIR`
writes the per-test simulated conversation artifacts plus `suite-report.md`,
`suite-report.html`, and `suite-run.json`. `--parallel N` caps the worker
threads for each scenario's matrix; the worker count never changes the
byte-identical result.

The runs are recorded into the fleet registry (`--registry`, default
`~/.hotato/fleet`; `--workspace`/`-w`, default `default`) as
Release / Suite / Scenario / Run / Conversation / Evaluation rows, so
`hotato serve -w <workspace>` renders them (see [WORKSPACE.md](WORKSPACE.md)).
Pass `--no-registry` to report only.

## Worked example: the reference agent

`examples/reference-agent/` ships a full suite: 25 realistic voice-agent jobs
x 5 caller behaviours x 3 audio environments = 375 offline simulated runs,
scored across the outcome / policy / conversation / speech dimensions. Its
`suite.json` has `inconclusive_policy: fail` and `required_for_release: true`.

The one-command path regenerates the files, runs the 375-run suite offline, and
records a browsable workspace:

```bash
cd examples/reference-agent
make reference
# then browse:
hotato serve --workspace reference --registry ./.workspace
```

The summary printed to stdout follows this shape (counts here are schematic,
illustrating the report's shape):

```
hotato suite run: reference-agent-suite (...) -- agent reference-agent-v1, release reference-agent-v1 -- exit_code=0
inconclusive_policy: fail  required_for_release: True
tests: 25  (25 pass, 0 fail, 0 refuse)
runs: 375 (375 valid, 0 SIMULATOR_INVALID -- broken fixtures, never an agent PASS/FAIL), origin=simulated
per-dimension (grouped across tests; never blended):
  outcome       ... pass / ... fail / ... inconclusive
  policy        ... pass / ... fail / ... inconclusive
  conversation  ... pass / ... fail / ... inconclusive
  speech        ... pass / ... fail / ... inconclusive
  reliability   ... pass / ... fail / ... inconclusive
reliability [origin=simulated]: pass@1=... pass@k=... pass^k=... (n=...)
per-test:
  [pass   ] refund-damaged-order            runs=15  ...
  ...
```

Origin is always labelled `simulated`, keeping a simulator's replay
reliability distinct from production reliability.

## Exit codes

| code | meaning |
| --- | --- |
| `0` | every test in the suite passed under the suite's `inconclusive_policy` (and no run was `SIMULATOR_INVALID`) |
| `1` | at least one test FAILed (a `success.required` condition failed, a deterministic assertion FAILed, or -- under `inconclusive_policy fail` -- an INCONCLUSIVE gated), or a run was `SIMULATOR_INVALID` (a broken fixture, kept separate from any agent PASS/FAIL) |
| `2` | under `inconclusive_policy refuse` a scored INCONCLUSIVE withheld the verdict (takes precedence over a FAIL); OR a usage error / unusable input: a malformed suite / conversation-test / scenario file, or an unresolvable test/scenario ref |

## See also

- [CONVERSATION-TEST.md](CONVERSATION-TEST.md) -- the conversation-test file each suite ref points at.
- [STATE-ADAPTERS.md](STATE-ADAPTERS.md) -- Authority 2 state grounding for `state` / `state_change` checks.
- [RELEASE-COMPARE.md](RELEASE-COMPARE.md) -- diff two releases recorded by `suite run`.
- [WORKSPACE.md](WORKSPACE.md) -- browse the recorded runs with `hotato serve`.
- [SUITES.md](SUITES.md) -- the `corpus/suites/` scenario-audio suites for the `hotato run` scoring path (a distinct mechanism).


================================================================================
FILE: docs/RELEASE-COMPARE.md
================================================================================

# `hotato release compare`: diff two releases, per dimension

`hotato release compare BASELINE CANDIDATE` reads two releases from the fleet
registry and reports what moved between them. `BASELINE` is first, `CANDIDATE`
second. It reports movement; gating a release is a separate step (use a
`required_for_release` suite for that).

Each release is a snapshot recorded by [`hotato suite run`](SUITE-RUN.md). To
compare, run the same suite twice under two stable `--release` ids in the same
workspace and registry, then diff them.

## What it reports

- **Per-dimension counts for each side, plus the per-count delta.** Each
  dimension (`outcome`, `policy`, `conversation`, `speech`, `reliability`)
  keeps its own three counts (pass / fail / inconclusive), each with its own
  delta -- every dimension scored on its own lane.
- **New failures** -- a `scenario x dimension` that PASSED on the baseline and
  FAILs on the candidate. **Fixed-since** -- the reverse. Both are diffed
  **only** where BOTH releases ran the same `scenario x dimension`. A scenario
  only one side ran counts as new coverage, separate from a regression.
- **Per-scenario status changes** -- every `scenario x dimension` whose status
  differs between the two, diffed only where both sides have a comparable
  result.

The releases' pinned digests are surfaced, so the reader knows exactly which
two snapshots were compared: an exact digest match, every time.

## Empty state

A side with no runs is stated plainly, as an empty state:

- a release id absent from the workspace is reported as unregistered;
- a release that is registered but has **zero runs** is reported plainly as
  an empty state.

When no `scenario x dimension` result is comparable across both releases, the
report says so: new-failures / fixed-since need a scenario BOTH releases ran.

## Compare

```bash
hotato release compare reference-agent-v1 reference-agent-rc2
```

`--workspace`/`-w ID` selects the fleet-registry workspace (default `default`);
`--registry PATH` sets the registry home (default `~/.hotato/fleet`);
`--format {text,json}` picks the output.

## Worked example: two reference-agent releases

Record two releases of the reference agent into the same registry and
workspace, then diff them. The reference example writes to
`examples/reference-agent/.workspace`, workspace `reference`:

```bash
cd examples/reference-agent

# record the baseline release
python run_reference.py --release reference-agent-v1

# edit the scenario / agent_mock fixtures (or point at a changed agent),
# regenerate, and record the candidate release
python run_reference.py --generate --release reference-agent-rc2

# diff them (same workspace + registry both runs recorded into)
hotato release compare reference-agent-v1 reference-agent-rc2 \
    --workspace reference --registry ./.workspace
```

Two runs over identical fixtures record byte-identical results, so the diff is
all-zero deltas and no status changes. Edit the fixtures between the two runs
to see movement.

The text output follows this shape (counts here are illustrative placeholders):

```
hotato release compare: baseline reference-agent-v1 -> candidate reference-agent-rc2  (workspace reference)
baseline runs=375 conv=375 eval=...  candidate runs=375 conv=375 eval=...
per-dimension counts (baseline -> candidate, delta; never blended):
  outcome       ...P/...F/...I  ->  ...P/...F/...I   (dP +.., dF -.., dI +..)
  policy        ...P/...F/...I  ->  ...P/...F/...I   (dP +.., dF -.., dI +..)
  conversation  ...P/...F/...I  ->  ...P/...F/...I   (dP +.., dF -.., dI +..)
  speech        ...P/...F/...I  ->  ...P/...F/...I   (dP +.., dF -.., dI +..)
  reliability   ...P/...F/...I  ->  ...P/...F/...I   (dP +.., dF -.., dI +..)
new failures (PASS on baseline -> FAIL on candidate): N
  - <scenario> / <dimension>
fixed since (FAIL on baseline -> PASS on candidate): N
  + <scenario> / <dimension>
all per-scenario status changes:
  ~ <scenario> / <dimension>: PASS -> FAIL
```

## Exit codes

| code | meaning |
| --- | --- |
| `0` | the two releases were compared (per-dimension deltas + new-failures / fixed-since printed; a side with no runs is reported as an empty state, exit stays `0`) |
| `2` | a usage error or an unreadable registry `--registry` |

## See also

- [SUITE-RUN.md](SUITE-RUN.md) -- records the releases this command diffs.
- [CONVERSATION-TEST.md](CONVERSATION-TEST.md) -- the per-dimension scorecard model.
- [WORKSPACE.md](WORKSPACE.md) -- browse the releases and their runs with `hotato serve`.


================================================================================
FILE: docs/WORKSPACE.md
================================================================================

# The team workspace: `hotato serve`

A self-hosted, local web app for a team to read a voice agent's conversation-QA
state: release readiness, the scenario matrix, a conversation inspector, failure
clusters, and production health. It is stdlib-only (`http.server` + `sqlite3`):
self-hosted, with no framework or build step, and nothing that phones home. It
serves the same fleet registry and evidence store the CLI writes, reading that
data directly instead of passing a database file around.

```
hotato serve --workspace default
```

On first start it prints where it is listening, the bearer token, and the URL to
open:

```
hotato serve: workspace 'default'
  registry:  /home/you/.hotato/fleet
  listening: http://127.0.0.1:8321
  token:     Ab3xQ_p1…                 (generated, stored 0600 at …/serve/default/token)
  open:      http://127.0.0.1:8321/?token=Ab3xQ_p1…
  audit log: /home/you/.hotato/fleet/serve/default/audit.jsonl   (append-only)
  read-only: the server issues only SELECTs; reviews/labels stay CLI-driven. No telemetry, no external calls.
```

Open the `open:` URL in a browser; the server sets a session cookie and redirects
to strip the token from the address bar (see [Auth](#auth)).

## Flags

| flag | default | meaning |
|---|---|---|
| `--workspace`, `-w` | `default` | workspace id to serve |
| `--host` | `127.0.0.1` | bind address (see [Binding](#binding-127001-by-default)) |
| `--port` | `8321` | listen port |
| `--registry` | `~/.hotato/fleet` | registry home directory |
| `--token` | none | supply the bearer token yourself |
| `--token-file` | none | read the bearer token from a file (first line) |

Exit codes: `0` clean shutdown (Ctrl-C); `2` usage error (unusable registry or
token, or the port was unavailable).

## The five views

Every view has a machine mirror at `?format=json` (same auth, same data) so
agents and scripts can drive the workspace without scraping HTML.

1. **Release readiness** (`/`): the pre-ship home screen. Per-release rollup
   from `suites`/`runs`/`evaluations`: whether required suites are complete,
   scenario and run counts, **failures by dimension** (outcome / policy /
   conversation / speech / reliability), the inconclusive count, the origin split
   (real vs simulated, kept separate), and **new-vs-fixed since the previous
   release** compared per (scenario, dimension). Small samples are flagged
   (`low sample, N=3`), never smoothed.
2. **Scenario matrix** (`/scenarios`): rows are scenarios, columns are the
   current and previous release, with a per-dimension status and **reliability**
   (`pass^k` where a scenario has repetitions). Filter by `agent`, `release`,
   `suite`, and `status` via query parameters (there is a filter form at the top).
3. **Conversation inspector** (`/conversation/<id>`): one conversation. Its
   evidence manifest (`conversation.v1`: origin, provider/caller provenance,
   child digests), transcript, trace spans, per-dimension evaluations with
   rationale and citations (deterministic checks and model-judged/advisory
   results shown in **separate lanes**), and reviewer decisions. Every digest is
   a link to the raw evidence blob (`/evidence/<digest>`) to drill straight to the
   source. Redacted transcript segments and trace spans render as `[redacted]`
   and the redacted text is scrubbed from both the HTML and the JSON mirror.
4. **Failure clusters** (`/clusters`): failed evaluations and assertions grouped
   by **observable signature** (dimension + assertion kind + reason-class), with
   counts and drill-through lists into the inspector. Labelled *clusters by
   observable signature*: it groups what was observed, and the cause stays
   yours to determine.
5. **Production health** (`/health`): ingest counts, evaluated coverage, and
   per-dimension failure rate over time, computed **separately for real and
   simulated** conversations, kept distinct. A day with no evaluated sample
   gets no point plotted, and a dimension with fewer than two days of data
   reads *not enough history*, matching the trend report. Each dimension
   keeps its own number -- there's no single combined quality score.

## Auth

Every request is authenticated against one shared **bearer token**:

- **Browser:** open `/?token=<token>` once; the server mints an in-memory,
  HttpOnly session cookie and redirects to remove the token from the URL.
- **Agent / API / `curl`:** send `Authorization: Bearer <token>`.

The token is compared in constant time (`hmac.compare_digest`). If you do not
pass `--token`/`--token-file`, one is generated with `secrets.token_urlsafe` on
first start and stored `0600` at `<registry>/serve/<workspace>/token`, so a
restart keeps the same URL. Sessions live only in memory (never persisted, never
cross-tenant).

## Audit log

Every request appends one JSONL line to
`<registry>/serve/<workspace>/audit.jsonl` (created `0600`):

```json
{"ts":"2026-07-12T18:04:11Z","who":"Ab3xQ_p1…","method":"GET","path":"/scenarios","query":"status=FAIL","status":200,"remote":"127.0.0.1"}
```

`who` is a token/session **prefix**, never the secret; the `token` parameter is
stripped from the recorded query. The audit log is the **only** file the server
writes.

## Read-only in v1

The server writes only the audit log. It issues only `SELECT`s against your
workspace, exposing reads and nothing more: no write endpoint. Reviews, labels, and
adjudications stay CLI-driven (`hotato fleet review …`, `hotato label …`); the UI
states this in its footer.

## Binding 127.0.0.1 by default

The server binds loopback (`127.0.0.1`) unless you explicitly pass `--host`. A
non-loopback bind (for example `--host 0.0.0.0`, to reach the workspace from
another machine) prints a prominent warning, because it exposes the workspace to
your local network. Token auth still applies, but prefer an SSH tunnel or a
reverse proxy you control over binding a wide interface directly.

## Zero egress

The server only opens a **listening** socket, making no outbound connection
and importing nothing that phones home -- audio, traces, and evaluations
stay on the machine. This is covered by a test that whitelists loopback and fails
if any view attempts an external connection, and by the threat-model row below.


================================================================================
FILE: docs/GUARDIAN-FLEET.md
================================================================================

# Hotato Guardian / Fleet

Guardian/Fleet turns Hotato's capture → scan → trust → label → contract → trial
primitives into one continuous workflow, wrapped in a private,
self-hosted control plane. It hardens the evidence kernel first, then layers a
fleet view on top. Every deploy to production stays a human call.

## The evidence kernel (hardened)

### An evidence vector, each dimension scored on its own
Every artifact carries a machine-readable **evidence vector** (`hotato/evidence.py`,
schema `evidence_vector.v1.json`). The public tier is the *weakest* tier any
required dimension allows: a minimum over an inspectable lattice, with each
dimension scored on its own. Tiers, ascending:

| tier | name | meaning |
|---|---|---|
| 0 | none | no usable evidence for a positive claim |
| 1 | asserted | envelope-only; not recomputed from audio |
| 2 | measured | recomputed from audio, one recording, input clean |
| 3 | paired | manifest-bound before/after, both sides recomputed under one pinned policy |
| 4 | attested | paired + signed policy/contract + runner-attested capture |

Dimensions include `score_integrity`, `audio_identity`, `policy_integrity`,
`fixture_set_integrity`, `input_health`, `channel_mapping`, `label_authority`,
`pairing_integrity`, `capture_origin`. One weak dimension pulls the whole tier
down; no renderer may raise it.

### Trial manifest (`hotato/manifest.py`, `trial_manifest.v1.json`)
An immutable pin created *before* an experiment from the battery: the scorer
config hash, one policy (applied to both sides), the complete fixture universe,
and each fixture's expectation, onset, and scripted-stimulus identity. Before and
after must reference the same `manifest_hash`; a changed policy, scorer, label,
onset, or fixture set refuses the comparison.

### Recompute-from-audio (`hotato/recompute.py`)
The proof gate re-derives every verdict from the on-disk audio under the
manifest, checking each stored `verdict.passed` against the recomputed
result, and refuses:

- **verdict tampering**: a stored verdict that disagrees with the recomputed one;
- **same-audio re-encode**: before and after decode to the same PCM;
- **dropped fixtures**: either side omits a pinned fixture;
- **unrelated audio**: the after-side caller stimulus does not match the pinned one and no capture receipt binds it.

A green *paired* proof additionally requires evidence tier ≥ paired.

### Capture receipt (`hotato/receipt.py`, `capture_receipt.v1.json`)
A fresh-recapture claim needs more than distinct PCM. A capture runner emits a
receipt binding the recording to a trial, agent, call id, timestamps, and decoded
PCM; an HMAC signature makes it *machine-verified*. A manual distinct WAV with no
receipt is *operator-asserted* -- a receipt is what promotes it to machine-verified
fresh recapture.

### Contract authenticity (`hotato/attest.py`)
Contracts embed a canonical digest over (schema + label + policy + audio + scorer
version). `contract verify`/`unpack` recompute it: a bundle edited after creation
(e.g. a loosened policy re-pack) reports **tampered** and fails; a matching but
unsigned bundle is *"unsigned, internally consistent evidence"*; a valid HMAC
signature promotes it to *authenticated*.

### Boundary sensitivity
Every event exposes `onset_frame_index`, `onset_effective_sec`,
`decision_margin_sec`, `decision_margin_hops`, and `boundary_sensitive`. A result
within one hop of flipping is flagged, so it no longer reads with the confidence
of one hundreds of milliseconds inside the limit.

### Trust headline
Any verdict-changing warning (low signal, possible channel swap, VAD-relevant
leakage) sets the headline to `scan with caution`. `input_health` is an
explicit three-state field (`clean` / `caution` / `not_scorable`). Leakage is
judged both against the fixed −40 dB bar and dynamically against the receiving
channel's effective VAD gate.

## Fleet control plane (local mode)

Local mode is zero-dependency: SQLite metadata plus a content-addressed artifact
directory (both stdlib).

- **Registry** (`hotato/fleet/registry.py`): workspace-scoped rows for agents,
  deployments, calls, recordings, candidates, labels, contracts, trials,
  decisions. No product-level cap on registered agents; every row carries
  `workspace_id`, keeping paths and call ids scoped to their own workspace.
- **Artifact store** (`hotato/fleet/store.py`): content-addressed blobs with
  dedup and lineage; audio is stored separately from any UI (privacy reversal).
- **Job queue** (`hotato/fleet/jobs.py`): leased jobs with deterministic
  idempotency keys, heartbeats, retries, and dead-lettering, so duplicate
  webhooks and worker crashes converge on one logical result.
- **Guardian API** (`hotato/fleet/api.py`): ingest → discover (trust preflight +
  scan) → human review + label → manifest-bound before/after experiment that
  **recommends a change for you to apply**, and refuses forged / same-audio /
  incomplete trials.

### CLI

```
hotato fleet init -w acme
hotato fleet agent add -w acme --agent-id support-bot --stack vapi --assistant-id asst_123
hotato fleet ingest -w acme --agent support-bot call.wav
hotato fleet discover -w acme --agent support-bot call.wav
hotato fleet review -w acme
hotato fleet label <candidate-id> --decision yield --reviewer you
hotato fleet experiment run -w acme --agent support-bot --trial-id t1 \
    --battery before/run.json --before before/ --after after/
hotato fleet status -w acme
```

Live clone/recapture/canary require a connected stack with credentials and a
tested rollback; this release recommends the change and leaves routing
production traffic as a manual, human-gated step.

## Synthetic perturbations (`hotato/synth.py`)
Deterministic acoustic transforms of real fixtures (resample, gain, noise,
leakage, channel invert, silence, onset offset, clip), each carrying its parent
hash, recipe, seed, and an explicit synthetic designation. Synthetic and real
stay on separate axes: only a real recapture raises the confidence tier.

## What fleet mode ships today
Evidence stays per-dimension, each one scored on its own lane, not folded into
a single fleet-wide score. Every contract needs human approval before it
exists. Live provider adapters (Vapi/Retell clone→apply→recapture) and canary
routing are gated on connected credentials and a tested rollback -- production
deploys stay a manual, credentialed step every time.


================================================================================
FILE: docs/START.md
================================================================================

# Start here: the guided first run

`hotato start --demo` is the zero-setup first run: one command, on your
machine. It sweeps the two bundled demo calls (the same recordings `hotato
demo` scores), creates and verifies one failure contract from them, writes
everything you need to see the flow, and then prints the exact next
commands.

```bash
hotato start --demo
```

## What it writes

Into the current directory (or `--dir DIR`):

- `hotato-sweep.json` -- the sweep result: every candidate timing moment across
  the two demo calls, ranked. This is an `analyze` result, so a `FILE#N` ref
  off it drives `hotato fixture promote` and `hotato card` unchanged.
- `hotato-sweep.html` -- a self-contained dashboard (embedded audio, no external
  assets) you can open in any browser.
- `hotato-no-single-threshold.svg` -- the threshold-funnel card: the demo battery
  misses an interruption **and** false-stops on a backchannel, so no single
  dial fixes both. See `docs/CARDS.md`.
- `contracts/demo-missed-interruption.hotato/` -- one failure contract,
  created from the sweep's missed-interruption candidate with `--expect
  yield` and verified immediately. See `docs/CONTRACTS.md`.

Everything is offline by construction: the demo pulls from packaged audio, and
the analyze/card/contract steps run on packaged audio and local files alone.

## The demo failure contract

`start --demo` goes beyond a ranked list of candidates: it runs the whole
loop once, on one of the bundled recordings:

```bash
hotato contract create --from-candidate hotato-sweep.json#2 --expect yield \
    --id demo-missed-interruption --out contracts
hotato contract verify contracts/
```

Candidate #2 in the bundled sweep is the missed-interruption call: the
agent talked over the caller instead of yielding. Scored against `--expect
yield`, it fails -- so `start --demo` prints:

```
verified contract: FAIL as expected -- the demo call missed the
interruption; a CI gate on this contract catches any change to the evidence
or policy -- catching the AGENT regressing requires a fresh recapture (see
docs/RECAPTURE.md)
(start --demo itself exits 0 because setup succeeded; run the next command to
see the contract's CI exit 1: hotato contract verify contracts/)
```

`hotato contract verify contracts/` re-scores the SAME bundled audio and
reports the SAME regression (exit code `1`), and a CI job wired to it fails
the same way if that bundled evidence or policy ever changes. On this frozen
recording, telling you whether a CURRENT deployed agent still has the bug
takes a fresh capture -- re-running the same caller stimulus against it and
verifying that; see
[`docs/RECAPTURE.md`](RECAPTURE.md). See [`docs/CONTRACTS.md`](CONTRACTS.md#two-lanes-what-verify-proves-depends-on-which-recording-you-feed-it)
for the full two-lane breakdown. Run it yourself:

```bash
hotato contract verify contracts/
```

## What it prints

The exact next commands, ready to copy:

1. **Save a candidate as a permanent regression test** (you choose the label --
   Hotato leaves that judgment to you):

   ```bash
   hotato fixture promote hotato-sweep.json#1 --expect <yield|hold> \
       --id my-first-fixture --out tests/hotato
   ```

2. **Run your fixtures in CI** (exits non-zero on a regression):

   ```bash
   hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio
   ```

3. **Render a shareable card** from any candidate:

   ```bash
   hotato card hotato-sweep.json#1 --out candidate.svg
   ```

4. **Re-verify the demo failure contract in CI** (or create your own from any
   candidate with `hotato contract create`):

   ```bash
   hotato contract verify contracts/
   ```

## Other modes

`--stack`, `--folder`, and `--stereo` are placeholders in this build. They print
the shipped command that does the job today:

- `--stack` -> `hotato sweep --stack <stack>` (connect once, sweep your own
  calls).
- `--folder` -> `hotato analyze <folder>` (scan a directory of recordings).
- `--stereo` -> `hotato run --stereo <call.wav>` (score one dual-channel call).

## Output and exit codes

`--format json` emits a machine object (the files written, the next commands,
and a `contract` block with the demo contract's id, expect, scorable, passed,
and `verified_fail_as_expected`) for an agent to drive.

- **0**: the guided first run completed (or a stubbed mode printed the shipped
  command to use instead). This holds even though the demo contract itself
  FAILS its policy -- `start --demo` finishing is a separate claim from the
  demo contract passing; `hotato contract verify contracts/` is the command
  that reports that (exit `1`, by design).
- **2**: usage error: no mode given, or `--dir` is not a directory.


================================================================================
FILE: docs/INVESTIGATE.md
================================================================================

# `hotato investigate`: one recording in, ranked candidate moments out

`hotato investigate` takes one recording -- a local dual-channel WAV, or a
live pull from a connected stack by call id -- and does four things, in order:
authenticates where the audio came from, runs the input-health / K6
verdict-eligibility gate, scans it for candidate turn-taking moments, and
prints the exact next command that turns each candidate into a signed,
CI-ready contract.

It is discovery and guidance: it surfaces candidates and prints the exact next
command. The one decision that matters --
which candidate is a bug, and whether the agent should have *yielded* or
*held* -- stays with you, and you make it by running the command this tool
prints:

```bash
hotato investigate label <candidate_ref> --expect yield|hold
```

Every step reuses a shipped primitive: audio-in is the same per-stack fetch
`hotato pull` uses; the gate is `hotato trust` in contract mode; discovery is
`hotato scan`; the label step wraps `hotato contract create --from-candidate`.

## The label comes from you

You label the expected behavior for the event:
`yield` means the agent should stop for the caller; `hold` means the agent
should keep speaking through a backchannel, noise, or acknowledgement. Hotato
then measures whether the timing matched that label. `"mhm"` and `"stop"` can
carry identical speech energy; no timing measurement tells them apart, so the
label is yours.

## Scan one recording

```bash
# a local dual-channel WAV you already have
hotato investigate call.wav

# or pull it live from a connected stack by call id
hotato investigate --stack vapi --call-id abc123
```

Give it either a local `SOURCE` path or `--stack STACK --call-id ID`, never
both. Connectable stacks: `vapi`, `twilio`, `retell`, `bland`, `elevenlabs`,
`synthflow`, `millis`, `cartesia`. LiveKit and Pipecat capture in your own
infrastructure -- run `hotato setup --stack <name>` and pass the resulting WAV
as `SOURCE`. `--allow-mono` permits a mono/mixed stack recording, but a summed
channel is degraded and is not candidate-eligible for separated scoring.

Useful flags: `--min-gap` (minimum response gap in seconds to surface, default
`2.0`), `--top` (how many top candidates to show and print label commands for,
default `10`; `0` shows all), `--caller-channel` / `--agent-channel`, and
`--state PATH` (default `.hotato/investigate-state.json`).

### What it reports

```
hotato investigate [run 1]: call.wav
  capture origin: operator-asserted local file (call.wav)
  input health: <trust recommendation>
  verdict path: eligible (a labeled event here can carry a yield/hold verdict)
  N candidate moment(s) (showing N):
    [1] t=42.18s <kind>  <durations>
        label: hotato investigate label .hotato/investigate-state.json#1 --expect yield|hold
  state remembered at: .hotato/investigate-state.json
```

The onset above is illustrative. Candidates are timing facts, ready for you to
label.

## Capture origin: tracked for every run

Every run records where the audio came from, one of three kinds:

- **`frozen_regression`** -- a previously-created hotato fixture clip (a sibling
  scenario file names this exact audio): a pinned regression, played back exactly.
- **`provider_pulled`** -- fetched just now from the stack's own recording API
  for a named call id. Stronger than an arbitrary file. For the stronger,
  signed, machine-verified claim, see [RECAPTURE.md](RECAPTURE.md).
- **`operator_asserted_local`** -- you handed hotato a local WAV path, taken at
  face value.

## The K6 verdict gate

Trust runs in contract mode -- the same stricter crosstalk/leakage bar
`hotato contract create` itself checks, since this command exists to produce a
contract. Two outcomes matter:

- **NOT SCORABLE** (mono, identical channels, a silent required channel): no
  candidates are scanned; the report names the reason, and the command exits
  `2`. Fix the input and re-run.
- **Verdict path REFUSED** (a suspected channel swap or crosstalk/leakage): the
  candidates below are still shown as timing facts, and a labeled event here
  carries a yield/hold verdict once you confirm the mapping with
  `--confirm-channels` or fix the crosstalk. See [TRUST.md](TRUST.md).

Note the two gates are different widths: `scan` runs whenever the input is
scorable, so candidates appear even when the verdict path is refused.

## State and candidate refs

State persists to `.hotato/investigate-state.json` (run-numbered, atomically
written, with a history log). It is written in the same shape `hotato analyze`
/ `hotato sweep` produce, so it is itself a valid `FILE#N` candidate ref:
`hotato fixture promote` and `hotato contract create --from-candidate` read it
directly. `#1` is the top-ranked candidate, `#2` the next, and so on.

## Label a candidate into a contract

`hotato investigate label` is the label step. Your `--expect` goes straight to
`hotato contract create --from-candidate`, which mints a signed label-record
bound to the exact decoded audio when a signing key is configured. Without a
signing key, the contract's `label_authority` floors at `asserted`.

```bash
hotato investigate label .hotato/investigate-state.json#1 \
    --expect yield \
    --max-talk-over 0.6 --max-time-to-yield 1.0 \
    --reviewer "$USER" --out contracts
```

This writes `contracts/<id>.hotato/`. The id defaults to a slug derived from
the source, onset, and label; pass `--id` to name it, `--force` to overwrite.
Clipping keeps `--pre` seconds before the onset (default `2.0`) and `--post`
after (default `6.0`); `--no-clip` keeps the full recording. If the verdict
path was refused, add `--confirm-channels` to carry the contract's verdict
through `contract verify`. Source basenames are redacted from the bundle by
default; pass `--include-identifiers` to keep them.

Building a contract (not a bare fixture) is deliberate: it carries the K6 trust
block, the CI policy, and the exact `hotato contract verify` command. That is
where investigate hands off to [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md) and
[CONTRACTS.md](CONTRACTS.md).

Scanning, the trust gate, and labeling run offline; audio stays on your machine
unless you explicitly pull it from your own stack ([THREAT-MODEL.md](THREAT-MODEL.md)).

## Exit codes

`hotato investigate`:

| code | meaning |
| --- | --- |
| `0` | the recording is candidate-eligible: trust + scan ran and candidates (if any) were persisted; a yield/hold VERDICT may still be refused (K6) -- see `verdict_status` |
| `2` | usage error (neither `SOURCE` nor `--stack/--call-id`, both given, a bad channel/`--min-gap` flag, a missing credential, or an unreadable state file), or the recording is NOT SCORABLE at all |

`hotato investigate label`:

| code | meaning |
| --- | --- |
| `0` | a signed, CI-ready contract was written from this candidate |
| `2` | usage error (a bad `--expect`, a bad candidate ref, an unresolved source recording, or an existing contract without `--force`), or the candidate turned out not scorable |

## See also

- [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md) -- the bad-call to CI-gate flow this feeds.
- [CONTRACTS.md](CONTRACTS.md) -- the failure-contract bundle.
- [TRUST.md](TRUST.md) -- the input-health / K6 scorability gate.
- [RECAPTURE.md](RECAPTURE.md) -- the stronger, signed fresh-recapture claim.
- [CONVERSATION-TEST.md](CONVERSATION-TEST.md), [SUITE-RUN.md](SUITE-RUN.md) -- the conversation-QA path.


================================================================================
FILE: docs/CONTRACTS.md
================================================================================

# Failure contracts

A failure contract turns ONE call moment into a portable, private,
vendor-neutral bundle: the audio, frame-level timing evidence, an
input-health report, a shareable card, a CI pass/fail policy, and the exact
commands to replay and re-verify it. It is the CI object: `hotato contract
verify` re-scores a directory of contracts and exits non-zero when one
regresses.

A contract's label always comes from a human call (`label_source` is frozen
to `"human"`); Hotato measures whether the recorded timing matched that
label, and `contract verify` re-measures the SAME recording later and
reports pass/fail.

> **A contract bundle contains call audio (`audio/event.wav`).** Do not
> commit a raw customer contract to a public repository -- treat it like any
> other recording of a caller. Use sanitized fixtures (synthetic or
> consent-cleared) for anything public, and put customer contracts in
> a private repository or controlled artifact storage instead.
> `--include-identifiers` additionally writes a source basename and
> candidate ref into `contract.json` and the card; leave it off by default
> for the same reason. That redaction covers metadata only -- the full
> audio-handling rules (what raw audio may contain, the limits of redaction,
> and the audio-free evidence summary for sharing proof minus the recording)
> are in [`SECURITY.md`](../SECURITY.md#audio-handling).

## Two lanes: what `verify` proves depends on which recording you feed it

| | `contract verify` on the frozen recording | `contract verify` on a fresh recapture |
| --- | --- | --- |
| **What it re-scores** | The SAME `audio/event.wav` the contract was created from | A new recording of the SAME stimulus against your CURRENT agent |
| **What a pass proves** | The evidence, the policy, and the scorer are still intact and still agree with the human label | The CURRENT agent's behavior on that stimulus still matches the label |
| **What a pass does NOT prove** | That the deployed agent's behavior has not changed since | Nothing extra -- this is the one that speaks to the live agent |
| **When it runs** | Every push, in the shipped `ci/github-action.yml` (`contract verify contracts/`) | Only when you recapture by hand or on a schedule; see [`docs/RECAPTURE.md`](RECAPTURE.md) |

A frozen-recording pass is necessary but not sufficient to know the agent bug
stayed fixed: it can only fail if someone edits the bundle's audio or policy,
because the recording never changes. Proving the CURRENT agent still yields
correctly requires re-running the same caller stimulus against it and
re-verifying the fresh capture, by hand, per
[`docs/RECAPTURE.md`](RECAPTURE.md). The two lanes are complementary: run the
frozen-recording gate on every push to catch evidence/policy drift for free,
and recapture periodically (or after an agent change) to catch what only a
fresh capture can.

## The bundle

`hotato contract create` writes one self-contained directory,
`<id>.hotato/`:

```
refund-cutoff-001.hotato/
  contract.json                      # the contract itself (schema hotato.contract.v1)
  audio/event.wav                    # the (clipped) two-channel recording, or the mono file
  evidence/
    frames.jsonl                     # per-frame timing evidence behind every measurement
    timeline.html                    # the to-scale caller/agent timeline, self-contained
    trust.json                       # the input-health (trust doctor) report
    card.svg                         # a shareable 1200x630 SVG card (redacted by default)
  traces/                            # empty until `hotato trace attach` (see docs/TRACE.md)
  source/
    call_metadata.json               # redacted-by-default: stack, category, expect
    stack_config_snapshot.json       # placeholder until populated by hand
  policy/verify.yaml                 # the SAME subset `hotato verify --policy` reads
  reports/
    initial.html                     # the full scored report at creation time
    after.html                       # placeholder until a fix is re-captured and verified
  provenance.json                    # who/when/how this contract was created
  ci/
    github-action.yml                # a weekly + on-push CI scaffold
    junit.xml                        # this ONE contract's JUnit result at creation time
```

Every path above is also recorded, machine-readable, in
`contract.json["bundle"]["paths"]`.

## Create

From a candidate a sweep or scan already surfaced:

```bash
hotato sweep --demo --format json > hotato-sweep.json
hotato contract create --from-candidate hotato-sweep.json#1 \
    --expect yield --id refund-cutoff-001 --out contracts
```

From a raw two-channel recording you already have:

```bash
hotato contract create --stereo bad-call.wav --onset 42.18 \
    --expect yield --id refund-cutoff-001 --out contracts \
    --max-talk-over 0.6 --max-time-to-yield 1.0
```

Both forms wrap the SAME round-trip guarantee `hotato fixture create` gives:
the moment is scored immediately, and a not-scorable input (the agent silent
at the onset, an unreadable file, a bad channel map) is refused with a clear
reason and exit code 2 -- no bundle is written. A single-channel
(mono) recording passed as `--stereo` is rejected the same way `fixture
create` rejects it: caller and agent cannot be told apart on one channel.

`--caller FILE --agent FILE` (two mono WAVs) is a third input form, scored
and clipped identically.

### Redaction

By default the bundle and the card hide a candidate ref and a source
recording's basename. Pass `--include-identifiers` to show them (in
`source/call_metadata.json`, `contract.json`, and `evidence/card.svg`).

### The opt-in diarized-mono path

A single-channel recording can still become a contract through the SAME
quality-gated diarizer front-end `hotato run --mono --diarize` uses:

```bash
hotato contract create --mono call.wav --diarize \
    --expect yield --id refund-cutoff-002 --out contracts
```

This keeps an indicative-only verdict in its tier: a `low`-confidence separation
tier carries `measurement.indicative_only: true` all the way into
`contract.json` and every renderer, and a `refuse`
tier (not two clean parties, extreme overlap, unstable segmentation, voices
too similar) is refused exactly like a plain mono file, with the specific
reason. Frame-level evidence (`evidence/frames.jsonl`, the to-scale timeline)
isn't produced for this path in this release; `evidence/timeline.html` states
that plainly and points at `evidence/trust.json`'s separation confidence
tier.

## Verify

```bash
hotato contract verify contracts/
hotato contract verify contracts/ --format json --junit contracts-junit.xml
hotato contract verify contracts/refund-cutoff-001.hotato --html verify.html
```

`DIR` is a contracts directory (every `*.hotato` subdirectory that carries a
`contract.json`) or one bundle directly. For each contract, `verify`
re-scores the SAME bundled audio against the SAME policy recorded in its own
`contract.json` -- this is what changes after an engine upgrade, a threshold
change, or a re-captured recording swapped into `audio/event.wav` -- and
reports pass/fail per contract and overall.

Exit codes are the CI contract: `0` every contract passes, `1` at least one
regressed (or is no longer scorable) OR at least one embedded assertion
deterministically FAILed (see below -- reported separately, but it still
contributes to this nonzero exit), `2` a usage error, an empty directory,
or a corrupt `contract.json`. `--junit` writes one `<testcase>` per contract
for a CI dashboard; the shipped `ci/github-action.yml` scaffold runs this on
push, on PR, and weekly, and publishes the JUnit file as an artifact.

Every text and HTML render of `verify` also prints, verbatim: *"This result
re-measures stored evidence. It does not test the current agent."* -- a
green CI run here means the evidence, policy, and scorer are still intact;
re-checking today's deployed agent needs the fresh-recapture lane above. See
[`docs/RECAPTURE.md`](RECAPTURE.md#claim-language-what-each-kind-of-evidence-lets-you-accurately-say)
for exactly what each kind of evidence lets you claim.

### Embedded assertions (optional)

A contract can carry its own `assertions` block (schema `hotato.contract.v1`
-- see `schema/contract.v1.json`), the same `{version, assertions}` document
`hotato assert` reads from an `assertions.yaml` file. When one is present,
`contract verify` evaluates it through the SAME `assert.v1` engine and reports
the result as a per-contract `assertions` field, a dimension kept separate
from the timing pass/fail in `summary`/`passed`. Pass
`--transcript FILE` (a plain JSON array of `{role, text, start, end}` turns,
or hotato's own `{"segments": [...]}` shape) to supply transcript context for
`phrase`/`pii`/`policy` assertions; `tool_call` assertions read the bundle's
own attached trace (`hotato trace attach`) if one exists. Missing context
(no `--transcript`, no attached trace) reports `INCONCLUSIVE`. A deterministic
assertion FAIL contributes to the batch's
nonzero exit code exactly like a timing regression; the batch result also
carries a separate `assertions_failed` count. See
[`schema/assert.v1.json`](../src/hotato/schema/assert.v1.json) for the result
shape and `hotato.assert_` for the five deterministic kinds.

## Inspect

```bash
hotato contract inspect contracts/refund-cutoff-001.hotato
hotato contract inspect contracts/refund-cutoff-001.hotato/contract.json --format json
```

## Pack / unpack

A bundle directory travels as one file:

```bash
hotato contract pack contracts/refund-cutoff-001.hotato
# -> contracts/refund-cutoff-001.hotato.pack (deterministic; a MANIFEST.sha256.json
#    of every member travels inside it)

hotato contract unpack contracts/refund-cutoff-001.hotato.pack --out contracts/refund-cutoff-001.hotato
# every member is verified against the packed sha256 manifest; any mismatch
# (a corrupt or tampered archive) is refused (exit 2) and nothing partial is
# left behind
```

Packing the SAME bundle directory twice produces byte-identical archives
(sorted member order, fixed timestamps, and every other value written into
each member's ZipInfo, including its `create_system` byte): a contract's
pack is a pure function of its bundle contents, deterministic for a fixed
hotato version. Byte-identical re-runs are verified in CI on Linux x86_64,
Python 3.10, 3.11, and 3.12; the same suite now also runs on macOS and
Windows in CI, not yet green -- see [VALIDATION.md](VALIDATION.md) Job 1.

### Security: unpack treats an archive as hostile input

A `.hotato` archive is meant to be sent between teams, so `contract unpack`
verifies everything about it before trusting a single byte. Before or during
extraction it refuses (exit 2) on any of the following, writing only inside a
scratch temp directory that's removed on any failure:

* path traversal (`..`), absolute paths, and Windows-style backslash /
  drive-letter paths (`C:\...`) in a member name;
* symlink members and encrypted members;
* duplicate member names;
* any member the archive carries that its own `MANIFEST.sha256.json` does
  not declare;
* more members than a legitimate bundle could plausibly need;
* a declared or measured decompressed size past the cap (default 512 MiB, set
  `HOTATO_CONTRACT_MAX_UNPACK_BYTES` or pass `--max-bytes` to raise it for a
  trusted archive) -- checked against the bytes measured live during
  extraction, beyond what the archive's own (untrusted) size metadata claims;
* a single member whose compression ratio is far beyond anything a legitimate
  bundle member produces (a zip-bomb signal), even if it is well under the
  total-bytes cap.

The existing sha256-per-member check (above) still runs on top of all of
this: a member can pass every hardening check above and still be refused for
not matching the packed manifest.

## CI

The shipped `ci/github-action.yml` is the minimal wiring:

```bash
uvx hotato contract verify contracts/ --junit contracts-junit.xml --format json > contracts-verify.json
```

Gate a pull request or a scheduled job on the process exit code; do not parse
stdout to decide pass or fail.

## What a contract proves, precisely

Hotato proves timing behavior against this explicit contract -- not
authorization, identity, compliance, or policy safety. `verify` reports
coincidence, not causation: a passing re-verify after a config change
coincides with the change, distinct from a controlled experiment. See
`docs/VALIDATION.md` and `docs/THREAT-MODEL.md`.

## Read more

- Proving the CURRENT agent, not just the frozen recording:
  [`docs/RECAPTURE.md`](RECAPTURE.md)
- The underlying regression-fixture primitive `contract create` wraps:
  [`docs/BAD-CALL-TO-CI.md`](BAD-CALL-TO-CI.md)
- Battery-scale before/after proof: [`docs/FIX-LOOP.md`](FIX-LOOP.md)
- Input-health (trust doctor): [`docs/TRUST.md`](TRUST.md) ·
  [`docs/TRUST-MATRIX.md`](TRUST-MATRIX.md)
- Diarized-mono scoring: [`docs/DIARIZE.md`](DIARIZE.md)
- Shareable cards: [`docs/CARDS.md`](CARDS.md)
- Attaching a voice trace (observability bridge): [`docs/TRACE.md`](TRACE.md) ·
  [`docs/OTEL.md`](OTEL.md)
- Audio-handling rules (what raw audio may contain, redaction limits,
  distribution): [`SECURITY.md`](../SECURITY.md#audio-handling)


================================================================================
FILE: docs/PYTEST.md
================================================================================

# Pytest: the fixture and the gate

Install hotato and the plugin registers itself automatically (a standard
`pytest11` entry point): zero `conftest.py` wiring, zero imports needed. It
adds one fixture and one opt-in session gate. Both score with the same engine as the CLI and return the same
envelope, the JSON result object every hotato surface emits.

```bash
pip install hotato
```

## The `hotato_score` fixture

Score a recording or a suite inside any test and assert on the envelope. Same
inputs as the CLI:

```python
def test_call_yields(hotato_score):
    env = hotato_score(stereo="call.wav", expect="yield")
    assert env["exit_code"] == 0
    assert all(e["verdict"]["passed"] for e in env["events"])

def test_split_channels(hotato_score):
    env = hotato_score(caller="caller.wav", agent="agent.wav", expect="yield")
    assert env["events"][0]["verdict"]["passed"]

def test_selftest_battery(hotato_score):
    env = hotato_score(suite="barge-in")
    assert env["exit_code"] == 0
```

The envelope is the standard machine shape (`schema_version` "1"): per event a
`verdict` (with `passed`, `did_yield`, `seconds_to_yield`, `talk_over_sec`), a
`signals` bus, a `fix` on every failure, and the `limits` block. Assert on
whichever measurement your test cares about.

## The `--hotato-suite` session gate

Opt in on the command line and the battery runs after your tests, printing a
summary and failing the whole session (exit 1) on a regression:

```bash
pytest --hotato-suite                    # default suite: barge-in
```

The bundled battery is a self-test of the harness. To gate on your own agent,
point the same flag at your own labelled scenario and audio directories:

```bash
pytest --hotato-suite \
  --hotato-suite-scenarios corpus/suites/gold/scenarios \
  --hotato-suite-audio corpus/suites/gold/audio
```

Any directory in the same scenario shape works, including your own labelled
calls (see `docs/SUITES.md` for the bundled tiers and `docs/SUBMITTING.md` for
building labelled fixtures from your own recordings).

## Where it fits

- The gate is one flag on a test run you already have, so turn-taking rides
  along with every `pytest` invocation, locally and in CI.
- The GitHub workflow (`docs/CI.md`) is the other ready-made gate: it scores on
  every pull request and posts a sticky results comment. Use either, or both.
- Everything runs offline; the plugin and its fixtures are deterministic and
  self-contained, so results stay stable run to run.


================================================================================
FILE: docs/SUITES.md
================================================================================

# Corpus suites: tiered, deterministic, explicit about what they prove

`corpus/suites/` ships four labelled scenario suites, 112 scenarios total,
every one synthetic shaped noise rendered deterministically from its own
labelled timings (seed `sha256(scenario_id)`). The timings are the ground
truth. Synthetic audio is the floor: these suites prove the scorer runs end to
end and catch regressions; validity on your system comes from your own labelled
calls (`docs/SUBMITTING.md`).

## The four suites

| Suite | Scenarios | Conditions | Expected exit |
|---------------------|-----------|-----------------------------------------------------------------------|---------------|
| `silver` | 40 | clean, 16 kHz, default noise floor | `0` |
| `silver-defects` | 16 | clean conditions, deliberate defect renders | `1` |
| `gold` | 40 | hard conditions: noise floors, 8 kHz, gain extremes, echo, edge timings, endurance | `0` |
| `gold-defects` | 16 | hard-condition defect renders, plus two labelled capture-defect cases | `1` |

The scenario families span the behaviours that matter on a real call: hard
interruptions (onset, speed, duration, resume), backchannels, short
acknowledgements like "mhm" that the agent should talk through (varied in
position, density, repeats, and length), double-talk, one-word interruptions,
stutter onsets, multi-turn exchanges, resume-then-reinterrupt, and latency
prompts.
`corpus/suites/manifest.json` is the machine-readable inventory: per suite the
family and category breakdown, sample rates, and the expected exit code.

## Defect suites fail by design

Every scenario in `silver-defects` and `gold-defects` is rendered to fail on
its labelled axis: an agent that keeps talking through a real interruption, or
stops for a backchannel, or misses its latency budget. A defect suite that
exits `1` is the scorer catching what it claims to catch; a defect suite that
exits `0` would be a bug. This is the negative control the positive suites
need to mean anything.

## Run a suite

Any suite is just a scenarios directory plus an audio directory:

```bash
hotato run --suite barge-in \
  --scenarios corpus/suites/gold/scenarios \
  --audio corpus/suites/gold/audio
```

The same directories plug into every other surface: `hotato report ... --out
report.html` for the visual report, `hotato export ... --out research/` for
CSVs, and `pytest --hotato-suite --hotato-suite-scenarios ... --hotato-suite-audio ...`
for the session gate (`docs/PYTEST.md`).

## Deterministic builder

The suites are generator-built end to end, and the generator is the proof:

```bash
python3 corpus/suites/build_suites.py           # rebuild in place
python3 corpus/suites/build_suites.py --check   # regenerate to a temp dir, byte-compare
```

`--check` regenerating byte-identical output is the reproducibility guarantee:
the audio on disk is exactly what the labelled timings say it is, deterministic
for a fixed hotato version (byte-identical re-runs verified in CI on Linux
x86_64, Python 3.10, 3.11, and 3.12; also now checked, not yet green, on macOS
and Windows -- see `.github/workflows/tests.yml`, jobs `portability` and
`determinism`). CI can run it as a drift gate.

## Additive scenario classes

`corpus/classes/` ships four small, deterministic classes on top of the four
suites above, built the same way (synthetic shaped noise, seed
`sha256(scenario_id)`, `--check` byte-compares a rebuild): `mid-utterance-pause`
(a multi-second thinking gap mid-turn, distinct from a true turn end), `backchannel-multilingual`
(short non-English acknowledgement tokens, since Hotato's energy-based VAD
treats them the same as English ones), `noise-hold` (sustained background
presence, distinct from a brief backchannel, that the agent should hold through), and
`telephony-degraded` (an existing gold scenario re-rendered through G.711
mu-law plus a fixed packet-loss schedule, to prove the verdict is stable
across codec degradation). Kept separate from `corpus/suites/` because
`mid-utterance-pause` needs a non-default `turn_end_silence_sec` beyond what the
generic suite tests apply. Full per-class detail: `corpus/classes/README.md`.

## Against a live stack

To run scenario audio through a real voice stack and score what comes back,
see [`docs/BENCHMARK-STACKS.md`](BENCHMARK-STACKS.md). Measurement-error
methodology for the scorer itself is in [`docs/BENCHMARK.md`](BENCHMARK.md).

## Contribute scenarios

New synthetic families are welcome through the normal PR path
(`CONTRIBUTING.md`). The highest-value contribution stays a real, consented,
labelled call: the walkthrough is [`docs/SUBMITTING.md`](SUBMITTING.md) and the
[corpus-submission issue form](https://github.com/attenlabs/hotato/issues/new?template=corpus_submission.yml)
opens the intake.


================================================================================
FILE: docs/CI.md
================================================================================

# CI: gate a PR on turn-taking

hotato scores turn-taking timing from call recordings, so a pull request can
carry a running score and fail when the agent gets slower to stop talking for
an interrupting caller. The workflow runs offline with zero extra dependencies.

## Drop it in

Copy [`.github/workflows/hotato.yml`](../.github/workflows/hotato.yml) into your
repository at the same path. That is the whole setup. On the next pull request it
will:

- install the package with `pip install .`
- score the bundled barge-in suite to a JSON envelope
- render a Markdown summary with [`scripts/pr_comment.py`](../scripts/pr_comment.py)
- post or update one sticky comment (found by a hidden marker, so it stays a single comment across runs)
- fail the job on any regression

The sticky comment shows a pass/fail line, a per scenario table (expect, yielded,
time to yield, talk over, result), and a short regressions section.

The job needs `pull-requests: write` to post the comment. The workflow already
requests it; if your org restricts the default `GITHUB_TOKEN`, allow that scope.

## Point it at your own recordings

The bundled suite is a self-test: it scores frozen synthetic fixtures, proving
the harness itself works. The strongest gate for your agent is a suite of
your OWN bad moments, pinned as fixtures with `hotato fixture create` and run
with `hotato run --scenarios DIR --audio DIR`; the full loop from one bad call
to this gate is [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md). Alternatively, replace
one step, `Score turn-taking (head)`, with your own capture and score:

1. play each corpus `*.caller.wav` into your agent
2. record your agent's reply
3. score the pair, caller on channel 0 and your agent on channel 1:

```bash
hotato run --stereo your_call.wav --expect yield --format json --no-fail > head.json
```

The envelope shape and exit codes are identical, so the render, comment, and gate
steps stay exactly as they are. Keep `--no-fail` on the score step so the comment
still posts on a regression; the true pass/fail lives in the envelope's
`exit_code`, which the `Fail on regression` step reads.

## Deltas against the base branch

When the workflow can install and score the base branch, it runs the same suite
there in an isolated venv and uses it as the baseline. Any scenario where the
overlap grew (talk-over up) or the agent stopped later (time to yield up) is
listed under Regressions with the delta. This step is best effort: if it cannot
run, the comment falls back to the current pass/fail table and the gate still
holds.

## Render a comment yourself

`scripts/pr_comment.py` is stdlib only and reads the same JSON the CLI emits, so
you can preview the comment locally:

```bash
hotato run --suite barge-in --format json | python3 scripts/pr_comment.py
```

Add `--base base.json` to include deltas against a saved baseline run.

## The other ready-made gate: pytest

If your CI already runs pytest, one flag adds the same regression gate to that
run instead: `pytest --hotato-suite` scores the battery after your tests and
fails the session on a regression. Point it at your own labelled sets with
`--hotato-suite-scenarios` and `--hotato-suite-audio`. Details:
[`PYTEST.md`](PYTEST.md). For richer artifacts on a gate failure,
`hotato report` renders the visual report and `hotato team` tracks the trend
across runs ([`REPORTS.md`](REPORTS.md)).


================================================================================
FILE: docs/MCP.md
================================================================================

# MCP: the hotato server

hotato ships an MCP server, `hotato-mcp`, that speaks MCP over stdio and
exposes twelve tools: one scoring tool, `voice_eval_run`, which returns the
identical JSON envelope (`schema_version` "1") the CLI emits, plus eleven fleet
tools. Eight read, verify, and propose over a local fleet workspace; three are
clone-scoped actions that recompute and hand the deploy decision back to you.
Everything, including audio, runs and stays on your machine.

Every tool response carries a uniform control envelope: four keys ride on every
response (pure reads included) so an autonomous caller parses one shape:
`evidence_status` (or null for a pure read with no verdict), `refusal_reason`
(or null), `artifact_digests` (a list, or `[]`), and
`pending_irreversible_action` (the exact human-gated action still pending, e.g.
deployment approval, or null).

## Run it (zero-install)

```bash
uvx --from "hotato[mcp]" hotato-mcp
```

**Common mistake:** `uvx hotato-mcp` (no `--from`) FAILS. uv then looks for a
package literally named `hotato-mcp` on PyPI, which does not exist; the
console script `hotato-mcp` lives inside the `hotato` distribution, installed
with its `mcp` extra. If you (or an agent) just tried the bare form and got a
"package not found" error, retry with `--from "hotato[mcp]"` exactly as
written above. If hotato is already installed in your environment,
`python -m hotato.mcp_server` also works and needs no extra invocation syntax.

## Add it to a client

Every block below is copy-paste exact; only the outer key (client-specific)
differs. All three use the same command and args.

### Claude Desktop

Edit `claude_desktop_config.json` (Settings -> Developer -> Edit Config):

```json
{
  "mcpServers": {
    "hotato": {
      "command": "uvx",
      "args": ["--from", "hotato[mcp]", "hotato-mcp"]
    }
  }
}
```

### Cursor

Project-scoped: `.cursor/mcp.json` in your repo root. User-scoped:
`~/.cursor/mcp.json`.

```json
{
  "mcpServers": {
    "hotato": {
      "command": "uvx",
      "args": ["--from", "hotato[mcp]", "hotato-mcp"]
    }
  }
}
```

### Codex CLI

Edit `~/.codex/config.toml`:

```toml
[mcp_servers.hotato]
command = "uvx"
args = ["--from", "hotato[mcp]", "hotato-mcp"]
```

## The scoring tool: `voice_eval_run`

Same two input modes as the CLI, all parameters optional beyond the mode you
pick:

| Parameter | Type | Default | Meaning |
| --- | --- | --- | --- |
| `stereo` | str | None | two-channel WAV path |
| `caller` | str | None | mono caller WAV (with `agent`) |
| `agent` | str | None | mono agent WAV (with `caller`) |
| `suite` | str | None | `"barge-in"` to run the bundled battery |
| `stack` | str | `"generic"` | livekit, pipecat, vapi, or generic |
| `expect` | str | `"yield"` | `"yield"` or `"hold"` |
| `onset_sec` | float | None | caller onset hint |
| `caller_channel` | int | 0 | caller channel index |
| `agent_channel` | int | 1 | agent channel index |
| `max_talk_over_sec` | float | None | pass threshold |
| `max_time_to_yield_sec` | float | None | pass threshold |
| `report_path` | str | None | also write the HTML report here; the envelope then carries `report_path` (absolute) |

Pass exactly one input mode: `stereo`, OR `caller` + `agent` together, OR
`suite`. Mixing modes (or passing none) returns a structured error you can
parse programmatically, with a stable `error_code` and message; see below.

## Output

On success, the identical envelope the CLI emits for `--format json`, schema
at [`https://hotato.dev/schema/envelope.v1.json`](https://hotato.dev/schema/envelope.v1.json)
(`schema_version` "1", additive-only: new keys may appear, the documented
core never changes).

Every expected failure (a missing / mono / mismatched / not-found file, an
unknown suite, an ambiguous input mode, or a well-formed input with no
scorable event) comes back as a structured error object, `ok: false` with a
stable `error_code` and an actionable `message`, schema at
[`https://hotato.dev/schema/error.v1.json`](https://hotato.dev/schema/error.v1.json).
The MCP tool and the CLI share this one error shape, so a caller (human or
agent) parses one contract across both surfaces.

## The fleet tools

Eleven tools drive the local, self-hosted fleet control plane
([`GUARDIAN-FLEET.md`](GUARDIAN-FLEET.md)). They read and reason over a
workspace, surfacing findings for a human to label and deploy.

| Tool | Scope | Does |
| --- | --- | --- |
| `fleet_status` | read-only | workspace counts + job-queue stats |
| `candidate_list` | read-only | top candidate moments awaiting a human label |
| `candidate_inspect` | read-only | one candidate's detail: onset, measured components (severity/input_health/recurrence/novelty/covered_by_contract), and trust findings |
| `contract_list` | read-only | contracts in a workspace |
| `trial_explain` | read-only | a recorded trial's verdict, evidence tier, and any pending human-gated action |
| `experiment_status` | read-only | a trial's current verdict, evidence tier, recommendation, and manifest hash |
| `artifact_verify` | read-only | verifies a contract bundle's authenticity + evidence on its own terms |
| `experiment_propose` | read-only | a bounded variant set with expected effects, ready for a human to clone, apply, or deploy |
| `experiment_create` | clone-scoped | precommit a trial manifest from a committed battery BEFORE any capture; capture and deploy stay separate, later steps |
| `experiment_run` | clone-scoped | recompute a before/after trial offline, entirely within the clone, and record a recommendation |
| `clone_cleanup` | clone-scoped | delete a STAGING clone an experiment created, scoped entirely to that clone |

## More

- Python API and the shared error contract: [`API.md`](API.md)
- What the envelope measures and the scope/ceiling: the top-level
  [`README.md`](../README.md) and [`METHODOLOGY.md`](../METHODOLOGY.md)
- Machine-readable index of every command (CLI and MCP): [`llms.txt`](../llms.txt)


================================================================================
FILE: docs/API.md
================================================================================

# Python API reference

Everything the CLI, the pytest plugin, and the MCP server do is a plain Python
call. The core is stdlib only, fully offline, and deterministic: the same audio
and config always produce the same envelope. Import and score:

```python
from hotato.core import run_single, run_suite

env = run_single(stereo="call.wav", expect="yield")
env = run_suite()  # bundled 8-scenario battery
```

The top-level package re-exports the essentials:
`from hotato import run_single, run_suite, LIMITS, SUITE_ID, __version__`.

All scoring functions take keyword arguments only.

## The envelope

`run_single` and `run_suite` return the same machine-readable dict
(JSON Schema: `src/hotato/schema/envelope.v1.json`):

```python
{
  "tool": "hotato",
  "schema_version": "1",
  "mode": "single" | "suite",
  "stack": "generic",            # normalized stack label
  "offline": True,
  "engine": {"name", "version", "upstream"},
  "limits": {...},               # scope and ceiling, hotato.core.LIMITS
  "summary": {"events", "passed", "failed", "regression"},
                                 # plus additive "not_scorable" (count) when
                                 # at least one event could not be judged
  "events": [...],               # one dict per scored event, below
  "fix_map": [...],              # one entry per failing event with a fix
  "funnel": {...} | None,        # systemic pointer, fires only when both axes fail
  "exit_code": 0 | 1,            # 1 when any scorable event failed; the CLI
                                 # process exits 2 for a single recording that
                                 # is not scorable (see Exit codes below)
  "suite": "barge-in",           # run_suite only
}
```

Each event:

```python
{
  "event_id": str,               # file basename or scenario id
  "scenario_id": str | None,
  "title": str | None,
  "category": str | None,        # e.g. "should_yield"
  "expected_yield": bool,
  "verdict": {
    "passed": bool,
    "did_yield": bool,
    "seconds_to_yield": float | None,
    "talk_over_sec": float,
    "reasons": [str],            # failure reasons, empty on pass
  },
  "measurements": {
    "caller_onset_sec": float,
    "agent_talking_at_onset": bool,
    "hop_sec": float,
    "notes": str,
  },
  "signals": {                   # namespaced signal bus, additive
    "barge_in": {"did_yield", "time_to_yield_sec", "talk_over_sec"},
    "latency": {"response_gap_sec", "premature_start_sec"},
    "echo": {                    # every event; cross-channel coherence,
                                 # deterministic, computed in hotato's own
                                 # layer (hotato/echo.py), never _engine
      "coherence", "lag_sec", "echo_suspected",
    },
    "resume": {                  # only on events where the agent yielded
      "resumed", "resume_gap_sec", "restart_suspected",
    },
  },
  "fix": None | {                # set on failing events
    "fix_class": "config" | "engagement-control",
    "title": str, "detail": str,
    "knob": str | None, "pointer": str | None,
  },
}
```

An event that lacks the input to be judged (a silent caller channel with no
onset label, or a should-yield expectation with the agent silent at onset)
additionally carries `"scorable": False` and a plain `"not_scorable_reason"`.
It's reported as an input problem: it sits outside `passed`/`failed` and
outside the fix router and the funnel. Envelopes for valid recordings are
byte-identical to before these keys existed.

## hotato.core

### run_single

```python
run_single(
    *,
    stereo: str | None = None,        # two-channel WAV path
    caller: str | None = None,        # mono WAV path (with agent=)
    agent: str | None = None,         # mono WAV path (with caller=)
    caller_channel: int = 0,
    agent_channel: int = 1,
    onset_sec: float | None = None,   # caller onset hint, seconds from start
    expect: str = "yield",            # "yield" or "hold" (backchannel)
    stack: str | None = None,         # livekit | pipecat | vapi | generic
    max_talk_over_sec: float | None = None,
    max_time_to_yield_sec: float | None = None,
    echo_gate: bool = False,          # hold an echo-suspected yield out of
                                      # the verdict (scorable: false) instead
                                      # of counting it as a clean pass
    cfg: ScoreConfig | None = None,
) -> dict
```

Scores one recording and returns the envelope. Provide either `stereo` or both
`caller` and `agent`. `expect="hold"` means the caller's speech is a
backchannel, a short acknowledgement like "mhm", and a correct agent keeps
talking through it. The two `max_*` thresholds tighten the pass criteria.
`echo_gate` is opt-in and off by default; `signals.echo` is always computed
and reported either way. Malformed or truncated WAVs raise a clean `ValueError` with the
ffmpeg export line to fix them.

### run_suite

```python
run_suite(
    *,
    suite: str = "barge-in",          # the only suite id (SUITE_ID)
    stack: str | None = None,
    scenarios_dir: str | None = None, # your scenario JSON labels
    audio_dir: str | None = None,     # your recordings, <scenario-id><suffix>
    suffix: str = ".example.wav",
    caller_channel: int = 0,
    agent_channel: int = 1,
    echo_gate: bool = False,          # see run_single
    cfg: ScoreConfig | None = None,
) -> dict
```

Runs a labelled battery and returns the envelope with a `suite` key. Defaults
to the bundled 8-scenario battery shipped inside the package, zero external
files. Point `scenarios_dir` and `audio_dir` at your own labelled set (for
example `corpus/suites/gold/scenarios` and `.../audio`). Suite audio must be
two-channel.

### dump_frames_for_input

```python
dump_frames_for_input(
    *,
    stereo: str | None = None,
    caller: str | None = None,
    agent: str | None = None,
    caller_channel: int = 0,
    agent_channel: int = 1,
    onset_sec: float | None = None,
    cfg: ScoreConfig | None = None,
) -> dict
```

The per-frame evidence behind every reported number: each channel's dBFS, VAD
activity, threshold, and noise floor, plus a self-describing `config` block.
Every reported signal is re-derivable by hand from this dump.

### LIMITS and SUITE_ID

`hotato.core.LIMITS` is the scope dict embedded in every envelope
(method, ceiling, best input, what it does not do). `SUITE_ID` is `"barge-in"`.

### ScoreConfig

Every threshold is an exposed parameter:

```python
from hotato._engine.score import ScoreConfig
from hotato._engine.vad import VADParams

cfg = ScoreConfig(
    frame_ms=20.0, hop_ms=10.0,
    yield_hangover_sec=0.20,       # agent quiet this long = yielded
    max_search_sec=3.0,            # yield search window after onset
    caller_proximity_sec=0.5,
    turn_end_silence_sec=0.20,
    premature_tolerance_sec=0.05,
    onset_min_run_sec=0.05,
    agent_onset_lookback_sec=0.10,
    caller_vad=VADParams(),        # rel_db=15.0, abs_gate_db=-60.0,
    agent_vad=VADParams(),         # hangover_sec=0.15, noise_percentile=0.10,
)                                  # dyn_margin_db=22.0, backend="energy"
```

`VADParams.backend` is `"energy"` (the deterministic reference behind every
published number) or `"neural"` (an optional Silero VAD cross-check via
`pip install 'hotato[neural]'`; without the extra it raises a clean
`BackendUnavailable`, surfacing the missing dependency immediately).

## hotato.report

Self-contained visual reports scored from the same measurements. All three
functions accept the full scoring parameter set (`stereo`, `caller`, `agent`,
`suite`, `scenarios_dir`, `audio_dir`, `suffix`, `caller_channel`,
`agent_channel`, `onset_sec`, `expect`, `stack`, `max_talk_over_sec`,
`max_time_to_yield_sec`, `cfg`) as keyword arguments.

```python
build_report_html(*, base: dict | None = None,
                  base_label: str | None = None, **kwargs) -> (str, dict)
build_report_md(*, base: dict | None = None,
                base_label: str | None = None, **kwargs) -> (str, dict)
write_report(path: str, fmt: str = "html", **kwargs) -> dict
```

`build_report_html` scores the input and returns `(html, envelope)`: one
self-contained file, inline CSS and SVG, per-event timelines, analytics, a
frame inspector, and print CSS for PDF. `build_report_md` mirrors it as
Markdown tables. `write_report` builds in `fmt` (`"html"` or `"md"`), writes to
`path`, and returns the envelope. Pass `base` (a previous envelope dict, for
example loaded from `hotato run --format json` output) to render per-scenario
regression deltas; `base_label` names it in the page.

```python
from hotato.report import write_report

env = write_report("report.html", suite="barge-in", stack="livekit")
```

## hotato.aggregate

Team mode: many run envelopes, one trend view.

```python
load_run_dir(dirpath: str, order: str = "mtime") -> dict
    # {"runs": [{"file", "path", "mtime", "env"}], "skipped": [...], "order"}
    # order: "mtime" (oldest first) or "name" (numeric prefix = explicit index)

aggregate_runs(runs: list, order: str = "mtime",
               skipped: list | None = None) -> dict
    # team envelope: kind "team-aggregate", runs, events_total,
    # talk_over_sec / seconds_to_yield distribution summaries (mean/median/p90),
    # pass_rate {latest, first, mean, direction}, pass_rate_over_time,
    # failure_classes, most_common_failure_class, skipped, exit_code 0.
    # Raises ValueError with fewer than 2 runs; each trend point corresponds
    # to one input run.

build_team_section_html(agg: dict) -> str   # embeddable section
build_team_page_html(agg: dict) -> str      # full self-contained page
```

```python
from hotato.aggregate import load_run_dir, aggregate_runs, build_team_page_html

loaded = load_run_dir("runs/")
agg = aggregate_runs(loaded["runs"], order=loaded["order"],
                     skipped=loaded["skipped"])
html = build_team_page_html(agg)
```

## hotato.export

Research-grade flat files from the same scorer.

```python
run_export(
    *,
    out_dir: str,
    # plus the full scoring parameter set: stereo, caller, agent,
    # caller_channel, agent_channel, onset_sec, expect, stack, suite,
    # scenarios_dir, audio_dir, suffix, max_talk_over_sec,
    # max_time_to_yield_sec, cfg
) -> dict   # {"env", "events_rows", "frames_rows", "paths"}
```

Writes `events.csv` (one row per event, columns in
`hotato.export.EVENT_COLUMNS`), `frames.csv` (one row per VAD frame, columns in
`FRAME_COLUMNS`), and `envelope.json` into `out_dir` (created if missing).
Column meanings are documented in `#` comment lines at the top of each CSV.
An empty cell marks a value that could not be derived from the input; every
filled cell is a direct measurement.

## hotato.stackbench

Identical scenarios, your stack, comparable result files: every number is a
measurement of the recordings you provide.

### run_stackbench

```python
run_stackbench(
    *,
    stack: str,                       # one of BENCH_STACKS:
                                      # vapi | twilio | livekit | pipecat | generic
    recordings_dir: str,              # <scenario-id><suffix> WAVs
    scenarios_dir: str | None = None, # default: bundled battery
    suffix: str | None = None,        # default: auto-detected
                                      # (.wav, .stereo.wav, .example.wav)
    caller_channel: int = 0,
    agent_channel: int = 1,
    cfg: ScoreConfig | None = None,
) -> dict
```

Returns a result dict (`kind: "stack-benchmark"`) with the envelope fields plus
`config`, `scenarios {total, captured, not_captured}`, and `provenance`.
Scoring is `run_suite` unchanged. Scenarios with no matching recording are
listed under `not_captured`, left out of both the scoring and the failure
counts. The timestamp derives from input file mtimes, so the same inputs
reproduce the same result file.

### load_result, compare_results, render_comparison_md

```python
load_result(path: str) -> dict
    # loads and validates one result JSON; anything else is a clean ValueError

compare_results(inputs: Sequence[tuple[str, dict]]) -> dict
    # inputs: (path, loaded_result) pairs, at least two.
    # Compares the intersection of scenarios scored in EVERY input;
    # the rest is listed under "skipped". Deltas are signed differences
    # against the FIRST input. Returns kind "stack-benchmark-comparison"
    # with inputs, compared, skipped, per_scenario, medians.

render_comparison_md(cmp_env: dict) -> str
    # the comparison as Markdown tables
```

```python
from hotato.stackbench import load_result, compare_results, render_comparison_md

cmp = compare_results([(p, load_result(p)) for p in ("a.json", "b.json")])
print(render_comparison_md(cmp))
```

## Pytest fixture

Installed automatically via the `pytest11` entry point (or load explicitly with
`-p hotato.pytest_plugin`). Inert unless used.

```python
def test_call_yields(hotato_score):
    env = hotato_score(stereo="call.wav", expect="yield")
    assert env["summary"]["regression"] is False
    assert env["events"][0]["verdict"]["seconds_to_yield"] < 1.0
```

`hotato_score(**kwargs)` takes the same keyword arguments as `run_single`; pass
`suite="barge-in"` (plus `run_suite` keywords) to score a battery instead. It
returns the envelope; write your own assertions against it.

Session gate flags: `pytest --hotato-suite` runs the battery after your tests
and fails the session (exit 1) on a regression; `--hotato-suite-scenarios DIR`
and `--hotato-suite-audio DIR` point it at your own labelled set. Detail:
`docs/PYTEST.md`.

## MCP tool

`hotato-mcp` (or `python -m hotato.mcp_server`) speaks MCP over stdio and
exposes nine tools. Its scoring tool, `voice_eval_run`, returns the identical
envelope the CLI emits; the other eight read, verify, and propose over a local
fleet workspace (`fleet_status`, `candidate_list`, `contract_list`,
`trial_explain`, `artifact_verify`, `experiment_propose`, `experiment_run`,
`clone_cleanup`) and are documented in [`MCP.md`](MCP.md). Install:
`uvx --from "hotato[mcp]" hotato-mcp`.

The parameters below are `voice_eval_run`'s, all optional:

| Parameter | Type | Default | Meaning |
| --- | --- | --- | --- |
| `stereo` | str | None | two-channel WAV path |
| `caller` | str | None | mono caller WAV (with `agent`) |
| `agent` | str | None | mono agent WAV (with `caller`) |
| `suite` | str | None | `"barge-in"` to run the bundled battery |
| `stack` | str | `"generic"` | livekit, pipecat, vapi, or generic |
| `expect` | str | `"yield"` | `"yield"` or `"hold"` |
| `onset_sec` | float | None | caller onset hint |
| `caller_channel` | int | 0 | caller channel index |
| `agent_channel` | int | 1 | agent channel index |
| `max_talk_over_sec` | float | None | pass threshold |
| `max_time_to_yield_sec` | float | None | pass threshold |
| `report_path` | str | None | also write the HTML report here; the envelope then carries `report_path` (absolute) |

## Exit codes and errors

Envelopes carry `exit_code`: 0 all scorable events passed, 1 regression.
`hotato.core.process_exit_code(env)` maps the finished envelope to the CLI
process exit: a single-recording run whose event is not scorable (silent
caller with no onset label, or agent silent at onset; the reason is in
`not_scorable_reason`) exits 2, because that is an input problem, not an
agent verdict. Suite runs count such events in `summary.not_scorable` and
keep their 0/1 semantics. Malformed input (bad WAV, out-of-range channel,
negative onset, unknown suite or stack) raises `ValueError`, which the CLI
surfaces as exit code 2. Only a file the scorer can fully read gets scored.


================================================================================
FILE: docs/SUBMITTING.md
================================================================================

# Submitting a recording

One consented, de-identified, human-labeled real call makes this eval more
credible. This is the whole path: record, label, validate, submit, and what
happens at intake.

The scorer measures speech energy over time. Your human label supplies the
meaning: which onset was a real bid for the floor, and whether a well-behaved
agent yields there. Energy is not intent, so the label is the ground truth,
and the label is what you are contributing.

## 1. Record dual-channel

Caller on one channel, agent on the other, separated at capture: the two legs
of a SIP bridge, or two streams kept fully separate. This separation makes overlap
a fact of the recording, exact to the sample. Save a WAV at the call's native
sample rate (8000 Hz telephony, 16000 Hz wideband).

Consent, PII, and PHI rules are normative in
[`CORPUS-GOVERNANCE.md`](CORPUS-GOVERNANCE.md): documented consent from every
audible party, identifiers redacted with same-duration tone or silence so the
timing survives, no PHI ever. Read it before you record; it includes a
reusable release paragraph.

## 2. Label it

The label is a JSON file next to your WAV: the bundled scenario shape plus
provenance and attestation. Spec of record:
[`corpus/label.schema.json`](../corpus/label.schema.json). Worked example:
[`corpus/examples/sample-contribution.json`](../corpus/examples/sample-contribution.json).

Minimal example:

```json
{
  "id": "call-021-address-change-interrupt",
  "title": "Caller interrupts the shipping recap to change the address",
  "category": "should_yield",
  "source_type": "real-call",
  "audio": "call-021-address-change-interrupt.wav",
  "channels": { "caller_channel": 0, "agent_channel": 1 },
  "sample_rate": 8000,
  "duration_sec": 9.2,
  "caller_onset_sec": 3.15,
  "expected": {
    "yield": true,
    "max_time_to_yield_sec": 0.7,
    "max_talk_over_sec": 0.8
  },
  "license": "MIT",
  "attestation": {
    "contributor": "Your Name <you@example.com>",
    "consent_on_file": true,
    "pii_removed": true,
    "no_phi": true,
    "right_to_release_mit": true
  }
}
```

For a hold case (the caller backchannels "mm-hm" and a good agent keeps
talking): set `category` to `should_not_yield`, `expected.yield` to `false`,
and both `max_*` bounds to `null`.

Label only what you can defend by hand. `caller_onset_sec` is required;
`reference_render` segment timings are welcome wherever you have them, and
the harness reports error for exactly the signals you supply.

## 3. Validate locally

```bash
python3 corpus/validate.py your_label.json
```

The WAV resolves next to the label via its `audio` field. Pass the audio path
as a second argument if it lives elsewhere:

```bash
python3 corpus/validate.py your_label.json your_recording.wav
```

PASS (exit code 0) means the pair conforms: fields well-typed, category and
expected bounds consistent, timings in range, attestation affirmed, and the
audio a readable WAV with two or more channels that matches the label. A
human still reviews consent and PII before anything merges.

## 4. Submit

Two doors, one intake:

- **Issue form.** Open
  ["Corpus submission: labeled recording"](https://github.com/attenlabs/hotato/issues/new?template=corpus_submission.yml).
  Attach the WAV as a .zip (GitHub accepts .zip attachments; a raw .wav
  upload is rejected) or link it, and paste your label JSON in the
  description.
- **Pull request.** Add the label and WAV under [`corpus/`](../corpus/), for
  example `corpus/call-021-address-change-interrupt.json` plus its `.wav`.
  State the source type in the PR body and confirm the attestation, per
  [`CONTRIBUTING.md`](../CONTRIBUTING.md).

## What maintainers do at intake

1. **Validate.** Re-run `corpus/validate.py` on the pair and check the label
   against [`corpus/label.schema.json`](../corpus/label.schema.json).
2. **Dedupe.** Hash the audio and compare call details against existing
   clips. The corpus stays small and high-integrity; every clip earns its
   place.
3. **Normalize.** Slug the `id`, align filenames, confirm the channel map and
   declared sample rate. Timing is preserved exactly: any audio edit leaves
   the label's timing untouched.
4. **Add to a suite.** Register the clip so the benchmark harness scores it
   and reports millisecond error distributions and the yield confusion matrix
   ([`BENCHMARK.md`](BENCHMARK.md)).
5. **Credit.** `attestation.contributor` is the credit of record, and
   contributors are named in the changelog when their clip lands.

Removal requests are honored in the next release, per
[`CORPUS-GOVERNANCE.md`](CORPUS-GOVERNANCE.md).


================================================================================
FILE: docs/CORPUS-GOVERNANCE.md
================================================================================

# Corpus governance

How this project assembles, labels, and reports on a small corpus of **real**
labeled voice-agent calls that augments the bundled synthetic fixtures.

This document is normative. To contribute real audio, follow it. To understand
what the numbers in a report mean, read the "Validity metrics" section at the end.

---

## Why real recordings matter

The synthetic fixtures in `src/hotato/data/` are a deliberately conservative floor. They
are rendered from known segment timings, so the ground truth is exact and the eval
is runnable by anyone, offline, in seconds. That makes them ideal for regression
tests and for demonstrating that the scorer behaves as specified.

Production validity is a different question. Synthetic audio has clean onsets,
controlled overlap, and no room acoustics or codec artifacts beyond what we inject.
A scorer that looks flawless on synthetic fixtures still needs proving on a real
8 kHz call with echo bleed, cross-talk, and a caller who trails off mid-word.

A small, carefully labeled corpus of real calls closes that gap. It is the
difference between "the tool does what the spec says" and "the tool measures what
happens on a phone line." We keep the corpus small and high-integrity on
purpose: every clip is consented, de-identified, and human-labeled. Ten trustworthy
calls beat ten thousand scraped ones.

---

## Scope and boundaries

- The corpus measures the **audio timing of turn-taking**: whether the agent
  stopped talking when the caller interrupted (yield), how long both talked at
  once (talk-over), and whether the agent talked through a short acknowledgement
  like "mhm" (backchannel handling).
- Labels are about **events in time**: did the agent stop, and when did the
  caller start.
- Everything here is redistributed under the project's MIT license. A clip enters
  the corpus only if it can be released under MIT with consent.

---

## Consent / release template

Every party audible on a real recording must agree, in a form you can produce on
request, to its inclusion. The following paragraph is intentionally short and
reusable. Send it to a caller and/or agent-operator and keep the signed or written
affirmative reply on file. Fill the brackets before use.

> **Recording release, hotato open test corpus**
>
> I, [name or role], took part in the audio recording made on [date] and described
> as [short description]. I understand this recording will be used as a test
> fixture in *hotato*, an open-source project, and grant a perpetual, worldwide,
> royalty-free right to store, redistribute, and publish the recording and its
> derived timing labels as part of that project's MIT-licensed corpus. I confirm
> that the recording contains no confidential, personal, health, or account
> information, or that any such information has been removed or replaced before
> submission. I understand I can request removal of the recording from future
> releases at any time by contacting the maintainers.
>
> Signed / affirmed: [name], [date], [contact]

Notes:

- Get consent from **all** identifiable parties: the human caller, and whoever
  operates the agent side when that is a real person or a proprietary system whose
  owner must agree.
- Consent must be **affirmative and documented**. A generic call-center "this call
  may be recorded" notice is not sufficient for redistribution under an open
  license.
- Honor removal requests promptly in the next release.

---

## PII policy

Real audio must be de-identified before it is committed. In order of preference:
**do not capture it, then remove it, then reject the clip.**

Strip or redact:

- **Names** of any real individual (caller, agent, third parties mentioned).
- **Phone numbers**, extensions, and any dialed digits (DTMF tones included).
- **Postal and email addresses.**
- **Account identifiers**: customer numbers, order numbers, card numbers, policy
  numbers, ticket IDs, anything that keys back to a real person or account.
- Any other free-text detail that, alone or combined, re-identifies someone.

Strongly preferred practice:

- Use **synthetic or role-played content**. Actors reading a script hit the same
  turn-taking dynamics (interruption, correction, backchannel) with zero PII risk.
  This is the cleanest source of real acoustics.
- **No PHI.** Do not submit any recording containing protected health information,
  regardless of consent. It is out of scope for this project.

Redaction guidance:

- Replace spoken PII with a short tone or silence of the **same duration** so the
  turn-taking timing is preserved. Cutting samples out shifts every onset after the
  edit and corrupts the labels.
- Redact the audio in **all channels**. A name muted on the mixed channel but
  audible on the caller channel is still present.
- Keep a private note of what was redacted and when (timestamps). Never commit the
  removed content.

---

## Data-handling policy

- **The tool is self-hosted.** `hotato` reads local audio and writes local
  reports, offline; your recordings stay on your machine. Contributing to the corpus is a deliberate,
  manual act of committing a de-identified, consented clip.
- **Storage.** Corpus audio lives in the repository alongside its scenario JSON,
  under the same MIT terms as the rest of the project. Keep working copies local
  until they are cleared for release; do not stage real audio in third-party cloud
  buckets, shared drives, or chat tools during preparation.
- **Raw originals stay out of the repo.** Only the final, redacted, consented clip
  is committed. The un-redacted original is the contributor's responsibility to
  secure or destroy per their own obligations.
- **Contributor attestation.** Each real-audio PR must include a short statement
  from the contributor affirming: (a) consent is on file for every party, (b) PII
  has been removed per the policy above, (c) the clip contains no PHI, and (d) the
  contributor has the right to release it under MIT. This attestation is part of
  the merge record.

---

## Validity metrics: what we publish

The corpus exists to quantify measurement quality. That shapes how results
are reported.

We publish:

- **Per-signal measurement error, in milliseconds.** For each labeled call we
  compare the scorer's measured event times to the human labels and report the
  error distribution per signal:
  - `time_to_yield`: signed and absolute error in ms between measured and labeled
    floor-yield.
  - `talk_over`: error in ms on measured overlap duration.
  - onset agreement: error in ms between measured caller onset and
    `caller_onset_sec`.
  We report these as distributions (median, spread, worst case), so a reader sees
  the real shape of the error.
- **A `did_yield` confusion matrix.** Against the human `category` (`should_yield`
  / `should_not_yield`) label, we report the four-cell matrix: correct yields,
  correct holds, false yields (yielded on a backchannel), and missed yields (kept
  talking over a real interruption). The two error cells are the ones operators
  feel, so we surface them directly.

The distribution and the four-cell matrix are the report -- each failure mode
scored on its own, so the trade-off between them stays visible. The scorer works
on speech energy over time, so validity is timing agreement and decision agreement.

### Reading a corpus report

A healthy result looks like: tight millisecond error distributions on
`time_to_yield` and `talk_over`, onset agreement within a small ms band, and a
confusion matrix whose off-diagonal (false-yield / missed-yield) cells are small
and individually inspectable. A poor result looks like wide error spread or a heavy
off-diagonal, and it should point you toward a concrete fix in the call.

Where a real call fails, the fix map names candidate remedies. A learned
engagement-control layer is one of them, referenced at that level. Audio-only is
its weaker modality and a fully local offline model remains a known gap. The corpus
is here to measure reality.


================================================================================
FILE: docs/BENCHMARK-STACKS.md
================================================================================

# Benchmarking voice stacks with hotato

`hotato benchmark` scores the recordings you captured by running one fixed
scenario set through YOUR configured voice stack. Every stack is scored on the
same scenarios, the same labels, and the same thresholds, so the result files
are directly comparable.

hotato measures timing on the recordings it is given: every number comes from
your own captures, scored the same way for every stack. Results depend on your
stack configuration and your captures; a benchmark result describes YOUR
setup, on the day you captured it.

## 1. Pick the scenario set

The bundled 8-scenario barge-in battery is the default. Any scenarios dir
works the same way, for example the larger tiered suites:

```
corpus/suites/gold/scenarios
corpus/suites/silver/scenarios
```

Each scenario JSON carries the caller transcript, the caller onset timing,
and the pass thresholds. The corpus suites also ship each scenario's caller
stimulus as `audio/<id>.caller.wav`.

## 2. Capture the battery through your stack

For every scenario, drive the caller side through your stack and record the
call dual-channel (caller on channel 0, agent on channel 1):

- Corpus suites: play the shipped `<id>.caller.wav` stimulus into your stack.
  For the bundled battery, reproduce the caller side from each scenario's
  transcript and `caller_onset_sec`.
- `hotato setup --stack livekit` or `--stack pipecat` prints the exact
  dual-channel recording scaffold for your infra.
- `hotato capture --stack vapi --call-id <id>` or
  `hotato capture --stack twilio --recording-sid RE...` pulls the finished
  dual-channel recording for you.

Name each recording by its scenario id, one directory per stack:

```
captures/livekit/01-hard-interruption.wav
captures/livekit/02-backchannel-mhm.wav
...
captures/vapi/01-hard-interruption.wav
```

## 3. Run the benchmark

```bash
hotato benchmark --stack livekit --recordings captures/livekit --out livekit.json
hotato benchmark --stack vapi    --recordings captures/vapi    --out vapi.json
```

Each result JSON records the stack, the scenario set, every measured signal
(the same events `hotato run` produces), the exact scoring thresholds, and a
provenance block: who ran it, on which files, with each file's mtime. The
result timestamp comes from the input files, not the wall clock, so the same
inputs reproduce the same result.

Scenarios without a matching recording are listed under `not_captured`, left
out of both the scoring and the failure count. The exit code is 0 when the
run completes; add `--fail-on-regression` to exit 1 when a scored event fails
its scenario thresholds.

## 4. Compare stacks

```bash
hotato benchmark compare livekit.json vapi.json
```

Side by side, per scenario: yielded, talk-over seconds, and time-to-yield
seconds for each input, with signed deltas against the first file, plus
summary medians. If the files cover different scenario sets, the intersection
is compared and everything else is listed as skipped, with the files it was
missing from.

## What the numbers are

Reproducible timing measurements of the recordings you provided, scored by
the same engine as `hotato run`, with every threshold exposed in the result.
They tell you how your configuration of a stack handled the battery you
captured.


================================================================================
FILE: docs/BENCHMARK.md
================================================================================

# Benchmark methodology

`hotato` ships a reproducible measurement-error harness: `src/hotato/benchmark.py`.
Given a set of labelled dual-channel recordings, it reports how far the scorer's
measured event times land from the rendered or labelled ground truth, and how its
yield/hold decisions line up with the human labels. That is the whole output.

```bash
PYTHONPATH=src python3 -m hotato.benchmark
```

This runs on every synthetic fixture in the checkout (the bundled battery, the
`examples/` reference set, and the deliberately-bad `funnel-demo` set), prints a
markdown table, and writes `benchmark-report/measurement-error.json` and
`benchmark-report/measurement-error.md`.

The harness is a standalone measurement tool you run against fixtures. The `hotato`
CLI scores a single call or the battery.

---

## What it measures

For every `(recording, label)` pair it reports two things.

### 1. Per-signal measurement error, in milliseconds

For each timing signal it computes `|measured - rendered|` in ms, where `rendered`
is the exact value the synthetic fixture was rendered from (its `reference_render`
block) or the value a contributor labelled by hand:

| signal | measured | rendered/true reference |
|---|---|---|
| **caller onset** | onset the VAD detects (the scorer is given no onset label) | start of the first caller segment |
| **time to yield** | seconds from caller onset to the agent going quiet | the agent's in-progress turn end minus the caller onset |
| **response gap** | endpointing dead-air from the caller's turn end to the agent's next onset | the fixture's rendered response gap |

Errors are reported as a distribution: median, mean, worst case, best case, and n.
A signal is scored where a rendered or true reference exists and the scorer
produced a value; a missing reference yields a `-`. (The echo-of-agent fixture has
no independent caller speech, so it has no onset or yield error: a gap.)

Onset is measured in **detect mode** (the scorer gets no onset hint), so it
tests the onset detector directly. Yield, talk-over, response gap, and `did_yield`
are measured in **label mode** (the scorer is given the human `caller_onset_sec`,
exactly as the shipped battery runs), so those are the numbers a user
sees.

### 2. A `did_yield` confusion matrix

Against the `should_yield` / `should_not_yield` label (each scenario's
`expected.yield`), it reports the four cells:

|  | measured **did_yield** | measured **held floor** |
|---|---|---|
| **should_yield** | correct yield | **missed yield** |
| **should_not_yield** | **false yield** | correct hold |

The two off-diagonal cells (missed yields, and false yields) are the failures an
operator feels, so the report surfaces them directly.

---

## Why milliseconds and a matrix

The report is a per-signal error distribution plus a four-cell confusion matrix.
The cells stay separate because a missed yield and a false yield are different
failures with different fixes; averaging them into one number would hide which
one you have. This is the rule the corpus governance doc enforces
(`docs/CORPUS-GOVERNANCE.md`, "Validity metrics"), applied to the tooling.

The reported error is what the default shipped config measures, so it is
the number you get. On the synthetic fixtures the yield error equals the exposed VAD
hangover and the onset/gap error is one frame hop, both documented `ScoreConfig`
parameters. Set the hangover to zero
(`caller_vad.hangover_sec = agent_vad.hangover_sec = 0`) and every signal collapses
to within one hop of the rendered ground truth; the test suite asserts exactly
this, so the claim is checkable.

The scorer works on speech energy over time, so the harness reports timing and
decisions.

---

## Quantization: a reported time is not infinitely precise

Every timing signal the scorer reports is quantized to the frame hop
(`ScoreConfig.hop_ms`, default `10.0` ms) plus, for yield/talk-over, the VAD
hangover (`caller_vad.hangover_sec` / `agent_vad.hangover_sec`, default `0.15`
s): the measured value can land up to one hop off the true underlying event
purely from where that event falls inside a 10 ms frame, before hangover is
even counted. This sub-frame-phase rounding is deterministic, driven purely
by where the event lands inside the frame, and it is the same one-hop
collapse the section above already pins down (hangover zero -> every signal
within one hop of ground truth).

The consequence for a `--max-time-to-yield` (or any other) policy bound: a
physically identical yield event, shifted by a few milliseconds of sub-frame
phase with nothing else about the audio changed, can cross an exact bound
purely from quantization. Reproduced case: a 250 ms yield event evaluated
against a 400 ms bound flips PASS/FAIL as the event's sub-frame phase is swept
through 3, 6, 12, and 16 ms offsets, with the underlying event unchanged --
the measured value moves by exactly one hop (10 ms) at each transition, and
no further. Phase alone can flip a bound set within one hop of the true value
either way on that recording.

**Read policy bounds accordingly: a margin of less than one hop (10 ms
default) from the true value sits inside the scorer's quantization noise.**
This is a disclosure of an existing property. `hotato` surfaces the margin
plainly (see `docs/FIX-PLANS.md`'s no-single-threshold rule, which the same
logic extends to quantization); the fix is to know the margin exists and set
bounds at least one hop away from a value you need to hold.

---

## Reproducing it

```bash
# render the synthetic fixtures deterministically (sha256-seeded; byte-identical
# for a fixed hotato version -- verified in CI on Linux x86_64, Python 3.10,
# 3.11, and 3.12; also now checked, not yet green, on macOS and Windows -- see
# .github/workflows/tests.yml), then run the harness over them
python3 examples/render_examples.py
PYTHONPATH=src python3 -m hotato.benchmark
```

The report has no wall-clock timestamp and the render is deterministic, so two runs
on the same code produce byte-identical artifacts. The JSON carries a `config`
snapshot of every threshold that produced the numbers, so a reader can re-derive
any value. `tests/test_benchmark.py` pins the whole thing: it asserts the harness
runs, the confusion matrix matches the known rendered behaviour, and every ms-error
is within its known, config-derived tolerance.

The synthetic fixtures are a floor: deterministic rendered audio with exact known
timings. They prove the scorer behaves as specified and guard against regressions.
See `examples/README.md` and `docs/CORPUS-GOVERNANCE.md`.

---

## Extending to your own recordings (bring your own labelled data)

The synthetic floor tells you the scorer does what the spec says. A labelled
recording from a live phone line tells you what it measures under production
conditions. The harness runs on your own labelled data with no code change:

```bash
PYTHONPATH=src python3 -m hotato.benchmark \
  --scenarios path/to/your/scenarios \
  --audio     path/to/your/audio \
  --name      my-corpus \
  --out       my-benchmark-report
```

- `--scenarios` is a directory of scenario JSON labels, same shape as
  `src/hotato/data/scenarios/*.json`. Each needs an `id`, an `expected.yield`
  label, and whatever reference timings you can defend: a `reference_render` with
  segment timings (as the synthetic fixtures carry), or at minimum a hand-labelled
  `caller_onset_sec` and the true event times you want error scored against. Supply
  a reference only where you have ground truth; the harness reports error for those
  signals and leaves the rest blank.
- `--audio` is a directory of dual-channel `<id>.example.wav` recordings (caller on
  channel 0, agent on channel 1). Record two physically separated channels whenever
  you can: that is what makes overlap ground-truthable.

When `--scenarios` is given, only that set is scored, so you get a clean report for
your corpus alone.

The path to a number from your own recordings is manual and consented: bring
your own labelled audio. Contributing that audio is governed by
`docs/CORPUS-GOVERNANCE.md` (consent, PII, data-handling) and the pipeline in
`corpus/` (a labelling schema and a validator). Synthetic fixtures keep their
synthetic label.


================================================================================
FILE: docs/CARDS.md
================================================================================

# Cards: a shareable image from any hotato result

`hotato card` turns a machine result into a self-contained SVG you can drop into
a pull request, an issue, or a slide. One command, offline: the card names the
measured timing moment -- a reproducible measurement, not a verdict about
intent.

```bash
hotato card INPUT[#REF] --out card.svg
```

Everything runs locally. The SVG is a pure function of the input JSON alone --
the same input renders the same bytes forever -- with every color inlined and
no font, image, stylesheet, script, or link to fetch. Drop it anywhere: it's
fully self-contained, no CDN or asset host required.

## The four cards, auto-detected

The input's kind decides the card; you do not pick.

| Input | Card |
|---|---|
| a `sweep`/`analyze` candidate ref, `FILE#N`, that is a talk-over moment | **talk-over candidate** |
| a `sweep`/`analyze` candidate ref, `FILE#N`, that is a false-stop moment | **false-stop candidate** |
| a fix plan whose `decision` is `do_not_tune_single_threshold` | **threshold funnel** (the hero) |
| a supported `hotato verify` before/after rollup that improved | **paired comparison** |

`#N` is the same 1-based rank the sweep report and dashboard show, and it is the
same ref `hotato fixture promote` takes, so a card and a fixture speak of the
exact same moment.

### A. talk-over candidate

An `overlap_while_agent_talking` (or `agent_start_during_caller`) moment: the
agent kept the floor while the caller was speaking. The card leads with the
measured overlap in seconds and closes with "Hotato reports timing candidates,
not intent."

```bash
hotato sweep --demo --format json > hotato-sweep.json
hotato card hotato-sweep.json#3 --out talk-over.svg
```

### B. false-stop candidate

An `agent_stop_no_caller` moment: the agent went quiet with no caller nearby to
explain the drop. The card leads with the measured trailing silence.

```bash
hotato card hotato-sweep.json#1 --out false-stop.svg
```

### C. threshold funnel (the hero)

The plan the both-axes case produces: the battery missed an interruption
**and** false-stopped on a backchannel, so satisfying both axes at once needs
more than a single sensitivity dial. The card states that Hotato refused
threshold tuning and names the fix class (`engagement-control`). This is the
card the project leads with.

```bash
hotato demo --format json > demo.json
hotato plan demo.json --out fix-plan.json
hotato card fix-plan.json --out no-single-threshold.svg
```

Only a `do_not_tune_single_threshold` plan renders this card; any other plan is
a clean exit-2 usage error (it is not one of the four kinds).

### D. paired comparison

A supported `hotato verify` before/after rollup where at least one previously-failing
fixture now passes and no hold/backchannel fixture regressed. This is paired
evidence tied to that specific before/after pair -- the card
reads "PAIRED FRESH-RECAPTURE IMPROVED" only when the recapture is runner-
attested and "PAIRED (OPERATOR-ASSERTED)" otherwise, never "verified" or "fix verified", and
closes with "Hotato reports coincidence, not causation." A verify result that
doesn't support that claim (too few previously-failing fixtures, nothing now
passing, or a regressed hold fixture) is refused with exit 2.

```bash
hotato card verify.json --out comparison.svg
```

## Redaction: safe to share by default

A card is a public image, so identifiers are hidden by default. A call id, a
filesystem path (only a basename is ever a candidate for display), and a vendor
recording name are omitted. A pulled recording named `STACK__ID.wav` carries the
call id inside its name; that name is only ever shown under
`--include-identifiers`.

```bash
# shows the source recording's basename on a candidate card
hotato card hotato-sweep.json#1 --out card.svg --include-identifiers
```

## Output and exit codes

Without `--out`, the SVG is written to stdout, so you can pipe it. With `--out`
it is written there atomically.

- **0**: the SVG card was rendered (to `--out`, or to stdout).
- **2**: usage error, unreadable input, a bad candidate ref, or an input that is
  not a fix plan / verify result / sweep candidate.

## Regenerating the committed cards

Three commit-ready examples live under `docs/assets/cards/`
(`no-single-threshold-card.svg`, `talk-over-card.svg`, `false-stop-card.svg`),
rendered from the two bundled demo calls. Regenerate them with:

```bash
PYTHONPATH=src python3 scripts/render_card_assets.py
```


================================================================================
FILE: docs/COMPARE.md
================================================================================

# Compare: Hotato vs broad QA platforms

Hotato is an open-source, self-hosted conversation-QA system; this page maps where it sits next to runtime layers and hosted platforms.

**Use Hamming, Cekura, Coval, Bluejay, Roark, Vapi, or Retell for:** broad
session QA, synthetic simulation, task success, transcript rubrics, compliance
workflows, production dashboards, load testing.

**Use Hotato for:** portable failure contracts from live calls, local/private
timing evidence, CI-enforced regression tests, trace-backed turn-taking proof,
refusing unsafe threshold bandaids.

**Hotato answers:** "Is the recorded evidence for this exact timing failure still intact (every push), and does the CURRENT agent still avoid it (on a fresh recapture, [`docs/RECAPTURE.md`](RECAPTURE.md))?"
**A broad QA platform answers:** "Was the whole call successful?"

These are complementary layers. A team running one of the
platforms above for broad conversation QA or as the agent runtime itself still
needs an answer to a narrower question: the specific talk-over or false-stop
moment a caller hit last week, is it still fixed after today's prompt
change? That is the layer Hotato owns, deliberately the only one.

## What each named platform is built for

Capability descriptions only, sourced entirely from each platform's own public
launch and product material.

| Platform | What it is built for |
|---|---|
| **Hamming** | Automated voice-agent testing: prompt-to-test generation, simulated call batteries, a multi-layer QA framework, and production-replay CI/CD regression across a broad test suite. |
| **Cekura** | AI voice/chat agent QA: production-conversation ingestion, test-case extraction from production calls, simulated scenario testing, and monitoring across fleets of agents, including regulated verticals. |
| **Coval** | Voice and chat agent simulation and evaluation, with published comparisons of testing approaches across the category. |
| **Bluejay** | Synthetic stress-testing for voice agents at scale. |
| **Roark** | Production-call replay for QA that preserves what the caller said, how, and when, for building regression scenarios. |
| **Vapi** | A voice agent orchestration platform: the runtime stack a team builds and deploys its agent on. |
| **Retell** | A voice agent orchestration platform for building and deploying voice agents. |

## What it measures, and the better fit

| What you need | Best fit | Why |
|---|---|---|
| Broad conversation QA: did the call succeed, was the transcript right, did it follow the rubric, simulate hundreds of scenarios, dashboards for the team | **Hamming, Cekura, Coval, Bluejay, Roark, Vapi, or Retell** | These grade the whole conversation, task success, and content, or they are the agent platform itself. Hotato's lane is the timing evidence underneath. |
| Prevent an interruption problem **in the moment**, at runtime: predict endpointing, suppress barge-in on noise, tune the live turn detector | **Pipecat, LiveKit** (and similar runtime layers) | These act during the live call. Hotato runs after the fact, from the recording. |
| Prove a specific timing bug is fixed, from a recorded call, portably, with every byte staying on your machine; a fresh recapture pins whether it **stays** fixed | **Hotato** | A private, deterministic fixture: audio, a human label, and an explicit policy, scored the same way everywhere with `hotato verify`. Ships as a single portable contract bundle -- audio, timing evidence, trace evidence, label, policy, CI command -- in the release that adds the contract layer. |

If your open question is "is my agent good?", start with a QA platform. If it
is "is my agent interrupting well right now?", start with a runtime layer. If
it is "is the evidence for the talk-over bug we fixed last month still
intact, and does this release's agent still avoid it?", that is Hotato: the
frozen check runs on every push, and the agent check needs a fresh recapture
([`docs/RECAPTURE.md`](RECAPTURE.md)).

## Runtime layer vs regression layer

These two are often confused, so it is worth being exact.

- A **runtime layer** (Pipecat, LiveKit turn detection) makes a decision
  *during* the call: should the agent stop talking now? It optimizes the live
  experience, moment to moment.
- A **regression layer** (Hotato) makes a decision *after* the call, from the
  recording: given this exact audio and this label, did the timing match, yes
  or no, byte-stable on every run? On the frozen recording it catches a
  change to that recorded evidence or policy on every push, and hands that
  proof to CI as a portable artifact; catching the day the AGENT itself
  regresses needs a fresh recapture through the same check
  ([`docs/RECAPTURE.md`](RECAPTURE.md)).

You want both. A runtime layer improves the median call. A regression layer
catches evidence and policy drift on every push, and, recaptured, proves
whether a good fix is still holding three releases later. Hotato takes the
moment a runtime layer got wrong, freezes it as a fixture with `hotato
fixture promote`, fails CI if the frozen evidence regresses, and recaptures
to prove the live agent still holds.

## What Hotato is for, precisely

- **Private.** Scoring, scanning, reports, fixtures, and verification run
  offline, and audio stays on your machine unless you explicitly pull it from
  your own stack. See [THREAT-MODEL.md](THREAT-MODEL.md).
- **Deterministic under pinned inputs.** The reference backend is rule-based
  end to end: with the same supported hotato version, the same audio, channel
  map, event onset, label, and scoring config, it produces the same timing
  numbers every run, so a changed result means at least one pinned input,
  policy, or scorer component changed -- most commonly the audio itself.
  Byte-identical re-runs are verified in CI on Linux x86_64, Python 3.10,
  3.11, and 3.12. The same check now also runs in CI on macOS and Windows,
  not yet green -- see [VALIDATION.md](VALIDATION.md) Job 1.
- **Portable.** A confirmed failure becomes a labelled fixture (audio, human
  label, explicit policy) that travels with the repository and verifies the
  same way with `hotato verify`, deterministic for a fixed hotato version
  (see above).
- **Narrow on purpose.** Three timing signals, and only those: talking over
  the caller, false-stopping on a backchannel, yielding too slowly.

## On provider-default examples

When a Hotato example shows a call failing on a named stack, it is labelled a
**provider-default** run: one assistant, one configuration, one date, one
scripted caller, on that provider's out-of-the-box interruption settings. It
demonstrates the threshold funnel (how a default config can miss an
interruption and false-stop on a backchannel in the same battery), scoped to
that one run. Any stack, tuned, can pass the same fixtures. Hotato publishes
fixtures and reproduction steps, never a scoreboard, of its own results or
anyone else's.

## The short version

Use a QA platform for broad quality, a runtime layer for live interruption
handling, and Hotato to prove, privately and portably, that a specific timing
bug from a call is fixed, gate every push on the frozen evidence, and
recapture to confirm it stays that way on your current agent.


================================================================================
FILE: docs/DIARIZE.md
================================================================================

# `hotato run --mono call.wav --diarize`: score a single-channel recording

Hotato's gold reference is a **two-channel** recording -- the caller on one
channel, the agent on the other, each channel one party, no separation needed.
That path, and every published/golden number, is unchanged. A **mono** (single,
mixed channel) recording is the coverage wall: by default it is rejected as not
scorable, because with both voices summed into one waveform the scorer cannot
attribute energy to a speaker.

The opt-in `[diarize]` front-end widens that coverage. It runs an off-the-shelf
**speaker diarizer** over the mono to recover *who was active when*, reconstructs
two caller/agent tracks, and feeds the **existing** scorer -- so a mono call
becomes scorable. It is **quality-gated** and labeled by tier: above the
confidence bar the verdict is a `diarized-mono` verdict; below it, the
verdict is labeled indicative only and no SLA gate fires; a non-separable file is
refused. A diarized-mono verdict is labeled and gated separately from a true
dual-channel measurement for sub-second talk-over, and the gate enforces that
per file.

**Turn-timing reconstruction, from anonymous labels.** Hotato scores *timing*
(who was active when, and overlap), which a diarizer's turn timestamps
reconstruct directly: it assigns anonymous `SPEAKER_00` / `SPEAKER_01` labels to
each turn and hands the boundaries to the existing scorer. The audio stays
mixed, and the labels carry only channel timing -- no name, no voice-print
match, no identity.

## Quickstart

```bash
# Install the default (local, offline) diarizer extra, plus a Hugging Face token
# with the gated model conditions accepted (one-time download, then offline):
pip install 'hotato[diarize]'
export HUGGINGFACE_TOKEN=hf_...            # accept the model card conditions first

# Check first whether the mono is confidently separable (no scoring):
hotato trust --stereo call.wav --diarize            # -> high / low / refuse tier

# Score the mono:
hotato run --mono call.wav --diarize --format json  # diarized-mono verdict
```

Without the extra (or the token/model), `--diarize` exits `2` with a clean
error: it always requires the diarizer to run before scoring.

## Backends and extras

A pluggable backend seam (mirroring the neural-VAD seam). Pick with
`--diarizer`; install only the extra you select. The default backend is chosen
by the downstream benchmark's measured results: `pyannote` is the accessible
local default, and a user who needs best-in-class accuracy on telephone audio
picks `sortformer` (local) or `pyannoteai` (hosted).

| `--diarizer` | extra | where it runs | notes |
|---|---|---|---|
| `pyannote` (default) | `[diarize]` | local, CPU-viable, offline | richest confidence signals (posterior + embedding margin); gated HF weights |
| `sortformer` | `[diarize-sortformer]` | local, GPU-leaning | best self-hostable on 2-speaker telephone; EEND (no embedding margin) |
| `pyannoteai` | `[diarize-hosted]` | HOSTED (audio leaves the machine) | best absolute accuracy; requires `--egress-opt-in` |

```toml
diarize            = ["pyannote.audio>=4.0", "torch>=2.8", "torchaudio>=2.8", "numpy>=1.21"]  # default; needs system ffmpeg
diarize-sortformer = ["nemo-toolkit[asr]>=2.7", "torch>=2.8", "numpy>=1.21"]                   # best self-hostable, GPU
diarize-hosted     = ["pyannoteai-sdk>=0.3"]                                                   # hosted, egress opt-in
```

The `[diarize]` path raises the effective Python floor to **>=3.10** (pyannote
4.x); the stdlib core stays >=3.9 -- this only constrains the optional path.

### Model licenses (log per FTO note)

Using off-the-shelf diarizers is integration, orthogonal to any Hotato IP claim,
but the dependency licenses are logged here and carried in the score envelope's
`diarization.licenses` block:

- `pyannote-audio` -- **MIT** (code)
- `speaker-diarization-community-1` weights -- **CC-BY-4.0** (attribution required)
- `segmentation-3.0` weights -- **MIT**; `wespeaker` embedding -- **CC-BY-4.0**
- `torch` -- **BSD-3-Clause**; `torchaudio` -- **BSD-2-Clause**
- `nemo-toolkit` -- **Apache-2.0**
- Sortformer **streaming v2** -- **CC-BY-4.0** (the offline v1 is **CC-BY-NC** ->
  non-commercial, and is never shipped)
- pyannoteAI hosted -- proprietary terms (verify before enabling egress)

If your licensing posture wants a permissive weights license, prefer
`speaker-diarization-3.1` (MIT weights) over community-1 (CC-BY-4.0) via
`HOTATO_DIARIZE_MODEL`.

## The confidence gate (the core safeguard)

Aggregate diarization error is a corpus statistic; the gate is **per file, at
runtime, with no ground truth**. Six signals feed a `separation_confidence` in
`[0, 1]` and one of three tiers:

| signal | flags low quality when |
|---|---|
| speaker count == 2 | != 2: not two clean parties (1 = couldn't separate; 3+ = extra voices / mis-cluster) -> refuse |
| both speakers >= 0.30s activity | a near-silent "speaker" is a spurious split of one party -> refuse |
| mean segmentation posterior | near-chance: the model is unsure who is speaking (uncalibrated; a relative signal) |
| embedding cluster margin (pyannote only) | small: two voices too similar to attribute confidently |
| overlap ratio in a sane band | extreme: heavy crosstalk / collapsed turns -> talk-over unreliable |
| segment churn (short turns/sec) | high: a jittery, unstable timeline -> noisy timing |

**Tiers:**

- **high** -- score normally; the verdict is always tagged
  `source: "diarized-mono"` and `confidence_tier: "high"`, distinct at every
  step from a dual-channel verdict.
- **low** -- score, but the envelope carries `indicative_only: true`: the verdict
  is "indicative only, reconstructed from single-channel diarization." The
  pass/fail SLA gate (`--max-talk-over` / `--max-time-to-yield`) stays off on a
  low tier.
- **refuse** -- `scorable: false`, a reason naming the failed signal, exit `2`
  (exactly like today's mono rejection).

The thresholds are provisional and **uncalibrated** -- they are pinned by the
downstream verdict-agreement benchmark, not asserted as accuracy -- and are
exposed as constants in `hotato/diarize.py`.

## Caller vs agent assignment (proposed, stated, overridable)

A diarizer returns anonymous `SPEAKER_00` / `SPEAKER_01`; Hotato needs
caller/agent. The mapping is proposed, stated as an assumption, and overridable:

- **default proposal** -- reuse the floor-dominance heuristic (`trust`'s
  possible-swap band): an agent usually holds the floor longer, so the
  higher-talk-time speaker is proposed as the agent. Ambiguous (balanced floor
  time) mappings are broken by who-speaks-first and flagged `balanced: true`,
  which downgrades the verdict to indicative until confirmed.
- **override** -- `--caller-speaker SPEAKER_00 --agent-speaker SPEAKER_01`; when
  both are given no heuristic runs.

The chosen mapping and its basis are emitted in
`diarization.speaker_map: {caller, agent, basis, balanced, confidence}`.

## Echo / crosstalk is N/A on this path

The two reconstructed tracks are slices of **one physical microphone**, so
`signals.echo` / crosstalk coherence carries no echo information (it is trivially
high in overlap). On the diarized-mono path the echo block is marked
`applicable: false`, since `--echo-gate` is meaningful only across two
physically separate channels.

## Limits (what stays indicative or refused)

- **very similar voices** (same gender/pitch, or one person on both ends) -> low
  embedding margin -> low/refuse.
- **heavy crosstalk / echo bleed** -> overlap balloons -> low/refuse.
- **>2 speakers** (a supervisor, hold music with vocals) -> refuse.
- **deep sustained overlap** and **sub-second boundary precision** are inherently
  weaker than dual-channel and are stamped, never hidden.
- **balanced speaker map** -> mapping uncertain -> indicative until confirmed.

A de-risk spike (with a *perfect* diarizer) confirmed the masked-reconstruction
path systematically inflates sub-second talk-over by ~0.1-0.36s and can bridge a
short backchannel gap -- an error intrinsic to single-channel masking that no
diarizer quality removes. That is precisely why elevated-overlap and short-yield
cases land in the fragile `low` zone (indicative, no SLA gate). Direct
diarization-timeline injection (skipping the reconstruction re-VAD) is the
recommended follow-up; under either approach the error budget is governed
by the gate above.

## JSON shape (agents)

A scored diarized-mono envelope is the SAME envelope as any single run, plus a
`diarization` provenance block and, when the tier is not high, `indicative_only:
true` on the event:

```json
{
  "mode": "single",
  "diarization": {
    "source": "diarized-mono",
    "backend": "pyannote",
    "model": "pyannote/speaker-diarization-community-1",
    "num_speakers": 2,
    "speaker_map": {"caller": "SPEAKER_00", "agent": "SPEAKER_01", "basis": "floor-dominance", "balanced": false},
    "separation_confidence": 0.86,
    "confidence_tier": "high",
    "overlap_ratio": 0.14,
    "licenses": {"pyannote-audio": "MIT (code)", "...": "..."}
  },
  "events": [{
    "verdict": {"did_yield": true, "seconds_to_yield": 0.5, "talk_over_sec": 0.5, "reasons": []},
    "diarization": { "...": "same block" },
    "scorability": {"separation": {"confidence_tier": "high", "separation_confidence": 0.86, "signals": {"...": "..."}}},
    "signals": {"echo": {"applicable": false, "reason": "single physical channel ..."}}
  }]
}
```

Branch on `diarization.confidence_tier`: an event carrying `indicative_only:
true` stays indicative-only, distinct from a confident dual-channel verdict. A
refused file is `scorable: false` with `not_scorable_reason` and exit `2`.


================================================================================
FILE: docs/DRIVE-A-CALL.md
================================================================================

# Drive-a-call: originate a call against a live agent, then score it

`hotato capture` scores a call you already ran. Drive-a-call closes the other
half: it PLACES the call against a live voice agent, waits for it to finish, and
feeds the recording into the same validated pull -> score pipeline. The produced
conversation carries the agent's own live turns, unscripted on that side, so
it flows through scoring unchanged.

It lives in `src/hotato/drive.py` and is wired into the fleet experiment loop as
the `run_scenario` step of the Vapi and Twilio adapters
(`src/hotato/fleet/adapters.py`).

## Worth stating plainly: the caller side runs from a script

The agent's half of the conversation is unscripted. The CALLER's half runs
from a script, and the origin records that plainly as `origin.caller`.

- **Twilio** renders your `scenario.v1` caller script into TwiML: one `<Say>`
  per `say`-turn, `<Pause>` between them. TwiML `<Say>` speaks at **fixed
  offsets and cannot react to what the agent says** -- so this is a
  FIXED-TIMELINE regression driver: deterministic, good for catching a
  regression on a scripted turn sequence, not a reactive caller that barges in
  the instant the agent starts talking. A turn's reactive `when_agent_asks` /
  `after` label is therefore NOT honored here (it is spoken unconditionally, in
  order). A reactive caller is later work. `origin.caller = "scripted-twiml"`.
- **Vapi** originates the call FROM the assistant (the agent under test -- a
  staging clone) TO a customer number. The direction is
  outbound-from-assistant: the assistant is the party being measured, and
  whoever/whatever answers the customer number is the other side. There is no
  scripted-TwiML caller on this path. `origin.caller = "assistant-originated"`.

Either way `origin.kind = "real"`, with `origin.provider` and
`origin.provider_call_id` (the provider's own call id). This is the invariant-5
axis: a real driven call is never conflated with a simulated one, and the origin
never overstates what drove the caller side.

## The endpoints implemented

| Provider | Originate | Poll until | Then pull |
| --- | --- | --- | --- |
| Twilio | `POST /2010-04-01/Accounts/{sid}/Calls.json` with `To`/`From`/`Twiml` + `Record=true`, `RecordingChannels=dual` (the REST equivalent of `<Dial record="record-from-answer-dual">`) | `GET .../Calls/{CallSid}.json` -> `status == completed` | `GET .../Recordings.json?CallSid=...` -> `RecordingSid` -> existing `capture_twilio` (`?RequestedChannels=2`) |
| Vapi | `POST https://api.vapi.ai/call` `{assistantId, phoneNumberId, customer:{number}}` | `GET /call/{id}` -> `status == ended` | existing `capture_vapi` -> `artifact.recording.stereoUrl` |

Only `POST` (create) and `GET` (poll + pull) are ever issued: drive-a-call can
create a call and read its status, and that is the whole surface -- a provider
config (an assistant, a number) stays untouched. For Vapi the call is driven
FROM the staging CLONE, so production stays untouched too.

A non-`completed` Twilio call (busy / failed / no-answer / canceled) is a
dead-end with no recording to score, so it raises a clear error.

## Credentials and the egress opt-in (both required)

Placing a call reaches the provider's REST API and **costs a real phone call**.
So `run_scenario` requires BOTH before it dials:

1. **Credentials** -- `VAPI_API_KEY`, or `TWILIO_ACCOUNT_SID` + `TWILIO_AUTH_TOKEN`
   (via `hotato connect` or the environment).
2. **An explicit egress opt-in** -- set `HOTATO_DRIVE_OPT_IN=1`, or pass
   `egress_opt_in: true` on the scenario. Absent it, `run_scenario` raises the
   same clean structured refusal a hosted op has always given, and dials nothing.

The drive parameters ride on the scenario (inline or under a `drive:` block) or
the environment:

- Twilio: `to_number` (the agent's number, `HOTATO_DRIVE_TO_NUMBER`),
  `from_number` (your Twilio number, `HOTATO_DRIVE_FROM_NUMBER`).
- Vapi: `phone_number_id` (`VAPI_PHONE_NUMBER_ID`), `customer_number`
  (`HOTATO_DRIVE_CUSTOMER_NUMBER`).

The recording download reuses `capture`'s validated path: http(s)-only scheme
allowlist, default-deny SSRF (a `127.0.0.1` local test recording server needs
`HOTATO_ALLOW_PRIVATE_URLS=1`, same as every other download), cross-host
credential strip, and atomic write. See [`docs/EGRESS.md`](EGRESS.md) and
[`docs/THREAT-MODEL.md`](THREAT-MODEL.md) for the per-command rows.

## What it costs, and what the recording proves

- It costs one real outbound phone call per scenario run, billed by your
  provider -- even the deterministic Twilio caller is a billed call.
- The recording captures a real agent conversation. Scoring it -- whether the
  agent "passed," whether the scripted caller's timing matched a real one --
  is the separate assert layer's job, run over that produced recording.

## Retell / LiveKit / Pipecat

Retell calls are captured after the fact with `hotato pull`, since Retell has
no confirmed create-call API to originate from. LiveKit and Pipecat capture
inside your own infra, so drive-a-call isn't wired for them either -- point
your own capture path at the recording instead.


================================================================================
FILE: docs/EGRESS.md
================================================================================

# Egress: what talks to the network, command by command

Derived from the code: every `urllib`/`http`/`subprocess`-to-a-network-tool
call site in `src/hotato/`, mapped to the CLI command that reaches it. Every
command listed under "Fully local" below runs entirely on your machine -- no
`urllib` import, no socket, no networked subprocess.

## Fully local -- everything runs on your machine

`run`, `report`, `doctor`, `team`, `export`, `benchmark`, `compare` (the
batch result-comparison command, `stackbench.py`), `demo`, `start`, `card`,
`diagnose`, `plan`, `explain`, `fixture create`, `fixture promote`,
`contract create` (stereo / caller+agent / local-file mono), `contract
verify`, `contract inspect`, `contract pack`, `contract unpack`, `trace
attach`, `trace export`, `scan`, `trust`, `analyze`, `patch`, `verify`, `fix
trial`, `loop`, `describe`, `init starter`, `init webhook` (the scaffold
generator -- the server it scaffolds is a local listener, see below), `setup`.

These read local files (recordings, scenarios, `hotato.yaml`, connection
files) and write local files. `connect` also belongs here: it validates and
stores credentials at `~/.hotato/connections.json` (mode `0600`) without a
network round trip (`src/hotato/connections.py`: "nothing in this module
makes a network call").

`rubric run`, `rubric calibrate`, and the `test run` rubric lane also belong
here **on the default path**: the model judge is a **LOCAL Ollama** daemon
(default `http://localhost:11434`), so everything stays on your machine. The
verdict cache (`~/.hotato/rubric-cache`) is local. Two ways a rubric command DOES reach
the network -- both explicit, both in the extras table below: a non-local
Ollama endpoint, or `--judge-provider hosted`.

## Reaches your configured vendor -- only when the command's job is to

| Command | Reaches | When | Code |
| --- | --- | --- | --- |
| `capture --stack vapi\|retell\|twilio` | The stack's REST API | Always (fetches the one call you named) | `capture.py`: `_http_get`/`_http_get_json`/`_download` via `urllib.request` |
| `capture --stack vapi\|retell\|twilio --demo` | Nothing | `--demo` scores a bundled reference file instead | `capture.py` |
| `capture --stack livekit\|pipecat` | Nothing from Hotato | The recording is produced by YOUR infra; Hotato only scores the local file `setup` pointed you at | `capture.py` |
| `pull` | The stack's list + download endpoints | Always (bulk-fetches recent recordings) | `capture.py` (same `_http_get`/`_download` path `capture` uses, looped) |
| `sweep --stack <stack>` | Same as `pull` | Whenever `--demo` is NOT passed | `capture.py` via the pull path |
| `sweep --demo` | Nothing | Sweeps the two bundled demo recordings | `analyze.py` on packaged audio |
| `inspect --stack vapi\|retell` | The stack's assistant-config read endpoint (GET only, never a write) | Always | `inspectcfg.py`: `_http_get_json`, `inspect_vapi`, `inspect_retell` |
| `inspect --stack livekit\|pipecat` | Nothing | Parses YOUR local config file with `ast`, no network | `inspectcfg.py`: `inspect_livekit_file`, `inspect_pipecat_file` |
| `ingest` (webhook worker) | Inbound only, from the stack you configured to call your webhook; outbound only if the event's `recording_url` needs a download | Per event, and only for the fetch step | `ingest.py`: `_resolve_recording` reuses `capture.py`'s fetch/download; payloads are always treated as data, never instructions |
| `apply` (default, no `--yes`) | Nothing | Dry run: prints the staging clone it WOULD create | `apply.py`: `build_apply` returns `dry_run: True`, never calls the networked function |
| `apply --clone --yes` | The stack's REST API (`vapi`, `retell` only) | Only with `--yes` and credentials | `apply.py`: `create_clone` / `_http_json` is "the only networked function" in the module (its own docstring) -- reads the source config via GET, then POSTs to create a NEW staging assistant. Never PUTs/PATCHes the source. |
| `issue create` (default, no `--yes`) | Nothing | Dry run: prints the issue body and the `gh` command it would run | `issuecmd.py` |
| `issue create --yes` | GitHub, through your local `gh` CLI's existing auth | Only with `--yes` | `issuecmd.py`: `subprocess.run(["gh", "issue", "create", ...])` |
| `pr create` (default, no `--yes`) | Nothing | Dry run: prints the PR body and the `git`/`gh` argv it would run | `prcmd.py` |
| `pr create --yes` | Git remote + GitHub, through your local `gh` CLI's existing auth | Only with `--yes` | `prcmd.py`: `subprocess.run([...git...])` then `subprocess.run(["gh", "pr", "create", ...])` |
| `test run --state F` (state-config with `adapter: http`) | Your configured system-of-record REST API | Per `state`/`state_change` assertion, and ONLY when the config sets `egress_opt_in: true`. What leaves: the mapped filter VALUES for that one query (in the URL or a JSON body); never audio, transcript, or the config. | `state_adapter.py`: `HttpStateAdapter.query` via `urllib.request` (credential-safe redirects reused from `capture.py`) |
| `test run --state F` (state-config with `adapter: sql` + a `dsn`) | Your database, over the network via the driver you install | Per assertion, and ONLY with `egress_opt_in: true`. A parameterized read-only SELECT; the filter VALUES are bound as data, never interpolated. A local `sqlite_path` opens no socket. | `state_adapter.py`: `SqlStateAdapter.query` (stdlib `sqlite3`, or a caller-installed DBAPI driver) |

`load_state_adapter` requires `egress_opt_in: true` before an `http` adapter or
a `sql` adapter over a `dsn` connects (absent it, it raises before any
connection -- the CLI's exit-2 usage-error path). The default `--state`
adapters -- the local mock JSON/SQLite sandbox and a local `sqlite_path` SQL DB
-- run entirely on your machine and need no opt-in.
Full config format: [`docs/STATE-ADAPTERS.md`](STATE-ADAPTERS.md).

## Optional extras that add a hosted call

| Extra | Adds | Gate |
| --- | --- | --- |
| `hotato[neural]` | Nothing network -- a local Silero VAD cross-check model, run offline | N/A |
| `hotato[transcribe]` (`run --transcribe`) | Nothing at inference -- a local `faster-whisper` ASR pass over the same recording, fully offline once the chosen model is cached. The first use of a model name not already cached downloads its weights from its public host (a one-time fetch, like installing any pip package with model weights); every run after that opens no socket. Context only -- kept separate from the score. | N/A |
| `hotato[livekit]` / `hotato[pipecat]` | Nothing from Hotato directly -- these SDKs run YOUR live capture infra; Hotato scores the file that infra writes | N/A |
| `--diarizer pyannoteai` (`contract create --mono --diarize`, `run --mono --diarize`) | Uploads the mono audio to `pyannote.ai` for diarization | Requires `--egress-opt-in` (exit 2 without it); the default diarizer (`pyannote`, local) stays offline. See `diarize.py`: `build_pyannoteai_backend`. |
| `hotato[judge]` (`rubric run`, `rubric calibrate`, `test run` rubric lane) | Nothing on the default path -- the judge is a LOCAL Ollama model (`http://localhost:11434`), reached with only the stdlib (`urllib`). The transcript stays on your machine. | N/A on the default (local) path. `rubric.py`: `OllamaJudge`. |
| `--judge-provider hosted --judge-endpoint URL` (any rubric command) | Sends the transcript + rubric criterion to a hosted OpenAI/Anthropic-compatible `/chat/completions` endpoint you name | Requires `--judge-egress-opt-in` (exit 2 without it); the default local judge stays on your machine. `rubric.py`: `HostedJudge` (raises `EgressRefused` without the flag). |
| A **non-local** `--judge-endpoint` (e.g. a remote Ollama host) | Reaches that host | Same gate: requires `--judge-egress-opt-in`. `localhost`/`127.0.0.1`/`::1` skip it. `rubric.py`: `OllamaJudge._is_local_endpoint`. |
| `--notify URL` (`sweep`, `fleet run`) | POSTs one JSON summary -- counts, top candidate moments (id, kind, timing numbers only), local artifact paths. Audio, credentials, and transcript text stay out of it. Plus a `text` line for Slack incoming webhooks. | Off by default; only fires with an explicit, repeatable `--notify URL`. A non-http(s) scheme is refused (exit 2) before any network attempt; once sent, delivery is fail-open -- a down or slow webhook logs one stderr warning and the run keeps going. See `notify.py`: `post_notification`. |

## The one credential-safety detail worth knowing

`capture.py` installs a process-wide `urllib` redirect handler
(`_CredentialSafeRedirectHandler`) so a 3xx redirect from a vendor API keeps
your Authorization header and API key scoped to the original host. This
applies to every fetch path in the table above that goes through
`capture.py`'s `_http_get`/`_download`.

## Read more

- Posture summary and reporting a vulnerability: [`SECURITY.md`](../SECURITY.md)
- Command-by-command threat model: [`docs/THREAT-MODEL.md`](THREAT-MODEL.md)
- Ingest's untrusted-payload handling: [`docs/INGEST.md`](INGEST.md)

## Drive-a-call -- originates a real call, so it is double-gated

`run_scenario` (the fleet experiment step; `src/hotato/drive.py`) ORIGINATES a
real, billable call against a live agent and pulls its recording. It is the one
path here that both reaches a vendor AND places an outbound call, so it carries a
gate on top of the usual credential requirement.

| Op | Reaches | When | Gate |
| --- | --- | --- | --- |
| `run_scenario` (Vapi adapter) | `POST https://api.vapi.ai/call`, then polls `GET /call/{id}`, then the existing `capture_vapi` download | Only when driven | Requires real credentials AND an explicit egress opt-in (`HOTATO_DRIVE_OPT_IN=1` or `egress_opt_in: true` on the scenario) both present before dialing. Only creates and reads (GET/POST); the call is driven from the staging CLONE, keeping production untouched. |
| `run_scenario` (Twilio adapter) | `POST .../Calls.json`, then polls `GET .../Calls/{sid}.json` + `GET .../Recordings.json`, then the existing `capture_twilio` download | Only when driven | Same double gate. TwiML `<Say>` is a fixed-timeline scripted caller; recording is dual-channel via `Record=true`/`RecordingChannels=dual`. GET/POST only, so config stays untouched. |

The recording download reuses `capture.py`'s validated fetch (scheme allowlist,
default-deny SSRF, cross-host credential strip, atomic write); a local test
recording server on `127.0.0.1` needs `HOTATO_ALLOW_PRIVATE_URLS=1` like every
other download. Full walkthrough: [`docs/DRIVE-A-CALL.md`](DRIVE-A-CALL.md).


================================================================================
FILE: docs/EVIDENCE-PACK.md
================================================================================

# The evidence pack

Hotato ships proof as reproducible commands, recorded audio, and deterministic,
byte-stable verdicts you can rerun on your own machine.

The standard every artifact in this pack is held to, and why it is ranked the
way it is, lives in [evidence/README.md](evidence/README.md). Read that first
if you are deciding how much to trust any single piece below.

## What's in the pack

| Artifact | What it shows | Where |
|---|---|---|
| Bundled demo | Two recorded calls a provider-default agent fails, scored offline in under a minute, one command | `hotato demo` · [START.md](START.md) |
| Recorded provider-default battery | 12 scripted calls against a live voice agent on its default settings; a missed interruption and a false stop fail in the same run | [`corpus/vapi-defaults/README.md`](../corpus/vapi-defaults/README.md) |
| Determinism check | Same recording, same numbers, every run, on every OS CI verifies (Linux proven; macOS/Windows now checked, pending a first green run) | [VALIDATION.md](VALIDATION.md) Job 1 |
| Not-scorable gallery | Eight input conditions and the exact verdict each produces, including three hard refusals | [GALLERY.md](GALLERY.md) · [TRUST-GALLERY.md](TRUST-GALLERY.md) |
| Trust contract | The input-condition table the gallery demonstrates | [TRUST-MATRIX.md](TRUST-MATRIX.md) |
| What is and is not validated | The three jobs Hotato is measured on, and the explicit does-not-claim list | [VALIDATION.md](VALIDATION.md) |
| Where Hotato fits next to a QA platform | Named-vendor routing guide | [COMPARE.md](COMPARE.md) |
| Case studies | Recorded-audio write-ups, each with a repro command and a mandatory "What Hotato did not prove" scope section | [`case-studies/`](case-studies/README.md) |
| Shareable cards | Self-contained SVGs of a candidate, a threshold-funnel finding, or a verify result | [CARDS.md](CARDS.md) · [`assets/cards/`](assets/cards/) |
| Launch-bar status | The checklist of what is done and what is still an open gap | [evidence/validation-plan.md](evidence/validation-plan.md) |

## How to check it yourself

Every artifact above is re-derivable from a command, not asserted from memory:

```bash
hotato demo --no-open --format text          # the two bundled failures
hotato run --stereo FILE.wav --expect yield  # diff two runs, get nothing
hotato trust --stereo FILE.wav               # the refusal contract, live
```

Every command above runs offline, on your machine -- reproduce it yourself
instead of taking the claim on faith.

## What counts as evidence here

Proof in this pack is reproducible: a command, a recording, a verdict you can
check yourself, ranked by the standard in
[evidence/README.md](evidence/README.md).


================================================================================
FILE: docs/EXPLAIN.md
================================================================================

# explain: root-cause-by-layer, composed from what already exists

`hotato explain` reads a finished result and turns it into a root-cause
attribution: which layer likely failed, whether a fix is safe to try, the
opposite-risk tradeoff, and a plain next command. It is compositional: no new
scoring engine. It reframes `hotato diagnose`'s per-event findings and the
same policy gate `hotato plan` enforces (a mapped knob, one bounded step, a
passing opposite-risk fixture in the battery, config-only-safe), plus, for a
contract bundle, the bundle's own trust report and policy bounds and an
attached voice trace when present.

When the evidence cannot support picking ONE root cause, explain states the
reason plainly.

```
hotato run --suite barge-in --format json > result.json
hotato explain result.json
```

## Three input shapes, auto-detected

| Input | Example | What happens |
|---|---|---|
| a run envelope | `hotato explain result.json` | full diagnose + policy-gate attribution, per failing event |
| a sweep/analyze candidate ref | `hotato explain hotato-sweep.json#1` | ALWAYS refused: a candidate carries no human label |
| a contract bundle directory | `hotato explain contracts/refund-cutoff-001.hotato` | attributed from the contract's own measurement + policy bounds, or refused when disambiguation needs a signal the bundle does not carry |

## The attribution shape

Layer-general by design, so a future layer (asr, tool, policy, latency,
handoff, ...) can be added without a version bump. Only `turn_taking` is
populated in this build:

```
{"event_id":         the failing event, or null for a battery-level finding,
 "failure_layer":    "turn_taking",
 "type":             missed_real_interruption | false_stop_on_backchannel |
                     false_stop_on_ambient_noise | slow_yield |
                     excess_talk_over | endpointing_miss | threshold_funnel,
 "turn_taking_layer": interruption_detection | endpointing,
 "confidence":       high | medium | low,
 "fixability":       safe_to_patch | needs_human | insufficient_evidence |
                     do_not_patch,
 "opposite_risk":    the tradeoff a fix on this axis trades against,
 "evidence_for":     measured fields and notes that support this attribution,
 "evidence_against":  the caveats against treating it as settled,
 "unknowns":         explicit gaps in the evidence (never silently assumed),
 "safe_next_action": one concrete next command, never an auto-apply}
```

`fixability` reuses the SAME gate `hotato plan` already enforces:

* `safe_to_patch` -- the failure maps to one setting AND the battery already
  contains a passing opposite-risk fixture on that axis (the same coverage
  `hotato plan` requires before it proposes a change).
* `insufficient_evidence` -- the failure maps to one setting but the
  opposite-risk fixture is missing, so a change could not be verified; or a
  contract bundle's own policy-bound comparison found a violation with no
  companion moment in the bundle to verify a change against.
* `needs_human` -- an audio-path problem (echo bleed).
* `do_not_patch` -- the threshold funnel: the battery missed a real
  interruption AND false-stopped on a backchannel in the SAME battery. No
  single sensitivity threshold can fix both; the fix class is
  engagement-control. This produces one COMPOSITE
  battery-level attribution (`event_id: null`, `type: threshold_funnel`) in
  addition to the two per-event attributions it explains.

## Refusals: a precise account of the gap

A refusal states exactly which gap in the evidence blocks attributing ONE root
cause:

* **a not-scorable event** -- an input problem.
* **an ambiguous slow yield** (`unknown_root_cause`, no passing opposite-risk
  fixture) -- TTS buffering, transport latency, and VAD smoothing are
  indistinguishable from one recording.
* **an echo-tagged false stop** -- the agent most likely heard its own TTS
  bleed, an audio-path problem.
* **a sweep/analyze candidate ref** -- ALWAYS refused. A candidate carries no
  human label (`yield` vs `hold`); explain prints the exact promote command
  for BOTH labels and lets a human choose.
* **a contract's false stop with no disambiguating `candidate_kind`** -- a
  `contract.json` does not carry the raw echo/ambient signal a full run
  envelope's `diagnose` has, so a false stop on `hold` could be a
  backchannel-discrimination miss, ambient non-speech noise, or echo bleed.
  Explain names the ambiguity and points back at
  `hotato run --dump-frames` / `hotato diagnose` on the original envelope for
  a definitive read.

```
{"event_id":         the refused event, or null,
 "reason":           why the evidence cannot support one attribution,
 "evidence_for":     what IS known,
 "unknowns":         the specific gap,
 "safe_next_action": one concrete next command}
```

## Contract bundles: attributed from the bundle's OWN fields, bounded

`contract.json` does not carry the scorer's raw `reasons` text a run
envelope's event does, so a contract-bundle attribution is deliberately
narrower than a full `diagnose`:

* a missed interruption (`expect: yield`, `did_yield: false`) is unambiguous
  and always attributed;
* a "yielded but still failed its policy" contract is attributed by comparing
  the MEASURED `seconds_to_yield` / `talk_over_sec` against the contract's OWN
  `policy.pass_conditions` bounds -- a bound comparison, never a guess;
* a false stop on `hold` is attributed as echo ONLY when the contract's
  `source.candidate_kind` says so (it was created `--from-candidate` an
  `echo_correlated_activity` moment); otherwise it is refused (see above);
* a single bundle carries coverage for one moment only, so a contract-bundle
  attribution caps at `insufficient_evidence` until a companion contract or
  fixture on the other axis exists.

When a voice trace is attached (`hotato trace attach`), its findings
(`traces/voice_trace.jsonl`) are folded into `evidence_for`; when it is not
attached, explain adds an explicit unknown saying so -- TTS-cancellation lag,
transport latency, and VAD smoothing stay indistinguishable from timing alone
without one. See [`TRACE.md`](TRACE.md).

## Output

* **text** (default): a plain summary, per attribution and per refusal.
* **`--format json`**: the full machine shape, schema
  `hotato.explain.v1` (`src/hotato/schema/explain.v1.json`).
* **`--html PATH`**: a self-contained report, reusing the same house style
  (`hotato.report`'s CSS and escaping) the other HTML reports use.

## Exit codes

| Code | Meaning |
|---|---|
| 0 | explained: nothing attributable (no failing or ambiguous events) |
| 1 | explained: at least one attribution or refusal was produced |
| 2 | usage error or unusable input (a bad candidate ref, a file that is not a hotato result, or an unreadable contract bundle) |

## What explain measures

Every attribution here is evidence-based -- a measurement scoped to root
cause, with its `evidence_against` and `unknowns` stated on every record.
`explain` is read-only, exactly like `diagnose` and `plan`: applying a change
stays a human decision in your own stack; see [`FIX-PLANS.md`](FIX-PLANS.md)
and [`FIX-LOOP.md`](FIX-LOOP.md) for the guarded ladder from a plan to a
proven fix.


================================================================================
FILE: docs/GALLERY.md
================================================================================

# Gallery

Every image and worked example Hotato ships, in one place, each one reproducible
with the command shown next to it.

## Cards: shareable findings

Three self-contained SVGs: offline, byte-stable from the same input, with every
font, image, and script inlined. Full spec: [CARDS.md](CARDS.md).

| Card | What it shows | Regenerate |
|---|---|---|
| ![threshold funnel](assets/cards/no-single-threshold-card.svg) | **The hero.** A battery where a missed interruption and a false stop fail together -- Hotato names the failure class (engagement-control) instead of picking one sensitivity dial. | `hotato card fix-plan.json --out card.svg` |
| ![talk-over candidate](assets/cards/talk-over-card.svg) | A measured talk-over moment: the agent kept the floor while the caller was speaking. | `hotato card hotato-sweep.json#N --out card.svg` |
| ![false-stop candidate](assets/cards/false-stop-card.svg) | A measured false-stop moment: the agent went quiet with no caller nearby to explain it. | `hotato card hotato-sweep.json#N --out card.svg` |

## The bundled demo, rendered

The self-contained dashboard `hotato demo` writes, with embedded audio and a
hear-the-bug playhead, and the terminal output next to it:

- Report: [`assets/hotato-demo-report.png`](assets/hotato-demo-report.png) ·
  [`assets/hotato-demo-report.html`](assets/hotato-demo-report.html)
- Terminal walkthrough: [`assets/hotato-demo.gif`](assets/hotato-demo.gif)

## Trust gallery: eight input conditions, eight verdicts

The full worked set, with verbatim CLI output for every row of the
[trust matrix](TRUST-MATRIX.md), lives in [TRUST-GALLERY.md](TRUST-GALLERY.md).
Summary:

| # | Condition | Verdict |
|---|---|---|
| 1 | Clean dual-channel | Scores, full confidence |
| 2 | Silent caller channel | `NOT SCORABLE`, exit 2 |
| 3 | Silent agent channel | `NOT SCORABLE`, exit 2 |
| 4 | Swapped channels | Scores, with a swap warning |
| 5 | Crosstalk / echo bleed | Scores, at lower confidence |
| 6 | Mono (mixed channel) | `NOT SCORABLE` by default, exit 2; two opt-in escapes, both indicative only |
| 7 | Backchannel candidate | Surfaced as a candidate; you label it |
| 8 | Noisy false-positive candidate | Surfaced, with the reason it is likely spurious |

## Case studies

Recorded-audio write-ups, each following
[case-study-TEMPLATE.md](evidence/case-study-TEMPLATE.md) and each carrying a
mandatory "What Hotato did not prove" scope section:

- [`case-studies/vapi-01-hard-interruption.md`](case-studies/vapi-01-hard-interruption.md)

The full index and the launch target (3 external or semi-external studies, 5
consented fixtures, 1 before/after, 1 public PR) is in
[`case-studies/README.md`](case-studies/README.md), tracked against the gap in
[evidence/validation-plan.md](evidence/validation-plan.md).

## Where this fits

This page is the visual survey. For the standard behind what counts as evidence
and how it is ranked, see [EVIDENCE-PACK.md](EVIDENCE-PACK.md) and
[evidence/README.md](evidence/README.md).


================================================================================
FILE: docs/RECAPTURE.md
================================================================================

# Recapture: proving the CURRENT agent, not just the frozen recording

`hotato contract verify` and a promoted fixture in CI both re-score the SAME
audio the fixture or contract was created from. That audio never changes, so
those checks can only fail if the recorded evidence, the policy, or the
scorer changes -- never because your deployed agent got worse. Proving the
CURRENT agent still avoids a bug requires generating a NEW recording of the
same caller stimulus and scoring that. This page is the manual walkthrough
for doing that. See the two-lane table in
[`docs/CONTRACTS.md`](CONTRACTS.md#two-lanes-what-verify-proves-depends-on-which-recording-you-feed-it)
for the guarantee each lane gives.

There is no `hotato recapture` command. Reproducing a caller's stimulus
against a live agent is not something Hotato can automate: it does not place
calls, drive a caller script, or know what a caller said. This is a human
task; Hotato's job starts again once you have a new recording.

## When you need this

- After a prompt, model, or config change you believe fixes a known bug, and
  you want proof beyond "the frozen contract still fails the same way it
  always did" (it will -- the recording never changes).
- On a schedule (weekly, before a release) to catch silent regressions the
  frozen-evidence CI gate structurally cannot see.
- Before telling a customer, a teammate, or a PR reviewer "this is fixed."

If your stack can create a staging clone (`vapi`, `retell`) and you already
have `hotato apply --clone --yes`, `hotato fix trial` automates the
before/after half of this (source vs. clone, re-captured) and folds in the
neighbouring-cases check. Use this manual walkthrough when that path is not
available, or when the recapture is on a live production agent you cannot
clone.

## Step 1: read the original contract's stimulus and policy

```bash
hotato contract inspect contracts/refund-cutoff-001.hotato
```

Note, from the printed fields (or `contract.json` directly with
`--format json`):

- `label.expected_behavior` (`yield` or `hold`) -- what the agent should do.
- `policy.pass_conditions` (`max_talk_over_sec`, `max_time_to_yield_sec`) --
  the SAME thresholds the fresh recapture must be scored against, or the
  comparison is not apples-to-apples.
- `source.category` / `source/call_metadata.json` (if `--include-identifiers`
  was used at creation) -- whatever context was recorded about the caller
  scenario. Hotato does not store a transcript or a script; if you need the
  exact words a caller used, that has to come from your own call notes,
  scripted test caller, or a human re-running the same scenario.

## Step 2: reproduce the caller stimulus against the CURRENT agent

Human-assisted is fine and expected: place the call (or run the scripted
test caller) against the agent as it is deployed TODAY, using the same
scenario that produced the original recording.

## Step 3: capture dual-channel audio

Record caller and agent on separate channels (two-channel WAV, or two
aligned mono files) -- the same input requirement every other Hotato command
has. A mixed mono recording cannot be scored reliably; see
[`docs/CONTRACTS.md`](CONTRACTS.md#the-opt-in-diarized-mono-path) for the
quality-gated diarized-mono fallback if separated channels are not
available.

## Step 4: create a NEW contract from the fresh recording, same policy

```bash
hotato contract create --stereo recapture-2026-07-10.wav --onset 41.90 \
    --expect yield --id refund-cutoff-001-recapture-2026-07-10 \
    --out contracts \
    --max-talk-over 0.6 --max-time-to-yield 1.0
```

Use the SAME `--expect` and the SAME `--max-talk-over` / `--max-time-to-yield`
values Step 1 read from the original contract's policy. `contract create`
scores immediately and refuses a not-scorable recording (exit 2) rather than
writing a meaningless bundle -- so a refusal here is itself information (the
recapture didn't produce a scorable moment, not that the agent passed).

## Step 5: verify the fresh contract

```bash
hotato contract verify contracts/refund-cutoff-001-recapture-2026-07-10.hotato
```

A pass here is the claim the frozen-recording gate cannot make: the CURRENT
agent, on a fresh recording of the same stimulus, still meets the same
labelled policy. Keep both contracts -- the original (the historical record
of the bug) and the recapture (today's evidence) -- they answer different
questions and neither substitutes for the other.

## How Hotato tells a recapture from a re-score

Every run envelope Step 3's capture produces (and every `hotato contract
create`) carries an `audio_provenance` block per event: a streamed sha256 of
the raw file bytes AND a streamed sha256 of the decoded PCM samples, plus
sample rate and frame count. This is the mechanical proof that Step 2/3
happened -- a NEW recording, not the old one replayed through a
looser threshold.

`hotato fix trial` enforces this automatically, and does not trust the string:
for every guarded fixture (the fail->pass targets AND the still-passing holds)
it VALIDATES the block (well-formed hex, plausible metadata, a top-level digest
consistent with the per-side digests), RECOMPUTES the raw and decoded-PCM
sha256 from the audio when it is present next to the envelope (a mismatch is
`refused`), and compares DECODED PCM before vs. after so a header-only edit or a
trailing-byte append cannot dress a re-score up as a fresh capture (identical
decoded audio is `refused`). It also refuses an incomplete after set (a dropped
target or hold), and downgrades to `inconclusive` -- never `improved` -- when
the identity is merely asserted: malformed, missing, or well-formed but not
recomputable because the audio was not present. An `improved` verdict is never
reachable on unverifiable evidence. See
[`docs/FIX-TRIAL.md`](FIX-TRIAL.md#fresh-capture-provenance-guard-a-re-score-is-never-a-fix)
for the full guard.

`hotato contract verify` on a frozen bundle does NOT run this guard -- by
design, per the two-lane table above, it re-scores the SAME recording on
purpose (a CI regression gate on labelled evidence, not a fix claim), so
identical audio identity there is expected, not a red flag. The guard exists
specifically where a "fix" is being claimed: `fix trial`'s before/after.

## Claim language: what each kind of evidence lets you honestly say

The same word ("verified", "fixed", "passed") means something different
depending on which of these five you are holding. Match what you say to what
you have:

| Evidence you have | How you get it | Accurate to say | Inaccurate (common overclaim) |
| --- | --- | --- | --- |
| **Historical contract only** | `hotato contract create` ran once; you are reading `contract.json` / `hotato contract inspect`, no `verify` run since | "On \[created_at], a human labeled this call and hotato measured \[timing] against that label; this is the frozen record of that one measurement." | "This proves our agent behaves correctly" -- nothing has been re-checked since capture; it speaks to that one recorded moment, not to now. |
| **Contract plus unchanged historical audio** | `hotato contract verify` re-scored the SAME `audio/event.wav` the contract was created from | "`hotato contract verify` re-measured stored evidence and it still meets its policy." (This is the literal caveat `contract verify` now prints -- see below.) | "The deployed agent no longer has this bug." Per the two-lane table in [`docs/CONTRACTS.md`](CONTRACTS.md#two-lanes-what-verify-proves-depends-on-which-recording-you-feed-it), a pass here can only fail if the evidence, policy, or scorer changed, never because the deployed agent changed. |
| **Separately captured current-agent take** | Steps 1 to 5 above: a fresh recording of the same stimulus, a new contract created from it, verified once, standalone (no paired before/after) | "The CURRENT agent, on a fresh recording of the same stimulus captured \[date], still meets the labeled policy. One data point." | "This proves the fix caused the improvement" (coincidence, not causation) or "this guarantees the next call passes too" (one recapture is one data point, not a rate -- see Limits below). |
| **Fresh take plus opposite-risk cases** | `hotato fix trial` `improved`: paired before/after with verified (recomputed, freshly distinct decoded-PCM) `audio_provenance` on every guarded target AND hold fixture (the provenance guard held) AND the hold/opposite-risk fixture still passed | "This proves the specific fresh capture scored above, at the revision it was captured from, including that a paired hold/opposite-risk case did not flip." (The literal caution `fix trial` prints -- see below.) | "This fix is now permanently verified" or "it will keep holding after future deploys." A later deploy is a new revision this report says nothing about, and nothing here re-runs itself. |
| **Production rerun after deploy** | You (there is no automatic trigger) recaptured again against the LIVE deployed agent post-deploy, per this page, on your own schedule | "As of \[date], a fresh production capture reverified the same labeled stimulus against the live agent." Still one data point per run. | "The fix is confirmed working in production, no further checks needed." This does not run in CI by itself (see Limits below); each rerun is one more independent data point, not a standing guarantee that survives the NEXT deploy. |

Two of these statements are not just guidance in this doc -- they are wired
into the command output, so a reader never has to take the caution on
faith:

- **`hotato contract verify`** (the stored-evidence row above) prints, in
  every text, HTML, and rollup render: *"This result re-measures stored
  evidence. It does not test the current agent."*
- **`hotato fix trial`**, wherever its audio-provenance section renders (an
  `improved` verdict, or a `refused`/`inconclusive` one the guard downgraded)
  prints: *"Provenance caution: this proves the specific fresh capture
  scored above, at the revision it was captured from. It does not certify a
  later deploy or every future call, and it does not re-run itself; recapture
  again after the next change."*

## Limits, stated plainly

- **This is not a controlled experiment.** A pass after a change coincides
  with the change; it does not prove causation. See
  [`docs/VALIDATION.md`](VALIDATION.md) and
  [`docs/THREAT-MODEL.md`](THREAT-MODEL.md).
- **The stimulus match is only as good as your reproduction of it.** A
  scripted test caller reproduces wording and timing more faithfully than an
  ad hoc human call; Hotato has no way to verify the fresh call
  matches the original scenario -- that judgment is yours, same as the
  original label.
- **One recapture is one data point.** A single fresh pass does not establish
  a rate; run it more than once, or fold it into a battery-scale check
  (`hotato verify --before --after`, [`docs/FIX-LOOP.md`](FIX-LOOP.md)) if
  you need a distribution rather than a single yes/no.
- **This does not run in CI by itself.** Unlike `contract verify` on the
  frozen recording, there is no automatic trigger for "go re-run the call
  against production" -- wiring that is a decision you make (a scheduled
  synthetic-caller job, a manual pre-release checklist item), not something
  Hotato ships turned on.

## What this does not stop

This is an offline tool: a user who controls every input can always lie to
themselves. Nothing on this page, and nothing `hotato fix trial`'s guard
recomputes, changes that. Specifically:

- **A fresh recording of a fabricated stimulus still passes.** If
  the "same scenario" you reproduced in Step 2 does not match the
  original bug, the audio identity check has nothing to say about it -- it
  verifies the bytes are freshly captured, never that the scenario is the
  one you claim.
- **Repacking a `.hotato` contract with a loosened policy still verifies.**
  `MANIFEST.sha256.json` is integrity (the archive agrees with itself), not
  authenticity (who approved the policy inside it). No signature is
  implemented yet.
- **A resample, re-encode, or gain change of the SAME call still reads as a
  distinct capture,** because the guard's freshness check is decoded-PCM
  difference, and those transforms change the decoded samples of a call that
  is otherwise identical. This is a known residual, not a claim broken.

See [`docs/FIX-TRIAL.md`](FIX-TRIAL.md#what-this-does-not-stop) for the same
note at length, in the context of the automated guard it applies to.

## Read more

- The two-lane distinction this page exists to close:
  [`docs/CONTRACTS.md`](CONTRACTS.md#two-lanes-what-verify-proves-depends-on-which-recording-you-feed-it)
- The automated before/after path for clone-appliable stacks:
  [`docs/FIX-TRIAL.md`](FIX-TRIAL.md) · [`docs/APPLY.md`](APPLY.md)
- Battery-scale before/after proof, coincidence not causation:
  [`docs/FIX-LOOP.md`](FIX-LOOP.md)
- What a contract does and does not prove:
  [`docs/CONTRACTS.md`](CONTRACTS.md#what-a-contract-does-not-prove)


================================================================================
FILE: docs/RELEASE-CHECKLIST.md
================================================================================

# Release checklist

Run top to bottom for every release. Each item is a gate: green means proceed.

## Code and tests

- [ ] `python3 -m pytest -q` fully green on a clean checkout.
- [ ] `python3 sync_engine.py --check` passes: the vendored `_engine` is byte-identical to upstream.
- [ ] CI is green on the release commit (`.github/workflows/tests.yml`).
- [ ] GitHub Actions stay SHA-pinned: every `uses:` in `.github/workflows/*.yml` references an action by full 40-char commit SHA with a trailing `# vX.Y.Z` comment (never a mutable `@v4` tag or `@release/v1` branch). When bumping an action, update the SHA and the comment together; do not revert to a floating tag.
- [ ] Version bumped in EVERY lockstep site: `pyproject.toml`, `src/hotato/__init__.py` (`__version__` -- this is what `hotato --version`, `hotato describe`, and stackbench provenance self-report), `server.json` (both `version` fields), `CITATION.cff` (`version` + `date-released`), the `llms.txt` version line; `CHANGELOG.md` has a dated entry for it. `tests/test_version_lockstep.py` gates the ones tests can see -- it now parses and compares `pyproject.toml`, `__init__.py`, the `describe` manifest, installed dist metadata, `llms.txt`'s `Version` line, `server.json` (top-level + each package), and `CITATION.cff`'s `version:`, so a missed bump in any of those reddens CI.

## README and assets

- [ ] Regenerate README assets before release: `python3 scripts/render_readme_assets.py`.
- [ ] Verify the README screenshot renders: `docs/assets/hotato-demo-report.png` is current, shows the failing demo summary, a timeline, and a fix card, and displays correctly in the GitHub README preview.
- [ ] README badges resolve (the `tests.yml` workflow badge reflects the current run).
- [ ] Verify the site hero matches the README: same tagline, same demo screenshot, same install command on hotato.dev.

## Adapters

- [ ] Adapter last-verified dates are fresh in `docs/ADAPTER-STATUS.md`: re-verify each capture path against the vendor's current API docs and update the date column.
- [ ] Fixmap knob names match the same verified APIs (`src/hotato/fixmap.py`).

## Package

- [ ] Build sdist and wheel from a clean tree; install the wheel in a fresh venv and run `hotato run --suite barge-in`. CI does this on every version tag in the `release.yml` `sanity` job.
- [ ] Generate + validate the SBOM(s) and attach them to the GitHub release: `python3 scripts/gen_sbom.py` writes `dist/hotato.sbom.cdx.json` (a minimal CycloneDX bill of materials for the core package and every declared dependency, generated offline straight from `pyproject.toml`, nothing to fetch). For a per-profile breakdown, `python3 scripts/gen_sbom.py --list-profiles` enumerates `core` plus each declared extra, and `python3 scripts/gen_sbom.py --profile <name>` writes `dist/hotato.sbom.<name>.cdx.json` (core alone, or core plus that one extra). Validate each with `python3 scripts/gen_sbom.py --check <file>`, then upload `dist/*.cdx.json` alongside the sdist/wheel on the GitHub release. CI produces and validates all of these automatically: `release.yml` uploads them in the `hotato-release-dist` artifact, and `publish-pypi-oidc.yml` uploads them in the `hotato-sbom` artifact.
- [ ] Publish to PyPI via **Trusted Publishing (OIDC)** -- the default path (see below): dispatch `publish-pypi-oidc.yml` with `version` = the release's `pyproject.toml` version and `confirm` = `PUBLISH`. That workflow builds reproducibly (a second build is rebuilt with a pinned backend and the two builds' `sha256sum`s are compared -- a mismatch fails the run), generates + validates the SBOMs, uploads the exact built artifacts, and then in a separate gated `publish` job downloads those exact bytes, attests build provenance over them, and uploads them to PyPI. Publishing authenticates with a short-lived OIDC token, generated fresh each run.
- [ ] Verify the build-provenance attestation for the published wheel and sdist: `gh attestation verify dist/hotato-<version>-py3-none-any.whl --repo attenlabs/hotato` (and again for `dist/hotato-<version>.tar.gz`). This checks the GitHub-signed provenance emitted by the `publish` job's `actions/attest-build-provenance` step, proving those exact bytes were built by this repo's workflow.
- [ ] Verify the published SBOM(s): re-run `python3 scripts/gen_sbom.py --check dist/hotato.sbom.cdx.json` (and each `dist/hotato.sbom.<profile>.cdx.json`) on the files attached to the release, and confirm each SBOM's `metadata.component.version` matches the release version.
- [ ] Verify `uvx hotato demo` works from the published package (fresh machine or cleared uv cache): it renders and opens the failing demo report.
- [ ] Verify `uvx hotato doctor` (self-test path) works from the published package.
- [ ] Tag the release commit and push the tag; GitHub release notes point at the CHANGELOG entry.

### Publishing path: PyPI Trusted Publishing (OIDC) -- DEFAULT

`.github/workflows/publish-pypi-oidc.yml` is the default publish path. It is a
`workflow_dispatch`-only workflow that builds (with a pinned, reproducible
backend), runs the full test suite, `twine check`s the artifacts, runs a
second-build `sha256sum` reproducibility check, generates + validates the
CycloneDX SBOMs, and publishes via
[PyPI Trusted Publishing](https://docs.pypi.org/trusted-publishers/) using a
short-lived GitHub OIDC token, minted fresh each run. It requires a `version`
input matching `pyproject.toml` and a `confirm` input equal to `PUBLISH`; the
`publish` job runs under the `pypi` GitHub Environment with `id-token: write`
and `attestations: write` granted only on that job, and attests build
provenance over the exact artifacts before uploading them.

**One-time operator action** (required before this path can publish): register
the Trusted Publisher on PyPI for `attenlabs/hotato` + workflow
`publish-pypi-oidc.yml`. Until this is done, PyPI rejects the OIDC token, so a
dispatch builds/tests/attests cleanly and stops at the upload step -- safe by
construction, since the workflow never holds a credential that could leak.
**hotato is already on PyPI**, so this is the existing-project flow, distinct
from the pending-publisher flow for unpublished names:

1. Sign in to PyPI as an owner/maintainer of the `hotato` project and open
   `https://pypi.org/manage/project/hotato/settings/publishing/`.
2. Under "Add a new publisher", choose GitHub and fill in:
   - Owner: `attenlabs`
   - Repository name: `hotato`
   - Workflow filename: `publish-pypi-oidc.yml`
   - Environment name: `pypi`
3. Save. Nothing is stored on either side; PyPI accepts only OIDC tokens
   minted by that exact repo/workflow/environment combination.
4. In the GitHub repo, create the `pypi` environment (Settings > Environments
   > New environment, name `pypi`) if it does not already exist. Optionally
   add required reviewers there for a human-approval gate in addition to the
   `confirm`/`version` input checks already in the workflow.
5. To cut a release this way: dispatch `publish-pypi-oidc.yml` with `version`
   set to the release's `pyproject.toml` version and `confirm` set to
   `PUBLISH`.

### Fallback: manual token upload (twine)

If Trusted Publishing is unavailable (e.g. the publisher is not yet registered
and a release must go out), publish by hand with a project-scoped API token:
`python3 -m twine upload dist/*` using a token scoped to the `hotato` project.
This path uses a long-lived credential and skips build-provenance attestation,
so prefer the OIDC path above; treat twine as the fallback, and still attach
the generated `dist/*.cdx.json` SBOMs to the GitHub release.

## After release

- [ ] Smoke-test the README quickstart commands exactly as written.
- [ ] File follow-ups for anything that needed a manual touch, so the next release is one pass.


================================================================================
FILE: docs/SELF-HOST.md
================================================================================

# Self-host hotato in your own cloud / VPC

Run the complete conversation-QA team workspace on infrastructure you control.
Call audio, transcripts, traces, and evaluations stay on your machine; the
default stack binds a local port and stops there. This page is the full
walk-through: build, bring it up, connect your own calls, add a local model
judge, back it up, and upgrade.

The stack is small on purpose:

- **hotato**: the `pip`-installable package. The core is stdlib-only (zero
  runtime dependencies), so the default image stays offline at run time, with
  zero added supply-chain surface.
- **`hotato serve`**: the read-only, token-authenticated team workspace (five
  views: release readiness, scenario matrix, conversation inspector, failure
  clusters, production health). See [`docs/WORKSPACE.md`](WORKSPACE.md).
- **Ollama** (optional): a local model judge for the rubric lane, opt-in behind
  a compose profile. The default path stays local end to end.

---

## Prerequisites

- Docker Engine 24+ and the Docker Compose v2 plugin, v2.24+ (`docker compose`,
  not the legacy `docker-compose`; the optional `env_file` uses the long-form
  `required:` field added in v2.24). Check with `docker compose version`.
- ~1 GB of disk for the default image; a persistent volume for `/data`.
- Build and run the default stack with Docker alone -- self-hosted, offline,
  nothing else to provide. (The optional judge model download is the one
  documented exception; see [Enable the local model judge](#enable-the-local-model-judge-optional).)

The files that make up the deployment:

| File | What it is |
|---|---|
| `Dockerfile` | Multi-stage, slim, non-root image that installs hotato from source |
| `docker-compose.yml` | The workspace service, an optional judge, and a one-shot demo seeder |
| `deploy/entrypoint.sh` | Injects the bearer token from a secret/env, then starts the server |
| `deploy/healthcheck.py` | The container HEALTHCHECK (authenticated GET over loopback) |
| `deploy/seed-demo.py` | Seeds a small, clearly-labelled example dataset (optional) |
| `deploy/verify-zero-egress.sh` | Proves the default stack makes no external calls (see below) |
| `deploy/hotato.env.example` | Optional environment (token, judge model name) |

---

## Build

From the repository root:

```bash
docker compose build
```

This builds `hotato-selfhost:local` from the local source. Two extras can be
built in with build args when you need them inside the workspace container:

```bash
# add a local faster-whisper ASR pass (offline once a model is cached):
docker compose build --build-arg WITH_TRANSCRIBE=1
# add the Ed25519 evidence-signing layer (cryptography):
docker compose build --build-arg WITH_SIGN=1
```

The default build installs the stdlib-only core only, which keeps the image
small and offline. The heavier live-capture and diarization extras run inside
your own voice pipeline instead, kept out of this container by design.

---

## Bring it up

```bash
docker compose up -d
```

The workspace is published to **host loopback only**:

```
http://127.0.0.1:8321
```

Only that one port is published, and only on `127.0.0.1`. Inside the container
the server binds `0.0.0.0:8321`, the interface a published port requires; it
prints a non-loopback-bind warning at start as an expected side effect of that
binding. The compose port mapping (`127.0.0.1:8321:8321`) is what keeps the
workspace on the host's loopback only. To reach it from your laptop, use an
SSH tunnel or a reverse proxy you control, and keep the published binding
as-is.

Open the workspace with the bearer token (see
[Credentials](#credentials-and-the-bearer-token)):

```bash
# print the token the server generated on first start
docker compose exec hotato-workspace cat /data/serve/default/token
# then open http://127.0.0.1:8321/?token=<token> once in a browser
```

Health: the container ships a HEALTHCHECK that authenticates to the server over
loopback and checks a view returns `200`. `docker compose ps` shows `healthy`
once it is serving.

---

## First boot: example data

A brand-new workspace is empty. To see the five views populated immediately with
a small, clearly-labelled **example** dataset (two releases, three scenarios, a
mix of pass / fail / inconclusive across the five dimensions, and both direct
and simulated origins), run the one-shot seeder:

```bash
docker compose --profile demo run --rm hotato-init
```

The seeder lives in the `demo` compose profile, so a plain `docker compose up -d`
leaves it out entirely. `docker compose run` targets that profiled service directly; the
`--profile demo` above makes the profile explicit.

> `hotato start --demo` writes a *sweep report* (`hotato-sweep.json` + an HTML
> dashboard) into a directory -- a separate thing from populating the workspace,
> which reads the fleet registry's entity model. The seeder writes that entity
> model through the same public API the CLI uses.

This is example data, labeled and scoped to itself -- it says nothing about
any agent you run. `origin` is set per conversation (direct vs simulated),
keeping the two in separate buckets.

To keep your own calls clear of the example rows, ingest into a **different
workspace id** (the demo lives in `default`), or reset the volume before you
start:

```bash
# your own data in its own workspace, leaving the demo in `default`:
docker compose exec hotato-workspace hotato fleet ingest --home /data -w acme ...
# or wipe everything (demo + all data) and start clean:
docker compose down -v
```

`python3 /opt/hotato-deploy/seed-demo.py --clear` prints the reset guidance; the
registry resets at the volume level, so a volume reset is the reliable way to
remove the example data.

---

## Connect your own data

The workspace reads the registry + evidence store under the `/data` volume. Your
CLI runs inside the same container, against the same `/data`, so everything the
workspace shows comes from calls you ingest.

Score and register a two-channel recording (caller on channel 0, agent on
channel 1). Mount the folder that holds your recordings, then run the CLI in the
container:

```bash
# make your recordings available at /calls inside the container
docker compose run --rm -v /path/to/your/recordings:/calls:ro \
  hotato-workspace hotato fleet ingest --home /data -w default \
  --agent support-bot /calls/one-call.wav
```

Then open the workspace: the conversation, its evidence, and any evaluations
appear in the inspector and the health view. The full lifecycle (register an
agent, run a suite, compare releases, review failures) uses the same
`docker compose exec hotato-workspace hotato …` pattern:

```bash
docker compose exec hotato-workspace hotato fleet agent add \
  --home /data -w default --agent-id support-bot --stack vapi
docker compose exec hotato-workspace hotato fleet status --home /data -w default
```

`--home /data` points the CLI at the same registry the server serves with
`--registry /data`. Reviews and labels stay CLI-driven; the workspace stays
read-only, writing only to its own audit log.

Fetching calls from a hosted voice provider (`hotato pull` / `capture`) reaches
that provider's API with credentials you supply -- an opt-in path, separate
from the default stack, listed in [`docs/EGRESS.md`](EGRESS.md).

---

## Enable the local model judge (optional)

The rubric lane can score a transcript against criteria with a **local** model.
The default judge is an Ollama daemon; enable it with the `judge` profile:

```bash
docker compose --profile judge up -d
```

The Ollama service stays off any published port: it is reachable only on the
private compose network, wired to the workspace via `HOTATO_JUDGE_ENDPOINT=http://ollama:11434`.
Pull the pinned judge model once (`qwen2.5vl:3b` is the model `hotato rubric run`
uses by default, so no `--judge-model` flag is needed later):

```bash
docker compose exec ollama ollama pull qwen2.5vl:3b
```

> **This pull downloads model weights from the internet**: the one documented
> download, like installing any package that carries model weights. It happens
> once, into the `ollama-models` volume; after that, inference runs offline.

Run the rubric lane inside the container, pointing it at a rubrics file and a
transcript you supply (author a rubric per [`docs/RUBRIC.md`](RUBRIC.md), then
mount or copy both into the `/data` volume). Because the judge endpoint hostname
(`ollama`, wired by `HOTATO_JUDGE_ENDPOINT` in the compose file) isn't loopback,
hotato's endpoint gate asks you to acknowledge it with `--judge-egress-opt-in`.
That traffic stays on the private compose network, inside the host the whole
way; the flag exists because the gate keys on the hostname, independent of
where the packet travels:

```bash
docker compose exec hotato-workspace hotato rubric run \
  --rubrics /data/rubric.json --transcript /data/transcript.json \
  --judge-egress-opt-in --judge-endpoint http://ollama:11434
```

The lane is advisory by default: a rubric FAIL is reported but the exit code
stays `0`; add `--gate` to fail CI on a FAIL. The judge uses the pinned
`qwen2.5vl:3b` model you pulled above; pass `--judge-model <id>` to pick another.
The rubric result is model-judged (`deterministic: false`) and is kept on its
own lane, apart from the deterministic assertion counts.

### Pre-seed the judge model for an air-gapped deploy

On a machine with network access, pull the model into a named volume, then move
that volume (or its backing directory) to the air-gapped host:

```bash
# on a connected machine
docker volume create ollama-models
docker run --rm -v ollama-models:/root/.ollama ollama/ollama:latest \
  sh -c "ollama serve & sleep 5 && ollama pull qwen2.5vl:3b"
# back up the volume and restore it as `hotato_ollama-models` on the target
docker run --rm -v ollama-models:/data -v "$PWD":/backup alpine \
  tar czf /backup/ollama-models.tgz -C /data .
```

With the model already in the volume, the air-gapped stack runs the judge with
no download.

---

## Credentials and the bearer token

Every request to the workspace is authenticated against one shared bearer token,
compared in constant time. Three ways to provide it (precedence high to low):

1. **A Docker secret** (best). Mount a secret file at `/run/secrets/hotato_token`;
   the entrypoint passes it as `--token-file`, so the token stays off the
   process command line. Example addition to `docker-compose.yml`:

   ```yaml
   services:
     hotato-workspace:
       secrets:
         - hotato_token
   secrets:
     hotato_token:
       file: ./deploy/hotato_token.txt   # chmod 0600
   ```

2. **An env var.** Set `HOTATO_SERVE_TOKEN` in `deploy/hotato.env` (copy it from
   `deploy/hotato.env.example`). The entrypoint writes it to a `0600` file in the
   container and passes `--token-file`, so it stays off the command line.

3. **Generated.** If you set nothing, the server generates a token with
   `secrets.token_urlsafe` on first start and stores it `0600` at
   `/data/serve/default/token`. Read it with
   `docker compose exec hotato-workspace cat /data/serve/default/token`.

The token file and the audit log are written with owner-only (`0600`)
permissions. Keep `deploy/hotato.env` and any token file `0600` on the host
too, and keep them out of version control.

---

## Backup and restore

Everything the workspace needs lives in the `/data` volume: the registry
(SQLite), the content-addressed evidence store, the serve token, and the audit
log. Back up that one volume:

```bash
# back up hotato_hotato-data to ./hotato-data-backup.tgz
docker run --rm -v hotato_hotato-data:/data -v "$PWD":/backup alpine \
  tar czf /backup/hotato-data-backup.tgz -C /data .

# restore into a fresh volume
docker volume create hotato_hotato-data
docker run --rm -v hotato_hotato-data:/data -v "$PWD":/backup alpine \
  sh -c "cd /data && tar xzf /backup/hotato-data-backup.tgz"
```

Because the evidence store is content-addressed (sha256), a restored artifact is
the same bytes that produced the original verdict; any digest mismatch is
caught and surfaced explicitly.

---

## Upgrade

The image installs hotato from the source in this repository, so upgrading is a
rebuild:

```bash
git pull                       # or check out the release tag you want
docker compose build --pull    # rebuild the image
docker compose up -d           # recreate the workspace container
```

The `/data` volume is untouched by a rebuild, so your registry and evidence
survive the upgrade. The registry re-asserts its (idempotent) schema on open. Back
up `/data` before a major upgrade, as with any stateful service.

---

## Zero-migration promise

The `/data` registry and the content-addressed evidence store use the
**same schemas** the managed cloud uses. Your conversation artifacts, conversation
tests, and dashboards move between self-hosted and cloud without changing a line.
Self-host is the same platform on your own infrastructure, with the whole QA
platform available there -- nothing here requires a hosted login.

---

## Air-gapped deployment

The default stack (the workspace alone) runs offline at run time, so it works
on an air-gapped host once the image is present. Bring the image over instead
of pulling it on the target:

```bash
# on a connected machine
docker compose build
docker save hotato-selfhost:local | gzip > hotato-selfhost.tar.gz
# move the tarball to the air-gapped host, then:
gunzip -c hotato-selfhost.tar.gz | docker load
docker compose up -d
```

This is air-gapped once its one-time steps are done: enabling the judge
profile needs the model pull described above unless you pre-seed the
`ollama-models` volume. With the image loaded and (if you use the judge) the
model volume pre-seeded, the whole stack runs fully offline.

---

## What "stays offline" covers

Scope, stated precisely so it holds up:

- The claim is about the **default stack's run-time behaviour**: the workspace
  server binds a listening socket and stops there -- it imports nothing that
  phones home, and keeps audio, traces, and evaluations on the machine. The
  workspace is read-only, writing only its own append-only audit log.
- The default workspace runs on a normal Docker bridge so its port can publish;
  the guarantee rests on the server's own behaviour, and you can verify it
  directly on your own machine, independent of any firewall:

  ```bash
  ./deploy/verify-zero-egress.sh
  ```

  It (1) confirms only `127.0.0.1:8321` is published, (2) runs the same image on
  an `internal` Docker network where egress is physically removed and shows the
  workspace still answers a view with `200` (proof the server does its job
  entirely on what's already on the machine), and (3) lists ESTABLISHED
  connections inside the running container and confirms every one stays
  internal.

- **Opt-in paths that reach the network are named explicitly, every one of
  them.** The local judge talks to the in-stack Ollama over the private
  network (and the one-time model pull downloads weights); `hotato pull` /
  `capture` fetch calls from a provider you configure; a hosted judge or the
  pyannoteAI diarizer send data off-box only behind an explicit
  `--judge-egress-opt-in` / `--egress-opt-in` flag. Every one of these is
  enumerated, command by command, in [`docs/EGRESS.md`](EGRESS.md) and
  [`docs/THREAT-MODEL.md`](THREAT-MODEL.md).

If a capability lacks its input for a given run, it returns INCONCLUSIVE --
plain and consistent, whether you self-host or not.

---

## See also

- [`docs/WORKSPACE.md`](WORKSPACE.md): the five views, auth, and the audit log
- [`docs/EGRESS.md`](EGRESS.md): every network call site, command by command
- [`docs/THREAT-MODEL.md`](THREAT-MODEL.md): the offline / opt-in split and the
  workspace's threat-model row
- [`SECURITY.md`](../SECURITY.md): posture summary and reporting a vulnerability


================================================================================
FILE: docs/SET-AND-FORGET.md
================================================================================

# Set and forget: passive turn-taking regression monitoring

`hotato sweep` turns from a command you remember to run into a job that runs
on its own schedule and only asks for your attention when it finds something
worth acting on.

Every command below is a shipped `hotato` command (verify with `hotato
<command> --help`). Try the whole loop right now with `--demo`, no
credentials needed, before pointing it at your own stack.

## The loop

```
connect (once) -> sweep (on a schedule) -> read the dashboard
   -> promote confirmed candidates into fixtures -> hotato run gates CI
```

Sweeping only reads: it lists candidate timing moments for you to judge. You
decide which ones are bugs and label them; `hotato fixture create` /
`hotato fixture promote` are what turn a candidate into a permanent test.

## 1. Connect once

```bash
hotato connect vapi --api-key <key>
# or: VAPI_API_KEY=<key> hotato connect vapi
```

This runs a lightweight live auth check (skip with `--no-verify`) and stores
the credential in `~/.hotato/connections.json`, file mode `0600`. The key
travels only to the vendor's own API, kept out of Hotato's hands entirely.
Full per-stack
credential table (Vapi, Twilio, Retell, Bland, ElevenLabs, Synthflow, Millis,
Cartesia): [`CONNECT.md`](CONNECT.md). LiveKit and Pipecat are
capture-in-your-infra; use `hotato setup --stack
livekit|pipecat` instead.

Once a stack is connected, `--stack` and the credential flags are optional
on `pull` / `sweep` whenever exactly one stack is connected.

## 2. Sweep on a schedule

Run the same command your terminal runs interactively, from cron or CI:

```bash
hotato sweep --stack vapi --since 7d --format json > hotato-sweep.json
hotato sweep --stack vapi --since 7d --out hotato-sweep.html --no-open
```

- `--format json` is the machine-readable candidate list `fixture promote`
  reads; redirect it to a file every time (`--out` writes the HTML
  dashboard; capture stdout for the JSON).
- `--out FILE.html --no-open` writes the shareable dashboard without popping
  a browser, the right mode when nothing is watching the screen.
- `--since 7d` scopes the pull to recent calls; a nightly job narrows this to
  `--since 1d` so it only re-scores what came in since the last run.

A crontab entry, 03:00 daily:

```
0 3 * * * cd /path/to/repo && hotato sweep --stack vapi --since 1d --format json > "reports/sweep-$(date +\%F).json" && hotato sweep --stack vapi --since 1d --out "reports/sweep-$(date +\%F).html" --no-open
```

See [`examples/set-and-forget/`](../examples/set-and-forget/README.md) for a
runnable version of this, plus the CI half of the loop.

No stack connected yet? Everything above works with `--demo` instead of
`--stack ... --since ...`, credential-less, against two bundled recorded calls:

```bash
hotato sweep --demo --format json > hotato-sweep.json
hotato sweep --demo --out hotato-sweep.html --no-open
```

## 3. Read the report

The HTML dashboard (the default `--format`) is one self-contained file:
every candidate moment across every swept call, ranked by salience, with the
hear-the-bug audio player embedded for the top `--audio-top` (default 8) so
you can listen before deciding anything. Calls that could not be scored
(mono/mixed stacks without `--allow-mono`, an unreadable file) list under
Skipped with the reason -- a fully logged skip.

The JSON (`--format json`) is the same candidate list as structured data.
Each entry carries the source recording, the timestamp (`t_sec`), the kind
(`agent_stop_no_caller`, `overlap_while_agent_talking`, ...), and a salience
score -- a timing fact for you to judge. You decide, per candidate, whether
the agent should have yielded or held.

## 4. Promote a confirmed bug into a fixture

Once you have listened to a candidate and decided what should have
happened, `hotato fixture promote` turns it into a permanent regression test
in one command, no `--stereo` / `--onset` needed, since the candidate ref
already carries the recording and the moment:

```bash
hotato fixture promote hotato-sweep.json#3 --expect yield \
    --id refund-cutoff-001 --out tests/hotato
```

`CANDIDATE_REF` is `FILE#N` (the Nth candidate, ranked order, matching the
report's numbering) or `FILE#CALL:N` (the Nth candidate from one call, named
by its source file or pulled call id), for example `hotato-sweep.json#3` or
`hotato-sweep.json#call_abc123:2`. `--expect yield` means the agent should
have stopped for the caller; `--expect hold` means it should have kept
talking through a backchannel. The fixture is scored immediately: a
candidate that isn't scorable is refused (exit 2), so only a scorable
candidate becomes a fixture.

This is the most important step in the loop. It is how a suspicious moment
becomes a fact your CI enforces forever, instead of something you noticed
once and forgot. (`hotato fixture create --stereo ...
--onset ...` does the same thing from a raw recording and a timestamp you
already know, if you are not starting from a sweep/analyze result.)

## 5. Gate CI on your fixtures

Every promoted fixture lives under `--out DIR` (`tests/hotato/scenarios` +
`tests/hotato/audio` above) and scores with the same command whether it runs
on your laptop or in CI:

```bash
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio --format json
```

Exit code `1` means a regression is pinned; `0` means every promoted fixture
still passes. Wire that into a GitHub Action (drop-in workflow at
[`.github/workflows/hotato.yml`](../.github/workflows/hotato.yml), guide at
[`docs/CI.md`](CI.md)) or into an existing pytest run with one flag
(`pytest --hotato-suite --hotato-suite-scenarios tests/hotato/scenarios
--hotato-suite-audio tests/hotato/audio`, see [`docs/PYTEST.md`](PYTEST.md)).
[`examples/set-and-forget/`](../examples/set-and-forget/README.md) has a
complete cron + CI pairing.

## Worked example (zero setup, verified end to end)

Every command above works right now on the bundled demo calls, so you can
see a failure get pinned before wiring anything to your own stack:

```bash
hotato sweep --demo --format json > hotato-sweep.json
hotato fixture promote hotato-sweep.json#2 --expect yield \
    --id demo-missed-interruption --out tests/hotato
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio
# exit code 1: the demo agent never yielded for an interruption -- pinned
```

That last `run` prints the fix card too (fix class, the config knob, the
direction to move it), because the fixture that just failed is a
labelled bad-agent moment from the bundled demo battery.

## What you control in this loop

- Labeling, fixture creation, and threshold tuning are each a command you
  run, for a candidate you listened to.
- `sweep` and `ingest` (the webhook-driven version of this same loop, see
  [`docs/INGEST.md`](INGEST.md)) both report candidates as timing facts;
  fixtures scored with `hotato run` are what produce a pass/fail verdict.
- This runs entirely as processes you control: cron, CI, and a webhook
  handler all just shell out to the same CLI, on a schedule that's yours to
  own.


================================================================================
FILE: docs/STATE-ADAPTERS.md
================================================================================

# State adapters: grounding a `state` assertion in your system of record

The `state` and `state_change` assertion kinds are **Authority 2** (post-call
state verification). They query a **system of record** after the call and
compare the actual state against what the agent said ("I issued the refund").
A state adapter is the small, pluggable seam that does the query.

`hotato test run --state FILE` loads a state adapter from `FILE`. Three adapters
ship, and the same one method backs all of them:

```
query(resource, **filters) -> dict | None
```

Return the record for `resource` matching every `filters` key, or `None` when
the system of record can be read and holds no such record.

## The three-outcome boundary (why this exists)

A `state` query verifies post-call system state. It has three outcomes, and the
distinction is the whole point:

- **The record is present and matches** -> `PASS`.
- **The system of record was read and holds no such record** (or a
  field does not match) -> a grounded `FAIL`. The agent claimed something the
  record does not confirm.
- **The system of record could not be reached or read** (network error,
  timeout, a 5xx, a non-JSON response, a DB error) -> `INCONCLUSIVE`, with a
  reason every time.

A `state` assertion is satisfied by the query result alone: a lookup plus a
dict comparison, deterministic, with no model path in a state adapter.

## The three adapters

### 1. `mock`: the local sandbox (default, offline)

A JSON or SQLite fixture: `{resource: rows}`. Runs offline, byte-stable, ready
to use with no opt-in. Use it for regression suites and for a captured before/after snapshot
(`state_change` reads a `{"before": [...], "after": [...]}` shape). This is the
adapter `--state some_sandbox.json` and `--state some.sqlite3` select today.

### 2. `http`: a REST system of record

Queries your API over stdlib `urllib`. A **resource map** turns a query into one
request:

```json
{
  "adapter": "http",
  "egress_opt_in": true,
  "base_url": "https://api.yourcompany.com/v1",
  "auth": { "type": "bearer", "token_env": "RECORDS_API_TOKEN" },
  "timeout": 30,
  "resources": {
    "appointment": {
      "path_template": "/patients/{patient_id}/appointment",
      "method": "GET",
      "params_map": { "status": "appt_status" },
      "response_pointer": "data/appointment"
    }
  }
}
```

`query("appointment", patient_id="P1", status="booked")` fills `{patient_id}`
into the path (URL-encoded), sends the remaining filters as query params (GET) or
a JSON body (POST) under their `params_map` wire names, then extracts the record
dict at `response_pointer` from the JSON response.

- `response_pointer` is a JSON-pointer-ish path (`/`- or `.`-separated; digits
  index a list; empty means the whole body is the record). A pointer that
  resolves to nothing on a well-formed response means "no such record" -> `None`.
- `method` is `GET` (default) or `POST`.
- **HTTPS is required by default.** A plain `http://` base URL is refused unless
  you set `"allow_http": true` (only for a trusted local endpoint); a state
  query would otherwise send filter values and the auth header in cleartext.
- A 404 means the addressed record does not exist -> `None` -> grounded FAIL.
  Every other non-2xx, and any network/timeout/non-JSON failure, is
  INCONCLUSIVE (`query` raises `StateAdapterError`, which the assertion engine
  turns into an INCONCLUSIVE result; the structured cause is on the adapter's
  `last_error`).

### 3. `sql`: a SQL system of record

Queries a database with a **parameterized, read-only** SELECT.

```json
{
  "adapter": "sql",
  "sqlite_path": "records.sqlite3",
  "resources": {
    "refund": {
      "query": "SELECT order_id, status, amount FROM refunds WHERE order_id = ? AND status = ?",
      "params_order": ["order_id", "status"]
    }
  }
}
```

`query("refund", order_id="O1", status="issued")` binds `[order_id, status]` as
the statement's parameters and returns the first row as a `{column: value}` dict,
or `None` when no row matches.

- Connection source (exactly one): `sqlite_path` (a local file DB, fully local,
  no egress), a `dsn` + `driver` (a DBAPI module to import and `connect`, e.g.
  `psycopg2` for PostgreSQL, a network DB, so it needs `egress_opt_in: true`;
  the driver is imported lazily and is never a hard dependency of hotato), or a
  caller-supplied `connection` object (Python API only).
- Use the placeholder style your driver expects (`?` for sqlite3, `%s` for
  psycopg2). Hotato only passes the bound value sequence through.
- A DB/driver error is INCONCLUSIVE (never a guessed FAIL).

## Injection safety and read-only discipline (enforced + tested)

- **Injection-safe by construction.** Filter values are ALWAYS bound as query
  parameters; they are never string-interpolated into the SQL. A filter value
  carrying SQL (`"O1'; DROP TABLE refunds; --"`) is treated as a literal: it
  matches no row and the table is untouched. Tested in
  `tests/test_state_adapters_real.py`.
- **Read-only.** Every mapped SQL query is validated at construction (and
  re-checked before each execute): it must begin with `SELECT` (or a
  `WITH ... SELECT` CTE), carry no second statement after a `;`, and contain no
  data-modifying keyword. A `DELETE`/`UPDATE`/`DROP`/… mapped query is rejected.

## Egress opt-in (the network adapters are refused without it)

The `http` adapter, and a `sql` adapter over a `dsn`, are **network paths** (see
[`docs/EGRESS.md`](EGRESS.md) and [`docs/THREAT-MODEL.md`](THREAT-MODEL.md)).
`load_state_adapter` **refuses** them unless the config sets
`"egress_opt_in": true`, an explicit, per-config opt-in, the same discipline
`--egress-opt-in` applies to the hosted diarizer and `--notify`. Without it, you
get a clear usage error (the CLI's exit-2 path) before any connection is made.
A local `sqlite_path` SQL DB and the mock sandbox run entirely on your
machine, opening no socket, with no opt-in needed.

When an `http` / non-local-`sql` query does fire, only the mapped filter VALUES
leave the machine (in the URL, query string, or JSON body). Audio, transcript,
and the config file itself stay on your machine.

## Credentials

Adapters take environment-variable **names** for credentials, keeping the
config file free of secrets so it can be committed or shared. Supply the
secret at run time from the environment (for example, `source` a `0600` file
that exports it):

- `bearer`: `{ "type": "bearer", "token_env": "RECORDS_API_TOKEN" }`
- `basic`: `{ "type": "basic", "username_env": "DB_USER", "password_env": "DB_PASS" }`
  (`username` may be inline since it is not a secret).
- `header`: `{ "type": "header", "headers": { "X-Api-Key": { "env": "API_KEY" }, "X-Tenant": { "value": "acme" } } }`
  where each header value is either `{ "env": NAME }` (a secret from the environment)
  or `{ "value": LITERAL }` (a non-secret constant).

A missing credential env var fails fast at construction with a message that
names the variable, not the value. Credential values stay out of logs and out
of `last_error`.

## `state_change` and before/after snapshots

`state_change` reads a `before` and an `after` snapshot to measure a delta. Only
the **mock** adapter provides both (from a captured `{"before": ..., "after":
...}` fixture, or `<table>__before` / `<table>__after` SQLite tables). The `http`
and `sql` adapters answer point-in-time `state` queries (a live API/DB exposes
only "now"), so their `before` snapshot is reported absent, keeping a
`state_change` against them grounded at INCONCLUSIVE instead of a fabricated
"no change".

## See also

- [`docs/ASSERTIONS.md`](ASSERTIONS.md): the `state` / `state_change` assertion kinds.
- [`docs/EGRESS.md`](EGRESS.md), [`docs/THREAT-MODEL.md`](THREAT-MODEL.md): the network rows.
- `src/hotato/state_adapter.py`: the adapters; `tests/test_state_adapters_real.py`: the tests.


================================================================================
FILE: docs/THREAT-MODEL.md
================================================================================

# Threat model

Hotato is built so that the sensitive thing (your call recordings) stays on your
machine, and every network action is one you named. This page is the precise
split: which commands are offline-only, which reach the network and only when you
ask, and the guarantees that hold either way.

## Three guarantees

1. **Every production change happens through a proposal you apply yourself.**
   `plan` and `patch` produce a proposal; `apply` operates on a fresh staging
   clone and dry-runs by default. Pushing a config to production is a step
   you take, not one any command takes on its own.
2. **Recordings stay on your machine.** Self-hosted, with audio moving only
   from your own stack to your own disk, and only when you run a
   pull/capture/sweep. The one exception is explicit and opt-in: the hosted
   `--diarizer pyannoteai` backend, which requires `--egress-opt-in` before any
   audio leaves the machine.
3. **Hotato treats a webhook payload strictly as data.** The `ingest` webhook
   worker reads a completed-call notification as **data**: it extracts a
   recording reference and scans it. Payload fields are read, never executed,
   shelled out, or used to choose what code runs.

## Core: fully offline, on your machine

These commands read and write only the local files you point them at, opening
no sockets. This is the whole scoring, analysis, and fixture surface.

| Command | What it touches |
|---|---|
| `run` | Score a local WAV (or the bundled self-test battery). |
| `scan` | List candidate moments in a local recording. |
| `trust` | Input-health check on a local recording. |
| `report` | Render a self-contained HTML report from local run data. |
| `fixture` | Create / promote a local moment into a regression fixture. |
| `compare` | Score a before/after pair of local takes. |
| `verify` | Roll up before/after run envelopes on disk. |
| `diagnose` | Explain a finished local run envelope (read-only). |
| `plan` | Combine a diagnosis + config into a proposed fix plan JSON. |
| `patch` | Render a fix plan into a paste-ready patch. Never applies it. |
| `demo` | Run the packaged two-call battery. Zero extra files. |
| `team`, `export`, `benchmark` | Aggregate / export local run data. |
| `setup`, `describe`, `init` | Print recording config, emit the CLI manifest, scaffold local integration files. |
| `rubric run`, `rubric calibrate` | Score a local transcript against a rubric with a **LOCAL** model (Ollama at `localhost`). Default path opens no off-box socket; the verdict cache is local. Reaches the network only with an explicit hosted / non-local judge (see below). |
| `test run` (rubric lane) | Same LOCAL model judge as `rubric run`, run inline on a conversation-test's `assertions.rubric` lane. Local by default. |

Default retention is local-only: reports, envelopes, and exports are written
exactly where you point them.

## Network: only when you explicitly request it

These commands reach outside the machine, and only because reaching out is their
job. Each requires you to name a stack, a repository, or a webhook you configured.

| Command | Network surface | Notes |
|---|---|---|
| `connect` | none at connect time | Stores a stack's credentials locally at `0600`. Setup for the pull path; no audio moves. |
| `pull` | your voice stack's API | Bulk-fetch recent recordings from a stack **you** connected, into a local folder. |
| `capture` | your voice stack's API | Fetch and score one real call from your stack. |
| `sweep` | your voice stack's API | `pull` + analyze in one command. |
| `inspect` | your voice stack's API | Read (never write) the current turn-taking config. Read-only. |
| `ingest` | a webhook endpoint you host | Worker that scans each completed call. Payloads are data, not instructions (guarantee 3). |
| `issue` | GitHub, via your local `gh` | File a sweep's candidates as an issue. Uses your existing `gh` auth. |
| `pr` | GitHub, via your local `gh` | Open a PR adding promoted fixtures. Uses your existing `gh` auth. |
| `apply` | a git clone you point it at | Applies a patch to a fresh **staging** clone only, never the source. Dry-run by default; refuses a both-axes threshold funnel. |
| `--diarizer pyannoteai` | Attention Labs hosted diarizer | The only AUDIO path that can send audio off-box, and only with `--egress-opt-in`. The default diarizer is local. |
| `--judge-provider hosted` / non-local `--judge-endpoint` (any rubric command) | a hosted or remote model host you name | Sends the transcript + rubric criterion off-box for judging. Refused (exit 2) unless `--judge-egress-opt-in`. The default judge is a LOCAL Ollama model that stays on the box. See [`docs/EGRESS.md`](EGRESS.md) and [`docs/RUBRIC.md`](RUBRIC.md). |
| `test run --state` **http adapter** | your system-of-record's REST API | Only when the state-config names `adapter: http`, and only with `egress_opt_in: true` in that config. Sends the mapped filter VALUES for one `state`/`state_change` query -- audio, transcript, and the config itself stay local. See [`docs/STATE-ADAPTERS.md`](STATE-ADAPTERS.md). |
| `test run --state` **sql adapter over a `dsn`** | your database, over the network | Only when the state-config names `adapter: sql` with a `dsn`, and only with `egress_opt_in: true`. A parameterized, read-only SELECT with the mapped filter values bound as data. A local `sqlite_path` opens no socket. |

The state adapters read a post-call **system of record** to ground an Authority-2
`state` assertion (did the refund/appointment get written?). A record
the system of record can be read and does not hold is a grounded FAIL; a system
of record Hotato could **not** reach or read (network error, timeout, 5xx,
non-JSON) is INCONCLUSIVE, never a fabricated verdict. Credentials come from
environment-variable **names** in the config, keeping the secret itself out of
it, and the HTTP adapter refuses a plain-`http://` base URL unless
`allow_http: true` is set for a trusted local endpoint. The default `--state`
path is the local mock sandbox (a JSON/SQLite fixture) and a local
`sqlite_path` SQL DB, both running entirely on-machine, with no opt-in needed.

Notify surfaces (Slack, GitHub) fire only through credentials you configured
(`gh`, a Slack token) and only for actions you invoked -- every integration is
one you triggered.

## Network trust: proxies and TLS

Every networked command above (`pull`, `capture`, `sweep`, `inspect`, `apply`,
and the credential probe in `connect`) makes its HTTP calls through Python's
standard `urllib`, which honors the `HTTP_PROXY` / `HTTPS_PROXY` / `NO_PROXY`
environment-variable convention, the same convention `curl`, `pip`, `git`, and
`docker` follow. That is deliberate: it is what lets Hotato work behind a
corporate proxy with no extra flags. It also means an environment variable set
for the Hotato process controls where its outbound credentialed requests are
routed, and that is a trust decision worth stating plainly instead of leaving
implicit.

Two things bound how far that ambient trust reaches:

- **TLS certificate validation is always enforced.** Every credentialed base
  URL in `capture.py` and `apply.py` is hardcoded `https://` (`api.vapi.ai`,
  `api.retellai.com`, `api.twilio.com`, and the other supported stacks). A
  proxy set via `HTTP_PROXY`/`HTTPS_PROXY` can see the `CONNECT` target host
  and port, and can refuse or stall the connection (a denial of service), but
  reading or rewriting the `Authorization` header or the response body
  requires also presenting a certificate the machine's own trust store
  already accepts for that vendor's domain, a much larger compromise (a rogue
  trusted root CA already installed) outside `HTTP_PROXY`'s reach and
  outside Hotato's control.
- **The threat prerequisite is already a compromised local environment.** An
  attacker able to set environment variables for the Hotato process can
  already read `VAPI_API_KEY` / `RETELL_API_KEY` / etc. straight out of the
  environment, or read the `0600` credentials file `connect` wrote, both
  strictly easier than standing up a TLS-valid proxy.

If you do not trust the ambient proxy environment a command will run in, set
`HOTATO_NO_PROXY=1` to make Hotato's HTTP opener ignore `HTTP_PROXY` /
`HTTPS_PROXY` for that run, or unset those variables before invoking the
command. The default is unchanged: proxy env vars are honored, matching
curl/pip/git, so a legitimate corporate-proxy setup keeps working with no
configuration.

## Hotato's containment guarantees

- **Recordings stay on the machine by default.** Audio leaves only when you run
  `pull`/`capture`/`sweep` with `--egress-opt-in`; the default diarizer is
  local.
- **A call webhook is read as data, never as commands.** `ingest` parses a
  payload for a recording reference and stops there: payload fields are read,
  never evaluated, so a malicious webhook body cannot make Hotato run
  arbitrary code (guarantee 3).
- **Production changes always go through you.** No command writes to a live
  stack's config directly; the furthest Hotato goes is a proposed patch you
  apply to a staging clone yourself.

## Verifying the posture

You can confirm the posture directly, without trusting this page:

- **Self-hosted, nothing to audit but the standard library.** `pip install
  hotato` pulls zero runtime dependencies; the only network code path is the
  one you explicitly invoke.
- **Credentials at `0600`.** `connect` writes stack credentials locally with
  owner-only permissions; check them with `ls -l` on your credentials path.
- **Local-only retention.** Reports, envelopes, and exports exist exactly where
  you wrote them.
- **Egress is opt-in.** The only off-box audio path is `--diarizer pyannoteai`,
  gated by `--egress-opt-in` -- audio stays on the machine until you set it.

## Drive-a-call (`run_scenario`): originating a real call

`run_scenario` (`src/hotato/drive.py`, wired into the Vapi and Twilio adapters)
places a REAL outbound call against a live agent and pulls the recording. Its
threat surface and the controls on it:

- **A real, billable call always requires explicit opt-in.** `run_scenario`
  places the call only when BOTH real credentials AND an explicit egress
  opt-in (`HOTATO_DRIVE_OPT_IN=1` or `egress_opt_in: true` on the scenario) are
  present. Absent either, it raises a clean structured refusal and dials
  nothing -- matching the opt-in posture of `--allow-mono` /
  `HOTATO_ALLOW_PRIVATE_URLS` / `--egress-opt-in`.
- **Production config stays untouched.** The surface is create-and-read only:
  the only verbs issued are `POST` (create the call) and `GET` (poll status,
  list the recording, download it), with no `PUT`/`PATCH`/`DELETE` path to
  alter an assistant, a phone number, or any other provider resource in place.
  For Vapi the call originates FROM the staging clone the experiment created,
  keeping the production source untouched.
- **The caller side is labelled precisely.** The produced conversation carries
  `origin.kind == "real"` with the provider and its call id, and
  `origin.caller` (`scripted-twiml` for the fixed-timeline Twilio caller,
  `assistant-originated` for a Vapi call the assistant placed), stating
  exactly what happened -- no claim that a human placed the call or that the
  scripted caller reacted to the agent.
- **The recording download inherits every capture-side control.** The vendor
  URL (`stereoUrl` / `RecordingSid` media) flows through the same validated
  download as `capture`: http(s)-only scheme allowlist, default-deny SSRF
  (loopback/private/metadata refused unless `HOTATO_ALLOW_PRIVATE_URLS=1`),
  cross-host `Authorization`-strip on redirect, and atomic write. The API key is
  never attached to a pre-signed media URL from the vendor's JSON response.

See [`docs/DRIVE-A-CALL.md`](DRIVE-A-CALL.md) and the drive-a-call rows in
[`docs/EGRESS.md`](EGRESS.md).

## Team workspace (`hotato serve`): a new local listening socket

`hotato serve` (`src/hotato/serve/`, wired into the CLI) opens a NEW local HTTP
listening socket to serve the five conversation-QA views over the fleet registry
+ evidence store. Its threat surface and the controls on it:

- **Localhost-default bind.** The server binds `127.0.0.1` unless the operator
  explicitly passes `--host`. A non-loopback bind (e.g. `--host 0.0.0.0`)
  always prints a prominent warning at start and always requires that explicit
  flag.
- **Token auth on every request.** A shared bearer token authenticates every
  request, compared in constant time (`hmac.compare_digest`). It is either
  operator-supplied (`--token` / `--token-file`) or generated with
  `secrets.token_urlsafe` and stored `0600` under the per-workspace state dir.
  Browsers bootstrap an in-memory, HttpOnly session cookie from the printed
  `/?token=…` URL (then the token is stripped from the address bar via a
  redirect); the token stays out of every response body. An unauthenticated
  request gets `401` before it is routed anywhere.
- **Read-only.** The server issues only `SELECT`s against the registry and reads
  evidence blobs by digest, exposing only reads: no write endpoint, no
  workspace mutation. Reviews and labels stay CLI-driven. The ONLY file it
  writes is the append-only audit log (`…/serve/<workspace>/audit.jsonl`,
  `0600`), which records who (token/session prefix, the secret stays out of
  it), what (method + path, the token stripped from the query), when, and the
  response status of every request.
- **Zero egress.** The server only binds a listening socket, opening no
  outbound connection and importing nothing that phones home -- audio,
  traces, and evaluations stay on the machine. A test whitelists loopback and
  fails if any view attempts an external connection.
- **Evidence renders as data, never as executable content.** The raw evidence
  endpoint (`/evidence/<digest>`) serves blobs as `text/plain` with
  `X-Content-Type-Options: nosniff` (and the pages set `X-Frame-Options: DENY`,
  `Referrer-Policy: no-referrer`), keeping a crafted evidence blob inert in the
  viewer. Redacted transcript/trace content is scrubbed at the data layer, so
  both the HTML and the JSON mirror render only the `[redacted]` placeholder.
- **Workspace id sanitized against path traversal.** The workspace id is used
  verbatim only in parameterized SQL; as a state-directory name it is
  sanitized to stay inside the registry home.

See [`docs/WORKSPACE.md`](WORKSPACE.md).

Security policy and reporting: [SECURITY.md](../SECURITY.md).


================================================================================
FILE: docs/TRANSCRIBE.md
================================================================================

# `hotato run --stereo call.wav --transcribe`: read a transcript next to the score

Hotato's gold reference is a deterministic **energy VAD** over the audio
waveform: `did_yield`, `talk_over_sec`, and `seconds_to_yield` come from frame
energy, nothing else, and every published/golden number is computed exactly
that way. The opt-in `[transcribe]` extra runs an off-the-shelf speech-to-text
model over the same recording and hands back plain text with per-segment
timestamps, so a human (or an agent) reading a report can see WHAT was said
next to WHEN the timing engine says it was said.

That is the entire feature. A transcript is attached as **context** beside
the score, computed strictly after scoring is final. The timing score stays
grounded in the audio: the transcript rides alongside it, never feeding back
into it, and carries no accuracy claim of its own.

## What stays unchanged

- **The timing score stays grounded in the audio.** `did_yield` /
  `talk_over_sec` / `seconds_to_yield` / every other verdict and measurement
  field is computed identically whether or not `--transcribe` is passed.
  Adding `--transcribe` to a run produces byte-identical timing numbers to
  the same run without it -- this is pinned by a test
  (`tests/test_transcribe.py`), not just asserted in prose.
- **Plain text, not a quality judgment.** The transcript hands back what was
  said, not a grade on whether it was said well; see "Not transcript
  scoring" in the README. What's reported for the ASR model is the plain
  text and its timestamps -- not a WER, accuracy, or quality number, here or
  anywhere in hotato.
- **Timestamps only, no speaker attribution.** The transcript is plain text
  with timestamps, with no caller/agent attribution -- anonymous speaker
  separation lives in a separate, unrelated extra, see
  [`docs/DIARIZE.md`](DIARIZE.md).

## Quickstart

```bash
# Install the opt-in extra (local, offline; no gated token, unlike [diarize]):
pip install 'hotato[transcribe]'

# Score a call and attach a transcript next to the verdict:
hotato run --stereo call.wav --transcribe --format json
```

With the extra absent, `--transcribe` always fails loud: a clean error, exit
`2`, never a silent skip of the transcript.

## Usage

```bash
hotato run --stereo call.wav --transcribe                                   # base.en, device auto-detected
hotato run --stereo call.wav --transcribe --transcribe-model small.en \
  --transcribe-device cpu                                                    # pick a model + device
hotato run --mono call.wav --diarize --transcribe                            # also works on the diarized-mono path
```

`--transcribe` needs a **single** audio file to run ASR over: `--stereo`, or
`--mono` when scoring through the opt-in `--diarize` front-end (it transcribes
the same mono file the diarizer separated). Two separate `--caller`/`--agent`
files raise a clean usage error naming `--stereo`, a clear signal instead of a
guess at which channel to transcribe, or a silent drop of one.

Combining `--transcribe` with the bundled self-test battery (`--suite` /
`--scenarios`+`--audio`) raises a clean usage error, not a silent no-op, so a
CI script never mistakes "flag ignored" for "flag applied."

## Models and device

- Default model: `base.en` (English-only, the fastest of the useful sizes).
  Override with `--transcribe-model NAME` -- any name or local path
  `faster-whisper`/CTranslate2 accepts.
- Default device: `auto` -- picks `cuda` if CTranslate2 reports a CUDA
  device, else `cpu`. `compute_type` defaults to `float16` on GPU and `int8`
  on CPU, CTranslate2's documented fast/accurate defaults for each device,
  not an accuracy claim; both are overridable.
- Every choice used is stamped in the output
  (`transcript.model` / `transcript.device` / `transcript.compute_type` /
  `transcript.language`), so a report is reproducible.

## Backend and extra

| extra | brings in | where it runs |
|---|---|---|
| `[transcribe]` | `faster-whisper` (a CTranslate2 re-implementation of OpenAI's Whisper) | local, CPU-viable, GPU-optional, fully offline once the model is cached |

```toml
transcribe = ["faster-whisper>=1.0"]
```

This extra keeps the core's Python floor at `>=3.9` (unlike `[diarize]`),
since `faster-whisper`/CTranslate2 support that same range. Unlike the gated
diarization model, there's no model card to accept here either (see the
egress note below).

### Model licenses (log per FTO note)

Using an off-the-shelf ASR model is integration, orthogonal to any Hotato IP
claim, but the dependency licenses are logged here:

- `faster-whisper` -- **MIT** (code)
- `CTranslate2` -- **MIT** (the inference runtime `faster-whisper` runs on)
- Whisper model weights -- **MIT** (OpenAI)

## The invariant: context beside the score, grounded in the audio

A transcript is computed strictly **after** the score is final, over the
same audio the score already used, and is attached as context on a new
top-level key of the envelope -- nothing already there is read or rewritten:

- `events`, `verdict`, `measurements`, and `signals` are the exact same
  values with or without `--transcribe`.
- The energy VAD's frame-level activity stays the ground truth for timing;
  a transcript word boundary is a separate thing -- ASR word timestamps are
  themselves a model estimate, read alongside it, never substituted in.
- A missing/broken `[transcribe]` extra raises a clean, actionable error --
  exactly like the `[neural]` and `[diarize]` seams: never a bare
  `ImportError`, a silent skip, or a fallback to a different backend.
- `hotato.transcribe` and the vendored `_engine` import nothing from each
  other, keeping the transcript path fully separate from the scorer.

## JSON shape (agents)

A `--transcribe` run is the SAME envelope as any single run, plus one
additive top-level `transcript` block. `events[0].verdict` is identical to a
run without `--transcribe`:

```json
{
  "mode": "single",
  "events": [{
    "verdict": {"did_yield": true, "seconds_to_yield": 0.42, "talk_over_sec": 0.18, "reasons": []}
  }],
  "transcript": {
    "text": "I need to check on my refund -- hold on, let me pull that up for you.",
    "segments": [
      {"start": 0.10, "end": 2.30, "text": "I need to check on my refund"},
      {"start": 2.60, "end": 4.95, "text": "hold on, let me pull that up for you."}
    ],
    "model": "base.en",
    "device": "cpu",
    "compute_type": "int8",
    "language": "en"
  }
}
```

Branch on the presence of `transcript`: present only when `--transcribe` was
passed, and additive only -- the meaning of everything under `events` stays
the same. On the diarized-mono path, `transcript` still attaches even when
the diarization confidence gate refused or downgraded the verdict; ASR runs
independent of whether diarization succeeds.

## Egress

Fully local at inference time: once the named model is cached, transcribing
opens no socket. The **first** run of a model you have not used before
downloads its weights from its public host (the same one-time fetch as
installing any pip package with model weights); every run after that is
offline. See [`docs/EGRESS.md`](EGRESS.md) for the full per-command network
table.


================================================================================
FILE: docs/TRUST-GALLERY.md
================================================================================

# Trust gallery: eight recordings, eight verdicts

Eight named input conditions, each with the `hotato` output it produces.
Every block below is verbatim CLI output from hotato 1.3.3, offline. The rows
correspond to the [trust matrix](TRUST-MATRIX.md).

The clean case and the echo case use the bundled examples
`01-hard-interruption.example.wav` and `07-echo-bleed.example.wav` (shipped in
`src/hotato/data/audio/`), and the backchannel case uses the bundled
`02-backchannel-mhm.example.wav`. The four defect cases (silent caller, silent
agent, swap, mono) are the deterministic synthetic two-channel fixtures the test
suite pins in `tests/test_trust.py`, each isolating one input defect, so every
verdict reproduces byte-for-byte from the shipped builders.

---

## 1. Safe stereo: the good case

Clean two-channel export, caller on ch0, agent on ch1. `trust` clears it.

```text
$ hotato trust --stereo 01-hard-interruption.example.wav
hotato trust: 01-hard-interruption.example.wav
  recording: 6.0s, 16000 Hz, 2 channels
  caller (ch0): 2.46s speech, first at 2.39s, peak -4.4 dBFS
  agent  (ch1): 2.71s speech, first at 0.19s, peak -4.4 dBFS
  leading silence: 0.19s
  crosstalk: coherence 0.306 (low) at 0.5s lag
  scorability: separated tracks yes, caller activity yes, agent activity yes
  => eligible for scan
```

**Verdict:** `eligible for scan`, exit 0. Score it. This is the only condition where
a full turn-taking verdict is trustworthy without a caveat.

---

## 2. Silent caller: nothing to measure against

The caller channel carries no speech (a dead leg, a wrong export, a muted line).

```text
$ hotato trust --stereo silent-caller.wav
hotato trust: silent-caller.wav
  recording: 6.0s, 16000 Hz, 2 channels
  caller (ch0): 0.00s speech, -, peak -120.0 dBFS
  agent  (ch1): 5.76s speech, first at 0.19s, peak -9.1 dBFS
  leading silence: 0.19s
  crosstalk: coherence 0.0 (low) at 0.0s lag
  scorability: separated tracks yes, caller activity no, agent activity yes
  => NOT SCORABLE: caller channel has no detected speech
     next step: verify channel mapping or export dual-channel again
```

**Verdict:** `NOT SCORABLE`, exit 2. A yield is a response to a caller; with no
caller there is nothing to be late to, so Hotato names the gap instead of
printing a hollow pass.

---

## 3. Silent agent: no floor to interrupt

The agent channel is empty. There is no floor for the caller to barge into.

```text
$ hotato trust --stereo silent-agent.wav
hotato trust: silent-agent.wav
  recording: 6.0s, 16000 Hz, 2 channels
  caller (ch0): 2.16s speech, first at 0.99s, peak -9.1 dBFS
  agent  (ch1): 0.00s speech, -, peak -120.0 dBFS
  leading silence: 0.99s
  crosstalk: coherence 0.0 (low) at 0.0s lag
  possible channel swap: channel 0 (mapped as caller) holds the floor 2.16s vs 0.0s on channel 1 (mapped as agent); an agent usually holds the floor longer, so the caller/agent channels may be reversed
  scorability: separated tracks yes, caller activity yes, agent activity no
  => NOT SCORABLE: agent channel has no detected speech
     next step: verify channel mapping or export dual-channel again
```

**Verdict:** `NOT SCORABLE`, exit 2. Note that Hotato also flags a possible
channel swap, because a caller-only recording looks like a mis-mapped agent. Two
independent checks point at the same fix: re-check the export.

---

## 4. Swapped channels: candidate-eligible, verdict pending confirmation

Both channels carry speech, but the long floor-holder is on the channel mapped
as the caller. `trust` keeps the input candidate-eligible and holds the verdict
until the mapping is confirmed.

```text
$ hotato trust --stereo swapped-warn.wav
hotato trust: swapped-warn.wav
  recording: 6.0s, 16000 Hz, 2 channels
  caller (ch0): 5.76s speech, first at 0.19s, peak -9.1 dBFS
  agent  (ch1): 0.66s speech, first at 1.99s, peak -9.1 dBFS
  leading silence: 0.19s
  crosstalk: coherence 0.31 (low) at 0.5s lag
  possible channel swap: channel 0 (mapped as caller) holds the floor 5.76s vs 0.66s on channel 1 (mapped as agent); an agent usually holds the floor longer, so the caller/agent channels may be reversed
  scorability: separated tracks yes, caller activity yes, agent activity yes
  => scan with caution: channel 0 (mapped as caller) holds the floor 5.76s vs 0.66s on channel 1 (mapped as agent); an agent usually holds the floor longer, so the caller/agent channels may be reversed
  [!] not verdict-eligible (scan mode): channel mapping unconfirmed: suspected swap/crosstalk; confirm mapping or supply provider metadata
```

**Verdict:** `scan with caution`, exit 0: candidate-eligible, with verdict
eligibility waiting on your confirmation. A swap would invert every yield into
a hold, so Hotato surfaces candidates for review and holds the turn-taking
verdict until the mapping is confirmed. Confirm the mapping (set
`--caller-channel` / `--agent-channel`, or supply provider metadata) to
restore verdict eligibility.

---

## 5. Crosstalk / echo bleed: candidate-eligible, verdict pending confirmation

The caller channel carries a delayed copy of the agent's own audio. Coherence
pegs at 1.0 and the measured leakage is -9.1 dB.

```text
$ hotato trust --stereo 07-echo-bleed.example.wav
hotato trust: 07-echo-bleed.example.wav
  recording: 6.0s, 16000 Hz, 2 channels
  caller (ch0): 5.69s speech, first at 0.31s, peak -13.5 dBFS
  agent  (ch1): 5.81s speech, first at 0.19s, peak -4.4 dBFS
  leading silence: 0.19s
  crosstalk: coherence 1.0 (HIGH) at 0.12s lag
  leakage: -9.1 dB (agent->caller, HIGH)
  scorability: separated tracks yes, caller activity yes, agent activity yes
  => scan with caution: cross-channel leakage: agent audio on the caller channel sits at -9.1 dB, a consistent delayed copy (lag 0.12s); leaked audio at this level can be counted as the other party's activity, so a downstream timing measurement may be wrong even though the tracks are separated
  [!] not verdict-eligible (scan mode): channel mapping unconfirmed: suspected swap/crosstalk; confirm mapping or supply provider metadata
```

**Verdict:** `scan with caution`, exit 0: candidate-eligible, with verdict
eligibility waiting on confirmation. The "caller" activity may be leaked TTS,
so Hotato still surfaces candidates for review (the scan step, example 8,
names the specific moment) and holds the turn-taking verdict until the
mapping is confirmed.

---

## 6. Mono: refused by default, tiered under `--diarize`

A single channel. The caller and the agent are mixed into one track.

```text
$ hotato trust --stereo mono-mixed.wav
hotato trust: mono-mixed.wav
  recording: 6.0s, 16000 Hz, 1 channel
  scorability: separated tracks no, caller activity no, agent activity no
  => NOT SCORABLE: the recording has a single channel, so the caller and the agent cannot be told apart
     next step: export a dual-channel recording with the caller on one channel and the agent on the other
```

**Verdict:** `NOT SCORABLE`, exit 2. This is the gold-standard refusal. There are
two opt-in escapes, both marked indicative only: `--allow-mono` on
`capture` / `pull` / `sweep` accepts a mono-only stack in degraded mode (talk-over
unattributable, no SLA gate), and the `--diarize` separation front-end reports a
`high` / `low` / `refuse` tier, with `hotato run --mono call.wav --diarize`
stamping the verdict `indicative_only` at the `low` tier. Neither is equivalent to
dual-channel. See [TRUST-MATRIX.md](TRUST-MATRIX.md) and [DIARIZE.md](DIARIZE.md).

---

## 7. Backchannel candidate: surfaced, yours to label

A scan of a call where the caller says "mhm" over the agent. `scan` lists the
overlaps as candidates and hands the intent decision to you.

```text
$ hotato scan --stereo 02-backchannel-mhm.example.wav --top 5
hotato scan: 02-backchannel-mhm.example.wav  (6.0s, 3 candidate moments)
Candidates are timing events. You decide the expected behavior; label with: hotato fixture create --onset <t> --expect yield|hold
  [ 1] t=2.09s  overlap_while_agent_talking  overlap=1.58s  agent did not go silent within 3.0s
  [ 2] t=3.19s  overlap_while_agent_talking  overlap=1.07s  agent did not go silent within 3.0s
  [ 3] t=4.29s  overlap_while_agent_talking  overlap=0.56s  agent did not go silent within 3.0s
```

**Verdict:** three candidates, exit 0. Whether "agent did not go silent" is
correct (it held the floor through a backchannel, good) or a bug (it talked over
a real interruption) is **your** call. `scan` reports the timing; you label
`hold` or `yield` -- the intent behind a caller sound is always your call.

---

## 8. Noisy false positive: a candidate that is really the agent

A scan of the echo-bleed call. One candidate is an overlap fact; the
second is Hotato flagging that the "caller" activity is leaked TTS.

```text
$ hotato scan --stereo 07-echo-bleed.example.wav --top 5
hotato scan: 07-echo-bleed.example.wav  (6.0s, 2 candidate moments)
Candidates are timing events. You decide the expected behavior; label with: hotato fixture create --onset <t> --expect yield|hold
  [ 1] t=0.31s  overlap_while_agent_talking  overlap=3.00s  agent did not go silent within 3.0s
  [ 2] t=0.31s  echo_correlated_activity     WARNING likely agent echo: coherence=1.00 at lag 0.12s  (caller channel looks like leaked TTS; a yield here may be the agent hearing itself)
```

**Verdict:** two candidates, exit 0. Candidate 1 looks like a bad talk-over.
Candidate 2 is Hotato telling you the overlap is probably the agent hearing its
own audio, not a real interruption. This is the failure mode of
candidate discovery: the net is wide, and Hotato labels the reason a wide-net
candidate is likely spurious instead of hiding it. You label it `hold` (or fix
the echo capture) and it never becomes a fixture.

---

## How to read the gallery

The eight cases fall into three buckets:

- **Score it** (1): clean stereo, verdict-eligible, full confidence.
- **Candidate-eligible, verdict pending confirmation** (4, 5): swap and
  crosstalk keep `scan` surfacing candidates (exit 0), with the turn-taking
  verdict waiting on your confirmation of the channel mapping. Example 8 is
  that same echo call under `scan`, where the specific leaked moment is
  named.
- **Refuse** (2, 3, 6): silent caller, silent agent, and mono are not scorable;
  Hotato names the reason and the next step and exits 2.

Case 7 is the everyday case: a candidate with no defect, waiting for your
label. The whole design goal is that a Hotato verdict you act on has already
survived this gauntlet.


================================================================================
FILE: docs/TRUST-MATRIX.md
================================================================================

# Trust matrix: what Hotato does for each input condition

Before Hotato scores a recording, `hotato trust` inspects the audio and decides
whether a score would be meaningful, catching a bad export before it can turn
into a confident-looking but hollow verdict. This page is the exact
contract: input condition on the left, Hotato's behavior on the right. This
table covers input health only; a turn-taking verdict is `scan`'s and `run`'s
job.

Worked examples with real CLI output for every row are in
[TRUST-GALLERY.md](TRUST-GALLERY.md).

## The contract

| Input condition | `trust` behavior | Exit | Scoring downstream |
|---|---|---|---|
| **Clean dual-channel (stereo)** | `eligible for scan` | 0 | **Full scoring.** `scan`, `run`, `compare`, `verify` all run normally. |
| **Silent caller channel** | `NOT SCORABLE: caller channel has no detected speech` | 2 | **Refused.** There is no caller to measure a yield against. |
| **Silent agent channel** | `NOT SCORABLE: agent channel has no detected speech` | 2 | **Refused.** There is no agent floor to interrupt. |
| **Channel swap risk** | `eligible for scan` **plus** `possible channel swap` **warning** | 0 | Scores, but confirm the mapping first (`--caller-channel` / `--agent-channel`). A swap silently inverts every yield/hold. |
| **High crosstalk / echo bleed** | `eligible for scan` **plus** high `crosstalk: coherence` **warning** | 0 | Scores at **lower confidence.** `scan` tags the moment `echo_correlated_activity`; a "yield" there may be the agent hearing itself. |
| **Mixed mono (single channel)** | `NOT SCORABLE: single channel, caller and agent cannot be told apart` | 2 | **Refused by default.** Export dual-channel, or take one of the two opt-in escapes (below). |
| **Mono, opt-in `--allow-mono`** | accepted in **degraded mode**, results **indicative only** | 0 | On `capture` / `pull` / `sweep` for a mono-only stack. Talk-over cannot be attributed, so no SLA gate fires. The dual-channel path stays the gold reference. |
| **Mono, opt-in `--diarize`** | separability **tier**: `high` / `low` / `refuse` | 0 (high/low), 2 (refuse) | The separation front-end. `hotato run --mono call.wav --diarize` scores it; at `low` the verdict is stamped `indicative_only`. The dual-channel path stays the gold reference. |
| **Short backchannel ("mhm") overlap** | outside `trust`'s scope; `scan` lists it as a **candidate** | 0 | **Candidate only, human labels.** `trust`/`scan` report the timing; you always label `yield` vs `hold`. |
| **Noisy / false-positive candidate** | `trust` warns (clipping, crosstalk, hot capture); `scan` still lists the candidate | 0 | Surfaced **with** the warning, as a candidate for you to inspect and label `hold`. |

**Reading the two axes.** `trust` answers one question: is this audio good enough
to score? The exit code is the machine contract, `0` = eligible for scan (possibly
with warnings), `2` = not scorable (with the reason and the next step). Warnings
(swap, crosstalk, clipping, leading silence) inform the read; the three hard
refusals (mono, identical channels, a silent required channel) are what
changes scorability.

## Why refuse instead of guessing

Every refusal above is a place where a lesser tool would still print a number.
A mono file "scored" by guessing which speaker is which produces a verdict that
looks identical to a real one and is worthless. A swapped-channel file scored
without the warning inverts caller and agent, so every "the agent yielded"
becomes "the caller yielded" with no visible sign. Hotato treats these as input
defects, reports them, and stops, so a green or red build always means something.

## Two ways past a mono refusal

Mono is refused by default because one channel cannot separate the two parties.
There are two opt-in escapes, both indicative only.

**1. `--allow-mono` (degraded mode).** On `capture`, `pull`, and `sweep`, the
`--allow-mono` flag (or `HOTATO_ALLOW_MONO=1`) accepts a mono-only recording from
a mono stack. Nothing is separated: talk-over cannot be attributed to caller or
agent, so the result is explicitly indicative and no SLA gate fires. Use this when
a stack only exports a mono mix and you want a rough signal anyway.

**2. `--diarize` (separation front-end).** The opt-in `[diarize]` extra
(`pip install 'hotato[diarize]'`) runs a local diarizer and reports a confidence
tier before you score:

- **high**: confidently separable. `hotato run --mono call.wav --diarize` gives a
  diarized-mono verdict. Exit 0.
- **low**: separable but only indicative (voices close, overlap elevated). The
  verdict is stamped `indicative_only`; no SLA gate fires. Exit 0.
- **refuse**: not confidently two clean parties. Not scorable, exit 2; record
  dual-channel.

The dual-channel path stays the gold reference. Neither a degraded-mono nor a
diarized-mono verdict is promoted to equivalence with it. Full front-end:
[DIARIZE.md](DIARIZE.md).

## Composing the gate

Because `trust` exits `2` on any unscorable input, it drops straight into a shell
gate ahead of a scan or a scheduled sweep:

```bash
hotato trust --stereo call.wav && hotato scan --stereo call.wav
```

For agents, `hotato trust --stereo call.wav --format json` emits one
machine-parseable report; branch on `scorable`, and on a defect read
`not_scorable_reason` and `next_step`. Details: [TRUST.md](TRUST.md).


================================================================================
FILE: docs/VALIDATION.md
================================================================================

# What Hotato validates

A turn handoff is not one number, so Hotato is validated on **three separate
jobs**, each measured on its own terms, each with an explicit reported output.
If you are judging whether to trust Hotato, judge these three jobs one at a time.

Everything below runs offline against recordings you control. Every threshold is
exposed and every frame is inspectable (`hotato run --dump-frames`).

---

## Job 1: timing reproducibility

**The question:** given the same recording and the same reference config, does
Hotato produce the same timing measurements every run? (Deterministic for a
fixed hotato version; byte-identical re-runs are verified in CI on Linux
x86_64, Python 3.10, 3.11, and 3.12 -- `.github/workflows/tests.yml`, job
`pytest`. The same double-run check now also runs in CI on macOS and Windows,
with the digest additionally compared ACROSS those OSes -- jobs `portability`
and `determinism` in the same file. Cross-OS agreement is measured and
reported there, not asserted: it has not yet run green, so it is not part of
this claim.)

**What is reported.** Per scored event: `did_yield` (true/false),
`seconds_to_yield`, and `talk_over_sec`, plus the exact thresholds used
(`max_talk_over`, `max_time_to_yield`) and the frame grid behind them. No
learned weights, no sampling, no RNG: the energy VAD and the reference framing
are deterministic, so the numbers are byte-stable.

**How to check it.** Score the same file twice and diff the output.

```text
$ hotato run --stereo 01-hard-interruption.example.wav --expect yield
hotato [single] stack=generic offline=True
  1/1 events pass  (failed=0)
  [PASS] 01-hard-interruption.example.wav: did_yield=True seconds_to_yield=0.51s talk_over=0.51s
  exit_code=0

$ hotato run --stereo 01-hard-interruption.example.wav --expect yield
  [PASS] 01-hard-interruption.example.wav: did_yield=True seconds_to_yield=0.51s talk_over=0.51s
```

`seconds_to_yield=0.51s` and `talk_over=0.51s` are identical across runs. This
is the property a regression test needs: under a fixed hotato version and the
same pinned audio, channel map, onset, label, and scoring config, a changed
result means one of those pinned inputs changed, not that the scorer drifted.

**What this job establishes.** That the measurement is stable and re-derivable
by hand from [`METHODOLOGY.md`](../METHODOLOGY.md) -- not that 0.51s is the
"true" yield latency in some absolute sense, or that the reference thresholds
are right for your product.

---

## Job 2: candidate-discovery usefulness

**The question:** when Hotato scans a whole recording, does it surface the
moments a human reviewer would want to look at, ranked by salience?

**What is reported.** A ranked list of candidate turn-taking moments as **timing
facts only**: overlap onsets (caller became active while the agent was talking,
with the overlap length and whether the agent went silent), agent starts during
caller activity, and long response gaps. Each candidate is a timestamp and a
measurement -- the timing fact, reported without a verdict or an intent label.

```text
$ hotato scan --stereo 02-backchannel-mhm.example.wav --top 5
hotato scan: 02-backchannel-mhm.example.wav  (6.0s, 3 candidate moments)
Candidates are timing events. You decide the expected behavior; label with: hotato fixture create --onset <t> --expect yield|hold
  [ 1] t=2.09s  overlap_while_agent_talking  overlap=1.58s  agent did not go silent within 3.0s
  [ 2] t=3.19s  overlap_while_agent_talking  overlap=1.07s  agent did not go silent within 3.0s
  [ 3] t=4.29s  overlap_while_agent_talking  overlap=0.56s  agent did not go silent within 3.0s
```

The usefulness bar is **recall of human-notable moments at a workable candidate
count**, not precision against a ground-truth intent label (there is no such
label at scan time, by design). A candidate that turns out to be a harmless
backchannel is simply one you label `hold` and move on. The validation
artifact is the [trust gallery](TRUST-GALLERY.md), which
includes a deliberate false positive so you can see what an unhelpful candidate
looks like and why Hotato still surfaces it.

**What this job establishes.** Scan widens the net for you to make the call --
not that every candidate is a bug, or that a quiet region is guaranteed clean.

---

## Job 3: contract verification

**The question:** once you have labelled a moment's expected behavior
(`yield` = stop for the caller, `hold` = keep the floor through a backchannel),
does Hotato's PASS/FAIL verdict agree with that label on the audio, against an
explicit, portable, CI-enforced policy?

Today this job runs on a fixture (`hotato fixture create` / `hotato run`): a
labelled recording plus an explicit threshold policy, scored the same way on
every machine covered by CI (Linux, proven; macOS and Windows now run the same
scoring code path too, pending a first green determinism run -- see Job 1
above). The portable contract bundle (`hotato contract create` /
`hotato contract verify`, audio plus timing evidence plus trace evidence plus
label plus policy plus a CI command in one artifact) carries this exact job
forward into a single self-contained object once it ships; the verdict this
job validates does not change shape, only the artifact it travels in.

**What is reported.** Per fixture: the verdict (`PASS`/`FAIL`), the measured
signals behind it, and the named fix class when the failure maps cleanly to a
config family. Agreement is checked against **your** label, not against an
opaque key.

```text
$ hotato demo --no-open --format text
hotato demo: recorded calls a provider's default agent fails
hotato [suite] stack=generic offline=True
  0/2 events pass  (failed=2)
  [FAIL] fd-01-missed-interruption: did_yield=False seconds_to_yield=- talk_over=0.25s
         fix[config]: Missed interruption: the agent kept talking over the caller
  [FAIL] fd-02-backchannel-yielded: did_yield=True seconds_to_yield=0.34s talk_over=0.32s
         fix[engagement-control]: False barge-in: a backchannel was treated as a bid for the floor
  note: no single sensitivity threshold satisfies this battery
  exit_code=1
```

Both labelled failures are caught, and the battery-level note reports the
disagreement plainly when a missed interruption and a false stop fail in the
same run -- no single threshold satisfies both. Reporting that disagreement
instead of inventing a fix is part of the validated behavior.

**What this job establishes.** That the verdict follows the audio and the
label consistently -- not that the label was correct (you own the label), or
that a passing fixture means the agent is good in general.

---

## What Hotato measures, and where it stops

Read this as the scope of the claim, stated once, plainly.

- **Timing, not semantic intent.** Hotato measures timing; whether a caller
  sound meant "stop" or "mhm, go on" is a label you supply.
- **A likely layer, not certainty.** A slow yield can be TTS buffering,
  transport, or VAD. `diagnose` names a likely layer and stays
  `unknown_root_cause` when one recording cannot separate them. A voice trace
  (once the trace layer ships) narrows the candidates further, short of
  turning one into a proof.
- **Timing, not task success.** Whether the call booked the appointment,
  resolved the ticket, or satisfied the caller is a QA platform's job (see
  [COMPARE.md](COMPARE.md)).
- **A demonstration, not a vendor ranking.** Hotato scores calls, not
  platforms. A provider-default example demonstrates the threshold funnel on
  one assistant, one config, one date, one scripted caller.
- **Timing, not tone.** Sentiment, satisfaction, and CSAT sit outside what
  Hotato measures.
- **Reproducible timing measurements, with the method exposed.** The three
  jobs above are the whole claim -- no headline percentage, by design.

The validation plan for the launch battery (external testers, consented
fixtures, before/after) lives in
[docs/evidence/validation-plan.md](evidence/validation-plan.md).


================================================================================
FILE: src/hotato/schema/envelope.v1.json
================================================================================

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://hotato.dev/schema/envelope.v1.json",
  "title": "hotato result envelope, schema_version 1",
  "description": "The single machine-readable shape emitted by the CLI, the one MCP tool, and the suite runner. Additive-only: new signal dimensions and sub-verdicts may appear as extra keys without a version bump, so consumers must ignore unknown fields. The stable core below is frozen for schema_version 1.",
  "type": "object",
  "required": ["tool", "schema_version", "mode", "stack", "offline", "engine", "limits", "summary", "events", "fix_map", "funnel", "exit_code"],
  "additionalProperties": true,
  "properties": {
    "tool": { "const": "hotato" },
    "schema_version": { "const": "1" },
    "mode": { "enum": ["single", "suite"] },
    "suite": { "type": "string" },
    "stack": { "type": "string" },
    "offline": { "const": true },
    "exit_code": { "enum": [0, 1] },
    "engine": {
      "type": "object",
      "required": ["name", "version", "upstream"],
      "additionalProperties": true,
      "properties": {
        "name": { "type": "string" },
        "version": { "type": "string" },
        "upstream": { "type": "string" }
      }
    },
    "limits": {
      "type": "object",
      "required": ["method", "accuracy_claim", "ceiling", "best_input", "does_not_do", "scope", "offline"],
      "additionalProperties": true,
      "properties": {
        "method": { "type": "string" },
        "accuracy_claim": { "type": "null", "description": "HONESTY INVARIANT: always null. This tool never emits an accuracy percentage." },
        "ceiling": { "type": "string" },
        "best_input": { "type": "string" },
        "does_not_do": { "type": "array", "items": { "type": "string" } },
        "scope": { "type": "string" }
      }
    },
    "summary": {
      "type": "object",
      "required": ["events", "passed", "failed", "regression"],
      "additionalProperties": true,
      "properties": {
        "events": { "type": "integer", "minimum": 0 },
        "passed": { "type": "integer", "minimum": 0 },
        "failed": { "type": "integer", "minimum": 0 },
        "regression": { "type": "boolean" },
        "not_scorable": { "type": "integer", "minimum": 1, "description": "Additive: count of events that could not be judged (malformed input, see each event's not_scorable_reason). Present ONLY when at least one event is not scorable; a run of valid recordings emits the exact pre-existing summary shape. Not-scorable events count in events but in neither passed nor failed." }
      }
    },
    "events": {
      "type": "array",
      "items": { "$ref": "#/definitions/event" }
    },
    "fix_map": {
      "type": "array",
      "items": { "$ref": "#/definitions/fix_map_entry" }
    },
    "funnel": {
      "oneOf": [
        { "type": "null" },
        {
          "type": "object",
          "required": ["reason", "pointer"],
          "additionalProperties": true,
          "properties": {
            "reason": { "type": "string" },
            "pointer": { "$ref": "#/definitions/engagement_pointer" }
          }
        }
      ]
    }
  },
  "definitions": {
    "event": {
      "type": "object",
      "required": ["event_id", "expected_yield", "verdict", "measurements"],
      "additionalProperties": true,
      "dependencies": {
        "scorable": ["not_scorable_reason"],
        "not_scorable_reason": ["scorable"]
      },
      "properties": {
        "event_id": { "type": "string" },
        "scorable": { "const": false, "description": "Additive: present ONLY on events that cannot be judged (and then always false); scorable events omit the key entirely, so every envelope for a valid recording is byte-identical to schema_version 1 before this key existed. A not-scorable event is excluded from passed/failed counting and can never carry a normal pass or fail." },
        "not_scorable_reason": { "type": "string", "description": "Plain-language reason the event cannot be judged (for example no detectable caller speech with no onset provided, or the agent silent at the caller onset under a should-yield expectation). Present exactly when scorable is present." },
        "scenario_id": { "type": ["string", "null"] },
        "title": { "type": ["string", "null"] },
        "category": { "type": ["string", "null"] },
        "expected_yield": { "type": "boolean" },
        "verdict": {
          "type": "object",
          "required": ["passed", "did_yield", "seconds_to_yield", "talk_over_sec", "reasons"],
          "additionalProperties": true,
          "properties": {
            "passed": { "type": "boolean" },
            "did_yield": { "type": "boolean" },
            "seconds_to_yield": { "type": ["number", "null"] },
            "talk_over_sec": { "type": "number" },
            "reasons": { "type": "array", "items": { "type": "string" } }
          }
        },
        "measurements": {
          "type": "object",
          "additionalProperties": true,
          "description": "Timing measurements behind the verdict. The core keys (caller_onset_sec, agent_talking_at_onset, hop_sec, notes) are frozen for schema_version 1. The boundary-sensitivity keys below are ADDITIVE (documented here, never in required): they expose where a near-threshold result sat on the frame grid and how far it was from flipping. A reader built before they existed simply ignores them.",
          "properties": {
            "caller_onset_sec": { "type": ["number", "null"] },
            "agent_talking_at_onset": { "type": "boolean" },
            "hop_sec": { "type": "number" },
            "notes": { "type": "string" },
            "onset_requested_sec": { "type": ["number", "null"], "description": "Additive: the caller_onset_sec the caller requested for this event, or null when the onset was auto-detected rather than supplied." },
            "onset_frame_index": { "type": ["integer", "null"], "description": "Additive: the integer hop index the onset was snapped to (null on a not-scorable placeholder)." },
            "onset_effective_sec": { "type": ["number", "null"], "description": "Additive: the quantized onset time actually used, equal to onset_frame_index * hop_sec (unrounded, so the equality holds exactly). Null on a not-scorable placeholder." },
            "yield_frame_index": { "type": ["integer", "null"], "description": "Additive: the hop index of the counted yield, or null when the agent did not yield (or the event was not scorable)." },
            "decision_margin_sec": { "type": ["number", "null"], "description": "Additive: signed slack in seconds from the tightest binding pass/fail threshold to the measured value (positive = inside the pass boundary, magnitude = how much slack; negative = over the line). Null when no numeric bound applies (pure yield/hold) or the event was not scorable." },
            "decision_margin_hops": { "type": ["integer", "null"], "description": "Additive: decision_margin_sec expressed in whole frame hops, round(decision_margin_sec / hop_sec). Null when decision_margin_sec is null." },
            "boundary_sensitive": { "type": "boolean", "description": "Additive: true when the result sits within one hop of flipping the verdict (abs(decision_margin_hops) <= 1). False when comfortably inside the limit, not derivable, or not scorable." }
          }
        },
        "signals": {
          "type": "object",
          "additionalProperties": true,
          "description": "Namespaced signal bus. barge_in is present whenever scoring ran; other dimensions (latency, and future overlap/resume/backchannel) are additive.",
          "properties": {
            "barge_in": {
              "type": "object",
              "required": ["did_yield", "time_to_yield_sec", "talk_over_sec"],
              "additionalProperties": true
            },
            "latency": {
              "type": "object",
              "additionalProperties": true,
              "properties": {
                "response_gap_sec": { "type": ["number", "null"] },
                "premature_start_sec": { "type": ["number", "null"] }
              }
            },
            "resume": {
              "type": "object",
              "description": "Post-yield behaviour on the agent VAD track; present only when the agent yielded. resumed: did the agent start speaking again within the window after the yield. resume_gap_sec: seconds from the yield to that fresh onset (null when it did not resume). restart_suspected: the longest post-resume run is unusually long, a timing proxy for re-answering from the top (a transcript repeat is out of scope).",
              "additionalProperties": true,
              "properties": {
                "resumed": { "type": "boolean" },
                "resume_gap_sec": { "type": ["number", "null"] },
                "restart_suspected": { "type": "boolean" }
              }
            }
          }
        },
        "fix": {
          "oneOf": [ { "type": "null" }, { "$ref": "#/definitions/fix" } ]
        },
        "audio_provenance": { "$ref": "#/definitions/audio_provenance" }
      }
    },
    "audio_provenance": {
      "type": "object",
      "description": "Additive: identity of the exact audio bytes this event was scored from (a streamed sha256 of the raw file, never a full in-memory decode) plus the sample rate and frame count read from the WAV header. Present on every event scored from a real audio file. Absent (never fabricated, never guessed) on an event with no file to hash -- for example a not-scorable / missing-audio placeholder, or an envelope from before this field existed. A reader (in particular hotato fix trial's before/after recapture guard) MUST treat absence as UNKNOWN provenance, never as proof that a fresh capture happened.",
      "required": ["schema_version", "sha256", "sides"],
      "additionalProperties": true,
      "properties": {
        "schema_version": { "type": "string" },
        "sha256": { "type": "string", "description": "The event's combined audio identity: one file's own sha256 for a single-file input (stereo/mono), or an order-stable combination of both sides' sha256 for a caller+agent dual-mono input." },
        "sides": {
          "type": "array",
          "items": {
            "type": "object",
            "required": ["role", "path", "sha256", "sample_rate", "num_samples"],
            "additionalProperties": true,
            "properties": {
              "role": { "type": "string" },
              "path": { "type": "string" },
              "sha256": { "type": "string", "description": "Streamed sha256 of this side's raw file bytes -- container identity. Changes on any byte anywhere in the file, including header-only metadata edits, a trailing-byte append, or a lossless repack/transcode that never touches a sample value." },
              "pcm_sha256": { "type": "string", "description": "OPTIONAL, additive: streamed sha256 of this side's DECODED PCM sample bytes only (the WAV data chunk, read the same way the scorer decodes it -- fixed width, little-endian, channel-interleaved, bounded by the header's own frame count). Identity of the SAMPLES, not the container: a header-only edit or a trailing-byte append leaves this unchanged even though sha256 above differs; an edit to a sample value changes both. Absent on an envelope from before this field existed -- a reader MUST treat absence as UNKNOWN, never as proof either way." },
              "sample_rate": { "type": "integer" },
              "num_samples": { "type": "integer" },
              "duration_sec": { "type": ["number", "null"] }
            }
          }
        }
      }
    },
    "fix": {
      "type": "object",
      "required": ["fix_class", "title", "detail", "knob", "pointer"],
      "additionalProperties": true,
      "properties": {
        "fix_class": { "enum": ["config", "engagement-control"] },
        "title": { "type": "string" },
        "detail": { "type": "string" },
        "knob": {
          "oneOf": [
            { "type": "null" },
            {
              "type": "object",
              "required": ["stack", "parameter", "direction", "trade_off"],
              "additionalProperties": true
            }
          ]
        },
        "pointer": {
          "oneOf": [ { "type": "null" }, { "$ref": "#/definitions/engagement_pointer" } ]
        }
      }
    },
    "fix_map_entry": {
      "type": "object",
      "required": ["event_id", "fix_class", "title", "detail", "knob", "pointer"],
      "additionalProperties": true,
      "properties": {
        "event_id": { "type": "string" },
        "scenario_id": { "type": ["string", "null"] },
        "fix_class": { "enum": ["config", "engagement-control"] },
        "title": { "type": "string" },
        "detail": { "type": "string" }
      }
    },
    "engagement_pointer": {
      "type": "object",
      "description": "VENDOR-NEUTRAL pointer for the engagement-control fix class. Names the problem class and the KIND of fix (a learned engagement-control / addressee-detection layer); names no vendor, no product, and nothing you can adopt/license/buy, and carries no numbers.",
      "required": ["layer", "what", "honest_scope"],
      "additionalProperties": true,
      "properties": {
        "layer": { "type": "string" },
        "what": { "type": "string" },
        "honest_scope": { "type": "string" }
      }
    }
  }
}
