================================================================================
FILE: README.md
================================================================================

<div align="center">

<img src=".github/assets/hotato-banner.svg" alt="hotato" width="440" style="max-width:100%;height:auto;">

<h1>hotato</h1>

</div>

<p align="center"><b>Open-source, self-hosted conversation QA for voice agents.</b></p>

<p align="center">
  Score every call across five dimensions kept apart (outcome, policy, conversation, speech, reliability), with the evidence behind each result. Deterministic checks and the model-judged rubric stay in separate columns.<br>
  Runs offline &middot; MIT &middot; zero dependencies.
</p>

<p align="center">
  <a href="https://pypi.org/project/hotato/"><img alt="PyPI version" src="https://img.shields.io/pypi/v/hotato.svg"></a>
  <a href="https://pypi.org/project/hotato/"><img alt="PyPI monthly downloads" src="https://img.shields.io/pypi/dm/hotato.svg"></a>
  <a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-blue.svg"></a>
  <img alt="Python 3.9 to 3.13" src="https://img.shields.io/badge/python-3.9%20to%203.13-blue.svg">
  <img alt="offline: yes" src="https://img.shields.io/badge/offline-yes-blue.svg">
  <img alt="runtime deps: zero" src="https://img.shields.io/badge/runtime%20deps-zero-blue.svg">
  <a href="https://github.com/attenlabs/hotato/actions/workflows/tests.yml"><img alt="tests" src="https://github.com/attenlabs/hotato/actions/workflows/tests.yml/badge.svg"></a>
</p>

Your voice agent passes every text assertion and still loses the call: it talks over the caller, skips a required disclosure, confirms a refund that never posted. hotato scores the conversation and the outcome from the two-channel audio, hands you the timing evidence behind every flag, and turns a caught bug into a CI contract that re-checks it on every push.

## Quickstart: one command, on your machine

```bash
uvx hotato start --demo
```

That runs hotato with [uv](https://docs.astral.sh/uv/), no install step, on any machine. To keep it in a project, use pipx or a virtualenv:

```bash
pipx install hotato && hotato start --demo
# or: python -m venv .venv && . .venv/bin/activate && pip install hotato && hotato start --demo
```

Offline, this sweeps two bundled calls a default agent failed, writes the candidate dashboard, and turns one missed-interruption moment into a demo contract it verifies on the spot:

```
[start] demo: swept 2 bundled calls, 5 candidate moments;
  wrote hotato-sweep.json, hotato-sweep.html,
  hotato-no-single-threshold.svg,
  contracts/demo-missed-interruption.hotato/contract.json
hotato start: swept the 2 bundled demo calls offline.
  sweep dashboard: hotato-sweep.html
  demo contract:   contracts/demo-missed-interruption.hotato
  verified contract: FAIL as expected -- the demo call
    missed the interruption
  [ ... then the exact next commands: promote a candidate,
    gate it in CI, re-check it ... ]
```

`hotato-sweep.html` ranks candidate moments by how far the timing missed, each with a hear-the-bug playhead. Screenshot the top one into a PR.

<p align="center">
  <img src="https://raw.githubusercontent.com/attenlabs/hotato/main/docs/assets/sweep-dashboard.png" alt="hotato candidate dashboard: 5 moments ranked by salience, each with a caller/agent timeline, a hear-the-bug playhead, and promote-as-yield / promote-as-hold / ignore buttons." width="760" style="max-width:100%;height:auto;">
</p>
<p align="center"><sub><code>hotato-sweep.html</code> &middot; candidates ranked by salience, each playhead synced to the audio. You label them into your verdict.</sub></p>

Point it at your own recording to walk the same loop:

```bash
# trust -> scan -> review -> label -> contract
hotato start --stereo my-call.wav
```

Every command takes a two-channel recording (caller on one channel, agent on the other). A mono file or a bad export is marked NOT SCORABLE, so every verdict rests on inputs that carry the timing evidence.

## Score a call: five dimensions, kept apart

<p align="center">
  <a href="https://hotato.dev"><img src="https://raw.githubusercontent.com/attenlabs/hotato/main/.github/banner.png" alt="hotato scorecard: one call graded across outcome, policy, conversation, speech, and reliability, deterministic checks kept separate from the model-judged rubric." width="760" style="max-width:100%;height:auto;"></a>
</p>

`hotato test run` grades one call against a conversation-test file, each dimension in its own count:

- **Outcome** &middot; did the job get done, graded on tool-call and state evidence, not the transcript's say-so.
- **Policy** &middot; required disclosures, PII handling, and the compliance phrases your team owns.
- **Conversation** &middot; the deterministic turn-taking core: did the agent yield when the caller took the floor, and how fast.
- **Speech** &middot; response latency and the timing around each turn.
- **Reliability** &middot; pass@1 / pass@k / pass^k across repeated runs with a Wilson interval, so a flaky check reads as flaky.

Deterministic and model-judged results never merge. The deterministic checks (phrase, PII, policy, tool-call, sequence, latency, outcome, all regex, checksum, span-lookup) set the gate; a rubric verdict is `deterministic: false`, advisory, and kept in its own count. Every dimension holds its own line, including under `--format json`, and the scored-output schemas reject an `overall_score` key outright.

```bash
# a starter you edit for your own call
hotato scenario init refund-check --out conversation-test.yaml
hotato test run conversation-test.yaml --agent support-bot
```

```
success: FAIL
  (required: all_deterministic_assertions_pass, no_rubric_failure)
per-dimension (grouped view; never blended):
  outcome       0 pass / 0 fail / 1 inconclusive
  policy        0 pass / 0 fail / 1 inconclusive
  conversation  0 pass / 0 fail / 1 inconclusive
  speech        0 pass / 0 fail / 1 inconclusive
  reliability   0 pass / 0 fail / 0 inconclusive
```

Supply the call as `--transcript`, `--trace`, `--state`, and/or `--audio`; each check turns pass or fail, and a check with no evidence stays INCONCLUSIVE. Walkthrough: [`docs/CONVERSATION-TEST.md`](docs/CONVERSATION-TEST.md).

The scored HTML report is the receipt: a per-event verdict, time-to-yield and talk-over histograms, failure clusters by fix class, and a two-channel timeline with the measured numbers, reproducible from the same audio and config.

<p align="center">
  <img src="https://raw.githubusercontent.com/attenlabs/hotato/main/docs/assets/report-scored.png" alt="Scored hotato report: 0 of 1 events pass with a REGRESSION verdict, an analytics panel (time to yield, talk-over histogram, failure clusters by fix class), and a per-event caller/agent timeline with the measured metrics." width="760" style="max-width:100%;height:auto;">
</p>
<p align="center"><sub>One recording, the pinned scorer, a FAIL against the labeled yield expectation. Share it in a PR with <code>hotato card hotato-sweep.json#1 --out finding.svg</code>.</sub></p>

## The loop: catch, confirm, gate, prove

Surface a moment, confirm what it should have done, pin it to CI, then re-check today's agent.

**1. Catch.** `sweep` ranks the talk-over and false-stop moments across your recent calls by how far the timing missed, each a reproducible measurement from the open scorer:

<p align="center">
  <img src="https://raw.githubusercontent.com/attenlabs/hotato/main/docs/assets/cards/talk-over-card.svg" alt="Level 1 candidate card: 0.32s of overlap while the agent was talking, at t=2s." width="480" style="max-width:100%;height:auto;">
</p>
<p align="center">
  <img src="https://raw.githubusercontent.com/attenlabs/hotato/main/docs/assets/cards/false-stop-card.svg" alt="Level 1 candidate card: 0.46s of silence after the agent stopped with no caller nearby, at t=1.28s." width="480" style="max-width:100%;height:auto;">
</p>
<p align="center"><sub><b>Level 1 -- candidate.</b> Measured, not judged (0.32s of overlap; 0.46s of trailing silence): a timing moment worth review.</sub></p>

**2. Confirm.** You label the expected behavior: `yield` (stop for the caller) or `hold` (keep talking through a backchannel). Intent stays yours. Because a threshold that stops missing interruptions starts false-stopping on backchannels, when both fail in one run `diagnose` surfaces the tradeoff rather than naming one threshold:

<p align="center">
  <img src="https://raw.githubusercontent.com/attenlabs/hotato/main/docs/assets/cards/no-single-threshold-card.svg" alt="One sensitivity dial trades a missed interruption against a false stop on a backchannel; hotato surfaces the tradeoff, fix class engagement-control." width="760" style="max-width:100%;height:auto;">
</p>
<p align="center"><sub><b>Level 2 -- human-labeled failure.</b> A reviewer confirms a broken yield-or-hold expectation; the fix lives in the engagement-control class.</sub></p>

**3. Gate.** `fixture promote` saves the labeled call as a permanent regression test, and hotato speaks CI natively:

- A deterministic fail exits non-zero, a pass exits zero, so a red build is a caught regression.
- `hotato contract verify contracts/ --junit contracts-junit.xml` writes JUnit XML your runner already renders.
- `--format json` carries an `exit_code` field, so an agent reads the verdict without parsing prose.
- The model-judged rubric is advisory by default and blocks a build only when you pass `--gate`.

Drop-in GitHub Action and pytest plugin: [`docs/CI.md`](docs/CI.md) &middot; [`docs/PYTEST.md`](docs/PYTEST.md). One bad call to a CI gate, step by step: [`docs/BAD-CALL-TO-CI.md`](docs/BAD-CALL-TO-CI.md) &middot; [`examples/bad-call-to-ci/`](examples/bad-call-to-ci/README.md).

**4. Prove.** The frozen recording catches evidence, threshold, or scorer drift. To check that the CURRENT agent still behaves, recapture the same scenario and score a NEW contract under the same policy:

```bash
# place the same call against today's agent, capture
# dual-channel, then:
hotato contract create --stereo fresh-call.wav --onset 41.90 --expect yield \
    --id refund-cutoff-001-recapture --out contracts
hotato contract verify contracts/refund-cutoff-001-recapture.hotato
```

<p align="center"><sub><b>Level 4 -- fresh-recapture comparison.</b> A newly captured call meets the same labeled policy and every submitted paired guard held. Walkthrough: <a href="docs/RECAPTURE.md"><code>docs/RECAPTURE.md</code></a>.</sub></p>

### Five levels of evidence, each on its own lane

Every card, report, and CLI result names its evidence level; the public tier is the weakest one its inputs support.

| Level | Name | What it means |
| --- | --- | --- |
| 1 | Candidate | A candidate timing moment worth human review. |
| 2 | Human-labeled failure | A reviewer confirmed this recording broke an explicit yield-or-hold expectation. |
| 3 | Stored-evidence check | The historical audio still produces the expected result under the pinned policy and scorer. |
| 4 | Fresh-recapture comparison | A newly captured call passed the same contract, and no submitted paired guard regressed. |
| 5 | External proof | An independent team confirms a caught regression or a fresh recapture. |

A before/after experiment (`hotato fix trial`, and the fleet loop) re-derives every verdict from the on-disk audio under one pinned trial manifest.

## Scale one call into a release gate

- `hotato suite run suite.yaml --agent support-bot` -- a whole `suite.v1` offline through the deterministic scripted-caller simulator, into the local registry.
- `hotato simulate --matrix scenario.yaml --out ./conv` -- expand a `scenario.v1` matrix into hundreds of seeded, byte-identical `origin=simulated` runs.
- `hotato rubric run --rubrics rubrics.yaml --transcript call.json` -- the model-judged lane on a pinned local model; zero egress, advisory unless `--gate`. [`docs/RUBRIC.md`](docs/RUBRIC.md).
- `hotato release compare BASELINE CANDIDATE` -- diff two recorded releases per dimension and scenario, digest-exact, surfacing new and fixed failures.
- `hotato serve` -- a read-only, token-authenticated web app over the local registry (release readiness, scenario matrix, conversation inspector, failure clusters, production health) on `127.0.0.1`. [`docs/WORKSPACE.md`](docs/WORKSPACE.md).

The bundled reference agent shows the shape at scale: 25 jobs x 5 caller behaviours x 3 audio environments = 375 offline simulated runs, scored and byte-reproducible on replay. Run it with `make reference` ([`examples/reference-agent/`](examples/reference-agent/README.md), [`docs/SUITE-RUN.md`](docs/SUITE-RUN.md)); the measurement-error harness is [`docs/BENCHMARK.md`](docs/BENCHMARK.md).

## Connect a production stack

Point hotato at your live calls: connect once, then sweep on a schedule.

```bash
# credentials stored 0600, local only
hotato connect vapi
# cron, CI, wherever
hotato sweep --stack vapi --since 7d --out hotato-sweep.html
```

Your audio stays on your machine unless you pull it from your stack. Full guide: [`docs/SET-AND-FORGET.md`](docs/SET-AND-FORGET.md) &middot; [`examples/set-and-forget/`](examples/set-and-forget/README.md).

`sweep` and `hotato fleet run` post a one-line JSON summary to a webhook when they finish, off by default, opt in with `--notify` (repeatable):

```bash
hotato sweep --stack vapi --since 7d \
    --notify https://hooks.slack.com/services/...
```

The payload is metadata-only: counts, top candidate moments (id, kind, timing numbers), artifact paths, and a Slack-ready `text` field; a down webhook leaves the run intact, a delivery failure is one stderr warning. Egress details: [`docs/EGRESS.md`](docs/EGRESS.md).

## Fleet: private, self-hosted

`hotato fleet` runs the loop across every agent from one local workspace: ingest calls, surface candidates, label them, and run a before/after experiment that recomputes both sides from audio under a pinned manifest. It recommends a change and leaves the deploy to you. Local mode is stdlib-only (SQLite plus a content-addressed store).

```bash
hotato fleet init -w acme
hotato fleet agent add -w acme --name support-bot \
    --stack vapi --assistant-id asst_123
hotato fleet ingest -w acme --agent support-bot call.wav
hotato fleet discover -w acme --agent support-bot call.wav
hotato fleet review -w acme
```

With history, `hotato fleet trend -w acme` writes one self-contained HTML page: per-agent talk-over and time-to-yield trends (p50/p95 per day), candidate moments over time, and experiment outcomes (improved/inconclusive/refused). A short series reports "not enough history to trend" over an interpolated line. Full guide: [`docs/GUARDIAN-FLEET.md`](docs/GUARDIAN-FLEET.md).

## Self-host in your own cloud or VPC

The whole team workspace ships as a container. One command stands up the read-only, token-authenticated `hotato serve` on host loopback over a private volume, keeping all traffic on your own infrastructure:

```bash
# workspace on 127.0.0.1:8321
docker compose up -d
# optional: seed example data
docker compose run --rm hotato-init
# optional: a local Ollama model judge
docker compose --profile judge up -d
```

Your own calls, the local judge, air-gap, backup, and the zero-migration promise (self-host and cloud share one set of schemas): [`docs/SELF-HOST.md`](docs/SELF-HOST.md). Confirm the offline posture on your own machine with [`deploy/verify-zero-egress.sh`](deploy/verify-zero-egress.sh).

## Built for coding agents

hotato is a first-class tool for an LLM or an agent to drive: machine JSON on every command, meaningful exit codes, a described capability manifest, [`llms.txt`](llms.txt), JSON-LD, and an MCP server.

```bash
# the voice_eval_run scorer + eleven fleet tools
uvx --from "hotato[mcp]" hotato-mcp
```

Configs and the tool contract: [`docs/MCP.md`](docs/MCP.md) &middot; [`AGENTS.md`](AGENTS.md) &middot; [`llms-full.txt`](llms-full.txt).

## Choose your path

| You want to | Run this |
| --- | --- |
| Try the full loop, no credentials | `hotato start --demo` |
| Sweep the bundled demo calls | `hotato sweep --demo` |
| Sweep recent calls from your stack | `hotato connect vapi` then `hotato sweep --stack vapi --since 7d` |
| Add hotato to an existing repo, CI gate included | `hotato init starter --stack vapi --out .` ([`docs/STARTER.md`](docs/STARTER.md)) |
| Turn a confirmed failure into a portable contract | `hotato contract create --from-candidate hotato-sweep.json#1 --expect yield --id refund-cutoff-001 --out contracts` ([`docs/CONTRACTS.md`](docs/CONTRACTS.md)) |
| Verify contracts in CI | `hotato contract verify contracts/ --junit contracts-junit.xml` |
| Attach observability traces to a contract | `hotato trace attach contracts/refund-cutoff-001.hotato --trace voice_trace.jsonl` ([`docs/TRACE.md`](docs/TRACE.md)) |
| Test a candidate fix, before/after, fail-closed | `hotato fix trial patch.json --name staging-x --before before/ --after after/` ([`docs/FIX-TRIAL.md`](docs/FIX-TRIAL.md)) |
| Reduce a scripted deterministic failure to a verified repro | `hotato counterexample compile --scenario case.json --test test.json --target assertion-id --out case.hotato-repro` ([`docs/COUNTEREXAMPLES.md`](docs/COUNTEREXAMPLES.md)) |
| Share a finding in a PR or slide | `hotato card hotato-sweep.json#1 --out finding.svg` |
| Drive it from a coding agent | `uvx --from "hotato[mcp]" hotato-mcp` ([`docs/MCP.md`](docs/MCP.md)) |

`contract verify` and a promoted fixture in CI are two different guarantees, depending on which recording goes in:

**On the frozen recording (every push)**
- Proves: The evidence, policy, and scorer are intact
- Does not prove: That the deployed agent has not changed

**On a fresh recapture** (by hand, see [`docs/RECAPTURE.md`](docs/RECAPTURE.md))
- Proves: The CURRENT agent's behavior still matches the label
- Does not prove: --

A contract bundle contains call audio. Do not commit a raw customer contract to a public repository; use sanitized fixtures for anything public. See [`docs/CONTRACTS.md`](docs/CONTRACTS.md).

## Install

Run any command zero-install with [`uvx`](https://docs.astral.sh/uv/), or add hotato to a project with pipx or pip in a virtualenv:

```bash
# zero-install, any command:
uvx hotato start --demo
# keep it in a project:
pipx install hotato
# extras (Silero VAD cross-check / ASR transcript / LiveKit / Pipecat capture):
pipx install 'hotato[neural]'
pipx install 'hotato[transcribe]'
pipx install 'hotato[livekit]'
pipx install 'hotato[pipecat]'
# run an extra zero-install:
uvx --from 'hotato[neural]' hotato start --demo
```

## Contribute a labeled call

The highest-value PR is one labeled dual-channel call: the corpus is the part of hotato that compounds, and every labeled moment sharpens every scorer. Add a clip: [`docs/SUBMITTING.md`](docs/SUBMITTING.md) &middot; corpus and schema in [`corpus/`](corpus/README.md), recorded battery in [`corpus/vapi-defaults/README.md`](corpus/vapi-defaults/README.md). Contributor guide: [`CONTRIBUTING.md`](CONTRIBUTING.md).

## Where hotato fits

- **Conversation QA that shows its work**, sitting next to your runtime voice layers: [`docs/COMPARE.md`](docs/COMPARE.md).
- **Audio-timing scoring.** The opt-in `--transcribe` flag attaches an ASR transcript as context beside the verdict; the timing score stays grounded in the audio: [`docs/TRANSCRIBE.md`](docs/TRANSCRIBE.md).
- **Offline, out-of-band, anonymous.** It reads recordings after the call over two channels, so your live audio path and running agent stay in your hands and the channels stay anonymous.

## Docs

- **Set-and-forget monitoring**: [`docs/SET-AND-FORGET.md`](docs/SET-AND-FORGET.md) &middot; [`examples/set-and-forget/`](examples/set-and-forget/README.md)
- **Bad call to CI, step by step**: [`docs/BAD-CALL-TO-CI.md`](docs/BAD-CALL-TO-CI.md) &middot; [`examples/bad-call-to-ci/`](examples/bad-call-to-ci/README.md)
- **What it measures** (three timing signals): [`METHODOLOGY.md`](METHODOLOGY.md) &middot; [`docs/API.md`](docs/API.md)
- **The fix ladder** (failure &rarr; fix class + setting direction): [`docs/FIX-PLANS.md`](docs/FIX-PLANS.md)
- **Rule out non-turn-taking bugs first** (STT, buffering, verbosity, refusals, language): [`docs/WHY.md`](docs/WHY.md)
- **Pull a call from your stack** (Vapi, Twilio, Retell, LiveKit, Pipecat): [`adapters/README.md`](adapters/README.md) &middot; [`docs/ADAPTER-STATUS.md`](docs/ADAPTER-STATUS.md)
- **CI gates**: [`docs/CI.md`](docs/CI.md) &middot; [`docs/PYTEST.md`](docs/PYTEST.md)
- **Recorded-call battery** (12 scripted calls on provider defaults): [`corpus/vapi-defaults/README.md`](corpus/vapi-defaults/README.md)
- **Failure contracts and traces**: [`docs/CONTRACTS.md`](docs/CONTRACTS.md) &middot; [`docs/TRACE.md`](docs/TRACE.md) &middot; [`docs/OTEL.md`](docs/OTEL.md)
- **Deterministic assertions** (phrase, PII, policy, tool-call, outcome): [`docs/ASSERTIONS.md`](docs/ASSERTIONS.md)
- **Checking the CURRENT agent, not just the frozen recording**: [`docs/RECAPTURE.md`](docs/RECAPTURE.md)
- **Egress** (per-command network table): [`docs/EGRESS.md`](docs/EGRESS.md)
- **Explain a failure, trial a fix**: [`docs/EXPLAIN.md`](docs/EXPLAIN.md) &middot; [`docs/FIX-TRIAL.md`](docs/FIX-TRIAL.md) &middot; [`docs/APPLY.md`](docs/APPLY.md) &middot; [`docs/FIX-LOOP.md`](docs/FIX-LOOP.md)
- **Evidence**: [`docs/VALIDATION.md`](docs/VALIDATION.md) &middot; [`docs/TRUST-MATRIX.md`](docs/TRUST-MATRIX.md) &middot; [`docs/GALLERY.md`](docs/GALLERY.md) &middot; [`docs/EVIDENCE-PACK.md`](docs/EVIDENCE-PACK.md) &middot; [`docs/COMPARE.md`](docs/COMPARE.md)
- **For coding agents**: [`AGENTS.md`](AGENTS.md) &middot; [`llms.txt`](llms.txt) &middot; [`llms-full.txt`](llms-full.txt) &middot; [`docs/MCP.md`](docs/MCP.md) &middot; [`SECURITY.md`](SECURITY.md)
- **Contributing** (a labeled call fixture): [`docs/SUBMITTING.md`](docs/SUBMITTING.md)

Why "hotato": good turn-taking is a game of hot potato. Speak, then pass the turn the moment the caller wants it. MIT licensed ([`LICENSE`](LICENSE)); the open core stays open.

mcp-name: io.github.attenlabs/hotato


================================================================================
FILE: docs/WHY.md
================================================================================

# Why Hotato

Four timing failures break a voice-agent call while every text-level test
still passes.

## Voice agents fail in ways your tests do not catch

Your test suite checks what the agent says; production calls fail on *when*
it speaks. Three failures are interruption patterns (missed interruption,
false stop, slow yield); the fourth is an endpointing gap:

1. **Missed interruption.** The caller says "stop, take that off" and the agent
   keeps talking. `did_yield` is false where it must be true. The caller
   repeats themselves, louder, then hangs up.
2. **False stop.** The caller says "mhm" -- a backchannel, an acknowledgement,
   not a request to take over. The agent stops mid-sentence anyway. `did_yield`
   is true where it must be false. The call stalls, restarts, stalls again.
3. **Slow yield.** The caller interrupts and the agent does stop, eventually.
   Every second of `talk_over_sec` before it stops is both speaking at once,
   and the caller hears all of it.
4. **Endpointing misses.** Endpointing is detecting that the caller finished
   speaking. When it misses, the caller gets dead air (`response_gap_sec`), or
   the agent starts before they are done (`premature_start_sec`). Both read as
   a broken conversation partner. Hotato measures how long the silence lasted,
   never what it meant (thinking, distracted, or gone quiet).

Each lives entirely in the audio timing, invisible to a transcript diff, an
LLM judge on text, or a unit test on the agent's reply -- all three score the
call as perfect: the transcript reads clean, and the caller calls back
anyway. That gap is what these four patterns close.

You label the expected behavior: `yield` (the agent should stop for the caller)
or `hold` (it should keep speaking through a backchannel/noise/acknowledgement).
Hotato measures whether the timing matched that label -- intent is always yours
to call.

## Is this even a turn-taking bug?

A lot of alleged barge-in bugs are not turn-taking bugs. Each pattern below
produces a symptom that reads exactly like a missed interruption or false
stop, but the fix lives in a different layer, and no VAD threshold touches it:

- **STT hallucination.** The transcript has words the caller never said, or
  drops words that mattered, so the agent responds to something unsaid. Reach
  for: an STT/ASR word-error-rate check against the raw audio.
- **Client-side audio buffering.** The caller's device or browser queues
  outgoing audio before it reaches the agent, so "the agent talked over me" is
  audio arriving late -- a transport ordering artifact. Reach for: client/WebRTC
  jitter-buffer and network-latency instrumentation.
- **LLM verbosity or tool-selection.** The agent keeps talking through the
  interruption because it is mid-tool-call or committed to finishing a long
  generation before re-checking for a stop signal. Reach for: response-length
  and tool-call latency tracing in your agent framework.
- **Safety false-refusal.** The agent stops abruptly because a moderation layer
  cut it off, upstream of any barge-in detection. Timing-wise this is
  indistinguishable from a false stop on "mhm". Reach for: your
  safety/moderation logs.
- **Wrong-language STT.** The caller speaks a language or accent the STT covers
  poorly, recognition comes back empty or garbled, and the response looks
  unrelated. That reads as a missed interruption; it is a language-coverage
  gap. Reach for: per-locale STT accuracy tooling. Hotato's own detector works
  from energy alone, so it measures the same regardless of language: see
  `corpus/classes/README.md`.

This is both a scope guard and a shortcut. Match one of these five and tuning
Hotato will not surface it -- you save the day you would have spent staring at
`turn_end_silence_sec`. Match none of them and agent-talks-over-caller /
false-stop-on-backchannel are exactly what Hotato measures, with the funnel --
no single config value fixes both directions at once -- demonstrated on
recorded calls, not synthetic fixtures (`corpus/vapi-defaults/README.md`).

## What makes Hotato different

- **It scores recordings you already have.** A dual-channel WAV from your stack
  is the whole input -- no test harness, synthetic caller, or re-architecture.
- **It runs offline.** Scoring is local and deterministic; audio, transcript,
  and result all stay on your machine.
- **It emits machine-readable timing.** One JSON envelope per run, the same
  shape from the CLI, the MCP tool, and the pytest fixture.
- **It fails CI.** Exit code 1 on a regression, 0 on pass, 2 when a recording
  isn't scorable yet (the caller channel is silent, or the agent wasn't talking
  when the caller started) -- so the verdict waits for a recording that can
  answer the question. A turn-taking regression blocks a merge like a failing
  unit test.
- **It routes every failure to a fix class.** When the failure maps cleanly to
  stack config, `config` names the setting family and the direction to
  investigate. `engagement-control` names the failure as a classification
  problem -- telling "mhm" apart from "stop" -- so you spend time on the fix
  that works instead of retuning a setting that cannot win.

## What Hotato measures, and where it stops

Hotato measures energy over time on two channels -- that is the whole method.
Transcription, emotion, and intent detection sit outside it; a diarizer, where
used, assigns anonymous SPEAKER_00/01 labels, never who a person is. A
single-channel (mono) recording scores via the opt-in, quality-gated
`[diarize]` front-end (`hotato run --mono call.wav --diarize`), labeled
indicative below the confidence bar, with dual-channel as the reference
standard. The method and its ceiling are stated in
[METHODOLOGY.md](../METHODOLOGY.md) and in the `limits` block of every result.

## Reproducible timing measurements, with the method exposed

A single blended percentage tells you nothing about which call failed, on
which axis, or what to change. Hotato reports three direct measurements per
event:

- `did_yield`: did the agent stop talking after the caller started
- `seconds_to_yield`: seconds between the caller starting and the agent stopping
- `talk_over_sec`: seconds the caller and the agent spoke at the same time

Every number is reproducible from the frame dump by hand, every threshold is
exposed, and a failure points at its fix. That is what you debug with.

Ready to pin a failure? The loop from one bad call to a CI gate, with `hotato
fixture create`, `compare`, and `plan`: [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md).


================================================================================
FILE: METHODOLOGY.md
================================================================================

# Methodology

How every number this tool reports is computed, why it is reproducible, and
where the method stops being trustworthy. There is no accuracy percentage in
this document and none is implied anywhere in the tool. What follows describes a
measurement, not a claim about any detector's internal quality.

Hotato scores the audio *timing* of turn-taking from a call recording: whether
the agent stopped talking once the caller started (a yield), how many seconds
that took, how many seconds it kept talking while the caller was talking
(talk-over), and, on the same two tracks, the endpointing timing: how the
agent's response sits around the moment the caller finished speaking. It
measures energy over time. It does not understand speech.

## Read this first: the honest ceiling

The limits come first because they bound everything below.

- **Energy is not intent.** The detector marks a frame "active" when its
 short-time energy crosses a threshold. A cough, a slammed door, or a burst of
 line noise reads as active exactly like a word does. Nothing here recovers
 *meaning*, it recovers *when there was speech-level energy*.
- **You supply the label.** Hotato does not infer intent. You label the
 expected behavior for the event: yield means the agent should stop for the
 caller. hold means the agent should keep speaking through a
 backchannel/noise/acknowledgement. Hotato then measures whether the timing
 matched that label. "mhm" and "stop" can carry identical speech energy; the
 verdict is only ever your label checked against measured timing.
- **Sub-second single-channel boundaries sit near the resolution limit.** At the
 default 10 ms hop a boundary is located to within a frame or two, but the
 hangover that keeps an utterance whole (default 150 ms) smears every trailing
 edge by up to that much, and on one mixed channel deciding *whose* energy
 crossed the threshold is not always possible.
- **Two channels is the gold reference; mono is scorable, quality-gated.** The
 reference input is separated caller and agent tracks: one two-channel WAV or
 two aligned mono WAVs, where each channel is one party and overlap is a fact of
 the recording (both tracks active at once, by construction). A single mixed
 mono call is scorable via the opt-in `[diarize]` front-end (`hotato run --mono
 call.wav --diarize`): a diarizer separates the mix into caller/agent activity,
 which is reconstructed and scored through the same path. It is quality-gated --
 above the confidence bar the verdict is labeled `diarized-mono`; below it, the
 verdict is labeled indicative only and no SLA gate fires; a non-separable file
 is refused. Diarized mono never equals a true dual-channel recording for
 sub-second talk-over attribution, and the gate is what keeps that honest per
 file.
- **Out of scope, permanently:** no speaker identification (a diarizer assigns
 anonymous SPEAKER_00/01; it never says who a person is), no speech-to-text, no
 emotion or intent detection, and no claim about any vendor's internal accuracy.
 Word-level semantics are not measured; only the timing of the yield is.

These same limits are emitted inline in the `limits` block of every JSON result
(`core.py`) and in the MCP tool description, so a consuming agent sees them.

## The pipeline, step by step

Everything runs offline, standard-library only (NumPy only accelerates the
per-frame RMS when importable; results are identical without it). Scoring is
deterministic: the same audio and `ScoreConfig` give the same output.

### Default parameters

Every value below is read directly from `ScoreConfig` (`score.py`) and
`VADParams` (`vad.py`); the same defaults apply to both channels unless overridden.

Framing and VAD (`VADParams`, applied per channel):

| Parameter | Default | Meaning |
|--------------------|-----------|---------|
| `frame_ms` | 20.0 ms | analysis window length for one RMS frame |
| `hop_ms` | 10.0 ms | step between frames (the time resolution) |
| `noise_percentile` | 0.10 | quietest 10% of frames estimates the noise floor |
| `rel_db` | 15.0 dB | a frame is active this many dB above the floor |
| `abs_gate_db` | -60.0 dBFS| nothing below this absolute level is ever active |
| `dyn_margin_db` | 22.0 dB | on a rarely-silent channel, hold the threshold at least this far below the loudest frames |
| `hangover_sec` | 0.15 s | stay "active" this long after energy drops |

Signal windows (`ScoreConfig`):

| Parameter | Default | Meaning |
|---------------------------|---------|---------|
| `yield_hangover_sec` | 0.20 s | agent must stay quiet this long to count as yielded |
| `max_search_sec` | 3.0 s | how long after onset a yield is searched for |
| `caller_proximity_sec` | 0.5 s | a yield only counts if the caller held the floor within this window of it |
| `turn_end_silence_sec` | 0.20 s | caller must stay quiet this long for the turn to count as ended |
| `premature_tolerance_sec` | 0.05 s | agent may lead the caller's turn end by up to this before it counts as premature |
| `onset_min_run_sec` | 0.05 s | minimum sustained active run for the caller onset to count |
| `agent_onset_lookback_sec`| 0.10 s | window before onset checked for whether the agent was already talking |

Both onset-detection windows are exposed on `ScoreConfig` as well:
`onset_min_run_sec` (the `0.05 s` sustained active run a caller onset must
clear, applied by `first_active_sec`) and `agent_onset_lookback_sec` (the
`0.10 s` window before onset checked for "was the agent talking at onset?").
Neither affects the per-frame active/threshold decision the frame dump records.
Every threshold the scorer uses is a named, overridable parameter -- there are
no hidden constants in the scoring path.

### Step 1, per-frame RMS

Each channel is cut into 20 ms windows stepped by 10 ms. For each window the
linear root-mean-square amplitude is computed (`frame_rms`, `audio.py`). At
16 kHz this is a 320-sample window with a 160-sample hop, so `hop_sec` is
exactly `160 / 16000 = 0.01 s`; it is derived from the integer sample hop over
the sample rate, so it matches `hop_ms` exactly at common rates.

### Step 2, RMS to dBFS

Each linear RMS value is converted to decibels relative to full scale:
`dBFS = 20 * log10(rms)`, with a floor of `-120.0 dBFS` substituted before the
log to avoid `log(0)` (`to_dbfs`, `audio.py`). This is the per-frame `*_dbfs`
column in the frame dump.

### Step 3, per-channel energy VAD

For each channel independently (`energy_vad`, `vad.py`):

1. Sort the frame dBFS values and take the `noise_percentile` (10th percentile)
 as the **noise floor**.
2. The base **threshold** is `max(noise_floor + rel_db, abs_gate_db)`, i.e.
 15 dB above the floor, but never below the -60 dBFS absolute gate.
3. **Dynamic-margin guard:** if a channel is almost never silent (e.g. an agent
 talking the whole clip), the 10th-percentile floor lands *inside* speech and
 would push the threshold above the speech itself. So compute
 `cap = max(dBFS) - dyn_margin_db`; if `cap` is above the absolute gate, lower
 the threshold to `min(threshold, cap)`. This cannot rescue a genuinely silent
 channel, because the guard only fires when loud content exists above the gate.
4. A frame is **raw-active** when its dBFS ≥ the threshold.
5. **Hangover:** after any raw-active frame, keep the channel active for
 `round(hangover_sec / hop_sec)` more frames, 15 frames (150 ms) at the
 defaults, so brief inter-word gaps do not fragment one utterance.

The resulting threshold and noise floor are constant across frames for a given
channel and are recorded verbatim in the frame dump.

### Step 4, caller onset

If a caller onset label is supplied (scenario `caller_onset_sec`, or `--onset`),
it is used directly. Otherwise the onset is the start of the caller's first
sustained active run, the first run of at least 0.05 s of active frames
(`first_active_sec`). The label is preferred because on mixed or noisy audio the
first energy is not always the caller. `onset_idx` is `round(onset / hop_sec)`,
clamped into range. No label and no detectable caller speech means there is no
caller event to score: the recording is reported not scorable (`scorable: false`
with a plain reason), never scored against frame 0, and a single-recording run
exits 2.

### Step 5, the three barge-in signals

Computed in `score_channels` (`score.py`):

- **`agent_talking_at_onset`**, was any agent frame active in the 0.10 s up to
 and including onset? If not, on a should-yield expectation there is nothing to
 yield: the event is reported not scorable (`scorable: false`, with the reason
 in `not_scorable_reason`) rather than passed or failed, and a single-recording
 run exits 2. The input is what is wrong (onset time, channel mapping, or
 expectation), and it is reported as such.
- **`did_yield`**, scanning from onset up to `onset + max_search_sec` (300
 frames), the first frame where the agent goes quiet and *stays* quiet for
 `yield_hangover_sec` (20 frames) **and** the caller held the floor within
 `caller_proximity_sec` (±50 frames) of that quiet point. The proximity
 condition is what stops an agent that merely finishes its own sentence seconds
 after an isolated backchannel from being scored as a barge-in response.
- **`time_to_yield_sec`**, `(yield_frame - onset_frame) * hop_sec`, floored at
 0; `None` if the agent never yielded within the search window.
- **`talk_over_sec`**, count of frames from onset to the yield point (or to the
 end of the search window if it never yielded) where the caller and the agent
 were *both* active, times `hop_sec`.

### Step 6, the two latency (endpointing) signals

Pure timing on the same two VAD tracks, no second model (`score.py`):

- **`caller turn end`**, the first frame at/after onset where the caller goes
 quiet and stays quiet for `turn_end_silence_sec` (20 frames). `None` if the
 caller never activates after onset or is still talking when the clip ends.
- **`agent response onset`**, skip any agent speech already in progress at
 onset (the pre-caller turn or the yield tail), then the first frame the agent
 becomes active again. `None` if the agent never starts a fresh run.
- With both defined, `lead = turn_end_frame - response_onset_frame` (positive
 when the agent starts *before* the caller finishes). If `lead` exceeds
 `premature_tolerance_sec` (5 frames), `premature_start_sec = lead * hop_sec`
 and `response_gap_sec` is null. Otherwise `premature_start_sec = 0.0` and
 `response_gap_sec = max(0, response_onset - turn_end) * hop_sec`. Both are
 null when not derivable from the tracks, never fabricated. Reported values
 are rounded to the millisecond (3 decimal places of seconds).

### Step 7, two additive, opt-in cross-checks: echo and resume

Both live entirely in hotato's own layer (`echo.py`, `resume.py`), never in
the vendored `_engine`, and both add an optional `signals` block without
changing `did_yield`, `seconds_to_yield`, or `talk_over_sec` for any existing
recording.

- **`signals.echo`** (`echo.py`), computed on every scored event. Leaked TTS
 (an agent that hears its own audio back on the caller channel) is, by
 construction, a delayed, scaled copy of the agent's own envelope. The check
 is a deterministic cross-channel coherence: at each lag from 0 up to
 `DEFAULT_MAX_LAG_SEC` (0.5 s), take the cosine similarity between the caller
 frame-RMS envelope and the agent envelope shifted by that lag; `coherence`
 is the best (highest) cosine found, `lag_sec` is the lag it occurred at, and
 `echo_suspected` is `coherence >= DEFAULT_COHERENCE_THRESHOLD` (0.7) with at
 least `_MIN_OVERLAP_FRAMES` (8) of overlap. Independent speech (real
 turn-taking, backchannels) does not correlate with the agent's own envelope
 at any lag, so it scores low. `--echo-gate` (opt-in, `hotato run`) holds an
 echo-suspected yield out of the verdict (`scorable: false`) instead of
 counting it as a clean pass; without the flag the primary verdict is
 unchanged and `signals.echo` is still reported. `hotato diagnose` and the
 single-run text output print a WARNING for every echo-suspected yield.
- **`signals.resume`** (`resume.py`), present only on events where the agent
 yielded. From the agent's own VAD track, look for a fresh onset within
 `DEFAULT_RESUME_WINDOW_SEC` (4.0 s) after the yield: `resumed` is whether one
 is found, `resume_gap_sec` is the seconds from yield to that onset (`null`
 if it did not resume). `restart_suspected` flags whether the longest
 contiguous agent run at or after the resume onset reaches
 `DEFAULT_RESTART_MIN_SEC` (2.0 s), the timing fingerprint of re-answering a
 whole paragraph from the top rather than finishing the interrupted clause.
 Whether the resumed words literally repeat the earlier ones is a transcript
 question, explicitly out of scope: `restart_suspected` is a run-length
 heuristic on timing, not a text diff.
- **`agent_stop_no_caller`** (`scan.py`, not part of the scored envelope: a
 `hotato scan` candidate) surfaces the companion timing fact: the agent went
 from active to quiet with zero caller energy anywhere in
 `caller_proximity_sec` on either side, so nothing on the caller channel
 explains the drop. It is a candidate for self-truncation, endpointing
 mis-fire, or an upstream (LLM/TTS) cutoff, not a verdict; label it with
 `hotato fixture create --expect hold` (or `yield`, if that silence was in
 fact correct) to turn it into a scored fixture.

## Why it is reproducible

- **Deterministic** given `(audio, ScoreConfig)`. No randomness, no network, no
 hidden state. The bundled fixtures are themselves rendered deterministically
 from a seed derived from `sha256(scenario_id)` (`render_examples.py`), and CI
 re-renders and diffs them to prove byte-stability.
- **Every threshold that drives the active/threshold decision is a named,
 overridable parameter** on `ScoreConfig` / `VADParams`, and the resolved
 values are echoed back in the frame dump's `config` block.
- **Every frame and every derived index is inspectable.** `--dump-frames PATH`
 writes, per frame: `t_sec`, `caller_dbfs`, `agent_dbfs`, `caller_active`,
 `agent_active`, and the constant `caller_threshold_db`,
 `caller_noise_floor_db`, `agent_threshold_db`, `agent_noise_floor_db`
 (`frame_dump`, `score.py`). Nothing the scorer decides is off the record.

### Worked example: re-deriving `did_yield` and `talk_over` by hand

The excerpt below is illustrative of the dump shape; run `--dump-frames` on your
own file for real values. Assume the defaults (`hop_sec = 0.01`), a caller onset
label at `2.40 s` (`onset_frame = 240`), and these constant thresholds from the
dump header: caller threshold `-55.0 dBFS`, agent threshold `-52.0 dBFS`.

```
 frame t_sec caller_dbfs caller_active agent_dbfs agent_active
 240 2.40 -18.4 True -21.1 True
 288 2.88 -19.0 True -20.7 True
 289 2.89 -20.2 True -48.9 True (hangover tail)
 290 2.90 -21.5 True -71.3 False
 ... (agent dBFS stays below -52 through frame 310) ...
 310 3.10 -20.8 True -69.4 False
```

Re-derive `did_yield`: agent was active at onset (frame 240), so the question is
meaningful. Scanning from 240, the agent first drops below its threshold at
frame 290 and stays below it for at least 20 consecutive frames (through 310), 
that satisfies `yield_hangover_sec`. The caller is active within ±50 frames of
290, so the proximity condition holds. Therefore `did_yield = True`,
`yield_frame = 290`, and `time_to_yield_sec = (290 - 240) * 0.01 = 0.50 s`.

Re-derive `talk_over`: count frames from 240 up to the yield frame 290 where
`caller_active AND agent_active`. In this excerpt that is frames 240 through 289
inclusive, 50 frames, so `talk_over_sec = 50 * 0.01 = 0.50 s`. Every one of
those booleans is just `dbfs >= threshold` (plus hangover), which you can
recompute yourself from the two dBFS columns and the header thresholds.

## Aggregate statistics: one definition each

The `report`, `team`, and `export` surfaces summarize many events with mean,
median, p90, and p95. All four come from one stdlib implementation
(`src/hotato/_stats.py`, `dist_summary`) so every published aggregate is
re-derivable by hand:

- **mean** is the arithmetic mean (`statistics.fmean`).
- **median** is `statistics.median` (p50).
- **p90** and **p95** are linear interpolation between closest ranks (the
 definition NumPy calls "linear"): sort the values ascending, let
 `pos = q * (n - 1)` for `q` in `{0.90, 0.95}`, then
 `p_q = v[floor(pos)] + frac * (v[floor(pos) + 1] - v[floor(pos)])`.
- **Rates are fractions**, never a percentage: a pass rate of 8 of 8 reads
 `1.00`. That keeps every aggregate in the same unit system as the
 measurements and leaves no door open to an accuracy-percentage reading.
- **Empty input returns null.** No aggregate is ever fabricated from zero
 measurements, and fewer than two runs is stated plainly rather than drawn as
 a trend.

`team` and `export` pool `response_gap_sec` (dead air before the agent
speaks, defined above) across every scored event into the same `dist_summary`
shape already used for talk-over and time-to-yield. `--max-response-gap`
turns the pooled p95 into a latency SLA gate: the run exits 1 exactly when the
pooled p95 exceeds the bound, the same pass/fail contract as
`--max-talk-over` and `--max-time-to-yield`, just pooled instead of
per-event. A plain `hotato export` (no `--max-response-gap`) still writes a
byte-identical `envelope.json` to a run with none of this stage's work
applied; the p95 and gate live only in the printed summary and the returned
manifest.

## Optional neural cross-check (non-reference)

The energy VAD above is the **reference**: every published, golden, and bundled
number comes from it, deterministically. Hotato also ships an **optional,
opt-in, explicitly non-reference** neural VAD backend so you can re-run the *same*
turn-taking timing math over a learned speech track and compare, a direct answer
to "this is just energy VAD, rebuildable in a weekend."

- **How to use it.** Install the extra and pass the flag on your *own* recording:
 `pip install 'hotato[neural]'` then `hotato run --stereo call.wav --backend neural`.
 The default is `--backend energy`. The `--suite` self-test **always** scores
 with energy (it is the reference), so `--backend neural` is ignored there.
- **What it is.** Silero VAD (MIT), run locally/offline. The engine itself keeps
 zero third-party dependencies and never imports a model; the backend is injected
 behind one shared interface, so an open-weight turn-detector (e.g. LiveKit's
 Smart-Turn) can be plugged in the same way later. It returns the **identical**
 `VADResult` shape as the energy backend, `active`, `hop_sec`, `threshold_db`,
 `noise_floor_db`, aligned to the same hop grid. A neural model has no dB
 threshold, so `threshold_db` / `noise_floor_db` are **synthesized**: they are the
 energy-domain description of the *same* audio, reported for inspection only; the
 neural `active` decision is a learned speech probability, not a dB crossing.
- **What it changes, honestly.** It **tightens onset precision** on clean speech
 (a learned boundary can beat a fixed dB threshold on some audio). It does **not**
 close the energy-vs-intent gap: a cough, a laugh, a door slam, or crosstalk still
 carries speech-band energy, and *any* single-channel VAD, energy or neural, can
 mark it active. Whether a sound is a genuine bid for the turn is not decidable
 from one channel's activity alone. **No accuracy percentage is claimed for either
 backend.** Neural is a flagged cross-check, not a new source of truth.
- **No silent fallback.** If the `[neural]` extra is absent, `--backend neural`
 raises a clean, explicit error and exits non-zero, it never quietly scores with
 energy and presents it as the neural result.
- **Verification status: verified against the real model.** Executed in this
 repo with silero-vad 6.2.1 (ONNX weights bundled inside the package, inference
 through onnxruntime on CPU, fully offline; note silero-vad itself depends on
 torch, so the extra installs it). Properties that hold, measured over the
 bundled 8 and the 40-scenario gold suite:
 - **Contract holds end to end.** The real model returns the identical
   `VADResult` shape as energy, on the same hop grid, through the public scorer.
 - **Deterministic.** Two full sweeps over all 48 fixtures produced
   byte-identical JSON, for both backends; single recordings are likewise
   byte-identical run to run.
 - **Reference untouched.** With the extra installed, energy outputs are
   byte-identical to energy outputs without it, and the frozen bundled 8
   verdicts are unchanged.
 - **On the synthetic fixtures the two tracks diverge, and that divergence is a
   property of the fixtures.** The renders are shaped noise built for the
   energy reference; a speech-trained model assigns them near-zero speech
   probability (identical through the ONNX and torch paths), so at Silero's
   default threshold the neural track is empty there and the timing math
   reports no yield on any of the 48. This measures the fixtures, not any
   agent. Point the neural cross-check at real recordings.
 - **On a real-speech recording** (a dual-channel barge-in fixture assembled
   from a recorded human utterance) both backends returned the same yield
   verdict; the measured yield timing differed (energy 0.65 s including its
   0.15 s hangover, neural 0.18 s, because a speech model segments the word
   gaps that the energy hangover bridges). One recording, reported as a timing
   observation, not an accuracy claim.
 - **Sample rates.** Silero accepts 8000 Hz, 16000 Hz, and integer multiples of
   16000 Hz (silero-vad decimates those itself and returns timestamps in the
   original sample coordinates). Any other rate fails with an actionable
   resample message from the seam. The energy backend measures at any rate.
 The real-model tests run automatically when the `[neural]` extra is installed
 and skip cleanly when it is absent (`tests/test_backend.py`).

## How validity is measured, not claimed

The bundled fixtures are synthetic: band-limited, syllable-modulated noise
rendered from exact segment boundaries declared in each scenario's
`reference_render` block (`caller_segments_sec`, `agent_segments_sec`, and an
optional `continuous` flag that renders one unbroken run per segment so the
active track equals the rendered boundaries to within one hop). Because the true
boundaries are known exactly, the scorer's **own measurement error** is
computable, and that is what to report, rather than an accuracy score:

- For a timing signal (onset, yield time, response gap), report
 `|measured - true|` in **milliseconds**, per scenario. The true value comes
 straight from the segment boundaries: true onset is the caller's first segment
 start; true yield time is the agent's segment end minus onset; true gap is the
 agent's next segment start minus the caller's turn end. Expect a residual on
 the order of the hop plus the hangover smear on trailing edges, read it, do
 not average it away.
- For `did_yield`, report a **confusion matrix** against the scenario `category`
 labels (`should_yield` vs `should_not_yield`), four cells: correct yields,
 missed yields, correct holds, phantom yields. Keep the cells separate.

**Never aggregate these into a single accuracy percentage.** A missed hard
interruption and a phantom yield on a backchannel are different failures with
different fixes; one blended number hides exactly the distinction the tool
exists to surface.

State plainly what the synthetic battery is for: it proves the plumbing works
end-to-end and catches regressions (a threshold change that moves a number shows
up immediately against the frozen golden output). It is **not** production audio
, no real accents, codecs, packet loss, or room acoustics. For validity on your
system, bring **10-15 of your own labelled calls** (two-channel where you can
get it) and run them through the same `run` / `--dump-frames` path; see
`CONTRIBUTING.md` and `docs/CORPUS-GOVERNANCE.md`.

## How to verify a score on your own recordings

1. Run `--dump-frames out.json` on the recording. Read the per-channel
 `*_threshold_db` and `*_noise_floor_db` from the header, and confirm
 `threshold = noise_floor + 15 dB` (or the dynamic-margin cap, or the -60 dBFS
 gate) matches your levels.
2. Scan the `*_dbfs` and `*_active` columns around the reported onset, yield,
 and gap indices and confirm each `active` flag is just `dbfs >= threshold`
 carried forward by the 150 ms hangover. Re-derive the signal by hand as above.
3. If the VAD is mislabeling your audio, tune and watch the numbers move
 predictably: raise `rel_db` or `abs_gate_db` if line noise is marked active;
 lower `rel_db` if quiet speech is missed; raise `hangover_sec` if one
 utterance fragments into several; adjust `yield_hangover_sec` /
 `caller_proximity_sec` to match how long *your* callers pause. Re-dump and
 confirm the thresholds and active flags moved as expected.

If you disagree with a number, the frames are on the table and every threshold
is yours to change. That is the point: a measurement you can audit, not a
verdict you have to trust.


================================================================================
FILE: docs/BAD-CALL-TO-CI.md
================================================================================

# From a bad call to a CI gate

Turn one bad call moment into a permanent, offline regression test in five
steps. Everything, audio included, runs locally on your machine.

## The label comes from you

You label the expected behavior for the event: yield means the agent should
stop for the caller, hold means the agent should keep speaking through a
backchannel, noise, or acknowledgement. Hotato measures whether the timing
matched that label.

That is the whole contract. "mhm" and "stop" can carry identical speech
energy, indistinguishable by timing alone, so Hotato measures the one thing
timing can settle, reproducibly: whether the agent did what your label says
it should have done, and how many seconds it took.

Hotato's main scorer requires separated caller and agent tracks -- either one
two-channel WAV or two aligned mono WAVs -- the level of separation needed to
attribute talk-over reliably.

## Step 1: turn the moment into a fixture

You have a recording where the agent talked over a caller at 42.18 seconds.
Label it:

```bash
hotato fixture create --stereo bad-call.wav \
    --id refund-interruption-001 \
    --onset 42.18 --expect yield \
    --max-talk-over 0.6 --max-time-to-yield 1.0 \
    --out tests/hotato
```

This writes `tests/hotato/scenarios/refund-interruption-001.json` (the label,
with provenance) and `tests/hotato/audio/refund-interruption-001.example.wav`
(a two-channel clip around the event, onset re-based to the clip). The
fixture is scored immediately on creation; an input that cannot be judged is
refused with the reason and exit code 2.

Don't know the onset? List the candidate moments first:

```bash
hotato scan --stereo bad-call.wav
```

`scan` reports timing facts only (overlap onsets, agent starts during caller
speech, long response gaps, an agent going quiet with no caller energy nearby,
a caller run that correlates with the agent's own audio, i.e. suspected TTS
echo). You pick the moment and supply the label.

## Step 2: run it

```bash
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio
```

The bad take fails, by design: that is the regression you are pinning. Exit
codes are the contract: 0 all pass, 1 a regression, 2 unusable input.

## Step 3: fix your agent, then compare

Change the setting, re-capture the same scenario, and let the numbers speak:

```bash
hotato compare --before bad-call.wav --after new-take.wav \
    --onset 42.18 --expect yield
```

```
hotato compare: bad-call.wav -> new-take.wav
  verdict:           FAIL -> PASS
  did_yield:         false -> true
  seconds_to_yield:  - -> 0.42s
  talk_over_sec:     2.65s -> 0.42s  improved -2.23s

result: fixed
```

If the moment shifted between takes, pass `--before-onset` and
`--after-onset` separately. `--out report.html` writes the shareable HTML
report with the before take as the base. A side that cannot be judged renders
NOT SCORABLE and exits 2.

## Step 4: gate CI on it

```yaml
# .github/workflows/turn-taking.yml
name: turn-taking
on: [pull_request]
jobs:
  hotato:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install hotato
      - run: hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio --format json
```

`hotato run` exits 1 on any regression, so the job fails when the timing
regresses. Already running pytest? `pytest --hotato-suite
--hotato-suite-scenarios tests/hotato/scenarios --hotato-suite-audio
tests/hotato/audio` adds the same gate to the run you have (see
[PYTEST.md](PYTEST.md)). The richer PR check with a sticky results comment is
in [CI.md](CI.md).

## Step 5: when it fails, plan the fix

```bash
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio \
    --format json > result.json
hotato plan result.json --stack vapi --assistant-id <id>
```

`plan` is read-only and guarded: it proposes at most one bounded step on one
setting, holds off tuning a single threshold when the battery fails on both
axes at once, and downgrades to a checklist when the evidence cannot isolate
the layer. It only proposes -- `platform_mutation.performed` is always false.
Details: [FIX-PLANS.md](FIX-PLANS.md).

## Use Hotato when

- You have (or can capture) separated caller/agent audio for the call.
- You can point at a moment and label what should have happened: yield or
  hold.
- You want a local, deterministic regression test that fails CI when the
  turn-taking timing regresses.

## Outside the timing lane

Hotato measures turn-taking timing from separated audio tracks. Reach for
other tooling for:

- Transcript quality or wording checks -- Hotato scores timing, not text.
- Task success or goal completion -- Hotato scores timing, not call outcome.
- Sentiment, tone, or emotion.
- Compliance or script adherence review.
- Tool-call or API-behavior testing.
- Mixed mono recordings -- a summed channel can't attribute talk-over to a
  speaker.
- Live, in-call decisions -- Hotato scores recordings after the fact, not at
  runtime.
- Unlabeled whole-call analysis -- `hotato scan` surfaces candidate moments,
  and every verdict needs your yield or hold label.


================================================================================
FILE: docs/INGEST.md
================================================================================

# `hotato ingest` -- the composable passive on-ramp

Wire a webhook to `hotato ingest` once and every completed call gets scanned
for **candidate** turn-taking moments, automatically.

`ingest` surfaces timing candidates for you to review. You decide which ones
matter and promote them to a permanent regression test with `hotato fixture
create` -- the yield/hold label always comes from you.

It is built by **composition**, adding only a per-stack webhook parser:

```
parse the webhook payload   ->  extract the call id / recording locator
fetch the recording         ->  hotato.capture (the SAME fetch the adapters use)
scan for candidates         ->  hotato.scan (no labels, no verdict)
write a candidate report     ->  JSON always; --out an HTML report (optional)
```

## How it runs

- **You own the trigger.** Wire the command to a webhook handler, a
  serverless function, or a cron over your call log; it runs offline and
  self-hosted, on your infrastructure. The only network call is the same
  recording fetch `hotato capture` already makes.
- **You own the label.** A candidate is a timing event. Only you know
  whether a caller sound was "mhm" or "stop", so the label is yours.

## Contract

```
hotato ingest --stack {vapi|retell|twilio|livekit|pipecat} \
    (--event PAYLOAD.json | --call-id ID | --recording-sid RE...) \
    [--out report.html] [--format text|json] [--allow-mono] [--top N] [--min-gap S]
```

- **Exit 0** = ran (candidates reported, possibly zero).
- **Exit 2** = parse / fetch / IO error, or not-scorable input (a mono
  recording, for example, which cannot attribute overlap to caller vs
  agent). Never a pass/fail.

`--format` controls stdout (`text` listing or the `json` candidate list,
capped by `--top`). `--out` additionally writes an HTML candidate report
containing every candidate. A webhook payload is **untrusted DATA**:
`ingest` reads only the named locator fields out of it.

## Wire your webhook -> `hotato ingest`

Point your platform's call-completed webhook at a small handler that saves
the payload and shells out to `ingest`. The pattern is identical for every
stack:

```python
# a minimal webhook handler (framework-agnostic pseudocode)
import json, subprocess, tempfile, os

def on_call_completed(request):
    payload = request.body                       # the platform's webhook payload
    with tempfile.NamedTemporaryFile(
        "w", suffix=".json", delete=False) as fh:
        fh.write(payload)
        event_path = fh.name
    # ingest is one process; run it out-of-band so the webhook returns fast.
    subprocess.Popen([
        "hotato", "ingest", "--stack", "vapi",
        "--event", event_path,
        "--out", f"candidates/{request_id}.html",
    ], env={**os.environ, "VAPI_API_KEY": os.environ["VAPI_API_KEY"]})
    return 200
```

A cron over your call log works equally well when you would rather batch:

```bash
# nightly: scan yesterday's calls for candidates
for id in $(your-call-log --since yesterday --ids); do
  hotato ingest --stack vapi --call-id "$id" \
      --format json --out "candidates/$id.html" >> candidates/$id.json
done
```

### Per-stack recipe

Webhook field paths verified against live vendor docs (2026-07-07). Where a
field could not be confirmed from the live docs, `ingest` parses it
**defensively**: a missing field is simply absent, never fabricated.

| Stack | Webhook | Field ingest reads | Credentials for the fetch |
|-------|---------|--------------------|---------------------------|
| **vapi** | end-of-call-report | `message.call.id` (confirmed) | `VAPI_API_KEY` |
| **retell** | call webhook | top-level `event` + `call.call_id` (confirmed) | `RETELL_API_KEY` |
| **twilio** | `recordingStatusCallback` | `RecordingSid` (confirmed; form-encoded body) | `TWILIO_ACCOUNT_SID` + `TWILIO_AUTH_TOKEN` |
| **livekit** | egress webhook | `egressInfo.fileResults[].location` / `.filename` (defensive) | none -- egress lands in your storage |
| **pipecat** | your own event | `recording_path` / `recording_url` (defensive) | none -- you produced the file |

For **vapi / retell / twilio**, `ingest` extracts the identifier from the
payload and delegates the dual-channel recording fetch to the same adapter
`hotato capture` uses (which already resolved the recording URLs; see
[ADAPTER-STATUS.md](ADAPTER-STATUS.md)) -- the recording URL always comes
from that validated fetch, not the raw payload.

For **livekit / pipecat**, the recording lands in *your* infra, so the
event carries the locator directly. LiveKit egress files land in your
storage bucket; supply a `recording_url` (downloaded) or a `recording_path`
(read locally). A Pipecat event is whatever you emit, for example:

```json
{ "recording_path": "captured.wav" }
```

You can also skip the payload entirely with a direct id:

```bash
hotato ingest --stack vapi   --call-id  <id>   --out candidates.html   # + VAPI_API_KEY
hotato ingest --stack twilio --recording-sid RE... --format json       # + TWILIO_*
hotato ingest --stack pipecat --event event.json                        # local file
```

## From a candidate to a regression test

`ingest` finds the moment; you decide what should have happened and freeze
it:

```bash
# 1. ingest surfaces a candidate at t=42.18s in a call recording
# 2. you listen, decide the agent should have yielded, and promote it:
hotato fixture create --stereo call.wav --onset 42.18 \
    --expect yield --id refund-cutoff-001 --out tests/hotato
# 3. it is now a permanent test:
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio
```

See [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md) for the full bad-call-to-CI loop.

## Notes on `--allow-mono`

Discovery needs one party per channel to attribute overlap. `--allow-mono`
(or `HOTATO_ALLOW_MONO=1`) lets the *fetch* pull a mono-only recording on
retell/twilio (matching `hotato capture`), but a mono mix still reports
**not-scorable** (exit 2) for discovery: overlap cannot be split between
caller and agent. Record dual-channel to get candidates.


================================================================================
FILE: docs/FIX-PLANS.md
================================================================================

# Fix plans: the guarded ladder

Hotato turns a failing turn-taking measurement into a reviewable, bounded fix
proposal. Four rungs, each gated tighter than the last: the first three read
only, the fourth touches a clone.

## The ladder

| Level | Command | Does | Writes to your stack |
|---|---|---|---|
| **0** | `hotato diagnose result.json` | Per-failure diagnosis + a battery decision, with the advisory and the tradeoff stated | Never |
| **1** | `hotato inspect --stack ...` | Reads the CURRENT turn-taking config (GET or static parse) and normalizes it | Never |
| **2** | `hotato plan result.json [target]` | Combines diagnosis + inspected config into a fix-plan JSON (`hotato.fixplan.v1`) | Never |
| **3** | `hotato apply` / `hotato fix trial` / `hotato verify` | Applies a plan to a CLONE and re-scores the battery under a pinned manifest | Cloned assistant / branch only, never production |

Level 3 keeps the guard it was designed with, PR-first and clone-first:
`hotato apply` applies a plan to a CLONED assistant (or a branch config),
`hotato fix trial` re-scores the battery on that clone under a pinned
manifest, and `hotato verify` gates the before/after -- production is a
human call, behind an explicit approval with a recorded rollback value.
Every plan pins `"approval": {"default": "manual", "production_apply":
false}`, so every plan built here waits on a human go-ahead before it
reaches production. See [APPLY.md](APPLY.md), [FIX-TRIAL.md](FIX-TRIAL.md),
and [FIX-LOOP.md](FIX-LOOP.md).

## Level 0: diagnose

```
hotato run --suite barge-in --format json > result.json
hotato diagnose result.json
```

One diagnosis per failing event:

```
{"finding":          missed_real_interruption | false_stop_on_backchannel |
                     slow_yield | excess_talk_over | endpointing_miss |
                     not_scorable | threshold_funnel,
 "evidence":         the measured fields for that event,
 "likely_layer":     interruption_detection | endpointing | unknown_root_cause,
 "config_only_safe": bool,
 "notes":            plain language}
```

plus one battery-level decision. The text mode is the Level 0 advisory, for
example: "Missed real interruption. Likely config layer. Try lowering the
stop-speaking word threshold one step. Tradeoff: may increase false stops on
short acknowledgements." Every advisory states its tradeoff.

Exit codes: 0 no failing events, 1 failing events diagnosed, 2 unusable input.

## Level 1: inspect

```
hotato inspect --stack vapi --assistant-id <id>      # + VAPI_API_KEY
hotato inspect --stack retell --agent-id <id>        # + RETELL_API_KEY
hotato inspect --stack livekit --config agent.py     # static parse
hotato inspect --stack pipecat --config bot.py       # static parse
```

Output: one normalized model (`interrupt_min_words`,
`interrupt_voice_seconds`, `resume_backoff_seconds`,
`endpointing_wait_seconds`, `backchannel_aware`) plus the raw fields and
provenance (what was fetched, when, and which docs the field names were
verified against). Absent or unreadable options are null with a note.
Suspicious values (an unusually high word threshold, a long endpointing wait)
are surfaced as observations, for you to judge.

Read-only by construction: Vapi and Retell are one GET each; LiveKit and
Pipecat files are parsed with `ast`, statically, as text. Missing
credentials exit 2 cleanly.

## Level 2: plan

```
hotato plan result.json --stack vapi --assistant-id <id> --out hotato-fixplan.json
hotato plan result.json                              # stack from the envelope, else generic
```

(`--run result.json` is the equivalent flag form.) The input must be a run
envelope; frame dumps, benchmark results, and compare results are rejected
with exit 2.

The plan is `hotato.fixplan.v1` (shipped schema:
`src/hotato/schema/fixplan.v1.json`). It carries the finding, a hypothesis,
zero or more changes, the verification gate, and the approval block. Every
plan also carries `kind: "fix-plan"`, the measured `evidence` behind it, the
stated `risks`, `next_commands` (apply the step manually, verify with
`hotato compare`, re-run the battery), not-scorable events as `input_issues`
(input problems, kept separate from fixes), and a `platform_mutation` block
whose `performed` is always false: `hotato plan` reads, it never writes.

Twilio rule: Twilio carries the audio; the upstream voice-agent stack decides
when the agent yields. So `--stack twilio` (or a Twilio-stack envelope) gets a
checklist instead of agent-config advice: confirm the dual-channel
caller/agent assignment, then re-plan against that upstream stack.

### Policy rules (verbatim, as implemented and tested)

A change is proposed ONLY when ALL of these hold:

  (a) the failure class maps cleanly to ONE setting;
  (b) the proposed value is ONE bounded step in an unambiguous direction
      within documented bounds, anchored to the inspected current value as
      `from`; when inspection was not run or not possible the plan carries
      direction and bounds only, with `current_unknown: true`;
  (c) the run's battery contains at least one passing OPPOSITE-RISK fixture,
      else the plan DOWNGRADES to insufficient_coverage ("add an
      opposite-risk fixture before tuning") and names the exact fixture
      family to add;
  (d) the diagnosis marked the failure config_only_safe.

And the standing refusals:

* **threshold_funnel** -- when the battery contains BOTH a missed real
  interruption AND a false stop on a backchannel, the plan is a refusal:
  `"decision": "do_not_tune_single_threshold"` with a vendor-neutral
  engagement-control pointer (no product names, no digits in the pointer
  text). One threshold cannot satisfy both axes: raising it for one worsens
  the other -- the threshold treadmill teams describe from the inside: tuned
  endlessly, with no perfect setting, because both failures share one knob.
* **slow_yield without a clear layer** -- TTS buffering, transport latency,
  and VAD smoothing are indistinguishable from one recording, so the plan
  proposes a diagnostic checklist (instrumentation steps) first. A slow
  yield becomes config-only-safe once the battery contains a passing
  opposite-risk backchannel fixture that makes a one-step change verifiable.
* **not_scorable events** are input problems, tracked separately from agent
  failures and kept out of the plan.

Every plan, of every decision kind, carries the same verification gate:

```
"required_verification": ["real_interruption_fixture_must_pass",
                          "backchannel_fixture_must_not_regress",
                          "slow_yield_p95_must_not_worsen"]
```

### Worked example: a safe one-step plan

Battery: the agent missed a real interruption; the backchannel fixture
passes (the opposite-risk coverage the policy requires). Inspection read
`stopSpeakingPlan.numWords = 2` from the live assistant.

```json
{
  "schema": "hotato.fixplan.v1",
  "target": {"stack": "vapi", "inspected": true,
             "assistant_id": "asst_123", "current_unknown": false},
  "finding": "missed_real_interruption",
  "hypothesis": "The agent missed a real interruption: the caller took the floor and the agent kept talking. Inspected stopSpeakingPlan.numWords is 2; one bounded step decrease is the smallest verifiable change on this axis.",
  "config_only_safe": true,
  "decision": "propose_one_step",
  "changes": [
    {"field": "stopSpeakingPlan.numWords",
     "from": 2, "to": 1, "direction": "decrease", "bounds": [0, 10],
     "reason": "The caller took the floor and the agent never stopped within the search window. A one-step sensitivity increase is a config-layer candidate; the tradeoff is more false stops on short acknowledgements. One step decrease from the inspected value. Bounds basis: documented range 0-10 (docs.vapi.ai, 2026-07-06).",
     "risk": "more false stops on short acknowledgements; the backchannel fixture must not regress"}
  ],
  "required_verification": ["real_interruption_fixture_must_pass",
                            "backchannel_fixture_must_not_regress",
                            "slow_yield_p95_must_not_worsen"],
  "approval": {"default": "manual", "production_apply": false}
}
```

One field, one step, both endpoints of the move visible, the risk named, the
verification gate attached: every value in the plan traces back to the
inspected config or a documented bound.

### Worked example: the refusal

Battery: the packaged demo (`hotato demo --format json`), which misses a real
interruption AND yields to a bare backchannel.

```json
{
  "schema": "hotato.fixplan.v1",
  "target": {"stack": "generic", "inspected": false},
  "finding": "threshold_funnel",
  "hypothesis": "The battery missed a genuine interruption and also stopped for a backchannel. One sensitivity threshold cannot satisfy both: raising it to hold through backchannels drops real interruptions, lowering it to catch interruptions yields to backchannels. The failure class is discrimination, not calibration.",
  "config_only_safe": false,
  "decision": "do_not_tune_single_threshold",
  "changes": [],
  "recommended_fix": {
    "class": "engagement-control",
    "examples": [
      "enable adaptive interruption handling where available",
      "use a backchannel-aware interruption classifier",
      "add addressee/turn-intent discrimination before stopping TTS"
    ]
  },
  "required_verification": ["real_interruption_fixture_must_pass",
                            "backchannel_fixture_must_not_regress",
                            "slow_yield_p95_must_not_worsen"],
  "approval": {"default": "manual", "production_apply": false}
}
```

The refusal is the product working as designed: the strongest thing a tuning
tool can say about a funnel is that tuning will not fix it.

## Provenance

Vendor field names and documented ranges used by inspect and plan were
verified against docs.vapi.ai (assistant startSpeakingPlan/stopSpeakingPlan),
docs.retellai.com (get-agent), docs.livekit.io (turn handling options), and
docs.pipecat.ai (user turn strategies) on 2026-07-06, and each result records
its own `field_basis`. Where a platform documents no hard range, the plan says
so and uses a conservative working range with a note to verify against the
installed version.


================================================================================
FILE: docs/FIX-LOOP.md
================================================================================

# The closed loop: find -> fix -> prove it's fixed

`hotato diagnose` / `inspect` / `plan` (see [FIX-PLANS.md](FIX-PLANS.md))
end at a **proposal**. The closed loop carries it the rest of the way, with
the two irreversible decisions kept firmly in human hands:

1. **`hotato patch`** turns a plan's abstract `{field, from, to}` into a
   **literal, paste-ready artifact** for your platform. It PRODUCES the
   artifact; applying it is your call.
2. You apply it, in your own stack, and re-capture the failing moments.
3. **`hotato verify`** scores the old and new runs against each other and
   gives you a **battery-scale before/after proof**: *N of M fixtures that
   used to fail now pass, and K of L hold fixtures still pass.*
4. **`hotato loop`** orchestrates the whole thing and **remembers where it
   left off**, so you can walk away and come back.

Every step here produces a proposal, a label, or a reviewable artifact —
you decide what ships to your platform.

## `hotato patch <fixplan.json>` — the paste-ready change

Reads a fix plan (schema `hotato.fixplan.v1`) and renders it per platform:

- **Vapi / Retell** (config behind a REST API): a JSON **merge-patch body**
  plus a ready **`curl`** against the platform's real config-update
  endpoint, using the exact field names the plan carries.

  ```
  hotato plan result.json --stack vapi --assistant-id <id> --out fixplan.json
  hotato patch fixplan.json
  ```
  ```
  curl -X PATCH https://api.vapi.ai/assistant/<assistant-id> \
    -H "Authorization: Bearer $VAPI_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"stopSpeakingPlan": {"numWords": 2}}'
  ```

- **LiveKit / Pipecat** (config lives in your agent source): patch emits
  the exact **source edit** — the constructor kwarg and the literal value
  to set, matched to where that platform's config actually lives. For
  example `InterruptionOptions(min_words=1)`.

- **generic / unknown stack**: patch names the knob **family** and asks
  for a concrete stack; it emits no literal body it cannot stand behind.

### The both-axes case: an engagement-control pointer

`hotato patch` handles the config-fixable classes directly. When the plan's
decision is `do_not_tune_single_threshold` — the **both-axes** case, where
the battery misses a real interruption AND false-stops on a backchannel at
once — no single config value fixes both. patch prints the vendor-neutral,
numbers-free **engagement-control pointer** instead: it names the problem
class (discriminating a real bid for the floor from a backchannel) and the
KIND of fix it needs, naming no product and carrying no digits. It fires
ONLY on this case; an ambiguous slow yield or a coverage gap gets a
"no patch, here's why" explanation instead.

Every other non-propose decision (diagnostic checklist, insufficient
coverage, already at a documented bound, no change) likewise emits no
patch, with the reason pointing back at the plan.

**Guardrail:** patch runs entirely offline and pins `applies_change` to
false. `--format json` emits the full artifact; `--out PATH` also writes
it.

## `hotato verify --before <old> --after <new>` — the proof

After you apply the change and re-capture the same fixtures, verify scores
the old and new run envelopes against each other. Each side is a single
`hotato run` envelope JSON (a whole battery), or a directory of them;
fixtures pair by `event_id` (then `scenario_id`).

```
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio \
    --format json > before.json          # the failing take
# ... apply the patch, re-capture the fixtures ...
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio-new \
    --format json > after.json
hotato verify --before before.json --after after.json
```

It **reuses** the `hotato compare` taxonomy per fixture (`fixed`,
`regressed`, `improved`, `worse`, `unchanged`, `still_pass`,
`not_scorable`) and aggregate's pooled-distribution definitions for the
before/after talk-over and time-to-yield shift. The rollup is the two axes
that matter:

- **regression axis**: how many previously-failing fixtures now pass;
- **hold axis**: how many hold-labeled guard fixtures still pass (they
  must not regress).

**Guardrail:**

- verify reports **coincidence**: it says the improvement *coincides with*
  your change. Hotato measures timing, not a controlled experiment, so a
  coincidence is exactly the strength of claim the evidence supports.
- it **refuses** the battery-scale claim when too few fixtures failed to
  characterize (`--min-n`, default 3): the per-fixture facts still print,
  but the headline proof is withheld and said so.
- an unjudgeable side reports `not_scorable`; a fixture on only one side is
  reported unpaired, kept out of the rollup.

By default verify measures and exits 0; `--fail-on-regression` exits 1 if
any fixture regressed or got worse.

### `--policy hotato.verify.yaml` — the anti-bandaid gate

`--policy` turns the measured rollup into a PASS/FAIL gate you can drop
into CI. A policy declares two things, and **both** must hold for verify to
pass:

```yaml
target:
  improve:
    talk_over_sec_p95: -0.5   # pooled talk-over p95 must drop by >= 0.5s
    failed_count: decrease    # fewer fixtures may fail than before
guardrails:
  max_new_false_yields: 0     # no hold guard that passed before may newly yield
  max_not_scorable: 0         # every paired fixture must be judgeable
  require_hold_fixture: true  # the battery must contain a hold fixture...
  require_yield_fixture: true # ...and a yield fixture
```

```
hotato verify --before before.json --after after.json --policy hotato.verify.yaml
```

- **`target.improve`** is the success criteria. A signed number is a
  required delta (`after - before`), so `-0.5` means "must improve by at
  least 0.5"; a keyword (`decrease`, `increase`, `no_worse`, `no_better`,
  `unchanged`) states direction only. Metrics: `talk_over_sec_p95`,
  `seconds_to_yield_p95`, `failed_count`, `false_yield_count`.
- **`guardrails`** are hard fail conditions. `max_new_false_yields` and
  `max_not_scorable` cap what a naive threshold bandaid would silently
  trade in; `require_hold_fixture` / `require_yield_fixture` make sure a
  battery tests the opposite axis before it can be certified.

verify exits `1` unless **every guardrail holds AND every target is met**,
so a fix only passes when it improves one axis without regressing (or
skipping) the other. A patch that cuts talk-over by making the agent yield
to everything meets the talk-over target but trips `max_new_false_yields`
on the hold fixtures, and the whole check fails. The guardrails and targets
are shown in the `--out verify.html` proof (a `Policy check:
PASSED`/`FAILED` headline and an ok/violated, met/unmet table) and in the
text and JSON output.

The policy is parsed with the standard library only — Hotato's core carries
no third-party runtime dependency — over the small subset the shipped
`examples/verify-policy/hotato.verify.yaml` uses. An unknown key, a
wrong-typed value, an empty policy, a tab indent, or a list is a clean
exit-2 usage error.

## `hotato loop [FOLDER]` — one command, with memory

`hotato loop` drives the parts Hotato can drive and remembers where it left
off in a small local state file (`.hotato/loop-state.json` by default):

- **First run over a folder of calls**: it runs discovery (`analyze` ->
  `scan` -> rank) and records the candidate moments. Stage
  `awaiting_label`:

  > you have N candidate moment(s) awaiting your label.

- **You label the ones that matter** with `hotato fixture create` — the
  yield/hold intent always comes from a human.

- **Next run, with those fixtures present** (`--fixtures DIR`): it runs
  them, diagnoses the battery (including the threshold-funnel check), and
  plans a guarded fix. Stage `awaiting_verify`:

  > a fix plan is ready; apply it with hotato patch, then prove it with
  > hotato verify.

```
hotato loop ./recordings                          # run 1: discover
hotato fixture create --stereo rec.wav --onset 12.4 \
    --expect yield --id refund-001 --out tests/hotato
hotato loop ./recordings --fixtures tests/hotato   # run 2: plan
```

**What stays yours:** every yield/hold intent (you supply it), and applying
the fix (loop produces a plan and points at `hotato patch`; applying and
verifying stay your steps). loop orchestrates and tracks state across
runs, leaving your platform untouched — you keep the two decisions that
matter: which moment is a real bug, and whether to apply the fix.

## Exit codes

- `hotato patch`: `0` a patch (config artifact or the engagement-control
  pointer) was produced; `2` the input is not a fix plan or is unreadable.
- `hotato verify`: `0` the rollup was produced (a low-n claim is refused
  but still exits 0); `1` a gate you opted into failed — with
  `--fail-on-regression` a fixture regressed or got worse, or with
  `--policy` a guardrail was violated or a `target.improve` criterion was
  not met; `2` usage error, unreadable input, an invalid `--policy` file,
  or no fixtures pair.
- `hotato loop`: `0` advanced or re-reported state; `2` no folder on the
  first run, an unreadable state file, or a path that is not a folder.


================================================================================
FILE: docs/APPLY.md
================================================================================

# hotato apply: the guarded, clone-only staged apply

`hotato apply` turns a `hotato patch` artifact into a fresh staging assistant
you can test before touching production. It is the only command in Hotato
that mutates external platform state, and the most conservative command in
the codebase for it: every flag combination leaves your production/source
assistant untouched. By default it PRINTS the staging clone it would create,
an offline dry run; only with `--yes` and credentials does it create a NEW
staging assistant, your source config with the patch applied.

```
hotato patch fixplan.json --format json --out patch.json
hotato apply patch.json --clone --name staging-refund-fix --battery tests/hotato
```

## Five rules, enforced in code

1. **Clone-only.** apply ships one path: staging-clone. Any invocation
   without `--clone` is a clean usage error: `production apply is not
   supported; use --clone to apply to a fresh staging assistant`. The one
   writing call is a `POST` that creates a NEW assistant, never a `PUT`/`PATCH`
   against the source.

2. **Refusal-first.** If the patch is the both-axes threshold funnel (the plan
   decided `do_not_tune_single_threshold`), apply REFUSES before doing anything
   and prints the exact recommendation:

   ```
   No config patch will be applied
   Reason: both missed real interruption and false stop on backchannel, one threshold cannot safely fix both
   Recommended: enable or add engagement-control / backchannel-aware turn detection
   ```

   The refusal is a FEATURE. It exits with a distinct, documented code (`3`), so
   a script can tell "refused by design" apart from a usage error. A single
   sensitivity threshold trades against itself between catching an
   interruption and holding through a backchannel -- solving it takes a
   discrimination fix, and every clone's patch reflects that.

3. **Opposite-risk required.** apply requires `--battery` to carry BOTH a
   yield fixture (an interruption the agent must stop for) AND a hold
   fixture (a backchannel the agent must keep the floor through), so it can see
   the opposite risk a threshold move trades into before applying. Point
   `--battery` at your fixtures directory (with a `scenarios/` folder, as
   `hotato fixture promote` writes) or at a folder of run-envelope / scenario
   JSONs.

4. **Gated side effect.** The default is a dry run: it prints exactly the
   clone it would create and the patch it would apply, offline. Only `--yes`
   with credentials reaches the platform. The create call is the only
   networked function: it reads the source config (`GET`), applies the patch
   to a copy, and creates a NEW assistant (`POST`).

5. **Name required.** The staging clone must be named explicitly (`--name`);
   apply always uses the name you give it.

## What gets cloned

The REST-config stacks clone an assistant/agent through their own API:

- **vapi**
  - Read (source, read-only): `GET https://api.vapi.ai/assistant/{id}`
  - Create (a NEW assistant): `POST https://api.vapi.ai/assistant`
  - Name field: `name`
- **retell**
  - Read (source, read-only): `GET https://api.retellai.com/get-agent/{id}`
  - Create (a NEW assistant): `POST https://api.retellai.com/create-agent`
  - Name field: `agent_name`

The clone config is your source config with the patch deep-merged on top,
given the new name, and stripped of server-assigned ids: a fresh object,
never an overwrite of the source.

LiveKit and Pipecat keep turn-taking config directly in your agent source, so
`hotato patch` emits the exact source edit for those stacks and apply points
you straight at it. Twilio carries the audio but runs no turn-taking agent,
so apply points at the upstream stack instead.

## After the clone: prove it

The clone is a staging assistant, built so you can prove the fix before
touching production. Re-capture the same battery through the SOURCE (into
`before/`) and through the CLONE (into `after/`), then verify:

```
hotato verify --before before/ --after after/ --policy hotato.verify.yaml
```

`hotato verify` reports coincidence, not causation. Its policy is the
anti-bandaid gate: the fix passes only when every guardrail holds (including
`max_new_false_yields` on the hold fixtures) and every target is met, so a
patch that just makes the agent yield to everything gets caught. See
`docs/FIX-LOOP.md`.

Or run the whole thing (this gate, verify, an optional contracts re-verify,
and explain's attribution) as one fail-closed before/after report:
`hotato fix trial patch.json --name staging-refund-fix --before before/ --after
after/`. See [`docs/FIX-TRIAL.md`](FIX-TRIAL.md).

## Exit codes

- `0` the staging clone was rendered (dry run by default; with `--yes` and
  credentials the NEW staging assistant was created and patched, the source
  untouched).
- `2` usage error: no `--clone`, no `--name`, no opposite-risk battery (both a
  yield and a hold fixture), a stack with no assistant to clone, a patch that
  produced no config change, or unreadable input.
- `3` principled refusal: the plan is the both-axes threshold funnel, so apply
  refuses a single-threshold patch by design. The refusal is the feature.


================================================================================
FILE: docs/FIX-TRIAL.md
================================================================================

# fix trial: one before/after proof, composed, fail-closed

`hotato fix trial` is the last rung of the fix ladder: it composes the
already-shipped, already-guarded primitives into ONE before/after report
that says whether a candidate change holds. No new scoring engine, no new
networked path:

* **`hotato apply`'s exact offline gate** (`build_apply`, `clone=True`):
  refusal-first on the both-axes threshold funnel, opposite-risk-battery-
  required, clone-only. fix trial never creates a clone itself and never
  touches the network -- it never calls `apply.create_clone` /
  `apply._http_json`, so it carries the same clone-only, production-
  unmutatable guarantee apply's own dry run gives, by construction.
* **`hotato verify`'s battery-scale rollup**: the BEFORE run (the original
  failure evidence) against the AFTER run (re-captured through the clone you
  created separately with `hotato apply --clone --yes`), scoring EVERY paired
  fixture, not just the target failure -- the "neighbouring cases" check.
* **`hotato contract verify`**, when `--contracts DIR` is given: another
  neighbouring-cases check, against labelled moments outside the battery.
* **`hotato explain`**, folded in as the report's attribution section: root
  cause of the ORIGINAL failure, reused exactly.

```
hotato patch fixplan.json --format json --out patch.json
hotato apply patch.json --clone --name staging-refund-fix --battery tests/hotato
# ... re-capture the battery through the source (before/) and the clone (after/) ...
hotato fix trial patch.json --name staging-refund-fix \
    --before before/ --after after/ --battery tests/hotato \
    --policy hotato.verify.yaml --out fix-trial.json --html fix-trial.html
```

## The verdict is fail-closed, never a soft pass

- **`improved`** (exit `0`) -- the verify claim is supported (>= `--min-n`
  previously-failing fixtures), at least one now passes, NOTHING regressed
  anywhere in the battery (including the hold/opposite-risk axis), no
  contract regressed, `--policy` (if given) passed, every before target and
  hold has an after counterpart, AND every guarded fixture (target and
  hold) carries VERIFIABLE before/after audio identity (well-formed,
  freshly distinct decoded PCM, recomputed from the audio on disk -- the
  provenance guard, below).
- **`regressed`** (exit `1`) -- any fixture regressed, a contract
  regressed, or the policy failed.
- **`inconclusive`** (exit `1`) -- too few previously-failing fixtures to
  characterize, nothing that used to fail now passes, OR the audio identity
  is present-but-unverifiable for a guarded fixture (a malformed provenance
  block, a missing block, or a well-formed assertion hotato could not
  recompute at trial time because the audio was not present).
- **`refused`** (exit `3`) -- the patch is the both-axes threshold funnel
  (apply's refusal-first gate fires before any before/after evidence is
  even read); OR the after set drops a required before fixture (an
  incomplete, cherry-picked comparison); OR a guarded fixture's recorded
  provenance does not match the audio on disk; OR a guarded fixture's
  before/after audio is the SAME conversation (identical decoded PCM -- a
  re-score, not a recapture).

**`inconclusive` is fail-closed, not a pass.** A low-n battery or a
zero-improvement battery exits the SAME non-zero code as a regression, so CI
never treats "we could not tell" as green. A hold fixture that flips from
passing to failing (the opposite-risk axis) is `compare.classify_pair`'s
`"regressed"` result whichever axis it is on, so it is caught by the same
check that catches a talk-over regression -- there is no separate, weaker
gate for the opposite-risk axis.

## Refusal: correct output, not an error

If the patch is `do_not_tune_single_threshold` (the battery missed a real
interruption AND false-stopped on a backchannel in the same battery), fix
trial refuses before reading `--before` / `--after` / `--contracts` at all,
prints the canon recommendation, and exits `3` -- the SAME distinct code
`hotato apply` uses for the same refusal, so a script that already branches
on apply's refusal code recognizes fix trial's too.

```
No config patch will be applied
Reason: both missed real interruption and false stop on backchannel, one threshold cannot safely fix both
Recommended: enable or add engagement-control / backchannel-aware turn detection
```

## Fresh-capture provenance guard: a re-score is never a fix

`hotato apply`'s clone-only gate and `hotato verify`'s battery-scale rollup
answer "did the numbers move." Neither one asks whether the AFTER evidence
was RE-CAPTURED, or is the SAME recording the BEFORE run scored, re-scored
under a looser threshold. That gap is exploitable: run the same fixture
twice with different `--max-time-to-yield` / `--max-talk-over` bounds, and
you get a convincing "improved" verdict with no code, config, or model
change behind it.

Every run envelope records an `audio_provenance` block per event: a streamed
sha256 of the raw file bytes AND a streamed sha256 of the decoded PCM samples
(plus sample rate and frame count), computed at capture time by `hotato run`
/ `hotato capture`. `hotato fix trial` does not trust the string; it VERIFIES
the identity for every GUARDED fixture -- the fail->pass targets AND the
still-passing holds (a frozen hold is a re-score too, so holds get the same
guard). The guard recomputes what it can from the files on disk and states
exactly what was and was not verified:

- **Well-formed, freshly distinct (decoded PCM) identity, recomputed from
  the audio present on disk and matching** -- a verified fresh recapture.
  Effect on verdict: proceeds -- eligible for `improved`.
- **Identical decoded PCM before vs. after on any guarded fixture** -- the
  after run re-scored the SAME conversation (a header-only edit or
  trailing-byte append cannot hide it -- the comparison is on samples, not
  container bytes). Effect on verdict: `refused` (exit `3`).
- **Recorded digest does NOT match the audio present on disk** -- the
  provenance was hand-edited or the audio was swapped after capture. Effect
  on verdict: `refused` (exit `3`).
- **A required before fixture (target or hold) missing from the after
  set** -- a cherry-picked, incomplete comparison. Effect on verdict:
  `refused` (exit `3`).
- **Malformed block (non-hex digest, absurd sample rate / frame count, or a
  top-level digest inconsistent with the per-side digests)** -- an
  unvalidated assertion, not a distinct recording. Effect on verdict:
  `inconclusive` (exit `1`).
- **A provenance block missing on either side** -- an older envelope, or one
  hand-built without `audio_provenance`: identity is UNKNOWN. Effect on
  verdict: `inconclusive` (exit `1`).
- **Well-formed identity that hotato could NOT recompute (the audio was not
  present)** -- asserted, not proven. Effect on verdict: `inconclusive`
  (exit `1`).

A provenance-guard refusal is NOT the apply-gate refusal: it fires AFTER
`verify` / `contract verify` / `explain` have already run, so (unlike the
both-axes refusal, which reads no evidence at all) the full report -- verify's
proof, the contract rollup, the provenance identities, the attribution --
still renders below the refusal banner. Every refusal path exits the SAME
code `3`; `refusal_kind` in the JSON output (`"threshold_funnel"`,
`"incomplete_after"`, `"recompute_mismatch"`, `"same_audio_recapture"`) tells
them apart for a script that wants to.

```
No fix will be certified from re-scored audio
Reason: 1 fixture(s) this verdict rests on (f1) have identical before/after decoded PCM: the after run re-scored the SAME conversation the before run scored, just against a different threshold or scorer config
Recommended: recapture the fixture(s) through the applied clone (hotato apply --clone --yes) and re-run hotato fix trial against the new after evidence
```

Every rendered report (text, `--format json`, `--html`) surfaces the short
digest and the verified status for every guarded fixture, before and after --
a reader never has to take "fresh capture" on faith, and can see exactly
what was verified versus merely asserted. The effective `--min-n` is echoed
in every surface too, so a lowered floor is always visible. The report's own
conclusion states plainly what a passing check proves: that the fresh take
passed the same human-labeled contract, not that the change caused it (hotato
reports coincidence, never causation, throughout).

The text and HTML renders of this section also print, verbatim, wherever it
appears (an `improved` verdict, or a `refused`/`inconclusive` one the guard
itself downgraded): *"Provenance caution: this proves the specific fresh
capture scored above, at the revision it was captured from. It does not
certify a later deploy or every future call, and it does not re-run itself;
recapture again after the next change."* See
[`docs/RECAPTURE.md`](RECAPTURE.md#claim-language-what-each-kind-of-evidence-lets-you-honestly-say)
for the fuller claim-language table this line is drawn from.

## What this does not stop

This is an offline tool: a user who controls every input can lie to
themselves. Recomputing identity from the audio (above) makes the specific
forgeries an external red-team demonstrated against a prior build -- hand-
written envelopes, a flipped header byte, a re-scored recording, a
cherry-picked after set -- impossible or loud. None of the following is a
bug the guard failed to catch; each is outside what an offline recompute
over supplied files can ever establish:

* **Fabricated inputs are still yours to fabricate.** Hand fix trial a
  fresh recording of a call that never happened, or one that does
  not match the bug you are claiming to fix, and the guard verifies
  the audio identity and still reaches `improved`. It checks that
  the bytes it scored are what they claim to be, never that the stimulus
  itself is real.
* **A contract's `MANIFEST.sha256.json` is integrity, not authenticity.** It
  proves the archive still agrees with itself after packing; it does not
  prove who approved the policy inside it. Loosen a `.hotato` bundle's
  policy (raise `max_talk_over_sec`, say) before `contract pack`, and
  `contract verify` on the repacked bundle still passes -- it is re-checking
  the archive against itself, not against an external record of what the
  policy was supposed to be. Only a trusted signature over the manifest
  closes this; none is implemented today.
* **A resample, re-encode, or gain change of the SAME call still changes the
  decoded PCM.** The freshness check above is exactly "is the decoded PCM
  different." A deliberately transcoded copy of the identical recording
  (resampled, gain-adjusted, round-tripped through a lossy codec) decodes to
  different samples, so it reads as a distinct capture -- it is not; it is
  the same call in a different container. This is a known, undetected
  residual of a PCM-identity check, not a claim the guard makes and breaks.
* **Signatures are not implemented.** Nothing here is cryptographically
  signed; a sha256 digest is a checksum, not an attestation of who produced
  it.

A green fix trial does not prove the audio was freshly captured for the
scenario claimed, that the same policy and labels were used throughout, that
any omitted fixture was safe to omit, that the named revision or clone
existed, that the patch was applied to it, or that the deployed agent
improved. The verdict states exactly what it recomputed and verified from
the files it was given; everything else is a boundary of the offline model,
not a promise this tool makes and quietly fails to keep.

## Flags

- **`PATCH_JSON`** -- a `hotato patch` artifact.
- **`--name NAME`** -- name of the staging clone this trial is proving
  (required for a non-refused patch; the same `--name` `hotato apply`
  takes -- fix trial evaluates the SAME clone-only gate, it never creates a
  clone itself).
- **`--before RUN.json|DIR`** -- the OLD run envelope(s): the original
  failure evidence. Also the default opposite-risk `--battery`, and the
  attribution source, when those are omitted.
- **`--after RUN.json|DIR`** -- the NEW run envelope(s), re-captured
  through the staging clone.
- **`--battery DIR`** -- the opposite-risk battery apply's gate checks
  (BOTH a yield and a hold fixture); defaults to `--before`.
- **`--contracts DIR`** -- also re-verify a directory of hotato contracts;
  any contract regression fails the trial.
- **`--policy hotato.verify.yaml`** -- gate verify's rollup (see
  `docs/FIX-LOOP.md`); a violation fails the trial.
- **`--min-n N`** -- minimum previously-failing fixtures needed to support
  the claim (default 3).
- **`--out PATH`** -- also write the full proof JSON.
- **`--html PATH`** -- also write a self-contained before/after HTML
  report.

## Output

Every surface renders the apply receipt right beside the verdict, not buried
in the raw `apply` sub-object: fix trial calls `apply.build_apply` and never
`apply.create_clone`, so `apply_dry_run` is `True` and `apply_created` /
`apply_applies_change` are `False` on EVERY run, including an `improved` one.
A green verdict never means "and the change was applied" -- the report says
so in the same breath as the verdict, in text (`apply: dry_run=True
created=False applies_change=False`, plus the plain-English sentence right
under it), JSON (top-level `apply_dry_run` / `apply_created` /
`apply_applies_change` / `apply_receipt_note`, alongside `verdict`), and HTML
(pills and a header line, in the `<header>` block itself). Proving the change
reached the clone or agent is `hotato apply --clone --yes`'s job, recorded in
its own receipt -- fix trial only proves what the before/after evidence shows
once that has already happened.

* **text** (default): the apply receipt, the verdict, verify's own rendered
  proof, the contract verify rollup when `--contracts` was given, and the
  attribution section (one `hotato explain` render per file under
  `--before`). When the trial's own verdict is not `improved` (a provenance,
  completeness, contract, or policy issue downgraded it), verify's nested
  `CLAIM` line is tagged `CLAIM (SUPERSEDED BY {VERDICT})` with a one-line
  restatement, whenever that sub-claim would otherwise read "supported" --
  because a fix-trial verdict of `regressed` / `refused` / `inconclusive`
  can still contain a verify claim that, read alone, looks like a pass; the
  parent verdict controls, and the render says so rather than leaving a
  clean-looking claim floating under a red result.
* **`--format json`**: the full machine shape, schema `hotato.fix_trial.v1`.
  Every sub-result is the nested result the underlying command already
  produces (`apply`, `verify`, `contract_verify`, `attribution`), not a
  re-derived summary. `apply_dry_run` / `apply_created` /
  `apply_applies_change` / `apply_receipt_note` sit at the top level next to
  `verdict`, not only inside the nested `apply` object.
* **`--html PATH`**: a self-contained report, reusing the same house style
  the other HTML reports use: the apply-receipt pills and note in the
  header, the verdict chip, the verify proof table (with the same
  superseded-claim label as text, when it applies), the contract-verify
  rollup, and the attribution cards.

## Exit codes

- **0** -- `improved`.
- **1** -- fail-closed: `regressed` or `inconclusive`.
- **2** -- usage error: the same gates `hotato apply` enforces (no
  `--name`, no opposite-risk battery, a stack with no clone target, a
  patch with no concrete change) or `hotato verify` / `contract verify`
  already enforce (no fixtures pair, an invalid `--policy`, a `--contracts`
  dir with no contracts, unreadable input).
- **3** -- principled refusal: the both-axes threshold funnel. Shared with
  `hotato apply`'s refusal code.

## What this is not

fix trial does not create, patch, or delete anything on your platform; it
never calls `apply.create_clone`. It does not prove authorization, identity,
compliance, or policy safety. It reports coincidence, never causation --
every underlying `verify` and `contract verify` number keeps that rule. See
[`FIX-LOOP.md`](FIX-LOOP.md) for the manual steps this composes, and
[`APPLY.md`](APPLY.md) / [`EXPLAIN.md`](EXPLAIN.md) for the primitives
themselves.


================================================================================
FILE: docs/ANALYZE.md
================================================================================

# `hotato analyze <folder>`: drop a folder, hear the bug

Point Hotato at a folder of dual-channel call recordings and it finds the
worst turn-taking moments across all of them, ranks them, and lets you hear
each one.

```bash
hotato analyze ./recordings
```

This writes one self-contained, offline HTML dashboard (`hotato-analyze.html`
by default) and opens it. A bare folder as the first argument routes to the
same command: `hotato ./recordings`.

## What it does

For every `.wav` under the folder (walked recursively, in sorted order),
`analyze`:

1. runs the same whole-call scanner as [`hotato scan`](../src/hotato/scan.py):
   it walks the caller and agent VAD activity tracks across the entire call,
   label-free, and emits candidate timing moments
   (`overlap_while_agent_talking`, `agent_start_during_caller`,
   `long_response_gap`, `agent_stop_no_caller`, `echo_correlated_activity`),
   each with a timestamp and a measured number;
2. aggregates the candidates across every call and ranks them by the
   scanner's own salience (overlap seconds / gap seconds / echo coherence),
   so the worst moments float to the top.

Then it emits three things.

### 1. A ranked dashboard

One card per top moment, in the [`hotato report`](REPORTS.md) house style: the
call file, the timestamp, the candidate kind, the measured number, and a
to-scale caller/agent timeline of that exact moment (the same SVG renderer the
report uses: activity spans, the shaded talk-over band, the onset marker, and
a yield marker where the scanner measured the agent going silent).

### 2. The hear-the-bug player

For the top `--audio-top` moments (default 8), the audio around the moment is
embedded inline as a base64 WAV data URI, so the page is fully self-contained
with zero external requests. Press play and a **playhead** sweeps that
moment's timeline in lockstep with `audio.currentTime` (via
`requestAnimationFrame`): you hear the agent talk over the caller, or the
dead-air gap, land exactly where the chart marks it. The playhead tracks
playback under `prefers-reduced-motion: reduce` too, riding `timeupdate`
instead of the smooth animation loop.

Only the top moments carry audio; the rest show the timeline. That, plus
`--pre` / `--post` (seconds of audio kept before/after each moment) and a
total-page audio budget, keeps the page a manageable size.

### 3. JSON for agents

```bash
hotato analyze ./recordings --format json
```

Prints the ranked candidates plus their metadata (source file, timestamp,
kind, salience, measured durations, and the audio/timeline window) to stdout,
capped by `--top`. Pass `--out FILE` to also write the full ranked JSON.

## Framing

Each result is a **measured candidate timing moment**: a timestamp and a
number, not a verdict on intent. Energy sounds the same whether a caller said
"mhm" or "stop", so you decide the expected behavior and label the moments
that matter:

```bash
hotato fixture create --stereo <call>.wav --onset <t> \
    --expect yield|hold --id found-moment-001 --out tests/hotato
```

Non-dual-channel or otherwise unreadable files are reported cleanly in a
"Skipped files" section with their reason (a mono mix cannot attribute
talk-over), and the run keeps going through the rest of the folder.

## Flags

| flag | default | meaning |
| --- | --- | --- |
| `FOLDER` | (required) | directory of dual-channel WAVs, walked recursively |
| `--top` | 25 | ranked moments shown in the dashboard / stdout JSON (0 = all) |
| `--audio-top` | 8 | top moments that get the embedded hear-the-bug player |
| `--pre` | 2.0 | seconds of audio/timeline kept before each moment |
| `--post` | 4.0 | seconds of audio/timeline kept after each moment |
| `--min-gap` | 2.0 | minimum response gap (seconds) to surface as a candidate |
| `--caller-channel` / `--agent-channel` | 0 / 1 | channel assignment |
| `--format` | `html` | `html` (dashboard) or `json` (ranked candidates) |
| `--out` | `hotato-analyze.html` | where to write the dashboard, or the JSON |
| `--no-open` | off | do not launch a browser for the dashboard |

## Exit codes

- **0**: ran -- candidate moments listed across the folder, possibly zero.
- **2**: usage error (the path is not a folder) or an IO error reading it.

Everything, audio included, stays on your machine.


================================================================================
FILE: docs/TRUST.md
================================================================================

# `hotato trust --stereo call.wav`: is this recording even scorable?

The input-health check, "trust doctor": inspect ONE recording and learn
whether the audio is good enough to score, before you scan or run it. `trust`
catches a bad export -- a mono file, a silent channel, a swapped channel map,
a hot capture -- up front, before it becomes a confident-looking but
meaningless turn-taking verdict downstream.

```bash
hotato trust --stereo your-call.wav
```

```text
hotato trust: your-call.wav
  recording: 32.0s, 44100 Hz, 2 channels
  caller (ch0): 3.32s speech, first at 3.27s, peak -2.1 dBFS
  agent  (ch1): 23.30s speech, first at 0.45s, peak -0.9 dBFS
  leading silence: 0.45s
  crosstalk: coherence 0.057 (low) at 0.39s lag
  scorability: separated tracks yes, caller activity yes, agent activity yes
  => eligible for scan
```

Exit code `0` means eligible for scan; exit code `2` means NOT SCORABLE (or a
usage error / unreadable file), so `trust` composes straight into a shell gate:

```bash
hotato trust --stereo call.wav && hotato scan --stereo call.wav
```

## What it checks

By convention the caller is on channel 0 and the agent on channel 1 (override
with `--caller-channel` / `--agent-channel`). `trust` reports INPUT health only:

- **per-channel activity**: how much speech each channel carries and when each
  first speaks;
- **possible channel swap**: a heuristic flag: if the channel mapped as the
  caller holds the floor far longer than the channel mapped as the agent (the
  reverse of the usual pattern, where an assistant answers in paragraphs), the
  caller/agent channels may be reversed;
- **sample rate and duration**: the basic recording facts;
- **clipping**: per-channel peak level (dBFS) and the fraction of samples at
  full scale, so a too-hot capture is visible;
- **leading silence**: dead air before the first speech on either channel;
- **crosstalk risk**: cross-channel echo coherence: is the caller channel
  carrying a delayed copy of the agent's own audio (echo bleed / missing echo
  cancellation)?
- **cross-channel leakage** (`crosstalk_risk.leakage_db`): the level, in dB
  below the source, of a consistent attenuated delayed COPY of one channel
  found on the other. The whole-clip coherence above is a single best-lag
  cosine over the entire envelope, diluted by unrelated activity elsewhere in
  the call, so bleed loud enough to corrupt a downstream timing verdict can
  sit under the coherence bar and go unflagged. Leakage is measured
  differently -- from the per-frame level ratio of the copy, constant across
  every frame the source speaks -- so it catches that regime. At or above
  `-40 dB` (calibrated to the level at which symmetric bleed was red-teamed
  into flipping a verdict) the copy is loud enough to be counted as the other
  party's activity, so it is flagged and the recommendation is downgraded off
  `eligible for scan`;
- **low signal level**: when even the loudest channel peaks below `-30 dBFS`, the
  capture is quiet enough that turn timing can be under-measured downstream; a
  warning, never a not-scorable condition;
- **scorability**: the three things a score needs: separated tracks, enough
  caller activity, and enough agent activity;
- **recommendation**: `eligible for scan`; `scan with caution` (scorable, but a loud
  cross-channel leak may corrupt the timing a scan produces); or `NOT SCORABLE`
  with the specific reason AND the next step to fix it.

## Scope: one question, and it stops there

`trust` answers exactly one question -- is this audio good enough to score? --
and stops: no `yield` / `hold`, no `pass` / `fail`, no `did_yield`, no
talk-over number, no intent label, no turn-taking verdict. Eligible for scan
means the input is clean, not that the agent behaved -- finding agent bugs is
what [`hotato scan`](../src/hotato/scan.py) and `hotato run` are for.

## Not scorable

Three input defects make a recording unscorable. Each is reported with
`scorable: false`, a plain reason, and the next step, and exits `2`:

- **mono**: a single channel cannot separate the caller from the agent on its
  own. Next step: export a dual-channel recording (the gold reference), OR score
  it via the opt-in diarization front-end (see "Mono via `--diarize`" below).
- **identical channels**: a mono recording duplicated into two channels: two
  channels, but not separated. Same fix as mono.
- **a silent required channel**: for example `caller channel has no detected
  speech`. Next step: verify channel mapping or export dual-channel again.

Clipping, high leading silence, crosstalk risk, cross-channel leakage, a very
low signal level, and a possible channel swap are **warnings**: surfaced as
signal, without changing scorability by themselves. A loud cross-channel leak
additionally downgrades the recommendation to `scan with caution` -- the
tracks stay separated and scorable, so the scorability gate and exit code
hold steady, and only the human-facing recommendation records that a scan's
timing may be wrong. `trust` adds signal and discloses limits; the
not-scorable boundary stays fixed.

## Mono via `--diarize`: is this mono file confidently separable?

A mono file is not scorable by default (above). The opt-in `[diarize]`
front-end changes the question: `hotato trust --stereo call.wav --diarize`
reports whether the mono is confidently SEPARABLE into caller/agent, still
without emitting a turn-taking verdict. It runs the selected diarizer
(`--diarizer pyannote|sortformer|pyannoteai`, default local `pyannote`), then
reports a `scorability.separation` sub-block and a confidence **tier**:

- **high**: confidently separable -- score it with `hotato run --mono call.wav
  --diarize` for a diarized-mono verdict. Exit `0`.
- **low**: separable but only indicative -- e.g. voices close, overlap elevated,
  or the caller/agent mapping balanced. `hotato run --mono ... --diarize` will
  still score it, but the verdict is stamped `indicative_only` and no SLA gate
  fires. Exit `0`.
- **refuse**: not confidently separable (not two clean parties, a near-silent
  speaker, extreme overlap, voices too similar). Not scorable, exit `2`; next
  step: record a dual-channel call.

The six signals behind the tier (speaker count, per-speaker activity, mean
segmentation posterior, embedding cluster-separation margin, overlap ratio,
segment churn) are in the `separation.signals` block. A missing extra / token /
model raises a clean error (exit 2) instead of a raw-mono guess. The
dual-channel path stays the gold reference; a diarized-mono verdict stays
distinct from it. See `docs/DIARIZE.md` for the full front-end.

## JSON for agents

`--format json` emits one machine-parseable report. Branch on `scorable` (and, on
a defect, read `not_scorable_reason` and `next_step`); `exit_code` mirrors the
process exit.

```bash
hotato trust --stereo call.wav --format json
```

```json
{
  "tool": "hotato",
  "kind": "input-health",
  "schema_version": "1",
  "source": "call.wav",
  "recording": {
    "sample_rate": 44100,
    "duration_sec": 32.0,
    "channels": 2,
    "clipping": {
      "caller": {
        "peak": 0.7906, "peak_dbfs": -2.1,
        "clipped_fraction": 0.0, "clipped": false
      },
      "agent": {
        "peak": 0.9016, "peak_dbfs": -0.9,
        "clipped_fraction": 0.0, "clipped": false
      }
    },
    "leading_silence_sec": 0.45
  },
  "channels": {
    "caller": {
      "channel": 0, "active_sec": 3.32, "first_speech_sec": 3.27,
      "has_speech": true, "enough_activity": true
    },
    "agent": {
      "channel": 1, "active_sec": 23.3, "first_speech_sec": 0.45,
      "has_speech": true, "enough_activity": true
    },
    "possible_swap": false,
    "swap_reason": null
  },
  "crosstalk_risk": {
    "coherence": 0.057, "lag_sec": 0.39, "suspected": false,
    "leakage_db": null, "leakage_direction": null
  },
  "scorability": {
    "separated_tracks": true, "enough_caller_activity": true,
    "enough_agent_activity": true
  },
  "warnings": [],
  "scorable": true,
  "recommendation": "eligible for scan",
  "not_scorable_reason": null,
  "next_step": null,
  "exit_code": 0
}
```

Everything runs offline and reuses hotato's existing primitives -- the
hardened WAV reader, the reference framing, the energy VAD, and the
cross-channel echo coherence -- all on your machine, reporting input-health
signals only, never an accuracy percentage.


================================================================================
FILE: docs/CONNECT.md
================================================================================

# Connect, pull, sweep: score every call across your stack

`hotato sweep` is the "connect once, see every turn-taking problem across
all your calls" flow. It runs in three steps you can also run on their
own:

1. **`hotato connect <stack>`**: store a stack's credentials once, locally.
2. **`hotato pull`**: bulk-fetch your recent recordings into a folder.
3. **`hotato sweep`**: pull, then run the zero-config `analyze` over them
   and write one offline dashboard of the ranked turn-taking moments.

Everything scores offline. The only network call is the direct recording
download from your vendor to your machine; your audio and your keys stay
between you and the vendor.

## 1. Connect once

```
hotato connect vapi --api-key <key>
```

This runs a lightweight live auth check (it lists one recent call), then
writes the credentials to `~/.hotato/connections.json` with file mode
`0600` (directory `0700`). The key stays out of any printed output and
travels only to the vendor's own API. Credentials also fall back to the
stack's environment variable, so this works too:

```
VAPI_API_KEY=<key> hotato connect vapi
```

Per-stack credentials:

| Stack | Flags (or env vars) |
| --- | --- |
| vapi | `--api-key` / `VAPI_API_KEY` |
| retell | `--api-key` / `RETELL_API_KEY` |
| twilio | `--account-sid` `--auth-token` / `TWILIO_ACCOUNT_SID` `TWILIO_AUTH_TOKEN` |
| bland | `--api-key` / `BLAND_API_KEY` |
| elevenlabs | `--api-key` / `ELEVENLABS_API_KEY` |
| synthflow | `--api-key` / `SYNTHFLOW_API_KEY` (+ `--model-id` / `SYNTHFLOW_MODEL_ID` to list) |
| millis | `--api-key` / `MILLIS_API_KEY` (+ `--base-url` for the EU region) |
| cartesia | `--api-key` / `CARTESIA_API_KEY` (+ `--agent-id` / `CARTESIA_AGENT_ID` to list) |

Retell has no list endpoint to verify against, so `connect retell` stores
the key and validates it on the first pull. Use `--no-verify` to skip the
live check for any stack. LiveKit and Pipecat capture in your own infra,
so connect them through `hotato setup --stack livekit|pipecat` instead of
a vendor pull.

After connecting, `--stack` and the credential flags are optional for
`pull` and `sweep`: when exactly one stack is connected, Hotato uses it.

## 2. Pull recent recordings

```
hotato pull --stack vapi --since 7d --limit 50
hotato pull                                   # the only connected stack
```

`pull` lists your recent recordings with the vendor's verified list
endpoint and downloads each one by looping the same single-call fetch
`hotato capture` uses, into `hotato-pull-<stack>/` (override with
`--out DIR`). `--since` accepts `7d`, `12h`, `30m`, `2w`. A recording that
cannot be fetched (missing URL, HTTP error, wrong channel count) is
reported as a clean skip with its reason, and the pull keeps going through
the rest.

**Retell has no verified list endpoint.** Pull it from explicit ids
instead (Hotato ships only verified endpoints):

```
hotato pull --stack retell --call-id c1 --call-id c2
```

For Twilio, `--call-id` values are Recording SIDs (`RE...`).

**Mono / mixed stacks** (bland, elevenlabs, synthflow, millis, cartesia)
produce a single combined recording with no per-party channel;
attributing talk-over needs the `--allow-mono` opt-in, and results stay
indicative only:

```
hotato pull --stack bland --allow-mono --limit 20
```

## 3. Sweep = pull + analyze

```
hotato sweep --stack vapi --since 7d
hotato sweep                                  # the only connected stack
```

`sweep` pulls (as above) into `hotato-sweep-<stack>/` (override with
`--dir`), then runs the exact same zero-config `analyze` over the folder
and writes one self-contained, offline HTML dashboard
(`hotato-sweep-<stack>.html`, override with `--out`) of the ranked
candidate turn-taking moments across every call, with the hear-the-bug
audio player on the top moments. `--format json` emits the ranked
candidates plus a pull summary instead.

Dual-channel stacks get separated scoring. Mono/mixed stacks sweep with
`--allow-mono`; without per-party separation, their calls surface in the
dashboard's Skipped section.

Candidates are MEASURED timing moments you review and label with
`hotato fixture create`: a timestamp and a number, not a verdict on
intent.

## What is and isn't pullable

See [`docs/ADAPTER-STATUS.md`](ADAPTER-STATUS.md) for the full map: which
stacks auto-pull dual-channel, which are mono-only behind `--allow-mono`,
which capture in your own infra, and which fall outside today's adapter
set, each against the exact verified endpoint it uses.


================================================================================
FILE: docs/ADAPTER-STATUS.md
================================================================================

# Adapter status

Which stack Hotato pulls recordings from, which endpoint it calls, and how it
gets separated caller/agent channels out of each one. Every entry is checked
verbatim against the vendor's live documentation on the date shown
(integration research: `hotato-launch/INTEGRATION-SPEC-2026-07-07.md`).

Terms:

- **auto-pull**: Hotato fetches the recording itself with your API key
  (`hotato capture` / `hotato pull` / `hotato sweep`).
- **capture-in-your-infra**: there is no vendor recording API; Hotato prints
  the recording config (`hotato setup`) and scores the file your deployment
  writes.
- **mono / mixed**: a single combined channel. Attributing overlap to the
  caller or the agent needs separated tracks, so Hotato scores a combined
  channel only behind an explicit `--allow-mono` / `HOTATO_ALLOW_MONO=1`
  opt-in and labels the result indicative only. Separated turn-taking
  analysis runs on a dual-channel file.

Each adapter ships with only the endpoints confirmed verbatim in the spec.
Where the spec marks a list-calls endpoint or a channel layout **unconfirmed
/ none**, Hotato falls back to explicit call ids and documents the gap here.

## Dual-channel: auto-pull, separated scoring

One entry per stack, verified 2026-07-07 unless noted otherwise. Each of
these auto-pulls a recording with per-party channels, so Hotato scores full
separated turn-taking without `--allow-mono`.

- **Vapi**
  - List recent calls: `GET https://api.vapi.ai/call` (params `limit`, `createdAtGt/Lt`) → JSON array of Call objects (`id`, `createdAt`)
  - Fetch recording: `GET /call/{id}` → `artifact.recording.stereoUrl` (current); deprecated fallbacks `artifact.stereoRecordingUrl`, `call.stereoRecordingUrl`
  - Channel basis: `stereoUrl` is a distinct 2-channel file (customer ch0, assistant ch1)
- **Twilio**
  - List recent calls: `GET .../Accounts/{Sid}/Recordings.json` (params `PageSize`, `DateCreatedAfter/Before`, `callSid`) → `recordings[].sid`
  - Fetch recording: `GET .../Recordings/{RE...}.wav?RequestedChannels=2` (HTTP Basic)
  - Channel basis: dual-channel only if the recording was created `RecordingChannels=dual`; 400 → clean stop or `--allow-mono` → `RequestedChannels=1`. Two-party order: ch0 = caller, ch1 = agent; conference: ch0 = first participant
- **Retell**
  - List recent calls: **none confirmed** -- the spec marks list-calls unconfirmed; do not fabricate one. Pull from explicit `--call-id`
  - Fetch recording: `GET https://api.retellai.com/v2/get-call/{id}` (Bearer) → `scrubbed_recording_multi_channel_url` preferred, then `recording_multi_channel_url`; plain mono `recording_url` rejected unless `--allow-mono`
  - Channel basis: per-party channels on the `*_multi_channel_url` fields
- **LiveKit**
  - List recent calls: **capture-in-your-infra** -- `ListEgress` is an RPC method name only; no REST list. `hotato setup --stack livekit`
  - Fetch recording: two audio-only Track egresses (one per party); recording location in `egress_info.file_results[].location`
  - Channel basis: separated by running one Track egress per participant
- **Pipecat**
  - List recent calls: **capture-in-your-infra** -- Pipecat Cloud session-list has no recording field; OSS has no list at all
  - Fetch recording: `AudioBufferProcessor(num_channels=2)` in-pipeline (user left, bot right); you write the WAV
  - Channel basis: 2-channel in-pipeline

## Mono / mixed: auto-pull, `--allow-mono`, indicative only

Each of these has a verified list + fetch, but the vendor produces a single
combined recording with **no documented per-party channel**. Hotato scores it
indicative-only, behind `--allow-mono`. One entry per stack, verified
2026-07-07.

- **Bland AI**
  - List recent calls: `GET https://api.bland.ai/v1/calls` → `calls[].call_id`
  - Fetch recording: `GET /v1/calls/{id}` → `recording_url`
  - Mono basis (verbatim): no stereo/channel field in the recording or call-details schema
- **ElevenLabs Conversational AI**
  - List recent calls: `GET https://api.elevenlabs.io/v1/convai/conversations` (params `page_size`, `call_start_after_unix`) → `conversations[].conversation_id`
  - Fetch recording: `GET /v1/convai/conversations/{id}/audio` → combined audio
  - Mono basis (verbatim): docs state the audio "does not include separate caller/agent channels, only a combined full conversation MP3"
- **Synthflow**
  - List recent calls: `GET https://api.synthflow.ai/v2/calls?model_id=…` (params `limit`, `from_date` epoch-ms) → `response.response.calls[].call_id`
  - Fetch recording: `GET /v2/calls/{id}` → `response.response.calls[0].recording_url` (a Twilio Recordings URL)
  - Mono basis (verbatim): Synthflow documents no dual-channel option; `recording_url` is a bare Twilio URL
- **Millis AI**
  - List recent calls: `GET {base}/call-logs` (US `api-west`, EU `api-eu-west`; param `limit`) → `histories[].session_id`
  - Fetch recording: `GET /call-logs/{session_id}` → `recording.recording_url`
  - Mono basis (verbatim): schema exposes only `recording_url`; `CallSettings` has only a boolean `enable_recording`
- **Cartesia (Line)**
  - List recent calls: `GET https://api.cartesia.ai/agents/calls?agent_id=…` (param `limit`; needs `Cartesia-Version` header) → `data[].id`
  - Fetch recording: `GET /agents/calls/{id}/audio` → `audio/wav`
  - Mono basis (verbatim): dual-vs-mono **unconfirmed** in the docs; treated as mono until a live channel-count check proves otherwise

`--stack synthflow` needs `--model-id` (its list endpoint requires `model_id`);
`--stack cartesia` needs `--agent-id`; `--stack millis` accepts `--base-url` for
the EU region. Single-call `capture`/`pull` by explicit id needs none of these.

## No vendor recording to pull

- **Deepgram Voice Agent API** (2026-07-07, confirmed absent) -- real-time
  WebSocket (`wss://agent.deepgram.com/v1/agent/converse`), with no REST
  list-calls, fetch-recording endpoint, or recording-ready webhook: the call
  itself is never stored vendor-side. Capture the recording in your own
  infra instead (the LiveKit/Pipecat pattern above).
- **PlayAI (formerly Play.ht)** (2026-07-07, dead endpoints verified) -- the
  conversational-agent product is retired: `play.ai` DNS does not resolve and
  `docs.play.ai` returns `DEPLOYMENT_NOT_FOUND`. The only live domain,
  `docs.play.ht`, is TTS-only with no calls/recordings API.

## Needs credentials or a live probe before shipping

Documented weakly or behind a login/SPA; the spec could not verify a recording
field-path or a channel layout for these, so they ship as fallbacks (explicit
call ids or the vendor's own webhook URL) rather than full adapters. Listed
here so the supported set stays exact.

- **Regal.ai**: no list-calls and no REST fetch-recording endpoint. The
  recording arrives only via the `call.recording.available` webhook's
  `properties.recording_link`, which is a literal Twilio Recordings URL. Feed
  that URL through Hotato's existing Twilio path -- the one shipped route for
  Regal-sourced recordings.
- **Thoughtly**: OpenAPI types responses as an untyped `GenericResponse.data`
  blob; no recording-URL field path could be confirmed live.
- **Sindarin**: no Sindarin-hosted recording endpoint found; `record` is
  submitted with the caller's own Twilio creds, implying the recording lands in
  the customer's own Twilio account (use the Twilio path).
- **xAI Grok Voice Agent Builder**: OpenAI-Realtime-compatible; no documented
  list-calls or fetch-recording endpoint (marketing mentions in-product
  recording, but no API contract was confirmable).
- **Cartesia dual-channel**: the `/agents/calls/{id}/audio` WAV's channel count
  was not confirmable in docs; treated as mono here pending a live check.
- **Twilio dual-channel edge behaviour**: the exact 400-on-unavailable and
  conference channel-ordering guarantees were carried forward from prior notes,
  not re-verified verbatim this pass.
- **Synthflow / Regal Twilio-URL dual-channel trick**: because their
  `recording_url` is a Twilio URL, `?RequestedChannels=2` *might* yield stereo,
  but neither vendor documents or guarantees it; unconfirmed until tested against
  a live account.

Also researched, login-gated or SPA-only docs or transcript-only APIs: Daily,
Ultravox, Hume EVI, Telnyx, Infobip, OpenAI Realtime, Amazon Connect, Parloa,
Sierra, Decagon, PolyAI, Voiceflow, Cognigy, Dialogflow CX, Genesys Cloud,
NICE CXone. The integration spec carries the per-platform verified facts and
gaps.

## What holds across every adapter

Every adapter validates that a dual-channel file it scores has one party per
channel (2 channels) before producing separated talk-over numbers; mono stacks
score degraded, only behind `--allow-mono`. Every live fetch/list path is
stdlib-only (`urllib`); stack SDKs import lazily, and only inside your own
infra. Credentials captured by `hotato connect` sit locally at
`~/.hotato/connections.json` (mode 0600) and go straight from your machine to
the vendor's own API, with Hotato out of that exchange. Scoring runs offline;
the only network call is the direct recording download.


================================================================================
FILE: docs/STARTER.md
================================================================================

# The starter kit: `hotato init starter`

The fastest way to add hotato to an existing voice-agent repo: one command
scaffolds the CI gate, a stack-tuned config file, and the three directories
the rest of the docs assume already exist. Skip the plumbing and go straight
to turning your first bad call into a contract.

```bash
hotato init starter --stack vapi --out .
```

`--stack` is one of `vapi`, `retell`, `twilio`, `livekit`, `pipecat` -- every
stack hotato has a shipped connector for (see
[`ADAPTER-STATUS.md`](ADAPTER-STATUS.md)). `--out` is usually `.`, the root of
the repo you are adding hotato to. Generation runs offline, in one command,
nothing to connect.

## What it writes

```
HOTATO.md                                # what was added, next steps (read this first)
hotato.yaml                              # config skeleton, tuned for --stack
.gitignore                               # excludes local/pulled recordings;
                                          #   keeps pinned fixture/contract clips committed
.github/workflows/hotato-contracts.yml   # the CI gate
fixtures/
  README.md
  scenarios/.gitkeep                     # -> hotato fixture create --out fixtures
  audio/.gitkeep
contracts/
  README.md
  .gitkeep                               # -> hotato contract create --out contracts
reports/
  README.md
  .gitkeep                               # local/CI scratch: doctor/report/sweep output
```

Every file writes cleanly only when it doesn't already exist (pass `--force`
to overwrite); each write lands whole or not at all. The generated names are
namespaced away from a repo's own files (`HOTATO.md`, distinct from
`README.md`; `hotato-contracts.yml`, distinct from `hotato.yml`), so a first
run drops in cleanly alongside files a voice-agent repo almost always
already has.

## Two input paths, chosen by `--stack`

**Auto-pull** (`vapi`, `retell`, `twilio`): hotato fetches the recording
itself once you connect a key. `hotato.yaml`'s `credentials.env` names the
exact environment variable(s) (`VAPI_API_KEY`; `RETELL_API_KEY`;
`TWILIO_ACCOUNT_SID` + `TWILIO_AUTH_TOKEN`) `hotato connect <stack>` also
reads. `recording.access` is `auto-pull`.

**Capture-in-your-infra** (`livekit`, `pipecat`): capture happens inside your
own deployment, so no credentials are needed. `hotato.yaml`'s
`credentials.env` is `[]` and `recording.access` is `capture-in-your-infra`;
`hotato setup --stack <stack>` prints the exact two-track capture scaffold,
and you point `hotato contract create --stereo` at the WAV your own
deployment writes.

## LiveKit and Pipecat runbook

LiveKit and Pipecat are the two stacks where capture and the turn-taking
config both live in your own code, ahead of any vendor API. This is the
operator runbook for both, capture through CI.

### LiveKit

1. **Capture.** Two audio-only Track egresses, one per participant --
   RoomComposite mixes both parties into one channel and cannot attribute
   overlap. `hotato setup --stack livekit` prints the copy-paste scaffold
   (Python `livekit-api`, `TrackEgressRequest` + `DirectFileOutput`); a
   ready-to-copy version also lives at `adapters/livekit_capture.py`.
2. **Find the turn-taking config.** It lives on
   `AgentSession(turn_handling=TurnHandlingOptions(...))`: `turn_detection`
   (`inference.TurnDetector()` / `"realtime_llm"` / `"vad"` / `"stt"` /
   `"manual"`), `endpointing` (`min_delay`, `max_delay`), and `interruption`
   (`enabled`, `mode`, `min_duration`, `min_words`,
   `false_interruption_timeout`, `resume_false_interruption`). Read what a
   given agent file is running, statically, before proposing any change:
   `hotato inspect --stack livekit --config agent.py`.
3. **Score it.**
   `hotato capture --stack livekit --caller caller.wav --agent agent.wav --onset <sec> --expect yield`
   (convert the egress output to WAV first, e.g. `ffmpeg -i caller.ogg caller.wav`).
4. **Fixture, contract, CI.** Same as every stack from here -- see "Turn
   your first bad call into a contract" below.

### Pipecat

1. **Capture.** A 2-channel `AudioBufferProcessor` in-pipeline (channel 0 =
   user/caller, channel 1 = bot/agent) -- do not mix down to one channel.
   `hotato setup --stack pipecat` prints the copy-paste scaffold; a
   ready-to-copy version also lives at `adapters/pipecat_capture.py`.
2. **Find the turn-taking config.** It lives on `PipelineTask`'s user-turn
   strategies: start strategies (`VADUserTurnStartStrategy`,
   `TranscriptionUserTurnStartStrategy`,
   `MinWordsUserTurnStartStrategy(min_words=...)`,
   `KrispVivaIPUserTurnStartStrategy(...)`) and stop strategies
   (`SpeechTimeoutUserTurnStopStrategy(user_speech_timeout=...)`,
   `TurnAnalyzerUserTurnStopStrategy(turn_analyzer=...)`); note
   `MinWordsInterruptionStrategy` is deprecated since pipecat 0.0.99 in
   favor of `MinWordsUserTurnStartStrategy`. Read what a given bot file is
   running, statically: `hotato inspect --stack pipecat --config bot.py`.
3. **Score it.**
   `hotato capture --stack pipecat --stereo captured.wav --expect yield`
   (write the WAV from the `AudioBufferProcessor`'s `on_audio_data` handler
   first).
4. **Fixture, contract, CI.** Same as every stack from here -- see "Turn
   your first bad call into a contract" below.

Both APIs move; `hotato setup` and `hotato inspect` state the verified-against
date. Full field-level detail and provenance: [`ADAPTER-STATUS.md`](ADAPTER-STATUS.md)
(capture) and [`FIX-PLANS.md`](FIX-PLANS.md) (inspect, Level 1 of the fix ladder).

## The CI gate

`.github/workflows/hotato-contracts.yml` runs on push, on pull request, and
weekly. It is two guarded steps that pass clean, as a no-op, until you have
added a first contract or fixture (a fresh scaffold's normal starting state):

```bash
hotato contract verify contracts --junit hotato.xml --format json > contracts-verify.json
hotato run --scenarios fixtures/scenarios --audio fixtures/audio --format json > fixtures-run.json
```

The JUnit file publishes as a build artifact on every run (`always()`),
whether the gate passed, failed, or had nothing to check yet.

For the three auto-pull stacks, the workflow also carries a `weekly-sweep`
job: a passive, candidate-only sweep of recent calls
(`hotato sweep --stack <stack>`), ranked by hotato's own salience -- a
candidate list for you to review and label. It ships disabled (`if: false`):
flip it to `true` once the stack's credential env var(s) are set as repo
secrets (Settings -> Secrets and variables -> Actions). A live pull against
your account runs only on your explicit say-so -- enabling this job is a
human decision, made once, in your own CI config. `livekit`/`pipecat` skip
this job entirely, since capture for those stacks already happens inside
your own deployment.

## Turn your first bad call into a contract

```bash
# auto-pull stacks
hotato connect vapi --api-key <key>
hotato sweep --stack vapi --out hotato-sweep.html
# open hotato-sweep.html, pick a candidate moment, then:
hotato contract create --from-candidate hotato-sweep.json#1 \
    --expect yield --id refund-cutoff-001 --out contracts

# capture-in-your-infra stacks
hotato setup --stack livekit
# once your deployment writes a two-channel WAV:
hotato contract create --stereo call.wav --onset 42.18 \
    --expect yield --id refund-cutoff-001 --out contracts
```

Commit the resulting `contracts/refund-cutoff-001.hotato/` directory. The
next push runs it through the CI gate above.

**A contract bundle contains call audio** (`audio/event.wav`). If this repo
is or could become public, commit a sanitized fixture (synthetic or
consent-cleared), and keep customer contracts in a private repository
or controlled artifact storage. See [`CONTRACTS.md`](CONTRACTS.md).

## Read more

- The bundle layout and the create/verify/inspect/pack/unpack commands:
  [`CONTRACTS.md`](CONTRACTS.md)
- The underlying fixture primitive, one bad call to a CI gate in five steps:
  [`BAD-CALL-TO-CI.md`](BAD-CALL-TO-CI.md)
- Per-stack connector support, verified against the vendor's live docs:
  [`ADAPTER-STATUS.md`](ADAPTER-STATUS.md)
- The connect-once bulk pull-and-analyze recipe: [`CONNECT.md`](CONNECT.md)
- An agent adding hotato to a repo end to end: [`../AGENTS.md`](../AGENTS.md)


================================================================================
FILE: docs/TRACE.md
================================================================================

# Voice traces

A voice trace is a timeline of discrete voice-pipeline events
(caller/agent audio activity, TTS cancel/stop, ASR partials, tool calls,
...) that adds a WHY layer beyond what a caller/agent audio track alone
shows: "the agent talked over the caller" becomes "evidence suggests TTS
cancellation lagged: cancel requested at 42.40s, audio stopped at 43.60s".

A voice trace reports coincidence: a pattern of events that lined up with
the measured timing, not root cause. See [`docs/OTEL.md`](OTEL.md) for the
ingest source formats.

Three commands:

```bash
hotato trace ingest --otel traces.jsonl --out voice_trace.jsonl
hotato trace attach contracts/refund-cutoff-001.hotato --trace voice_trace.jsonl
hotato trace export contracts/refund-cutoff-001.hotato --format otel --out otel.jsonl
```

## Schema

`voice_trace.jsonl` is JSONL (one meta line, then one line per span -- the
same convention `evidence/frames.jsonl` uses), validating against
`hotato.voice_trace.v1` (`src/hotato/schema/voice_trace.v1.json`) once
reassembled by `hotato.trace.load_voice_trace_jsonl`:

```json
{
  "schema": "hotato.voice_trace.v1",
  "call_id": null,
  "deployment": {
    "stack": "vapi",
    "agent_id": null,
    "git_sha": "deadbeef",
    "config_hash": "sha256:..."
  },
  "spans": [
    {"type": "agent_audio_active", "start_sec": 0.0, "end_sec": 2.9},
    {"type": "tts_cancel_requested", "time_sec": 2.6},
    {"type": "tts_audio_stopped", "time_sec": 2.9},
    {
      "type": "asr_partial",
      "start_sec": 2.4,
      "end_sec": 2.95,
      "text_redacted": true
    },
    {
      "type": "tool_call",
      "start_sec": 1.1,
      "end_sec": 1.42,
      "name": "lookup_order",
      "latency_ms": 320
    }
  ],
  "source": {"format": "otel-jsonl-bridge", "input_span_count": 6}
}
```

A span carries an open `type` string; the common ones are
`caller_audio_active`, `agent_audio_active`, `tts_cancel_requested`,
`tts_audio_stopped`, `asr_partial`, `tool_call`, `llm_first_token`,
`handoff`. An unrecognized type passes through unchanged (additive,
forward-compatible with a pipeline that emits a span type this release does
not name). Exactly one time shape applies per span: an interval span
carries `start_sec`/`end_sec`, a point event carries `time_sec`.

## Redaction by default

`call_id` and `deployment.agent_id` are dropped (`null`) unless
`--include-identifiers` is passed at ingest time. An `asr_partial` span's
transcript text is dropped (`text_redacted: true`, no `text` key) unless
`--include-text` is passed. `deployment.stack` / `git_sha` / `config_hash`
are not treated as identifiers and are kept by default.

## Ingest

```bash
hotato trace ingest --otel traces.jsonl --out voice_trace.jsonl
hotato trace ingest --otel export.json --out voice_trace.jsonl \
    --stack vapi --include-identifiers --include-text
```

`--otel FILE` accepts either a standard OTel JSON export (a document with a
top-level `resourceSpans` array) or hotato's own documented OTel bridge
JSONL (see [`docs/OTEL.md`](OTEL.md)). `--stack` / `--call-id` /
`--agent-id` / `--git-sha` / `--config-hash` override or fill in whatever
the source's own resource attributes carried. Refused (exit 2, nothing
written) for an unreadable file, an empty file, or a source with zero
spans.

## Attach

```bash
hotato trace attach contracts/refund-cutoff-001.hotato --trace voice_trace.jsonl
```

Copies the trace into `<bundle>/traces/voice_trace.jsonl` and re-renders
`evidence/timeline.html` with the trace's events drawn as an additional
row, aligned to the same [0, duration] scale as the existing caller/agent
timeline. This reads the bundle's own `evidence/frames.jsonl` and
`contract.json` back in, reusing the existing scored evidence as-is:
attaching a trace needs no diarization extra installed and no re-scoring
of the audio. On a diarized-mono bundle (no frame-level evidence), the
base timeline states that plainly, and the trace row still renders on its
own scale.

`contract.json` records the attachment (additive, schema-safe: `trace:
{attached, path, span_count, attached_at, source_format}`). Refused (exit
2) for a missing bundle, a trace file that does not validate as a
`hotato.voice_trace.v1` JSONL, or an already-attached trace without
`--force`.

### Report wording

When a `tts_cancel_requested` / `tts_audio_stopped` pair is present, the
timeline states the measured delta plainly:

> Evidence suggests TTS cancellation delay: cancel requested at 2.60s,
> audio stopped at 2.90s (delta 0.30s).
> Hotato does not prove root cause.
> Unknowns: no client-side playout trace was attached.

The last line is always present in this release: a client-side audio
playout trace (the point where the caller's device stopped rendering
audio, as opposed to when the server issued the stop) sits outside what
this release's span types collect, so the gap is always named plainly.

## Export

```bash
hotato trace export contracts/refund-cutoff-001.hotato --format otel --out otel.jsonl
```

Writes the bundle's attached trace back out as hotato's OTel bridge JSONL
-- the exact shape `trace ingest` reads, so `ingest -> attach -> export ->
ingest` round-trips the identical spans. `--format` is claimed by the
export format on this subcommand (only `otel` is supported today); pass
`--json` for the machine result summary instead of the default text line.
Refused (exit 2) when the bundle has no attached trace, or `--out` exists
without `--force`.

## What a voice trace adds

A voice trace adds timing correlation to a failure contract's existing
timing measurement: a pattern of events lined up with it, named plainly.
Authorization, identity, compliance, policy safety, intent, and root cause
stay outside its claims, and a client-side playout moment is named only
when one was attached.

## Read more

- Source formats and the OTel bridge JSONL shape:
  [`docs/OTEL.md`](OTEL.md)
- Failure contracts a trace attaches to: [`docs/CONTRACTS.md`](CONTRACTS.md)


================================================================================
FILE: docs/OTEL.md
================================================================================

# OTel ingest: two source shapes

`hotato trace ingest --otel FILE` turns an OTel trace into hotato's own
`hotato.voice_trace.v1` spans. It reads a file you already have -- an
exported trace, or a small script's own event log -- offline, once, and
translates `name`, `startTimeUnixNano`/`endTimeUnixNano`, `attributes`, and
span `events` from a standard export into hotato's span shape. It
recognizes two input shapes; `source.format` records which one it used
(`"otel-json"` or `"otel-jsonl-bridge"`).

## 1. Standard OTel JSON export (`otel-json`)

A single JSON document with a top-level `resourceSpans` array (the shape a
standard OTel exporter/collector writes):

```json
{
  "resourceSpans": [{
    "resource": {"attributes": [
      {"key": "service.name", "value": {"stringValue": "vapi"}},
      {"key": "git_sha", "value": {"stringValue": "cafebabe"}}
    ]},
    "scopeSpans": [{"spans": [
      {
        "name": "agent_audio_active",
        "startTimeUnixNano": "1000000000",
        "endTimeUnixNano": "4400000000"
      },
      {
        "name": "tts.cancel_requested",
        "startTimeUnixNano": "2600000000",
        "events": []
      }
    ]}]
  }]
}
```

- Resource attributes flatten into a plain dict: `service.name` (or a custom
  `stack` attribute) becomes `deployment.stack`, and `git_sha` /
  `config_hash` attributes, when present, fill the matching deployment
  fields. Only the FIRST `resourceSpans` entry's resource supplies
  deployment metadata; every entry's spans are still walked.
- Timestamps convert to seconds relative to the EARLIEST timestamp anywhere
  in the file, matching the audio-relative-seconds convention every other
  hotato timestamp uses.
- A span's own `name` maps to a hotato span `type` via a small documented
  table (`caller_audio_active`, `agent_audio_active`,
  `tts.cancel_requested` -> `tts_cancel_requested`,
  `tts.audio_stopped` -> `tts_audio_stopped`, `asr.partial` -> `asr_partial`,
  `llm.first_token` -> `llm_first_token`); an unmapped name passes through
  unchanged.
- Span `events` -- OTel's own point-in-time markers nested inside a span,
  the natural place a production pipeline puts a `tts.cancel_requested`
  marker inside a broader `tts_playback` span -- flatten into their own
  point events the same way top-level spans do.
- A `tool_call`-mapped span takes its `name` field from the `tool.name` /
  `gen_ai.tool.name` attribute; a `latency_ms` attribute is used when
  present, otherwise computed from the span's own start/end.
- An `asr_partial`-mapped span's `text` / `asr.transcript.partial` attribute
  is captured and redacted by default (`text_redacted: true`); pass
  `--include-text` to keep it in the output.

## 2. Hotato's OTel bridge JSONL (`otel-jsonl-bridge`)

The documented shape for a script or test fixture that skips a full OTel
exporter: one JSON object per line (or one bare JSON array of the same
objects). Each line is either a span or a meta/resource line. A span line
uses `type` directly; `name` on a span line is reserved for `tool_call`'s
own tool name, distinct from the span kind:

```
{"type": "caller_audio_active", "start_sec": 2.40, "end_sec": 4.10}
{"type": "agent_audio_active", "start_sec": 0.00, "end_sec": 2.90}
{"type": "tts_cancel_requested", "time_sec": 2.60}
{"type": "tts_audio_stopped", "time_sec": 2.90}
{"type": "asr_partial", "start_sec": 2.40, "end_sec": 2.95, "text": "wait, I need a refund"}
{"type": "tool_call", "start_sec": 1.10, "end_sec": 1.42, "name": "lookup_order", "latency_ms": 320}
```

An optional meta/resource line (no `type` key) attaches deployment
metadata and a call id:

```
{"call_id": "demo-call-001", "deployment": {"stack": "vapi", "agent_id": "agent-demo-1", "git_sha": "deadbeefcafe", "config_hash": "sha256:0f1e2d3c"}}
```

`--call-id` / `--stack` / `--agent-id` / `--git-sha` / `--config-hash` on
`trace ingest` override whatever a meta line (or a standard export's
resource) provided. `hotato trace export` writes exactly this bridge
shape, so `ingest -> attach -> export -> ingest` round-trips the same
spans; the shipped test fixture (`tests/data/otel/demo-trace.otel.jsonl`)
uses this shape.

## Which one to use

Already running an OTel collector or SDK? Point `--otel` at that trace
export directly -- the standard-export path reads it, best-effort. Wiring a
quick script (a webhook handler, a log-line scraper) outside a full OTel
pipeline? Write the bridge JSONL directly -- the same information, already
flat.


================================================================================
FILE: docs/REPORTS.md
================================================================================

# Reports: doctor, report, team, export

Four surfaces, one scorer: reproducible timing measurements read straight
from the envelope, with the method exposed at every layer.

## `hotato doctor`: the 5-minute path

One command: pass a recording and it scores that, or run it bare and it runs
the bundled self-test battery. Either way it writes a self-contained HTML
report and opens it in your browser -- on a headless box, it prints the path
instead.

```bash
uvx hotato doctor --stereo call.wav     # score your call, open the report
uvx hotato doctor                       # self-test fallback, same flow
uvx hotato doctor --demo --no-open --out report.html
```

It wraps the scorer and report, and runs offline end to end. Exit codes
match `run`: `0` all pass, `1` a regression (`--no-fail` forces `0`), `2`
usage or IO error, or a not-scorable recording. Not-scorable means the
recording lacks a moment to measure -- the caller channel is silent, or the
agent wasn't talking when the caller started -- reported plainly instead of
a verdict.

**Your own recording gets its audio embedded in the report by default.**
Scoring a call with `--stereo` / `--caller`+`--agent` writes the exact scored
audio into the HTML file as a base64 data URI, so hearing the moment next to
its timeline works offline. The bundled self-test fallback stays unembedded.
Sharing, mailing, or posting the resulting `report.html` shares the raw
recording -- treat it with the same care.

## `hotato report`: the visual report

One self-contained file -- inline CSS, inline SVG, zero external requests --
that opens by double-click and survives being mailed around.

```bash
uvx hotato report --stereo call.wav --out report.html
uvx hotato report --suite barge-in --out selftest.html
uvx hotato report --stereo call.wav --format md --out report.md
uvx hotato report --stereo call.wav --embed-audio --out report.html  # opt-in: embed the audio too
```

`--embed-audio` embeds the exact scored audio (base64, under a size cap) so
the report is a fully self-contained, hearable artifact. The same caution
applies here: a report built with `--embed-audio` (or `hotato doctor` on
your own recording, which sets it by default) carries the call audio inside
the HTML file -- give it the same care as the raw recording before posting
it publicly or attaching it to a public issue/PR.

Per event it draws a to-scale caller/agent activity timeline from the frame
data -- overlap shaded, caller-onset and yield markers, measured talk-over
seconds, expected vs. measured, and a PASS or FAIL chip.

Once the page has at least three event cards, an analytics rollup follows,
computed from the same measurements (fewer events, and the rollup is
skipped):

- a **time-to-yield distribution** strip, one dot per measured yield, with
  mean, median, and p90 (definitions in `METHODOLOGY.md`);
- a **talk-over histogram**, per-event seconds bucketed on a fixed grid;
- **failure clustering by fix class**, so a batch of failures reads as "these
  five share one config setting" instead of five separate mysteries.

Every timeline carries a collapsible **frame inspector**: the full frame dump
behind that event as a table (`t_sec`, per-channel dBFS, active flags,
thresholds), so any pixel on the page can be re-derived by hand. The exact
`ScoreConfig` thresholds sit in one collapsed "Thresholds used" panel at the
end of the page -- the run is reproducible without stamping the parameter
table above every render.

### Voice-trace context with `--trace`

Pass `--trace voice_trace.jsonl` (a `hotato.voice_trace.v1` file, written by
`hotato trace ingest`) to attach a voice trace as **context** next to the
timing:

```bash
hotato trace ingest --otel spans.otel.jsonl --out voice_trace.jsonl
hotato report --stereo call.wav --trace voice_trace.jsonl --out report.html
```

The report gains one collapsed, clearly-labelled "Trace (context, not a
score)" section: the trace's discrete voice-pipeline events -- TTS
cancel/stop, ASR partials, tool calls -- as a mono span table (type, name,
start, end, detail). Like `--base` and an attached `assert.v1` envelope, the
report renders the already-produced trace artifact as data, context
alongside the score. The section stays scoped to context: `did_yield`,
`talk_over_sec`, `seconds_to_yield`, and the PASS/FAIL verdict come from the
scorer alone, and the trace folds into the machine envelope as an additive
top-level `trace_context` key. A report built without `--trace` is
byte-identical to one built before the flag existed.

Redaction carries through: a span with `text_redacted: true` (e.g. an
`asr_partial` ingested without `--include-text`) shows a `[redacted]`
placeholder in place of its text, so a report shared outside the fleet
carries only what the trace already chose to keep. The same `trace=`
parameter is on `build_report_html` / `build_report_md`; see `docs/TRACE.md`
for the trace format and `hotato trace ingest`.

### Reliability in the scorecard (pass@1 / pass@k / pass^k)

When a report carries an `assert.v1` envelope with dimension-tagged results,
the "Deterministic" shelf renders as a per-dimension **scorecard** (outcome /
policy / conversation / speech / reliability). The **Reliability** dimension
is pass^k's home, rendering the repetition data you thread in:

```python
from hotato import report, simulate

summary = simulate.run_matrix(scenario, conversation_test=ct)   # a matrix aggregate
html, _ = report.build_report_html(stereo="call.wav",
                                   assertions=env, reliability=summary)
```

`reliability=` accepts a `simulate.run_matrix` summary, a bare
`simulate.reliability()` dict, or a `{"aggregate": <reliability dict>, "origin":
...}` wrapper. The dimension shows each number **labeled**, tabular mono:

- **pass@1** (single-run pass rate), **pass@k** (>=1 of k passed), and **pass^k**
  (ALL k passed), with `n`, `k`, `passes`, and a **Wilson 95% CI** on pass@1;
- a **per-variation-cell** breakdown (each cell its own pass^k) when the summary
  carries one;
- a **SIMULATOR_INVALID** bucket -- broken fixtures, shown separately and
  **excluded from n**, kept distinct from an agent PASS/FAIL.

pass^k stays its OWN number, on its own lane -- there's no `overall_score`
field for it to blend into. Runs from simulation are labeled
**origin=simulated**: a simulator's replay reliability, scoped apart from
production reliability.

`hotato test run --repetitions N` (with `N > 1`) computes this aggregate over the
N deterministic runs and threads it into `report.{html,md}` automatically. With
no repetition data (`reliability=None`, or `--repetitions 1`) the dimension shows
the empty-state -- "not measured: no repeated runs in this report" -- and
the report is byte-identical to one built without the parameter.

### Regression deltas with `--base`

Save an envelope, then compare a later run against it:

```bash
hotato run --suite barge-in --format json > base.json
hotato report --suite barge-in --base base.json --out report.html
```

The report renders per-scenario talk-over and time-to-yield deltas, worse
and better marks clearly flagged. The same `--base` flag drives
`scripts/pr_comment.py`'s CI comment (`docs/CI.md`).

### PDF

The page ships print CSS: print it from any browser and the interactive
parts collapse into a clean paper layout -- print-to-PDF is the PDF export.

## `hotato team`: the trend view

Aggregates a directory of run envelopes into one trend.

```bash
hotato run --suite barge-in --format json > runs/001.json
# ... more runs over days or branches ...
hotato team runs/ --html team.html --out agg.json
```

It reports: number of runs; mean/median/p90 talk-over and time-to-yield
pooled across all events; mean/median/p90/p95 response gap (dead air before
the agent speaks) pooled the same way; pass rate per run over time; the most
common failure class; and a pass-rate trend line in the HTML page.
`--order mtime` (default) orders runs by file time; `--order name` uses the
filename, so a numeric prefix acts as an explicit index.

`--max-response-gap SECONDS` turns the pooled p95 response gap into a
latency SLA: the run exits `1` exactly when p95 exceeds the bound, the same
pass/fail contract as a talk-over or time-to-yield regression (`--no-fail`
always exits `0`). Percentile definitions: `METHODOLOGY.md`; the pooling
shape is `dist_summary` in `src/hotato/_stats.py`.

Fewer than two runs is stated plainly, and exits `0`; a trend line renders
once there are enough points to mean something.

## `hotato export`: research-grade CSVs

Scores a recording (or the bundled battery) exactly like `hotato run`, and
writes three files into a directory:

```bash
uvx hotato export --stereo call.wav --out research/
uvx hotato export --suite barge-in --out research/
```

- `events.csv`: one row per scored event, every measured signal plus verdict.
- `frames.csv`: one row per VAD frame, the evidence behind every number.
- `envelope.json`: the standard machine envelope, unchanged.

Column meanings are documented in comment lines at the top of each CSV, so the
files are self-describing when they land in a notebook or a stats package
months later. An empty cell means "not derivable", distinct from a zero.
Stdlib only, offline.

`export` also prints mean/median/p90/p95 response gap pooled across the
exported events, and accepts the same `--max-response-gap SECONDS` latency
SLA gate as `team` (exit `1` when the pooled p95 exceeds the bound). A plain
export with no `--max-response-gap` writes a byte-identical `envelope.json`;
the pooled numbers and the gate live only in the printed summary and the
returned manifest, never in the CSVs or the envelope file.


================================================================================
FILE: docs/ASSERTIONS.md
================================================================================

# Assertions (`assert.v1`): deterministic, typed, each scored on its own lane

`hotato.assert_` checks a call's transcript, ingested trace, and timing
against a fixed set of typed assertions, each kind scored on its own lane by
construction. Every kind is a regex, checksum, or span/dict lookup: deterministic
end to end, with no model call in the loop.

```python
from hotato import assert_ as A

ctx = A.build_context(
    transcript_path="transcript.json",   # or transcript=[...] turns already in hand
    trace_path="voice_trace.jsonl",      # or spans=[...] already in hand
    timing=envelope,                     # optional: a run's envelope.v1 events
)
env = A.run_assertions_from_file("assertions.yaml", ctx)
print(env["exit_code"])   # 0 pass, 1 fail
```

Python API: [`docs/API.md`](API.md) covers the scoring envelope this
complements; the schema is `src/hotato/schema/assert.v1.json`.

## The core deterministic kinds

Every kind is deterministic, so every result carries `deterministic: true`,
including an `INCONCLUSIVE` one. The five below are the original core; the
full vocabulary is under [the whole `kind` vocabulary](#the-whole-kind-vocabulary).

- **`phrase`** -- checks: a regex is present (or, in `absent` mode, never
  present), with an optional `role` filter and `position`
  (`first`/`last`/`any`). Reads: transcript text.
- **`pii`** -- checks: deterministic detectors (`ssn`, `card_luhn` with full
  Luhn validation, `email`, `phone`) find nothing, `mode: must_not_leak`.
  Reads: transcript text.
- **`policy`** -- checks: a named, versioned, offline rule pack's
  banned-language and required-disclosure rules. Reads: transcript text.
- **`tool_call`** -- checks: a tool was (or was not) called, with an optional
  argument subset, a count bound, a required order across tools, or a "never
  before" ordering constraint. Reads: ingested `voice_trace.v1` spans only.
- **`outcome`** -- checks: task success as `all_of`/`any_of` a list of the
  sub-predicates above (`tool_called`, `phrase`, `field_present`), reported as
  a `met`/`of` fraction. Reads: whichever context each sub-predicate needs.

`tool_call` checks the ingested trace (`hotato trace ingest`, [`docs/TRACE.md`](TRACE.md))
alone -- that's the evidence a tool ran; an agent's own words claiming it ran
don't move the needle. `pii` surfaces only a `[REDACTED]` transcript artifact
plus hit metadata (detector name, transcript turn index, role) -- never the
matched text.

### The whole `kind` vocabulary

The full `assert.v1` `kind` vocabulary is **17 deterministic kinds** -- all
`deterministic: true`, all model-free. Alongside the five core kinds:
`tool_result` and `tool_error` (a tool's returned value or raised error, read
from the trace), `state` and `state_change` (a state adapter's snapshot or
transition -- see [STATE-ADAPTERS.md](STATE-ADAPTERS.md)), `handoff`, `dtmf`,
`termination`, `latency`, `timing_contract`, `entity_accuracy`, `sequence`, and
`count`. Two further NAMED kinds, `human_rubric` and `judge_rubric`, belong to
the SEPARATE model-judged rubric lane ([RUBRIC.md](RUBRIC.md)); inside a raw
`assert.v1` document they resolve to a deterministic `INCONCLUSIVE` that points
at that lane, so no model runs here and the deterministic guarantee holds.

## Context: transcript, trace, timing

`build_context` assembles the three inputs an assertion run is evaluated
against, each built from hotato's existing primitives:

- **transcript**: `hotato.transcribe` (the opt-in `[transcribe]` extra,
  faster-whisper) produces one, or pass `--transcript FILE` / `transcript_path=`
  with a JSON file you already have -- a plain array of `{role, text, start,
  end}` turns, or the `{"segments": [...]}` shape `hotato.transcribe` and the
  MCP surface write.
- **trace**: `hotato trace ingest --otel FILE --out voice_trace.jsonl`
  ([`docs/TRACE.md`](TRACE.md), [`docs/OTEL.md`](OTEL.md)) produces the
  `hotato.voice_trace.v1` spans `tool_call` reads (`name`, `arguments`, and --
  when the source trace carries them -- `result`/`error`).
- **timing**: a scoring run's own envelope (`hotato run --format json`,
  [`docs/API.md`](API.md)) passed straight through as read-only context for
  `outcome`'s `field_present` sub-predicate. Nothing here recomputes it.

Context you never supply stays `None`, distinct from `[]` or `{}`: a value
that was supplied and happens to be empty. An assertion whose required input
is absent reports `INCONCLUSIVE`. `tool_call` with `spans=[]` (a trace was
ingested, it just has zero spans) is a `FAIL`, distinct from `tool_call` with
no trace at all, which reports `INCONCLUSIVE`.

## `assertions.yaml`

A small, dependency-free YAML subset (block mappings/sequences, flow
`[...]`/`{...}`, quoted or bare scalars, `#` comments) -- or a document that is
already valid JSON, accepted directly. Hotato parses this subset itself, so the
core stays zero third-party dependency either way.

```yaml
version: 1
assertions:
  - id: refund-confirmed
    kind: outcome
    all_of: [{tool_called: issue_refund}, {phrase: "confirmation number", role: agent}]
  - id: tool-order
    kind: tool_call
    require_order: [verify_identity, lookup_account, issue_refund]
    never_before: {tool: issue_refund, until: verify_identity}
  - id: disclosure
    kind: phrase
    regex: "recorded for quality"
    role: agent
    position: first
  - id: no-ssn-leak
    kind: pii
    detectors: [ssn, card_luhn]
    mode: must_not_leak
```

Every assertion needs a unique `id` and a recognized `kind`. The kind-specific
fields are validated (bad regex, unknown detector, missing required field, an
unsupported `version`) up front, before anything is evaluated -- a malformed
file is caught whole, before partial results exist.

## The deterministic/judge split

This is the entire point of the module, and it is structural, not a convention
someone can quietly break:

- Every result carries `kind` and `deterministic: true` -- true on every
  deterministic kind and every status, including `INCONCLUSIVE`, itself a
  deterministic read of missing required input.
- The envelope's `summary` **splits** `deterministic` (`{pass, fail,
  inconclusive}`) from `judge` (`{pass, fail}`), each in its own count -- the
  schema (`src/hotato/schema/assert.v1.json`) enforces this with
  `"overall_score": false` and a `not: {required: [overall_score]}` on the
  summary object.
- `judge` -- an LLM-scored rubric kind -- is kept structurally quarantined
  from the deterministic count, so a model-scored result can never blend
  into it. `summary.judge` reports `{"pass": 0, "fail": 0}`, and
  `summary.note` states how many judge-scored assertions ran.
- Same inputs, same file, same result, every time: `run_assertions` is
  byte-stable across repeated calls on identical input -- no wall-clock
  timestamp or random id in the mix.

The report (below) renders this as two visually separate shelves instead of one
number, so it is visible on the page, not just in the JSON.

## The report: two shelves, each counted on its own

`hotato.report.build_report_html` / `build_report_md` accept an optional
`assertions=` parameter: an already-evaluated `assert.v1` envelope (build one
with `run_assertions` / `run_assertions_from_file` / `run_assertions_from_yaml`
above). Exactly like `base` (a previous run envelope) and `transcript` (an
already-produced ASR artifact), the report purely renders an assertion result
handed to it.

```python
from hotato import assert_ as A, report

env = A.run_assertions_from_file("assertions.yaml", ctx)
html, _ = report.build_report_html(suite="barge-in", assertions=env)
```

When present, it adds one "Assertions" section:

- **The headline.** Two counts side by side, each scored on its own:
  `N deterministic pass / M fail  K judge-scored (advisory)`.
- **Deterministic** (audio / timing / transcript / trace derived): one
  PER-DIMENSION TYPED card per result -- a kind tag, the `deterministic` flag,
  the PASS/FAIL/INCONCLUSIVE chip, and that result's kind-specific fields (a
  `pii` card's hit detail and redacted transcript, a `policy` card's matched
  rules and pack name, a `tool_call` card's grounding span ids, an `outcome`
  card's met/of fraction).
- **Model-assisted (advisory, quarantined)**: stays empty here by design,
  with a note pointing at the model-judged rubric lane
  ([RUBRIC.md](RUBRIC.md)) where that scoring runs.

`assertions=None` (the default) is byte-identical to a report built before this
parameter existed.

## `inconclusive_policy`: making missing input gate CI

By default an `INCONCLUSIVE` result -- a check whose required input was
absent -- leaves the run's exit code unaffected, the right default for an
exploratory run. `inconclusive_policy` lets a suite opt into gating on that,
so a transcript or trace that never arrived fails loudly instead of leaving
the suite silently green:

| value | how `INCONCLUSIVE` gates | `exit_code` |
| --- | --- | --- |
| `report` (default) | reports missing input and leaves the gate unchanged | `1` if any `FAIL`, else `0` |
| `fail` | gates exactly like a `FAIL` | `1` if any `FAIL` **or** `INCONCLUSIVE`, else `0` |
| `refuse` | refuses to return a verdict at all | `2` if any `INCONCLUSIVE`, else `1` if any `FAIL`, else `0` |

**`refuse` precedence.** Under `refuse`, an `INCONCLUSIVE` result exits `2`
*even if another assertion also `FAIL`ed* -- the exit-2 refusal takes precedence
over the FAIL. A run that cannot fully see its inputs withholds its verdict.

The **default is `report`**, so a suite that sets nothing gates exactly as
before this field existed -- fully backward-compatible. **CI and compliance
suites should set `fail` or `refuse`**, so a missing transcript or trace fails
loudly.

Set it as an optional top-level key:

```yaml
version: 1
inconclusive_policy: fail   # or: refuse | report (default)
assertions:
  - id: disclosure
    kind: phrase
    regex: "recorded for quality"
    role: agent
```

or override the document's key from the CLI (the flag wins):

```bash
hotato assert run --assertions assertions.yaml --transcript call.json \
    --inconclusive-policy refuse
```

or from Python (an explicit argument overrides the document's key; absent both,
`report` applies):

```python
env = A.run_assertions_from_file("assertions.yaml", ctx, inconclusive_policy="fail")
```

A bad value (anything but `report`/`fail`/`refuse`), in the document or passed
explicitly, is a usage error (`ValueError`, exit `2`) raised during validation,
before any assertion is evaluated. The envelope always carries the applied
`inconclusive_policy` and states it, with the counts, in `summary.note`.

## Mapping to a CI gate

Same exit-code convention as every hotato command: `0` every assertion passed
(under `report`, an `INCONCLUSIVE` reports missing input and leaves the exit at
`0`; under `fail` it gates like `FAIL`; under `refuse` it exits `2`), `1` at
least one deterministic status is `FAIL` (or, under `fail`, `INCONCLUSIVE`), `2`
a refusal under `refuse` (an `INCONCLUSIVE`, taking precedence over a `FAIL`) or
a malformed file / bad input, raised before any assertion is evaluated.

```bash
python3 - <<'PY'
from hotato import assert_ as A
import sys

ctx = A.build_context(transcript_path="transcript.json", trace_path="voice_trace.jsonl")
env = A.run_assertions_from_file("assertions.yaml", ctx)
print(env["summary"]["note"])
sys.exit(env["exit_code"])
PY
```

A gate on `assert.v1` and a gate on the timing scorer's own `exit_code`
(`hotato run` / `hotato verify`, [`docs/CI.md`](CI.md)) are two different,
composable guarantees: one gates turn-taking timing, the other gates
transcript/trace content. Run both; neither exit code substitutes for the
other's.


================================================================================
FILE: docs/CONVERSATION-TEST.md
================================================================================

# Conversation tests (`hotato test run`): one file, one call, a per-dimension scorecard

A **conversation test** (`hotato.conversation-test.v1`) is one file that
defines one testable conversation: the agent under test, the simulated
caller, the environment, two SEPARATE assertion lanes, and an explicit
success condition. `hotato test run` evaluates a supplied call against
that file and produces a **conversation artifact**
(`hotato.conversation.v1`) whose evidence is bound by sha256, plus a
**per-dimension scorecard**. Success is a boolean over a small closed
vocabulary of named conditions, with every dimension counted on its own,
including in `--format json`.

This is the single documented end-to-end conversation-QA workflow:

```
hotato scenario init   ->   hotato test run   ->   hotato conversation verify
   (author a file)          (evaluate a call)        (digest-check the artifact)
```

## 1. Author a conversation test

```bash
hotato scenario init refund-flow --agent support-v3 --out refund.yaml
```

The starter carries the two lanes, tagged checks across the five report
dimensions, and a boolean `success`:

```yaml
kind: hotato.conversation-test
version: 1
id: refund-flow
agent: support-v3
caller:
  persona: a customer whose order arrived damaged and wants a refund
  goal: get a refund for order A-1001
  facts: {order_id: A-1001}
assertions:
  # DETERMINISTIC lane: pure, offline, no model. Each result is TAGGED with one
  # of the five report dimensions (a grouping key, never a weight).
  deterministic:
    - id: disclosure-said
      kind: phrase
      regex: "recorded for quality"
      role: agent
      dimension: policy
    - id: refund-tool-called
      kind: tool_call
      name: issue_refund
      dimension: outcome
    - id: lookup-then-refund
      kind: sequence
      steps: [{tool: lookup_order}, {tool: issue_refund}]
      dimension: conversation
    - id: refund-latency
      kind: latency
      tool: issue_refund
      max_ms: 1500
      dimension: speech
  # RUBRIC (model-judged) lane lands in Phase 3. In Phase 1 each is INCONCLUSIVE
  # (no model runs); it never counts toward the deterministic result or exit code.
  rubric:
    - id: was-empathetic
      kind: judge_rubric
      dimension: conversation
# inconclusive_policy: fail  # CI/compliance suites should set fail or refuse
success:
  # Success is the conjunction of these named conditions -- never a score.
  required: [all_deterministic_assertions_pass, no_rubric_failure]
  report_dimensions: [outcome, policy, conversation, speech, reliability]
```

Validate one file or a whole directory (exit 2 on any malformed file):

```bash
hotato scenario validate refund.yaml
hotato scenario validate ./scenarios --format json
```

The closed vocabularies are enforced structurally
(`hotato.conversation_test`): `success.required` is drawn from
`all_deterministic_assertions_pass`, `no_deterministic_fail`,
`no_rubric_failure`, `no_inconclusive`; `dimension` is one of `outcome`,
`policy`, `conversation`, `speech`, `reliability`; an `overall_score` key
anywhere is rejected.

## 2. Evaluate a call

`hotato test run` takes the file plus whatever evidence you have for the
call -- a scored recording (`--audio`), an ingested trace (`--trace`, see
[`docs/TRACE.md`](TRACE.md)), a transcript (`--transcript`), and a
post-call state sandbox (`--state`, Authority 2). Each supplied piece
feeds the assertions; each ABSENT piece leaves the checks that need it
`INCONCLUSIVE`.

```bash
hotato test run refund.yaml --agent support-v3 \
    --audio call.wav \
    --trace voice_trace.jsonl \
    --transcript call.transcript.json \
    --out ./conv-artifact --format html
```

Flow: load and validate the file -> build the evaluation context from the
supplied transcript / trace / state / (scored) timing -> evaluate the
DETERMINISTIC lane (the rubric lane stays quarantined ->
`INCONCLUSIVE`) -> evaluate `success.required` over the results -> bind
the evidence into a `hotato.conversation.v1` artifact -> render the
unified report + per-dimension scorecard. The per-dimension summary
printed to stdout:

```
hotato test run: refund-flow (agent support-v3) -- exit_code=0
inconclusive_policy: report
success: PASS  (required: all_deterministic_assertions_pass, no_rubric_failure)
  [ok] all_deterministic_assertions_pass
  [ok] no_rubric_failure
per-dimension (grouped view; never blended):
  outcome       1 pass / 0 fail / 0 inconclusive
  policy        1 pass / 0 fail / 0 inconclusive
  conversation  1 pass / 0 fail / 0 inconclusive
  speech        1 pass / 0 fail / 0 inconclusive
  reliability   0 pass / 0 fail / 0 inconclusive
reliability over 1 repeated run(s) of the deterministic lane on the same supplied real recording; the lane has zero variance, so pass^k == pass@1 -- reported honestly, not fabricated variance
rubric lane: 1 assertion(s) INCONCLUSIVE (quarantined, Phase 3 -- no model ran)
deterministic: 4 pass, 0 fail, 0 inconclusive
judge: 0 pass, 0 fail (no judge/rubric kind is built in this release)
```

`--format`:

| format | stdout | writes into `--out` |
| --- | --- | --- |
| `text` (default) | the per-dimension summary | the conversation artifact |
| `html` / `md` | the summary (+ where files landed) | the artifact **and** `report.{html,md}` (needs `--audio`) |
| `json` | the full machine result | the conversation artifact |

### Exit code

The exit code honors the file's `inconclusive_policy` exactly as
`assert run` does, then is raised to non-zero when a `success.required`
condition fails:

| `inconclusive_policy` | an `INCONCLUSIVE` (missing-input) result | a `FAIL` |
| --- | --- | --- |
| `report` (default) | does not gate (exit 0) | exit 1 |
| `fail` | gates like a FAIL (exit 1) | exit 1 |
| `refuse` | withholds the verdict (exit 2, precedence) | exit 1 |

A `success.required` failure makes an otherwise-passing run non-zero; a
refuse (exit 2) is never downgraded.

### Reliability and repetitions

`--repetitions N` runs the deterministic lane N times and reports the
per-run results, the run count, and a reliability aggregate: **pass@1**
(single-run pass rate), **pass@k** (>=1 of k passed), **pass^k** (all k
passed), plus a Wilson 95% CI. Every run scores the same recording, so
the deterministic lane has zero variance and `pass^k == pass@1`. With
`N > 1` the aggregate is threaded into the report's Reliability dimension
(`--format html/md`); with no repetition data that dimension shows the
empty-state ("not measured: no repeated runs in this report"). pass^k
stays its OWN number, kept separate from every other dimension and from
`overall_score`.

## 3. Verify the artifact

The conversation artifact directory binds each child by sha256:

```
conv-artifact/
  conversation.json      # the manifest (origin real|simulated + bound digests)
  audio/call.wav         # the recording (single dual-channel form)
  transcript.json
  trace.jsonl
  timing.json            # the scored envelope
  assertions.json        # the evaluated assert.v1 envelope
  report.html            # the unified report + per-dimension scorecard
```

`hotato conversation verify` re-hashes every bound child and REFUSES
(exit 2) on any digest mismatch or missing file -- a tampered or absent
artifact is refused outright:

```bash
hotato conversation verify ./conv-artifact
# conversation refund-flow: VERIFIED
#   verified: assertions, audio, timing, trace, transcript
#   all 5 bound artifact(s) re-hashed to their recorded digest
```

`origin.kind` is `real` by default (a supplied recording is evaluated
as-is) and `simulated` only when the test file carries a `simulator`
block, which must declare its `model_id` / `scenario_id` / `seed` --
synthetic and live origins are never conflated.

## The deterministic/judge split (structural)

* **Each dimension scored on its own.** Success is a boolean conjunction
  of named conditions; the scorecard groups the same results by
  dimension, each with its own counts.
* **Two separate lanes.** Deterministic checks (regex / checksum / span /
  state lookup -- no model) are evaluated; the model-judged rubric lane
  is quarantined until Phase 3, reported `INCONCLUSIVE` and kept out of
  the deterministic summary and the exit code.
* **Grounded in authority.** `tool_result` / `tool_error` (Authority 1)
  read the ingested trace spans; `state` / `state_change` (Authority 2)
  query a post-call state adapter -- the trace and the adapter are the
  evidence a tool ran or a state changed, deterministic and model-free.
* **Missing input is `INCONCLUSIVE`.** A check whose evidence was not
  supplied reports `INCONCLUSIVE`.

## The bundled end-to-end example

`tests/data/conversation/` ships one worked call -- a conversation-test
file, a transcript, and a `voice_trace.v1` trace -- evaluated against the
bundled recording `01-hard-interruption.example.wav`.
`tests/test_test_run_cli.py`
(`test_bundled_call_one_file_end_to_end_scorecard_and_artifact`) drives
the whole workflow: one call evaluated for outcome / policy / timing /
transcript-facts / tool-behaviour from one file, producing an artifact
that `conversation verify` passes and a five-dimension scorecard.

## See also

* [`docs/ASSERTIONS.md`](ASSERTIONS.md) -- the deterministic assertion
  kinds.
* [`docs/TRACE.md`](TRACE.md) -- ingesting a `voice_trace.v1` trace.
* [`docs/REPORTS.md`](REPORTS.md) -- the unified report and the
  scorecard.


================================================================================
FILE: docs/RUBRIC.md
================================================================================

# Rubric evaluation: the model-judged lane (`rubric.v1`)

`hotato rubric` scores a call against a **user-authored rubric** with a
pinned **local** model, structurally separate from the deterministic
`assert.v1` wall. Every rubric result carries `deterministic: false` with
full provenance and lives on its own report shelf, counted apart from any
overall number.

- `assert.v1`'s kinds stay `deterministic: {const: true}` forever. A model
  verdict lives in its own `rubric.v1` lane, structurally apart from
  `assert.v1`.
- The default judge is a **local Ollama** model (`http://localhost:11434`),
  reached with only the stdlib. Zero egress; a hosted judge is opt-in (see
  below).
- **Advisory by default**: the deterministic checks set the gate, and a
  rubric FAIL is reported while the model's read stays advisory. `--gate`
  (or `--gate-judge` on `test run`) opts a team into gating on a rubric FAIL
  too.
- Missing or insufficient evidence resolves to `INCONCLUSIVE`, always
  labeled plainly. A `human_rubric` stays `INCONCLUSIVE` and human-required,
  by design.

## The rubric object

```yaml
version: 1
rubrics:
  - id: acknowledged-frustration
    kind: judge_rubric            # or human_rubric (human-scored only)
    dimension: conversation       # optional report-dimension tag
    criterion: "Did the agent acknowledge the caller's frustration before proposing a fix?"
    evidence: [transcript]        # transcript and/or tool_trace; absent -> INCONCLUSIVE
    examples: {pass: "...", fail: "..."}      # optional
    evaluation:
      model: qwen2.5vl:3b         # optional; else --judge-model / the default
      repetitions: 1              # N model calls, aggregated
      aggregation: unanimous_or_inconclusive
      confidence_required: 0.85
    review:
      human_required_on: [fail, disagreement, confidence_below_threshold]
```

## Run it

```bash
# advisory (exit 0 regardless of verdicts)
hotato rubric run --rubrics rubrics.yaml --transcript call.json --trace trace.jsonl

# opt into CI gating on a rubric FAIL
hotato rubric run --rubrics rubrics.yaml --transcript call.json --gate

# re-query the model and DIFF against the cached verdict (surface drift)
hotato rubric run --rubrics rubrics.yaml --transcript call.json --no-cache
```

Inside a conversation-test, author the rubric in the `assertions.rubric`
lane; `hotato test run` scores it inline (advisory; `--gate-judge` to gate)
and the unified report shows a populated **Model-assisted (advisory)** shelf
beside the deterministic one -- two counts side by side, each on its own
lane.

## Result provenance (`rubric.v1`)

Every result records: the pinned `model` + its content `model_digest`,
`provider`, `prompt_id` / `prompt_version` / `prompt_sha256`,
`temperature: 0`, the `input_sha256`, the `cache_key`, `cached`, the raw
`votes`, `disagreement`, `confidence`, and `citations` to the exact
transcript turns / trace events -- its own number, scored on its own lane;
the `rubric.v1` schema rejects an `overall_score` key.

## Reproducibility (stated precisely)

**What's guaranteed is replay.** Every verdict is content-addressed by
`sha256(provider:model + prompt_sha256 + input_sha256)` and cached (reusing
the content-addressed artifact store), so a cache hit reproduces the same
`verdict_sha256` every time. A fresh model call is a separate question,
checked explicitly with `--no-cache`, which re-queries the model and
**diffs** the result against the cached verdict, surfacing any drift
directly. `--sign` optionally signs a cached verdict as a "judge-record"
(Ed25519 `human` tier via the `[sign]` extra, else HMAC `human-shared`), so
a stored verdict stays provably unmutated.

## Egress

The default local judge stays on the box. A **hosted** judge
(`--judge-provider hosted --judge-endpoint URL`) or a **non-local**
`--judge-endpoint` sends the transcript off-box, and requires
`--judge-egress-opt-in` to proceed (exit 2 without it) -- the same posture
as `--diarizer pyannoteai --egress-opt-in`. See
[`docs/EGRESS.md`](EGRESS.md) and [`docs/THREAT-MODEL.md`](THREAT-MODEL.md).

## Calibration: measured credibility

```bash
hotato rubric calibrate --labeled ./labeled --out agreement.json
```

Each `*.json` item is `{rubric, transcript, trace?, label: pass|fail|inconclusive,
split?: train|held_out}`. Human labels are **mandatory** here: humans author
the labels, and the model is scored *against* them. The command computes
**agreement** (held-out items where the model verdict equals the human
label) and **selective accuracy** (agreement restricted to items where the
model committed to a verdict), and writes a reproducible artifact of raw
counts, method, and provenance. Re-running on the same corpus with the same
model reproduces the same split and verdicts -- the numbers travel with
their method and provenance built in.


================================================================================
FILE: docs/SIMULATE.md
================================================================================

# Simulate (`hotato simulate`): a scenario to a deterministic, labelled conversation

`hotato simulate` renders a **scenario** (`hotato.scenario.v1`) through a
deterministic scripted caller into one or more **conversation artifacts**
(`hotato.conversation.v1`), each labelled `origin=simulated`. It runs fully
offline with a scripted caller only -- the deterministic input side of a
simulation: the ground truth a caller holds, the caller's scripted turns,
the environment, and an optional variation matrix.

Use it to build the regression foundation you want in place before any
generative caller: a fixed `(scenario, seed)` reproduces the transcript
byte-for-byte (the produced transcript is content-hashed), so a run is a
stable, reusable fixture.

## Quickstart (zero-install with uvx)

```bash
# run zero-install with uvx (or: pipx install hotato)
uvx hotato --help

# 1. Write a minimal scenario simulate accepts as-is:
hotato simulate --init demo.scenario.json

# 2. Render it into a labelled origin=simulated conversation:
hotato simulate demo.scenario.json --out ./sim

# 3. (optional) digest-verify the produced artifact:
hotato conversation verify ./sim
```

Prefer zero files? The package ships a minimal scenario you can run
directly, no file on disk:

```bash
hotato simulate --example --out ./sim
```

Step 2 prints:

```
wrote 1 simulated artifact(s) under ./sim/
hotato simulate: demo -- 1 run(s), origin=simulated (never real)
  run 1: seed=0 77f1eb9e6994 sim=ok  -> ./sim
reliability: pass@1=1.000 pass@k=1.000 pass^k=1.000 (n=1)
```

`./sim` holds a `conversation.json` plus its bound transcript and
`voice_trace.jsonl`, all `origin.kind=simulated`.

> **Two different `scenario` concepts, one name.** `hotato simulate` consumes a
> `hotato.scenario.v1` doc -- author one with `hotato simulate --init`.
> `hotato scenario init` writes a separate file, a `hotato.conversation-test.v1`
> that `hotato test run` consumes -- see
> [CONVERSATION-TEST.md](CONVERSATION-TEST.md). Feed a conversation-test file
> to `simulate` and you get an actionable error that points you back to
> `--init`.

## What `--init` writes

`hotato simulate --init demo.scenario.json` writes a minimal, valid
`hotato.scenario.v1` doc -- a starter you edit for your own agent, shaping
the caller turns to match your own call. The scenario id derives from the
filename stem.

```json
{
  "kind": "hotato.scenario",
  "version": 1,
  "id": "demo",
  "goal": { "type": "get_refund", "target": "order A-1001" },
  "facts": { "order_id": "A-1001" },
  "caller": {
    "script": [
      { "say": "Hi, my order A-1001 arrived damaged and I would like a refund." },
      { "say": "Yes, please refund it to my card." }
    ],
    "behavior": { "backchannels": { "probability": 0.0 } }
  },
  "environment": { "locale": "en-US", "route": "phone" },
  "seed": 0
}
```

The caller's script holds only the caller's own turns -- a `say` is the
caller speaking. The schema enforces this structurally: there's no field
for the agent's words, so a scenario stays scoped to the caller's side by
construction, at the schema level.

Required fields: `kind`, `version`, `id`, `goal` (`type` + `target`), and
`caller.script` (at least one `say`). The full schema, including
`variation_matrix` and the optional deterministic `agent_mock`, is
[`schema/scenario.v1.json`](../src/hotato/schema/scenario.v1.json).

## The invariants, enforced structurally

- **`origin=simulated` on every produced conversation**, kept in its own
  bucket, apart from real calls. `write_artifact` only writes artifacts
  whose origin is simulated.
- **A bad rendering is `SIMULATOR_INVALID`, kept distinct from an agent
  PASS/FAIL.** The simulator only decides whether the produced conversation
  faithfully renders its scenario. Scoring is the separate assert layer's
  job, over the produced artifact.
- **A seeded replay is byte-identical** -- there's no model in this path. A
  fixed `(scenario, seed)` produces the same transcript bytes every time;
  different seeds differ only where the scenario allows it (probabilistic
  backchannels).
- **Each dimension scores on its own lane**, enforced structurally in both
  the schema, which rejects an `overall_score` key, and the code path.

## Reliability: pass@1 / pass@k / pass^k

`simulate` reports Reliability as its own dimension, scored on its own lane:

- `pass@1` -- the fraction of runs that rendered faithfully.
- `pass@k` -- at-least-one-passes across `k` runs.
- `pass^k` -- all-`k`-pass.

For the scripted deterministic caller, `pass^k == pass@1`: a seeded replay
is byte-identical, so every run has the same outcome. Variance shows up
only where the scenario introduces it. `--repetitions N` expands the
variation matrix so Reliability is measured over `N` runs.

## Simulate many scenarios in parallel (`--matrix`)

```bash
# expand the scenario's FULL variation matrix (locale x speaking_rate x noise x
# behavior x repetitions), render + validate each in a bounded pool:
hotato simulate --matrix demo.scenario.json --out ./matrix

# score each produced conversation against a conversation-test's DETERMINISTIC
# assertions; SIMULATOR_INVALID runs are bucketed separately, kept distinct
# from a PASS/FAIL:
hotato simulate --matrix demo.scenario.json \
    --conversation-test refund.test.yaml --parallel 8 --format json
```

The summary stays byte-identical no matter the worker count. Every result is
attributed to its own variation cell, each scored on its own lane.

## Exit codes

- **`0`** -- every produced conversation is `origin=simulated` and validated
  as a faithful rendering (and, under `--matrix --conversation-test`, every
  scored aggregate passed).
- **`1`** -- at least one produced simulation was `SIMULATOR_INVALID` -- a
  broken fixture, kept distinct from an agent PASS/FAIL -- or, under
  `--matrix --conversation-test`, a scored aggregate FAILed.
- **`2`** -- usage error / unusable input: a malformed or unreadable
  scenario / conversation-test file (or, under `--matrix
  --conversation-test` with `inconclusive_policy refuse`, a withheld
  verdict).

## Related

- [CONVERSATION-TEST.md](CONVERSATION-TEST.md) -- `hotato scenario init` /
  `hotato test run`: score a real call against a conversation-test.
- [`schema/scenario.v1.json`](../src/hotato/schema/scenario.v1.json) -- the
  full scenario schema.


================================================================================
FILE: docs/SUITE-RUN.md
================================================================================

# `hotato suite run`: execute a suite of conversation-tests

`hotato suite run` loads a `hotato.suite` file (a named set of
conversation-test refs, each scored on its own), resolves each ref relative
to the suite file, executes every test, and emits a per-dimension +
reliability report. It also records the run into the fleet registry so
`hotato serve` can browse it.

A conversation-test is one file that defines one testable conversation; see
[CONVERSATION-TEST.md](CONVERSATION-TEST.md). A suite groups those files
under a `suite_id`, a `purpose`, an `inconclusive_policy`, and a
`required_for_release` flag. The suite file is JSON or YAML.

## How a test runs

- A test that declares a `scenario` runs **offline** through the
  deterministic scripted-caller simulator. The scenario's variation matrix
  expands into concrete runs, each rendered, validated, and scored against
  the test's deterministic lane. Authority-1 tool spans and the Authority-2
  mock-state sandbox come from the scenario's `agent_mock`, fully simulated
  and offline. (Authority 2 is grounded state; see
  [STATE-ADAPTERS.md](STATE-ADAPTERS.md).)
- A test with no scenario is evaluated once against an empty context: every
  input-dependent check reports INCONCLUSIVE, never a guessed pass/fail.

## Every dimension scored on its own

The report carries per-dimension counts (`outcome`, `policy`,
`conversation`, `speech`, `reliability`) plus a reliability aggregate --
`pass@1`, `pass@k`, `pass^k`. Per-dimension counts group results across
tests; reliability stands as its own dimension over every valid simulated
run. Each dimension keeps its own count and its own verdict, exposed as-is,
including in `--format json`; the schemas reject an `overall_score` key.

A `SIMULATOR_INVALID` run flags a broken fixture. It is bucketed separately
and reported by test id and run id, kept out of every dimension and the
reliability aggregate.

## The suite's policy is authoritative

The suite's `inconclusive_policy` is the effective policy for every test it
names, keeping a required CI/compliance suite's gate consistent regardless
of a test's own policy. A suite with `inconclusive_policy: fail` makes an
INCONCLUSIVE (absent required input) FAIL the gate; `refuse` withholds the
verdict (exit `2`, precedence).

The suite exit code is the **worst** test outcome under that policy
(`refuse` > `fail` > `pass`), raised to at least `1` by any
`SIMULATOR_INVALID` run.

## Run it

```bash
hotato suite run examples/reference-agent/suite.json \
    --agent reference-agent-v1 \
    --release reference-agent-v1 \
    --out ./suite-out
```

`--agent ID` (required) is recorded on every conversation and the release.
`--release ID` names the release the runs are recorded under (default
`<agent>@<suite_id>`); use a stable id per release so
[`hotato release compare`](RELEASE-COMPARE.md) can diff two of them. `--out
DIR` writes the per-test simulated conversation artifacts plus
`suite-report.md`, `suite-report.html`, and `suite-run.json`. `--parallel N`
caps the worker threads for each scenario's matrix; the worker count never
changes the byte-identical result.

The runs are recorded into the fleet registry (`--registry`, default
`~/.hotato/fleet`; `--workspace`/`-w`, default `default`) as Release / Suite
/ Scenario / Run / Conversation / Evaluation rows, so `hotato serve -w
<workspace>` renders them (see [WORKSPACE.md](WORKSPACE.md)). Pass
`--no-registry` to report only.

## Worked example: the reference agent

`examples/reference-agent/` ships a full suite: 25 realistic voice-agent
jobs x 5 caller behaviours x 3 audio environments = 375 offline simulated
runs, scored across the outcome / policy / conversation / speech
dimensions. Its `suite.json` has `inconclusive_policy: fail` and
`required_for_release: true`.

The one-command path regenerates the files, runs the 375-run suite offline,
and records a browsable workspace:

```bash
cd examples/reference-agent
make reference
# then browse:
hotato serve --workspace reference --registry ./.workspace
```

The summary printed to stdout follows this shape (counts here are
schematic, illustrating the report's shape):

```
hotato suite run: reference-agent-suite (...) -- agent reference-agent-v1, release reference-agent-v1 -- exit_code=0
inconclusive_policy: fail  required_for_release: True
tests: 25  (25 pass, 0 fail, 0 refuse)
runs: 375 (375 valid, 0 SIMULATOR_INVALID -- broken fixtures, never an agent PASS/FAIL), origin=simulated
per-dimension (grouped across tests; never blended):
  outcome       ... pass / ... fail / ... inconclusive
  policy        ... pass / ... fail / ... inconclusive
  conversation  ... pass / ... fail / ... inconclusive
  speech        ... pass / ... fail / ... inconclusive
  reliability   ... pass / ... fail / ... inconclusive
reliability [origin=simulated]: pass@1=... pass@k=... pass^k=... (n=...)
per-test:
  [pass   ] refund-damaged-order            runs=15  ...
  ...
```

Origin is always labelled `simulated`, keeping a simulator's replay
reliability distinct from production reliability.

## Exit codes

- **`0`** -- every test in the suite passed under the suite's
  `inconclusive_policy` (and no run was `SIMULATOR_INVALID`).
- **`1`** -- at least one test FAILed (a `success.required` condition
  failed, a deterministic assertion FAILed, or -- under
  `inconclusive_policy fail` -- an INCONCLUSIVE gated), or a run was
  `SIMULATOR_INVALID` (a broken fixture, kept separate from any agent
  PASS/FAIL).
- **`2`** -- under `inconclusive_policy refuse` a scored INCONCLUSIVE
  withheld the verdict (takes precedence over a FAIL); OR a usage error /
  unusable input: a malformed suite / conversation-test / scenario file, or
  an unresolvable test/scenario ref.

## See also

- [CONVERSATION-TEST.md](CONVERSATION-TEST.md) -- the conversation-test file each suite ref points at.
- [STATE-ADAPTERS.md](STATE-ADAPTERS.md) -- Authority 2 state grounding for `state` / `state_change` checks.
- [RELEASE-COMPARE.md](RELEASE-COMPARE.md) -- diff two releases recorded by `suite run`.
- [WORKSPACE.md](WORKSPACE.md) -- browse the recorded runs with `hotato serve`.
- [SUITES.md](SUITES.md) -- the `corpus/suites/` scenario-audio suites for the `hotato run` scoring path (a distinct mechanism).


================================================================================
FILE: docs/RELEASE-COMPARE.md
================================================================================

# `hotato release compare`: diff two releases, per dimension

`hotato release compare BASELINE CANDIDATE` reads two releases from the fleet
registry and reports what moved between them -- `BASELINE` first, `CANDIDATE`
second. It reports movement; gating a release on that movement is a separate
step (use a `required_for_release` suite for that).

Each release is a snapshot recorded by [`hotato suite run`](SUITE-RUN.md).
Run the same suite twice under two stable `--release` ids in the same
workspace and registry, then diff them.

## What it reports

- **Per-dimension counts for each side, plus the per-count delta.** Each
  dimension (`outcome`, `policy`, `conversation`, `speech`, `reliability`)
  keeps its own three counts (pass / fail / inconclusive) and its own delta
  -- every dimension scored on its own lane.
- **New failures** -- a `scenario x dimension` that PASSED on the baseline
  and FAILs on the candidate. **Fixed-since** -- the reverse. Both diff
  **only** where BOTH releases ran the same `scenario x dimension`; a
  scenario only one side ran counts as new coverage, separate from a
  regression.
- **Per-scenario status changes** -- every `scenario x dimension` whose
  status differs between the two, diffed only where both sides have a
  comparable result.

The releases' pinned digests are surfaced, so the reader knows exactly which
two snapshots were compared: an exact digest match, every time.

## Empty state

A side with no runs is stated plainly:

- a release id absent from the workspace is reported as unregistered;
- a release that is registered but has **zero runs** is reported as an
  empty state.

When no `scenario x dimension` result is comparable across both releases, the
report says so: new-failures / fixed-since needs a scenario BOTH releases
ran.

## Compare

```bash
hotato release compare reference-agent-v1 reference-agent-rc2
```

`--workspace`/`-w ID` selects the fleet-registry workspace (default `default`);
`--registry PATH` sets the registry home (default `~/.hotato/fleet`);
`--format {text,json}` picks the output.

## Worked example: two reference-agent releases

Record two releases of the reference agent into the same registry and
workspace, then diff them. The reference example writes to
`examples/reference-agent/.workspace`, workspace `reference`:

```bash
cd examples/reference-agent

# record the baseline release
python run_reference.py --release reference-agent-v1

# edit the scenario / agent_mock fixtures (or point at a changed agent),
# regenerate, and record the candidate release
python run_reference.py --generate --release reference-agent-rc2

# diff them (same workspace + registry both runs recorded into)
hotato release compare reference-agent-v1 reference-agent-rc2 \
    --workspace reference --registry ./.workspace
```

Two runs over identical fixtures record byte-identical results, so the diff
shows all-zero deltas and no status changes. Edit the fixtures between the
two runs to see movement.

The text output follows this shape (counts here are illustrative placeholders):

```
hotato release compare: baseline reference-agent-v1 -> candidate reference-agent-rc2  (workspace reference)
baseline runs=375 conv=375 eval=...  candidate runs=375 conv=375 eval=...
per-dimension counts (baseline -> candidate, delta; never blended):
  outcome       ...P/...F/...I  ->  ...P/...F/...I   (dP +.., dF -.., dI +..)
  policy        ...P/...F/...I  ->  ...P/...F/...I   (dP +.., dF -.., dI +..)
  conversation  ...P/...F/...I  ->  ...P/...F/...I   (dP +.., dF -.., dI +..)
  speech        ...P/...F/...I  ->  ...P/...F/...I   (dP +.., dF -.., dI +..)
  reliability   ...P/...F/...I  ->  ...P/...F/...I   (dP +.., dF -.., dI +..)
new failures (PASS on baseline -> FAIL on candidate): N
  - <scenario> / <dimension>
fixed since (FAIL on baseline -> PASS on candidate): N
  + <scenario> / <dimension>
all per-scenario status changes:
  ~ <scenario> / <dimension>: PASS -> FAIL
```

## Exit codes

- **`0`** -- the two releases were compared (per-dimension deltas and
  new-failures / fixed-since printed; a side with no runs is reported as an
  empty state, exit stays `0`).
- **`2`** -- a usage error or an unreadable registry `--registry`.

## See also

- [SUITE-RUN.md](SUITE-RUN.md) -- records the releases this command diffs.
- [CONVERSATION-TEST.md](CONVERSATION-TEST.md) -- the per-dimension scorecard model.
- [WORKSPACE.md](WORKSPACE.md) -- browse the releases and their runs with `hotato serve`.


================================================================================
FILE: docs/WORKSPACE.md
================================================================================

# The team workspace: `hotato serve`

A self-hosted, local web app for a team to read a voice agent's
conversation-QA state: release readiness, the scenario matrix, a conversation
inspector, failure clusters, and production health. It is stdlib-only
(`http.server` + `sqlite3`): self-hosted, no framework or build step, nothing
that phones home. It serves the same fleet registry and evidence store the
CLI writes, reading that data directly instead of passing a database file
around.

```
hotato serve --workspace default
```

On first start it prints where it is listening, the bearer token, and the URL to
open:

```
hotato serve: workspace 'default'
  registry:  /home/you/.hotato/fleet
  listening: http://127.0.0.1:8321
  token:     Ab3xQ_p1…                 (generated, stored 0600 at …/serve/default/token)
  open:      http://127.0.0.1:8321/?token=Ab3xQ_p1…
  audit log: /home/you/.hotato/fleet/serve/default/audit.jsonl   (append-only)
  read-only: the server issues only SELECTs; reviews/labels stay CLI-driven. No telemetry, no external calls.
```

Open the `open:` URL in a browser; the server sets a session cookie and
redirects to strip the token from the address bar (see [Auth](#auth)).

## Flags

| flag | default | meaning |
|---|---|---|
| `--workspace`, `-w` | `default` | workspace id to serve |
| `--host` | `127.0.0.1` | bind address (see [Binding](#binding-127001-by-default)) |
| `--port` | `8321` | listen port |
| `--registry` | `~/.hotato/fleet` | registry home directory |
| `--token` | none | supply the bearer token yourself |
| `--token-file` | none | read the bearer token from a file (first line) |

Exit codes: `0` clean shutdown (Ctrl-C); `2` usage error (unusable registry or
token, or the port was unavailable).

## The five views

Every view has a machine mirror at `?format=json` (same auth, same data) so
agents and scripts can drive the workspace without scraping HTML.

1. **Release readiness** (`/`): the pre-ship home screen. Per-release rollup
   from `suites`/`runs`/`evaluations`: whether required suites are complete,
   scenario and run counts, **failures by dimension** (outcome / policy /
   conversation / speech / reliability), the inconclusive count, the origin split
   (real vs simulated, kept separate), and **new-vs-fixed since the previous
   release** compared per (scenario, dimension). Small samples are flagged
   (`low sample, N=3`), never smoothed.
2. **Scenario matrix** (`/scenarios`): rows are scenarios, columns are the
   current and previous release, with a per-dimension status and **reliability**
   (`pass^k` where a scenario has repetitions). Filter by `agent`, `release`,
   `suite`, and `status` via query parameters (there is a filter form at the top).
3. **Conversation inspector** (`/conversation/<id>`): one conversation. Its
   evidence manifest (`conversation.v1`: origin, provider/caller provenance,
   child digests), transcript, trace spans, per-dimension evaluations with
   rationale and citations (deterministic checks and model-judged/advisory
   results shown in **separate lanes**), and reviewer decisions. Every digest is
   a link to the raw evidence blob (`/evidence/<digest>`) to drill straight to the
   source. Redacted transcript segments and trace spans render as `[redacted]`,
   scrubbed from both the HTML and the JSON mirror.
4. **Failure clusters** (`/clusters`): failed evaluations and assertions grouped
   by **observable signature** (dimension + assertion kind + reason-class), with
   counts and drill-through lists into the inspector. Labelled *clusters by
   observable signature*: it groups what was observed, and the cause stays
   yours to determine.
5. **Production health** (`/health`): ingest counts, evaluated coverage, and
   per-dimension failure rate over time, computed **separately for real and
   simulated** conversations, kept distinct. A day with no evaluated sample
   gets no point plotted, and a dimension with fewer than two days of data
   reads *not enough history*, matching the trend report. Each dimension
   keeps its own number -- there's no single combined quality score.

## Auth

Every request is authenticated against one shared **bearer token**:

- **Browser:** open `/?token=<token>` once; the server mints an in-memory,
  HttpOnly session cookie and redirects to remove the token from the URL.
- **Agent / API / `curl`:** send `Authorization: Bearer <token>`.

The token is compared in constant time (`hmac.compare_digest`). If you do not
pass `--token`/`--token-file`, one is generated with `secrets.token_urlsafe` on
first start and stored `0600` at `<registry>/serve/<workspace>/token`, so a
restart keeps the same URL. Sessions live only in memory, never persisted and
never cross-tenant.

## Audit log

Every request appends one JSONL line to
`<registry>/serve/<workspace>/audit.jsonl` (created `0600`):

```json
{"ts":"2026-07-12T18:04:11Z","who":"Ab3xQ_p1…","method":"GET","path":"/scenarios","query":"status=FAIL","status":200,"remote":"127.0.0.1"}
```

`who` is a token/session **prefix**, never the secret; the `token` parameter is
stripped from the recorded query. The audit log is the **only** file the server
writes.

## Binding 127.0.0.1 by default

The server binds loopback (`127.0.0.1`) unless you explicitly pass `--host`. A
non-loopback bind (for example `--host 0.0.0.0`, to reach the workspace from
another machine) prints a prominent warning, because it exposes the workspace to
your local network. Token auth still applies; an SSH tunnel or a reverse proxy
you control is the tighter choice over binding a wide interface directly.

## Zero egress

The server only opens a **listening** socket: no outbound connection, and
nothing it imports phones home -- audio, traces, and evaluations stay on the
machine. A test whitelists loopback and fails if any view attempts an
external connection, backed by the threat-model row below.


================================================================================
FILE: docs/GUARDIAN-FLEET.md
================================================================================

# Hotato Guardian / Fleet

Guardian/Fleet runs Hotato's capture → scan → trust → label → contract →
trial primitives as one continuous workflow inside a private, self-hosted
control plane: a hardened evidence kernel with a fleet view on top. Every
deploy to production stays a human call.

## The evidence kernel

### An evidence vector, scored dimension by dimension
Every artifact carries a machine-readable **evidence vector**
(`hotato/evidence.py`, schema `evidence_vector.v1.json`): each dimension
scored on its own, the public tier set to the *weakest* tier any required
dimension allows -- a minimum over an inspectable lattice. Tiers, ascending:

| tier | name | meaning |
|---|---|---|
| 0 | none | no usable evidence for a positive claim |
| 1 | asserted | envelope-only; not recomputed from audio |
| 2 | measured | recomputed from audio, one recording, input clean |
| 3 | paired | manifest-bound before/after, both sides recomputed under one pinned policy |
| 4 | attested | paired + signed policy/contract + runner-attested capture |

Dimensions: `score_integrity`, `audio_identity`, `policy_integrity`,
`fixture_set_integrity`, `input_health`, `channel_mapping`,
`label_authority`, `pairing_integrity`, `capture_origin`. One weak dimension
pulls the whole tier down; no renderer may raise it.

### Trial manifest (`hotato/manifest.py`, `trial_manifest.v1.json`)
An immutable pin created *before* an experiment from the battery: the scorer
config hash, one policy (applied to both sides), the complete fixture
universe, and each fixture's expectation, onset, and scripted-stimulus
identity. Before and after must reference the same `manifest_hash`; a
changed policy, scorer, label, onset, or fixture set refuses the comparison.

### Recompute-from-audio (`hotato/recompute.py`)
The proof gate re-derives every verdict from the on-disk audio under the
manifest, checks each stored `verdict.passed` against the recomputed
result, and refuses:

- **verdict tampering** -- a stored verdict that disagrees with the recomputed one;
- **same-audio re-encode** -- before and after decode to the same PCM;
- **dropped fixtures** -- either side omits a pinned fixture;
- **unrelated audio** -- the after-side caller stimulus does not match the pinned one and no capture receipt binds it.

A green *paired* proof additionally requires evidence tier >= paired.

### Capture receipt (`hotato/receipt.py`, `capture_receipt.v1.json`)
A fresh-recapture claim needs more than distinct PCM. A capture runner emits
a receipt binding the recording to a trial, agent, call id, timestamps, and
decoded PCM; an HMAC signature makes it *machine-verified*. A manual
distinct WAV with no receipt is *operator-asserted* -- the receipt is what
promotes it to machine-verified fresh recapture.

### Contract authenticity (`hotato/attest.py`)
Contracts embed a canonical digest over schema + label + policy + audio +
scorer version. `contract verify`/`unpack` recompute it: a bundle edited
after creation (a loosened policy re-pack, say) reports **tampered** and
fails; a matching but unsigned bundle is *"unsigned, internally consistent
evidence"*; a valid HMAC signature promotes it to *authenticated*.

### Boundary sensitivity
Every event exposes `onset_frame_index`, `onset_effective_sec`,
`decision_margin_sec`, `decision_margin_hops`, and `boundary_sensitive`. A
result within one hop of flipping is flagged, so it never reads with the
confidence of a result a hundred milliseconds inside the limit.

### Trust headline
Any verdict-changing warning (low signal, possible channel swap,
VAD-relevant leakage) sets the headline to `scan with caution`.
`input_health` is an explicit three-state field (`clean` / `caution` /
`not_scorable`). Leakage is judged both against a fixed -40 dB bar and
dynamically against the receiving channel's own VAD gate.

## Fleet control plane (local mode)

Local mode is zero-dependency: SQLite metadata plus a content-addressed
artifact directory, both stdlib.

- **Registry** (`hotato/fleet/registry.py`) -- workspace-scoped rows for
  agents, deployments, calls, recordings, candidates, labels, contracts,
  trials, decisions. Agents scale per workspace with no product-level cap;
  every row carries `workspace_id`, keeping paths and call ids scoped to
  their own workspace.
- **Artifact store** (`hotato/fleet/store.py`) -- content-addressed blobs
  with dedup and lineage; audio is stored separately from any UI (privacy
  reversal).
- **Job queue** (`hotato/fleet/jobs.py`) -- leased jobs with deterministic
  idempotency keys, heartbeats, retries, and dead-lettering, so duplicate
  webhooks and worker crashes converge on one logical result.
- **Guardian API** (`hotato/fleet/api.py`) -- ingest → discover (trust
  preflight + scan) → human review + label → manifest-bound before/after
  experiment that **recommends a change for you to apply**, and refuses
  forged / same-audio / incomplete trials.

### CLI

```
hotato fleet init -w acme
hotato fleet agent add -w acme --agent-id support-bot --stack vapi --assistant-id asst_123
hotato fleet ingest -w acme --agent support-bot call.wav
hotato fleet discover -w acme --agent support-bot call.wav
hotato fleet review -w acme
hotato fleet label <candidate-id> --decision yield --reviewer you
hotato fleet experiment run -w acme --agent support-bot --trial-id t1 \
    --battery before/run.json --before before/ --after after/
hotato fleet status -w acme
```

Live clone/recapture/canary run against a connected stack with credentials
and a tested rollback; every release recommends the change and keeps
routing production traffic a manual, human-gated step.

## Synthetic perturbations (`hotato/synth.py`)
Deterministic acoustic transforms of real fixtures -- resample, gain, noise,
leakage, channel invert, silence, onset offset, clip -- each carrying its
parent hash, recipe, seed, and an explicit synthetic designation. Synthetic
and real stay on separate axes: only a real recapture raises the confidence
tier.

## What fleet mode ships today
Evidence stays per-dimension, each one scored on its own lane, never folded
into a single fleet-wide score. Every contract needs human approval before
it exists. Live provider adapters (Vapi/Retell clone→apply→recapture) and
canary routing are gated on connected credentials and a tested rollback --
production deploys stay a manual, credentialed step every time.


================================================================================
FILE: docs/START.md
================================================================================

# Start here: the guided first run

`hotato start --demo` runs the whole hotato loop on one bundled call and
shows you the result: it sweeps the two demo recordings (the same calls
`hotato demo` scores), builds and verifies one failure contract from them,
writes every artifact you need to see the flow, and prints the exact next
commands.

```bash
hotato start --demo
```

## What it writes

Into the current directory (or `--dir DIR`):

- `hotato-sweep.json` -- the sweep result: every candidate timing moment across
  the two demo calls, ranked. This is an `analyze` result, so a `FILE#N` ref
  off it drives `hotato fixture promote` and `hotato card` unchanged.
- `hotato-sweep.html` -- a self-contained dashboard (embedded audio, no external
  assets) that opens in any browser.
- `hotato-no-single-threshold.svg` -- the threshold-funnel card: the demo battery
  misses an interruption **and** false-stops on a backchannel, so no single
  dial fixes both. See `docs/CARDS.md`.
- `contracts/demo-missed-interruption.hotato/` -- one failure contract, built
  from the sweep's missed-interruption candidate with `--expect yield` and
  verified on the spot. See `docs/CONTRACTS.md`.

Everything runs offline: the demo pulls from packaged audio, and the
analyze/card/contract steps run on packaged audio and local files alone.

## The demo failure contract

`start --demo` runs the whole loop once, on one of the bundled recordings --
not just a ranked list of candidates:

```bash
hotato contract create --from-candidate hotato-sweep.json#2 --expect yield \
    --id demo-missed-interruption --out contracts
hotato contract verify contracts/
```

Candidate #2 in the bundled sweep is the missed-interruption call: the agent
talked over the caller instead of yielding. Scored against `--expect yield`,
it fails -- so `start --demo` prints:

```
verified contract: FAIL as expected -- the demo call missed the
interruption; a CI gate on this contract catches any change to the evidence
or policy -- catching the AGENT regressing requires a fresh recapture (see
docs/RECAPTURE.md)
(start --demo itself exits 0 because setup succeeded; run the next command to
see the contract's CI exit 1: hotato contract verify contracts/)
```

`hotato contract verify contracts/` re-scores that same bundled audio and
reports the same regression (exit code `1`); a CI job wired to it fails the
same way if the bundled evidence or policy ever changes. On this frozen
recording, telling whether a currently deployed agent still has the bug
takes a fresh capture -- re-running the same caller stimulus against it and
verifying that; see [`docs/RECAPTURE.md`](RECAPTURE.md). The full two-lane
breakdown is in
[`docs/CONTRACTS.md`](CONTRACTS.md#two-lanes-what-verify-proves-depends-on-which-recording-you-feed-it).
Run it yourself:

```bash
hotato contract verify contracts/
```

## What it prints

The exact next commands, ready to copy:

1. **Save a candidate as a permanent regression test** -- you choose the
   label, hotato leaves that judgment to you:

   ```bash
   hotato fixture promote hotato-sweep.json#1 --expect <yield|hold> \
       --id my-first-fixture --out tests/hotato
   ```

2. **Run your fixtures in CI** (exits non-zero on a regression):

   ```bash
   hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio
   ```

3. **Render a shareable card** from any candidate:

   ```bash
   hotato card hotato-sweep.json#1 --out candidate.svg
   ```

4. **Re-verify the demo failure contract in CI** (or build your own from any
   candidate with `hotato contract create`):

   ```bash
   hotato contract verify contracts/
   ```

## Other modes

`--stack`, `--folder`, and `--stereo` each hand you the shipped command that
does the job:

- `--stack` -> `hotato sweep --stack <stack>` (connect once, sweep your own
  calls).
- `--folder` -> `hotato analyze <folder>` (scan a directory of recordings).
- `--stereo` -> `hotato run --stereo <call.wav>` (score one dual-channel call).

## Output and exit codes

`--format json` emits a machine object (the files written, the next commands,
and a `contract` block with the demo contract's id, expect, scorable, passed,
and `verified_fail_as_expected`) for an agent to drive.

- **0**: the guided first run completed -- including when `--stack`,
  `--folder`, or `--stereo` printed the shipped command to use instead. This
  holds even though the demo contract itself FAILS its policy: `start --demo`
  finishing is a separate claim from the demo contract passing, which
  `hotato contract verify contracts/` reports (exit `1`, by design).
- **2**: usage error -- no mode given, or `--dir` is not a directory.


================================================================================
FILE: docs/INVESTIGATE.md
================================================================================

# `hotato investigate`: one recording in, ranked candidate moments out

Point `hotato investigate` at one recording -- a local dual-channel WAV, or a
live pull from a connected stack by call id -- and it authenticates where the
audio came from, runs the input-health / K6 verdict-eligibility gate, scans
for candidate turn-taking moments, and prints the exact next command that
turns each candidate into a signed, CI-ready contract.

It surfaces candidates and prints the command; the one decision that matters
-- which candidate is a bug, and whether the agent should have *yielded* or
*held* -- stays with you:

```bash
hotato investigate label <candidate_ref> --expect yield|hold
```

Every step reuses a shipped primitive: audio-in is the same per-stack fetch
`hotato pull` uses; the gate is `hotato trust` in contract mode; discovery is
`hotato scan`; the label step wraps `hotato contract create --from-candidate`.

## The label comes from you

`yield` means the agent should stop for the caller; `hold` means the agent
should keep speaking through a backchannel, noise, or acknowledgement.
Hotato measures whether the timing matched your label. `"mhm"` and `"stop"`
can carry identical speech energy -- no timing measurement tells them apart,
so the label is yours to make.

## Scan one recording

```bash
# a local dual-channel WAV you already have
hotato investigate call.wav

# or pull it live from a connected stack by call id
hotato investigate --stack vapi --call-id abc123
```

Give it either a local `SOURCE` path or `--stack STACK --call-id ID`, never
both. Connectable stacks: `vapi`, `twilio`, `retell`, `bland`, `elevenlabs`,
`synthflow`, `millis`, `cartesia`. LiveKit and Pipecat capture in your own
infrastructure -- run `hotato setup --stack <name>` and pass the resulting WAV
as `SOURCE`. `--allow-mono` permits a mono/mixed stack recording, but a
summed channel is degraded and not candidate-eligible for separated scoring.

Useful flags: `--min-gap` (minimum response gap in seconds to surface,
default `2.0`), `--top` (how many top candidates to show and print label
commands for, default `10`; `0` shows all), `--caller-channel` /
`--agent-channel`, and `--state PATH` (default
`.hotato/investigate-state.json`).

### What it reports

```
hotato investigate [run 1]: call.wav
  capture origin: operator-asserted local file (call.wav)
  input health: <trust recommendation>
  verdict path: eligible (a labeled event here can carry a yield/hold verdict)
  N candidate moment(s) (showing N):
    [1] t=42.18s <kind>  <durations>
        label: hotato investigate label .hotato/investigate-state.json#1 --expect yield|hold
  state remembered at: .hotato/investigate-state.json
```

The onset above is illustrative -- candidates are timing facts, ready for
you to label.

## Capture origin: tracked for every run

Every run records where the audio came from, one of three kinds:

- **`frozen_regression`** -- a previously-created hotato fixture clip (a
  sibling scenario file names this exact audio): a pinned regression, played
  back exactly.
- **`provider_pulled`** -- fetched just now from the stack's own recording
  API for a named call id, a stronger claim than an arbitrary file. For the
  stronger, signed, machine-verified claim, see [RECAPTURE.md](RECAPTURE.md).
- **`operator_asserted_local`** -- you handed hotato a local WAV path, taken
  at face value.

## The K6 verdict gate

Trust runs in contract mode -- the same, stricter crosstalk/leakage bar
`hotato contract create` itself checks, since this command exists to produce
a contract. Two outcomes matter:

- **NOT SCORABLE** (mono, identical channels, a silent required channel): no
  candidates are scanned, the report names the reason, and the command exits
  `2`. Fix the input and re-run.
- **Verdict path REFUSED** (a suspected channel swap or crosstalk/leakage):
  candidates still surface as timing facts, and a labeled event carries a
  yield/hold verdict once you confirm the mapping with `--confirm-channels`
  or fix the crosstalk. See [TRUST.md](TRUST.md).

The two gates are different widths: `scan` runs whenever the input is
scorable, so candidates appear even when the verdict path is refused.

## State and candidate refs

State persists to `.hotato/investigate-state.json` (run-numbered, atomically
written, with a history log) in the same shape `hotato analyze` / `hotato
sweep` produce, so it is itself a valid `FILE#N` candidate ref: `hotato
fixture promote` and `hotato contract create --from-candidate` read it
directly. `#1` is the top-ranked candidate, `#2` the next, and so on.

## Label a candidate into a contract

`hotato investigate label` is the label step. Your `--expect` goes straight
to `hotato contract create --from-candidate`, which mints a signed
label-record bound to the exact decoded audio when a signing key is
configured. Without a signing key, the contract's `label_authority` floors
at `asserted`.

```bash
hotato investigate label .hotato/investigate-state.json#1 \
    --expect yield \
    --max-talk-over 0.6 --max-time-to-yield 1.0 \
    --reviewer "$USER" --out contracts
```

This writes `contracts/<id>.hotato/`. The id defaults to a slug derived from
the source, onset, and label; pass `--id` to name it, `--force` to
overwrite. Clipping keeps `--pre` seconds before the onset (default `2.0`)
and `--post` after (default `6.0`); `--no-clip` keeps the full recording. If
the verdict path was refused, add `--confirm-channels` to carry the
contract's verdict through `contract verify`. Source basenames are redacted
from the bundle by default; pass `--include-identifiers` to keep them.

Building a contract, not a bare fixture, is deliberate: it carries the K6
trust block, the CI policy, and the exact `hotato contract verify` command
-- the handoff point from investigate into
[BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md) and [CONTRACTS.md](CONTRACTS.md).

Scanning, the trust gate, and labeling run offline; audio stays on your
machine unless you explicitly pull it from your own stack
([THREAT-MODEL.md](THREAT-MODEL.md)).

## Exit codes

`hotato investigate`:

- **`0`** -- the recording is candidate-eligible: trust + scan ran and
  candidates (if any) were persisted; a yield/hold VERDICT may still be
  refused (K6) -- see `verdict_status`.
- **`2`** -- usage error (neither `SOURCE` nor `--stack/--call-id`, both
  given, a bad channel/`--min-gap` flag, a missing credential, or an
  unreadable state file), or the recording is NOT SCORABLE at all.

`hotato investigate label`:

- **`0`** -- a signed, CI-ready contract was written from this candidate.
- **`2`** -- usage error (a bad `--expect`, a bad candidate ref, an
  unresolved source recording, or an existing contract without `--force`),
  or the candidate turned out not scorable.

## See also

- [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md) -- the bad-call to CI-gate flow this feeds.
- [CONTRACTS.md](CONTRACTS.md) -- the failure-contract bundle.
- [TRUST.md](TRUST.md) -- the input-health / K6 scorability gate.
- [RECAPTURE.md](RECAPTURE.md) -- the stronger, signed fresh-recapture claim.
- [CONVERSATION-TEST.md](CONVERSATION-TEST.md), [SUITE-RUN.md](SUITE-RUN.md) -- the conversation-QA path.


================================================================================
FILE: docs/CONTRACTS.md
================================================================================

# Failure contracts

A failure contract turns ONE call moment into a portable, private,
vendor-neutral bundle: the audio, frame-level timing evidence, an
input-health report, a shareable card, a CI pass/fail policy, and the
exact commands to replay and re-verify it. It is the CI object:
`hotato contract verify` re-scores a directory of contracts and exits
non-zero when one regresses.

A contract's label always comes from a human call (`label_source` is
frozen to `"human"`); Hotato measures whether the recorded timing matched
that label, and `contract verify` re-measures the SAME recording later
and reports pass/fail.

> **A contract bundle contains call audio (`audio/event.wav`).** Do not
> commit a raw customer contract to a public repository -- treat it like
> any other recording of a caller. Use sanitized fixtures (synthetic or
> consent-cleared) for anything public, and put customer contracts in a
> private repository or controlled artifact storage.
> `--include-identifiers` additionally writes a source basename and
> candidate ref into `contract.json` and the card; leave it off by
> default for the same reason. That redaction covers metadata only -- the
> full audio-handling rules are in
> [`SECURITY.md`](../SECURITY.md#audio-handling).

## Two lanes: what `verify` proves depends on which recording you feed it

**Lane A -- `contract verify` on the frozen recording:**

- **What it re-scores:** the SAME `audio/event.wav` the contract was
  created from.
- **What a pass proves:** the evidence, the policy, and the scorer are
  still intact and still agree with the human label.
- **What a pass leaves open:** whether the deployed agent's behavior has
  changed since.
- **When it runs:** every push, in the shipped `ci/github-action.yml`
  (`contract verify contracts/`).

**Lane B -- `contract verify` on a fresh recapture:**

- **What it re-scores:** a new recording of the SAME stimulus against
  your CURRENT agent.
- **What a pass proves:** the CURRENT agent's behavior on that stimulus
  still matches the label.
- **What a pass leaves open:** nothing extra -- this is the lane that
  speaks to the live agent.
- **When it runs:** only when you recapture by hand or on a schedule; see
  [`docs/RECAPTURE.md`](RECAPTURE.md).

A frozen-recording pass is necessary but not sufficient to know the agent
bug stayed fixed: it can only fail if someone edits the bundle's audio or
policy, because the recording never changes. Confirming the CURRENT agent
still yields correctly takes re-running the same caller stimulus against
it and re-verifying the fresh capture, by hand
([`docs/RECAPTURE.md`](RECAPTURE.md)). Run the frozen gate on every push
to catch evidence/policy drift for free, and recapture periodically (or
after an agent change) to catch what only a fresh capture can.

## The bundle

`hotato contract create` writes one self-contained directory, `<id>.hotato/`:

```
refund-cutoff-001.hotato/
  contract.json                      # the contract itself (schema hotato.contract.v1)
  audio/event.wav                    # the (clipped) two-channel recording, or the mono file
  evidence/
    frames.jsonl                     # per-frame timing evidence behind every measurement
    timeline.html                    # the to-scale caller/agent timeline, self-contained
    trust.json                       # the input-health (trust doctor) report
    card.svg                         # a shareable 1200x630 SVG card (redacted by default)
  traces/                            # empty until `hotato trace attach` (see docs/TRACE.md)
  source/
    call_metadata.json               # redacted-by-default: stack, category, expect
    stack_config_snapshot.json       # placeholder until populated by hand
  policy/verify.yaml                 # the SAME subset `hotato verify --policy` reads
  reports/
    initial.html                     # the full scored report at creation time
    after.html                       # placeholder until a fix is re-captured and verified
  provenance.json                    # who/when/how this contract was created
  ci/
    github-action.yml                # a weekly + on-push CI scaffold
    junit.xml                        # this ONE contract's JUnit result at creation time
```

Every path above is also recorded, machine-readable, in
`contract.json["bundle"]["paths"]`.

## Create

From a candidate a sweep or scan already surfaced:

```bash
hotato sweep --demo --format json > hotato-sweep.json
hotato contract create --from-candidate hotato-sweep.json#1 \
    --expect yield --id refund-cutoff-001 --out contracts
```

From a raw two-channel recording you already have:

```bash
hotato contract create --stereo bad-call.wav --onset 42.18 \
    --expect yield --id refund-cutoff-001 --out contracts \
    --max-talk-over 0.6 --max-time-to-yield 1.0
```

Both forms wrap the SAME round-trip guarantee `hotato fixture create`
gives: the moment is scored immediately, and a not-scorable input (the
agent silent at the onset, an unreadable file, a bad channel map) is
refused with a clear reason and exit code 2 -- no bundle is written. A
single-channel (mono) recording passed as `--stereo` is rejected the same
way `fixture create` rejects it: caller and agent cannot be told apart on
one channel.

`--caller FILE --agent FILE` (two mono WAVs) is a third input form,
scored and clipped identically.

### Redaction

By default the bundle and card hide a candidate ref and a source
recording's basename. Pass `--include-identifiers` to show them (in
`source/call_metadata.json`, `contract.json`, and `evidence/card.svg`).

### The opt-in diarized-mono path

A single-channel recording can still become a contract through the SAME
quality-gated diarizer front-end `hotato run --mono --diarize` uses:

```bash
hotato contract create --mono call.wav --diarize \
    --expect yield --id refund-cutoff-002 --out contracts
```

This keeps an indicative-only verdict in its tier: a `low`-confidence
separation tier carries `measurement.indicative_only: true` all the way
into `contract.json` and every renderer, and a `refuse` tier (not two
clean parties, extreme overlap, unstable segmentation, voices too
similar) is refused exactly like a plain mono file, with the specific
reason. This path's evidence lives in `evidence/trust.json`'s separation
confidence tier: `evidence/timeline.html` points there instead of
rendering the frame-level to-scale timeline (`evidence/frames.jsonl`)
that a dual-channel contract carries.

## Verify

```bash
hotato contract verify contracts/
hotato contract verify contracts/ --format json --junit contracts-junit.xml
hotato contract verify contracts/refund-cutoff-001.hotato --html verify.html
```

`DIR` is a contracts directory (every `*.hotato` subdirectory that
carries a `contract.json`) or one bundle directly. For each contract,
`verify` re-scores the SAME bundled audio against the SAME policy
recorded in its own `contract.json` -- this is what changes after an
engine upgrade, a threshold change, or a re-captured recording swapped
into `audio/event.wav` -- and reports pass/fail per contract and overall.

Exit codes are the CI contract: `0` every contract passes, `1` at least
one regressed (or is no longer scorable) OR at least one embedded
assertion deterministically FAILed (reported separately below, but it
still contributes to this nonzero exit), `2` a usage error, an empty
directory, or a corrupt `contract.json`. `--junit` writes one
`<testcase>` per contract; the shipped `ci/github-action.yml` scaffold
runs this on push, on PR, and weekly, and publishes the JUnit file as an
artifact.

Every text and HTML render of `verify` also prints, verbatim: *"This
result re-measures stored evidence. It does not test the current
agent."* -- a green CI run here means the evidence, policy, and scorer
are still intact; checking today's deployed agent takes the
fresh-recapture lane above. See
[`docs/RECAPTURE.md`](RECAPTURE.md#claim-language-what-each-kind-of-evidence-lets-you-accurately-say)
for exactly what each kind of evidence lets you claim.

### Embedded assertions (optional)

A contract can carry its own `assertions` block (schema
`hotato.contract.v1` -- see `schema/contract.v1.json`), the same
`{version, assertions}` document `hotato assert` reads from an
`assertions.yaml` file. When present, `contract verify` evaluates it
through the SAME `assert.v1` engine and reports the result as a
per-contract `assertions` field, kept separate from the timing pass/fail
in `summary`/`passed`. Pass `--transcript FILE` (a plain JSON array of
`{role, text, start, end}` turns, or hotato's own `{"segments": [...]}`
shape) to supply transcript context for `phrase`/`pii`/`policy`
assertions; `tool_call` assertions read the bundle's own attached trace
(`hotato trace attach`) if one exists. Missing context (no `--transcript`,
no attached trace) reports `INCONCLUSIVE`. A deterministic assertion FAIL
contributes to the batch's nonzero exit code exactly like a timing
regression; the batch result also carries a separate `assertions_failed`
count. See [`schema/assert.v1.json`](../src/hotato/schema/assert.v1.json)
for the result shape and `hotato.assert_` for the five deterministic
kinds.

## Inspect

```bash
hotato contract inspect contracts/refund-cutoff-001.hotato
hotato contract inspect contracts/refund-cutoff-001.hotato/contract.json --format json
```

## Pack / unpack

A bundle directory travels as one file:

```bash
hotato contract pack contracts/refund-cutoff-001.hotato
# -> contracts/refund-cutoff-001.hotato.pack (deterministic; a MANIFEST.sha256.json
#    of every member travels inside it)

hotato contract unpack contracts/refund-cutoff-001.hotato.pack \
    --out contracts/refund-cutoff-001.hotato
# every member is verified against the packed sha256 manifest; any mismatch
# (a corrupt or tampered archive) is refused (exit 2) and nothing partial is
# left behind
```

Packing the SAME bundle directory twice produces byte-identical archives
(sorted member order, fixed timestamps, and every other value written
into each member's ZipInfo, including its `create_system` byte): a
contract's pack is a pure function of its bundle contents, deterministic
for a fixed hotato version. Byte-identical re-runs are verified in CI on
Linux x86_64, Python 3.10, 3.11, and 3.12 -- see
[VALIDATION.md](VALIDATION.md) Job 1.

### Security: unpack treats an archive as hostile input

A `.hotato` archive is meant to be sent between teams, so `contract
unpack` verifies everything about it before trusting a single byte.
Before or during extraction it refuses (exit 2) on any of the following,
writing only inside a scratch temp directory that's removed on any
failure:

* path traversal (`..`), absolute paths, and Windows-style backslash /
  drive-letter paths (`C:\...`) in a member name;
* symlink members and encrypted members;
* duplicate member names;
* any member the archive carries that its own `MANIFEST.sha256.json` does
  not declare;
* more members than a legitimate bundle could plausibly need;
* a declared or measured decompressed size past the cap (default 512 MiB,
  set `HOTATO_CONTRACT_MAX_UNPACK_BYTES` or pass `--max-bytes` to raise it
  for a trusted archive) -- checked against the bytes measured live during
  extraction, beyond what the archive's own (untrusted) size metadata
  claims;
* a single member whose compression ratio is far beyond anything a
  legitimate bundle member produces (a zip-bomb signal), even under the
  total-bytes cap.

The sha256-per-member check (above) still runs on top of all of this: a
member can pass every hardening check and still be refused for not
matching the packed manifest.

## CI

The shipped `ci/github-action.yml` is the minimal wiring:

```bash
uvx hotato contract verify contracts/ --junit contracts-junit.xml \
    --format json > contracts-verify.json
```

Gate a pull request or a scheduled job on the process exit code; do not
parse stdout to decide pass or fail.

## What a contract proves, precisely

Hotato proves timing behavior against this explicit contract -- not
authorization, identity, compliance, or policy safety. `verify` reports
coincidence, not causation: a passing re-verify after a config change
coincides with the change, distinct from a controlled experiment. See
`docs/VALIDATION.md` and `docs/THREAT-MODEL.md`.

## Read more

- Proving the CURRENT agent, not just the frozen recording:
  [`docs/RECAPTURE.md`](RECAPTURE.md)
- The underlying regression-fixture primitive `contract create` wraps:
  [`docs/BAD-CALL-TO-CI.md`](BAD-CALL-TO-CI.md)
- Battery-scale before/after proof: [`docs/FIX-LOOP.md`](FIX-LOOP.md)
- Input-health (trust doctor): [`docs/TRUST.md`](TRUST.md) ·
  [`docs/TRUST-MATRIX.md`](TRUST-MATRIX.md)
- Diarized-mono scoring: [`docs/DIARIZE.md`](DIARIZE.md)
- Shareable cards: [`docs/CARDS.md`](CARDS.md)
- Attaching a voice trace (observability bridge): [`docs/TRACE.md`](TRACE.md) ·
  [`docs/OTEL.md`](OTEL.md)
- Audio-handling rules (what raw audio may contain, redaction limits,
  distribution): [`SECURITY.md`](../SECURITY.md#audio-handling)


================================================================================
FILE: docs/COUNTEREXAMPLES.md
================================================================================

# Counterexamples: reduce one scripted failure to a verified repro

`hotato counterexample` takes one failing deterministic scripted scenario,
deletes inputs that are unnecessary to that exact failure, and writes a local
`.hotato-repro` directory.

```text
scenario + conversation test + target assertion
                    |
                    v
       hotato counterexample compile
                    |
                    v
  private runnable capsule + deletion certificate
```

Every accepted deletion is evaluated through Hotato's existing scripted
simulator and deterministic assertion engine. A candidate is retained only
when the same assertion still contains the source-selected structured failure
branch and therefore retains the same failure fingerprint.

The v1 boundary is narrow:

- input is one local `hotato.scenario` plus one local
  `hotato.conversation-test`;
- the target is one assertion in `assertions.deterministic`;
- the scripted simulator supplies the transcript, trace, mock tools, and mock
  state;
- transformations only delete units named by `hotato.reducers.v1`;
- model-judged targets, provider sessions, captured recordings, external
  timing bundles, DTMF targets, and custom policy-pack paths are refused.

The scripted proof lane covers 15 deterministic assertion kinds through 48
closed, schema-coupled failure branches: `phrase`, `pii`, `policy`,
`tool_call`, `outcome`, `tool_result`, `tool_error`, `state`, `state_change`,
`handoff`, `termination`, `latency`, `entity_accuracy`, `sequence`, and
`count`.

The compiler does not invoke a voice agent, provider, model, TTS service, STT
service, command oracle, or network endpoint. Its result describes the
scripted fixture under the recorded Hotato evaluator source identity. It does
not establish the behavior of a deployed agent.

## Run the example

The repository example starts with three caller turns, two mock tools, and two
state records. The selected assertion fails because order `A-1` remains
`pending` instead of `posted`.

```bash
sh examples/counterexample/run.sh
```

The script compiles a private capsule in a new temporary directory, verifies
the proof, reproduces the reduced fixture, checks predicate semantics, and
exports a non-runnable share-safe projection. It leaves both directories in
place and prints their paths.

Run the steps directly:

```bash
hotato counterexample compile \
  --scenario examples/counterexample/refund-not-posted.scenario.json \
  --test examples/counterexample/refund-not-posted.test.json \
  --target refund-posted \
  --workspace examples/counterexample \
  --out /tmp/refund-not-posted.hotato-repro

hotato counterexample verify /tmp/refund-not-posted.hotato-repro
hotato counterexample reproduce /tmp/refund-not-posted.hotato-repro
```

`--out` must name a path that does not exist. Compilation builds a sibling
temporary directory and promotes it with an operating-system no-replace
primitive. Linux, macOS, and Windows have explicit implementations; an
unsupported platform refuses instead of falling back to an overwriting move.

## Source and target selection

### One base scenario

The compiler evaluates the scenario document passed to `--scenario` directly.
It does not expand `variation_matrix` and does not select a matrix row.
`capsule.json` records:

```json
{
  "scenario_selection": {
    "mode": "base-scenario",
    "variation_matrix_applied": false
  }
}
```

`--seed` selects the scripted replay seed. Its default is `scenario.seed`, or
`0` when the scenario has no seed. A seed does not select a variation-matrix
combination.

To minimize one generated variation, first materialize that variation as its
own complete scenario document, then pass the concrete file to `compile`.
Leaving a matrix in the source only preserves its bytes in source provenance;
the reducer can delete the matrix when it has no effect on the base-scenario
failure.

### One deterministic assertion

`--target` must match exactly one assertion ID in
`assertions.deterministic`. Compilation first confirms twice that the selected
assertion fails with identical result, content, and trace identities. It
refuses a passing, inconclusive, advisory, missing, duplicated, or unsupported
target.

### Exact failure branch identity

A deterministic `FAIL` can contain more than one reason. The oracle maps those
reasons into a closed set of payload-free failure atoms, sorts and de-duplicates
them canonically, and selects the first source atom as the preservation anchor.
The private capsule records both that selected atom and the complete ordered
source atom set. A candidate is `PRESERVED` only when the selected source atom
is still present in the candidate's atom set.

An atom carries only the discriminator needed to keep failure branches apart:
for example, an assertion index, detector, policy rule/type, state field, or
entity key. A wrong tool argument therefore becomes
`tool-argument-value-mismatch` plus its key. Deleting that value changes the
branch to `tool-argument-field-missing`; deleting the named call changes it to
`tool-missing`. Either candidate is `DRIFTED`, even though the assertion still
reports `FAIL`. Results and termination attributes use the same
missing-versus-value distinction. Required-order and sequence assertions keep
present-but-out-of-order evidence separate from an absent step. Latency atoms
distinguish a declared mock measurement from the simulator default. Payload
values, transcript text, and broad diagnostic counts are not part of an atom.

The fingerprint binds the test and assertion identity, assertion bytes, kind,
dimension, deterministic authority, required `FAIL` status, and selected
source atom. It does not bind every diagnostic field in the assertion result.
This keeps irrelevant evidence reducible without allowing the preserved
failure to collapse into a different branch.

The emitted `input/conversation-test.json` is a canonical projection containing
only the selected deterministic assertion. Other source assertions and the
rubric lane are outside the preservation oracle. Their removal is normalization
and is not evidence that those checks passed.

### Workspace boundary

`--workspace` is the directory both input files must resolve within. Input
files and path components must be regular local paths without symlink escape.
When omitted, the workspace is the common parent of the scenario and test
files. Set it explicitly in automation so the filesystem boundary is visible
in review.

## What the compiler deletes

`hotato.reducers.v1` is a closed, versioned, deletion-only algebra over the
scripted scenario. It can attempt to remove:

- variation-matrix, facts, and environment entries;
- caller behavior and interruption declarations;
- caller turns while retaining at least one turn;
- mock tool calls, optional tool fields, and nested argument/result entries;
- handoff and termination records;
- mock-state resources, rows, snapshots, and nested leaves;
- optional additive scenario fields.

It cannot add content, replace a scalar, reorder a list, rewrite speech, or
change the target assertion. `certificate.json` records the digest-bound
delete-only transform for every accepted step. `reduction.jsonl` records every
candidate attempt.

## The 1-minimal claim

A capsule with:

```json
{"minimality": {"status": "one_minimal"}}
```

supports this exact statement:

> The fixture is 1-minimal under `hotato.reducers.v1` with the recorded
> observation-scope freezes.

After hierarchical reduction, the compiler repeatedly tries every remaining
single-unit deletion. It reaches `one_minimal` only when none of those
deletions preserves the selected source failure branch. `verify` recomputes the
single-unit check instead of trusting `minimality.json`.

This is a local claim over the named deletion algebra. It is not a global
minimum, a root-cause determination, or a semantic-equivalence claim. A
remaining deletion can make the target `PASS`, change the selected failure
branch, invalidate the scenario, or make required evidence unavailable. All
of those outcomes mean that deletion did not preserve the recorded exact
failure identity; the individual outcome remains recorded in
`minimality.json`.

## Budgets

`--budget` limits uncached candidate evaluations during reduction:

```bash
hotato counterexample compile ... --budget 512
```

- default: `512`;
- minimum: `1`;
- hard maximum: `100000`;
- cache hits do not consume the budget;
- the two source and two final qualification executions are reported
  separately and do not consume the candidate budget.

When the budget ends before the final deletion pass completes, the compiler
still emits a capsule if the reduced fixture preserves the exact failure on
both final executions. Its status is `budget_exhausted`, and its permitted
claim is limited to failure preservation. It carries no 1-minimal claim.

Compilation exits `0` for `one_minimal` and `1` for `budget_exhausted`, while
writing a complete capsule in either case. Automation can require a completed
proof from the process status without parsing a nested field. `verify` can
verify the integrity and preserved-failure evidence of a
`budget_exhausted` capsule, but it does not upgrade that capsule to
`one_minimal`.

## Capsule contents

A private runnable capsule contains:

```text
case.hotato-repro/
  capsule.json
  oracle.json
  certificate.json
  minimality.json
  reduction.jsonl
  source/
    scenario.json
    scenario.original
    conversation-test.json
    conversation-test.original
  input/
    scenario.json
    conversation-test.json
  expected/assertion-result.json
  report.md
  report.html
  card.svg
  reproduce.sh
  predicate.sh
  MANIFEST.sha256.json
```

`MANIFEST.sha256.json` binds every other file. The capsule ID also binds the
proof-relevant documents and their digests. Strict verification rejects
missing, changed, extra, symlinked, or unsafe members and independently
re-renders `report.md`, `report.html`, `card.svg`, `reproduce.sh`, and
`predicate.sh` before accepting their bytes.

The Markdown, HTML, and SVG reports summarize identifiers, proof status, and
reduction counts. They are bound, reproducible projections of the capsule;
they do not replace verification.

### Resource and local-filesystem boundary

Verification refuses a capsule before replay when it exceeds any of these
limits:

- 1,024 files;
- 4,096 directories;
- 64 directory levels;
- 64 MiB for one member;
- 256 MiB across all members.

Each source JSON file is limited to 16 MiB and 96 JSON levels. The scripted
scenario also caps caller turns and mock tools at 10,000 each.

Counterexample compilation and replay add a narrower proof-lane budget:

- the selected assertion is at most 256 KiB in canonical JSON;
- a candidate scenario is at most 2 MiB in canonical JSON;
- rendered transcript text is at most 256 KiB of UTF-8;
- one deterministic assertion result is at most 2 MiB in canonical JSON;
- `hits` and `matched_rules` evidence collections are capped at 10,000 rows;
- an accepted proof chain contains at most 512 deletion steps and one step
  contains at most 10,000 deletion operations;
- a completed minimality proof contains at most 512 remaining deletion units;
- each proof-lane regex is at most 1,024 UTF-8 bytes and belongs to a closed,
  fixed-width replay subset: groups, alternation, backreferences, and variable
  quantifiers are refused.

The assertion and regex byte bounds run before the general assertion validator
can invoke Python's regex parser.

The selected structured failure branch is still the preservation identity;
these limits bound the repeated work needed to establish it. A candidate that
crosses an execution limit is `UNRESOLVED` with
`resource_limit_exceeded`, never treated as preserved or absent. These are
counterexample compiler/replay limits. They do not narrow the regex or result
surface of Hotato's deterministic evaluator outside the proof lane.

The capsule directory must remain unchanged for the duration of `verify`,
`reproduce`, `inspect`, or `export`. Persistent mutation, a root replacement,
and symlink or special-file substitution are detected. A privileged local
process that can swap bytes between individual reads and restore them before
the final manifest pass is outside this version's proof snapshot. Copy an
untrusted capsule into a private, non-writable workspace before verification.

### What the journal proves

`certificate.json` and the `PRESERVED` rows in `reduction.jsonl` carry the
accepted delete-only chain. Strict verification reconstructs those transforms
and reruns the final single-unit deletion inventory for a `one_minimal` claim.
`ABSENT`, `DRIFTED`, and `UNRESOLVED` journal rows are diagnostic reduction
history. Their candidate payloads are not part of the accepted proof chain and
are not independently replayed by `verify`.

## Verify, reproduce, and inspect

These commands answer different questions.

| Command | Question | Evaluator rule | Work performed |
|---|---|---|---|
| `verify` | Is this the same intact proof produced by the recorded evaluator source? | Recorded package version and evaluator source digest must match the installed implementation. | Replays source and final cases twice, reconstructs every accepted deletion, checks the selected source failure branch and expected result, re-renders derived artifacts, and recomputes claimed single-unit minimality. |
| `reproduce` | Does the reduced fixture still produce the selected source failure branch under the installed evaluator? | Evaluator drift is allowed and reported. | Checks capsule integrity and the delete-only chain, then runs the reduced fixture twice. It does not reassert historical intermediate verdicts or renew the original minimality proof under changed code. |
| `inspect` | What does the intact capsule claim? | No evaluator execution. | Checks the closed member inventory, source/oracle/artifact bindings, canonical human files, manifest, and capsule schema, then prints target, reduction, minimality, preservation, and profile metadata. |

Use `verify` for proof audit and artifact integrity with the recorded Hotato
package version and evaluator source closure. The evaluator digest does not
bind the Python interpreter build, operating system, CPU, or native libraries.
Strict replay still compares the recorded result, content, and trace hashes,
so a runtime difference that changes evaluator behavior fails verification.
Use `reproduce` when evaluating the frozen reduced fixture after Hotato
evaluator or simulator code changes.

`reproduce` returning `0` means the failure was reproduced. That success code
answers the reproduction question; it is the inverse of a conventional
quality gate. Use `predicate` or the generated `predicate.sh` when the desired
shell result is “failure present means bad.”

## Private and share-safe forms

Compilation writes `private-runnable-v1`. It contains complete source and
reduced scenario/test inputs, including scripted speech and mock tool/state
payloads. Treat the directory as sensitive even when its origin is simulated.
The compiler applies restrictive POSIX modes where supported, but permissions
do not establish publication rights or de-identification.

Create a payload-free projection only after the private capsule verifies:

```bash
hotato counterexample export /tmp/refund-not-posted.hotato-repro \
  --profile share-safe-v1 \
  --out /tmp/refund-not-posted.share-safe
```

The exported directory has one closed inventory: `capsule.json`, `report.md`,
`report.html`, `card.svg`, `README.md`, and `MANIFEST.sha256.json`. Extra,
missing, renamed, symlinked, or special-file members are refused even when a
manifest declares them. `inspect` re-renders every human-facing file and
requires byte-for-byte equality with its canonical rendering, so rebinding
modified report text into a new manifest does not make the projection valid.

Those canonical renderer bytes are part of the capsule v1 exchange contract.
A compatible implementation must retain the v1 renderer; changing its output
requires a versioned renderer/profile or a capsule format version bump.

The projection omits runnable inputs, scenario and assertion bodies,
transcript or audio content, tool payloads, state values, credentials, provider
identifiers, absolute paths, and source-derived reducer paths. Private
single-unit rows become an aggregate count by outcome; the projection contains
no reducer paths or per-candidate digests.

The share target and canonical reports expose the selected failure code, such
as `tool-argument-value-mismatch` or `state-field-value-mismatch`, so the
failure category remains readable without a payload. Atom discriminators such
as a field, key, rule, detector, or index stay private.
`failure_atom_digest` binds the complete selected atom without publishing those
discriminator values.

`share-safe-v1` is an engineering access boundary, not an anonymity claim.
Capsule, source, assertion, evaluator, fingerprint, and failure-atom digests
remain correlators. Low-entropy or externally known source material can be
tested against those values. Keep a projection within the same review,
artifact-retention, and disclosure controls used for engineering metadata.

The projection cannot reproduce the failure. Keep the private capsule when a
reviewer, CI job, or coding agent must execute the fixture.

## Exit codes

### `compile`

| Exit | Meaning |
|---:|---|
| `0` | An atomic private capsule was written and earned `one_minimal`. |
| `1` | An atomic private capsule was written with the exact failure preserved, but the candidate budget ended before one-minimality was proved. |
| `2` | Refused: input, target, workspace, output, replay, or proof safety requirements were not met. No completed destination is left. |

### `verify`

| Exit | Meaning |
|---:|---|
| `0` | Integrity, provenance, source/final replay, accepted deletion chain, exact failure identity, and the stated minimality level verified. |
| `1` | The capsule is intact, but the target failure or claimed single-unit minimality no longer reproduces. |
| `2` | No proof verdict: malformed, tampered, unsafe, evaluator-incompatible, unsupported, or inconclusive capsule. |

### `reproduce`

| Exit | Meaning |
|---:|---|
| `0` | The selected source failure branch reproduced twice under the installed evaluator. |
| `1` | The capsule is intact, but that exact failure is absent. |
| `2` | No reproduction verdict: malformed, tampered, unsafe, unsupported, disagreeing, or inconclusive capsule. |

`inspect` and `export` return `0` on success and `2` on refusal.

### `predicate`

`hotato counterexample predicate DIR` and `DIR/predicate.sh` map reproduction
to `git bisect run` semantics:

| Exit | Meaning |
|---:|---|
| `1` | Bad: the exact target failure is present. |
| `0` | Good: the exact target failure is absent. |
| `125` | Skip: the fixture is unusable, unsafe, unsupported, disagreeing, or inconclusive at this revision. |

## CI and evaluator bisect boundaries

### Recorded-evaluator proof audit

Use `verify` when CI installs the Hotato evaluator recorded in the capsule:

```bash
hotato counterexample verify tests/repros/refund-not-posted.hotato-repro
```

This detects capsule mutation and evaluator-source drift. A changed evaluator
digest is a refusal (`2`), because a different implementation cannot silently
validate the historical proof. Interpreter and platform identity are not part
of that digest; the replayed result, content, and trace hashes remain the
behavioral check across runtimes.

### Current-evaluator regression

Use the generated predicate when a reproduced failure should fail the job:

```bash
tests/repros/refund-not-posted.hotato-repro/predicate.sh
```

The predicate executes the frozen reduced mock scenario. It does not call an
application's current voice agent, prompt, tools, provider, or telephony path.
V1 therefore gates Hotato evaluator/simulator behavior over the capsule; it is
not an end-to-end agent release gate.

### `git bisect run`

```bash
git bisect start BAD_REVISION GOOD_REVISION
git bisect run /absolute/path/to/refund-not-posted.hotato-repro/predicate.sh
```

This can locate the revision where Hotato evaluator or scripted-simulator
behavior begins reproducing the recorded failure. The capsule retains its own
source and reduced inputs, so changes to external scenario/test files are not
part of the predicate. A revision that cannot load or evaluate the capsule is
skipped with `125`. As with `git bisect run` generally, skipped revisions near
the transition can leave a candidate range instead of identifying one first
bad revision.

Bisecting agent implementations, hosted models, provider configurations, or
prompt revisions requires a separately defined adapter that executes those
revisions and emits the evidence consumed by an assertion. That adapter is
outside counterexample v1.

## Refusals worth designing around

The compiler fails closed when:

- the target is in the rubric lane or does not fail twice identically;
- required evidence is missing and the target is inconclusive;
- a `timing_contract` depends on an external bundle;
- a DTMF assertion requests trace evidence the scripted simulator cannot emit;
- an outcome predicate reads an external timing field;
- a latency assertion reads an external timing field or a span type other than
  a scripted `tool_call`;
- a keyed tool argument/result target uses an empty key, or an entity reference
  uses an empty key or a null expected value;
- a policy assertion uses an external `pack_path`;
- an input escapes the workspace or traverses a symlink;
- the output path already exists;
- source or final replays disagree;
- a capsule member, manifest, certificate, or evaluator provenance is invalid.

Refusal is evidence that the compiler cannot make its promised statement for
that input. It is never converted into a passing or failing agent verdict.

## Machine output

Add `--format json` to `compile`, `verify`, `reproduce`, `inspect`, or `export`.
The JSON carries the command kind, exit code, capsule/failure identities, and
the command-specific proof or reduction fields. `predicate` communicates only
through its process exit.

See the complete runnable fixture in
[`examples/counterexample/`](../examples/counterexample/).


================================================================================
FILE: docs/PYTEST.md
================================================================================

# Pytest: the fixture and the gate

Install hotato and the plugin registers itself: a standard `pytest11` entry
point, zero `conftest.py` wiring, zero imports. It adds one fixture and one
opt-in session gate, both scoring with the same engine as the CLI and
returning the same envelope -- the JSON result object every hotato surface
emits.

```bash
# zero-install with uvx, or: pipx install hotato
uvx hotato --help
```

## The `hotato_score` fixture

Score a recording or a suite inside any test and assert on the envelope --
the same inputs the CLI takes:

```python
def test_call_yields(hotato_score):
    env = hotato_score(stereo="call.wav", expect="yield")
    assert env["exit_code"] == 0
    assert all(e["verdict"]["passed"] for e in env["events"])

def test_split_channels(hotato_score):
    env = hotato_score(caller="caller.wav", agent="agent.wav", expect="yield")
    assert env["events"][0]["verdict"]["passed"]

def test_selftest_battery(hotato_score):
    env = hotato_score(suite="barge-in")
    assert env["exit_code"] == 0
```

The envelope is the standard machine shape (`schema_version` "1"): per event
a `verdict` (`passed`, `did_yield`, `seconds_to_yield`, `talk_over_sec`), a
`signals` bus, a `fix` on every failure, and the `limits` block. Assert on
whichever measurement your test cares about.

## The `--hotato-suite` session gate

Opt in on the command line and the battery runs after your tests, printing a
summary and failing the whole session (exit 1) on a regression:

```bash
pytest --hotato-suite                    # default suite: barge-in
```

The bundled battery is a self-test of the harness. Point the same flag at
your own labelled scenario and audio directories to gate on your own agent:

```bash
pytest --hotato-suite \
  --hotato-suite-scenarios corpus/suites/gold/scenarios \
  --hotato-suite-audio corpus/suites/gold/audio
```

Any directory in the same scenario shape works, including your own labelled
calls -- see `docs/SUITES.md` for the bundled tiers and `docs/SUBMITTING.md`
for building labelled fixtures from your own recordings.

## Where it fits

- The gate is one flag on a test run you already have: turn-taking rides
  along with every `pytest` invocation, locally and in CI.
- The GitHub workflow (`docs/CI.md`) is the other ready-made gate -- it
  scores every pull request and posts a sticky results comment. Use either,
  or both.
- The plugin runs offline, and its fixtures are deterministic and
  self-contained, so results stay stable run to run.


================================================================================
FILE: docs/SUITES.md
================================================================================

# Corpus suites: tiered, deterministic, explicit about what they prove

`corpus/suites/` ships four labelled scenario suites, 112 scenarios total,
every one synthetic shaped noise rendered deterministically from its own
labelled timings (seed `sha256(scenario_id)`). The timings are the ground
truth. Synthetic audio is the floor: these suites prove the scorer runs end
to end and catch regressions. Validity on your system comes from your own
labelled calls ([`docs/SUBMITTING.md`](SUBMITTING.md)).

## The four suites

| Suite | Scenarios | Conditions | Expected exit |
| --- | --- | --- | --- |
| `silver` | 40 | clean, 16 kHz, default noise floor | `0` |
| `silver-defects` | 16 | clean conditions, deliberate defect renders | `1` |
| `gold` | 40 | hard conditions: noise floors, 8 kHz, gain extremes, echo, edge timings, endurance | `0` |
| `gold-defects` | 16 | hard-condition defect renders, plus two labelled capture-defect cases | `1` |

Scenario families span what shows up on a live call: hard interruptions
(onset, speed, duration, resume), backchannels, short acknowledgements like
"mhm" the agent should talk through (varied in position, density, repeats,
length), double-talk, one-word interruptions, stutter onsets, multi-turn
exchanges, resume-then-reinterrupt, and latency prompts.
`corpus/suites/manifest.json` is the machine-readable inventory: per suite
the family and category breakdown, sample rates, and expected exit code.

## Defect suites fail by design

Every `silver-defects` / `gold-defects` scenario is rendered to fail on its
labelled axis: an agent that keeps talking through a true interruption, or
stops for a backchannel, or misses its latency budget. A defect suite that
exits `1` is the scorer catching what it claims to; a defect suite that
exits `0` would be a bug. This is the negative control the positive suites
need to mean anything.

## Run a suite

A suite is just a scenarios directory plus an audio directory:

```bash
hotato run --suite barge-in \
  --scenarios corpus/suites/gold/scenarios \
  --audio corpus/suites/gold/audio
```

The same directories plug into every other surface: `hotato report ... --out
report.html` for the visual report, `hotato export ... --out research/` for
CSVs, and `pytest --hotato-suite --hotato-suite-scenarios ... --hotato-suite-audio ...`
for the session gate ([`docs/PYTEST.md`](PYTEST.md)).

## Deterministic builder

The suites are generator-built end to end, and the generator is the proof:

```bash
python3 corpus/suites/build_suites.py           # rebuild in place
python3 corpus/suites/build_suites.py --check   # regenerate to a temp dir, byte-compare
```

`--check` regenerating byte-identical output is the reproducibility
guarantee: the audio on disk is exactly what the labelled timings say,
deterministic for a fixed hotato version (byte-identical re-runs verified in
CI on Linux x86_64, Python 3.10, 3.11, 3.12 -- see
`.github/workflows/tests.yml`, job `determinism`). CI can run it as a drift
gate.

## Additive scenario classes

`corpus/classes/` ships four small, deterministic classes on top of the
suites above, built the same way (synthetic shaped noise, seed
`sha256(scenario_id)`, `--check` byte-compares a rebuild):

- `mid-utterance-pause` -- a multi-second thinking gap mid-turn, distinct
  from a true turn end.
- `backchannel-multilingual` -- short non-English acknowledgement tokens,
  since Hotato's energy-based VAD treats them the same as English ones.
- `noise-hold` -- sustained background presence, distinct from a brief
  backchannel, that the agent should hold through.
- `telephony-degraded` -- an existing gold scenario re-rendered through
  G.711 mu-law plus a fixed packet-loss schedule, to prove the verdict is
  stable across codec degradation.

Kept separate from `corpus/suites/` because `mid-utterance-pause` needs a
non-default `turn_end_silence_sec` beyond what the generic suite tests
apply. Full per-class detail: `corpus/classes/README.md`.

## Against a live stack

To run scenario audio through a live voice stack and score what comes back,
see [`docs/BENCHMARK-STACKS.md`](BENCHMARK-STACKS.md). Measurement-error
methodology for the scorer itself is in [`docs/BENCHMARK.md`](BENCHMARK.md).

## Contribute scenarios

New synthetic families are welcome through the normal PR path
(`CONTRIBUTING.md`). The highest-value contribution stays a consented,
labelled call from your own traffic: the walkthrough is
[`docs/SUBMITTING.md`](SUBMITTING.md) and the
[corpus-submission issue form](https://github.com/attenlabs/hotato/issues/new?template=corpus_submission.yml)
opens the intake.


================================================================================
FILE: docs/CI.md
================================================================================

# CI: gate a PR on turn-taking

hotato scores turn-taking timing from call recordings, so a pull request
carries a running score and fails when the agent gets slower to stop
talking for an interrupting caller. The workflow runs offline with zero
extra dependencies.

## The root Action: run a committed suite from any repository

The repository root ships a composite GitHub Action, so a repository with
no hotato source can run a committed suite, conversation test, or contract
verification and gate on hotato's exit status. The default run is offline:
it runs the pinned Action revision itself off PYTHONPATH (no pip, no
package index), installs no model, no ASR, no Node tool, and calls no
external judge. No secret is read or needed.

The Action ships as a composite Action from release v1.4.0 onward. Adopt
the current release (v1.6.1) and pin the revision you adopt by its full
commit SHA; resolve a tag to its SHA first:

```bash
git ls-remote https://github.com/attenlabs/hotato refs/tags/v1.6.1
```

Then commit this workflow (replace the `attenlabs/hotato` pin with the SHA
the command printed):

```yaml
name: conversation QA

on:
  pull_request:

permissions:
  contents: read

jobs:
  conversation-qa:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
      - id: hotato
        # Pin by full commit SHA (immutable); the comment names the release.
        uses: attenlabs/hotato@<full-commit-sha>  # v1.6.1
        with:
          suite: tests/voice/qa.suite.json
          agent: support-agent
      # Artifact upload is an explicit consumer step, pinned by full commit
      # SHA; the Action itself never uploads, comments, or notifies.
      - if: always()
        uses: actions/upload-artifact@834a144ee995460fba8ed112a2fc961b36a5ec5a # v4.3.6
        with:
          name: hotato-conversation-qa
          path: ${{ steps.hotato.outputs.output }}
          if-no-files-found: error
```

`permissions: contents: read` is sufficient. The Action never posts a
comment, uploads an artifact, or sends a notification; retention and
publication stay under the consumer workflow's control, as above.

### Inputs

Exactly one of `suite`, `test`, `contracts` selects what runs; each is a
workspace-relative committed path. `agent` names the agent under test for
suite and test runs. Everything else is optional: `release` (defaults to
the commit SHA), `output` (defaults to `.hotato/results`), `parallel`
(suite worker cap; never changes result bytes), `transcript` / `trace` /
`state` (evidence files for a test run), `gate-advisory` (test runs only;
passes hotato's own `--gate-judge` so the model-judged rubric lane gates),
and `hotato-version`.

`hotato-version` pins the install to one of three forms:

- `action` (default): install the pinned Action revision itself, with
  `--no-deps` and no package-index egress. The code that runs is exactly
  the revision your workflow pinned.
- an exact version such as `1.6.1`: `pip install --no-deps hotato==1.6.1`.
- `preinstalled`: skip installation (hotato is already on the runner).

A range or `latest` is refused, so the pin always names one exact
revision. Suite policy lives in the committed suite file; the Action never
overrides a suite's `inconclusive_policy`.

### Outputs and the exit contract

The step exit IS hotato's exit code, so the job gates without any extra
step:

| Exit | Meaning |
|---|---|
| 0 | every deterministic check passed under the committed policy |
| 1 | a deterministic check or a `success.required` condition failed |
| 2 | the verdict was withheld (refuse policy), or a usage/validation error |

The machine JSON stays primary. Outputs: `output` (results directory),
`suite-result` (the machine result JSON path), `summary` (the rendered
Markdown), `records` (Failure Record directory when produced, else empty),
`exit-code` (as a string), `status` (`pass`, `fail`, `inconclusive`, or
`error`, read from the machine result; presentation only), and
`hotato-version` (the executed package version).

The five-lane job summary (Outcome, Policy, Conversation, Speech,
Reliability) is appended to the job page on pass AND on failure, with the
exact reproduce command and the evaluated check ids. A lane with no
evaluated check renders NOT_RUN; a lane whose checks lack required
evidence renders INCONCLUSIVE, never PASS. The model-judged rubric lane
reports in its own advisory section with `gate enabled: true|false` and
changes the exit only when the run opted in with `gate-advisory: true`;
when no local judge is reachable it reports ERROR, holding the verdict
rather than guessing at one.

The conformance fixture for all of this is
[`tests/fixtures/action-consumer/`](../tests/fixtures/action-consumer/),
and `.github/workflows/tests.yml` runs the same consumer shape against the
local checkout on every pull request (job `action-smoke`).

## Drop it in

Copy [`.github/workflows/hotato.yml`](../.github/workflows/hotato.yml) into
your repository at the same path. That is the whole setup. On the next
pull request it will:

- install the package with `pip install .`
- score the bundled barge-in suite to a JSON envelope
- render a Markdown summary with [`scripts/pr_comment.py`](../scripts/pr_comment.py)
- post or update one sticky comment (found by a hidden marker, so it stays
  a single comment across runs)
- fail the job on any regression

The sticky comment shows a pass/fail line, a per scenario table (expect,
yielded, time to yield, talk over, result), and a short regressions
section.

The job needs `pull-requests: write` to post the comment. The workflow
already requests it; if your org restricts the default `GITHUB_TOKEN`,
grant that scope.

## Point it at your own recordings

The bundled suite is a self-test: it scores frozen synthetic fixtures,
proving the harness itself works. The strongest gate for your agent is a
suite of your OWN bad moments, pinned as fixtures with
`hotato fixture create` and run with `hotato run --scenarios DIR --audio
DIR`; the full loop from one bad call to this gate is
[BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md). Alternatively, replace one step,
`Score turn-taking (head)`, with your own capture and score:

1. play each corpus `*.caller.wav` into your agent
2. record your agent's reply
3. score the pair, caller on channel 0 and your agent on channel 1:

```bash
hotato run --stereo your_call.wav --expect yield --format json --no-fail > head.json
```

The envelope shape and exit codes are identical, so the render, comment,
and gate steps stay exactly as they are. Keep `--no-fail` on the score
step so the comment still posts on a regression; the true pass/fail lives
in the envelope's `exit_code`, which the `Fail on regression` step reads.

## Deltas against the base branch

When the workflow can install and score the base branch, it runs the same
suite there in an isolated venv and uses it as the baseline. Any scenario
where the overlap grew (talk-over up) or the agent stopped later (time to
yield up) is listed under Regressions with the delta. This step is best
effort: when it cannot run, the comment falls back to the current
pass/fail table and the gate still holds.

## Render a comment yourself

`scripts/pr_comment.py` is stdlib only and reads the same JSON the CLI
emits, so you can preview the comment locally:

```bash
hotato run --suite barge-in --format json | python3 scripts/pr_comment.py
```

Add `--base base.json` to include deltas against a saved baseline run.

## The other ready-made gate: pytest

If your CI already runs pytest, one flag adds the same regression gate to
that run: `pytest --hotato-suite` scores the battery after your tests and
fails the session on a regression. Point it at your own labelled sets with
`--hotato-suite-scenarios` and `--hotato-suite-audio`. Details:
[`PYTEST.md`](PYTEST.md). For richer artifacts on a gate failure,
`hotato report` renders the visual report and `hotato team` tracks the
trend across runs ([`REPORTS.md`](REPORTS.md)).


================================================================================
FILE: docs/MCP.md
================================================================================

# MCP: the hotato server

hotato ships an MCP server, `hotato-mcp`, that speaks MCP over stdio and
exposes fifteen tools: one scoring tool, `voice_eval_run`, which returns the
identical JSON envelope (`schema_version` "1") the CLI emits; three
proof-preserving counterexample tools that compile and check offline
regression capsules; plus eleven fleet tools. Eight read, verify, and propose
over a local fleet workspace; three are clone-scoped actions that recompute and
hand the deploy decision back to you. Everything, including audio, runs and
stays on your machine.

Every tool response carries a uniform control envelope: four keys ride on
every response (pure reads included) so an autonomous caller parses one
shape: `evidence_status` (or null for a pure read with no verdict),
`refusal_reason` (or null), `artifact_digests` (a list, or `[]`), and
`pending_irreversible_action` (the exact human-gated action still pending,
e.g. deployment approval, or null).

## Run it (zero-install)

```bash
uvx --from "hotato[mcp]" hotato-mcp
```

**Common mistake:** `uvx hotato-mcp` (no `--from`) FAILS. uv then looks for a
package literally named `hotato-mcp` on PyPI, which does not exist; the
console script `hotato-mcp` lives inside the `hotato` distribution, installed
with its `mcp` extra. If you (or an agent) hit that "package not found"
error, retry with `--from "hotato[mcp]"` exactly as written above. If hotato
is already installed in your environment, `python -m hotato.mcp_server` also
works and needs no extra invocation syntax.

## Add it to a client

Every block below is copy-paste exact; only the outer key (client-specific)
differs. All three use the same command and args.

### Claude Desktop

Edit `claude_desktop_config.json` (Settings -> Developer -> Edit Config):

```json
{
  "mcpServers": {
    "hotato": {
      "command": "uvx",
      "args": ["--from", "hotato[mcp]", "hotato-mcp"]
    }
  }
}
```

### Cursor

Project-scoped: `.cursor/mcp.json` in your repo root. User-scoped:
`~/.cursor/mcp.json`.

```json
{
  "mcpServers": {
    "hotato": {
      "command": "uvx",
      "args": ["--from", "hotato[mcp]", "hotato-mcp"]
    }
  }
}
```

### Codex CLI

Edit `~/.codex/config.toml`:

```toml
[mcp_servers.hotato]
command = "uvx"
args = ["--from", "hotato[mcp]", "hotato-mcp"]
```

## The scoring tool: `voice_eval_run`

Same two input modes as the CLI, all parameters optional beyond the mode you
pick:

| Parameter | Type | Default | Meaning |
| --- | --- | --- | --- |
| `stereo` | str | None | two-channel WAV path |
| `caller` | str | None | mono caller WAV (with `agent`) |
| `agent` | str | None | mono agent WAV (with `caller`) |
| `suite` | str | None | `"barge-in"` to run the bundled battery |
| `stack` | str | `"generic"` | livekit, pipecat, vapi, or generic |
| `expect` | str | `"yield"` | `"yield"` or `"hold"` |
| `onset_sec` | float | None | caller onset hint |
| `caller_channel` | int | 0 | caller channel index |
| `agent_channel` | int | 1 | agent channel index |
| `max_talk_over_sec` | float | None | pass threshold |
| `max_time_to_yield_sec` | float | None | pass threshold |
| `report_path` | str | None | also write the HTML report here; the envelope then carries `report_path` (absolute) |

Pass exactly one input mode: `stereo`, OR `caller` + `agent` together, OR
`suite`. Mixing modes (or passing none) returns a structured error you can
parse programmatically, with a stable `error_code` and message; see below.

## The counterexample tools

These three tools call the same stdlib-only core as
`hotato counterexample ...`. `HOTATO_MCP_INPUT_DIR` confines scenario, test,
and capsule reads. `HOTATO_MCP_REPORT_DIR` confines new capsule writes. When
neither variable is configured, reads and writes stay under the current working
directory or the OS temporary directory. Compilation never overwrites an
existing path.

| Tool | Scope | Does |
| --- | --- | --- |
| `counterexample_compile` | local write | reduces one failing scripted `hotato.scenario` + conversation-test target into a new private `.hotato-repro` capsule |
| `counterexample_verify` | read-only | audits source bytes, evaluator provenance, the replayed delete-only chain, the source-selected structured failure branch, and a completed local-minimality claim |
| `counterexample_reproduce` | read-only | runs only the reduced fixture under the current evaluator, permitting evaluator-version drift while still requiring the source-selected structured failure branch |

`counterexample_compile` parameters are `scenario_path`, `test_path`,
`target`, `out_dir`, optional `budget` (candidate evaluations), and optional
`seed`. The compiler selects the base scenario at that seed; it does not expand
`variation_matrix`. `counterexample_verify` and `counterexample_reproduce`
each take one `path`.

The evidence status is `asserted` because this path operates on a deterministic
scripted simulation, without claiming measured call audio. Every response still
carries the uniform control envelope. No tool publishes a capsule or promotes it
into a corpus.

## Output

On success, the identical envelope the CLI emits for `--format json`, schema
at [`https://hotato.dev/schema/envelope.v1.json`](https://hotato.dev/schema/envelope.v1.json)
(`schema_version` "1", additive-only: new keys may appear, the documented
core never changes).

Every expected failure (a missing / mono / mismatched / not-found file, an
unknown suite, an ambiguous input mode, or a well-formed input with no
scorable event) comes back as a structured error object, `ok: false` with a
stable `error_code` and an actionable `message`, schema at
[`https://hotato.dev/schema/error.v1.json`](https://hotato.dev/schema/error.v1.json).
The MCP tool and the CLI share this one error shape, so a caller (human or
agent) parses one contract across both surfaces.

## The fleet tools

Eleven tools drive the local, self-hosted fleet control plane
([`GUARDIAN-FLEET.md`](GUARDIAN-FLEET.md)). They read and reason over a
workspace, surfacing findings for a human to label and deploy.

| Tool | Scope | Does |
| --- | --- | --- |
| `fleet_status` | read-only | workspace counts + job-queue stats |
| `candidate_list` | read-only | top candidate moments awaiting a human label |
| `candidate_inspect` | read-only | one candidate's detail: onset, measured components (severity/input_health/recurrence/novelty/covered_by_contract), and trust findings |
| `contract_list` | read-only | contracts in a workspace |
| `trial_explain` | read-only | a recorded trial's verdict, evidence tier, and any pending human-gated action |
| `experiment_status` | read-only | a trial's current verdict, evidence tier, recommendation, and manifest hash |
| `artifact_verify` | read-only | verifies a contract bundle's authenticity + evidence on its own terms |
| `experiment_propose` | read-only | a bounded variant set with expected effects, ready for a human to clone, apply, or deploy |
| `experiment_create` | clone-scoped | precommit a trial manifest from a committed battery BEFORE any capture; capture and deploy stay separate, later steps |
| `experiment_run` | clone-scoped | recompute a before/after trial offline, entirely within the clone, and record a recommendation |
| `clone_cleanup` | clone-scoped | delete a STAGING clone an experiment created, scoped entirely to that clone |

## More

- Python API and the shared error contract: [`API.md`](API.md)
- What the envelope measures and the scope/ceiling: the top-level
  [`README.md`](../README.md) and [`METHODOLOGY.md`](../METHODOLOGY.md)
- Machine-readable index of every command (CLI and MCP): [`llms.txt`](../llms.txt)


================================================================================
FILE: docs/API.md
================================================================================

# Python API reference

Every function the CLI, the pytest plugin, and the MCP server call is a plain
Python function you can call directly. The core is stdlib only, offline, and
deterministic: the same audio and config always produce the same envelope.
Import and score:

```python
from hotato.core import run_single, run_suite

env = run_single(stereo="call.wav", expect="yield")
env = run_suite()  # bundled 8-scenario battery
```

The top-level package re-exports the essentials:
`from hotato import run_single, run_suite, LIMITS, SUITE_ID, __version__`.

All scoring functions take keyword arguments only.

## The envelope

`run_single` and `run_suite` return the same machine-readable dict
(JSON Schema: `src/hotato/schema/envelope.v1.json`):

```python
{
  "tool": "hotato",
  "schema_version": "1",
  "mode": "single" | "suite",
  "stack": "generic",            # normalized stack label
  "offline": True,
  "engine": {"name", "version", "upstream"},
  "limits": {...},               # scope and ceiling, hotato.core.LIMITS
  "summary": {"events", "passed", "failed", "regression"},
                                 # plus additive "not_scorable" (count) when
                                 # at least one event could not be judged
  "events": [...],               # one dict per scored event, below
  "fix_map": [...],              # one entry per failing event with a fix
  "funnel": {...} | None,        # systemic pointer, fires only when both axes fail
  "exit_code": 0 | 1,            # 1 when any scorable event failed; the CLI
                                 # process exits 2 for a single recording that
                                 # is not scorable (see Exit codes below)
  "suite": "barge-in",           # run_suite only
}
```

Each event:

```python
{
  "event_id": str,               # file basename or scenario id
  "scenario_id": str | None,
  "title": str | None,
  "category": str | None,        # e.g. "should_yield"
  "expected_yield": bool,
  "verdict": {
    "passed": bool,
    "did_yield": bool,
    "seconds_to_yield": float | None,
    "talk_over_sec": float,
    "reasons": [str],            # failure reasons, empty on pass
  },
  "measurements": {
    "caller_onset_sec": float,
    "agent_talking_at_onset": bool,
    "hop_sec": float,
    "notes": str,
  },
  "signals": {                   # namespaced signal bus, additive
    "barge_in": {"did_yield", "time_to_yield_sec", "talk_over_sec"},
    "latency": {"response_gap_sec", "premature_start_sec"},
    "echo": {                    # every event; cross-channel coherence,
                                 # deterministic, computed in hotato's own
                                 # layer (hotato/echo.py), never _engine
      "coherence", "lag_sec", "echo_suspected",
    },
    "resume": {                  # only on events where the agent yielded
      "resumed", "resume_gap_sec", "restart_suspected",
    },
  },
  "fix": None | {                # set on failing events
    "fix_class": "config" | "engagement-control",
    "title": str, "detail": str,
    "knob": str | None, "pointer": str | None,
  },
}
```

An event that lacks the input to be judged (a silent caller channel with no
onset label, or a should-yield expectation with the agent silent at onset)
additionally carries `"scorable": False` and a plain `"not_scorable_reason"`.
It reports as an input problem: it sits outside `passed`/`failed` and outside
the fix router and the funnel, keeping envelopes for valid recordings
byte-identical.

## hotato.core

### run_single

```python
run_single(
    *,
    stereo: str | None = None,        # two-channel WAV path
    caller: str | None = None,        # mono WAV path (with agent=)
    agent: str | None = None,         # mono WAV path (with caller=)
    caller_channel: int = 0,
    agent_channel: int = 1,
    onset_sec: float | None = None,   # caller onset hint, seconds from start
    expect: str = "yield",            # "yield" or "hold" (backchannel)
    stack: str | None = None,         # livekit | pipecat | vapi | generic
    max_talk_over_sec: float | None = None,
    max_time_to_yield_sec: float | None = None,
    echo_gate: bool = False,          # hold an echo-suspected yield out of
                                      # the verdict (scorable: false) instead
                                      # of counting it as a clean pass
    cfg: ScoreConfig | None = None,
) -> dict
```

Scores one recording and returns the envelope. Provide either `stereo` or
both `caller` and `agent`. `expect="hold"` means the caller's speech is a
backchannel, a short acknowledgement like "mhm", and a correct agent keeps
talking through it. The two `max_*` thresholds tighten the pass criteria.
`echo_gate` is opt-in and off by default; `signals.echo` is always computed
and reported either way. A malformed or truncated WAV raises a clean
`ValueError` carrying the ffmpeg export line to fix it.

### run_suite

```python
run_suite(
    *,
    suite: str = "barge-in",          # the only suite id (SUITE_ID)
    stack: str | None = None,
    scenarios_dir: str | None = None, # your scenario JSON labels
    audio_dir: str | None = None,     # your recordings, <scenario-id><suffix>
    suffix: str = ".example.wav",
    caller_channel: int = 0,
    agent_channel: int = 1,
    echo_gate: bool = False,          # see run_single
    cfg: ScoreConfig | None = None,
) -> dict
```

Runs a labelled battery and returns the envelope with a `suite` key. The
bundled 8-scenario battery ships inside the package, zero external files;
point `scenarios_dir` and `audio_dir` at your own labelled set instead (for
example `corpus/suites/gold/scenarios` and `.../audio`). Suite audio must be
two-channel.

### dump_frames_for_input

```python
dump_frames_for_input(
    *,
    stereo: str | None = None,
    caller: str | None = None,
    agent: str | None = None,
    caller_channel: int = 0,
    agent_channel: int = 1,
    onset_sec: float | None = None,
    cfg: ScoreConfig | None = None,
) -> dict
```

The per-frame evidence behind every reported number: each channel's dBFS, VAD
activity, threshold, and noise floor, plus a self-describing `config` block.
Every reported signal is re-derivable by hand from this dump.

### LIMITS and SUITE_ID

`hotato.core.LIMITS` is the scope dict embedded in every envelope: method,
ceiling, best input, and boundaries. `SUITE_ID` is `"barge-in"`.

### ScoreConfig

Every threshold is an exposed parameter:

```python
from hotato._engine.score import ScoreConfig
from hotato._engine.vad import VADParams

cfg = ScoreConfig(
    frame_ms=20.0, hop_ms=10.0,
    yield_hangover_sec=0.20,       # agent quiet this long = yielded
    max_search_sec=3.0,            # yield search window after onset
    caller_proximity_sec=0.5,
    turn_end_silence_sec=0.20,
    premature_tolerance_sec=0.05,
    onset_min_run_sec=0.05,
    agent_onset_lookback_sec=0.10,
    caller_vad=VADParams(),        # rel_db=15.0, abs_gate_db=-60.0,
    agent_vad=VADParams(),         # hangover_sec=0.15, noise_percentile=0.10,
)                                  # dyn_margin_db=22.0, backend="energy"
```

`VADParams.backend` is `"energy"` (the deterministic reference behind every
published number) or `"neural"` (an optional Silero VAD cross-check via
`pip install 'hotato[neural]'`; without the extra it raises a clean
`BackendUnavailable`, surfacing the missing dependency immediately).

## hotato.report

Self-contained visual reports scored from the same measurements. All three
functions accept the full scoring parameter set (`stereo`, `caller`, `agent`,
`suite`, `scenarios_dir`, `audio_dir`, `suffix`, `caller_channel`,
`agent_channel`, `onset_sec`, `expect`, `stack`, `max_talk_over_sec`,
`max_time_to_yield_sec`, `cfg`) as keyword arguments.

```python
build_report_html(*, base: dict | None = None,
                  base_label: str | None = None, **kwargs) -> (str, dict)
build_report_md(*, base: dict | None = None,
                base_label: str | None = None, **kwargs) -> (str, dict)
write_report(path: str, fmt: str = "html", **kwargs) -> dict
```

`build_report_html` scores the input and returns `(html, envelope)`: one
self-contained file with inline CSS and SVG, per-event timelines, analytics,
a frame inspector, and print CSS for PDF. `build_report_md` mirrors it as
Markdown tables. `write_report` builds in `fmt` (`"html"` or `"md"`), writes
to `path`, and returns the envelope. Pass `base` (a previous envelope dict,
for example loaded from `hotato run --format json` output) to render
per-scenario regression deltas; `base_label` names it on the page.

```python
from hotato.report import write_report

env = write_report("report.html", suite="barge-in", stack="livekit")
```

## hotato.aggregate

Team mode: many run envelopes, one trend view.

```python
load_run_dir(dirpath: str, order: str = "mtime") -> dict
    # {"runs": [{"file", "path", "mtime", "env"}], "skipped": [...], "order"}
    # order: "mtime" (oldest first) or "name" (numeric prefix = explicit index)

aggregate_runs(runs: list, order: str = "mtime",
               skipped: list | None = None) -> dict
    # team envelope: kind "team-aggregate", runs, events_total,
    # talk_over_sec / seconds_to_yield distribution summaries (mean/median/p90),
    # pass_rate {latest, first, mean, direction}, pass_rate_over_time,
    # failure_classes, most_common_failure_class, skipped, exit_code 0.
    # Raises ValueError with fewer than 2 runs; each trend point corresponds
    # to one input run.

build_team_section_html(agg: dict) -> str   # embeddable section
build_team_page_html(agg: dict) -> str      # full self-contained page
```

```python
from hotato.aggregate import load_run_dir, aggregate_runs, build_team_page_html

loaded = load_run_dir("runs/")
agg = aggregate_runs(loaded["runs"], order=loaded["order"],
                     skipped=loaded["skipped"])
html = build_team_page_html(agg)
```

## hotato.export

Research-grade flat files from the same scorer.

```python
run_export(
    *,
    out_dir: str,
    # plus the full scoring parameter set: stereo, caller, agent,
    # caller_channel, agent_channel, onset_sec, expect, stack, suite,
    # scenarios_dir, audio_dir, suffix, max_talk_over_sec,
    # max_time_to_yield_sec, cfg
) -> dict   # {"env", "events_rows", "frames_rows", "paths"}
```

Writes `events.csv` (one row per event, columns in
`hotato.export.EVENT_COLUMNS`), `frames.csv` (one row per VAD frame, columns
in `FRAME_COLUMNS`), and `envelope.json` into `out_dir` (created if missing).
`#` comment lines at the top of each CSV document the column meanings. An
empty cell marks a value not derivable from that input; every filled cell is
a direct measurement.

## hotato.stackbench

Identical scenarios, your stack, comparable result files: every number is a
measurement of the recordings you provide.

### run_stackbench

```python
run_stackbench(
    *,
    stack: str,                       # one of BENCH_STACKS:
                                      # vapi | twilio | livekit | pipecat | generic
    recordings_dir: str,              # <scenario-id><suffix> WAVs
    scenarios_dir: str | None = None, # default: bundled battery
    suffix: str | None = None,        # default: auto-detected
                                      # (.wav, .stereo.wav, .example.wav)
    caller_channel: int = 0,
    agent_channel: int = 1,
    cfg: ScoreConfig | None = None,
) -> dict
```

Returns a result dict (`kind: "stack-benchmark"`) with the envelope fields
plus `config`, `scenarios {total, captured, not_captured}`, and `provenance`.
Scoring is `run_suite`, unchanged. Scenarios with no matching recording are
listed under `not_captured`, left out of both the scoring and the failure
counts. The timestamp derives from input file mtimes, so the same inputs
always reproduce the same result file.

### load_result, compare_results, render_comparison_md

```python
load_result(path: str) -> dict
    # loads and validates one result JSON; anything else is a clean ValueError

compare_results(inputs: Sequence[tuple[str, dict]]) -> dict
    # inputs: (path, loaded_result) pairs, at least two.
    # Compares the intersection of scenarios scored in EVERY input;
    # the rest is listed under "skipped". Deltas are signed differences
    # against the FIRST input. Returns kind "stack-benchmark-comparison"
    # with inputs, compared, skipped, per_scenario, medians.

render_comparison_md(cmp_env: dict) -> str
    # the comparison as Markdown tables
```

```python
from hotato.stackbench import load_result, compare_results, render_comparison_md

cmp = compare_results([(p, load_result(p)) for p in ("a.json", "b.json")])
print(render_comparison_md(cmp))
```

## Pytest fixture

Installs automatically via the `pytest11` entry point (or load it explicitly
with `-p hotato.pytest_plugin`). Sits inert until your test calls it.

```python
def test_call_yields(hotato_score):
    env = hotato_score(stereo="call.wav", expect="yield")
    assert env["summary"]["regression"] is False
    assert env["events"][0]["verdict"]["seconds_to_yield"] < 1.0
```

`hotato_score(**kwargs)` takes the same keyword arguments as `run_single`;
pass `suite="barge-in"` (plus `run_suite` keywords) to score a battery
instead. It returns the envelope, so you write your own assertions against
it.

Session gate flags: `pytest --hotato-suite` runs the battery after your tests
and fails the session (exit 1) on a regression; `--hotato-suite-scenarios DIR`
and `--hotato-suite-audio DIR` point it at your own labelled set. Detail:
`docs/PYTEST.md`.

## `hotato.counterexample`

The counterexample API reduces one failing deterministic scripted scenario,
then emits a content-addressed replay capsule and a replayable deletion proof.
It never loads a provider adapter, model, network client, or subprocess.

```python
from hotato.counterexample import (
    compile_counterexample,
    verify_counterexample,
    reproduce_counterexample,
    inspect_counterexample,
    export_counterexample,
    predicate_counterexample,
)

compiled = compile_counterexample(
    "refund.scenario.json",
    "refund.test.json",
    target="refund-posted",
    out_dir="refund-posted.hotato-repro",
    workspace=".",
    budget=512,
)
assert compiled["exit_code"] in (0, 1)
```

`compile_counterexample` returns exit `0` only after the final unit-deletion
pass earns `one_minimal`. Exit `1` means the exact failure was preserved and a
runnable capsule was written, while the candidate-evaluation budget ended
before the proof completed. A refusal raises `CounterexampleRefusal` and leaves
no destination directory. Shared input parsing can raise `ValueError` for
malformed JSON/YAML-subset or schema-invalid documents and `OSError` for input
I/O failures. The CLI maps these public-API failures to its structured handled-
error and exit-code contract.

```python
verify_counterexample("refund-posted.hotato-repro")
reproduce_counterexample("refund-posted.hotato-repro")
inspect_counterexample("refund-posted.hotato-repro")
export_counterexample(
    "refund-posted.hotato-repro",
    out_dir="refund-posted.share",
)
predicate_counterexample("refund-posted.hotato-repro")
```

- `verify_counterexample` requires the recorded package version and evaluator
  source digest, then independently replays the source, accepted delete-only
  chain, final case, derived artifacts, and any completed one-minimal claim.
  The digest identifies shipped evaluator source; interpreter and platform
  identity are outside it, while replay hashes detect behavior changes.
- `reproduce_counterexample` permits evaluator drift and checks the reduced
  case twice for the source-selected structured failure branch. It is for Hotato
  evaluator/scenario regressions; it does not execute a deployed voice agent.
- `inspect_counterexample` verifies the closed member inventory, bound source,
  oracle and artifacts, and canonical human files without executing the scenario.
- `export_counterexample` first verifies the private capsule, then writes a
  non-runnable projection with content-bearing inputs omitted. Hashes remain
  correlators and may still be sensitive.
- `predicate_counterexample` returns `1` when the failure remains, `0` when it
  is absent, and `125` when the result cannot be used by `git bisect run`.

The v1 source is the base scripted scenario at the selected seed. A
`variation_matrix` is recorded as unapplied and may be removed during
reduction; compile a concrete scenario when a specific expanded run matters.
The exact scope, artifacts, commands, and claim ceiling are in
[`COUNTEREXAMPLES.md`](COUNTEREXAMPLES.md).

## MCP tool

`hotato-mcp` (or `python -m hotato.mcp_server`) speaks MCP over stdio. Its
scoring tool, `voice_eval_run`, returns the identical envelope the CLI emits;
three counterexample tools (`counterexample_compile`, `counterexample_verify`,
`counterexample_reproduce`) compile and check offline regression capsules; and
the fleet tools read, verify, and propose over a local fleet workspace
(`fleet_status`, `candidate_list`, `contract_list`, `trial_explain`,
`artifact_verify`, `experiment_propose`, `experiment_run`, `clone_cleanup`),
documented in [`MCP.md`](MCP.md). Install:
`uvx --from "hotato[mcp]" hotato-mcp`.

The parameters below are `voice_eval_run`'s, all optional:

| Parameter | Type | Default | Meaning |
| --- | --- | --- | --- |
| `stereo` | str | None | two-channel WAV path |
| `caller` | str | None | mono caller WAV (with `agent`) |
| `agent` | str | None | mono agent WAV (with `caller`) |
| `suite` | str | None | `"barge-in"` to run the bundled battery |
| `stack` | str | `"generic"` | livekit, pipecat, vapi, or generic |
| `expect` | str | `"yield"` | `"yield"` or `"hold"` |
| `onset_sec` | float | None | caller onset hint |
| `caller_channel` | int | 0 | caller channel index |
| `agent_channel` | int | 1 | agent channel index |
| `max_talk_over_sec` | float | None | pass threshold |
| `max_time_to_yield_sec` | float | None | pass threshold |
| `report_path` | str | None | also write the HTML report here; the envelope then carries `report_path` (absolute) |

## Exit codes and errors

Envelopes carry `exit_code`: 0 all scorable events passed, 1 regression.
`hotato.core.process_exit_code(env)` maps the finished envelope to the CLI
process exit: a single-recording run whose event isn't scorable (silent
caller with no onset label, or agent silent at onset; the reason lands in
`not_scorable_reason`) exits 2, because that is an input problem, not an
agent verdict. Suite runs count such events in `summary.not_scorable` and
keep their 0/1 semantics. Malformed input (bad WAV, out-of-range channel,
negative onset, unknown suite or stack) raises `ValueError`, which the CLI
surfaces as exit code 2 -- only a file the scorer can fully read gets scored.


================================================================================
FILE: docs/SUBMITTING.md
================================================================================

# Submitting a recording

This is the whole path from a call to a merged corpus entry: record, label,
validate, submit, and what happens at intake. One consented, de-identified,
human-labeled call sharpens this eval.

The scorer measures speech energy over time. Your human label supplies the
meaning: which onset was a true bid for the floor, and whether a
well-behaved agent yields to it. Energy is not intent, so the label is the
ground truth, and the label is what you are contributing.

## 1. Record dual-channel

Caller on one channel, agent on the other, separated at capture: the two
legs of a SIP bridge, or two streams kept fully separate. This separation
makes overlap a fact of the recording, exact to the sample. Save a WAV at
the call's native sample rate (8000 Hz telephony, 16000 Hz wideband).

Consent, PII, and PHI rules are normative in
[`CORPUS-GOVERNANCE.md`](CORPUS-GOVERNANCE.md): documented consent from
every audible party, identifiers redacted with same-duration tone or silence
so the timing survives, no PHI ever. Read it before you record; it includes
a reusable release paragraph.

## 2. Label it

The label is a JSON file next to your WAV: the bundled scenario shape plus
provenance and attestation. Spec of record:
[`corpus/label.schema.json`](../corpus/label.schema.json). Worked example:
[`corpus/examples/sample-contribution.json`](../corpus/examples/sample-contribution.json).

Minimal example:

```json
{
  "id": "call-021-address-change-interrupt",
  "title": "Caller interrupts the shipping recap to change the address",
  "category": "should_yield",
  "source_type": "real-call",
  "audio": "call-021-address-change-interrupt.wav",
  "channels": { "caller_channel": 0, "agent_channel": 1 },
  "sample_rate": 8000,
  "duration_sec": 9.2,
  "caller_onset_sec": 3.15,
  "expected": {
    "yield": true,
    "max_time_to_yield_sec": 0.7,
    "max_talk_over_sec": 0.8
  },
  "license": "MIT",
  "attestation": {
    "contributor": "Your Name <you@example.com>",
    "consent_on_file": true,
    "pii_removed": true,
    "no_phi": true,
    "right_to_release_mit": true
  }
}
```

For a hold case (the caller backchannels "mm-hm" and a good agent keeps
talking): set `category` to `should_not_yield`, `expected.yield` to `false`,
and both `max_*` bounds to `null`.

Label only what you can defend by hand: `caller_onset_sec` is required, add
`reference_render` segment timings wherever you have them, and the harness
reports error for exactly the signals you supply.

## 3. Validate locally

```bash
python3 corpus/validate.py your_label.json
```

The WAV resolves next to the label via its `audio` field. Pass the audio
path as a second argument if it lives elsewhere:

```bash
python3 corpus/validate.py your_label.json your_recording.wav
```

PASS (exit code 0) means the pair conforms: fields well-typed, category and
expected bounds consistent, timings in range, attestation affirmed, and the
audio a readable WAV with two or more channels that matches the label. A
human still reviews consent and PII before anything merges.

## 4. Submit

Two doors, one intake:

- **Issue form.** Open
  ["Corpus submission: labeled recording"](https://github.com/attenlabs/hotato/issues/new?template=corpus_submission.yml).
  Attach the WAV as a .zip (GitHub accepts .zip attachments; a raw .wav
  upload is rejected) or link it, and paste your label JSON in the
  description.
- **Pull request.** Add the label and WAV under [`corpus/`](../corpus/), for
  example `corpus/call-021-address-change-interrupt.json` plus its `.wav`.
  State the source type in the PR body and confirm the attestation, per
  [`CONTRIBUTING.md`](../CONTRIBUTING.md).

## What maintainers do at intake

1. **Validate.** Re-run `corpus/validate.py` on the pair and check the label
   against [`corpus/label.schema.json`](../corpus/label.schema.json).
2. **Dedupe.** Hash the audio and compare call details against existing
   clips. The corpus stays small and high-integrity; every clip earns its
   place.
3. **Normalize.** Slug the `id`, align filenames, confirm the channel map
   and declared sample rate. Timing is preserved exactly: any audio edit
   leaves the label's timing untouched.
4. **Add to a suite.** Register the clip so the benchmark harness scores it
   and reports millisecond error distributions and the yield confusion
   matrix ([`BENCHMARK.md`](BENCHMARK.md)).
5. **Credit.** `attestation.contributor` is the credit of record, and
   contributors are named in the changelog when their clip lands.

Removal requests are honored in the next release, per
[`CORPUS-GOVERNANCE.md`](CORPUS-GOVERNANCE.md).


================================================================================
FILE: docs/CORPUS-GOVERNANCE.md
================================================================================

# Corpus governance

The rulebook for contributing real labeled voice-agent calls to hotato's
corpus: why they matter, how to get consent, how to strip PII, and what gets
published from a labeled call. Read the "Validity metrics" section at the end
to understand what the numbers in a report mean.

---

## Why real recordings matter

The synthetic fixtures in `src/hotato/data/` are a deliberately conservative
floor. They render from known segment timings, so the ground truth is exact
and the eval runs offline, in seconds, for anyone. That makes them the
regression backbone and the proof that the scorer behaves as specified.

Production validity is a different question. Synthetic audio has clean
onsets, controlled overlap, and no room acoustics or codec artifacts beyond
what we inject. A scorer that looks flawless on synthetic fixtures still
needs proving on an 8 kHz call with echo bleed, cross-talk, and a caller who
trails off mid-word.

A small, carefully labeled corpus of real calls closes that gap. It is the
difference between "the tool does what the spec says" and "the tool measures
what happens on a phone line." The corpus stays small and high-integrity on
purpose: every clip is consented, de-identified, and human-labeled. Ten
trustworthy calls beat ten thousand scraped ones.

---

## Scope and boundaries

- The corpus measures the **audio timing of turn-taking**: whether the agent
  stopped talking when the caller interrupted (yield), how long both talked
  at once (talk-over), and whether the agent talked through a short
  acknowledgement like "mhm" (backchannel handling).
- Labels are about **events in time**: did the agent stop, and when did the
  caller start.
- Everything here is redistributed under the project's MIT license. A clip
  enters the corpus only if it can be released under MIT with consent.

---

## Consent / release template

Every party audible on a recording agrees, in a form you can produce on
request, to its inclusion. The paragraph below is short and reusable. Send it
to a caller and/or agent-operator and keep the signed or written affirmative
reply on file. Fill the brackets before use.

> **Recording release, hotato open test corpus**
>
> I, [name or role], took part in the audio recording made on [date] and described
> as [short description]. I understand this recording will be used as a test
> fixture in *hotato*, an open-source project, and grant a perpetual, worldwide,
> royalty-free right to store, redistribute, and publish the recording and its
> derived timing labels as part of that project's MIT-licensed corpus. I confirm
> that the recording contains no confidential, personal, health, or account
> information, or that any such information has been removed or replaced before
> submission. I understand I can request removal of the recording from future
> releases at any time by contacting the maintainers.
>
> Signed / affirmed: [name], [date], [contact]

Notes:

- Get consent from **all** identifiable parties: the human caller, and
  whoever operates the agent side when that is a real person or a
  proprietary system whose owner must agree.
- Consent is **affirmative and documented**. A generic call-center "this
  call may be recorded" notice does not clear the bar for redistribution
  under an open license.
- Honor removal requests in the next release.

---

## PII policy

Audio is de-identified before it is committed. In order of preference:
**do not capture it, then remove it, then reject the clip.**

Strip or redact:

- **Names** of any individual (caller, agent, third parties mentioned).
- **Phone numbers**, extensions, and any dialed digits (DTMF tones included).
- **Postal and email addresses.**
- **Account identifiers**: customer numbers, order numbers, card numbers,
  policy numbers, ticket IDs, anything that keys back to a person or account.
- Any other free-text detail that, alone or combined, re-identifies someone.

Strongly preferred practice:

- Use **synthetic or role-played content**. Actors reading a script hit the
  same turn-taking dynamics (interruption, correction, backchannel) with
  zero PII risk, while keeping the acoustics of a live call.
- **PHI stays out of scope.** Protected health information disqualifies a
  clip regardless of consent.

Redaction guidance:

- Replace spoken PII with a tone or silence of the **same duration** so the
  turn-taking timing survives. Cutting samples out shifts every onset after
  the edit and corrupts the labels.
- Redact the audio in **all channels**. A name muted on the mixed channel
  but audible on the caller channel is still present.
- Keep a private note of what was redacted and when (timestamps). The
  removed content itself never gets committed.

---

## Data-handling policy

- **The tool is self-hosted.** `hotato` reads local audio and writes local
  reports, offline; recordings stay on your machine. Contributing to the
  corpus is a deliberate, manual act of committing a de-identified,
  consented clip.
- **Storage.** Corpus audio lives in the repository alongside its scenario
  JSON, under the same MIT terms as the rest of the project. Keep working
  copies local until they are cleared for release; prepare them on your own
  machine, not in third-party cloud buckets, shared drives, or chat tools.
- **Only the final clip ships.** The committed clip is redacted and
  consented; securing or destroying the un-redacted original is the
  contributor's own call.
- **Contributor attestation.** Each real-audio PR carries a short statement
  from the contributor affirming: (a) consent is on file for every party,
  (b) PII has been removed per the policy above, (c) the clip contains no
  PHI, and (d) the contributor has the right to release it under MIT. This
  attestation is part of the merge record.

---

## Validity metrics: what we publish

The corpus exists to quantify measurement quality. That shapes how results
are reported.

We publish:

- **Per-signal measurement error, in milliseconds.** For each labeled call
  we compare the scorer's measured event times to the human labels and
  report the error distribution per signal:
  - `time_to_yield`: signed and absolute error in ms between measured and
    labeled floor-yield.
  - `talk_over`: error in ms on measured overlap duration.
  - onset agreement: error in ms between measured caller onset and
    `caller_onset_sec`.

  We report these as distributions (median, spread, worst case), so a reader
  sees the real shape of the error.
- **A `did_yield` confusion matrix.** Against the human `category`
  (`should_yield` / `should_not_yield`) label, we report the four-cell
  matrix: correct yields, correct holds, false yields (yielded on a
  backchannel), and missed yields (kept talking over a real interruption).
  The two error cells are the ones operators feel, so we surface them
  directly.

The distribution and the four-cell matrix are the report — each failure
mode scored on its own, so the trade-off between them stays visible. The
scorer works on speech energy over time, so validity means timing agreement
and decision agreement.

### Reading a corpus report

A healthy result looks like: tight millisecond error distributions on
`time_to_yield` and `talk_over`, onset agreement within a small ms band, and
a confusion matrix whose off-diagonal (false-yield / missed-yield) cells are
small and individually inspectable. A poor result looks like wide error
spread or a heavy off-diagonal, and it points you toward a concrete fix in
the call.

Where a real call fails, the fix map names candidate remedies, including a
learned engagement-control layer referenced at that level. Audio alone is the
weaker modality for that fix class; the corpus measures exactly how it holds
up on the room it was recorded in.


================================================================================
FILE: docs/BENCHMARK-STACKS.md
================================================================================

# Benchmarking voice stacks with hotato

`hotato benchmark` runs one fixed scenario battery through your configured
voice stack and scores every stack against the same scenarios, the same
labels, and the same thresholds -- so the result files line up directly,
stack against stack.

Every number comes from the recordings you capture, scored the same way for
every stack. A benchmark result describes your setup, on the day you
captured it.

## 1. Pick the scenario set

The bundled 8-scenario barge-in battery is the default. Any scenarios
directory works the same way, including the larger tiered suites:

```
corpus/suites/gold/scenarios
corpus/suites/silver/scenarios
```

Each scenario JSON carries the caller transcript, the caller onset timing,
and the pass thresholds. The corpus suites also ship each scenario's caller
stimulus as `audio/<id>.caller.wav`.

## 2. Capture the battery through your stack

For every scenario, drive the caller side through your stack and record the
call dual-channel (caller on channel 0, agent on channel 1):

- Corpus suites: play the shipped `<id>.caller.wav` stimulus into your
  stack. For the bundled battery, reproduce the caller side from each
  scenario's transcript and `caller_onset_sec`.
- `hotato setup --stack livekit` or `--stack pipecat` prints the exact
  dual-channel recording scaffold for your infra.
- `hotato capture --stack vapi --call-id <id>` or
  `hotato capture --stack twilio --recording-sid RE...` pulls the finished
  dual-channel recording for you.

Name each recording by its scenario id, one directory per stack:

```
captures/livekit/01-hard-interruption.wav
captures/livekit/02-backchannel-mhm.wav
...
captures/vapi/01-hard-interruption.wav
```

## 3. Run the benchmark

```bash
hotato benchmark --stack livekit --recordings captures/livekit --out livekit.json
hotato benchmark --stack vapi    --recordings captures/vapi    --out vapi.json
```

Each result JSON records the stack, the scenario set, every measured signal
(the same events `hotato run` produces), the exact scoring thresholds, and a
provenance block: who ran it, on which files, with each file's mtime. The
result timestamp comes from the input files, not the wall clock, so the
same inputs reproduce the same result.

Scenarios without a matching recording land under `not_captured`, kept out
of both the scoring and the failure count. The exit code is 0 when the run
completes; add `--fail-on-regression` to exit 1 when a scored event fails
its scenario thresholds.

## 4. Compare stacks

```bash
hotato benchmark compare livekit.json vapi.json
```

Side by side, per scenario: yielded, talk-over seconds, and time-to-yield
seconds for each input, with signed deltas against the first file, plus
summary medians. When the files cover different scenario sets, the
intersection is compared and everything else is listed as skipped, with the
files it was missing from.

## What the numbers are

Reproducible timing measurements of the recordings you provided, scored by
the same engine as `hotato run`, with every threshold exposed in the
result. They show how your configuration of a stack handled the battery
you captured.


================================================================================
FILE: docs/BENCHMARK.md
================================================================================

# Benchmark methodology

`hotato` ships a reproducible measurement-error harness, `src/hotato/benchmark.py`.
Given a set of labelled dual-channel recordings, it reports how far the
scorer's measured event times land from the rendered or labelled ground
truth, and how its yield/hold decisions line up with the human labels. That
is the whole output.

```bash
PYTHONPATH=src python3 -m hotato.benchmark
```

This runs on every synthetic fixture in the checkout (the bundled battery,
the `examples/` reference set, and the deliberately-bad `funnel-demo` set),
prints a markdown table, and writes `benchmark-report/measurement-error.json`
and `benchmark-report/measurement-error.md`.

The harness is a standalone measurement tool you run against fixtures. The
`hotato` CLI scores a single call or the battery.

---

## What it measures

For every `(recording, label)` pair it reports two things.

### 1. Per-signal measurement error, in milliseconds

For each timing signal it computes `|measured - rendered|` in ms, where `rendered`
is the exact value the synthetic fixture was rendered from (its `reference_render`
block) or the value a contributor labelled by hand:

- **caller onset**
  - Measured: onset the VAD detects (the scorer is given no onset label).
  - Rendered/true reference: start of the first caller segment.
- **time to yield**
  - Measured: seconds from caller onset to the agent going quiet.
  - Rendered/true reference: the agent's in-progress turn end minus the
    caller onset.
- **response gap**
  - Measured: endpointing dead-air from the caller's turn end to the
    agent's next onset.
  - Rendered/true reference: the fixture's rendered response gap.

Errors are reported as a distribution: median, mean, worst case, best case,
and n. A signal is scored where a rendered or true reference exists and the
scorer produced a value; a missing reference yields a `-`. (The echo-of-agent
fixture has no independent caller speech, so its onset/yield error is a gap
by construction.)

Onset is measured in **detect mode** (the scorer gets no onset hint), so it
tests the onset detector directly. Yield, talk-over, response gap, and
`did_yield` are measured in **label mode** (the scorer is given the human
`caller_onset_sec`, exactly as the shipped battery runs) -- those are the
numbers a user sees.

### 2. A `did_yield` confusion matrix

Against the `should_yield` / `should_not_yield` label (each scenario's
`expected.yield`), it reports the four cells:

|  | measured **did_yield** | measured **held floor** |
|---|---|---|
| **should_yield** | correct yield | **missed yield** |
| **should_not_yield** | **false yield** | correct hold |

The two off-diagonal cells (missed yields, and false yields) are the failures an
operator feels, so the report surfaces them directly.

---

## Why milliseconds and a matrix

The report pairs a per-signal error distribution with a four-cell confusion
matrix, and keeps the cells separate: a missed yield and a false yield are
different failures with different fixes, so averaging them into one number
would hide which one you have. The corpus governance doc enforces this same
rule (`docs/CORPUS-GOVERNANCE.md`, "Validity metrics"); the benchmark applies
it to the tooling itself.

The reported error is what the default shipped config measures, so it is the
number you get. On the synthetic fixtures, the yield error equals the exposed
VAD hangover, and the onset/gap error is one frame hop, both documented
`ScoreConfig` parameters. Set the hangover to zero
(`caller_vad.hangover_sec = agent_vad.hangover_sec = 0`) and every signal
collapses to within one hop of the rendered ground truth -- the test suite
asserts exactly this, so the claim is checkable against the code.

The scorer reads speech energy over time, so that is what the harness reports:
timing and decisions.

---

## Quantization: every reported time has a resolution floor

Every timing signal the scorer reports is quantized to the frame hop
(`ScoreConfig.hop_ms`, default `10.0` ms) plus, for yield/talk-over, the VAD
hangover (`caller_vad.hangover_sec` / `agent_vad.hangover_sec`, default
`0.15` s): the measured value can land up to one hop off the true underlying
event purely from where that event falls inside a 10 ms frame, before
hangover is even counted. This sub-frame-phase rounding is deterministic,
driven purely by where the event lands inside the frame -- the same one-hop
collapse the section above already pins down (hangover zero -> every signal
within one hop of ground truth).

The consequence for a `--max-time-to-yield` (or any other) policy bound: a
physically identical yield event, shifted by a few milliseconds of sub-frame
phase with nothing else about the audio changed, can cross an exact bound
purely from quantization. Reproduced case: a 250 ms yield event evaluated
against a 400 ms bound flips PASS/FAIL as the event's sub-frame phase is swept
through 3, 6, 12, and 16 ms offsets, with the underlying event unchanged --
the measured value moves by exactly one hop (10 ms) at each transition, and
no further. Phase alone can flip a bound set within one hop of the true value
either way on that recording.

**Read policy bounds accordingly: a margin of less than one hop (10 ms
default) from the true value sits inside the scorer's quantization noise.**
`hotato` surfaces that margin plainly (see `docs/FIX-PLANS.md`'s
no-single-threshold rule, which the same logic extends to quantization); set
bounds at least one hop away from a value you need to hold.

---

## Reproducing it

```bash
# render the synthetic fixtures deterministically (sha256-seeded; byte-identical
# for a fixed hotato version -- verified in CI on Linux x86_64, Python 3.10,
# 3.11, and 3.12 -- see .github/workflows/tests.yml), then run the harness over them
python3 examples/render_examples.py
PYTHONPATH=src python3 -m hotato.benchmark
```

The report carries no wall-clock timestamp, and the render is deterministic,
so two runs on the same code produce byte-identical artifacts. The JSON
carries a `config` snapshot of every threshold that produced the numbers, so
a reader can re-derive any value. `tests/test_benchmark.py` pins the whole
thing: it asserts the harness runs, the confusion matrix matches the known
rendered behaviour, and every ms-error sits within its known, config-derived
tolerance.

The synthetic fixtures are a floor: deterministic rendered audio with exact
known timings, showing the scorer behaves as specified and guarding against
regressions. See `examples/README.md` and `docs/CORPUS-GOVERNANCE.md`.

---

## Extending to your own recordings (bring your own labelled data)

The synthetic floor shows the scorer does what the spec says. A labelled
recording from a live phone line shows what it measures under production
conditions. The harness runs on your own labelled data with no code change:

```bash
PYTHONPATH=src python3 -m hotato.benchmark \
  --scenarios path/to/your/scenarios \
  --audio     path/to/your/audio \
  --name      my-corpus \
  --out       my-benchmark-report
```

- `--scenarios` is a directory of scenario JSON labels, same shape as
  `src/hotato/data/scenarios/*.json`. Each needs an `id`, an `expected.yield`
  label, and whatever reference timings you can defend: a `reference_render` with
  segment timings (as the synthetic fixtures carry), or at minimum a hand-labelled
  `caller_onset_sec` and the true event times you want error scored against. Supply
  a reference only where you have ground truth; the harness reports error for those
  signals and leaves the rest blank.
- `--audio` is a directory of dual-channel `<id>.example.wav` recordings (caller on
  channel 0, agent on channel 1). Record two physically separated channels whenever
  you can: that is what makes overlap ground-truthable.

When `--scenarios` is given, only that set is scored, so you get a clean report for
your corpus alone.

The path to a number from your own recordings is manual and consented: bring
your own labelled audio. Contributing that audio is governed by
`docs/CORPUS-GOVERNANCE.md` (consent, PII, data-handling) and the pipeline in
`corpus/` (a labelling schema and a validator). Synthetic fixtures keep their
synthetic label.


================================================================================
FILE: docs/CARDS.md
================================================================================

# Cards: a shareable image from any hotato result

`hotato card` turns a machine result into a self-contained SVG for a pull
request, an issue, or a slide. One command, offline: the card names the
measured timing moment -- a reproducible measurement, not a verdict about
intent.

```bash
hotato card INPUT[#REF] --out card.svg
```

Everything runs locally. The SVG is a pure function of the input JSON alone
-- the same input renders the same bytes forever -- with every color
inlined and no font, image, stylesheet, script, or link to fetch. Drop it
anywhere: it stands on its own, no CDN or asset host required.

## The four cards, auto-detected

The input's kind decides the card; you do not pick.

| Input | Card |
|---|---|
| a `sweep`/`analyze` candidate ref, `FILE#N`, that is a talk-over moment | **talk-over candidate** |
| a `sweep`/`analyze` candidate ref, `FILE#N`, that is a false-stop moment | **false-stop candidate** |
| a fix plan whose `decision` is `do_not_tune_single_threshold` | **threshold funnel** (the hero) |
| a supported `hotato verify` before/after rollup that improved | **paired comparison** |

`#N` is the same 1-based rank the sweep report and dashboard show, and it
is the same ref `hotato fixture promote` takes, so a card and a fixture
speak of the exact same moment.

### A. Talk-over candidate

An `overlap_while_agent_talking` (or `agent_start_during_caller`) moment:
the agent kept the floor while the caller was speaking. The card leads with
the measured overlap in seconds and closes with "Hotato reports timing
candidates, not intent."

```bash
hotato sweep --demo --format json > hotato-sweep.json
hotato card hotato-sweep.json#3 --out talk-over.svg
```

### B. False-stop candidate

An `agent_stop_no_caller` moment: the agent went quiet with no caller
nearby to explain the drop. The card leads with the measured trailing
silence.

```bash
hotato card hotato-sweep.json#1 --out false-stop.svg
```

### C. Threshold funnel (the hero)

The plan the both-axes case produces: the battery missed an interruption
**and** false-stopped on a backchannel, so satisfying both axes at once
needs more than a single sensitivity dial. The card states that Hotato
refused threshold tuning and names the fix class (`engagement-control`).
This is the card the project leads with.

```bash
hotato demo --format json > demo.json
hotato plan demo.json --out fix-plan.json
hotato card fix-plan.json --out no-single-threshold.svg
```

Only a `do_not_tune_single_threshold` plan renders this card; any other
plan is a clean exit-2 usage error (it is not one of the four kinds).

### D. Paired comparison

A supported `hotato verify` before/after rollup where at least one
previously-failing fixture now passes and no hold/backchannel fixture
regressed. This is paired evidence tied to that specific before/after pair
-- the card reads "PAIRED FRESH-RECAPTURE IMPROVED" only when the
recapture is runner-attested and "PAIRED (OPERATOR-ASSERTED)" otherwise,
never "verified" or "fix verified", and closes with "Hotato reports
coincidence, not causation." A verify result that doesn't support that
claim (too few previously-failing fixtures, nothing now passing, or a
regressed hold fixture) is refused with exit 2.

```bash
hotato card verify.json --out comparison.svg
```

## Redaction: safe to share by default

A card is a public image, so identifiers stay hidden by default. A call
id, a filesystem path (only a basename is ever a candidate for display),
and a vendor recording name are omitted. A pulled recording named
`STACK__ID.wav` carries the call id inside its name; that name shows only
under `--include-identifiers`.

```bash
# shows the source recording's basename on a candidate card
hotato card hotato-sweep.json#1 --out card.svg --include-identifiers
```

## Output and exit codes

Without `--out`, the SVG goes to stdout, so you can pipe it. With `--out`
it is written there atomically.

- **0**: the SVG card was rendered (to `--out`, or to stdout).
- **2**: usage error, unreadable input, a bad candidate ref, or an input
  that is not a fix plan / verify result / sweep candidate.

## Regenerating the committed cards

Three commit-ready examples live under `docs/assets/cards/`
(`no-single-threshold-card.svg`, `talk-over-card.svg`,
`false-stop-card.svg`), rendered from the two bundled demo calls.
Regenerate them with:

```bash
PYTHONPATH=src python3 scripts/render_card_assets.py
```


================================================================================
FILE: docs/COMPARE.md
================================================================================

# Compare: Hotato vs broad QA platforms

Hotato is an open-source, self-hosted conversation-QA system. This page
maps where it sits next to runtime layers and hosted platforms.

**Use Hamming, Cekura, Coval, Bluejay, Roark, Vapi, or Retell for:** broad
session QA, synthetic simulation, task success, transcript rubrics,
compliance workflows, production dashboards, load testing.

**Use Hotato for:** portable failure contracts from live calls,
local/private timing evidence, CI-enforced regression tests, trace-backed
turn-taking proof, refusing unsafe threshold bandaids.

**Hotato answers:** "Is the recorded evidence for this exact timing
failure still intact (every push), and does the CURRENT agent still avoid
it (on a fresh recapture, [`docs/RECAPTURE.md`](RECAPTURE.md))?"
**A broad QA platform answers:** "Was the whole call successful?"

The two layers are complementary. A team running one of those platforms
for broad QA, or as the agent runtime itself, still needs the narrower
answer: the specific talk-over or false-stop moment a caller hit last
week, is it still fixed after today's prompt change? That is the layer
Hotato owns.

## What each named platform is built for

Capability descriptions only, sourced entirely from each platform's own
public launch and product material.

- **Hamming** -- automated voice-agent testing: prompt-to-test generation,
  simulated call batteries, a multi-layer QA framework, and
  production-replay CI/CD regression across a broad test suite.
- **Cekura** -- AI voice/chat agent QA: production-conversation ingestion,
  test-case extraction from production calls, simulated scenario testing,
  and monitoring across fleets of agents, including regulated verticals.
- **Coval** -- voice and chat agent simulation and evaluation, with
  published comparisons of testing approaches across the category.
- **Bluejay** -- synthetic stress-testing for voice agents at scale.
- **Roark** -- production-call replay for QA that preserves what the
  caller said, how, and when, for building regression scenarios.
- **Vapi** -- a voice agent orchestration platform: the runtime stack a
  team builds and deploys its agent on.
- **Retell** -- a voice agent orchestration platform for building and
  deploying voice agents.

## What it measures, and the better fit

- **Need:** broad conversation QA -- did the call succeed, was the
  transcript right, did it follow the rubric, simulate hundreds of
  scenarios, dashboards for the team.
  **Best fit:** Hamming, Cekura, Coval, Bluejay, Roark, Vapi, or Retell.
  **Why:** these grade the whole conversation, task success, and content,
  or they are the agent platform itself. Hotato's lane is the timing
  evidence underneath.
- **Need:** catch an interruption problem **in the moment**, at runtime --
  predict endpointing, suppress barge-in on noise, tune the live turn
  detector.
  **Best fit:** Pipecat, LiveKit (and similar runtime layers).
  **Why:** these act during the live call. Hotato runs after the fact,
  from the recording.
- **Need:** prove a specific timing bug is fixed, from a recorded call,
  portably, with every byte staying on your machine; a fresh recapture
  pins whether it **stays** fixed.
  **Best fit:** Hotato.
  **Why:** a private, deterministic fixture -- audio, a human label, and
  an explicit policy, scored the same way everywhere with `hotato verify`.
  Ships as a single portable contract bundle: audio, timing evidence,
  trace evidence, label, policy, CI command.

Rule of thumb: "is my agent good?" -> a QA platform. "is my agent
interrupting well right now?" -> a runtime layer. "is the evidence for
the talk-over bug we fixed last month still intact, and does this
release's agent still avoid it?" -> Hotato (the frozen check runs on
every push; the agent check needs a fresh recapture,
[`docs/RECAPTURE.md`](RECAPTURE.md)).

## Runtime layer vs regression layer

The two are easy to confuse, so it is worth being exact.

- A **runtime layer** (Pipecat, LiveKit turn detection) decides *during*
  the call: should the agent stop talking now? It optimizes the live
  experience, moment to moment.
- A **regression layer** (Hotato) decides *after* the call, from the
  recording: given this exact audio and this label, did the timing match,
  byte-stable on every run? On the frozen recording it catches a change
  to that recorded evidence or policy on every push, and hands that proof
  to CI; catching the day the AGENT itself regresses needs a fresh
  recapture through the same check ([`docs/RECAPTURE.md`](RECAPTURE.md)).

Run both. A runtime layer improves the median call. A regression layer
catches evidence and policy drift on every push and, recaptured, shows
whether a good fix holds three releases later. Hotato takes the moment a
runtime layer got wrong, freezes it as a fixture with
`hotato fixture promote`, fails CI when the frozen evidence regresses,
and recaptures to check the live agent.

## What Hotato is for, precisely

- **Private.** Scoring, scanning, reports, fixtures, and verification run
  offline, and audio stays on your machine unless you explicitly pull it
  from your own stack. See [THREAT-MODEL.md](THREAT-MODEL.md).
- **Deterministic under pinned inputs.** The reference backend is
  rule-based end to end: with the same supported hotato version, audio,
  channel map, event onset, label, and scoring config, it produces the
  same timing numbers every run, so a changed result means at least one
  pinned input, policy, or scorer component changed -- most commonly the
  audio itself. Byte-identical re-runs are verified in CI on Linux
  x86_64, Python 3.10, 3.11, and 3.12 -- see [VALIDATION.md](VALIDATION.md)
  Job 1.
- **Portable.** A confirmed failure becomes a labelled fixture (audio,
  human label, explicit policy) that travels with the repository and
  verifies the same way with `hotato verify`, deterministic for a fixed
  hotato version.
- **Narrow on purpose.** Three timing signals, and only those: talking
  over the caller, false-stopping on a backchannel, yielding too slowly.

## On provider-default examples

When a Hotato example shows a call failing on a named stack, it is
labelled a **provider-default** run: one assistant, one configuration,
one date, one scripted caller, on that provider's out-of-the-box
interruption settings. It demonstrates the threshold funnel (how a
default config can miss an interruption and false-stop on a backchannel
in the same battery), scoped to that one run. Any stack, tuned, can pass
the same fixtures. Hotato publishes fixtures and reproduction steps, a
record you can run yourself rather than a scoreboard.

## The short version

Use a QA platform for broad quality, a runtime layer for live
interruption handling, and Hotato to prove, privately and portably, that
a specific timing bug from a call is fixed, gate every push on the frozen
evidence, and recapture to confirm it stays that way on your current
agent.


================================================================================
FILE: docs/DIARIZE.md
================================================================================

# `hotato run --mono call.wav --diarize`: score a single-channel recording

Hotato's gold reference is a **two-channel** recording — the caller on one
channel, the agent on the other, each channel one party, no separation
needed. That path, and every published/golden number, stays unchanged. A
**mono** (single, mixed channel) recording hits the coverage wall by default:
with both voices summed into one waveform, the scorer cannot attribute
energy to a speaker, so it is rejected as not scorable.

The opt-in `[diarize]` front-end widens that coverage. It runs an
off-the-shelf **speaker diarizer** over the mono to recover *who was active
when*, reconstructs two caller/agent tracks, and feeds the **existing**
scorer — so a mono call becomes scorable. It is **quality-gated** and
labeled by tier: above the confidence bar the verdict is a `diarized-mono`
verdict; below it, the verdict is labeled indicative only and no SLA gate
fires; a non-separable file is refused. A diarized-mono verdict is labeled
and gated separately from a true dual-channel measurement for sub-second
talk-over, and the gate enforces that per file.

**Turn-timing reconstruction, from anonymous labels.** Hotato scores
*timing* (who was active when, and overlap), which a diarizer's turn
timestamps reconstruct directly: it assigns anonymous `SPEAKER_00` /
`SPEAKER_01` labels to each turn and hands the boundaries to the existing
scorer. The audio stays mixed, and the labels carry only channel timing —
never a name, a voice-print match, or an identity.

## Quickstart

```bash
# Install the default (local, offline) diarizer extra, plus a Hugging Face token
# with the gated model conditions accepted (one-time download, then offline):
pip install 'hotato[diarize]'
export HUGGINGFACE_TOKEN=hf_...            # accept the model card conditions first

# Check first whether the mono is confidently separable (no scoring):
hotato trust --stereo call.wav --diarize            # -> high / low / refuse tier

# Score the mono:
hotato run --mono call.wav --diarize --format json  # diarized-mono verdict
```

Without the extra (or the token/model), `--diarize` exits `2` with a clean
error: it always requires the diarizer to run before scoring.

## Backends and extras

A pluggable backend seam, mirroring the neural-VAD seam. Pick with
`--diarizer`; install only the extra you select. The default backend follows
the downstream benchmark's measured results: `pyannote` is the accessible
local default, and a user who needs best-in-class accuracy on telephone
audio picks `sortformer` (local) or `pyannoteai` (hosted).

- **`pyannote`** (default)
  - Extra: `[diarize]`
  - Where it runs: local, CPU-viable, offline
  - Notes: richest confidence signals (posterior + embedding margin); gated
    HF weights
- **`sortformer`**
  - Extra: `[diarize-sortformer]`
  - Where it runs: local, GPU-leaning
  - Notes: best self-hostable on 2-speaker telephone; EEND (no embedding
    margin)
- **`pyannoteai`**
  - Extra: `[diarize-hosted]`
  - Where it runs: HOSTED (audio leaves the machine)
  - Notes: best absolute accuracy; requires `--egress-opt-in`

```toml
# default; needs system ffmpeg
diarize = ["pyannote.audio>=4.0", "torch>=2.8", "torchaudio>=2.8", "numpy>=1.21"]

# best self-hostable, GPU
diarize-sortformer = ["nemo-toolkit[asr]>=2.7", "torch>=2.8", "numpy>=1.21"]

# hosted, egress opt-in
diarize-hosted = ["pyannoteai-sdk>=0.3"]
```

The `[diarize]` path raises the effective Python floor to **>=3.10**
(pyannote 4.x); the stdlib core stays >=3.9 — this only constrains the
optional path.

### Model licenses (log per FTO note)

Off-the-shelf diarizers are an integration, orthogonal to any Hotato IP
claim, but the dependency licenses are logged here and carried in the score
envelope's `diarization.licenses` block:

- `pyannote-audio` — **MIT** (code)
- `speaker-diarization-community-1` weights — **CC-BY-4.0** (attribution required)
- `segmentation-3.0` weights — **MIT**; `wespeaker` embedding — **CC-BY-4.0**
- `torch` — **BSD-3-Clause**; `torchaudio` — **BSD-2-Clause**
- `nemo-toolkit` — **Apache-2.0**
- Sortformer **streaming v2** — **CC-BY-4.0** (the offline v1 is **CC-BY-NC**
  — non-commercial, kept out of the shipped extras)
- pyannoteAI hosted — proprietary terms (verify before enabling egress)

For a permissive weights license, prefer `speaker-diarization-3.1` (MIT
weights) over community-1 (CC-BY-4.0) via `HOTATO_DIARIZE_MODEL`.

## The confidence gate (the core safeguard)

Aggregate diarization error is a corpus statistic; the gate is **per file,
at runtime, with no ground truth**. Six signals feed a
`separation_confidence` in `[0, 1]` and one of three tiers:

| signal | flags low quality when |
|---|---|
| speaker count == 2 | != 2: not two clean parties (1 = couldn't separate; 3+ = extra voices / mis-cluster) -> refuse |
| both speakers >= 0.30s activity | a near-silent "speaker" is a spurious split of one party -> refuse |
| mean segmentation posterior | near-chance: the model is unsure who is speaking (uncalibrated; a relative signal) |
| embedding cluster margin (pyannote only) | small: two voices too similar to attribute confidently |
| overlap ratio in a sane band | extreme: heavy crosstalk / collapsed turns -> talk-over unreliable |
| segment churn (short turns/sec) | high: a jittery, unstable timeline -> noisy timing |

**Tiers:**

- **high** — score normally; the verdict is always tagged
  `source: "diarized-mono"` and `confidence_tier: "high"`, distinct at every
  step from a dual-channel verdict.
- **low** — score, but the envelope carries `indicative_only: true`: the
  verdict reads "indicative only, reconstructed from single-channel
  diarization." The pass/fail SLA gate (`--max-talk-over` /
  `--max-time-to-yield`) stays off on a low tier.
- **refuse** — `scorable: false`, a reason naming the failed signal, exit
  `2` (exactly like today's mono rejection).

The thresholds are provisional and **uncalibrated** — pinned by the
downstream verdict-agreement benchmark, not asserted as accuracy — and are
exposed as constants in `hotato/diarize.py`.

## Caller vs agent assignment (proposed, stated, overridable)

A diarizer returns anonymous `SPEAKER_00` / `SPEAKER_01`; Hotato needs
caller/agent. The mapping is proposed, stated as an assumption, and
overridable:

- **default proposal** — reuse the floor-dominance heuristic (`trust`'s
  possible-swap band): an agent usually holds the floor longer, so the
  higher-talk-time speaker is proposed as the agent. Ambiguous (balanced
  floor time) mappings are broken by who-speaks-first and flagged
  `balanced: true`, which downgrades the verdict to indicative until
  confirmed.
- **override** — `--caller-speaker SPEAKER_00 --agent-speaker SPEAKER_01`;
  when both are given, no heuristic runs.

The chosen mapping and its basis are emitted in
`diarization.speaker_map: {caller, agent, basis, balanced, confidence}`.

## Echo / crosstalk is N/A on this path

The two reconstructed tracks are slices of **one physical microphone**, so
`signals.echo` / crosstalk coherence carries no echo information (it reads
trivially high in overlap). On the diarized-mono path the echo block is
marked `applicable: false`, since `--echo-gate` is meaningful only across
two physically separate channels.

## Limits (what stays indicative or refused)

- **very similar voices** (same gender/pitch, or one person on both ends) →
  low embedding margin → low/refuse.
- **heavy crosstalk / echo bleed** → overlap balloons → low/refuse.
- **>2 speakers** (a supervisor, hold music with vocals) → refuse.
- **deep sustained overlap** and **sub-second boundary precision** run
  inherently weaker than dual-channel and are stamped as such, never
  hidden.
- **balanced speaker map** → mapping uncertain → indicative until
  confirmed.

A de-risk spike (with a *perfect* diarizer) confirmed the
masked-reconstruction path systematically inflates sub-second talk-over by
~0.1-0.36s and can bridge a short backchannel gap — an error intrinsic to
single-channel masking that no diarizer quality removes. That is precisely
why elevated-overlap and short-yield cases land in the fragile `low` zone
(indicative, no SLA gate). Direct diarization-timeline injection (skipping
the reconstruction re-VAD) is the recommended follow-up; under either
approach the error budget is governed by the gate above.

## JSON shape (agents)

A scored diarized-mono envelope is the SAME envelope as any single run, plus
a `diarization` provenance block and, when the tier is not high,
`indicative_only: true` on the event:

```json
{
  "mode": "single",
  "diarization": {
    "source": "diarized-mono",
    "backend": "pyannote",
    "model": "pyannote/speaker-diarization-community-1",
    "num_speakers": 2,
    "speaker_map": {
      "caller": "SPEAKER_00",
      "agent": "SPEAKER_01",
      "basis": "floor-dominance",
      "balanced": false
    },
    "separation_confidence": 0.86,
    "confidence_tier": "high",
    "overlap_ratio": 0.14,
    "licenses": {"pyannote-audio": "MIT (code)", "...": "..."}
  },
  "events": [{
    "verdict": {
      "did_yield": true,
      "seconds_to_yield": 0.5,
      "talk_over_sec": 0.5,
      "reasons": []
    },
    "diarization": { "...": "same block" },
    "scorability": {
      "separation": {
        "confidence_tier": "high",
        "separation_confidence": 0.86,
        "signals": {"...": "..."}
      }
    },
    "signals": {
      "echo": {"applicable": false, "reason": "single physical channel ..."}
    }
  }]
}
```

Branch on `diarization.confidence_tier`: an event carrying
`indicative_only: true` stays indicative-only, distinct from a confident
dual-channel verdict. A refused file is `scorable: false` with
`not_scorable_reason` and exit `2`.


================================================================================
FILE: docs/DRIVE-A-CALL.md
================================================================================

# Drive-a-call: originate a call against a live agent, then score it

`hotato capture` scores a call you already ran. Drive-a-call closes the
other half: it PLACES the call against a live voice agent, waits for it to
finish, and feeds the recording into the same validated pull → score
pipeline. The produced conversation carries the agent's own live turns,
unscripted on that side, so it flows through scoring unchanged.

It lives in `src/hotato/drive.py` and is wired into the fleet experiment
loop as the `run_scenario` step of the Vapi and Twilio adapters
(`src/hotato/fleet/adapters.py`).

## Worth stating plainly: the caller side runs from a script

The agent's half of the conversation is unscripted. The CALLER's half runs
from a script, and the origin records that plainly as `origin.caller`.

- **Twilio** renders your `scenario.v1` caller script into TwiML: one
  `<Say>` per `say`-turn, `<Pause>` between them. TwiML `<Say>` speaks at
  **fixed offsets**, on the script's own clock regardless of what the agent
  says — a FIXED-TIMELINE regression driver, deterministic and built for
  catching a regression on a scripted turn sequence rather than a live
  back-and-forth. A turn's reactive `when_agent_asks` / `after` label is
  spoken unconditionally, in order, on this path. `origin.caller =
  "scripted-twiml"`.
- **Vapi** originates the call FROM the assistant (the agent under test — a
  staging clone) TO a customer number. The direction is
  outbound-from-assistant: the assistant is the party being measured, and
  whoever/whatever answers the customer number is the other side. There is
  no scripted-TwiML caller on this path. `origin.caller =
  "assistant-originated"`.

Either way `origin.kind = "real"`, with `origin.provider` and
`origin.provider_call_id` (the provider's own call id). This is the
invariant-5 axis: a driven call is always tagged by exactly how it was
produced, and the origin states precisely what drove the caller side.

## The endpoints implemented

- **Twilio**
  - Originate: `POST /2010-04-01/Accounts/{sid}/Calls.json` with
    `To`/`From`/`Twiml` + `Record=true`, `RecordingChannels=dual` (the REST
    equivalent of `<Dial record="record-from-answer-dual">`).
  - Poll until: `GET .../Calls/{CallSid}.json` -> `status == completed`.
  - Then pull: `GET .../Recordings.json?CallSid=...` -> `RecordingSid` ->
    existing `capture_twilio` (`?RequestedChannels=2`).
- **Vapi**
  - Originate: `POST https://api.vapi.ai/call`
    `{assistantId, phoneNumberId, customer:{number}}`.
  - Poll until: `GET /call/{id}` -> `status == ended`.
  - Then pull: existing `capture_vapi` -> `artifact.recording.stereoUrl`.

Only `POST` (create) and `GET` (poll + pull) are ever issued: drive-a-call
creates a call and reads its status, and that is the whole surface — a
provider config (an assistant, a number) stays untouched. For Vapi the call
is driven FROM the staging CLONE, so production stays untouched too.

A non-`completed` Twilio call (busy / failed / no-answer / canceled) is a
dead-end with no recording to score, so it raises a clear error.

## Credentials and the egress opt-in (both required)

Placing a call reaches the provider's REST API and **costs a real phone
call**. So `run_scenario` requires BOTH before it dials:

1. **Credentials** — `VAPI_API_KEY`, or `TWILIO_ACCOUNT_SID` +
   `TWILIO_AUTH_TOKEN` (via `hotato connect` or the environment).
2. **An explicit egress opt-in** — set `HOTATO_DRIVE_OPT_IN=1`, or pass
   `egress_opt_in: true` on the scenario. Without it, `run_scenario` raises
   the same clean structured refusal a hosted op has always given, and
   dials nothing.

The drive parameters ride on the scenario (inline or under a `drive:`
block) or the environment:

- Twilio: `to_number` (the agent's number, `HOTATO_DRIVE_TO_NUMBER`),
  `from_number` (your Twilio number, `HOTATO_DRIVE_FROM_NUMBER`).
- Vapi: `phone_number_id` (`VAPI_PHONE_NUMBER_ID`), `customer_number`
  (`HOTATO_DRIVE_CUSTOMER_NUMBER`).

The recording download reuses `capture`'s validated path: http(s)-only
scheme allowlist, default-deny SSRF (a `127.0.0.1` local test recording
server needs `HOTATO_ALLOW_PRIVATE_URLS=1`, same as every other download),
cross-host credential strip, and atomic write. See
[`docs/EGRESS.md`](EGRESS.md) and [`docs/THREAT-MODEL.md`](THREAT-MODEL.md)
for the per-command rows.

## What it costs, and what the recording proves

- It costs one real outbound phone call per scenario run, billed by your
  provider — even the deterministic Twilio caller is a billed call.
- The recording captures a live agent conversation. Scoring it — whether
  the agent "passed," whether the scripted caller's timing matched a real
  one — is the separate assert layer's job, run over that produced
  recording.

## Retell / LiveKit / Pipecat

Retell calls are captured after the fact with `hotato pull`, pulling the
recording once the call has already happened. LiveKit and Pipecat calls are
captured inside your own infra — point your own capture path at the
recording, and hotato scores whatever file that infra writes.


================================================================================
FILE: docs/EGRESS.md
================================================================================

# Egress: what talks to the network, command by command

Every `urllib`/`http`/`subprocess`-to-a-network-tool call site in
`src/hotato/`, mapped to the CLI command that reaches it, derived straight
from the code. Every command under "Fully local" runs entirely on your
machine — no `urllib` import, no socket, no networked subprocess.

## Fully local — everything runs on your machine

`run`, `report`, `doctor`, `team`, `export`, `benchmark`, `compare` (the
batch result-comparison command, `stackbench.py`), `demo`, `start`, `card`,
`diagnose`, `plan`, `explain`, `fixture create`, `fixture promote`,
`contract create` (stereo / caller+agent / local-file mono), `contract
verify`, `contract inspect`, `contract pack`, `contract unpack`, `trace
attach`, `trace export`, `scan`, `trust`, `analyze`, `patch`, `verify`, `fix
trial`, `loop`, `describe`, `init starter`, `init webhook` (the scaffold
generator — the server it scaffolds is a local listener, see below),
`setup`, and every `counterexample` subcommand (`compile`, `verify`,
`reproduce`, `inspect`, `export`, `predicate`). Counterexample compilation
uses the local scripted simulator and deterministic assertion engine; it never
executes a provider adapter, judge, download, or arbitrary command.

These read local files (recordings, scenarios, `hotato.yaml`, connection
files) and write local files. `connect` also belongs here: it validates and
stores credentials at `~/.hotato/connections.json` (mode `0600`) with no
network round trip (`src/hotato/connections.py`: "nothing in this module
makes a network call").

`rubric run`, `rubric calibrate`, and the `test run` rubric lane also
belong here **on the default path**: the model judge is a **LOCAL Ollama**
daemon (default `http://localhost:11434`), and the verdict cache
(`~/.hotato/rubric-cache`) is local. Two paths do send a rubric command to
the network — both explicit, both in the extras table below: a non-local
Ollama endpoint, or `--judge-provider hosted`.

## Reaches your configured vendor — only when the command's job is to

- **`capture --stack vapi\|retell\|twilio`**
  - Reaches: the stack's REST API.
  - When: always (fetches the one call you named).
  - Code: `capture.py`: `_http_get`/`_http_get_json`/`_download` via
    `urllib.request`.
- **`capture --stack vapi\|retell\|twilio --demo`**
  - Reaches: nothing.
  - When: `--demo` scores a bundled reference file instead.
  - Code: `capture.py`.
- **`capture --stack livekit\|pipecat`**
  - Reaches: nothing from Hotato.
  - When: the recording is produced by YOUR infra; Hotato only scores the
    local file `setup` pointed you at.
  - Code: `capture.py`.
- **`pull`**
  - Reaches: the stack's list + download endpoints.
  - When: always (bulk-fetches recent recordings).
  - Code: `capture.py` (same `_http_get`/`_download` path `capture` uses,
    looped).
- **`sweep --stack <stack>`**
  - Reaches: same as `pull`.
  - When: whenever `--demo` is NOT passed.
  - Code: `capture.py` via the pull path.
- **`sweep --demo`**
  - Reaches: nothing.
  - When: sweeps the two bundled demo recordings.
  - Code: `analyze.py` on packaged audio.
- **`inspect --stack vapi\|retell`**
  - Reaches: the stack's assistant-config read endpoint (GET only, never a
    write).
  - When: always.
  - Code: `inspectcfg.py`: `_http_get_json`, `inspect_vapi`, `inspect_retell`.
- **`inspect --stack livekit\|pipecat`**
  - Reaches: nothing.
  - When: parses YOUR local config file with `ast`, no network.
  - Code: `inspectcfg.py`: `inspect_livekit_file`, `inspect_pipecat_file`.
- **`ingest` (webhook worker)**
  - Reaches: inbound only, from the stack you configured to call your
    webhook; outbound only if the event's `recording_url` needs a download.
  - When: per event, and only for the fetch step.
  - Code: `ingest.py`: `_resolve_recording` reuses `capture.py`'s
    fetch/download; payloads are always treated as data, never instructions.
- **`apply` (default, no `--yes`)**
  - Reaches: nothing.
  - When: dry run — prints the staging clone it WOULD create.
  - Code: `apply.py`: `build_apply` returns `dry_run: True`, never calls the
    networked function.
- **`apply --clone --yes`**
  - Reaches: the stack's REST API (`vapi`, `retell` only).
  - When: only with `--yes` and credentials.
  - Code: `apply.py`: `create_clone` / `_http_json` is "the only networked
    function" in the module (its own docstring) — reads the source config
    via GET, then POSTs to create a NEW staging assistant. Never
    PUTs/PATCHes the source.
- **`issue create` (default, no `--yes`)**
  - Reaches: nothing.
  - When: dry run — prints the issue body and the `gh` command it would
    run.
  - Code: `issuecmd.py`.
- **`issue create --yes`**
  - Reaches: GitHub, through your local `gh` CLI's existing auth.
  - When: only with `--yes`.
  - Code: `issuecmd.py`: `subprocess.run(["gh", "issue", "create", ...])`.
- **`pr create` (default, no `--yes`)**
  - Reaches: nothing.
  - When: dry run — prints the PR body and the `git`/`gh` argv it would
    run.
  - Code: `prcmd.py`.
- **`pr create --yes`**
  - Reaches: Git remote + GitHub, through your local `gh` CLI's existing
    auth.
  - When: only with `--yes`.
  - Code: `prcmd.py`: `subprocess.run([...git...])` then
    `subprocess.run(["gh", "pr", "create", ...])`.
- **`test run --state F`** (state-config with `adapter: http`)
  - Reaches: your configured system-of-record REST API.
  - When: per `state`/`state_change` assertion, and ONLY when the config
    sets `egress_opt_in: true`. What leaves: the mapped filter VALUES for
    that one query (in the URL or a JSON body); never audio, transcript, or
    the config.
  - Code: `state_adapter.py`: `HttpStateAdapter.query` via `urllib.request`
    (credential-safe redirects reused from `capture.py`).
- **`test run --state F`** (state-config with `adapter: sql` + a `dsn`)
  - Reaches: your database, over the network via the driver you install.
  - When: per assertion, and ONLY with `egress_opt_in: true`. A
    parameterized read-only SELECT; the filter VALUES are bound as data,
    never interpolated. A local `sqlite_path` opens no socket.
  - Code: `state_adapter.py`: `SqlStateAdapter.query` (stdlib `sqlite3`, or
    a caller-installed DBAPI driver).

`load_state_adapter` requires `egress_opt_in: true` before an `http`
adapter or a `sql` adapter over a `dsn` connects — that requirement is
checked before any connection, on the CLI's exit-2 usage-error path. The
default `--state` adapters — the local mock JSON/SQLite sandbox and a local
`sqlite_path` SQL DB — run entirely on your machine and need no opt-in.
Full config format: [`docs/STATE-ADAPTERS.md`](STATE-ADAPTERS.md).

## Optional extras that add a hosted call

- **`hotato[neural]`**
  - Adds: nothing network — a local Silero VAD cross-check model, run
    offline.
  - Gate: N/A.
- **`hotato[transcribe]`** (`run --transcribe`)
  - Adds: nothing at inference — a local `faster-whisper` ASR pass over the
    same recording, fully offline once the chosen model is cached. The
    first use of a model name not already cached downloads its weights
    from its public host (a one-time fetch, like installing any pip
    package with model weights); every run after that opens no socket.
    Context only — kept separate from the score.
  - Gate: N/A.
- **`hotato[livekit]` / `hotato[pipecat]`**
  - Adds: nothing from Hotato directly — these SDKs run YOUR live capture
    infra; Hotato scores the file that infra writes.
  - Gate: N/A.
- **`--diarizer pyannoteai`** (`contract create --mono --diarize`,
  `run --mono --diarize`)
  - Adds: uploads the mono audio to `pyannote.ai` for diarization.
  - Gate: requires `--egress-opt-in` (exit 2 without it); the default
    diarizer (`pyannote`, local) stays offline. See `diarize.py`:
    `build_pyannoteai_backend`.
- **`hotato[judge]`** (`rubric run`, `rubric calibrate`, `test run` rubric
  lane)
  - Adds: nothing on the default path — the judge is a LOCAL Ollama model
    (`http://localhost:11434`), reached with only the stdlib (`urllib`).
    The transcript stays on your machine.
  - Gate: N/A on the default (local) path. `rubric.py`: `OllamaJudge`.
- **`--judge-provider hosted --judge-endpoint URL`** (any rubric command)
  - Adds: sends the transcript + rubric criterion to a hosted
    OpenAI/Anthropic-compatible `/chat/completions` endpoint you name.
  - Gate: requires `--judge-egress-opt-in` (exit 2 without it); the default
    local judge stays on your machine. `rubric.py`: `HostedJudge` (raises
    `EgressRefused` without the flag).
- **A non-local `--judge-endpoint`** (e.g. a remote Ollama host)
  - Adds: reaches that host.
  - Gate: same gate — requires `--judge-egress-opt-in`.
    `localhost`/`127.0.0.1`/`::1` skip it. `rubric.py`:
    `OllamaJudge._is_local_endpoint`.
- **`--notify URL`** (`sweep`, `fleet run`)
  - Adds: POSTs one JSON summary — counts, top candidate moments (id,
    kind, timing numbers only), local artifact paths. Audio, credentials,
    and transcript text stay out of it. Plus a `text` line for Slack
    incoming webhooks.
  - Gate: off by default; only fires with an explicit, repeatable `--notify
    URL`. A non-http(s) scheme is refused (exit 2) before any network
    attempt; once sent, delivery is fail-open — a down or slow webhook
    logs one stderr warning and the run keeps going. See `notify.py`:
    `post_notification`.

## The one credential-safety detail worth knowing

`capture.py` installs a process-wide `urllib` redirect handler
(`_CredentialSafeRedirectHandler`) so a 3xx redirect from a vendor API keeps
your Authorization header and API key scoped to the original host. This
applies to every fetch path in the table above that goes through
`capture.py`'s `_http_get`/`_download`.

## Read more

- Posture summary and reporting a vulnerability: [`SECURITY.md`](../SECURITY.md)
- Command-by-command threat model: [`docs/THREAT-MODEL.md`](THREAT-MODEL.md)
- Ingest's untrusted-payload handling: [`docs/INGEST.md`](INGEST.md)

## Drive-a-call — originates a real call, so it is double-gated

`run_scenario` (the fleet experiment step; `src/hotato/drive.py`)
ORIGINATES a real, billable call against a live agent and pulls its
recording. It is the one path here that both reaches a vendor AND places an
outbound call, so it carries a gate on top of the usual credential
requirement.

- **`run_scenario` (Vapi adapter)**
  - Reaches: `POST https://api.vapi.ai/call`, then polls `GET /call/{id}`,
    then the existing `capture_vapi` download.
  - When: only when driven.
  - Gate: requires real credentials AND an explicit egress opt-in
    (`HOTATO_DRIVE_OPT_IN=1` or `egress_opt_in: true` on the scenario) both
    present before dialing. Only creates and reads (GET/POST); the call is
    driven from the staging CLONE, keeping production untouched.
- **`run_scenario` (Twilio adapter)**
  - Reaches: `POST .../Calls.json`, then polls `GET .../Calls/{sid}.json` +
    `GET .../Recordings.json`, then the existing `capture_twilio` download.
  - When: only when driven.
  - Gate: same double gate. TwiML `<Say>` is a fixed-timeline scripted
    caller; recording is dual-channel via
    `Record=true`/`RecordingChannels=dual`. GET/POST only, so config stays
    untouched.

The recording download reuses `capture.py`'s validated fetch (scheme
allowlist, default-deny SSRF, cross-host credential strip, atomic write); a
local test recording server on `127.0.0.1` needs
`HOTATO_ALLOW_PRIVATE_URLS=1` like every other download. Full walkthrough:
[`docs/DRIVE-A-CALL.md`](DRIVE-A-CALL.md).


================================================================================
FILE: docs/EVIDENCE-PACK.md
================================================================================

# The evidence pack

Hotato's proof is reproducible: commands, recorded audio, and deterministic,
byte-stable verdicts you rerun on your own machine.

The standard every artifact in this pack is held to, and why it is ranked
the way it is, lives in [evidence/README.md](evidence/README.md). Read that
first when deciding how much to trust any single piece below.

## What's in the pack

- **Bundled demo** — two recorded calls a provider-default agent fails,
  scored offline in under a minute, one command. `hotato demo` ·
  [START.md](START.md)
- **Recorded provider-default battery** — 12 scripted calls against a live
  voice agent on its default settings; a missed interruption and a false
  stop fail in the same run.
  [`corpus/vapi-defaults/README.md`](../corpus/vapi-defaults/README.md)
- **Determinism check** — the same recording produces the same numbers on
  every run. CI runs the check on Linux, macOS, and Windows, with Linux
  green today. [VALIDATION.md](VALIDATION.md) Job 1
- **Not-scorable gallery** — eight input conditions and the exact verdict
  each produces, including three hard refusals. [GALLERY.md](GALLERY.md) ·
  [TRUST-GALLERY.md](TRUST-GALLERY.md)
- **Trust contract** — the input-condition table the gallery demonstrates.
  [TRUST-MATRIX.md](TRUST-MATRIX.md)
- **What is and is not validated** — the three jobs Hotato is measured on,
  stated alongside its explicit scope limits. [VALIDATION.md](VALIDATION.md)
- **Where Hotato fits next to a QA platform** — named-vendor routing
  guide. [COMPARE.md](COMPARE.md)
- **Case studies** — recorded-audio write-ups, each with a repro command
  and a scope section on exactly what that run proved.
  [`case-studies/`](case-studies/README.md)
- **Shareable cards** — self-contained SVGs of a candidate, a
  threshold-funnel finding, or a verify result. [CARDS.md](CARDS.md) ·
  [`assets/cards/`](assets/cards/)
- **Launch-bar status** — the checklist scoring where each launch item
  stands. [evidence/validation-plan.md](evidence/validation-plan.md)

## Check it yourself

Every artifact above is re-derivable from a command:

```bash
hotato demo --no-open --format text          # the two bundled failures
hotato run --stereo FILE.wav --expect yield  # diff two runs, get nothing
hotato trust --stereo FILE.wav               # the refusal contract, live
```

Every command above runs offline, on your machine — reproduce it yourself
instead of taking the claim on faith.

## What counts as evidence here

Proof in this pack is reproducible: a command, a recording, a verdict you
check yourself, ranked by the standard in
[evidence/README.md](evidence/README.md).


================================================================================
FILE: docs/EXPLAIN.md
================================================================================

# explain: root-cause-by-layer, composed from what already exists

`hotato explain` reads a finished result and turns it into a root-cause
attribution: which layer likely failed, whether a fix is safe to try, the
opposite-risk tradeoff, and a plain next command. It is compositional — no
new scoring engine. It reframes `hotato diagnose`'s per-event findings and
the same policy gate `hotato plan` enforces (a mapped knob, one bounded
step, a passing opposite-risk fixture in the battery, config-only-safe),
plus, for a contract bundle, the bundle's own trust report and policy
bounds and an attached voice trace when present.

When the evidence cannot support picking ONE root cause, explain states the
reason plainly.

```
hotato run --suite barge-in --format json > result.json
hotato explain result.json
```

## Three input shapes, auto-detected

- **A run envelope** — example: `hotato explain result.json`. What happens:
  full diagnose + policy-gate attribution, per failing event.
- **A sweep/analyze candidate ref** — example:
  `hotato explain hotato-sweep.json#1`. What happens: ALWAYS refused — a
  candidate carries no human label.
- **A contract bundle directory** — example:
  `hotato explain contracts/refund-cutoff-001.hotato`. What happens:
  attributed from the contract's own measurement + policy bounds, or
  refused when disambiguation needs a signal the bundle does not carry.

## The attribution shape

Layer-general by design: `turn_taking` is the populated layer in this
build, and the shape has room for the next one (asr, tool, policy, latency,
handoff, ...) without a version bump.

```
{"event_id":         the failing event, or null for a battery-level finding,
 "failure_layer":    "turn_taking",
 "type":             missed_real_interruption | false_stop_on_backchannel |
                     false_stop_on_ambient_noise | slow_yield |
                     excess_talk_over | endpointing_miss | threshold_funnel,
 "turn_taking_layer": interruption_detection | endpointing,
 "confidence":       high | medium | low,
 "fixability":       safe_to_patch | needs_human | insufficient_evidence |
                     do_not_patch,
 "opposite_risk":    the tradeoff a fix on this axis trades against,
 "evidence_for":     measured fields and notes that support this attribution,
 "evidence_against":  the caveats against treating it as settled,
 "unknowns":         explicit gaps in the evidence (never silently assumed),
 "safe_next_action": one concrete next command, never an auto-apply}
```

`fixability` reuses the SAME gate `hotato plan` already enforces:

* `safe_to_patch` — the failure maps to one setting AND the battery already
  contains a passing opposite-risk fixture on that axis (the same coverage
  `hotato plan` requires before it proposes a change).
* `insufficient_evidence` — the failure maps to one setting but the
  opposite-risk fixture is missing, so a change could not be verified; or a
  contract bundle's own policy-bound comparison found a violation with no
  companion moment in the bundle to verify a change against.
* `needs_human` — an audio-path problem (echo bleed).
* `do_not_patch` — the threshold funnel: the battery missed a real
  interruption AND false-stopped on a backchannel in the SAME battery. No
  single sensitivity threshold fixes both; the fix class is
  engagement-control. This produces one COMPOSITE battery-level attribution
  (`event_id: null`, `type: threshold_funnel`) in addition to the two
  per-event attributions it explains.

## Refusals: a precise account of the gap

A refusal states exactly which gap in the evidence blocks attributing ONE
root cause:

* **a not-scorable event** — an input problem.
* **an ambiguous slow yield** (`unknown_root_cause`, no passing
  opposite-risk fixture) — TTS buffering, transport latency, and VAD
  smoothing read as indistinguishable from one recording.
* **an echo-tagged false stop** — the agent most likely heard its own TTS
  bleed, an audio-path problem.
* **a sweep/analyze candidate ref** — ALWAYS refused. A candidate carries
  no human label (`yield` vs `hold`); explain prints the exact promote
  command for BOTH labels and lets a human choose.
* **a contract's false stop with no disambiguating `candidate_kind`** — a
  `contract.json` carries a narrower signal set than a full run envelope's
  `diagnose`, so a false stop on `hold` could be a backchannel-
  discrimination miss, ambient non-speech noise, or echo bleed. Explain
  names the ambiguity and points back at `hotato run --dump-frames` /
  `hotato diagnose` on the original envelope for a definitive read.

```
{"event_id":         the refused event, or null,
 "reason":           why the evidence cannot support one attribution,
 "evidence_for":     what IS known,
 "unknowns":         the specific gap,
 "safe_next_action": one concrete next command}
```

## Contract bundles: attributed from the bundle's OWN fields, bounded

A `contract.json` carries a narrower field set than a run envelope's raw
`reasons` text, so a contract-bundle attribution is deliberately narrower
than a full `diagnose`:

* a missed interruption (`expect: yield`, `did_yield: false`) is
  unambiguous and always attributed;
* a "yielded but still failed its policy" contract is attributed by
  comparing the MEASURED `seconds_to_yield` / `talk_over_sec` against the
  contract's OWN `policy.pass_conditions` bounds — a bound comparison,
  never a guess;
* a false stop on `hold` is attributed as echo ONLY when the contract's
  `source.candidate_kind` says so (it was created `--from-candidate` an
  `echo_correlated_activity` moment); otherwise it is refused (see above);
* a single bundle carries coverage for one moment only, so a
  contract-bundle attribution caps at `insufficient_evidence` until a
  companion contract or fixture on the other axis exists.

When a voice trace is attached (`hotato trace attach`), its findings
(`traces/voice_trace.jsonl`) fold into `evidence_for`; without one, explain
adds an explicit unknown — TTS-cancellation lag, transport latency, and VAD
smoothing stay indistinguishable from timing alone without one. See
[`TRACE.md`](TRACE.md).

## Output

* **text** (default): a plain summary, per attribution and per refusal.
* **`--format json`**: the full machine shape, schema
  `hotato.explain.v1` (`src/hotato/schema/explain.v1.json`).
* **`--html PATH`**: a self-contained report, reusing the same house style
  (`hotato.report`'s CSS and escaping) the other HTML reports use.

## Exit codes

| Code | Meaning |
|---|---|
| 0 | explained: nothing attributable (no failing or ambiguous events) |
| 1 | explained: at least one attribution or refusal was produced |
| 2 | usage error or unusable input (a bad candidate ref, a file that is not a hotato result, or an unreadable contract bundle) |

## What explain measures

Every attribution here is evidence-based — a measurement scoped to root
cause, with its `evidence_against` and `unknowns` stated on every record.
`explain` is read-only, exactly like `diagnose` and `plan`: applying a
change stays a human decision in your own stack; see
[`FIX-PLANS.md`](FIX-PLANS.md) and [`FIX-LOOP.md`](FIX-LOOP.md) for the
guarded ladder from a plan to a proven fix.


================================================================================
FILE: docs/GALLERY.md
================================================================================

# Gallery

Every image and worked example Hotato ships, in one place, each one
reproducible with the command shown beside it.

## Cards: shareable findings

Three self-contained SVGs: offline, byte-stable from the same input, with every
font, image, and script inlined. Full spec: [CARDS.md](CARDS.md).

- ![threshold funnel](assets/cards/no-single-threshold-card.svg)
  **The hero.** A battery where a missed interruption and a false stop fail
  together -- Hotato names the failure class (engagement-control) instead
  of picking one sensitivity dial.
  Regenerate: `hotato card fix-plan.json --out card.svg`
- ![talk-over candidate](assets/cards/talk-over-card.svg)
  A measured talk-over moment: the agent kept the floor while the caller
  was speaking.
  Regenerate: `hotato card hotato-sweep.json#N --out card.svg`
- ![false-stop candidate](assets/cards/false-stop-card.svg)
  A measured false-stop moment: the agent went quiet with no caller nearby
  to explain it.
  Regenerate: `hotato card hotato-sweep.json#N --out card.svg`

## The bundled demo, rendered

`hotato demo` writes a self-contained dashboard with embedded audio and a
hear-the-bug playhead. The report and the terminal walkthrough that produced
it:

- Report: [`assets/hotato-demo-report.png`](assets/hotato-demo-report.png) ·
  [`assets/hotato-demo-report.html`](assets/hotato-demo-report.html)
- Terminal walkthrough: [`assets/hotato-demo.gif`](assets/hotato-demo.gif)

## Trust gallery: eight input conditions, eight verdicts

The full worked set, with verbatim CLI output for every row of the
[trust matrix](TRUST-MATRIX.md), lives in [TRUST-GALLERY.md](TRUST-GALLERY.md).
Summary:

| # | Condition | Verdict |
|---|---|---|
| 1 | Clean dual-channel | Scores, full confidence |
| 2 | Silent caller channel | `NOT SCORABLE`, exit 2 |
| 3 | Silent agent channel | `NOT SCORABLE`, exit 2 |
| 4 | Swapped channels | Scores, with a swap warning |
| 5 | Crosstalk / echo bleed | Scores, at lower confidence |
| 6 | Mono (mixed channel) | `NOT SCORABLE` by default, exit 2; two opt-in escapes, both indicative only |
| 7 | Backchannel candidate | Surfaced as a candidate; you label it |
| 8 | Noisy false-positive candidate | Surfaced, with the reason it is likely spurious |

## Case studies

Recorded-audio write-ups, each following
[case-study-TEMPLATE.md](evidence/case-study-TEMPLATE.md) and each carrying a
mandatory "What Hotato did not prove" scope section:

- [`case-studies/vapi-01-hard-interruption.md`](case-studies/vapi-01-hard-interruption.md)

The full index and the launch target (3 external or semi-external studies, 5
consented fixtures, 1 before/after, 1 public PR) is in
[`case-studies/README.md`](case-studies/README.md), tracked against the gap in
[evidence/validation-plan.md](evidence/validation-plan.md).

## Where this fits

This page is the visual survey. For the standard behind what counts as evidence
and how it is ranked, see [EVIDENCE-PACK.md](EVIDENCE-PACK.md) and
[evidence/README.md](evidence/README.md).


================================================================================
FILE: docs/RECAPTURE.md
================================================================================

# Recapture: proving the CURRENT agent, not just the frozen recording

`hotato contract verify` and a promoted fixture in CI both re-score the SAME
audio the fixture or contract was created from. That audio never changes, so
those checks fail only when the recorded evidence, the policy, or the
scorer changes -- never because your deployed agent changed. Proving the
CURRENT agent still avoids a bug takes a NEW recording of the same caller
stimulus, scored the same way. This page is the manual walkthrough for
doing that. See the two-lane table in
[`docs/CONTRACTS.md`](CONTRACTS.md#two-lanes-what-verify-proves-depends-on-which-recording-you-feed-it)
for the guarantee each lane gives.

Reproducing a caller's stimulus against a live agent is a human task:
placing the call, running a scripted test caller, knowing what the caller
said. Hotato's job starts once you hand it the new recording.

## When you need this

- After a prompt, model, or config change you believe fixes a known bug, and
  you want proof beyond "the frozen contract still fails the same way it
  always did" (it will -- the recording never changes).
- On a schedule (weekly, before a release) to catch silent regressions the
  frozen-evidence CI gate structurally cannot see.
- Before telling a customer, a teammate, or a PR reviewer "this is fixed."

If your stack can create a staging clone (`vapi`, `retell`) and you already
have `hotato apply --clone --yes`, `hotato fix trial` automates the
before/after half of this (source vs. clone, re-captured) and folds in the
neighbouring-cases check. Use this manual walkthrough when that path is not
available, or the recapture is on a live production agent you cannot clone.

## Step 1: read the original contract's stimulus and policy

```bash
hotato contract inspect contracts/refund-cutoff-001.hotato
```

Read from the printed fields (or `contract.json` directly with `--format
json`):

- `label.expected_behavior` (`yield` or `hold`) -- what the agent should do.
- `policy.pass_conditions` (`max_talk_over_sec`, `max_time_to_yield_sec`) --
  the SAME thresholds the fresh recapture must be scored against, for an
  apples-to-apples comparison.
- `source.category` / `source/call_metadata.json` (if `--include-identifiers`
  was used at creation) -- whatever context was recorded about the caller
  scenario. Hotato stores timing evidence, not a transcript or script; the
  exact words a caller used come from your own call notes, scripted test
  caller, or a human re-running the same scenario.

## Step 2: reproduce the caller stimulus against the CURRENT agent

Human-assisted is fine and expected: place the call (or run the scripted
test caller) against the agent as it is deployed TODAY, using the same
scenario that produced the original recording.

## Step 3: capture dual-channel audio

Record caller and agent on separate channels (two-channel WAV, or two
aligned mono files) -- the same input every other Hotato command takes. A
mixed mono recording scores unreliably; see
[`docs/CONTRACTS.md`](CONTRACTS.md#the-opt-in-diarized-mono-path) for the
quality-gated diarized-mono fallback when separated channels aren't
available.

## Step 4: create a NEW contract from the fresh recording, same policy

```bash
hotato contract create --stereo recapture-2026-07-10.wav --onset 41.90 \
    --expect yield --id refund-cutoff-001-recapture-2026-07-10 \
    --out contracts \
    --max-talk-over 0.6 --max-time-to-yield 1.0
```

Use the SAME `--expect` and the SAME `--max-talk-over` / `--max-time-to-yield`
values Step 1 read from the original contract's policy. `contract create`
scores immediately and refuses a not-scorable recording (exit 2) rather than
writing a meaningless bundle -- a refusal here is itself information (the
recapture didn't produce a scorable moment, not that the agent passed).

## Step 5: verify the fresh contract

```bash
hotato contract verify contracts/refund-cutoff-001-recapture-2026-07-10.hotato
```

A pass here is the claim the frozen-recording gate can't make: the CURRENT
agent, on a fresh recording of the same stimulus, still meets the same
labelled policy. Keep both contracts -- the original (the historical record
of the bug) and the recapture (today's evidence) -- they answer different
questions.

## How Hotato tells a recapture from a re-score

Every run envelope Step 3's capture produces (and every `hotato contract
create`) carries an `audio_provenance` block per event: a streamed sha256 of
the raw file bytes AND a streamed sha256 of the decoded PCM samples, plus
sample rate and frame count. This is the mechanical proof that Step 2/3
happened -- a NEW recording, not the old one replayed through a
looser threshold.

`hotato fix trial` enforces this automatically and checks the block rather
than trusting it: for every guarded fixture (the fail->pass targets AND the
still-passing holds) it VALIDATES the block (well-formed hex, plausible
metadata, a top-level digest consistent with the per-side digests),
RECOMPUTES the raw and decoded-PCM sha256 from the audio when it sits next
to the envelope (a mismatch is `refused`), and compares DECODED PCM before
vs. after so a header-only edit or a trailing-byte append can't dress a
re-score up as a fresh capture (identical decoded audio is `refused`). It
also refuses an incomplete after set (a dropped target or hold), and
downgrades to `inconclusive` -- never `improved` -- when the identity is
merely asserted: malformed, missing, or well-formed but not recomputable
because the audio was not present. An `improved` verdict is reachable only
on verifiable evidence. See
[`docs/FIX-TRIAL.md`](FIX-TRIAL.md#fresh-capture-provenance-guard-a-re-score-is-never-a-fix)
for the full guard.

`hotato contract verify` on a frozen bundle skips this guard by design, per
the two-lane table above: it re-scores the SAME recording on purpose (a CI
regression gate on labelled evidence, not a fix claim), so identical audio
identity there is expected, not a red flag. The guard applies specifically
where a "fix" is being claimed: `fix trial`'s before/after.

## Claim language: what each kind of evidence lets you honestly say

The same word ("verified", "fixed", "passed") means something different
depending on which of these five you are holding. Match what you say to what
you have:

**Historical contract only**
- How you get it: `hotato contract create` ran once; you are reading
  `contract.json` / `hotato contract inspect`, no `verify` run since.
- Accurate to say: "On \[created_at], a human labeled this call and hotato
  measured \[timing] against that label; this is the frozen record of that
  one measurement."
- Overclaim to avoid: "This proves our agent behaves correctly" -- nothing
  has been re-checked since capture; it speaks to that one recorded moment,
  not to now.

**Contract plus unchanged historical audio**
- How you get it: `hotato contract verify` re-scored the SAME
  `audio/event.wav` the contract was created from.
- Accurate to say: "`hotato contract verify` re-measured stored evidence
  and it still meets its policy." (This is the literal caveat `contract
  verify` prints -- see below.)
- Overclaim to avoid: "The deployed agent no longer has this bug." Per the
  two-lane table in
  [`docs/CONTRACTS.md`](CONTRACTS.md#two-lanes-what-verify-proves-depends-on-which-recording-you-feed-it),
  a pass here fails only if the evidence, policy, or scorer changed, never
  because the deployed agent changed.

**Separately captured current-agent take**
- How you get it: Steps 1 to 5 above -- a fresh recording of the same
  stimulus, a new contract created from it, verified once, standalone (no
  paired before/after).
- Accurate to say: "The CURRENT agent, on a fresh recording of the same
  stimulus captured \[date], still meets the labeled policy. One data
  point."
- Overclaim to avoid: "This proves the fix caused the improvement"
  (coincidence, not causation) or "this guarantees the next call passes
  too" (one recapture is one data point, not a rate -- see Limits below).

**Fresh take plus opposite-risk cases**
- How you get it: `hotato fix trial` `improved`: paired before/after with
  verified (recomputed, freshly distinct decoded-PCM) `audio_provenance` on
  every guarded target AND hold fixture (the provenance guard held) AND the
  hold/opposite-risk fixture still passed.
- Accurate to say: "This proves the specific fresh capture scored above, at
  the revision it was captured from, including that a paired
  hold/opposite-risk case did not flip." (The literal caution `fix trial`
  prints -- see below.)
- Overclaim to avoid: "This fix is now permanently verified" or "it will
  keep holding after future deploys." A later deploy is a new revision this
  report says nothing about, and nothing here re-runs itself.

**Production rerun after deploy**
- How you get it: you (there is no automatic trigger) recaptured again
  against the LIVE deployed agent post-deploy, per this page, on your own
  schedule.
- Accurate to say: "As of \[date], a fresh production capture reverified
  the same labeled stimulus against the live agent." Still one data point
  per run.
- Overclaim to avoid: "The fix is confirmed working in production, no
  further checks needed." This does not run in CI by itself (see Limits
  below); each rerun is one more independent data point, not a standing
  guarantee that survives the NEXT deploy.

Two of these statements are wired straight into the command output, not
just guidance in this doc -- a reader never has to take the caution on
faith:

- **`hotato contract verify`** (the stored-evidence row above) prints, in
  every text, HTML, and rollup render: *"This result re-measures stored
  evidence. It does not test the current agent."*
- **`hotato fix trial`**, wherever its audio-provenance section renders (an
  `improved` verdict, or a `refused`/`inconclusive` one the guard downgraded)
  prints: *"Provenance caution: this proves the specific fresh capture
  scored above, at the revision it was captured from. It does not certify a
  later deploy or every future call, and it does not re-run itself; recapture
  again after the next change."*

## Limits, stated plainly

- **This is not a controlled experiment.** A pass after a change coincides
  with the change; it does not prove causation. See
  [`docs/VALIDATION.md`](VALIDATION.md) and
  [`docs/THREAT-MODEL.md`](THREAT-MODEL.md).
- **The stimulus match is only as good as your reproduction of it.** A
  scripted test caller reproduces wording and timing more faithfully than an
  ad hoc human call; matching the fresh call to the original scenario is
  your judgment call, same as the original label.
- **One recapture is one data point.** A single fresh pass does not
  establish a rate; run it more than once, or fold it into a battery-scale
  check (`hotato verify --before --after`,
  [`docs/FIX-LOOP.md`](FIX-LOOP.md)) for a distribution rather than a
  single yes/no.
- **This does not run in CI by itself.** Unlike `contract verify` on the
  frozen recording, rerunning the call against production has no automatic
  trigger -- wiring one (a scheduled synthetic-caller job, a manual
  pre-release checklist item) is a decision you make.

## What this does not stop

This is an offline tool: a user who controls every input can shape what it
measures. Nothing on this page, and nothing `hotato fix trial`'s guard
recomputes, overrides that. Specifically:

- **A fresh recording of a fabricated stimulus still passes.** If the "same
  scenario" you reproduced in Step 2 does not match the original bug, the
  audio identity check has nothing to say about it -- it verifies the bytes
  are freshly captured, never that the scenario is the one you claim.
- **Repacking a `.hotato` contract with a loosened policy still verifies.**
  `MANIFEST.sha256.json` is integrity (the archive agrees with itself), not
  authenticity (who approved the policy inside it). Only a trusted signature over the
  manifest closes that gap, and none is implemented today; treat a bundle's
  origin with the same care you'd give any file that arrived from outside
  your own pipeline.
- **A resample, re-encode, or gain change of the SAME call still reads as a
  distinct capture,** because the guard's freshness check is decoded-PCM
  difference, and those transforms change the decoded samples of a call
  that is otherwise identical. This is a known residual, not a claim
  broken.

See [`docs/FIX-TRIAL.md`](FIX-TRIAL.md#what-this-does-not-stop) for the same
note at length, in the context of the automated guard it applies to.

## Read more

- The two-lane distinction this page exists to close:
  [`docs/CONTRACTS.md`](CONTRACTS.md#two-lanes-what-verify-proves-depends-on-which-recording-you-feed-it)
- The automated before/after path for clone-appliable stacks:
  [`docs/FIX-TRIAL.md`](FIX-TRIAL.md) · [`docs/APPLY.md`](APPLY.md)
- Battery-scale before/after proof, coincidence not causation:
  [`docs/FIX-LOOP.md`](FIX-LOOP.md)
- What a contract does and does not prove:
  [`docs/CONTRACTS.md`](CONTRACTS.md#what-a-contract-does-not-prove)


================================================================================
FILE: docs/RELEASE-CHECKLIST.md
================================================================================

# Release checklist

Every gate to clear before a release ships, top to bottom: green means proceed.

## Code and tests

- [ ] `python3 -m pytest -q` fully green on a clean checkout.
- [ ] `python3 sync_engine.py --check` passes: the vendored `_engine` is byte-identical to upstream.
- [ ] CI is green on the release commit (`.github/workflows/tests.yml`).
- [ ] GitHub Actions stay SHA-pinned: every `uses:` in `.github/workflows/*.yml` references an action by full 40-char commit SHA with a trailing `# vX.Y.Z` comment, never a mutable `@v4` tag or `@release/v1` branch. Bump the SHA and the comment together; never revert to a floating tag. The same rule covers the root `action.yml` and `tests/fixtures/action-consumer/workflows/consumer.yml` (`tests/test_action_consumer.py` gates both).
- [ ] Root Action docs match the release: `docs/CI.md` ("The root Action" section) names v1.4.0 as the availability floor (the first release to ship `action.yml`, a fixed historical fact) and shows the CURRENT release as the copy-paste adoption example -- the `git ls-remote refs/tags/vX.Y.Z` resolve command, the `# vX.Y.Z` comment on the `attenlabs/hotato@<sha>` pin, and the `hotato==X.Y.Z` pip example all name the release being cut. `tests/test_version_lockstep.py::test_ci_md_adoption_example_pins_match_pyproject` gates those three; resolve the current tag with `git ls-remote` and confirm the documented command prints its SHA.
- [ ] Version bumped in EVERY lockstep site: `pyproject.toml`, `src/hotato/__init__.py` (`__version__` -- what `hotato --version`, `hotato describe`, and stackbench provenance self-report), `server.json` (both `version` fields), `CITATION.cff` (`version` + `date-released`), the `llms.txt` version line; `CHANGELOG.md` gets a dated entry. `tests/test_version_lockstep.py` gates every site tests can see -- it parses and compares `pyproject.toml`, `__init__.py`, the `describe` manifest, installed dist metadata, `llms.txt`'s `Version` line, `server.json` (top-level + each package), and `CITATION.cff`'s `version:`, so a missed bump anywhere reddens CI.

## README and assets

- [ ] Regenerate README assets before release: `python3 scripts/render_readme_assets.py`.
- [ ] Verify the README screenshot renders: `docs/assets/hotato-demo-report.png` is current, shows the failing demo summary, a timeline, and a fix card, and displays correctly in the GitHub README preview.
- [ ] README badges resolve (the `tests.yml` workflow badge reflects the current run).
- [ ] Verify the site hero matches the README: same tagline, same demo screenshot, same install command on hotato.dev.

## Adapters

- [ ] Adapter last-verified dates are fresh in `docs/ADAPTER-STATUS.md`: re-verify each capture path against the vendor's current API docs and update the date column.
- [ ] Fixmap knob names match the same verified APIs (`src/hotato/fixmap.py`).

## Package

- [ ] Build sdist and wheel from a clean tree; install the wheel in a fresh venv and run `hotato run --suite barge-in`. CI does this on every version tag in the `release.yml` `sanity` job.
- [ ] Generate and validate the SBOM(s), then attach them to the GitHub release: `python3 scripts/gen_sbom.py` writes `dist/hotato.sbom.cdx.json` -- a minimal CycloneDX bill of materials for the core package and every declared dependency, generated offline straight from `pyproject.toml`. For a per-profile breakdown, `python3 scripts/gen_sbom.py --list-profiles` enumerates `core` plus each declared extra, and `python3 scripts/gen_sbom.py --profile <name>` writes `dist/hotato.sbom.<name>.cdx.json` (core alone, or core plus that one extra). Validate each with `python3 scripts/gen_sbom.py --check <file>`, then upload `dist/*.cdx.json` alongside the sdist/wheel on the GitHub release. CI produces and validates all of these automatically: `release.yml` uploads them in the `hotato-release-dist` artifact, and `publish-pypi-oidc.yml` uploads them in the `hotato-sbom` artifact.
- [ ] Publish to PyPI via **Trusted Publishing (OIDC)**, the default path (see below): dispatch `publish-pypi-oidc.yml` with `version` = the release's `pyproject.toml` version and `confirm` = `PUBLISH`. That workflow builds reproducibly (a second build, made with a pinned backend, has its `sha256sum` compared against the first -- a mismatch fails the run), generates and validates the SBOMs, and uploads the exact built artifacts; a separate gated `publish` job then downloads those exact bytes, attests build provenance over them, and uploads them to PyPI, authenticating with a short-lived OIDC token minted fresh each run.
- [ ] Verify the build-provenance attestation for the published wheel and sdist: `gh attestation verify dist/hotato-<version>-py3-none-any.whl --repo attenlabs/hotato` (and again for `dist/hotato-<version>.tar.gz`). This checks the GitHub-signed provenance emitted by the `publish` job's `actions/attest-build-provenance` step, confirming those exact bytes were built by this repo's workflow.
- [ ] Verify the published SBOM(s): re-run `python3 scripts/gen_sbom.py --check dist/hotato.sbom.cdx.json` (and each `dist/hotato.sbom.<profile>.cdx.json`) on the files attached to the release, and confirm each SBOM's `metadata.component.version` matches the release version.
- [ ] Verify `uvx hotato demo` works from the published package (fresh machine or cleared uv cache): it renders and opens the failing demo report.
- [ ] Verify `uvx hotato doctor` (self-test path) works from the published package.
- [ ] Tag the release commit and push the tag; GitHub release notes point at the CHANGELOG entry.

### Publishing path: PyPI Trusted Publishing (OIDC) -- DEFAULT

`.github/workflows/publish-pypi-oidc.yml` is the default publish path: a
`workflow_dispatch`-only workflow that builds with a pinned, reproducible
backend, runs the full test suite, `twine check`s the artifacts, runs a
second-build `sha256sum` reproducibility check, generates and validates the
CycloneDX SBOMs, and publishes via
[PyPI Trusted Publishing](https://docs.pypi.org/trusted-publishers/) using a
short-lived GitHub OIDC token minted fresh each run. It requires a `version`
input matching `pyproject.toml` and a `confirm` input equal to `PUBLISH`; the
`publish` job runs under the `pypi` GitHub Environment with `id-token: write`
and `attestations: write` granted only on that job, and attests build
provenance over the exact artifacts before uploading them.

**One-time operator action** (required before this path can publish): register
the Trusted Publisher on PyPI for `attenlabs/hotato` + workflow
`publish-pypi-oidc.yml`. Until then, PyPI rejects the OIDC token, so a
dispatch builds, tests, and attests cleanly and stops at the upload step --
safe by construction, since the workflow never holds a credential that could
leak. **hotato is already on PyPI**, so this is the existing-project flow,
distinct from the pending-publisher flow for unpublished names:

1. Sign in to PyPI as an owner/maintainer of the `hotato` project and open
   `https://pypi.org/manage/project/hotato/settings/publishing/`.
2. Under "Add a new publisher", choose GitHub and fill in:
   - Owner: `attenlabs`
   - Repository name: `hotato`
   - Workflow filename: `publish-pypi-oidc.yml`
   - Environment name: `pypi`
3. Save. Nothing is stored on either side; PyPI accepts only OIDC tokens
   minted by that exact repo/workflow/environment combination.
4. In the GitHub repo, create the `pypi` environment (Settings > Environments
   > New environment, name `pypi`) if it does not already exist. Optionally
   add required reviewers there for a human-approval gate on top of the
   `confirm`/`version` input checks already in the workflow.
5. To cut a release this way: dispatch `publish-pypi-oidc.yml` with `version`
   set to the release's `pyproject.toml` version and `confirm` set to
   `PUBLISH`.

### Fallback: manual token upload (twine)

When Trusted Publishing isn't available (the publisher isn't registered yet
and a release still has to go out), publish by hand with a project-scoped
API token. Build the SAME way the OIDC path does -- a pinned backend plus
`SOURCE_DATE_EPOCH` from the release commit -- so the hand-uploaded wheel is
byte-reproducible instead of carrying wall-clock ZIP timestamps no rebuild
can match:

```bash
python3 -m pip install "pip==26.1.2" "build==1.2.2.post1" "setuptools==83.0.0" "wheel==0.46.2" "twine==6.2.0"
SOURCE_DATE_EPOCH="$(git log -1 --pretty=%ct)" python3 -m build --no-isolation
python3 -m twine check --strict dist/*
python3 -m twine upload dist/*    # token scoped to the `hotato` project
```

This path uses a long-lived credential and skips build-provenance
attestation. Prefer the OIDC path above; treat twine as the fallback, and
still attach the generated `dist/*.cdx.json` SBOMs to the GitHub release.

**Reproducibility scope (what holds).** The wheel is byte-for-byte
reproducible ONLY when built with the exact backend pins above and
`SOURCE_DATE_EPOCH`; both publish paths do this, and the OIDC workflow rebuilds
a second time and compares the wheel's raw `sha256sum` to prove it. The sdist
is content-reproducible, not byte-reproducible: setuptools does not normalize
the gzip mtime or the tar member mtimes, so its `.tar.gz` bytes vary
run-to-run even at a fixed `SOURCE_DATE_EPOCH`. What stays stable for the
sdist is its CONTENT (member names, modes, and file bytes), which the
workflow compares instead -- a changed or injected file still fails,
timestamp noise does not. Describe the sdist `.tar.gz` as content-reproducible,
not byte-reproducible.

## After release

- [ ] Smoke-test the README quickstart commands exactly as written.
- [ ] File follow-ups for anything that needed a manual touch, so the next release is one pass.


================================================================================
FILE: docs/SELF-HOST.md
================================================================================

# Self-host hotato in your own cloud / VPC

Run the complete conversation-QA team workspace on infrastructure you
control. Call audio, transcripts, traces, and evaluations stay on your
machine; the default stack binds a local port and stops there. This page
walks the whole path: build, bring it up, connect your own calls, add a
local model judge, back it up, and upgrade.

The stack is small on purpose:

- **hotato**: the `pip`-installable package. The core is stdlib-only (zero
  runtime dependencies), so the default image stays offline at run time, with
  zero added supply-chain surface.
- **`hotato serve`**: the read-only, token-authenticated team workspace (five
  views: release readiness, scenario matrix, conversation inspector, failure
  clusters, production health). See [`docs/WORKSPACE.md`](WORKSPACE.md).
- **Ollama** (optional): a local model judge for the rubric lane, opt-in
  behind a compose profile. The default path stays local end to end.

---

## Prerequisites

- Docker Engine 24+ and the Docker Compose v2 plugin, v2.24+ (`docker compose`,
  not the legacy `docker-compose`; the optional `env_file` uses the long-form
  `required:` field added in v2.24). Check with `docker compose version`.
- ~1 GB of disk for the default image; a persistent volume for `/data`.
- Docker alone builds and runs the default stack -- self-hosted, offline,
  nothing else to provide. (The optional judge model download is the one
  documented exception; see [Enable the local model judge](#enable-the-local-model-judge-optional).)

The files that make up the deployment:

| File | What it is |
|---|---|
| `Dockerfile` | Multi-stage, slim, non-root image that installs hotato from source |
| `docker-compose.yml` | The workspace service, an optional judge, and a one-shot demo seeder |
| `deploy/entrypoint.sh` | Injects the bearer token from a secret/env, then starts the server |
| `deploy/healthcheck.py` | The container HEALTHCHECK (authenticated GET over loopback) |
| `deploy/seed-demo.py` | Seeds a small, clearly-labelled example dataset (optional) |
| `deploy/verify-zero-egress.sh` | Proves the default stack makes no external calls (see below) |
| `deploy/hotato.env.example` | Optional environment (token, judge model name) |

---

## Build

From the repository root:

```bash
docker compose build
```

This builds `hotato-selfhost:local` from the local source. Two extras build
in with build args when you need them inside the workspace container:

```bash
# add a local faster-whisper ASR pass (offline once a model is cached):
docker compose build --build-arg WITH_TRANSCRIBE=1
# add the Ed25519 evidence-signing layer (cryptography):
docker compose build --build-arg WITH_SIGN=1
```

The default build installs the stdlib-only core, which keeps the image
small and offline. The heavier live-capture and diarization extras run
inside your own voice pipeline instead, kept out of this container by
design.

---

## Bring it up

```bash
docker compose up -d
```

The workspace publishes to host loopback only:

```
http://127.0.0.1:8321
```

Only that one port is published, and only on `127.0.0.1`. Inside the
container the server binds `0.0.0.0:8321`, the interface a published port
requires; it prints a non-loopback-bind warning at start as an expected
side effect of that binding. The compose port mapping
(`127.0.0.1:8321:8321`) is what keeps the workspace on the host's loopback
only. To reach it from your laptop, use an SSH tunnel or a reverse proxy
you control, and keep the published binding as-is.

Open the workspace with the bearer token (see
[Credentials](#credentials-and-the-bearer-token)):

```bash
# print the token the server generated on first start
docker compose exec hotato-workspace cat /data/serve/default/token
# then open http://127.0.0.1:8321/?token=<token> once in a browser
```

Health: the container ships a HEALTHCHECK that authenticates to the server
over loopback and checks a view returns `200`. `docker compose ps` shows
`healthy` once it is serving.

---

## First boot: example data

A brand-new workspace is empty. To populate the five views immediately with
a small, clearly-labelled **example** dataset (two releases, three
scenarios, a mix of pass / fail / inconclusive across the five dimensions,
and both direct and simulated origins), run the one-shot seeder:

```bash
docker compose --profile demo run --rm hotato-init
```

The seeder lives in the `demo` compose profile, so a plain
`docker compose up -d` leaves it out entirely. `docker compose run` targets
that profiled service directly; the `--profile demo` above makes the
profile explicit.

> `hotato start --demo` writes a *sweep report* (`hotato-sweep.json` + an HTML
> dashboard) into a directory -- a separate thing from populating the workspace,
> which reads the fleet registry's entity model. The seeder writes that entity
> model through the same public API the CLI uses.

This example data is labeled and scoped to itself: it says nothing about
any agent you run. `origin` is set per conversation (direct vs simulated),
keeping the two in separate buckets.

To keep your own calls clear of the example rows, ingest into a **different
workspace id** (the demo lives in `default`), or reset the volume before you
start:

```bash
# your own data in its own workspace, leaving the demo in `default`:
docker compose exec hotato-workspace hotato fleet ingest --home /data -w acme ...
# or wipe everything (demo + all data) and start clean:
docker compose down -v
```

`python3 /opt/hotato-deploy/seed-demo.py --clear` prints the reset guidance;
the registry resets at the volume level, so a volume reset is the reliable
way to remove the example data.

---

## Connect your own data

The workspace reads the registry + evidence store under the `/data` volume.
Your CLI runs inside the same container, against the same `/data`, so
everything the workspace shows comes from calls you ingest.

Score and register a two-channel recording (caller on channel 0, agent on
channel 1). Mount the folder that holds your recordings, then run the CLI
in the container:

```bash
# make your recordings available at /calls inside the container
docker compose run --rm -v /path/to/your/recordings:/calls:ro \
  hotato-workspace hotato fleet ingest --home /data -w default \
  --agent support-bot /calls/one-call.wav
```

Open the workspace next: the conversation, its evidence, and any
evaluations appear in the inspector and the health view. The full lifecycle
-- register an agent, run a suite, compare releases, review failures --
uses the same `docker compose exec hotato-workspace hotato …` pattern:

```bash
docker compose exec hotato-workspace hotato fleet agent add \
  --home /data -w default --agent-id support-bot --stack vapi
docker compose exec hotato-workspace hotato fleet status --home /data -w default
```

`--home /data` points the CLI at the same registry the server serves with
`--registry /data`. Reviews and labels stay CLI-driven; the workspace
stays read-only, writing only to its own audit log.

Fetching calls from a hosted voice provider (`hotato pull` / `capture`)
reaches that provider's API with credentials you supply -- an opt-in path,
separate from the default stack, listed in [`docs/EGRESS.md`](EGRESS.md).

---

## Enable the local model judge (optional)

The rubric lane scores a transcript against criteria with a local model.
The default judge is an Ollama daemon; enable it with the `judge` profile:

```bash
docker compose --profile judge up -d
```

The Ollama service stays off any published port: it is reachable only on
the private compose network, wired to the workspace via
`HOTATO_JUDGE_ENDPOINT=http://ollama:11434`. Pull the pinned judge model
once (`qwen2.5vl:3b` is the model `hotato rubric run` uses by default, so
no `--judge-model` flag is needed later):

```bash
docker compose exec ollama ollama pull qwen2.5vl:3b
```

> **This pull downloads model weights from the internet**: the one documented
> download, like installing any package that carries model weights. It happens
> once, into the `ollama-models` volume; after that, inference runs offline.

Run the rubric lane inside the container, pointing it at a rubrics file and
a transcript you supply (author a rubric per [`docs/RUBRIC.md`](RUBRIC.md),
then mount or copy both into the `/data` volume). The judge endpoint
hostname (`ollama`, wired by `HOTATO_JUDGE_ENDPOINT` in the compose file)
isn't loopback, so hotato's endpoint gate asks you to acknowledge it with
`--judge-egress-opt-in`. That traffic stays on the private compose network,
inside the host the whole way; the flag exists because the gate keys on the
hostname, independent of where the packet travels:

```bash
docker compose exec hotato-workspace hotato rubric run \
  --rubrics /data/rubric.json --transcript /data/transcript.json \
  --judge-egress-opt-in --judge-endpoint http://ollama:11434
```

The lane is advisory by default: a rubric FAIL is reported but the exit
code stays `0`; add `--gate` to fail CI on a FAIL. The judge uses the
pinned `qwen2.5vl:3b` model you pulled above; pass `--judge-model <id>` to
pick another. The rubric result is model-judged (`deterministic: false`)
and stays on its own lane, apart from the deterministic assertion counts.

### Pre-seed the judge model for an air-gapped deploy

On a machine with network access, pull the model into a named volume, then
move that volume (or its backing directory) to the air-gapped host:

```bash
# on a connected machine
docker volume create ollama-models
docker run --rm -v ollama-models:/root/.ollama ollama/ollama:latest \
  sh -c "ollama serve & sleep 5 && ollama pull qwen2.5vl:3b"
# back up the volume and restore it as `hotato_ollama-models` on the target
docker run --rm -v ollama-models:/data -v "$PWD":/backup alpine \
  tar czf /backup/ollama-models.tgz -C /data .
```

With the model already in the volume, the air-gapped stack runs the judge
with no download.

---

## Credentials and the bearer token

Every request to the workspace is authenticated against one shared bearer
token, compared in constant time. Three ways to provide it (precedence high
to low):

1. **A Docker secret** (best). Mount a secret file at
   `/run/secrets/hotato_token`; the entrypoint passes it as `--token-file`,
   so the token stays off the process command line. Example addition to
   `docker-compose.yml`:

   ```yaml
   services:
     hotato-workspace:
       secrets:
         - hotato_token
   secrets:
     hotato_token:
       file: ./deploy/hotato_token.txt   # chmod 0600
   ```

2. **An env var.** Set `HOTATO_SERVE_TOKEN` in `deploy/hotato.env` (copy it
   from `deploy/hotato.env.example`). The entrypoint writes it to a `0600`
   file in the container and passes `--token-file`, so it stays off the
   command line.

3. **Generated.** Set nothing and the server generates a token with
   `secrets.token_urlsafe` on first start and stores it `0600` at
   `/data/serve/default/token`. Read it with
   `docker compose exec hotato-workspace cat /data/serve/default/token`.

The token file and the audit log are written with owner-only (`0600`)
permissions. Keep `deploy/hotato.env` and any token file `0600` on the host
too, and keep them out of version control.

---

## Backup and restore

Everything the workspace needs lives in the `/data` volume: the registry
(SQLite), the content-addressed evidence store, the serve token, and the
audit log. Back up that one volume:

```bash
# back up hotato_hotato-data to ./hotato-data-backup.tgz
docker run --rm -v hotato_hotato-data:/data -v "$PWD":/backup alpine \
  tar czf /backup/hotato-data-backup.tgz -C /data .

# restore into a fresh volume
docker volume create hotato_hotato-data
docker run --rm -v hotato_hotato-data:/data -v "$PWD":/backup alpine \
  sh -c "cd /data && tar xzf /backup/hotato-data-backup.tgz"
```

Because the evidence store is content-addressed (sha256), a restored
artifact is the same bytes that produced the original verdict; any digest
mismatch is caught and surfaced explicitly.

---

## Upgrade

The image installs hotato from the source in this repository, so upgrading
is a rebuild:

```bash
git pull                       # or check out the release tag you want
docker compose build --pull    # rebuild the image
docker compose up -d           # recreate the workspace container
```

The `/data` volume is untouched by a rebuild, so your registry and evidence
survive the upgrade. The registry re-asserts its (idempotent) schema on
open. Back up `/data` before a major upgrade, as with any stateful service.

---

## Zero-migration promise

The `/data` registry and the content-addressed evidence store use the same
schemas the managed cloud uses. Your conversation artifacts, conversation
tests, and dashboards move between self-hosted and cloud without changing
a line. Self-host is the same platform on your own infrastructure, with
the whole QA platform available there -- no hosted login required.

---

## Air-gapped deployment

The default stack (the workspace alone) runs offline at run time, so it
works on an air-gapped host once the image is present. Bring the image over
instead of pulling it on the target:

```bash
# on a connected machine
docker compose build
docker save hotato-selfhost:local | gzip > hotato-selfhost.tar.gz
# move the tarball to the air-gapped host, then:
gunzip -c hotato-selfhost.tar.gz | docker load
docker compose up -d
```

This is air-gapped once its one-time steps are done: enabling the judge
profile needs the model pull described above unless you pre-seed the
`ollama-models` volume. With the image loaded and (if you use the judge)
the model volume pre-seeded, the whole stack runs fully offline.

---

## What "stays offline" covers

Scope, stated precisely so it holds up:

- The claim is about the default stack's run-time behaviour: the workspace
  server binds a listening socket and stops there -- it imports nothing
  that phones home, and keeps audio, traces, and evaluations on the
  machine. The workspace is read-only, writing only its own append-only
  audit log.
- The default workspace runs on a normal Docker bridge so its port can
  publish; the guarantee rests on the server's own behaviour, and you can
  verify it directly on your own machine, independent of any firewall:

  ```bash
  ./deploy/verify-zero-egress.sh
  ```

  It (1) confirms only `127.0.0.1:8321` is published, (2) runs the same
  image on an `internal` Docker network where egress is physically removed
  and shows the workspace still answers a view with `200` (proof the
  server does its job entirely on what's already on the machine), and (3)
  lists ESTABLISHED connections inside the running container and confirms
  every one stays internal.

- **Every opt-in path that reaches the network is named explicitly.** The
  local judge talks to the in-stack Ollama over the private network (and
  the one-time model pull downloads weights); `hotato pull` / `capture`
  fetch calls from a provider you configure; a hosted judge or the
  pyannoteAI diarizer send data off-box only behind an explicit
  `--judge-egress-opt-in` / `--egress-opt-in` flag. Every one of these is
  enumerated, command by command, in [`docs/EGRESS.md`](EGRESS.md) and
  [`docs/THREAT-MODEL.md`](THREAT-MODEL.md).

A capability that lacks its input for a given run returns INCONCLUSIVE --
plain and consistent, whether you self-host or not.

---

## See also

- [`docs/WORKSPACE.md`](WORKSPACE.md): the five views, auth, and the audit log
- [`docs/EGRESS.md`](EGRESS.md): every network call site, command by command
- [`docs/THREAT-MODEL.md`](THREAT-MODEL.md): the offline / opt-in split and the
  workspace's threat-model row
- [`SECURITY.md`](../SECURITY.md): posture summary and reporting a vulnerability


================================================================================
FILE: docs/SET-AND-FORGET.md
================================================================================

# Set and forget: passive turn-taking regression monitoring

`hotato sweep` turns from a command you remember to run into a job that runs
on its own schedule and asks for your attention only when it finds something
worth acting on.

Every command below is a shipped `hotato` command (verify with `hotato
<command> --help`). Run the whole loop right now with `--demo`, no
credentials needed, before pointing it at your own stack.

## The loop

```
connect (once) -> sweep (on a schedule) -> read the dashboard
   -> promote confirmed candidates into fixtures -> hotato run gates CI
```

Sweeping only reads: it lists candidate timing moments for you to judge. You
decide which ones are bugs and label them; `hotato fixture create` /
`hotato fixture promote` turn a candidate into a permanent test.

## 1. Connect once

```bash
hotato connect vapi --api-key <key>
# or: VAPI_API_KEY=<key> hotato connect vapi
```

This runs a lightweight live auth check (skip with `--no-verify`) and stores
the credential in `~/.hotato/connections.json`, file mode `0600`. The key
travels only to the vendor's own API, kept out of hotato's hands entirely.
Full per-stack credential table (Vapi, Twilio, Retell, Bland, ElevenLabs,
Synthflow, Millis, Cartesia): [`CONNECT.md`](CONNECT.md). LiveKit and
Pipecat are capture-in-your-infra; use `hotato setup --stack
livekit|pipecat` instead.

Once a stack is connected, `--stack` and the credential flags are optional
on `pull` / `sweep` whenever exactly one stack is connected.

## 2. Sweep on a schedule

Run the same command your terminal runs interactively, from cron or CI:

```bash
hotato sweep --stack vapi --since 7d --format json > hotato-sweep.json
hotato sweep --stack vapi --since 7d --out hotato-sweep.html --no-open
```

- `--format json` is the machine-readable candidate list `fixture promote`
  reads; redirect it to a file every time (`--out` writes the HTML
  dashboard; capture stdout for the JSON).
- `--out FILE.html --no-open` writes the shareable dashboard without popping
  a browser -- the right mode when nothing is watching the screen.
- `--since 7d` scopes the pull to recent calls; a nightly job narrows this
  to `--since 1d` so it only re-scores what came in since the last run.

A crontab entry, 03:00 daily:

```
0 3 * * * cd /path/to/repo && hotato sweep --stack vapi --since 1d --format json > "reports/sweep-$(date +\%F).json" && hotato sweep --stack vapi --since 1d --out "reports/sweep-$(date +\%F).html" --no-open
```

See [`examples/set-and-forget/`](../examples/set-and-forget/README.md) for a
runnable version of this, plus the CI half of the loop.

No stack connected yet? Everything above works with `--demo` instead of
`--stack ... --since ...`, credential-free, against two bundled recorded
calls:

```bash
hotato sweep --demo --format json > hotato-sweep.json
hotato sweep --demo --out hotato-sweep.html --no-open
```

## 3. Read the report

The HTML dashboard (the default `--format`) is one self-contained file:
every candidate moment across every swept call, ranked by salience, with the
hear-the-bug audio player embedded for the top `--audio-top` (default 8) so
you can listen before deciding anything. Calls that could not be scored
(mono/mixed stacks without `--allow-mono`, an unreadable file) list under
Skipped with the reason -- a fully logged skip.

The JSON (`--format json`) is the same candidate list as structured data.
Each entry carries the source recording, the timestamp (`t_sec`), the kind
(`agent_stop_no_caller`, `overlap_while_agent_talking`, ...), and a salience
score -- a timing fact for you to judge. You decide, per candidate, whether
the agent should have yielded or held.

## 4. Promote a confirmed bug into a fixture

Once you have listened to a candidate and decided what should have
happened, `hotato fixture promote` turns it into a permanent regression test
in one command -- no `--stereo` / `--onset` needed, since the candidate ref
already carries the recording and the moment:

```bash
hotato fixture promote hotato-sweep.json#3 --expect yield \
    --id refund-cutoff-001 --out tests/hotato
```

`CANDIDATE_REF` is `FILE#N` (the Nth candidate, ranked order, matching the
report's numbering) or `FILE#CALL:N` (the Nth candidate from one call, named
by its source file or pulled call id), for example `hotato-sweep.json#3` or
`hotato-sweep.json#call_abc123:2`. `--expect yield` means the agent should
have stopped for the caller; `--expect hold` means it should have kept
talking through a backchannel. The fixture is scored immediately: a
candidate that isn't scorable is refused (exit 2), so only a scorable
candidate becomes a fixture.

This is the loop's most important step -- the moment a suspicious call
becomes a fact your CI enforces forever instead of something you noticed
once and forgot. (`hotato fixture create --stereo ... --onset ...` does the
same thing from a raw recording and a timestamp you already know, when you
are not starting from a sweep/analyze result.)

## 5. Gate CI on your fixtures

Every promoted fixture lives under `--out DIR` (`tests/hotato/scenarios` +
`tests/hotato/audio` above) and scores with the same command whether it runs
on your laptop or in CI:

```bash
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio --format json
```

Exit code `1` means a regression is pinned; `0` means every promoted fixture
still passes. Wire that into a GitHub Action (drop-in workflow at
[`.github/workflows/hotato.yml`](../.github/workflows/hotato.yml), guide at
[`docs/CI.md`](CI.md)) or into an existing pytest run with one flag
(`pytest --hotato-suite --hotato-suite-scenarios tests/hotato/scenarios
--hotato-suite-audio tests/hotato/audio`, see [`docs/PYTEST.md`](PYTEST.md)).
[`examples/set-and-forget/`](../examples/set-and-forget/README.md) has a
complete cron + CI pairing.

## Worked example (zero setup, verified end to end)

Every command above works right now on the bundled demo calls, so you can
watch a failure get pinned before wiring anything to your own stack:

```bash
hotato sweep --demo --format json > hotato-sweep.json
hotato fixture promote hotato-sweep.json#2 --expect yield \
    --id demo-missed-interruption --out tests/hotato
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio
# exit code 1: the demo agent never yielded for an interruption -- pinned
```

That last `run` also prints the fix card (fix class, the config knob, the
direction to move it), because the fixture that just failed is a labelled
bad-agent moment from the bundled demo battery.

## What you control in this loop

- Labeling, fixture creation, and threshold tuning are each a command you
  run, for a candidate you listened to.
- `sweep` and `ingest` (the webhook-driven version of this same loop, see
  [`docs/INGEST.md`](INGEST.md)) both report candidates as timing facts;
  fixtures scored with `hotato run` are what produce a pass/fail verdict.
- This runs entirely as processes you control: cron, CI, and a webhook
  handler all shell out to the same CLI, on a schedule that's yours to own.


================================================================================
FILE: docs/STATE-ADAPTERS.md
================================================================================

# State adapters: ground a `state` assertion in your system of record

`state` and `state_change` assertions are **Authority 2**: post-call state
verification. They query a **system of record** after the call and compare
system state against what the agent claimed ("I issued the refund"). A state
adapter is the small, pluggable seam that runs that query.

`hotato test run --state FILE` loads a state adapter from `FILE`. Three
adapters ship, and one method backs all three:

```
query(resource, **filters) -> dict | None
```

Return the record for `resource` matching every `filters` key, or `None`
when the system of record was read and holds no such record.

## The three-outcome boundary

A `state` query lands on exactly three outcomes, and the distinction is the
whole point:

- **The record is present and matches** -> `PASS`.
- **The system of record was read and holds no matching record** (or a field
  does not match) -> a grounded `FAIL`. The agent claimed something the
  record does not confirm.
- **The system of record could not be reached or read** (network error,
  timeout, a 5xx, a non-JSON response, a DB error) -> `INCONCLUSIVE`, with a
  reason every time.

A `state` assertion is satisfied by the query result alone: a lookup plus a
dict comparison, deterministic, with no model path in a state adapter.

## The three adapters

### 1. `mock`: the local sandbox (default, offline)

A JSON or SQLite fixture: `{resource: rows}`. Runs offline, byte-stable, and
ready to use with no opt-in. Use it for regression suites and for a captured
before/after snapshot (`state_change` reads a `{"before": [...], "after":
[...]}` shape). This is the adapter `--state some_sandbox.json` and `--state
some.sqlite3` select today.

### 2. `http`: a REST system of record

Queries your API over stdlib `urllib`. A **resource map** turns a query into
one request:

```json
{
  "adapter": "http",
  "egress_opt_in": true,
  "base_url": "https://api.yourcompany.com/v1",
  "auth": { "type": "bearer", "token_env": "RECORDS_API_TOKEN" },
  "timeout": 30,
  "resources": {
    "appointment": {
      "path_template": "/patients/{patient_id}/appointment",
      "method": "GET",
      "params_map": { "status": "appt_status" },
      "response_pointer": "data/appointment"
    }
  }
}
```

`query("appointment", patient_id="P1", status="booked")` fills `{patient_id}`
into the path (URL-encoded), sends the remaining filters as query params (GET)
or a JSON body (POST) under their `params_map` wire names, then extracts the
record dict at `response_pointer` from the JSON response.

- `response_pointer` is a JSON-pointer-ish path (`/`- or `.`-separated; digits
  index a list; empty means the whole body is the record). A pointer that
  resolves to nothing on a well-formed response means "no such record" ->
  `None`.
- `method` is `GET` (default) or `POST`.
- **HTTPS is required by default.** A plain `http://` base URL is refused
  unless you set `"allow_http": true` (only for a trusted local endpoint);
  otherwise a state query would send filter values and the auth header in
  cleartext.
- A 404 means the addressed record does not exist -> `None` -> grounded FAIL.
  Every other non-2xx, and any network/timeout/non-JSON failure, is
  INCONCLUSIVE (`query` raises `StateAdapterError`, which the assertion
  engine turns into an INCONCLUSIVE result; the structured cause lands on
  the adapter's `last_error`).

### 3. `sql`: a SQL system of record

Queries a database with a **parameterized, read-only** SELECT.

```json
{
  "adapter": "sql",
  "sqlite_path": "records.sqlite3",
  "resources": {
    "refund": {
      "query": "SELECT order_id, status, amount FROM refunds WHERE order_id = ? AND status = ?",
      "params_order": ["order_id", "status"]
    }
  }
}
```

`query("refund", order_id="O1", status="issued")` binds `[order_id, status]`
as the statement's parameters and returns the first row as a `{column:
value}` dict, or `None` when no row matches.

- Connection source (exactly one): `sqlite_path` (a local file DB, fully
  local, no egress); a `dsn` + `driver` (a DBAPI module to import and
  `connect`, e.g. `psycopg2` for PostgreSQL -- a network DB, so it needs
  `egress_opt_in: true`; the driver is imported lazily and is never a hard
  dependency of hotato); or a caller-supplied `connection` object (Python
  API only).
- Use the placeholder style your driver expects (`?` for sqlite3, `%s` for
  psycopg2). Hotato only passes the bound value sequence through.
- A DB/driver error resolves to INCONCLUSIVE -- a query never guesses a FAIL.

## Injection safety and read-only discipline

Enforced by construction and covered in `tests/test_state_adapters_real.py`:

- **Injection-safe.** Filter values are always bound as query parameters,
  never string-interpolated into the SQL. A filter value carrying SQL
  (`"O1'; DROP TABLE refunds; --"`) is treated as a literal: it matches no
  row, and the table stays untouched.
- **Read-only.** Every mapped SQL query is validated at construction (and
  re-checked before each execute): it must begin with `SELECT` (or a `WITH
  ... SELECT` CTE), carry no second statement after a `;`, and contain no
  data-modifying keyword. A `DELETE`/`UPDATE`/`DROP`/... mapped query is
  rejected.

## Egress opt-in

The `http` adapter, and a `sql` adapter over a `dsn`, are **network paths**
(see [`docs/EGRESS.md`](EGRESS.md) and
[`docs/THREAT-MODEL.md`](THREAT-MODEL.md)). `load_state_adapter` **refuses**
them unless the config sets `"egress_opt_in": true` -- an explicit,
per-config opt-in, the same discipline `--egress-opt-in` applies to the
hosted diarizer and `--notify`. Without it, you get a clear usage error (the
CLI's exit-2 path) before any connection is made. A local `sqlite_path` SQL
DB and the mock sandbox run entirely on your machine, opening no socket,
with no opt-in needed.

When an `http` / non-local-`sql` query does fire, only the mapped filter
VALUES leave the machine (in the URL, query string, or JSON body). Audio,
transcript, and the config file itself stay on your machine.

## Credentials

Adapters take environment-variable **names** for credentials, keeping the
config file free of secrets so it can be committed or shared. Supply the
secret at run time from the environment (for example, `source` a `0600` file
that exports it):

- `bearer`: `{ "type": "bearer", "token_env": "RECORDS_API_TOKEN" }`
- `basic`: `{ "type": "basic", "username_env": "DB_USER", "password_env": "DB_PASS" }`
  (`username` may be inline since it is not a secret).
- `header`: `{ "type": "header", "headers": { "X-Api-Key": { "env": "API_KEY" }, "X-Tenant": { "value": "acme" } } }`,
  where each header value is either `{ "env": NAME }` (a secret from the
  environment) or `{ "value": LITERAL }` (a non-secret constant).

A missing credential env var fails fast at construction with a message that
names the variable, not the value. Credential values stay out of logs and
out of `last_error`.

## `state_change` and before/after snapshots

`state_change` reads a `before` and an `after` snapshot to measure a delta.
Only the **mock** adapter provides both (from a captured `{"before": ...,
"after": ...}` fixture, or `<table>__before` / `<table>__after` SQLite
tables). The `http` and `sql` adapters answer point-in-time `state` queries
(a live API/DB exposes only "now"), so their `before` snapshot is reported
absent -- keeping a `state_change` against them grounded at INCONCLUSIVE
instead of guessing "no change."

## See also

- [`docs/ASSERTIONS.md`](ASSERTIONS.md): the `state` / `state_change`
  assertion kinds.
- [`docs/EGRESS.md`](EGRESS.md), [`docs/THREAT-MODEL.md`](THREAT-MODEL.md):
  the network rows.
- `src/hotato/state_adapter.py`: the adapters; `tests/test_state_adapters_real.py`: the tests.


================================================================================
FILE: docs/THREAT-MODEL.md
================================================================================

# Threat model

The sensitive thing here -- your call recordings -- stays on your machine,
and every network action is one you named. This page draws the precise
line: which commands are offline-only, which reach the network and only
when you ask, and the guarantees that hold either way.

## Three guarantees

1. **Every production change happens through a proposal you apply
   yourself.** `plan` and `patch` produce a proposal; `apply` operates on a
   fresh staging clone and dry-runs by default. Pushing a config to
   production is a step you take, not one any command takes on its own.
2. **Recordings stay on your machine.** Self-hosted, with audio moving only
   from your own stack to your own disk, and only when you run a
   pull/capture/sweep. The one exception is explicit and opt-in: the hosted
   `--diarizer pyannoteai` backend, which requires `--egress-opt-in` before
   any audio leaves the machine.
3. **A webhook payload is read strictly as data.** The `ingest` webhook
   worker reads a completed-call notification as **data**: it extracts a
   recording reference and scans it. Payload fields are read, never
   executed, shelled out, or used to choose what code runs.

## Core: fully offline, on your machine

These commands read and write only the local files you point them at,
opening no sockets. This is the whole scoring, analysis, and fixture
surface.

- **`run`** -- score a local WAV (or the bundled self-test battery).
- **`scan`** -- list candidate moments in a local recording.
- **`trust`** -- input-health check on a local recording.
- **`report`** -- render a self-contained HTML report from local run data.
- **`fixture`** -- create / promote a local moment into a regression
  fixture.
- **`compare`** -- score a before/after pair of local takes.
- **`verify`** -- roll up before/after run envelopes on disk.
- **`diagnose`** -- explain a finished local run envelope (read-only).
- **`plan`** -- combine a diagnosis + config into a proposed fix plan
  JSON.
- **`patch`** -- render a fix plan into a paste-ready patch. Never applies
  it.
- **`demo`** -- run the packaged two-call battery. Zero extra files.
- **`team`, `export`, `benchmark`** -- aggregate / export local run data.
- **`setup`, `describe`, `init`** -- print recording config, emit the CLI
  manifest, scaffold local integration files.
- **`rubric run`, `rubric calibrate`** -- score a local transcript against
  a rubric with a **local** model (Ollama at `localhost`). Default path
  opens no off-box socket; the verdict cache is local. Reaches the network
  only with an explicit hosted / non-local judge (see below).
- **`test run` (rubric lane)** -- same local model judge as `rubric run`,
  run inline on a conversation-test's `assertions.rubric` lane. Local by
  default.
- **`counterexample compile|verify|reproduce|inspect|export|predicate`** --
  read local scenario/test/capsule files and write a new local capsule or
  share-safe projection. The proof path loads no provider adapter, model,
  subprocess, or network client.

Default retention is local-only: reports, envelopes, and exports land
exactly where you point them.

### Counterexample capsule boundary

A private `.hotato-repro` contains the source scenario and test, reduced
fixture, target assertion result, and proof journal. Treat it as sensitive
source material. The compiler creates it with owner-only modes where the host
supports POSIX permissions, never overwrites an existing destination, and
promotes a sibling staging directory only after all files and hashes exist.

The verifier rejects path traversal, symlinked members and directories,
special files, undeclared files, oversized/deep inputs, digest mismatches,
broken deletion chains, evaluator-source drift on the strict proof path, and a
minimality claim that admits a preserving unit deletion. The preflight caps a
capsule at 1,024 files, 4,096 directories, 64 directory levels, 64 MiB per
member, and 256 MiB total. `reproduce` is a separate current-evaluator check:
it permits evaluator drift, but still validates capsule integrity and the
source-to-final delete-only chain.

Replay also bounds proof-specific work: 256 KiB per selected assertion, 2 MiB
per canonical candidate scenario, 256 KiB of rendered transcript text, 2 MiB
per assertion result, 10,000 evidence rows, 512 accepted proof-chain steps,
10,000 deletion operations per accepted step, and 512 remaining deletion
units in a completed minimality proof. Accepted steps require fresh oracle
evaluations; cached preserved journal rows cannot inflate the chain. Proof
regexes use a closed, fixed-width 1,024-byte subset that refuses groups,
alternation, backreferences, and variable quantifiers. These limits apply to
counterexample compilation and replay; they are not global evaluator limits.
The selected-assertion and proof-regex byte checks run before the general
assertion validator can invoke Python's regex parser.

Each profile also has an exact member allowlist. A manifest cannot authorize a
new file outside that profile. For `share-safe-v1`, the only members are
`capsule.json`, `report.md`, `report.html`, `card.svg`, `README.md`, and
`MANIFEST.sha256.json`; each human-facing file is reconstructed canonically and
compared byte for byte during inspection. This prevents a modified report,
README, or card from becoming accepted merely because its replacement digest
was written into a new manifest.

The canonical share renderer bytes are frozen as part of the capsule v1
exchange contract. A future renderer must retain v1 output or use a versioned
profile/format rather than silently changing bytes under an existing proof.

The evaluator digest binds the shipped Python source closure and package
version, not the interpreter build, host platform, CPU, or native libraries.
Strict replay compares the recorded result, content, and trace hashes, so a
runtime difference that changes behavior is rejected. Accepted transforms and
the final single-unit inventory are proof-bearing; non-`PRESERVED` journal
rows are diagnostic history and are not independently replayed.

The capsule directory is required to remain unchanged during a command.
Persistent mutation, root replacement, and symlink or special-file
substitution are detected. A privileged local process that can swap and
restore bytes between individual reads is outside the v1 proof snapshot; move
untrusted capsules into a private, non-writable workspace before verification.

`counterexample export` derives a non-runnable projection after strict private
verification. It omits scenario/test bodies, transcript content, tool payloads,
state values, provider identifiers, reducer paths, and per-candidate digests.
Private minimality rows are reduced to aggregate outcome counts. The projection
exposes the payload-free selected failure code for review while keeping its
field/key/rule/detector/index discriminator private. It also contains
identifiers and hashes that remain correlators; low-entropy or
externally known source material can be tested against them. `share-safe-v1`
therefore belongs inside the team's normal engineering-artifact access
boundary and does not claim anonymous public disclosure.

## Network: only when you explicitly request it

These commands reach outside the machine, and only because reaching out is
their job. Each requires you to name a stack, a repository, or a webhook
you configured.

- **`connect`**
  - Network surface: none at connect time.
  - Notes: stores a stack's credentials locally at `0600`. Setup for the
    pull path; no audio moves.
- **`pull`**
  - Network surface: your voice stack's API.
  - Notes: bulk-fetch recent recordings from a stack **you** connected,
    into a local folder.
- **`capture`**
  - Network surface: your voice stack's API.
  - Notes: fetch and score one call from your stack.
- **`sweep`**
  - Network surface: your voice stack's API.
  - Notes: `pull` + analyze in one command.
- **`inspect`**
  - Network surface: your voice stack's API.
  - Notes: read (never write) the current turn-taking config. Read-only.
- **`ingest`**
  - Network surface: a webhook endpoint you host.
  - Notes: worker that scans each completed call. Payloads are data, not
    instructions (guarantee 3).
- **`issue`**
  - Network surface: GitHub, via your local `gh`.
  - Notes: file a sweep's candidates as an issue. Uses your existing `gh`
    auth.
- **`pr`**
  - Network surface: GitHub, via your local `gh`.
  - Notes: open a PR adding promoted fixtures. Uses your existing `gh`
    auth.
- **`apply`**
  - Network surface: a git clone you point it at.
  - Notes: applies a patch to a fresh **staging** clone only, never the
    source. Dry-run by default; refuses a both-axes threshold funnel.
- **`--diarizer pyannoteai`**
  - Network surface: Attention Labs hosted diarizer.
  - Notes: the only audio path that can send audio off-box, and only with
    `--egress-opt-in`. The default diarizer is local.
- **`--judge-provider hosted` / non-local `--judge-endpoint`** (any rubric
  command)
  - Network surface: a hosted or remote model host you name.
  - Notes: sends the transcript + rubric criterion off-box for judging.
    Refused (exit 2) unless `--judge-egress-opt-in`. The default judge is a
    local Ollama model that stays on the box. See
    [`docs/EGRESS.md`](EGRESS.md) and [`docs/RUBRIC.md`](RUBRIC.md).
- **`test run --state` http adapter**
  - Network surface: your system-of-record's REST API.
  - Notes: only when the state-config names `adapter: http`, and only with
    `egress_opt_in: true` in that config. Sends the mapped filter VALUES for
    one `state`/`state_change` query -- audio, transcript, and the config
    itself stay local. See [`docs/STATE-ADAPTERS.md`](STATE-ADAPTERS.md).
- **`test run --state` sql adapter over a `dsn`**
  - Network surface: your database, over the network.
  - Notes: only when the state-config names `adapter: sql` with a `dsn`,
    and only with `egress_opt_in: true`. A parameterized, read-only SELECT
    with the mapped filter values bound as data. A local `sqlite_path`
    opens no socket.

The state adapters read a post-call **system of record** to ground an
Authority-2 `state` assertion (did the refund/appointment get written?). A
record the system of record can be read and does not hold is a grounded
FAIL; a system of record Hotato could **not** reach or read (network error,
timeout, 5xx, non-JSON) is INCONCLUSIVE, never a guessed verdict.
Credentials come from environment-variable **names** in the config, keeping
the secret itself out of it, and the HTTP adapter refuses a plain-`http://`
base URL unless `allow_http: true` is set for a trusted local endpoint. The
default `--state` path is the local mock sandbox (a JSON/SQLite fixture)
and a local `sqlite_path` SQL DB, both running entirely on-machine, with no
opt-in needed.

Notify surfaces (Slack, GitHub) fire only through credentials you
configured (`gh`, a Slack token) and only for actions you invoked -- every
integration is one you triggered.

## Network trust: proxies and TLS

Every networked command above (`pull`, `capture`, `sweep`, `inspect`,
`apply`, and the credential probe in `connect`) makes its HTTP calls through
Python's standard `urllib`, which honors the `HTTP_PROXY` / `HTTPS_PROXY` /
`NO_PROXY` environment-variable convention, the same convention `curl`,
`pip`, `git`, and `docker` follow. That is deliberate: it is what lets
Hotato work behind a corporate proxy with no extra flags. It also means an
environment variable set for the Hotato process controls where its outbound
credentialed requests are routed, and that is a trust decision worth
stating plainly instead of leaving implicit.

Two things bound how far that ambient trust reaches:

- **TLS certificate validation is always enforced.** Every credentialed
  base URL in `capture.py` and `apply.py` is hardcoded `https://`
  (`api.vapi.ai`, `api.retellai.com`, `api.twilio.com`, and the other
  supported stacks). A proxy set via `HTTP_PROXY`/`HTTPS_PROXY` can see the
  `CONNECT` target host and port, and can refuse or stall the connection (a
  denial of service), but reading or rewriting the `Authorization` header
  or the response body requires also presenting a certificate the
  machine's own trust store already accepts for that vendor's domain -- a
  much larger compromise (a rogue trusted root CA already installed)
  outside `HTTP_PROXY`'s reach and outside Hotato's control.
- **The threat prerequisite is already a compromised local environment.**
  An attacker able to set environment variables for the Hotato process can
  already read `VAPI_API_KEY` / `RETELL_API_KEY` / etc. straight out of the
  environment, or read the `0600` credentials file `connect` wrote -- both
  strictly easier than standing up a TLS-valid proxy.

If you do not trust the ambient proxy environment a command will run in,
set `HOTATO_NO_PROXY=1` to make Hotato's HTTP opener ignore `HTTP_PROXY` /
`HTTPS_PROXY` for that run, or unset those variables before invoking the
command. The default is unchanged: proxy env vars are honored, matching
curl/pip/git, so a corporate-proxy setup keeps working with no
configuration.

## Hotato's containment guarantees

- **Recordings stay on the machine by default.** Audio leaves only when you
  run `pull`/`capture`/`sweep` with `--egress-opt-in`; the default diarizer
  is local.
- **A call webhook is read as data, never as commands.** `ingest` parses a
  payload for a recording reference and stops there: payload fields are
  read, never evaluated, so a malicious webhook body cannot make Hotato run
  arbitrary code (guarantee 3).
- **Production changes always go through you.** No command writes to a live
  stack's config directly; the furthest Hotato goes is a proposed patch you
  apply to a staging clone yourself.

## Verifying the posture

Confirm the posture directly, without trusting this page:

- **Self-hosted, nothing to audit but the standard library.** `pip install
  hotato` pulls zero runtime dependencies; the only network code path is
  the one you explicitly invoke.
- **Credentials at `0600`.** `connect` writes stack credentials locally
  with owner-only permissions; check them with `ls -l` on your credentials
  path.
- **Local-only retention.** Reports, envelopes, and exports exist exactly
  where you wrote them.
- **Egress is opt-in.** The only off-box audio path is `--diarizer
  pyannoteai`, gated by `--egress-opt-in` -- audio stays on the machine
  until you set it.

## Drive-a-call (`run_scenario`): placing a live call

`run_scenario` (`src/hotato/drive.py`, wired into the Vapi and Twilio
adapters) places an outbound call against a live agent and pulls the
recording. Its threat surface and the controls on it:

- **A billable, outbound call always requires explicit opt-in.**
  `run_scenario` places the call only when BOTH live credentials AND an
  explicit egress opt-in (`HOTATO_DRIVE_OPT_IN=1` or `egress_opt_in: true`
  on the scenario) are present. Absent either, it raises a clean
  structured refusal and dials nothing -- matching the opt-in posture of
  `--allow-mono` / `HOTATO_ALLOW_PRIVATE_URLS` / `--egress-opt-in`.
- **Production config stays untouched.** The surface is create-and-read
  only: the only verbs issued are `POST` (create the call) and `GET` (poll
  status, list the recording, download it), with no `PUT`/`PATCH`/`DELETE`
  path to alter an assistant, a phone number, or any other provider
  resource in place. For Vapi the call originates FROM the staging clone
  the experiment created, keeping the production source untouched.
- **The caller side is labelled precisely.** The produced conversation
  carries `origin.kind == "real"` with the provider and its call id, and
  `origin.caller` (`scripted-twiml` for the fixed-timeline Twilio caller,
  `assistant-originated` for a Vapi call the assistant placed), stating
  exactly what happened -- no claim that a human placed the call or that
  the scripted caller reacted to the agent.
- **The recording download inherits every capture-side control.** The
  vendor URL (`stereoUrl` / `RecordingSid` media) flows through the same
  validated download as `capture`: http(s)-only scheme allowlist,
  default-deny SSRF (loopback/private/metadata refused unless
  `HOTATO_ALLOW_PRIVATE_URLS=1`), cross-host `Authorization`-strip on
  redirect, and atomic write. The API key is never attached to a
  pre-signed media URL from the vendor's JSON response.

See [`docs/DRIVE-A-CALL.md`](DRIVE-A-CALL.md) and the drive-a-call rows in
[`docs/EGRESS.md`](EGRESS.md).

## Team workspace (`hotato serve`): a new local listening socket

`hotato serve` (`src/hotato/serve/`, wired into the CLI) opens a new local
HTTP listening socket to serve the five conversation-QA views over the
fleet registry + evidence store. Its threat surface and the controls on
it:

- **Localhost-default bind.** The server binds `127.0.0.1` unless the
  operator explicitly passes `--host`. A non-loopback bind (e.g. `--host
  0.0.0.0`) always prints a prominent warning at start and always requires
  that explicit flag.
- **Token auth on every request.** A shared bearer token authenticates
  every request, compared in constant time (`hmac.compare_digest`). It is
  either operator-supplied (`--token` / `--token-file`) or generated with
  `secrets.token_urlsafe` and stored `0600` under the per-workspace state
  dir. Browsers bootstrap an in-memory, HttpOnly session cookie from the
  printed `/?token=…` URL (then the token is stripped from the address bar
  via a redirect); the token stays out of every response body. An
  unauthenticated request gets `401` before it is routed anywhere.
- **Read-only.** The server issues only `SELECT`s against the registry and
  reads evidence blobs by digest, exposing only reads: no write endpoint,
  no workspace mutation. Reviews and labels stay CLI-driven. The only file
  it writes is the append-only audit log (`…/serve/<workspace>/audit.jsonl`,
  `0600`), which records who (token/session prefix, the secret stays out of
  it), what (method + path, the token stripped from the query), when, and
  the response status of every request.
- **Zero egress.** The server only binds a listening socket, opening no
  outbound connection and importing nothing that phones home -- audio,
  traces, and evaluations stay on the machine. A test whitelists loopback
  and fails if any view attempts an external connection.
- **Evidence renders as data, never as executable content.** The raw
  evidence endpoint (`/evidence/<digest>`) serves blobs as `text/plain`
  with `X-Content-Type-Options: nosniff` (and the pages set
  `X-Frame-Options: DENY`, `Referrer-Policy: no-referrer`), keeping a
  crafted evidence blob inert in the viewer. Redacted transcript/trace
  content is scrubbed at the data layer, so both the HTML and the JSON
  mirror render only the `[redacted]` placeholder.
- **Workspace id sanitized against path traversal.** The workspace id is
  used verbatim only in parameterized SQL; as a state-directory name it is
  sanitized to stay inside the registry home.

See [`docs/WORKSPACE.md`](WORKSPACE.md).

Security policy and reporting: [SECURITY.md](../SECURITY.md).


================================================================================
FILE: docs/TRANSCRIBE.md
================================================================================

# `hotato run --stereo call.wav --transcribe`: read a transcript next to the score

Hotato's gold reference is a deterministic **energy VAD** over the audio
waveform: `did_yield`, `talk_over_sec`, and `seconds_to_yield` come from
frame energy, nothing else, and every published/golden number is computed
that way. The opt-in `[transcribe]` extra runs an off-the-shelf
speech-to-text model over the same recording and hands back plain text with
per-segment timestamps, so a human (or an agent) reading a report can see
WHAT was said next to WHEN the timing engine says it was said.

That is the entire feature. A transcript is attached as **context** beside
the score, computed strictly after scoring is final. The timing score stays
grounded in the audio: the transcript rides alongside it, never feeding
back into it, and carries no accuracy claim of its own.

## What stays unchanged

- **The timing score stays grounded in the audio.** `did_yield` /
  `talk_over_sec` / `seconds_to_yield` / every other field is computed
  identically with or without `--transcribe` -- adding it produces
  byte-identical timing numbers, pinned by a test
  (`tests/test_transcribe.py`), not just prose.
- **Plain text, not a quality judgment.** The transcript hands back what
  was said, not a grade on whether it was said well. What's reported for
  the ASR model is the plain text and its timestamps -- no WER, accuracy,
  or quality number, here or anywhere in hotato.
- **Timestamps only, no speaker attribution.** Plain text with timestamps,
  no caller/agent attribution -- anonymous speaker separation lives in a
  separate, unrelated extra ([`docs/DIARIZE.md`](DIARIZE.md)).

## Quickstart

```bash
# Install the opt-in extra (local, offline; no gated token, unlike [diarize]):
pip install 'hotato[transcribe]'

# Score a call and attach a transcript next to the verdict:
hotato run --stereo call.wav --transcribe --format json
```

With the extra absent, `--transcribe` always fails loud: a clean error,
exit `2`, never a silent skip of the transcript.

## Usage

```bash
# base.en, device auto-detected:
hotato run --stereo call.wav --transcribe

# pick a model + device:
hotato run --stereo call.wav --transcribe --transcribe-model small.en \
  --transcribe-device cpu

# also works on the diarized-mono path:
hotato run --mono call.wav --diarize --transcribe
```

`--transcribe` needs a **single** audio file: `--stereo`, or `--mono` when
scoring through the opt-in `--diarize` front-end (it transcribes the same
mono file the diarizer separated). Two separate `--caller`/`--agent` files
raise a clean usage error naming `--stereo`, rather than guessing which
channel to transcribe or silently dropping one. Combining `--transcribe`
with the bundled self-test battery (`--suite` / `--scenarios`+`--audio`)
also raises a clean usage error, not a silent no-op, so CI never mistakes
"flag ignored" for "flag applied."

## Models and device

- Default model: `base.en` (English-only, the fastest useful size).
  Override with `--transcribe-model NAME` -- any name or local path
  `faster-whisper`/CTranslate2 accepts.
- Default device: `auto` -- `cuda` if CTranslate2 reports a CUDA device,
  else `cpu`. `compute_type` defaults to `float16` on GPU and `int8` on CPU
  (CTranslate2's documented fast defaults per device, not an accuracy
  claim); both overridable.
- Every choice used is stamped in the output (`transcript.model` /
  `.device` / `.compute_type` / `.language`), so a report is reproducible.

## Backend and extra

- **`[transcribe]`**
  - Brings in: `faster-whisper` (a CTranslate2 re-implementation of
    OpenAI's Whisper).
  - Where it runs: local, CPU-viable, GPU-optional, fully offline once the
    model is cached.

```toml
transcribe = ["faster-whisper>=1.0"]
```

This extra keeps the core's Python floor at `>=3.9` (unlike `[diarize]`),
since `faster-whisper`/CTranslate2 support that range. No model card to
accept here either.

### Model licenses (log per FTO note)

Using an off-the-shelf ASR model is integration, orthogonal to any Hotato
IP claim, but the dependency licenses are logged here:

- `faster-whisper` -- **MIT** (code)
- `CTranslate2` -- **MIT** (the inference runtime `faster-whisper` runs on)
- Whisper model weights -- **MIT** (OpenAI)

## The invariant: context beside the score, grounded in the audio

A transcript is computed strictly **after** the score is final, over the
same audio the score already used, and attached on a new top-level key of
the envelope -- nothing already there is read or rewritten:

- `events`, `verdict`, `measurements`, and `signals` are the exact same
  values with or without `--transcribe`.
- The energy VAD's frame-level activity stays the ground truth for timing;
  a transcript word boundary is a separate thing -- ASR word timestamps are
  themselves a model estimate, read alongside it, never substituted in.
- A missing/broken `[transcribe]` extra raises a clean, actionable error --
  exactly like the `[neural]` and `[diarize]` seams: never a bare
  `ImportError`, a silent skip, or a fallback to a different backend.
- `hotato.transcribe` and the vendored `_engine` import nothing from each
  other, keeping the transcript path fully separate from the scorer.

## JSON shape (agents)

A `--transcribe` run is the same envelope as any single run, plus one
additive top-level `transcript` block. `events[0].verdict` is identical to
a run without it:

```json
{
  "mode": "single",
  "events": [{
    "verdict": {
      "did_yield": true,
      "seconds_to_yield": 0.42,
      "talk_over_sec": 0.18,
      "reasons": []
    }
  }],
  "transcript": {
    "text": "I need to check on my refund -- hold on, let me pull that up for you.",
    "segments": [
      {"start": 0.10, "end": 2.30, "text": "I need to check on my refund"},
      {"start": 2.60, "end": 4.95, "text": "hold on, let me pull that up for you."}
    ],
    "model": "base.en",
    "device": "cpu",
    "compute_type": "int8",
    "language": "en"
  }
}
```

Branch on the presence of `transcript`: present only when `--transcribe`
was passed, additive only -- the meaning of everything under `events`
stays the same. On the diarized-mono path, `transcript` still attaches
even when the diarization confidence gate refused or downgraded the
verdict; ASR runs independent of whether diarization succeeds.

## Egress

Fully local at inference time: once the named model is cached,
transcribing opens no socket. The **first** run of a model you have not
used before downloads its weights from its public host (the same one-time
fetch as installing any pip package with model weights); every run after
that is offline. See [`docs/EGRESS.md`](EGRESS.md) for the full per-command
network table.


================================================================================
FILE: docs/TRUST-GALLERY.md
================================================================================

# Trust gallery: eight recordings, eight verdicts

Eight named input conditions, each with the `hotato` output it produces.
Every block below is verbatim CLI output from hotato 1.6.1, offline. The rows
correspond to the [trust matrix](TRUST-MATRIX.md).

The clean case and the echo case use the bundled examples
`01-hard-interruption.example.wav` and `07-echo-bleed.example.wav` (shipped in
`src/hotato/data/audio/`), and the backchannel case uses the bundled
`02-backchannel-mhm.example.wav`. The four defect cases (silent caller, silent
agent, swap, mono) are the deterministic synthetic two-channel fixtures the test
suite pins in `tests/test_trust.py`, each isolating one input defect, so every
verdict reproduces byte-for-byte from the shipped builders.

---

## 1. Safe stereo: the good case

Clean two-channel export, caller on ch0, agent on ch1. `trust` clears it.

```text
$ hotato trust --stereo 01-hard-interruption.example.wav
hotato trust: 01-hard-interruption.example.wav
  recording: 6.0s, 16000 Hz, 2 channels
  caller (ch0): 2.46s speech, first at 2.39s, peak -4.4 dBFS
  agent  (ch1): 2.71s speech, first at 0.19s, peak -4.4 dBFS
  leading silence: 0.19s
  crosstalk: coherence 0.306 (low) at 0.5s lag
  scorability: separated tracks yes, caller activity yes, agent activity yes
  => eligible for scan
```

**Verdict:** `eligible for scan`, exit 0. Score it. This is the only condition
where a full turn-taking verdict is trustworthy without a caveat.

---

## 2. Silent caller: nothing to measure against

The caller channel carries no speech (a dead leg, a wrong export, a muted line).

```text
$ hotato trust --stereo silent-caller.wav
hotato trust: silent-caller.wav
  recording: 6.0s, 16000 Hz, 2 channels
  caller (ch0): 0.00s speech, -, peak -120.0 dBFS
  agent  (ch1): 5.76s speech, first at 0.19s, peak -9.1 dBFS
  leading silence: 0.19s
  crosstalk: coherence 0.0 (low) at 0.0s lag
  scorability: separated tracks yes, caller activity no, agent activity yes
  => NOT SCORABLE: caller channel has no detected speech
     next step: verify channel mapping or export dual-channel again
```

**Verdict:** `NOT SCORABLE`, exit 2. A yield is a response to a caller; with no
caller there is nothing to be late to, so Hotato names the gap instead of
printing a hollow pass.

---

## 3. Silent agent: no floor to interrupt

The agent channel is empty. There is no floor for the caller to barge into.

```text
$ hotato trust --stereo silent-agent.wav
hotato trust: silent-agent.wav
  recording: 6.0s, 16000 Hz, 2 channels
  caller (ch0): 2.16s speech, first at 0.99s, peak -9.1 dBFS
  agent  (ch1): 0.00s speech, -, peak -120.0 dBFS
  leading silence: 0.99s
  crosstalk: coherence 0.0 (low) at 0.0s lag
  possible channel swap: channel 0 (mapped as caller) holds the floor 2.16s vs 0.0s on channel 1 (mapped as agent); an agent usually holds the floor longer, so the caller/agent channels may be reversed
  scorability: separated tracks yes, caller activity yes, agent activity no
  => NOT SCORABLE: agent channel has no detected speech
     next step: verify channel mapping or export dual-channel again
```

**Verdict:** `NOT SCORABLE`, exit 2. Hotato also flags a possible channel
swap here, because a caller-only recording looks like a mis-mapped agent. Two
independent checks point at the same fix: re-check the export.

---

## 4. Swapped channels: candidate-eligible, verdict pending confirmation

Both channels carry speech, but the long floor-holder is on the channel mapped
as the caller. `trust` keeps the input candidate-eligible and holds the verdict
until the mapping is confirmed.

```text
$ hotato trust --stereo swapped-warn.wav
hotato trust: swapped-warn.wav
  recording: 6.0s, 16000 Hz, 2 channels
  caller (ch0): 5.76s speech, first at 0.19s, peak -9.1 dBFS
  agent  (ch1): 0.66s speech, first at 1.99s, peak -9.1 dBFS
  leading silence: 0.19s
  crosstalk: coherence 0.31 (low) at 0.5s lag
  possible channel swap: channel 0 (mapped as caller) holds the floor 5.76s vs 0.66s on channel 1 (mapped as agent); an agent usually holds the floor longer, so the caller/agent channels may be reversed
  scorability: separated tracks yes, caller activity yes, agent activity yes
  => scan with caution: channel 0 (mapped as caller) holds the floor 5.76s vs 0.66s on channel 1 (mapped as agent); an agent usually holds the floor longer, so the caller/agent channels may be reversed
  [!] not verdict-eligible (scan mode): channel mapping unconfirmed: suspected swap/crosstalk; confirm mapping or supply provider metadata
```

**Verdict:** `scan with caution`, exit 0: candidate-eligible, with verdict
eligibility waiting on your confirmation. A swap would invert every yield into
a hold, so Hotato surfaces candidates for review and holds the turn-taking
verdict until the mapping is confirmed. Confirm the mapping (set
`--caller-channel` / `--agent-channel`, or supply provider metadata) to
restore verdict eligibility.

---

## 5. Crosstalk / echo bleed: candidate-eligible, verdict pending confirmation

The caller channel carries a delayed copy of the agent's own audio. Coherence
pegs at 1.0 and the measured leakage is -9.1 dB.

```text
$ hotato trust --stereo 07-echo-bleed.example.wav
hotato trust: 07-echo-bleed.example.wav
  recording: 6.0s, 16000 Hz, 2 channels
  caller (ch0): 5.69s speech, first at 0.31s, peak -13.5 dBFS
  agent  (ch1): 5.81s speech, first at 0.19s, peak -4.4 dBFS
  leading silence: 0.19s
  crosstalk: coherence 1.0 (HIGH) at 0.12s lag
  leakage: -9.1 dB (agent->caller, HIGH)
  scorability: separated tracks yes, caller activity yes, agent activity yes
  => scan with caution: cross-channel leakage: agent audio on the caller channel sits at -9.1 dB, a consistent delayed copy (lag 0.12s); leaked audio at this level can be counted as the other party's activity, so a downstream timing measurement may be wrong even though the tracks are separated
  [!] not verdict-eligible (scan mode): channel mapping unconfirmed: suspected swap/crosstalk; confirm mapping or supply provider metadata
```

**Verdict:** `scan with caution`, exit 0: candidate-eligible, with verdict
eligibility waiting on confirmation. The "caller" activity may be leaked TTS,
so Hotato still surfaces candidates for review (the scan step, example 8,
names the specific moment) and holds the turn-taking verdict until the
mapping is confirmed.

---

## 6. Mono: refused by default, tiered under `--diarize`

A single channel. The caller and the agent are mixed into one track.

```text
$ hotato trust --stereo mono-mixed.wav
hotato trust: mono-mixed.wav
  recording: 6.0s, 16000 Hz, 1 channel
  scorability: separated tracks no, caller activity no, agent activity no
  => NOT SCORABLE: the recording has a single channel, so the caller and the agent cannot be told apart
     next step: export a dual-channel recording with the caller on one channel and the agent on the other
```

**Verdict:** `NOT SCORABLE`, exit 2. This is the gold-standard refusal. Two
opt-in escapes exist, both marked indicative only: `--allow-mono` on
`capture` / `pull` / `sweep` accepts a mono-only stack in degraded mode (talk-over
unattributable, no SLA gate), and the `--diarize` separation front-end reports a
`high` / `low` / `refuse` tier, with `hotato run --mono call.wav --diarize`
stamping the verdict `indicative_only` at the `low` tier. Neither stands in
for dual-channel. See [TRUST-MATRIX.md](TRUST-MATRIX.md) and
[DIARIZE.md](DIARIZE.md).

---

## 7. Backchannel candidate: surfaced, yours to label

A scan of a call where the caller says "mhm" over the agent. `scan` lists the
overlaps as candidates and hands the intent decision to you.

```text
$ hotato scan --stereo 02-backchannel-mhm.example.wav --top 5
hotato scan: 02-backchannel-mhm.example.wav  (6.0s, 3 candidate moments)
Candidates are timing events. You decide the expected behavior; label with: hotato fixture create --onset <t> --expect yield|hold
  [ 1] t=2.09s  overlap_while_agent_talking  overlap=1.58s  agent did not go silent within 3.0s
  [ 2] t=3.19s  overlap_while_agent_talking  overlap=1.07s  agent did not go silent within 3.0s
  [ 3] t=4.29s  overlap_while_agent_talking  overlap=0.56s  agent did not go silent within 3.0s
```

**Verdict:** three candidates, exit 0. Whether "agent did not go silent" is
correct (it held the floor through a backchannel, good) or a bug (it talked
over a caller trying to take the floor) is **your** call. `scan` reports the
timing; you label `hold` or `yield` -- the intent behind a caller sound is
always your call.

---

## 8. Noisy false positive: a candidate that is really the agent

A scan of the echo-bleed call. One candidate is an overlap fact; the
second is Hotato flagging that the "caller" activity is leaked TTS.

```text
$ hotato scan --stereo 07-echo-bleed.example.wav --top 5
hotato scan: 07-echo-bleed.example.wav  (6.0s, 2 candidate moments)
Candidates are timing events. You decide the expected behavior; label with: hotato fixture create --onset <t> --expect yield|hold
  [ 1] t=0.31s  overlap_while_agent_talking  overlap=3.00s  agent did not go silent within 3.0s
  [ 2] t=0.31s  echo_correlated_activity     WARNING likely agent echo: coherence=1.00 at lag 0.12s  (caller channel looks like leaked TTS; a yield here may be the agent hearing itself)
```

**Verdict:** two candidates, exit 0. Candidate 1 looks like a bad talk-over.
Candidate 2 is Hotato telling you the overlap is probably the agent hearing
its own audio rather than the caller interrupting. This is the failure mode
of candidate discovery: the net is wide, and Hotato labels the reason a
wide-net candidate is likely spurious instead of hiding it. You label it
`hold` (or fix
the echo capture) and it never becomes a fixture.

---

## How to read the gallery

The eight cases fall into three buckets:

- **Score it** (1): clean stereo, verdict-eligible, full confidence.
- **Candidate-eligible, verdict pending confirmation** (4, 5): swap and
  crosstalk keep `scan` surfacing candidates (exit 0), with the turn-taking
  verdict waiting on your confirmation of the channel mapping. Example 8 is
  that same echo call under `scan`, where the specific leaked moment is
  named.
- **Refuse** (2, 3, 6): silent caller, silent agent, and mono are not scorable;
  Hotato names the reason and the next step and exits 2.

Case 7 is the everyday case: a candidate with no defect, waiting for your
label. The design goal is a Hotato verdict you act on has already survived
this gauntlet.


================================================================================
FILE: docs/TRUST-MATRIX.md
================================================================================

# Trust matrix: what Hotato does for each input condition

Before Hotato scores a recording, `hotato trust` inspects the audio and
decides whether a score would be meaningful, catching a bad export before it
turns into a confident-looking but hollow verdict. This page is the exact
contract: input condition on the left, Hotato's behavior on the right. This
table covers input health only; a turn-taking verdict is `scan`'s and `run`'s
job.

Worked examples with CLI output for every row are in
[TRUST-GALLERY.md](TRUST-GALLERY.md).

## The contract

- **Clean dual-channel (stereo)**
  - `trust` behavior: `eligible for scan`.
  - Exit: 0.
  - Scoring downstream: **full scoring.** `scan`, `run`, `compare`,
    `verify` all run normally.
- **Silent caller channel**
  - `trust` behavior: `NOT SCORABLE: caller channel has no detected
    speech`.
  - Exit: 2.
  - Scoring downstream: **refused.** There is no caller to measure a
    yield against.
- **Silent agent channel**
  - `trust` behavior: `NOT SCORABLE: agent channel has no detected
    speech`.
  - Exit: 2.
  - Scoring downstream: **refused.** There is no agent floor to
    interrupt.
- **Channel swap risk**
  - `trust` behavior: `eligible for scan` **plus** `possible channel
    swap` **warning**.
  - Exit: 0.
  - Scoring downstream: scores, but confirm the mapping first
    (`--caller-channel` / `--agent-channel`). A swap silently inverts
    every yield/hold.
- **High crosstalk / echo bleed**
  - `trust` behavior: `eligible for scan` **plus** high `crosstalk:
    coherence` **warning**.
  - Exit: 0.
  - Scoring downstream: scores at **lower confidence.** `scan` tags the
    moment `echo_correlated_activity`; a "yield" there may be the agent
    hearing itself.
- **Mixed mono (single channel)**
  - `trust` behavior: `NOT SCORABLE: single channel, caller and agent
    cannot be told apart`.
  - Exit: 2.
  - Scoring downstream: **refused by default.** Export dual-channel, or
    take one of the two opt-in escapes (below).
- **Mono, opt-in `--allow-mono`**
  - `trust` behavior: accepted in **degraded mode**, results **indicative
    only**.
  - Exit: 0.
  - Scoring downstream: on `capture` / `pull` / `sweep` for a mono-only
    stack. Talk-over cannot be attributed, so no SLA gate fires. The
    dual-channel path stays the gold reference.
- **Mono, opt-in `--diarize`**
  - `trust` behavior: separability **tier**: `high` / `low` / `refuse`.
  - Exit: 0 (high/low), 2 (refuse).
  - Scoring downstream: the separation front-end. `hotato run --mono
    call.wav --diarize` scores it; at `low` the verdict is stamped
    `indicative_only`. The dual-channel path stays the gold reference.
- **Short backchannel ("mhm") overlap**
  - `trust` behavior: outside `trust`'s scope; `scan` lists it as a
    **candidate**.
  - Exit: 0.
  - Scoring downstream: **candidate only, human labels.** `trust`/`scan`
    report the timing; you always label `yield` vs `hold`.
- **Noisy / false-positive candidate**
  - `trust` behavior: `trust` warns (clipping, crosstalk, hot capture);
    `scan` still lists the candidate.
  - Exit: 0.
  - Scoring downstream: surfaced **with** the warning, as a candidate for
    you to inspect and label `hold`.

**Reading the two axes.** `trust` answers one question: is this audio good
enough to score? The exit code is the machine contract -- `0` means eligible
for scan (possibly with warnings), `2` means not scorable (with the reason
and the next step). Warnings (swap, crosstalk, clipping, leading silence)
inform the read; the three hard refusals (mono, identical channels, a silent
required channel) are what changes scorability.

## Why refuse instead of guessing

Every refusal above is a place where a less careful tool would still print a
number. A mono file "scored" by guessing which speaker is which produces a
verdict indistinguishable from a properly-scored one, and it is worthless. A
swapped-channel file scored without the warning inverts caller and agent, so
every "the agent yielded" becomes "the caller yielded" with no visible sign.
Hotato treats these as input defects, reports them, and stops, so a green or
red build always means something.

## Two ways past a mono refusal

Mono is refused by default because one channel cannot separate the two
parties. Two opt-in escapes exist, both indicative only.

**1. `--allow-mono` (degraded mode).** On `capture`, `pull`, and `sweep`, the
`--allow-mono` flag (or `HOTATO_ALLOW_MONO=1`) accepts a mono-only recording
from a mono stack. Nothing is separated: talk-over cannot be attributed to
caller or agent, so the result is explicitly indicative and no SLA gate
fires. Use this when a stack only exports a mono mix and a rough signal is
still worth having.

**2. `--diarize` (separation front-end).** The opt-in `[diarize]` extra
(`pip install 'hotato[diarize]'`) runs a local diarizer and reports a
confidence tier before you score:

- **high**: confidently separable. `hotato run --mono call.wav --diarize` gives a
  diarized-mono verdict. Exit 0.
- **low**: separable but only indicative (voices close, overlap elevated). The
  verdict is stamped `indicative_only`; no SLA gate fires. Exit 0.
- **refuse**: not confidently two clean parties. Not scorable, exit 2; record
  dual-channel.

The dual-channel path stays the gold reference. Neither a degraded-mono nor a
diarized-mono verdict is promoted to equivalence with it. Full front-end:
[DIARIZE.md](DIARIZE.md).

## Composing the gate

Because `trust` exits `2` on any unscorable input, it drops straight into a
shell gate ahead of a scan or a scheduled sweep:

```bash
hotato trust --stereo call.wav && hotato scan --stereo call.wav
```

For agents, `hotato trust --stereo call.wav --format json` emits one
machine-parseable report; branch on `scorable`, and on a defect read
`not_scorable_reason` and `next_step`. Details: [TRUST.md](TRUST.md).


================================================================================
FILE: docs/VALIDATION.md
================================================================================

# What Hotato validates

A turn handoff is not one number. Hotato is validated on **three separate
jobs**, each measured on its own terms, each with an explicit reported
output. Judge each job on its own terms when you decide whether to trust it.

Everything below runs offline against recordings you control. Every threshold
is exposed and every frame is inspectable (`hotato run --dump-frames`).

---

## Job 1: timing reproducibility

**The question:** given the same recording and the same reference config, does
Hotato produce the same timing measurements every run? (Deterministic for a
fixed hotato version; byte-identical re-runs are verified in CI on Linux
x86_64, Python 3.10, 3.11, and 3.12 -- `.github/workflows/tests.yml`, job
`pytest`. The same double-run check also runs in CI on macOS and Windows,
with the digest additionally compared ACROSS those OSes -- jobs `portability`
and `determinism` in the same file. Cross-OS agreement is measured and
reported there; it is a separate finding, not part of this claim.)

**What is reported.** Per scored event: `did_yield` (true/false),
`seconds_to_yield`, and `talk_over_sec`, plus the exact thresholds used
(`max_talk_over`, `max_time_to_yield`) and the frame grid behind them. No
learned weights, no sampling, no RNG: the energy VAD and the reference framing
are deterministic, so the numbers are byte-stable.

**How to check it.** Score the same file twice and diff the output.

```text
$ hotato run --stereo 01-hard-interruption.example.wav --expect yield
hotato [single] stack=generic offline=True
  1/1 events pass  (failed=0)
  [PASS] 01-hard-interruption.example.wav: did_yield=True seconds_to_yield=0.51s talk_over=0.51s
  exit_code=0

$ hotato run --stereo 01-hard-interruption.example.wav --expect yield
  [PASS] 01-hard-interruption.example.wav: did_yield=True seconds_to_yield=0.51s talk_over=0.51s
```

`seconds_to_yield=0.51s` and `talk_over=0.51s` are identical across runs. This
is the property a regression test needs: under a fixed hotato version and the
same pinned audio, channel map, onset, label, and scoring config, a changed
result means one of those pinned inputs changed, not that the scorer drifted.

**What this job establishes.** That the measurement is stable and re-derivable
by hand from [`METHODOLOGY.md`](../METHODOLOGY.md). It is a claim about
stability, not that 0.51s is the absolute "true" yield latency or that the
reference thresholds fit your product.

---

## Job 2: candidate-discovery usefulness

**The question:** when Hotato scans a whole recording, does it surface the
moments a human reviewer would want to look at, ranked by salience?

**What is reported.** A ranked list of candidate turn-taking moments as **timing
facts only**: overlap onsets (caller became active while the agent was talking,
with the overlap length and whether the agent went silent), agent starts during
caller activity, and long response gaps. Each candidate is a timestamp and a
measurement -- the timing fact, reported without a verdict or an intent label.

```text
$ hotato scan --stereo 02-backchannel-mhm.example.wav --top 5
hotato scan: 02-backchannel-mhm.example.wav  (6.0s, 3 candidate moments)
Candidates are timing events. You decide the expected behavior; label with: hotato fixture create --onset <t> --expect yield|hold
  [ 1] t=2.09s  overlap_while_agent_talking  overlap=1.58s  agent did not go silent within 3.0s
  [ 2] t=3.19s  overlap_while_agent_talking  overlap=1.07s  agent did not go silent within 3.0s
  [ 3] t=4.29s  overlap_while_agent_talking  overlap=0.56s  agent did not go silent within 3.0s
```

The usefulness bar is **recall of human-notable moments at a workable candidate
count**, not precision against a ground-truth intent label (there is no such
label at scan time, by design). A candidate that turns out to be a harmless
backchannel is simply one you label `hold` and move on. The validation
artifact is the [trust gallery](TRUST-GALLERY.md), which
includes a deliberate false positive so you can see what an unhelpful candidate
looks like and why Hotato still surfaces it.

**What this job establishes.** That scan widens the net for you to make the
call. It is a claim about recall, not that every candidate is a bug or that a
quiet region is clean.

---

## Job 3: contract verification

**The question:** once you have labelled a moment's expected behavior
(`yield` = stop for the caller, `hold` = keep the floor through a backchannel),
does Hotato's PASS/FAIL verdict agree with that label on the audio, against an
explicit, portable, CI-enforced policy?

Today this job runs on a fixture (`hotato fixture create` / `hotato run`): a
labelled recording plus an explicit threshold policy, scored the same way on
every machine covered by CI (Linux, established; macOS and Windows now run the
same scoring code path too, pending a first green determinism run -- see Job 1
above). The portable contract bundle (`hotato contract create` /
`hotato contract verify`, audio plus timing evidence plus trace evidence plus
label plus policy plus a CI command in one artifact) carries this exact job
forward into a single self-contained object once it ships; the verdict this
job validates keeps its shape, only the artifact it travels in changes.

**What is reported.** Per fixture: the verdict (`PASS`/`FAIL`), the measured
signals behind it, and the named fix class when the failure maps cleanly to a
config family. Agreement is checked against **your** label, not against an
opaque key.

```text
$ hotato demo --no-open --format text
hotato demo: recorded calls a provider's default agent fails
hotato [suite] stack=generic offline=True
  0/2 events pass  (failed=2)
  [FAIL] fd-01-missed-interruption: did_yield=False seconds_to_yield=- talk_over=0.25s
         fix[config]: Missed interruption: the agent kept talking over the caller
  [FAIL] fd-02-backchannel-yielded: did_yield=True seconds_to_yield=0.34s talk_over=0.32s
         fix[engagement-control]: False barge-in: a backchannel was treated as a bid for the floor
  note: no single sensitivity threshold satisfies this battery
  exit_code=1
```

Both labelled failures are caught, and the battery-level note reports the
disagreement plainly when a missed interruption and a false stop fail in the
same run -- no single threshold satisfies both. Reporting that disagreement
instead of inventing a fix is part of the validated behavior.

**What this job establishes.** That the verdict follows the audio and the
label consistently. It is a claim about consistency, not that the label was
correct (you own the label) or that a passing fixture means the agent is good
in general.

---

## What Hotato measures, and where it stops

Read this as the scope of the claim, stated once, plainly.

- **Timing, not semantic intent.** Hotato measures timing; whether a caller
  sound meant "stop" or "mhm, go on" is a label you supply.
- **A likely layer, not certainty.** A slow yield can be TTS buffering,
  transport, or VAD. `diagnose` names a likely layer and stays
  `unknown_root_cause` when one recording cannot separate them. A voice trace
  (once the trace layer ships) narrows the candidates further, short of
  turning one into a proof.
- **Timing, not task success.** Whether the call booked the appointment,
  resolved the ticket, or satisfied the caller is a QA platform's job (see
  [COMPARE.md](COMPARE.md)).
- **A demonstration, not a vendor ranking.** Hotato scores calls, not
  platforms. A provider-default example demonstrates the threshold funnel on
  one assistant, one config, one date, one scripted caller.
- **Timing, not tone.** Sentiment, satisfaction, and CSAT sit outside what
  Hotato measures.
- **Reproducible timing measurements, with the method exposed.** The three
  jobs above are the whole claim -- deliberately no headline percentage.

The validation plan for the launch battery (external testers, consented
fixtures, before/after) lives in
[docs/evidence/validation-plan.md](evidence/validation-plan.md).


================================================================================
FILE: src/hotato/schema/counterexample-oracle.v1.json
================================================================================

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://hotato.dev/schema/counterexample-oracle.v1.json",
  "title": "Hotato deterministic counterexample oracle",
  "type": "object",
  "additionalProperties": false,
  "required": ["kind", "version", "authority", "ci_gate_eligible", "target", "observation_scope"],
  "properties": {
    "kind": {"const": "hotato.counterexample-oracle.v1"},
    "version": {"const": 1},
    "authority": {"const": "deterministic"},
    "ci_gate_eligible": {"const": true},
    "target": {"$ref": "#/definitions/target"},
    "observation_scope": {"$ref": "#/definitions/observation_scope"}
  },
  "definitions": {
    "digest": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"},
    "nonnegative_integer": {"type": "integer", "minimum": 0},
    "dimension": {
      "type": ["string", "null"],
      "enum": ["outcome", "policy", "conversation", "speech", "reliability", null]
    },
    "assertion_kind": {
      "enum": [
        "phrase", "pii", "policy", "tool_call", "outcome", "tool_result",
        "tool_error", "state", "state_change", "handoff",
        "termination", "latency", "entity_accuracy", "sequence", "count"
      ]
    },
    "failure_atom": {
      "$comment": "Closed payload-free branch identity; kind coupling is enforced by the target schema.",
      "oneOf": [
        {
          "type": "object", "additionalProperties": false,
          "required": ["code"],
          "properties": {
            "code": {"enum": [
              "forbidden-match", "no-qualifying-turns", "required-match-missing",
              "tool-missing", "tool-arguments-missing", "tool-count-below",
              "tool-count-above", "never-before-boundary-missing",
              "never-before-order-violation",
              "no-predicate-met", "result-missing", "unexpected-tool-error",
              "tool-error-missing", "tool-error-pattern-mismatch", "state-record-missing",
              "unexpected-handoff", "handoff-missing", "handoff-target-mismatch",
              "unexpected-termination", "termination-missing",
              "latency-declared-threshold-exceeded", "latency-default-threshold-exceeded",
              "count-below", "count-above"
            ]}
          }
        },
        {
          "type": "object", "additionalProperties": false,
          "required": ["code", "index"],
          "properties": {
            "code": {"enum": ["order-step-absent", "order-step-out-of-order", "predicate-unmet", "sequence-step-absent", "sequence-step-out-of-order"]},
            "index": {"$ref": "#/definitions/nonnegative_integer"}
          }
        },
        {
          "type": "object", "additionalProperties": false,
          "required": ["code", "detector"],
          "properties": {
            "code": {"const": "pii-detected"},
            "detector": {"enum": ["ssn", "card_luhn", "email", "phone"]}
          }
        },
        {
          "type": "object", "additionalProperties": false,
          "required": ["code", "rule", "type"],
          "properties": {
            "code": {"const": "policy-violation"},
            "rule": {"type": "string", "minLength": 1},
            "type": {"enum": ["banned", "required_disclosure_missing"]}
          }
        },
        {
          "type": "object", "additionalProperties": false,
          "required": ["code", "field"],
          "properties": {
            "code": {"enum": ["state-field-missing", "state-field-value-mismatch", "before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged", "termination-attribute-missing", "termination-attribute-value-mismatch"]},
            "field": {"type": "string", "minLength": 1}
          }
        },
        {
          "type": "object", "additionalProperties": false,
          "required": ["code", "key"],
          "properties": {
            "code": {"enum": ["entity-missing", "entity-value-mismatch", "tool-argument-field-missing", "tool-argument-value-mismatch", "result-field-missing", "result-field-value-mismatch"]},
            "key": {"type": "string", "minLength": 1}
          }
        }
      ]
    },
    "target_kind_atom_coupling": {
      "allOf": [
        {
          "if": {"properties": {"kind": {"const": "phrase"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_phrase"}
        },
        {
          "if": {"properties": {"kind": {"const": "pii"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_pii"}
        },
        {
          "if": {"properties": {"kind": {"const": "policy"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_policy"}
        },
        {
          "if": {"properties": {"kind": {"const": "tool_call"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_tool_call"}
        },
        {
          "if": {"properties": {"kind": {"const": "outcome"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_outcome"}
        },
        {
          "if": {"properties": {"kind": {"const": "tool_result"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_tool_result"}
        },
        {
          "if": {"properties": {"kind": {"const": "tool_error"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_tool_error"}
        },
        {
          "if": {"properties": {"kind": {"const": "state"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_state"}
        },
        {
          "if": {"properties": {"kind": {"const": "state_change"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_state_change"}
        },
        {
          "if": {"properties": {"kind": {"const": "handoff"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_handoff"}
        },
        {
          "if": {"properties": {"kind": {"const": "termination"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_termination"}
        },
        {
          "if": {"properties": {"kind": {"const": "latency"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_latency"}
        },
        {
          "if": {"properties": {"kind": {"const": "entity_accuracy"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_entity_accuracy"}
        },
        {
          "if": {"properties": {"kind": {"const": "sequence"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_sequence"}
        },
        {
          "if": {"properties": {"kind": {"const": "count"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_count"}
        }
      ]
    },
    "target_atom_codes_phrase": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["forbidden-match", "no-qualifying-turns", "required-match-missing"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["forbidden-match", "no-qualifying-turns", "required-match-missing"]}}}}
      }
    },
    "target_atom_codes_pii": {
      "properties": {
        "failure_atom": {"properties": {"code": {"const": "pii-detected"}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"const": "pii-detected"}}}}
      }
    },
    "target_atom_codes_policy": {
      "properties": {
        "failure_atom": {"properties": {"code": {"const": "policy-violation"}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"const": "policy-violation"}}}}
      }
    },
    "target_atom_codes_tool_call": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["tool-missing", "tool-arguments-missing", "tool-argument-field-missing", "tool-argument-value-mismatch", "tool-count-below", "tool-count-above", "order-step-absent", "order-step-out-of-order", "never-before-boundary-missing", "never-before-order-violation"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["tool-missing", "tool-arguments-missing", "tool-argument-field-missing", "tool-argument-value-mismatch", "tool-count-below", "tool-count-above", "order-step-absent", "order-step-out-of-order", "never-before-boundary-missing", "never-before-order-violation"]}}}}
      }
    },
    "target_atom_codes_outcome": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["predicate-unmet", "no-predicate-met"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["predicate-unmet", "no-predicate-met"]}}}}
      }
    },
    "target_atom_codes_tool_result": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["tool-missing", "result-missing", "result-field-missing", "result-field-value-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["tool-missing", "result-missing", "result-field-missing", "result-field-value-mismatch"]}}}}
      }
    },
    "target_atom_codes_tool_error": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["tool-missing", "unexpected-tool-error", "tool-error-missing", "tool-error-pattern-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["tool-missing", "unexpected-tool-error", "tool-error-missing", "tool-error-pattern-mismatch"]}}}}
      }
    },
    "target_atom_codes_state": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["state-record-missing", "state-field-missing", "state-field-value-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["state-record-missing", "state-field-missing", "state-field-value-mismatch"]}}}}
      }
    },
    "target_atom_codes_state_change": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged"]}}}}
      }
    },
    "target_atom_codes_handoff": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["unexpected-handoff", "handoff-missing", "handoff-target-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["unexpected-handoff", "handoff-missing", "handoff-target-mismatch"]}}}}
      }
    },
    "target_atom_codes_termination": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["unexpected-termination", "termination-missing", "termination-attribute-missing", "termination-attribute-value-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["unexpected-termination", "termination-missing", "termination-attribute-missing", "termination-attribute-value-mismatch"]}}}}
      }
    },
    "target_atom_codes_latency": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["latency-declared-threshold-exceeded", "latency-default-threshold-exceeded"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["latency-declared-threshold-exceeded", "latency-default-threshold-exceeded"]}}}}
      }
    },
    "target_atom_codes_entity_accuracy": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["entity-missing", "entity-value-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["entity-missing", "entity-value-mismatch"]}}}}
      }
    },
    "target_atom_codes_sequence": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["sequence-step-absent", "sequence-step-out-of-order"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["sequence-step-absent", "sequence-step-out-of-order"]}}}}
      }
    },
    "target_atom_codes_count": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["count-below", "count-above"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["count-below", "count-above"]}}}}
      }
    },
    "target": {
      "$comment": "Runtime validation also requires canonical UTF-8 atom ordering, de-duplication, selected atom first, and a matching fingerprint.",
      "type": "object", "additionalProperties": false,
      "required": [
        "test_id", "assertion_digest", "assertion_id", "kind", "dimension",
        "authority", "required_status", "failure_atom", "source_failure_atoms",
        "fingerprint"
      ],
      "properties": {
        "test_id": {"type": "string", "minLength": 1},
        "assertion_digest": {"$ref": "#/definitions/digest"},
        "assertion_id": {"type": "string", "minLength": 1},
        "kind": {"$ref": "#/definitions/assertion_kind"},
        "dimension": {"$ref": "#/definitions/dimension"},
        "authority": {"const": "deterministic"}, "required_status": {"const": "FAIL"},
        "failure_atom": {"$ref": "#/definitions/failure_atom"},
        "source_failure_atoms": {
          "type": "array", "minItems": 1, "uniqueItems": true,
          "items": {"$ref": "#/definitions/failure_atom"}
        },
        "fingerprint": {"$ref": "#/definitions/digest"}
      },
      "allOf": [{"$ref": "#/definitions/target_kind_atom_coupling"}]
    },
    "observation_scope": {
      "type": "object", "additionalProperties": false,
      "required": ["frozen_components", "rule", "minimum_caller_turns", "transforms"],
      "properties": {
        "frozen_components": {
          "type": "array", "uniqueItems": true,
          "items": {"enum": ["script", "tools", "state", "handoff", "termination"]}
        },
        "rule": {
          "const": "candidate is a complete schema-valid scripted session and must preserve the source-selected structured failure branch"
        },
        "minimum_caller_turns": {"const": 1},
        "transforms": {"const": "deletion-only"}
      }
    }
  }
}


================================================================================
FILE: src/hotato/schema/reduction-certificate.v1.json
================================================================================

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://hotato.dev/schema/reduction-certificate.v1.json",
  "title": "Hotato counterexample reduction certificate",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "kind", "version", "algorithm", "reducer_set", "source_scenario_digest",
    "final_scenario_digest", "source_test_digest", "final_test_digest",
    "failure_fingerprint", "accepted_steps", "journal_sha256", "budget",
    "candidate_evaluations", "cache_hits", "termination"
  ],
  "properties": {
    "kind": {"const": "hotato.reduction-certificate.v1"},
    "version": {"const": 1},
    "algorithm": {"const": "hierarchical-ddmin"},
    "reducer_set": {"const": "hotato.reducers.v1"},
    "source_scenario_digest": {"$ref": "#/definitions/digest"},
    "final_scenario_digest": {"$ref": "#/definitions/digest"},
    "source_test_digest": {"$ref": "#/definitions/digest"},
    "final_test_digest": {"$ref": "#/definitions/digest"},
    "failure_fingerprint": {"$ref": "#/definitions/digest"},
    "accepted_steps": {
      "type": "array", "maxItems": 512,
      "items": {"$ref": "#/definitions/accepted_step"}
    },
    "journal_sha256": {"$ref": "#/definitions/digest"},
    "budget": {"type": "integer", "minimum": 1, "maximum": 100000},
    "candidate_evaluations": {"type": "integer", "minimum": 0, "maximum": 100000},
    "cache_hits": {"type": "integer", "minimum": 0},
    "termination": {"enum": ["one_minimal", "budget_exhausted"]}
  },
  "definitions": {
    "digest": {
      "type": "string",
      "pattern": "^sha256:[0-9a-f]{64}$"
    },
    "raw_digest": {
      "type": "string",
      "pattern": "^[0-9a-f]{64}$"
    },
    "path_segment": {
      "oneOf": [
        {"type": "string"},
        {"type": "integer", "minimum": 0}
      ]
    },
    "deletion_operation": {
      "type": "object",
      "additionalProperties": false,
      "required": ["op", "path", "removed_digest"],
      "properties": {
        "op": {"const": "remove"},
        "path": {
          "type": "array", "minItems": 1, "maxItems": 96,
          "items": {"$ref": "#/definitions/path_segment"}
        },
        "removed_digest": {"$ref": "#/definitions/digest"}
      }
    },
    "deletion_transform": {
      "type": "object",
      "additionalProperties": false,
      "required": ["kind", "operations"],
      "properties": {
        "kind": {"const": "hotato.delete-only.v1"},
        "operations": {
          "type": "array", "minItems": 1, "maxItems": 10000,
          "uniqueItems": true,
          "items": {"$ref": "#/definitions/deletion_operation"}
        }
      }
    },
    "remove_field": {
      "type": "object",
      "additionalProperties": false,
      "required": ["kind", "phase", "path"],
      "properties": {
        "kind": {"const": "remove-field"},
        "phase": {"type": "string", "minLength": 1},
        "path": {"type": "string", "minLength": 1}
      }
    },
    "remove_path_set": {
      "type": "object",
      "additionalProperties": false,
      "required": ["kind", "phase", "paths"],
      "properties": {
        "kind": {"const": "remove-path-set"},
        "phase": {"type": "string", "minLength": 1},
        "paths": {
          "type": "array", "minItems": 1, "maxItems": 10000,
          "uniqueItems": true,
          "items": {"type": "string", "minLength": 1}
        }
      }
    },
    "component": {
      "enum": [
        "top-level-optional", "goal-optional", "caller-optional",
        "variation_matrix", "facts", "environment", "interruptions", "behavior",
        "script", "script-field", "tools", "tool-field", "handoff-field", "handoff",
        "termination-field", "termination", "state", "agent-mock-optional", "agent_mock"
      ]
    },
    "remove_single_unit": {
      "type": "object",
      "additionalProperties": false,
      "required": ["kind", "phase", "path", "component"],
      "properties": {
        "kind": {"const": "remove-single-unit"},
        "phase": {"enum": ["one-minimality", "minimality-proof-restart"]},
        "path": {"type": "string", "minLength": 1},
        "component": {"$ref": "#/definitions/component"}
      }
    },
    "reducer_operation": {
      "oneOf": [
        {"$ref": "#/definitions/remove_field"},
        {"$ref": "#/definitions/remove_path_set"},
        {"$ref": "#/definitions/remove_single_unit"}
      ]
    },
    "accepted_step": {
      "type": "object",
      "additionalProperties": false,
      "required": [
        "step", "parent_digest", "child_digest", "operation", "transform",
        "oracle_result_digest", "failure_atom_digest"
      ],
      "properties": {
        "step": {"type": "integer", "minimum": 1, "maximum": 512},
        "parent_digest": {"$ref": "#/definitions/raw_digest"},
        "child_digest": {"$ref": "#/definitions/raw_digest"},
        "operation": {"$ref": "#/definitions/reducer_operation"},
        "transform": {"$ref": "#/definitions/deletion_transform"},
        "oracle_result_digest": {"$ref": "#/definitions/digest"},
        "failure_atom_digest": {"$ref": "#/definitions/digest"}
      }
    }
  }
}


================================================================================
FILE: src/hotato/schema/counterexample.v1.json
================================================================================

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://hotato.dev/schema/counterexample.v1.json",
  "title": "Hotato proof-preserving counterexample capsule",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "kind", "version", "counterexample_id", "source", "target",
    "reduction", "minimality", "preservation", "privacy", "provenance"
  ],
  "properties": {
    "kind": {"const": "hotato.counterexample.v1"},
    "version": {"const": 1},
    "counterexample_id": {"$ref": "#/definitions/digest"},
    "source": {},
    "target": {},
    "oracle": {"$ref": "#/definitions/oracle_reference"},
    "artifacts": {"$ref": "#/definitions/artifacts"},
    "artifact_digests": {"$ref": "#/definitions/artifact_digests"},
    "reduction": {"$ref": "#/definitions/reduction"},
    "minimality": {},
    "preservation": {"$ref": "#/definitions/preservation"},
    "privacy": {},
    "provenance": {}
  },
  "allOf": [
    {
      "oneOf": [
        {
          "required": ["oracle", "artifacts", "artifact_digests"],
          "properties": {
            "source": {"$ref": "#/definitions/private_source"},
            "target": {"$ref": "#/definitions/private_target"},
            "minimality": {"$ref": "#/definitions/private_minimality"},
            "privacy": {"$ref": "#/definitions/private_privacy"},
            "provenance": {"$ref": "#/definitions/private_provenance"}
          }
        },
        {
          "not": {
            "anyOf": [
              {"required": ["oracle"]}, {"required": ["artifacts"]},
              {"required": ["artifact_digests"]}
            ]
          },
          "properties": {
            "source": {"$ref": "#/definitions/share_source"},
            "target": {"$ref": "#/definitions/share_target"},
            "minimality": {"$ref": "#/definitions/share_minimality"},
            "privacy": {"$ref": "#/definitions/share_privacy"},
            "provenance": {"$ref": "#/definitions/share_provenance"}
          }
        }
      ]
    },
    {
      "oneOf": [
        {
          "properties": {
            "reduction": {"properties": {"termination": {"const": "one_minimal"}}},
            "minimality": {"properties": {"status": {"const": "one_minimal"}}}
          }
        },
        {
          "properties": {
            "reduction": {"properties": {"termination": {"const": "budget_exhausted"}}},
            "minimality": {"properties": {"status": {"const": "budget_exhausted"}}}
          }
        }
      ]
    }
  ],
  "definitions": {
    "digest": {
      "type": "string",
      "pattern": "^sha256:[0-9a-f]{64}$"
    },
    "raw_digest": {
      "type": "string",
      "pattern": "^[0-9a-f]{64}$"
    },
    "dimension": {
      "type": ["string", "null"],
      "enum": ["outcome", "policy", "conversation", "speech", "reliability", null]
    },
    "assertion_kind": {
      "enum": [
        "phrase", "pii", "policy", "tool_call", "outcome", "tool_result",
        "tool_error", "state", "state_change", "handoff",
        "termination", "latency", "entity_accuracy", "sequence", "count"
      ]
    },
    "nonnegative_integer": {"type": "integer", "minimum": 0},
    "failure_atom": {
      "$comment": "Closed payload-free branch identity; kind coupling is enforced by the target schema.",
      "oneOf": [
        {
          "type": "object", "additionalProperties": false,
          "required": ["code"],
          "properties": {
            "code": {"enum": [
              "forbidden-match", "no-qualifying-turns", "required-match-missing",
              "tool-missing", "tool-arguments-missing", "tool-count-below",
              "tool-count-above", "never-before-boundary-missing",
              "never-before-order-violation",
              "no-predicate-met", "result-missing", "unexpected-tool-error",
              "tool-error-missing", "tool-error-pattern-mismatch", "state-record-missing",
              "unexpected-handoff", "handoff-missing", "handoff-target-mismatch",
              "unexpected-termination", "termination-missing",
              "latency-declared-threshold-exceeded", "latency-default-threshold-exceeded",
              "count-below", "count-above"
            ]}
          }
        },
        {
          "type": "object", "additionalProperties": false,
          "required": ["code", "index"],
          "properties": {
            "code": {"enum": ["order-step-absent", "order-step-out-of-order", "predicate-unmet", "sequence-step-absent", "sequence-step-out-of-order"]},
            "index": {"$ref": "#/definitions/nonnegative_integer"}
          }
        },
        {
          "type": "object", "additionalProperties": false,
          "required": ["code", "detector"],
          "properties": {
            "code": {"const": "pii-detected"},
            "detector": {"enum": ["ssn", "card_luhn", "email", "phone"]}
          }
        },
        {
          "type": "object", "additionalProperties": false,
          "required": ["code", "rule", "type"],
          "properties": {
            "code": {"const": "policy-violation"},
            "rule": {"type": "string", "minLength": 1},
            "type": {"enum": ["banned", "required_disclosure_missing"]}
          }
        },
        {
          "type": "object", "additionalProperties": false,
          "required": ["code", "field"],
          "properties": {
            "code": {"enum": ["state-field-missing", "state-field-value-mismatch", "before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged", "termination-attribute-missing", "termination-attribute-value-mismatch"]},
            "field": {"type": "string", "minLength": 1}
          }
        },
        {
          "type": "object", "additionalProperties": false,
          "required": ["code", "key"],
          "properties": {
            "code": {"enum": ["entity-missing", "entity-value-mismatch", "tool-argument-field-missing", "tool-argument-value-mismatch", "result-field-missing", "result-field-value-mismatch"]},
            "key": {"type": "string", "minLength": 1}
          }
        }
      ]
    },
    "target_kind_atom_coupling": {
      "allOf": [
        {
          "if": {"properties": {"kind": {"const": "phrase"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_phrase"}
        },
        {
          "if": {"properties": {"kind": {"const": "pii"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_pii"}
        },
        {
          "if": {"properties": {"kind": {"const": "policy"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_policy"}
        },
        {
          "if": {"properties": {"kind": {"const": "tool_call"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_tool_call"}
        },
        {
          "if": {"properties": {"kind": {"const": "outcome"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_outcome"}
        },
        {
          "if": {"properties": {"kind": {"const": "tool_result"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_tool_result"}
        },
        {
          "if": {"properties": {"kind": {"const": "tool_error"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_tool_error"}
        },
        {
          "if": {"properties": {"kind": {"const": "state"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_state"}
        },
        {
          "if": {"properties": {"kind": {"const": "state_change"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_state_change"}
        },
        {
          "if": {"properties": {"kind": {"const": "handoff"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_handoff"}
        },
        {
          "if": {"properties": {"kind": {"const": "termination"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_termination"}
        },
        {
          "if": {"properties": {"kind": {"const": "latency"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_latency"}
        },
        {
          "if": {"properties": {"kind": {"const": "entity_accuracy"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_entity_accuracy"}
        },
        {
          "if": {"properties": {"kind": {"const": "sequence"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_sequence"}
        },
        {
          "if": {"properties": {"kind": {"const": "count"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_count"}
        }
      ]
    },
    "target_atom_codes_phrase": {
      "properties": {
        "failure_code": {"enum": ["forbidden-match", "no-qualifying-turns", "required-match-missing"]},
        "failure_atom": {"properties": {"code": {"enum": ["forbidden-match", "no-qualifying-turns", "required-match-missing"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["forbidden-match", "no-qualifying-turns", "required-match-missing"]}}}}
      }
    },
    "target_atom_codes_pii": {
      "properties": {
        "failure_code": {"const": "pii-detected"},
        "failure_atom": {"properties": {"code": {"const": "pii-detected"}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"const": "pii-detected"}}}}
      }
    },
    "target_atom_codes_policy": {
      "properties": {
        "failure_code": {"const": "policy-violation"},
        "failure_atom": {"properties": {"code": {"const": "policy-violation"}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"const": "policy-violation"}}}}
      }
    },
    "target_atom_codes_tool_call": {
      "properties": {
        "failure_code": {"enum": ["tool-missing", "tool-arguments-missing", "tool-argument-field-missing", "tool-argument-value-mismatch", "tool-count-below", "tool-count-above", "order-step-absent", "order-step-out-of-order", "never-before-boundary-missing", "never-before-order-violation"]},
        "failure_atom": {"properties": {"code": {"enum": ["tool-missing", "tool-arguments-missing", "tool-argument-field-missing", "tool-argument-value-mismatch", "tool-count-below", "tool-count-above", "order-step-absent", "order-step-out-of-order", "never-before-boundary-missing", "never-before-order-violation"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["tool-missing", "tool-arguments-missing", "tool-argument-field-missing", "tool-argument-value-mismatch", "tool-count-below", "tool-count-above", "order-step-absent", "order-step-out-of-order", "never-before-boundary-missing", "never-before-order-violation"]}}}}
      }
    },
    "target_atom_codes_outcome": {
      "properties": {
        "failure_code": {"enum": ["predicate-unmet", "no-predicate-met"]},
        "failure_atom": {"properties": {"code": {"enum": ["predicate-unmet", "no-predicate-met"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["predicate-unmet", "no-predicate-met"]}}}}
      }
    },
    "target_atom_codes_tool_result": {
      "properties": {
        "failure_code": {"enum": ["tool-missing", "result-missing", "result-field-missing", "result-field-value-mismatch"]},
        "failure_atom": {"properties": {"code": {"enum": ["tool-missing", "result-missing", "result-field-missing", "result-field-value-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["tool-missing", "result-missing", "result-field-missing", "result-field-value-mismatch"]}}}}
      }
    },
    "target_atom_codes_tool_error": {
      "properties": {
        "failure_code": {"enum": ["tool-missing", "unexpected-tool-error", "tool-error-missing", "tool-error-pattern-mismatch"]},
        "failure_atom": {"properties": {"code": {"enum": ["tool-missing", "unexpected-tool-error", "tool-error-missing", "tool-error-pattern-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["tool-missing", "unexpected-tool-error", "tool-error-missing", "tool-error-pattern-mismatch"]}}}}
      }
    },
    "target_atom_codes_state": {
      "properties": {
        "failure_code": {"enum": ["state-record-missing", "state-field-missing", "state-field-value-mismatch"]},
        "failure_atom": {"properties": {"code": {"enum": ["state-record-missing", "state-field-missing", "state-field-value-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["state-record-missing", "state-field-missing", "state-field-value-mismatch"]}}}}
      }
    },
    "target_atom_codes_state_change": {
      "properties": {
        "failure_code": {"enum": ["before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged"]},
        "failure_atom": {"properties": {"code": {"enum": ["before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged"]}}}}
      }
    },
    "target_atom_codes_handoff": {
      "properties": {
        "failure_code": {"enum": ["unexpected-handoff", "handoff-missing", "handoff-target-mismatch"]},
        "failure_atom": {"properties": {"code": {"enum": ["unexpected-handoff", "handoff-missing", "handoff-target-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["unexpected-handoff", "handoff-missing", "handoff-target-mismatch"]}}}}
      }
    },
    "target_atom_codes_termination": {
      "properties": {
        "failure_code": {"enum": ["unexpected-termination", "termination-missing", "termination-attribute-missing", "termination-attribute-value-mismatch"]},
        "failure_atom": {"properties": {"code": {"enum": ["unexpected-termination", "termination-missing", "termination-attribute-missing", "termination-attribute-value-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["unexpected-termination", "termination-missing", "termination-attribute-missing", "termination-attribute-value-mismatch"]}}}}
      }
    },
    "target_atom_codes_latency": {
      "properties": {
        "failure_code": {"enum": ["latency-declared-threshold-exceeded", "latency-default-threshold-exceeded"]},
        "failure_atom": {"properties": {"code": {"enum": ["latency-declared-threshold-exceeded", "latency-default-threshold-exceeded"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["latency-declared-threshold-exceeded", "latency-default-threshold-exceeded"]}}}}
      }
    },
    "target_atom_codes_entity_accuracy": {
      "properties": {
        "failure_code": {"enum": ["entity-missing", "entity-value-mismatch"]},
        "failure_atom": {"properties": {"code": {"enum": ["entity-missing", "entity-value-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["entity-missing", "entity-value-mismatch"]}}}}
      }
    },
    "target_atom_codes_sequence": {
      "properties": {
        "failure_code": {"enum": ["sequence-step-absent", "sequence-step-out-of-order"]},
        "failure_atom": {"properties": {"code": {"enum": ["sequence-step-absent", "sequence-step-out-of-order"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["sequence-step-absent", "sequence-step-out-of-order"]}}}}
      }
    },
    "target_atom_codes_count": {
      "properties": {
        "failure_code": {"enum": ["count-below", "count-above"]},
        "failure_atom": {"properties": {"code": {"enum": ["count-below", "count-above"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["count-below", "count-above"]}}}}
      }
    },
    "private_target": {
      "$comment": "Runtime validation also requires canonical UTF-8 atom ordering, de-duplication, selected atom first, and a matching fingerprint.",
      "type": "object", "additionalProperties": false,
      "required": [
        "test_id", "assertion_digest", "assertion_id", "kind", "dimension",
        "authority", "required_status", "failure_atom", "source_failure_atoms",
        "fingerprint"
      ],
      "properties": {
        "test_id": {"type": "string", "minLength": 1},
        "assertion_digest": {"$ref": "#/definitions/digest"},
        "assertion_id": {"type": "string", "minLength": 1},
        "kind": {"$ref": "#/definitions/assertion_kind"},
        "dimension": {"$ref": "#/definitions/dimension"},
        "authority": {"const": "deterministic"},
        "required_status": {"const": "FAIL"},
        "failure_atom": {"$ref": "#/definitions/failure_atom"},
        "source_failure_atoms": {
          "type": "array", "minItems": 1, "uniqueItems": true,
          "items": {"$ref": "#/definitions/failure_atom"}
        },
        "fingerprint": {"$ref": "#/definitions/digest"}
      },
      "allOf": [{"$ref": "#/definitions/target_kind_atom_coupling"}]
    },
    "share_target": {
      "$comment": "failure_code is the selected private atom code; failure_atom_digest binds the complete private atom.",
      "type": "object", "additionalProperties": false,
      "required": ["assertion_ref", "kind", "dimension", "authority", "required_status", "fingerprint", "failure_code", "failure_atom_digest"],
      "properties": {
        "assertion_ref": {"$ref": "#/definitions/digest"},
        "kind": {"$ref": "#/definitions/assertion_kind"},
        "dimension": {"$ref": "#/definitions/dimension"},
        "authority": {"const": "deterministic"},
        "required_status": {"const": "FAIL"},
        "fingerprint": {"$ref": "#/definitions/digest"},
        "failure_code": {"type": "string", "minLength": 1},
        "failure_atom_digest": {"$ref": "#/definitions/digest"}
      },
      "allOf": [{"$ref": "#/definitions/target_kind_atom_coupling"}]
    },
    "private_source": {
      "type": "object", "additionalProperties": false,
      "required": ["scenario_file_sha256", "test_file_sha256", "scenario_digest", "test_digest"],
      "properties": {
        "scenario_file_sha256": {"$ref": "#/definitions/digest"},
        "test_file_sha256": {"$ref": "#/definitions/digest"},
        "scenario_digest": {"$ref": "#/definitions/digest"},
        "test_digest": {"$ref": "#/definitions/digest"}
      }
    },
    "share_source": {
      "type": "object", "additionalProperties": false,
      "required": ["scenario_digest", "test_digest"],
      "properties": {
        "scenario_digest": {"$ref": "#/definitions/digest"},
        "test_digest": {"$ref": "#/definitions/digest"}
      }
    },
    "oracle_reference": {
      "type": "object", "additionalProperties": false,
      "required": ["path", "digest"],
      "properties": {"path": {"const": "oracle.json"}, "digest": {"$ref": "#/definitions/digest"}}
    },
    "artifacts": {
      "type": "object", "additionalProperties": false,
      "required": [
        "source_scenario", "source_conversation_test", "source_scenario_file",
        "source_conversation_test_file", "scenario", "conversation_test",
        "expected_result", "certificate", "journal", "minimality",
        "report_markdown", "report_html", "share_card",
        "reproduce_script", "predicate_script"
      ],
      "properties": {
        "source_scenario": {"const": "source/scenario.json"},
        "source_conversation_test": {"const": "source/conversation-test.json"},
        "source_scenario_file": {"const": "source/scenario.original"},
        "source_conversation_test_file": {"const": "source/conversation-test.original"},
        "scenario": {"const": "input/scenario.json"},
        "conversation_test": {"const": "input/conversation-test.json"},
        "expected_result": {"const": "expected/assertion-result.json"},
        "certificate": {"const": "certificate.json"},
        "journal": {"const": "reduction.jsonl"},
        "minimality": {"const": "minimality.json"},
        "report_markdown": {"const": "report.md"},
        "report_html": {"const": "report.html"},
        "share_card": {"const": "card.svg"},
        "reproduce_script": {"const": "reproduce.sh"},
        "predicate_script": {"const": "predicate.sh"}
      }
    },
    "artifact_digests": {
      "type": "object", "additionalProperties": false,
      "required": [
        "source_scenario", "source_conversation_test", "source_scenario_file",
        "source_conversation_test_file", "scenario", "conversation_test",
        "expected_result", "certificate", "journal", "minimality",
        "report_markdown", "report_html", "share_card",
        "reproduce_script", "predicate_script"
      ],
      "properties": {
        "source_scenario": {"$ref": "#/definitions/digest"},
        "source_conversation_test": {"$ref": "#/definitions/digest"},
        "source_scenario_file": {"$ref": "#/definitions/digest"},
        "source_conversation_test_file": {"$ref": "#/definitions/digest"},
        "scenario": {"$ref": "#/definitions/digest"},
        "conversation_test": {"$ref": "#/definitions/digest"},
        "expected_result": {"$ref": "#/definitions/digest"},
        "certificate": {"$ref": "#/definitions/digest"},
        "journal": {"$ref": "#/definitions/digest"},
        "minimality": {"$ref": "#/definitions/digest"},
        "report_markdown": {"$ref": "#/definitions/digest"},
        "report_html": {"$ref": "#/definitions/digest"},
        "share_card": {"$ref": "#/definitions/digest"},
        "reproduce_script": {"$ref": "#/definitions/digest"},
        "predicate_script": {"$ref": "#/definitions/digest"}
      }
    },
    "stats": {
      "type": "object", "additionalProperties": false,
      "required": ["bytes", "turns", "tools", "state_leaves", "transcript_segments", "trace_spans"],
      "properties": {
        "bytes": {"type": "integer", "minimum": 1},
        "turns": {"type": "integer", "minimum": 1, "maximum": 10000},
        "tools": {"type": "integer", "minimum": 0, "maximum": 10000},
        "state_leaves": {"$ref": "#/definitions/nonnegative_integer"},
        "transcript_segments": {"$ref": "#/definitions/nonnegative_integer"},
        "trace_spans": {"$ref": "#/definitions/nonnegative_integer"}
      }
    },
    "reduction": {
      "type": "object", "additionalProperties": false,
      "required": [
        "algorithm", "reducer_set", "initial", "final", "attempts",
        "candidate_evaluations", "qualification_evaluations", "total_evaluations",
        "accepted", "cache_hits", "budget", "termination"
      ],
      "properties": {
        "algorithm": {"const": "hierarchical-ddmin"},
        "reducer_set": {"const": "hotato.reducers.v1"},
        "initial": {"$ref": "#/definitions/stats"},
        "final": {"$ref": "#/definitions/stats"},
        "attempts": {"$ref": "#/definitions/nonnegative_integer"},
        "candidate_evaluations": {"type": "integer", "minimum": 0, "maximum": 100000},
        "qualification_evaluations": {"const": 4},
        "total_evaluations": {"type": "integer", "minimum": 4, "maximum": 100004},
        "accepted": {"type": "integer", "minimum": 0, "maximum": 512},
        "cache_hits": {"$ref": "#/definitions/nonnegative_integer"},
        "budget": {"type": "integer", "minimum": 1, "maximum": 100000},
        "termination": {"enum": ["one_minimal", "budget_exhausted"]}
      }
    },
    "minimality_check": {
      "type": "object", "additionalProperties": false,
      "required": ["path", "component", "outcome", "code", "candidate_digest"],
      "properties": {
        "path": {"type": "string", "minLength": 1},
        "component": {"enum": [
          "top-level-optional", "goal-optional", "caller-optional",
          "variation_matrix", "facts", "environment", "interruptions",
          "behavior", "script", "script-field", "tools", "tool-field",
          "handoff-field", "handoff", "termination-field", "termination",
          "state", "agent-mock-optional", "agent_mock"
        ]},
        "outcome": {"enum": ["PRESERVED", "ABSENT", "DRIFTED", "UNRESOLVED"]},
        "code": {"enum": [
          null, "budget_exhausted", "simulator_invalid",
          "assertion_contract_invalid", "assertion_inconclusive",
          "target_failed", "target_absent", "candidate_invalid",
          "failure_identity_drift", "failure_atom_unavailable",
          "resource_limit_exceeded"
        ]},
        "candidate_digest": {"$ref": "#/definitions/raw_digest"}
      }
    },
    "private_minimality": {
      "type": "object", "additionalProperties": false,
      "required": ["status", "reducer_set", "claim", "remaining_unit_checks", "frozen_components"],
      "properties": {
        "status": {"enum": ["one_minimal", "budget_exhausted"]},
        "reducer_set": {"const": "hotato.reducers.v1"},
        "claim": {"type": "string", "minLength": 1},
        "remaining_unit_checks": {"type": "array", "maxItems": 512, "items": {"$ref": "#/definitions/minimality_check"}},
        "frozen_components": {
          "type": "array", "uniqueItems": true,
          "items": {"enum": ["script", "tools", "state", "handoff", "termination"]}
        }
      },
      "allOf": [
        {
          "if": {"properties": {"status": {"const": "one_minimal"}}},
          "then": {"properties": {"claim": {"const": "1-minimal under hotato.reducers.v1 with the recorded observation-scope freezes"}}}
        },
        {
          "if": {"properties": {"status": {"const": "budget_exhausted"}}},
          "then": {"properties": {"claim": {"const": "failure preserved; minimality incomplete because the candidate budget ended"}}}
        }
      ]
    },
    "share_minimality": {
      "type": "object", "additionalProperties": false,
      "required": ["status", "reducer_set", "claim", "check_summary", "frozen_components"],
      "properties": {
        "status": {"enum": ["one_minimal", "budget_exhausted"]},
        "reducer_set": {"const": "hotato.reducers.v1"},
        "claim": {"type": "string", "minLength": 1},
        "check_summary": {
          "type": "object", "additionalProperties": false,
          "required": ["count", "outcomes"],
          "properties": {
            "count": {"type": "integer", "minimum": 0, "maximum": 512},
            "outcomes": {
              "type": "object", "additionalProperties": false,
              "required": ["PRESERVED", "ABSENT", "DRIFTED", "UNRESOLVED"],
              "properties": {
                "PRESERVED": {"$ref": "#/definitions/nonnegative_integer"},
                "ABSENT": {"$ref": "#/definitions/nonnegative_integer"},
                "DRIFTED": {"$ref": "#/definitions/nonnegative_integer"},
                "UNRESOLVED": {"$ref": "#/definitions/nonnegative_integer"}
              }
            }
          }
        },
        "frozen_components": {
          "type": "array", "uniqueItems": true,
          "items": {"enum": ["script", "tools", "state", "handoff", "termination"]}
        }
      },
      "allOf": [
        {
          "if": {"properties": {"status": {"const": "one_minimal"}}},
          "then": {"properties": {"claim": {"const": "1-minimal under hotato.reducers.v1 with the recorded observation-scope freezes"}}}
        },
        {
          "if": {"properties": {"status": {"const": "budget_exhausted"}}},
          "then": {"properties": {"claim": {"const": "failure preserved; minimality incomplete because the candidate budget ended"}}}
        }
      ]
    },
    "preservation": {
      "type": "object", "additionalProperties": false,
      "required": [
        "source_executions", "source_matching_failures", "final_executions",
        "final_matching_failures", "source_result_digests", "final_result_digests",
        "source_content_hashes", "final_content_hashes", "source_trace_digests",
        "final_trace_digests", "same_failure_fingerprint"
      ],
      "properties": {
        "source_executions": {"const": 2},
        "source_matching_failures": {"const": 2},
        "final_executions": {"const": 2},
        "final_matching_failures": {"const": 2},
        "source_result_digests": {
          "type": "array", "minItems": 2, "maxItems": 2,
          "items": {"$ref": "#/definitions/digest"}
        },
        "final_result_digests": {
          "type": "array", "minItems": 2, "maxItems": 2,
          "items": {"$ref": "#/definitions/digest"}
        },
        "source_content_hashes": {
          "type": "array", "minItems": 2, "maxItems": 2,
          "items": {"$ref": "#/definitions/raw_digest"}
        },
        "final_content_hashes": {
          "type": "array", "minItems": 2, "maxItems": 2,
          "items": {"$ref": "#/definitions/raw_digest"}
        },
        "source_trace_digests": {
          "type": "array", "minItems": 2, "maxItems": 2,
          "items": {"$ref": "#/definitions/digest"}
        },
        "final_trace_digests": {
          "type": "array", "minItems": 2, "maxItems": 2,
          "items": {"$ref": "#/definitions/digest"}
        },
        "same_failure_fingerprint": {"const": true}
      }
    },
    "private_privacy": {
      "type": "object", "additionalProperties": false,
      "required": ["profile", "content_included", "network_egress", "hashes_are_correlators"],
      "properties": {
        "profile": {"const": "private-runnable-v1"},
        "content_included": {"const": ["scenario", "conversation_test", "assertion_result"]},
        "network_egress": {"const": false},
        "hashes_are_correlators": {"const": true}
      }
    },
    "share_privacy": {
      "type": "object", "additionalProperties": false,
      "required": ["profile", "content_included", "omitted", "runnable", "hashes_are_correlators"],
      "properties": {
        "profile": {"const": "share-safe-v1"},
        "content_included": {"const": []},
        "omitted": {
          "const": [
            "audio", "transcript_body", "scenario_body", "assertion_body",
            "tool_payload", "state_value", "credentials", "absolute_paths",
            "provider_identifiers", "deletion_paths"
          ]
        },
        "runnable": {"const": false},
        "hashes_are_correlators": {"const": true}
      }
    },
    "evaluator_digest": {"$ref": "#/definitions/digest"},
    "version_string": {
      "type": "string",
      "pattern": "^[0-9]+(?:\\.[0-9]+){1,3}(?:[A-Za-z0-9._+-]*)?$"
    },
    "scenario_selection": {
      "type": "object", "additionalProperties": false,
      "required": ["mode", "seed", "variation_matrix_applied"],
      "properties": {
        "mode": {"const": "base-scenario"},
        "seed": {"type": "integer", "minimum": 0},
        "variation_matrix_applied": {"const": false}
      }
    },
    "private_provenance": {
      "type": "object", "additionalProperties": false,
      "required": ["hotato_version", "evaluator_digest", "seed", "scenario_selection"],
      "properties": {
        "hotato_version": {"$ref": "#/definitions/version_string"},
        "evaluator_digest": {"$ref": "#/definitions/evaluator_digest"},
        "seed": {"type": "integer", "minimum": 0},
        "scenario_selection": {"$ref": "#/definitions/scenario_selection"}
      }
    },
    "share_provenance": {
      "type": "object", "additionalProperties": false,
      "required": ["hotato_version", "evaluator_digest", "scenario_selection"],
      "properties": {
        "hotato_version": {"$ref": "#/definitions/version_string"},
        "evaluator_digest": {"$ref": "#/definitions/evaluator_digest"},
        "scenario_selection": {"$ref": "#/definitions/scenario_selection"}
      }
    }
  }
}


================================================================================
FILE: src/hotato/schema/envelope.v1.json
================================================================================

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://hotato.dev/schema/envelope.v1.json",
  "title": "hotato result envelope, schema_version 1",
  "description": "The single machine-readable shape emitted by the CLI, the one MCP tool, and the suite runner. Additive-only: new signal dimensions and sub-verdicts may appear as extra keys without a version bump, so consumers must ignore unknown fields. The stable core below is frozen for schema_version 1.",
  "type": "object",
  "required": ["tool", "schema_version", "mode", "stack", "offline", "engine", "limits", "summary", "events", "fix_map", "funnel", "exit_code"],
  "additionalProperties": true,
  "properties": {
    "tool": { "const": "hotato" },
    "schema_version": { "const": "1" },
    "mode": { "enum": ["single", "suite"] },
    "suite": { "type": "string" },
    "stack": { "type": "string" },
    "offline": { "const": true },
    "exit_code": { "enum": [0, 1] },
    "engine": {
      "type": "object",
      "required": ["name", "version", "upstream"],
      "additionalProperties": true,
      "properties": {
        "name": { "type": "string" },
        "version": { "type": "string" },
        "upstream": { "type": "string" }
      }
    },
    "limits": {
      "type": "object",
      "required": ["method", "accuracy_claim", "ceiling", "best_input", "does_not_do", "scope", "offline"],
      "additionalProperties": true,
      "properties": {
        "method": { "type": "string" },
        "accuracy_claim": { "type": "null", "description": "HONESTY INVARIANT: always null. This tool never emits an accuracy percentage." },
        "ceiling": { "type": "string" },
        "best_input": { "type": "string" },
        "does_not_do": { "type": "array", "items": { "type": "string" } },
        "scope": { "type": "string" }
      }
    },
    "summary": {
      "type": "object",
      "required": ["events", "passed", "failed", "regression"],
      "additionalProperties": true,
      "properties": {
        "events": { "type": "integer", "minimum": 0 },
        "passed": { "type": "integer", "minimum": 0 },
        "failed": { "type": "integer", "minimum": 0 },
        "regression": { "type": "boolean" },
        "not_scorable": { "type": "integer", "minimum": 1, "description": "Additive: count of events that could not be judged (malformed input, see each event's not_scorable_reason). Present ONLY when at least one event is not scorable; a run of valid recordings emits the exact pre-existing summary shape. Not-scorable events count in events but in neither passed nor failed." }
      }
    },
    "events": {
      "type": "array",
      "items": { "$ref": "#/definitions/event" }
    },
    "fix_map": {
      "type": "array",
      "items": { "$ref": "#/definitions/fix_map_entry" }
    },
    "funnel": {
      "oneOf": [
        { "type": "null" },
        {
          "type": "object",
          "required": ["reason", "pointer"],
          "additionalProperties": true,
          "properties": {
            "reason": { "type": "string" },
            "pointer": { "$ref": "#/definitions/engagement_pointer" }
          }
        }
      ]
    }
  },
  "definitions": {
    "event": {
      "type": "object",
      "required": ["event_id", "expected_yield", "verdict", "measurements"],
      "additionalProperties": true,
      "dependencies": {
        "scorable": ["not_scorable_reason"],
        "not_scorable_reason": ["scorable"]
      },
      "properties": {
        "event_id": { "type": "string" },
        "scorable": { "const": false, "description": "Additive: present ONLY on events that cannot be judged (and then always false); scorable events omit the key entirely, so every envelope for a valid recording is byte-identical to schema_version 1 before this key existed. A not-scorable event is excluded from passed/failed counting and can never carry a normal pass or fail." },
        "not_scorable_reason": { "type": "string", "description": "Plain-language reason the event cannot be judged (for example no detectable caller speech with no onset provided, or the agent silent at the caller onset under a should-yield expectation). Present exactly when scorable is present." },
        "scenario_id": { "type": ["string", "null"] },
        "title": { "type": ["string", "null"] },
        "category": { "type": ["string", "null"] },
        "expected_yield": { "type": "boolean" },
        "verdict": {
          "type": "object",
          "required": ["passed", "did_yield", "seconds_to_yield", "talk_over_sec", "reasons"],
          "additionalProperties": true,
          "properties": {
            "passed": { "type": "boolean" },
            "did_yield": { "type": "boolean" },
            "seconds_to_yield": { "type": ["number", "null"] },
            "talk_over_sec": { "type": "number" },
            "reasons": { "type": "array", "items": { "type": "string" } }
          }
        },
        "measurements": {
          "type": "object",
          "additionalProperties": true,
          "description": "Timing measurements behind the verdict. The core keys (caller_onset_sec, agent_talking_at_onset, hop_sec, notes) are frozen for schema_version 1. The boundary-sensitivity keys below are ADDITIVE (documented here, never in required): they expose where a near-threshold result sat on the frame grid and how far it was from flipping. A reader built before they existed simply ignores them.",
          "properties": {
            "caller_onset_sec": { "type": ["number", "null"] },
            "agent_talking_at_onset": { "type": "boolean" },
            "hop_sec": { "type": "number" },
            "notes": { "type": "string" },
            "onset_requested_sec": { "type": ["number", "null"], "description": "Additive: the caller_onset_sec the caller requested for this event, or null when the onset was auto-detected rather than supplied." },
            "onset_frame_index": { "type": ["integer", "null"], "description": "Additive: the integer hop index the onset was snapped to (null on a not-scorable placeholder)." },
            "onset_effective_sec": { "type": ["number", "null"], "description": "Additive: the quantized onset time actually used, equal to onset_frame_index * hop_sec (unrounded, so the equality holds exactly). Null on a not-scorable placeholder." },
            "yield_frame_index": { "type": ["integer", "null"], "description": "Additive: the hop index of the counted yield, or null when the agent did not yield (or the event was not scorable)." },
            "decision_margin_sec": { "type": ["number", "null"], "description": "Additive: signed slack in seconds from the tightest binding pass/fail threshold to the measured value (positive = inside the pass boundary, magnitude = how much slack; negative = over the line). Null when no numeric bound applies (pure yield/hold) or the event was not scorable." },
            "decision_margin_hops": { "type": ["integer", "null"], "description": "Additive: decision_margin_sec expressed in whole frame hops, round(decision_margin_sec / hop_sec). Null when decision_margin_sec is null." },
            "boundary_sensitive": { "type": "boolean", "description": "Additive: true when the result sits within one hop of flipping the verdict (abs(decision_margin_hops) <= 1). False when comfortably inside the limit, not derivable, or not scorable." }
          }
        },
        "signals": {
          "type": "object",
          "additionalProperties": true,
          "description": "Namespaced signal bus. barge_in is present whenever scoring ran; other dimensions (latency, and future overlap/resume/backchannel) are additive.",
          "properties": {
            "barge_in": {
              "type": "object",
              "required": ["did_yield", "time_to_yield_sec", "talk_over_sec"],
              "additionalProperties": true
            },
            "latency": {
              "type": "object",
              "additionalProperties": true,
              "properties": {
                "response_gap_sec": { "type": ["number", "null"] },
                "premature_start_sec": { "type": ["number", "null"] }
              }
            },
            "resume": {
              "type": "object",
              "description": "Post-yield behaviour on the agent VAD track; present only when the agent yielded. resumed: did the agent start speaking again within the window after the yield. resume_gap_sec: seconds from the yield to that fresh onset (null when it did not resume). restart_suspected: the longest post-resume run is unusually long, a timing proxy for re-answering from the top (a transcript repeat is out of scope).",
              "additionalProperties": true,
              "properties": {
                "resumed": { "type": "boolean" },
                "resume_gap_sec": { "type": ["number", "null"] },
                "restart_suspected": { "type": "boolean" }
              }
            }
          }
        },
        "fix": {
          "oneOf": [ { "type": "null" }, { "$ref": "#/definitions/fix" } ]
        },
        "audio_provenance": { "$ref": "#/definitions/audio_provenance" }
      }
    },
    "audio_provenance": {
      "type": "object",
      "description": "Additive: identity of the exact audio bytes this event was scored from (a streamed sha256 of the raw file, never a full in-memory decode) plus the sample rate and frame count read from the WAV header. Present on every event scored from a real audio file. Absent (never fabricated, never guessed) on an event with no file to hash -- for example a not-scorable / missing-audio placeholder, or an envelope from before this field existed. A reader (in particular hotato fix trial's before/after recapture guard) MUST treat absence as UNKNOWN provenance, never as proof that a fresh capture happened.",
      "required": ["schema_version", "sha256", "sides"],
      "additionalProperties": true,
      "properties": {
        "schema_version": { "type": "string" },
        "sha256": { "type": "string", "description": "The event's combined audio identity: one file's own sha256 for a single-file input (stereo/mono), or an order-stable combination of both sides' sha256 for a caller+agent dual-mono input." },
        "sides": {
          "type": "array",
          "items": {
            "type": "object",
            "required": ["role", "path", "sha256", "sample_rate", "num_samples"],
            "additionalProperties": true,
            "properties": {
              "role": { "type": "string" },
              "path": { "type": "string" },
              "sha256": { "type": "string", "description": "Streamed sha256 of this side's raw file bytes -- container identity. Changes on any byte anywhere in the file, including header-only metadata edits, a trailing-byte append, or a lossless repack/transcode that never touches a sample value." },
              "pcm_sha256": { "type": "string", "description": "OPTIONAL, additive: streamed sha256 of this side's DECODED PCM sample bytes only (the WAV data chunk, read the same way the scorer decodes it -- fixed width, little-endian, channel-interleaved, bounded by the header's own frame count). Identity of the SAMPLES, not the container: a header-only edit or a trailing-byte append leaves this unchanged even though sha256 above differs; an edit to a sample value changes both. Absent on an envelope from before this field existed -- a reader MUST treat absence as UNKNOWN, never as proof either way." },
              "sample_rate": { "type": "integer" },
              "num_samples": { "type": "integer" },
              "duration_sec": { "type": ["number", "null"] }
            }
          }
        }
      }
    },
    "fix": {
      "type": "object",
      "required": ["fix_class", "title", "detail", "knob", "pointer"],
      "additionalProperties": true,
      "properties": {
        "fix_class": { "enum": ["config", "engagement-control"] },
        "title": { "type": "string" },
        "detail": { "type": "string" },
        "knob": {
          "oneOf": [
            { "type": "null" },
            {
              "type": "object",
              "required": ["stack", "parameter", "direction", "trade_off"],
              "additionalProperties": true
            }
          ]
        },
        "pointer": {
          "oneOf": [ { "type": "null" }, { "$ref": "#/definitions/engagement_pointer" } ]
        }
      }
    },
    "fix_map_entry": {
      "type": "object",
      "required": ["event_id", "fix_class", "title", "detail", "knob", "pointer"],
      "additionalProperties": true,
      "properties": {
        "event_id": { "type": "string" },
        "scenario_id": { "type": ["string", "null"] },
        "fix_class": { "enum": ["config", "engagement-control"] },
        "title": { "type": "string" },
        "detail": { "type": "string" }
      }
    },
    "engagement_pointer": {
      "type": "object",
      "description": "VENDOR-NEUTRAL pointer for the engagement-control fix class. Names the problem class and the KIND of fix (a learned engagement-control / addressee-detection layer); names no vendor, no product, and nothing you can adopt/license/buy, and carries no numbers.",
      "required": ["layer", "what", "honest_scope"],
      "additionalProperties": true,
      "properties": {
        "layer": { "type": "string" },
        "what": { "type": "string" },
        "honest_scope": { "type": "string" }
      }
    }
  }
}
