================================================================================
FILE: README.md
================================================================================

<div align="center">

<img src=".github/assets/hotato-banner.svg" alt="hotato" width="442" style="max-width:100%;height:auto;">

<p>
<a href="https://pypi.org/project/hotato/"><img src="https://img.shields.io/pypi/v/hotato?style=flat-square&color=c23c07&label=pypi" alt="PyPI version"></a>
<a href="https://pypistats.org/packages/hotato"><img src="https://img.shields.io/pypi/dm/hotato?style=flat-square&color=c23c07&label=downloads" alt="Downloads per month"></a>
<a href="https://pypi.org/project/hotato/"><img src="https://img.shields.io/pypi/pyversions/hotato?style=flat-square&color=6f5d44" alt="Python versions"></a>
<a href="https://github.com/attenlabs/hotato/actions/workflows/tests.yml"><img src="https://github.com/attenlabs/hotato/actions/workflows/tests.yml/badge.svg?branch=main" alt="CI status"></a>
<a href="https://github.com/attenlabs/hotato/blob/main/LICENSE"><img src="https://img.shields.io/pypi/l/hotato?style=flat-square&color=6f5d44" alt="MIT license"></a></p>
<!-- Add a stars badge (shields.io github/stars/attenlabs/hotato) here once the repo reaches ~25 stars; below that it advertises the low number. -->

# hotato

**Find what broke in your agent calls. Pin it so it never ships again.**

```bash
pip install hotato
hotato vapi health              # analyze your last 100 Vapi calls
hotato autopsy ./call.wav       # or analyze one local file
```

Zero config. Works with Vapi, Retell, Bland, Synthflow, Millis, or local audio.
No judges. No cloud. No bill. MIT.

**[hotato.dev](https://hotato.dev)**

</div>

## What it finds

- **Barge-in → Say-do gaps**: Caller interrupts to cancel; agent says "canceled" but the booking tool still fires. The most expensive voice AI bug. (Timing from the audio; the tool-fire check reads your call's tool log: hotato ingests Vapi/OTel traces.)
- **Latency spikes**: 800ms → 5s unpredictability that makes users hang up.
- **Dead air**: Long silences that kill conversation flow.
- **Talk-over**: Agent speaks over the caller; never yields.

## Quickstart

### Vapi

```bash
pip install hotato
export VAPI_API_KEY=...
hotato vapi health --last 7d --output report.html
```

Open `report.html`. See your Voice Stability Score and every critical incident.

### Retell

```bash
export RETELL_API_KEY=...
hotato retell health --call-id CALL_ID
```

`--call-id` is required and repeatable: Retell has no verified
list-recent-calls endpoint, so hotato never guesses one. `hotato bland health`,
`hotato synthflow health`, and `hotato millis health` follow the Vapi shape;
those stacks export one mixed channel, so their reports carry the
measured-confidence mono observations block.

### Local audio

```bash
hotato autopsy ./call.wav
```

Writes a detailed, self-contained HTML incident report under `hotato-output/`;
open it in your browser.

## From finding bugs to preventing them

`autopsy` finds bugs. `scan` tracks trends across a folder of calls. When you
are ready, pin incidents to your CI so they never ship again: `hotato pin`
turns one incident into a portable failure check, and `hotato prove` is the CI
check that re-runs every stored piece of evidence and fails closed. Every
verdict carries its evidence across five dimensions (outcome, policy,
conversation, speech, reliability).

[Read more →](docs/CI.md)

For continuous use: run `hotato vapi health` on a schedule, and open
`hotato console --production-db DB` to inspect stored runs locally.

## Wire it into CI

The step's exit code **is** the verdict: `0` pass, `1` fail, `2` refuse.

```yaml
# .github/workflows/voice-qa.yml
on: [pull_request]
jobs:
  hotato:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: attenlabs/hotato@v1.16.0
        with:
          contracts: contracts/
          hotato-version: 1.16.0
```

Copy-paste workflow with a commit-SHA pin: [`docs/CI.md`](docs/CI.md).

## Point your agent at it

Point Claude Code, Cursor, or any coding agent at this repo: it reads
[`AGENTS.md`](AGENTS.md) and runs the loop end to end, offline, no key. The MCP
server exposes the scorer plus read/verify/propose tools over local stdio:
`uvx --from "hotato[mcp]" hotato-mcp` ([`docs/MCP.md`](docs/MCP.md)).

## Nothing leaves your machine

hotato runs offline, on the machine that invokes it. The core is stdlib-only
Python: no account, no key, no network call of its own. Your traces, prompts,
and audio stay local, and the local-judge lane is opt-in and quality-gated,
separate from the deterministic core.

## Go deeper

The whole loop, command by command: [`docs/LIFECYCLE.md`](docs/LIFECYCLE.md).
First touch to a CI gate: [`docs/GETTING-STARTED.md`](docs/GETTING-STARTED.md).
Feed it what you already have: [`docs/CONNECT.md`](docs/CONNECT.md) &#183;
[`docs/TRACE.md`](docs/TRACE.md) &#183; [`docs/SIMULATE.md`](docs/SIMULATE.md).
Next to the hosted alternatives: [`docs/COMPARE.md`](docs/COMPARE.md).

## Specifications

| Property | Value |
| :-- | :-- |
| Footprint | ~10 MiB installed, 0 runtime dependencies (stdlib-only) |
| Reproducibility | byte-for-byte, content-addressed checks |
| Exit codes | `0` pass &#183; `1` fail &#183; `2` refuse |
| Release integrity | OIDC Trusted Publishing + build-provenance attested |
| Runtime | offline, off the production data path |

<details>
<summary><b>Verify the measurement yourself</b></summary>

```bash
PYTHONPATH=src python3 -m hotato.benchmark \
  --scenarios corpus/real/scenarios --audio corpus/real/audio
```

On 13 recorded AMI Meeting Corpus clips, the median error between measured caller-onset and the human word-alignment label is **20 ms**. Provenance: [`corpus/real/README.md`](corpus/real) &#183; method: [`METHODOLOGY.md`](METHODOLOGY.md).

Timing is measurable only when the two voices arrive on separate channels; a mono or mixed export is marked **NOT SCORABLE** and refused (`hotato trust --stereo call.wav`).

</details>

## Contribute

Issues and PRs welcome: [`CONTRIBUTING.md`](CONTRIBUTING.md) &#183; [`SECURITY.md`](SECURITY.md) &#183; [`CHANGELOG`](CHANGELOG.md) &#183; [`docs/`](docs/)

## License

MIT ([`LICENSE`](LICENSE))

<div align="center"><sub>Know when to pass it on.</sub></div>

mcp-name: io.github.attenlabs/hotato


================================================================================
FILE: docs/GETTING-STARTED.md
================================================================================

# Getting started

One path, no forks: from first touch to a CI gate that guards every pull request.
Follow the five steps in order. Each command prints the exact next command, so
you can chain the whole loop from what hotato tells you.

The transcript passed and the call still failed: the agent talked over the caller,
ran through the interruption, or left a beat of dead air handing the floor back.
None of that is in the words. Hotato scores the turn timing between the two voices
of a recording, pins a caught moment as a contract, and re-runs it in CI forever.

## Install

Zero install, run it now:

```bash
uvx hotato start --demo
```

Keep it around for daily use:

```bash
pipx install hotato        # or: uv tool install hotato, or: pip install hotato
```

Scoring needs two separate channels (caller on one, agent on the other). A mono
or mixed export is marked NOT SCORABLE and refused, exit 2. Confirm a file is
scorable before scoring it with `hotato trust --stereo call.wav`.

## The five steps

### 1. See it catch a failure

```bash
hotato start --demo
```

Sweeps the two bundled demo calls, builds one failure contract, and runs one
say-do conversation check. It exits 0 because setup finished. The gate command it
points at, `hotato contract verify contracts/`, exits 1 by design.

### 2. Score your own recording

```bash
hotato investigate ./call.wav
```

```console
hotato investigate [run 1]: call.wav
  input health: eligible for scan
  verdict path: eligible (a labeled event here can carry a real yield/hold verdict)
  most likely failure (top-ranked candidate):
    [1] t=7.63s agent_stop_no_caller  trailing_silence_sec=0.37, caller_proximity_sec=0.5
  next: label it (use --expect hold instead if the agent was right to keep talking):
    hotato investigate label '.hotato/investigate-state.json#1' --expect yield
```

Hotato ranks the timing moments, marks the top one `most likely failure`, and
hands you one command to pin it. It infers no intent. Have a provider call id
instead of a WAV? `hotato investigate --stack vapi --call-id <id>` pulls it first,
then ranks the same way.

### 3. Commit the catch as a regression

Yield means the agent should have stopped for the caller. Hold means it should
have kept the floor through a backchannel or noise. The label is your decision;
hotato measures whether the timing matched it.

```bash
hotato investigate label '.hotato/investigate-state.json#1' --expect yield
```

```console
created hotato contract: call-8s-yield
  dir:      contracts/call-8s-yield.hotato
  expect:   yield
  passed:   False
  measured: did_yield=False seconds_to_yield=n/a talk_over=0.00s
next:
  hotato contract verify contracts

open the pull request that adds it to your repo's CI gate:
  hotato pr create --fixtures contracts/call-8s-yield.hotato --repo OWNER/REPO --title 'Add hotato contract call-8s-yield'
```

The contract bundle is content-addressed: the clipped audio, the frame-level
evidence, the policy, and its own manifest, committed byte-identical.

### 4. Open the pull request

```bash
hotato pr create --fixtures contracts/call-8s-yield.hotato --repo OWNER/REPO --title 'Add hotato contract call-8s-yield'
```

Stages the bundle byte-identical under `tests/hotato/contracts/` and opens the
PR. It is a dry run by default; add `--yes` to run git and gh.

### 5. Let the gate re-run the evidence

```bash
hotato contract verify contracts/
```

```console
hotato contract verify: contracts (1 contract)
  [FAIL] call-8s-yield (expect yield): did_yield=False seconds_to_yield=n/a talk_over=0.00s | integrity: intact
  0/1 contracts pass; exit_code=1
  These contracts pin known failures. Each stays red until you fix the agent and recapture the call, the same way a snapshot test stays red until you update the snapshot.
  Path to green: fix the agent, then recapture with `hotato drive <bundle>` (vapi/twilio), or the manual path in docs/RECAPTURE.md.
```

This re-measures the stored evidence deterministically. It is the CI gate: exit 0
pass, exit 1 fail.

## When the gate is red

A committed contract is a pinned bad call, so it is *meant* to stay exit 1, the
way a snapshot test stays red until you update the snapshot. The frozen audio
never changes, so the gate goes green only after you fix the agent and recapture
the call. That still-red state is a review checkpoint, not a broken test.

To get to green, fix the agent, then recapture:

```bash
hotato drive <bundle>        # re-run a fresh call against the live agent (vapi/twilio)
```

`hotato drive` originates a new call for vapi and twilio and reports a before and
after verdict. For every other stack, or by hand, follow [`RECAPTURE.md`](RECAPTURE.md).

## Reference

The five steps are the whole loop. When you need more depth, follow these one at
a time; none of them is required to complete the loop above.

- [`START.md`](START.md): the guided demo, both acts (timing and say-do), in detail.
- [`INVESTIGATE.md`](INVESTIGATE.md): capture-origin authentication and candidate ranking.
- [`CONTRACTS.md`](CONTRACTS.md): what a contract bundle holds and what `verify` proves.
- [`CI.md`](CI.md): the copy-paste GitHub Action with a commit-SHA pin and a PR results comment.
- [`RECAPTURE.md`](RECAPTURE.md): the path from a red gate to a green one.
- [`ASSERTIONS.md`](ASSERTIONS.md) and [`TRACE.md`](TRACE.md): the say-do check from traces you already log.
- [`MCP.md`](MCP.md): drive the loop from Claude Code, Cursor, or any MCP client.
- [`../AGENTS.md`](../AGENTS.md): the same loop, written for a coding agent, plus the machine contract.

Symptom-first walkthroughs, each opening with the direct answer and running on
what ships in the repo:

- [`scenarios/browser-vs-pstn.md`](scenarios/browser-vs-pstn.md): passes in the browser, fails on the phone; score the same moment clean and codec-degraded.
- [`latency-waterfall.md`](latency-waterfall.md): perceived latency worse than the dashboard; the per-hop waterfall from your traces.
- [`scenarios/load-and-recovery.md`](scenarios/load-and-recovery.md): breaks under concurrent call load; the evidence-preserving load staircase.
- [`scenarios/dtmf-verification.md`](scenarios/dtmf-verification.md): verifying DTMF from the evidence your pipeline logs.
- [`scenarios/echo-self-interruption.md`](scenarios/echo-self-interruption.md): the agent interrupts itself; TTS bleed measured and gated.
- [`scenarios/false-interruption-replay.md`](scenarios/false-interruption-replay.md): stops talking on "mhm"; pin the false yield as a CI contract.


================================================================================
FILE: docs/WHY.md
================================================================================

# Why Hotato

Four timing failures break a voice-agent call while every text-level test
still passes.

## Voice agents fail in ways your tests do not catch

Your test suite checks what the agent says. Production calls fail on *when*
it speaks:

1. **Missed interruption.** The caller says "stop, take that off"; the
   agent keeps talking. `did_yield` is false where it should be true --
   they repeat themselves, louder, then hang up.
2. **False stop.** The caller says "mhm" -- a backchannel, not a takeover
   request -- and the agent stops mid-sentence anyway. `did_yield` is true
   where it should be false.
3. **Slow yield.** The agent does stop, eventually. Every second of
   `talk_over_sec` before it does, the caller hears both voices at once.
4. **Endpointing misses.** The agent misjudges when the caller finished
   talking: dead air (`response_gap_sec`), or a start before they are done
   (`premature_start_sec`). Hotato measures how long the silence lasted,
   never what it meant.

Each lives in audio timing, invisible to a transcript diff, an LLM judge, or
a unit test on the reply -- all three score the call clean, and the caller
calls back anyway.

You label the expected behavior -- `yield` (stop for the caller) or `hold`
(keep speaking through a backchannel or noise). Hotato measures whether the
timing matched that label; intent is always yours to call.

## Rule out five look-alikes first

These produce the same symptom as a turn-taking bug, but no VAD threshold
touches them:

- **STT hallucination** -- the transcript has words the caller never said.
  Check ASR word-error-rate.
- **Client-side audio buffering** -- the caller's device queues audio
  before it reaches the agent, so "talked over me" is a transport delay,
  not a turn-taking one. Check WebRTC jitter-buffer/network latency.
- **LLM verbosity or tool-selection** -- the agent is mid-tool-call and
  doesn't re-check for a stop signal. Check response-length and tool-call
  latency.
- **Safety false-refusal** -- a moderation layer cut the agent off. Check
  your safety/moderation logs.
- **Wrong-language STT** -- the caller's language or accent isn't covered
  well. Check per-locale STT accuracy -- Hotato's own detector runs on
  audio energy alone, so it scores the same regardless of language
  (`corpus/classes/README.md`).

Match one of these and Hotato will not surface it -- a day saved. Match
none, and agent-talks-over-caller / false-stop-on-backchannel is exactly
what Hotato measures: no single threshold fixes both directions, shown on
recorded calls, not synthetic fixtures (`corpus/vapi-defaults/README.md`).

## What makes Hotato different

- **Scores recordings you already have.** A dual-channel WAV from your
  stack is the whole input -- no test harness, synthetic caller, or
  re-architecture.
- **Runs offline.** Scoring is local and deterministic; audio, transcript,
  and result stay on your machine.
- **Emits machine-readable timing.** One JSON envelope, the same shape from
  the CLI, the MCP tool, and the pytest fixture.
- **Fails CI.** Exit code 1 on a regression, 0 on pass, 2 when a recording
  isn't scorable yet -- a turn-taking regression blocks a merge like a
  failing unit test.
- **Routes every failure to a fix class.** `config` names the setting to
  change; `engagement-control` names it as a classification problem
  (telling "mhm" from "stop") so you don't retune a setting that can't win.

## What Hotato measures, and where it stops

Hotato measures energy over time on two channels -- that is the whole
method. Transcription, emotion, and intent sit outside it. A diarizer,
where used, assigns anonymous SPEAKER_00/01 labels, never who a person is.
A single-channel (mono) recording scores via the opt-in,
quality-gated `[diarize]` front-end (`hotato run --mono call.wav
--diarize`), labeled indicative below the confidence bar, with dual-channel
as the reference standard. Method and ceiling: [METHODOLOGY.md](../METHODOLOGY.md)
and the `limits` block of every result.

## Three numbers, reproducible by hand

A single blended percentage tells you nothing about which call failed or
what to change. Hotato reports three measurements per event:

- `did_yield`: did the agent stop talking after the caller started
- `seconds_to_yield`: seconds between the caller starting and the agent stopping
- `talk_over_sec`: seconds the caller and the agent spoke at the same time

Every number is reproducible from the frame dump by hand, every threshold
exposed -- that is what you debug with.

Ready to pin a failure? The loop from one bad call to a CI gate, with `hotato
fixture create`, `compare`, and `plan`: [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md).


================================================================================
FILE: METHODOLOGY.md
================================================================================

# Methodology

How every number this tool reports is computed, why it is reproducible, and
where the method stops being trustworthy. There is no accuracy percentage in
this document and none is implied anywhere in the tool. What follows describes a
measurement, not a claim about any detector's internal quality.

Hotato scores the audio *timing* of turn-taking from a call recording: whether
the agent stopped talking once the caller started (a yield), how many seconds
that took, how many seconds it kept talking while the caller was talking
(talk-over), and, on the same two tracks, the endpointing timing: how the
agent's response sits around the moment the caller finished speaking. It
measures energy over time. It does not understand speech.

## Read this first: the honest ceiling

The limits come first because they bound everything below.

- **Energy is not intent.** The detector marks a frame "active" when its
 short-time energy crosses a threshold. A cough, a slammed door, or a burst of
 line noise reads as active exactly like a word does. Nothing here recovers
 *meaning*, it recovers *when there was speech-level energy*.
- **You supply the label.** Hotato does not infer intent. You label the
 expected behavior for the event: yield means the agent should stop for the
 caller. hold means the agent should keep speaking through a
 backchannel/noise/acknowledgement. Hotato then measures whether the timing
 matched that label. "mhm" and "stop" can carry identical speech energy; the
 verdict is only ever your label checked against measured timing.
- **Sub-second single-channel boundaries sit near the resolution limit.** At the
 default 10 ms hop a boundary is located to within a frame or two, but the
 hangover that keeps an utterance whole (default 150 ms) smears every trailing
 edge by up to that much, and on one mixed channel deciding *whose* energy
 crossed the threshold is not always possible.
- **Two channels is the gold reference; mono is scorable, quality-gated.** The
 reference input is separated caller and agent tracks: one two-channel WAV or
 two aligned mono WAVs, where each channel is one party and overlap is a fact of
 the recording (both tracks active at once, by construction). A single mixed
 mono call is scorable via the opt-in `[diarize]` front-end (`hotato run --mono
 call.wav --diarize`): a diarizer separates the mix into caller/agent activity,
 which is reconstructed and scored through the same path. It is quality-gated --
 above the confidence bar the verdict is labeled `diarized-mono`; below it, the
 verdict is labeled indicative only and no SLA gate fires; a non-separable file
 is refused. Diarized mono never equals a true dual-channel recording for
 sub-second talk-over attribution, and the gate is what keeps that honest per
 file.
- **Out of scope, permanently:** no speaker identification (a diarizer assigns
 anonymous SPEAKER_00/01; it never says who a person is), no speech-to-text, no
 emotion or intent detection, and no claim about any vendor's internal accuracy.
 Word-level semantics are not measured; only the timing of the yield is.

These same limits are emitted inline in the `limits` block of every JSON result
(`core.py`) and in the MCP tool description, so a consuming agent sees them.

## The pipeline, step by step

Everything runs offline, standard-library only (NumPy only accelerates the
per-frame RMS when importable; results are identical without it). Scoring is
deterministic: the same audio and `ScoreConfig` give the same output.

### Default parameters

Every value below is read directly from `ScoreConfig` (`score.py`) and
`VADParams` (`vad.py`); the same defaults apply to both channels unless overridden.

Framing and VAD (`VADParams`, applied per channel):

| Parameter | Default | Meaning |
|--------------------|-----------|---------|
| `frame_ms` | 20.0 ms | analysis window length for one RMS frame |
| `hop_ms` | 10.0 ms | step between frames (the time resolution) |
| `noise_percentile` | 0.10 | quietest 10% of frames estimates the noise floor |
| `rel_db` | 15.0 dB | a frame is active this many dB above the floor |
| `abs_gate_db` | -60.0 dBFS| nothing below this absolute level is ever active |
| `dyn_margin_db` | 22.0 dB | on a rarely-silent channel, hold the threshold at least this far below the loudest frames |
| `hangover_sec` | 0.15 s | stay "active" this long after energy drops |

Signal windows (`ScoreConfig`):

| Parameter | Default | Meaning |
|---------------------------|---------|---------|
| `yield_hangover_sec` | 0.20 s | agent must stay quiet this long to count as yielded |
| `max_search_sec` | 3.0 s | how long after onset a yield is searched for |
| `caller_proximity_sec` | 0.5 s | a yield only counts if the caller held the floor within this window of it |
| `turn_end_silence_sec` | 0.20 s | caller must stay quiet this long for the turn to count as ended |
| `premature_tolerance_sec` | 0.05 s | agent may lead the caller's turn end by up to this before it counts as premature |
| `onset_min_run_sec` | 0.05 s | minimum sustained active run for the caller onset to count |
| `agent_onset_lookback_sec`| 0.10 s | window before onset checked for whether the agent was already talking |

Both onset-detection windows are exposed on `ScoreConfig` as well:
`onset_min_run_sec` (the `0.05 s` sustained active run a caller onset must
clear, applied by `first_active_sec`) and `agent_onset_lookback_sec` (the
`0.10 s` window before onset checked for "was the agent talking at onset?").
Neither affects the per-frame active/threshold decision the frame dump records.
Every threshold the scorer uses is a named, overridable parameter -- there are
no hidden constants in the scoring path.

### Step 1, per-frame RMS

Each channel is cut into 20 ms windows stepped by 10 ms. For each window the
linear root-mean-square amplitude is computed (`frame_rms`, `audio.py`). At
16 kHz this is a 320-sample window with a 160-sample hop, so `hop_sec` is
exactly `160 / 16000 = 0.01 s`; it is derived from the integer sample hop over
the sample rate, so it matches `hop_ms` exactly at common rates.

### Step 2, RMS to dBFS

Each linear RMS value is converted to decibels relative to full scale:
`dBFS = 20 * log10(rms)`, with a floor of `-120.0 dBFS` substituted before the
log to avoid `log(0)` (`to_dbfs`, `audio.py`). This is the per-frame `*_dbfs`
column in the frame dump.

### Step 3, per-channel energy VAD

For each channel independently (`energy_vad`, `vad.py`):

1. Sort the frame dBFS values and take the `noise_percentile` (10th percentile)
 as the **noise floor**.
2. The base **threshold** is `max(noise_floor + rel_db, abs_gate_db)`, i.e.
 15 dB above the floor, but never below the -60 dBFS absolute gate.
3. **Dynamic-margin guard:** if a channel is almost never silent (e.g. an agent
 talking the whole clip), the 10th-percentile floor lands *inside* speech and
 would push the threshold above the speech itself. So compute
 `cap = max(dBFS) - dyn_margin_db`; if `cap` is above the absolute gate, lower
 the threshold to `min(threshold, cap)`. This cannot rescue a genuinely silent
 channel, because the guard only fires when loud content exists above the gate.
 The cap is also the scorer's noise ceiling: once a channel's noise floor
 climbs to within `dyn_margin_db` of its loudest frame the verdict flips
 rather than degrades; `docs/BENCHMARK.md` ("Noise floor and the verdict
 cliff") gives the measured flip points and the opt-in `--snr-gate-db`
 scorability gate.
4. A frame is **raw-active** when its dBFS ≥ the threshold.
5. **Hangover:** after any raw-active frame, keep the channel active for
 `round(hangover_sec / hop_sec)` more frames, 15 frames (150 ms) at the
 defaults, so brief inter-word gaps do not fragment one utterance.

The resulting threshold and noise floor are constant across frames for a given
channel and are recorded verbatim in the frame dump.

### Step 4, caller onset

If a caller onset label is supplied (scenario `caller_onset_sec`, or `--onset`),
it is used directly. Otherwise the onset is the start of the caller's first
sustained active run, the first run of at least 0.05 s of active frames
(`first_active_sec`). The label is preferred because on mixed or noisy audio the
first energy is not always the caller. `onset_idx` is `round(onset / hop_sec)`,
clamped into range. No label and no detectable caller speech means there is no
caller event to score: the recording is reported not scorable (`scorable: false`
with a plain reason), never scored against frame 0, and a single-recording run
exits 2.

### Step 5, the three barge-in signals

Computed in `score_channels` (`score.py`):

- **`agent_talking_at_onset`**, was any agent frame active in the 0.10 s up to
 and including onset? If not, on a should-yield expectation there is nothing to
 yield: the event is reported not scorable (`scorable: false`, with the reason
 in `not_scorable_reason`) rather than passed or failed, and a single-recording
 run exits 2. The input is what is wrong (onset time, channel mapping, or
 expectation), and it is reported as such.
- **`did_yield`**, scanning from onset up to `onset + max_search_sec` (300
 frames), the first frame where the agent goes quiet and *stays* quiet for
 `yield_hangover_sec` (20 frames) **and** the caller held the floor within
 `caller_proximity_sec` (±50 frames) of that quiet point. The proximity
 condition is what stops an agent that merely finishes its own sentence seconds
 after an isolated backchannel from being scored as a barge-in response.
- **`time_to_yield_sec`**, `(yield_frame - onset_frame) * hop_sec`, floored at
 0; `None` if the agent never yielded within the search window.
- **`talk_over_sec`**, count of frames from onset to the yield point (or to the
 end of the search window if it never yielded) where the caller and the agent
 were *both* active, times `hop_sec`. Both activity tracks are the hangover
 smoothed tracks, so against a label placed at the raw end of speech energy
 the measured end sits late by at most `hangover_sec` plus one hop and the
 measured start sits early by at most one frame; the bias is deterministic
 and one sided, and `hangover_sec = 0` removes the hangover term
 (see docs/BENCHMARK.md, Quantization).

### Step 6, the two latency (endpointing) signals

Pure timing on the same two VAD tracks, no second model (`score.py`):

- **`caller turn end`**, the first frame at/after onset where the caller goes
 quiet and stays quiet for `turn_end_silence_sec` (20 frames). `None` if the
 caller never activates after onset or is still talking when the clip ends.
- **`agent response onset`**, skip any agent speech already in progress at
 onset (the pre-caller turn or the yield tail), then the first frame the agent
 becomes active again. `None` if the agent never starts a fresh run.
- With both defined, `lead = turn_end_frame - response_onset_frame` (positive
 when the agent starts *before* the caller finishes). If `lead` exceeds
 `premature_tolerance_sec` (5 frames), `premature_start_sec = lead * hop_sec`
 and `response_gap_sec` is null. Otherwise `premature_start_sec = 0.0` and
 `response_gap_sec = max(0, response_onset - turn_end) * hop_sec`. Both are
 null when not derivable from the tracks, never fabricated. Reported values
 are rounded to the millisecond (3 decimal places of seconds).

### Step 7, two additive, opt-in cross-checks: echo and resume

Both live entirely in hotato's own layer (`echo.py`, `resume.py`), never in
the vendored `_engine`, and both add an optional `signals` block without
changing `did_yield`, `seconds_to_yield`, or `talk_over_sec` for any existing
recording.

- **`signals.echo`** (`echo.py`), computed on every scored event. Leaked TTS
 (an agent that hears its own audio back on the caller channel) is, by
 construction, a delayed, scaled copy of the agent's own envelope. The check
 is a deterministic cross-channel coherence: at each lag from 0 up to
 `DEFAULT_MAX_LAG_SEC` (0.5 s), take the cosine similarity between the caller
 frame-RMS envelope and the agent envelope shifted by that lag; `coherence`
 is the best (highest) cosine found, `lag_sec` is the lag it occurred at, and
 `echo_suspected` is `coherence >= DEFAULT_COHERENCE_THRESHOLD` (0.7) with at
 least `_MIN_OVERLAP_FRAMES` (8) of overlap. Independent speech (real
 turn-taking, backchannels) does not correlate with the agent's own envelope
 at any lag, so it scores low. `--echo-gate` (opt-in, `hotato run`) holds an
 echo-suspected yield out of the verdict (`scorable: false`) instead of
 counting it as a clean pass; without the flag the primary verdict is
 unchanged and `signals.echo` is still reported. `hotato diagnose` and the
 single-run text output print a WARNING for every echo-suspected yield.
- **`signals.resume`** (`resume.py`), present only on events where the agent
 yielded. From the agent's own VAD track, look for a fresh onset within
 `DEFAULT_RESUME_WINDOW_SEC` (4.0 s) after the yield: `resumed` is whether one
 is found, `resume_gap_sec` is the seconds from yield to that onset (`null`
 if it did not resume). `restart_suspected` flags whether the longest
 contiguous agent run at or after the resume onset reaches
 `DEFAULT_RESTART_MIN_SEC` (2.0 s), the timing fingerprint of re-answering a
 whole paragraph from the top rather than finishing the interrupted clause.
 Whether the resumed words literally repeat the earlier ones is a transcript
 question, explicitly out of scope: `restart_suspected` is a run-length
 heuristic on timing, not a text diff.
- **`agent_stop_no_caller`** (`scan.py`, not part of the scored envelope: a
 `hotato scan` candidate) surfaces the companion timing fact: the agent went
 from active to quiet with zero caller energy anywhere in
 `caller_proximity_sec` on either side, so nothing on the caller channel
 explains the drop. It is a candidate for self-truncation, endpointing
 mis-fire, or an upstream (LLM/TTS) cutoff, not a verdict; label it with
 `hotato fixture create --expect hold` (or `yield`, if that silence was in
 fact correct) to turn it into a scored fixture.

## Why it is reproducible

- **Deterministic** given `(audio, ScoreConfig)`. No randomness, no network, no
 hidden state. The bundled fixtures are themselves rendered deterministically
 from a seed derived from `sha256(scenario_id)` (`render_examples.py`), and CI
 re-renders and diffs them to prove byte-stability.
- **Every threshold that drives the active/threshold decision is a named,
 overridable parameter** on `ScoreConfig` / `VADParams`, and the resolved
 values are echoed back in the frame dump's `config` block.
- **Every frame and every derived index is inspectable.** `--dump-frames PATH`
 writes, per frame: `t_sec`, `caller_dbfs`, `agent_dbfs`, `caller_active`,
 `agent_active`, and the constant `caller_threshold_db`,
 `caller_noise_floor_db`, `agent_threshold_db`, `agent_noise_floor_db`
 (`frame_dump`, `score.py`). Nothing the scorer decides is off the record.

### Worked example: re-deriving `did_yield` and `talk_over` by hand

The excerpt below is illustrative of the dump shape; run `--dump-frames` on your
own file for real values. Assume the defaults (`hop_sec = 0.01`), a caller onset
label at `2.40 s` (`onset_frame = 240`), and these constant thresholds from the
dump header: caller threshold `-55.0 dBFS`, agent threshold `-52.0 dBFS`.

```
 frame t_sec caller_dbfs caller_active agent_dbfs agent_active
 240 2.40 -18.4 True -21.1 True
 288 2.88 -19.0 True -20.7 True
 289 2.89 -20.2 True -48.9 True (hangover tail)
 290 2.90 -21.5 True -71.3 False
 ... (agent dBFS stays below -52 through frame 310) ...
 310 3.10 -20.8 True -69.4 False
```

Re-derive `did_yield`: agent was active at onset (frame 240), so the question is
meaningful. Scanning from 240, the agent first drops below its threshold at
frame 290 and stays below it for at least 20 consecutive frames (through 310), 
that satisfies `yield_hangover_sec`. The caller is active within ±50 frames of
290, so the proximity condition holds. Therefore `did_yield = True`,
`yield_frame = 290`, and `time_to_yield_sec = (290 - 240) * 0.01 = 0.50 s`.

Re-derive `talk_over`: count frames from 240 up to the yield frame 290 where
`caller_active AND agent_active`. In this excerpt that is frames 240 through 289
inclusive, 50 frames, so `talk_over_sec = 50 * 0.01 = 0.50 s`. Every one of
those booleans is just `dbfs >= threshold` (plus hangover), which you can
recompute yourself from the two dBFS columns and the header thresholds.

## Aggregate statistics: one definition each

The `report`, `team`, and `export` surfaces summarize many events with mean,
median, p90, and p95. All four come from one stdlib implementation
(`src/hotato/_stats.py`, `dist_summary`) so every published aggregate is
re-derivable by hand:

- **mean** is the arithmetic mean (`statistics.fmean`).
- **median** is `statistics.median` (p50).
- **p90** and **p95** are linear interpolation between closest ranks (the
 definition NumPy calls "linear"): sort the values ascending, let
 `pos = q * (n - 1)` for `q` in `{0.90, 0.95}`, then
 `p_q = v[floor(pos)] + frac * (v[floor(pos) + 1] - v[floor(pos)])`.
- **Rates are fractions**, never a percentage: a pass rate of 8 of 8 reads
 `1.00`. That keeps every aggregate in the same unit system as the
 measurements and leaves no door open to an accuracy-percentage reading.
- **Empty input returns null.** No aggregate is ever fabricated from zero
 measurements, and fewer than two runs is stated plainly rather than drawn as
 a trend.

`team` and `export` pool `response_gap_sec` (dead air before the agent
speaks, defined above) across every scored event into the same `dist_summary`
shape already used for talk-over and time-to-yield. `--max-response-gap`
turns the pooled p95 into a latency SLA gate: the run exits 1 exactly when the
pooled p95 exceeds the bound, the same pass/fail contract as
`--max-talk-over` and `--max-time-to-yield`, just pooled instead of
per-event. A plain `hotato export` (no `--max-response-gap`) still writes a
byte-identical `envelope.json` to a run with none of this stage's work
applied; the p95 and gate live only in the printed summary and the returned
manifest.

## Optional neural cross-check (non-reference)

The energy VAD above is the **reference**: every published, golden, and bundled
number comes from it, deterministically. Hotato also ships an **optional,
opt-in, explicitly non-reference** neural VAD backend so you can re-run the *same*
turn-taking timing math over a learned speech track and compare, a direct answer
to "this is just energy VAD, rebuildable in a weekend."

- **How to use it.** Install the extra and pass the flag on your *own* recording:
 `pip install 'hotato[neural]'` then `hotato run --stereo call.wav --backend neural`.
 The default is `--backend energy`. The `--suite` self-test **always** scores
 with energy (it is the reference), so `--backend neural` is ignored there.
- **What it is.** Silero VAD (MIT), run locally/offline. The engine itself keeps
 zero third-party dependencies and never imports a model; the backend is injected
 behind one shared interface, so an open-weight turn-detector (e.g. LiveKit's
 Smart-Turn) can be plugged in the same way. It returns the **identical**
 `VADResult` shape as the energy backend, `active`, `hop_sec`, `threshold_db`,
 `noise_floor_db`, aligned to the same hop grid. A neural model has no dB
 threshold, so `threshold_db` / `noise_floor_db` are **synthesized**: they are the
 energy-domain description of the *same* audio, reported for inspection only; the
 neural `active` decision is a learned speech probability, not a dB crossing.
- **What it changes, honestly.** It **tightens onset precision** on clean speech
 (a learned boundary can beat a fixed dB threshold on some audio). It does **not**
 close the energy-vs-intent gap: a cough, a laugh, a door slam, or crosstalk still
 carries speech-band energy, and *any* single-channel VAD, energy or neural, can
 mark it active. Whether a sound is a genuine bid for the turn is not decidable
 from one channel's activity alone. **No accuracy percentage is claimed for either
 backend.** Neural is a flagged cross-check, not a new source of truth.
- **No silent fallback.** If the `[neural]` extra is absent, `--backend neural`
 raises a clean, explicit error and exits non-zero, it never quietly scores with
 energy and presents it as the neural result.
- **Verification status: verified against the real model.** Executed in this
 repo with silero-vad 6.2.1 (ONNX weights bundled inside the package, inference
 through onnxruntime on CPU, fully offline; note silero-vad itself depends on
 torch, so the extra installs it). Properties that hold, measured over the
 bundled 8 and the 40-scenario gold suite:
 - **Contract holds end to end.** The real model returns the identical
   `VADResult` shape as energy, on the same hop grid, through the public scorer.
 - **Deterministic.** Two full sweeps over all 48 fixtures produced
   byte-identical JSON, for both backends; single recordings are likewise
   byte-identical run to run.
 - **Reference untouched.** With the extra installed, energy outputs are
   byte-identical to energy outputs without it, and the frozen bundled 8
   verdicts are unchanged.
 - **On the synthetic fixtures the two tracks diverge, and that divergence is a
   property of the fixtures.** The renders are shaped noise built for the
   energy reference; a speech-trained model assigns them near-zero speech
   probability (identical through the ONNX and torch paths), so at Silero's
   default threshold the neural track is empty there and the timing math
   reports no yield on any of the 48. This measures the fixtures, not any
   agent. Point the neural cross-check at real recordings.
 - **On a real-speech recording** (a dual-channel barge-in fixture assembled
   from a recorded human utterance) both backends returned the same yield
   verdict; the measured yield timing differed (energy 0.65 s including its
   0.15 s hangover, neural 0.18 s, because a speech model segments the word
   gaps that the energy hangover bridges). One recording, reported as a timing
   observation, not an accuracy claim.
 - **Sample rates.** Silero accepts 8000 Hz, 16000 Hz, and integer multiples of
   16000 Hz (silero-vad decimates those itself and returns timestamps in the
   original sample coordinates). Any other rate fails with an actionable
   resample message from the seam. The energy backend measures at any rate.
 The real-model tests run automatically when the `[neural]` extra is installed
 and skip cleanly when it is absent (`tests/test_backend.py`).

## How validity is measured, not claimed

The bundled fixtures are synthetic: band-limited, syllable-modulated noise
rendered from exact segment boundaries declared in each scenario's
`reference_render` block (`caller_segments_sec`, `agent_segments_sec`, and an
optional `continuous` flag that renders one unbroken run per segment so the
active track equals the rendered boundaries to within one hop). Because the true
boundaries are known exactly, the scorer's **own measurement error** is
computable, and that is what to report, rather than an accuracy score:

- For a timing signal (onset, yield time, response gap), report
 `|measured - true|` in **milliseconds**, per scenario. The true value comes
 straight from the segment boundaries: true onset is the caller's first segment
 start; true yield time is the agent's segment end minus onset; true gap is the
 agent's next segment start minus the caller's turn end. Expect a residual on
 the order of the hop plus the hangover smear on trailing edges, read it, do
 not average it away.
- For `did_yield`, report a **confusion matrix** against the scenario `category`
 labels (`should_yield` vs `should_not_yield`), four cells: correct yields,
 missed yields, correct holds, phantom yields. Keep the cells separate.

**Never aggregate these into a single accuracy percentage.** A missed hard
interruption and a phantom yield on a backchannel are different failures with
different fixes; one blended number hides exactly the distinction the tool
exists to surface.

State plainly what the synthetic battery is for: it proves the plumbing works
end-to-end and catches regressions (a threshold change that moves a number shows
up immediately against the frozen golden output). It is **not** production audio
, no real accents, codecs, packet loss, or room acoustics. For validity on your
system, bring **10-15 of your own labelled calls** (two-channel where you can
get it) and run them through the same `run` / `--dump-frames` path; see
`CONTRIBUTING.md` and `docs/CORPUS-GOVERNANCE.md`.

## Say-do verification methodology

Everything above describes the audio timing scorer. Say-do verification is
the other deterministic lane: it checks what the agent *said* (transcript)
against what the backend *did* (recorded evidence), and it reads recorded
artifacts, never audio energy.

- **Authority 1 is the trace.** `tool_call`, `tool_result`, `tool_error`, and
 `http_result` assertions read only the ingested `hotato.voice_trace.v1`
 spans (`hotato trace ingest`, `docs/TRACE.md`). A span is the evidence a
 tool ran or an HTTP exchange happened; an agent's own words claiming it
 happened can never satisfy these kinds. `outcome` combines span
 sub-predicates with transcript phrases into the say-do check itself: the
 agent said "your refund is on its way" *and* the `issue_refund` span is in
 the trace.
- **Deterministic, no model.** Every `assert.v1` kind is a regex, checksum,
 or span/dict lookup (`docs/ASSERTIONS.md`); every result carries
 `deterministic: true`, and `run_assertions` is byte-stable across repeated
 calls on identical input. The model-judged rubric lane is structurally
 quarantined in its own count and never blends in.
- **INCONCLUSIVE without evidence.** An assertion whose required input is
 absent (no trace, no transcript) reports `INCONCLUSIVE`, never a guess, and
 `inconclusive_policy: fail | refuse` turns that into a gate so missing
 evidence fails loudly in CI.

**Worked example: the reference-agent suite.**
`examples/reference-agent` is the runnable ground truth for this lane: a
375-run offline suite (25 scenarios, 5 caller behaviours, 3 audio
environments) where each scenario's deterministic `agent_mock` renders
`tool_call` spans (Authority 1) and a post-call state sandbox (Authority 2).
Four scenarios carry seeded agent defects (a refund claimed but never
issued, identity skipped before a lookup, an escalation never handed off, a
declined payment handled wrong), and the suite surfaces each as an outcome
or policy FAIL from trace and state evidence alone. Two seeded runs write
byte-identical conversation artifacts, pinned by
`tests/test_determinism_reference.py`.

**Where the five dimensions stand.** The dimensions (outcome, policy,
conversation, speech, reliability) are scored lanes: each Failure Record
carries all five, each with its own status and never a blended score
(`docs/CARDS.md`, `src/hotato/failure_record.py`). Timing (the conversation
and speech lanes) is the dimension family with a frozen physics benchmark:
measurement error against rendered and hand-labelled ground truth
(`docs/BENCHMARK.md`). Say-do (the outcome and policy lanes) is the
dimension family with deterministic trace evaluation, grounded by the
reference-agent suite above. Reliability aggregates repeated runs as
`pass@1` / `pass@k` / `pass^k` on its own axis.

## How to verify a score on your own recordings

1. Run `--dump-frames out.json` on the recording. Read the per-channel
 `*_threshold_db` and `*_noise_floor_db` from the header, and confirm
 `threshold = noise_floor + 15 dB` (or the dynamic-margin cap, or the -60 dBFS
 gate) matches your levels.
2. Scan the `*_dbfs` and `*_active` columns around the reported onset, yield,
 and gap indices and confirm each `active` flag is just `dbfs >= threshold`
 carried forward by the 150 ms hangover. Re-derive the signal by hand as above.
3. If the VAD is mislabeling your audio, tune and watch the numbers move
 predictably: raise `rel_db` or `abs_gate_db` if line noise is marked active;
 lower `rel_db` if quiet speech is missed; raise `hangover_sec` if one
 utterance fragments into several; adjust `yield_hangover_sec` /
 `caller_proximity_sec` to match how long *your* callers pause. Re-dump and
 confirm the thresholds and active flags moved as expected.

If you disagree with a number, the frames are on the table and every threshold
is yours to change. That is the point: a measurement you can audit, not a
verdict you have to trust.


================================================================================
FILE: docs/BAD-CALL-TO-CI.md
================================================================================

# From a bad call to a CI gate

Turn one bad call into a permanent, offline regression test. Five steps,
audio included, all local.

## Start from your own provider call

Have a call id instead of a WAV? Pull, review, label, and open the pull
request in three commands:

```bash
hotato investigate --stack vapi --call-id <id>
hotato investigate label .hotato/investigate-state.json#<n> --expect yield --reviewer you
hotato pr create --fixtures contracts/<contract-id>.hotato --repo you/your-repo --title "Add hotato contract <contract-id>"
```

`investigate` prints a ranked candidate for each timing moment with the exact
label command. `investigate label` writes a signed contract bundle to
`contracts/<contract-id>.hotato/` and prints the exact `pr create` command
for it. `pr create` accepts that bundle directly: it stages the bundle
byte-identical under `tests/hotato/contracts/` and opens the pull request
(dry run by default; `--yes` runs git and gh). CI then gates on
`hotato contract verify tests/hotato/contracts/`, which exits non-zero when
a contract regresses.

Prefer a plain scenario + audio fixture over a contract bundle? The same
labeled moment lands as one through the alternate sequence, and the
`hotato run` CI gate below applies to it:

```bash
hotato fixture promote .hotato/investigate-state.json#<n> --expect yield --out tests/hotato
hotato pr create --fixtures tests/hotato --repo you/your-repo --title "Add turn-taking regression fixtures"
```

## The label comes from you

Yield means the agent should stop for the caller. Hold means it should keep
talking through a backchannel, noise, or acknowledgement. Hotato measures
whether the timing matched your label.

`"mhm"` and `"stop"` can carry identical speech energy -- no timing
measurement tells them apart. So Hotato measures the one thing timing can
settle: did the agent do what your label says, and how fast.

The main scorer needs separated tracks: one two-channel WAV, or two aligned
mono WAVs, enough to attribute talk-over reliably.

## Step 1: turn the moment into a fixture

Say the agent talked over a caller at 42.18 seconds. Label it:

```bash
hotato fixture create --stereo bad-call.wav \
    --id refund-interruption-001 \
    --onset 42.18 --expect yield \
    --max-talk-over 0.6 --max-time-to-yield 1.0 \
    --out tests/hotato
```

Writes the labeled scenario with provenance
(`tests/hotato/scenarios/refund-interruption-001.json`) and a two-channel
audio clip (`tests/hotato/audio/refund-interruption-001.example.wav`),
scored on creation. An unjudgeable input is refused with the reason, exit
code 2.

Don't know the onset? List the candidates first:

```bash
hotato scan --stereo bad-call.wav
```

`scan` reports timing facts only -- overlap, an agent starting over caller
speech, long gaps, dead air, or suspected TTS echo. You pick the moment and
supply the label.

## Step 2: run it

```bash
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio
```

The bad take fails, by design: that's the regression you're pinning. Exit
codes are the contract: 0 all pass, 1 a regression, 2 unusable input.

## Step 3: fix your agent, then compare

Change the setting, re-capture the same scenario, let the numbers speak:

```bash
hotato compare --before bad-call.wav --after new-take.wav \
    --onset 42.18 --expect yield
```

```
hotato compare: bad-call.wav -> new-take.wav
  verdict:           FAIL -> PASS
  did_yield:         false -> true
  seconds_to_yield:  - -> 0.42s
  talk_over_sec:     2.65s -> 0.42s  improved -2.23s

result: fixed
```

Moment shifted between takes? Pass `--before-onset` and `--after-onset`
separately. `--out report.html` writes a shareable HTML report. An
unjudgeable side renders NOT SCORABLE and exits 2.

## Step 4: gate CI on it

```yaml
# .github/workflows/turn-taking.yml
name: turn-taking
on: [pull_request]
jobs:
  hotato:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install hotato
      - run: hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio --format json
```

`hotato run` exits 1 on a regression, failing the job, and exits 2 when the
battery could not tell anything: an empty `--scenarios` directory, or a
battery whose every event is not scorable (a wrong `--audio` path, audio
artifacts the job never fetched). Both read red -- a battery that scored
nothing gates exactly like `hotato prove`'s "a proof of nothing is refused".
Already running pytest? `pytest --hotato-suite --hotato-suite-scenarios
tests/hotato/scenarios --hotato-suite-audio tests/hotato/audio` adds the
same gate (see [PYTEST.md](PYTEST.md)). The richer PR check with a sticky
results comment is in [CI.md](CI.md).

## Step 5: when it fails, plan the fix

```bash
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio \
    --format json > result.json
hotato plan result.json --stack vapi --assistant-id <id>
```

`plan` is read-only and guarded: one bounded step on one setting at most,
never a single-threshold fix when the battery fails on both axes at once,
and a checklist when the evidence can't isolate the layer. It only proposes
-- `platform_mutation.performed` is always false. Details:
[FIX-PLANS.md](FIX-PLANS.md).

## Use Hotato when

- You have (or can capture) separated caller/agent audio.
- You can point at a moment and label it: yield or hold.
- You want a local, deterministic CI regression test on turn-taking timing.

## What Hotato measures

Turn-taking timing from separated audio tracks: talk-over, response gaps,
and yield/hold timing against your label. Not transcript wording, call
outcome, sentiment, compliance, or tool-call behavior. Not live, in-call
decisions either -- Hotato scores recordings after the fact. And `hotato
scan` surfaces candidates only; every verdict still needs your yield or
hold label.


================================================================================
FILE: docs/INGEST.md
================================================================================

# `hotato ingest` -- the composable passive on-ramp

Wire a webhook to `hotato ingest` once, and every completed call is
scanned for **candidate** turn-taking moments, automatically.

`ingest` surfaces candidates; you decide which matter and promote them
with `hotato fixture create` -- the yield/hold label always comes from
you.

Built by **composition**, adding only a per-stack webhook parser:

```
parse the webhook payload   ->  extract the call id / recording locator
fetch the recording         ->  hotato.capture (the SAME fetch the adapters use)
scan for candidates         ->  hotato.scan (no labels, no verdict)
write a candidate report     ->  JSON always; --out an HTML report (optional)
```

## How it runs

- **You own the trigger.** Wire it to a webhook handler, a serverless
  function, or a cron over your call log. It runs offline, self-hosted;
  the only network call is the same fetch `hotato capture` already makes.
- **You own the label.** A candidate is a timing event. Only you know
  whether a caller sound was "mhm" or "stop," so the label is yours.

## Contract

```
hotato ingest --stack {vapi|retell|twilio|livekit|pipecat} \
    (--event PAYLOAD.json | --call-id ID | --recording-sid RE...) \
    [--out report.html] [--format text|json] [--allow-mono] [--top N] [--min-gap S]
```

- **Exit 0** -- ran (candidates reported, possibly zero).
- **Exit 2** -- parse / fetch / IO error, or not-scorable input (a mono
  recording, for example, which can't attribute overlap to caller vs
  agent). Never a pass/fail.

`--format` controls stdout (`text` listing, or `json` candidates capped by
`--top`). `--out` additionally writes an HTML report with every
candidate. A webhook payload is **untrusted DATA**: `ingest` reads only
the named locator fields.

## Wire your webhook -> `hotato ingest`

Point your platform's call-completed webhook at a small handler that saves
the payload and shells out to `ingest` -- identical for every stack:

```python
# a minimal webhook handler (framework-agnostic pseudocode)
import json, subprocess, tempfile, os

def on_call_completed(request):
    payload = request.body                       # the platform's webhook payload
    with tempfile.NamedTemporaryFile(
        "w", suffix=".json", delete=False) as fh:
        fh.write(payload)
        event_path = fh.name
    # ingest is one process; run it out-of-band so the webhook returns fast.
    subprocess.Popen([
        "hotato", "ingest", "--stack", "vapi",
        "--event", event_path,
        "--out", f"candidates/{request_id}.html",
    ], env={**os.environ, "VAPI_API_KEY": os.environ["VAPI_API_KEY"]})
    return 200
```

A cron over your call log works just as well when you'd rather batch:

```bash
# nightly: scan yesterday's calls for candidates
for id in $(your-call-log --since yesterday --ids); do
  hotato ingest --stack vapi --call-id "$id" \
      --format json --out "candidates/$id.html" >> candidates/$id.json
done
```

### Per-stack recipe

Webhook field paths verified against live vendor docs (2026-07-07); an
unconfirmed field is parsed **defensively** -- missing, never fabricated.

| Stack | Webhook | Field ingest reads | Credentials for the fetch |
|-------|---------|--------------------|---------------------------|
| **vapi** | end-of-call-report | `message.call.id` (confirmed) | `VAPI_API_KEY` |
| **retell** | call webhook | top-level `event` + `call.call_id` (confirmed) | `RETELL_API_KEY` |
| **twilio** | `recordingStatusCallback` | `RecordingSid` (confirmed; form-encoded body) | `TWILIO_ACCOUNT_SID` + `TWILIO_AUTH_TOKEN` |
| **livekit** | egress webhook | `egressInfo.fileResults[].location` / `.filename` (defensive) | none -- egress lands in your storage |
| **pipecat** | your own event | `recording_path` / `recording_url` (defensive) | none -- you produced the file |

For **vapi / retell / twilio**, `ingest` extracts the identifier and
delegates the fetch to the same adapter `hotato capture` uses
([ADAPTER-STATUS.md](ADAPTER-STATUS.md)) -- the recording URL always comes
from that validated fetch, not the raw payload.

For **livekit / pipecat**, the recording lands in *your* infra, so the
event carries the locator directly: `recording_url` (downloaded) or
`recording_path` (read locally). A Pipecat event is whatever you emit:

```json
{ "recording_path": "captured.wav" }
```

Or skip the payload entirely with a direct id:

```bash
hotato ingest --stack vapi   --call-id  <id>   --out candidates.html   # + VAPI_API_KEY
hotato ingest --stack twilio --recording-sid RE... --format json       # + TWILIO_*
hotato ingest --stack pipecat --event event.json                        # local file
```

## From a candidate to a regression test

`ingest` finds the moment; you decide what should have happened and
freeze it:

```bash
# 1. ingest surfaces a candidate at t=42.18s in a call recording
# 2. you listen, decide the agent should have yielded, and promote it:
hotato fixture create --stereo call.wav --onset 42.18 \
    --expect yield --id refund-cutoff-001 --out tests/hotato
# 3. it is now a permanent test:
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio
```

See [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md) for the full loop.

## Notes on `--allow-mono`

Discovery needs one party per channel to attribute overlap. `--allow-mono`
(or `HOTATO_ALLOW_MONO=1`) lets the *fetch* pull a mono-only recording on
retell/twilio (matching `hotato capture`), but a mono mix still reports
**not-scorable** (exit 2): overlap can't split between caller and agent.
Record dual-channel for candidates.


================================================================================
FILE: docs/FIX-PLANS.md
================================================================================

# Fix plans: the guarded ladder

Hotato turns a failing turn-taking measurement into a reviewable, bounded
fix proposal: four rungs, each gated tighter than the last. The first
three read only; the fourth touches a clone.

## The ladder

| Level | Command | Does | Writes to your stack |
|---|---|---|---|
| **0** | `hotato diagnose result.json` | Per-failure diagnosis + a battery decision, with the advisory and the tradeoff stated | Never |
| **1** | `hotato inspect --stack ...` | Reads the CURRENT turn-taking config (GET or static parse) and normalizes it | Never |
| **2** | `hotato plan result.json [target]` | Combines diagnosis + inspected config into a fix-plan JSON (`hotato.fixplan.v1`) | Never |
| **3** | `hotato apply` / `hotato fix trial` / `hotato verify` | Applies a plan to a CLONE and re-scores the battery under a pinned manifest | Cloned assistant / branch only, never production |

Level 3 keeps the same guard throughout: `hotato apply` targets a CLONED
assistant (or a branch config) only, `hotato fix trial` re-scores the
battery on that clone under a pinned manifest, and `hotato verify` gates
the before/after. Production is a human call: every plan pins
`"approval": {"default": "manual", "production_apply": false}`, so it
waits on a human go-ahead. See [APPLY.md](APPLY.md),
[FIX-TRIAL.md](FIX-TRIAL.md), and [FIX-LOOP.md](FIX-LOOP.md).

## Level 0: diagnose

```
hotato run --suite barge-in --format json > result.json
hotato diagnose result.json
```

One diagnosis per failing event:

```
{"finding":          missed_real_interruption | false_stop_on_backchannel |
                     slow_yield | excess_talk_over | endpointing_miss |
                     not_scorable | threshold_funnel,
 "evidence":         the measured fields for that event,
 "likely_layer":     interruption_detection | endpointing | unknown_root_cause,
 "config_only_safe": bool,
 "notes":            plain language}
```

plus one battery-level decision. Text mode prints the Level 0 advisory,
for example: "Missed real interruption. Likely config layer. Try lowering
the stop-speaking word threshold one step. Tradeoff: may increase false
stops on short acknowledgements." Every advisory states its tradeoff.

Exit codes: 0 no failing events, 1 failing events diagnosed, 2 unusable input.

## Level 1: inspect

```
hotato inspect --stack vapi --assistant-id <id>      # + VAPI_API_KEY
hotato inspect --stack retell --agent-id <id>        # + RETELL_API_KEY
hotato inspect --stack livekit --config agent.py     # static parse
hotato inspect --stack pipecat --config bot.py       # static parse
```

Output: one normalized model (`interrupt_min_words`,
`interrupt_voice_seconds`, `resume_backoff_seconds`,
`endpointing_wait_seconds`, `backchannel_aware`) plus the raw fields and
provenance (what was fetched, when, and which docs the field names were
verified against). Absent or unreadable options are null with a note;
suspicious values (an unusually high word threshold, a long endpointing
wait) are surfaced as observations, for you to judge.

Read-only by construction: Vapi and Retell are one GET each; LiveKit and
Pipecat files are parsed with `ast`, statically, as text. Missing
credentials exit 2 cleanly.

## Level 2: plan

```
hotato plan result.json --stack vapi --assistant-id <id> --out hotato-fixplan.json
hotato plan result.json                              # stack from the envelope, else generic
```

(`--run result.json` is the equivalent flag form.) The input must be a run
envelope; frame dumps, benchmark results, and compare results are rejected
with exit 2.

The plan is `hotato.fixplan.v1` (shipped schema:
`src/hotato/schema/fixplan.v1.json`): the finding, a hypothesis, zero or
more changes, the verification gate, and the approval block. It also
carries `kind: "fix-plan"`, the measured `evidence` behind it, the stated
`risks`, `next_commands` (apply the step manually, verify with
`hotato compare`, re-run the battery), not-scorable events as
`input_issues` (input problems, kept separate from fixes), and a
`platform_mutation` block whose `performed` is always false: `hotato plan`
reads, it never writes.

Twilio rule: Twilio carries the audio; the upstream voice-agent stack
decides when the agent yields. `--stack twilio` (or a Twilio-stack
envelope) gets a checklist instead of agent-config advice: confirm the
dual-channel caller/agent assignment, then re-plan against that upstream
stack.

### Policy rules (as implemented and tested)

A change is proposed ONLY when ALL of these hold:

  (a) the failure class maps cleanly to ONE setting;
  (b) the proposed value is ONE bounded step, an unambiguous direction
      within documented bounds, anchored to the inspected value as `from`;
      without inspection, direction and bounds only, `current_unknown:
      true`;
  (c) the run's battery contains at least one passing OPPOSITE-RISK
      fixture, else the plan DOWNGRADES to insufficient_coverage ("add an
      opposite-risk fixture before tuning") and names the exact fixture
      family to add;
  (d) the diagnosis marked the failure config_only_safe.

Standing refusals:

* **threshold_funnel** -- when the battery contains BOTH a missed real
  interruption AND a false stop on a backchannel, the plan is a refusal:
  `"decision": "do_not_tune_single_threshold"` with a vendor-neutral
  engagement-control pointer (no product names, no digits). One threshold
  cannot satisfy both axes: raising it for one worsens the other -- the
  threshold treadmill teams describe from the inside, tuned endlessly with
  no perfect setting, because both failures share one knob.
* **slow_yield without a clear layer** -- TTS buffering, transport
  latency, and VAD smoothing are indistinguishable from one recording, so
  the plan proposes a diagnostic checklist (instrumentation steps) first.
  A slow yield becomes config-only-safe once the battery has a passing
  opposite-risk backchannel fixture that makes a one-step change
  verifiable.
* **not_scorable events** are input problems, tracked separately from
  agent failures and kept out of the plan.

Every plan, whatever its decision, carries the same verification gate:

```
"required_verification": ["real_interruption_fixture_must_pass",
                          "backchannel_fixture_must_not_regress",
                          "slow_yield_p95_must_not_worsen"]
```

### Worked example: a safe one-step plan

Battery: the agent missed a real interruption; the backchannel fixture
passes (the opposite-risk coverage the policy requires). Inspection read
`stopSpeakingPlan.numWords = 2` from the live assistant.

```json
{
  "schema": "hotato.fixplan.v1",
  "target": {"stack": "vapi", "inspected": true,
             "assistant_id": "asst_123", "current_unknown": false},
  "finding": "missed_real_interruption",
  "hypothesis": "The agent missed a real interruption: the caller took the floor and the agent kept talking. Inspected stopSpeakingPlan.numWords is 2; one bounded step decrease is the smallest verifiable change on this axis.",
  "config_only_safe": true,
  "decision": "propose_one_step",
  "changes": [
    {"field": "stopSpeakingPlan.numWords",
     "from": 2, "to": 1, "direction": "decrease", "bounds": [0, 10],
     "reason": "The caller took the floor and the agent never stopped within the search window. A one-step sensitivity increase is a config-layer candidate; the tradeoff is more false stops on short acknowledgements. One step decrease from the inspected value. Bounds basis: documented range 0-10 (docs.vapi.ai, 2026-07-06).",
     "risk": "more false stops on short acknowledgements; the backchannel fixture must not regress"}
  ],
  "required_verification": ["real_interruption_fixture_must_pass",
                            "backchannel_fixture_must_not_regress",
                            "slow_yield_p95_must_not_worsen"],
  "approval": {"default": "manual", "production_apply": false}
}
```

One field, one step, both endpoints of the move visible, the risk named,
the verification gate attached: every value in the plan traces back to
the inspected config or a documented bound.

### Worked example: the refusal

Battery: the packaged demo (`hotato demo --format json`), which misses a
real interruption AND yields to a bare backchannel.

```json
{
  "schema": "hotato.fixplan.v1",
  "target": {"stack": "generic", "inspected": false},
  "finding": "threshold_funnel",
  "hypothesis": "The battery missed a genuine interruption and also stopped for a backchannel. One sensitivity threshold cannot satisfy both: raising it to hold through backchannels drops real interruptions, lowering it to catch interruptions yields to backchannels. The failure class is discrimination, not calibration.",
  "config_only_safe": false,
  "decision": "do_not_tune_single_threshold",
  "changes": [],
  "recommended_fix": {
    "class": "engagement-control",
    "examples": [
      "enable adaptive interruption handling where available",
      "use a backchannel-aware interruption classifier",
      "add addressee/turn-intent discrimination before stopping TTS"
    ]
  },
  "required_verification": ["real_interruption_fixture_must_pass",
                            "backchannel_fixture_must_not_regress",
                            "slow_yield_p95_must_not_worsen"],
  "approval": {"default": "manual", "production_apply": false}
}
```

The refusal is the product working as designed: the strongest thing a
tuning tool can say about a funnel is that tuning will not fix it.

## Provenance

Vendor field names and documented ranges were verified against
docs.vapi.ai (assistant startSpeakingPlan/stopSpeakingPlan),
docs.retellai.com (get-agent), docs.livekit.io (turn handling options),
and docs.pipecat.ai (user turn strategies) on 2026-07-06; each result
records its own `field_basis`. Where a platform documents no hard range,
the plan says so and uses a conservative working range, with a note to
verify against the installed version.


================================================================================
FILE: docs/FIX-LOOP.md
================================================================================

# The closed loop: find -> fix -> prove it's fixed

`hotato diagnose` / `inspect` / `plan` (see [FIX-PLANS.md](FIX-PLANS.md))
end at a **proposal**. The closed loop carries it the rest of the way, with
the two irreversible decisions kept in human hands:

1. **`hotato patch`** turns a plan's abstract `{field, from, to}` into a
   **literal, paste-ready artifact** for your platform. It produces the
   artifact; applying it is your call.
2. You apply it, in your own stack, and re-capture the failing moments.
3. **`hotato verify`** scores the old and new runs against each other: a
   **battery-scale before/after proof** -- *N of M fixtures that used to
   fail now pass, K of L hold fixtures still pass.*
4. **`hotato loop`** orchestrates the whole thing and **remembers where it
   left off**, so you can walk away and come back.

Every step produces a proposal, a label, or a reviewable artifact -- you
decide what ships to your platform.

## `hotato patch <fixplan.json>`: the paste-ready change

Reads a fix plan (schema `hotato.fixplan.v1`) and renders it per platform:

- **Vapi / Retell** (config behind a REST API): a JSON **merge-patch body**
  plus a ready **`curl`** against the platform's config-update endpoint,
  using the exact field names the plan carries.

  ```
  hotato plan result.json --stack vapi --assistant-id <id> --out fixplan.json
  hotato patch fixplan.json
  ```
  ```
  curl -X PATCH https://api.vapi.ai/assistant/<assistant-id> \
    -H "Authorization: Bearer $VAPI_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"stopSpeakingPlan": {"numWords": 2}}'
  ```

- **LiveKit / Pipecat** (config lives in your agent source): patch emits
  the exact **source edit** -- the constructor kwarg and literal value to
  set, e.g. `InterruptionOptions(min_words=1)`.

- **generic / unknown stack**: patch names the knob **family** and asks
  for a concrete stack; it emits no literal body it cannot stand behind.

### The both-axes case: no single knob fixes it

`hotato patch` handles config-fixable classes directly. When the plan's
decision is `do_not_tune_single_threshold` -- the **both-axes** case,
where the battery misses a real interruption AND false-stops on a
backchannel at once -- no single config value fixes both. patch prints a
vendor-neutral, numbers-free **engagement-control pointer** instead: the
problem class (telling a real bid for the floor apart from a backchannel)
and the KIND of fix it needs, no product named, no digits. It fires ONLY
here; an ambiguous slow yield or a coverage gap gets a "no patch, here's
why" explanation instead. Every other non-propose decision (diagnostic
checklist, insufficient coverage, already at a documented bound, no
change) likewise emits no patch, the reason pointing back at the plan.

**Guardrail:** patch runs entirely offline and pins `applies_change` to
false. `--format json` emits the full artifact; `--out PATH` also writes
it.

## `hotato verify --before <old> --after <new>`: the proof

After you apply the change and re-capture the same fixtures, verify scores
the old and new run envelopes against each other. Each side is a single
`hotato run` envelope JSON (a whole battery) or a directory of them;
fixtures pair by `event_id`, then `scenario_id`.

```
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio \
    --format json > before.json          # the failing take
# ... apply the patch, re-capture the fixtures ...
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio-new \
    --format json > after.json
hotato verify --before before.json --after after.json
```

It reuses the `hotato compare` taxonomy per fixture (`fixed`, `regressed`,
`improved`, `worse`, `unchanged`, `still_pass`, `not_scorable`) and
aggregate's pooled-distribution definitions for the before/after talk-over
and time-to-yield shift. Two axes matter:

- **regression axis** -- how many previously-failing fixtures now pass.
- **hold axis** -- how many hold-labeled guard fixtures still pass (they
  must not regress).

**Guardrail:**

- verify reports **coincidence**: the improvement *coincides with* your
  change. Hotato measures timing, not a controlled experiment, so
  coincidence is the strongest claim the evidence supports.
- it **refuses** the battery-scale claim when too few fixtures failed to
  characterize (`--min-n`, default 3): per-fixture facts still print, but
  the headline proof is withheld, and said so.
- an unjudgeable side reports `not_scorable`; a fixture on only one side is
  unpaired, kept out of the rollup.

By default verify measures and exits 0; `--fail-on-regression` exits 1 if
any fixture regressed or got worse.

### `--policy hotato.verify.yaml`: the anti-bandaid gate

`--policy` turns the measured rollup into a PASS/FAIL gate you can drop
into CI. A policy declares two things, and **both** must hold for verify
to pass:

```yaml
target:
  improve:
    talk_over_sec_p95: -0.5   # pooled talk-over p95 must drop by >= 0.5s
    failed_count: decrease    # fewer fixtures may fail than before
guardrails:
  max_new_false_yields: 0     # no hold guard that passed before may newly yield
  max_not_scorable: 0         # every paired fixture must be judgeable
  require_hold_fixture: true  # the battery must contain a hold fixture...
  require_yield_fixture: true # ...and a yield fixture
```

```
hotato verify --before before.json --after after.json --policy hotato.verify.yaml
```

- **`target.improve`** is the success criteria: a signed number is a
  required delta (`after - before` -- `-0.5` means "must improve by at
  least 0.5"); a keyword (`decrease`, `increase`, `no_worse`, `no_better`,
  `unchanged`) states direction only. Metrics: `talk_over_sec_p95`,
  `seconds_to_yield_p95`, `failed_count`, `false_yield_count`.
- **`guardrails`** are hard fail conditions: `max_new_false_yields` and
  `max_not_scorable` cap what a naive threshold bandaid would silently
  trade away; `require_hold_fixture` / `require_yield_fixture` make sure
  the battery tests the opposite axis before it can be certified.

verify exits `1` unless **every guardrail holds and every target is met**,
so a fix passes only when it improves one axis without regressing the
other. A patch that cuts talk-over by making the agent yield to everything
meets the talk-over target but trips `max_new_false_yields` on the hold
fixtures -- the whole check fails. The `--out verify.html` proof shows the
guardrails and targets (a `Policy check: PASSED`/`FAILED` headline, an
ok/violated, met/unmet table), same as text and JSON.

The policy is parsed with the standard library only -- Hotato's core
carries no third-party runtime dependency -- over the small subset the
shipped `examples/verify-policy/hotato.verify.yaml` uses. An unknown key,
a wrong-typed value, an empty policy, a tab indent, or a list is a clean
exit-2 usage error.

## `hotato loop [FOLDER]`: one command, with memory

`hotato loop` drives the parts Hotato can drive and remembers where it
left off, in a small local state file (`.hotato/loop-state.json` by
default):

- **First run over a folder of calls**: runs discovery (`analyze` ->
  `scan` -> rank) and records the candidate moments. Stage
  `awaiting_label`:

  > you have N candidate moment(s) awaiting your label.

- **You label the ones that matter** with `hotato fixture create` -- the
  yield/hold intent always comes from a human.

- **Next run, with those fixtures present** (`--fixtures DIR`): runs them,
  diagnoses the battery (including the threshold-funnel check), and plans
  a guarded fix. Stage `awaiting_verify`:

  > a fix plan is ready; apply it with hotato patch, then prove it with
  > hotato verify.

```
hotato loop ./recordings                          # run 1: discover
hotato fixture create --stereo rec.wav --onset 12.4 \
    --expect yield --id refund-001 --out tests/hotato
hotato loop ./recordings --fixtures tests/hotato   # run 2: plan
```

**What stays yours:** every yield/hold intent, and applying the fix (loop
produces a plan and points at `hotato patch`; applying and verifying stay
your steps). loop tracks state across runs, leaving your platform
untouched -- you keep the two decisions that matter: which moment is
real, and whether to fix it.

## Exit codes

| Command | Exit | Meaning |
|---|---|---|
| `hotato patch` | `0` | a patch (config artifact or the engagement-control pointer) was produced |
| `hotato patch` | `2` | input is not a fix plan, or is unreadable |
| `hotato verify` | `0` | the rollup was produced (a low-n claim is refused but still exits 0) |
| `hotato verify` | `1` | a gate you opted into failed: `--fail-on-regression` and a fixture regressed or got worse, or `--policy` and a guardrail was violated or a `target.improve` criterion wasn't met |
| `hotato verify` | `2` | usage error, unreadable input, an invalid `--policy` file, or no fixtures pair |
| `hotato loop` | `0` | advanced or re-reported state |
| `hotato loop` | `2` | no folder on the first run, an unreadable state file, or a path that isn't a folder |


================================================================================
FILE: docs/APPLY.md
================================================================================

# hotato apply: the guarded, clone-only staged apply

`hotato apply` turns a `hotato patch` artifact into a staging assistant
you test before touching production. It's the only Hotato command that
mutates external platform state, and the most conservative one for it:
every flag combination leaves your source assistant untouched. By
default it PRINTS the clone it would create, an offline dry run; only
`--yes` with credentials creates the new one.

```
hotato patch fixplan.json --format json --out patch.json
hotato apply patch.json --clone --name staging-refund-fix --battery tests/hotato
```

## Five rules, enforced in code

1. **Clone-only.** apply ships one path: staging-clone. Any call without
   `--clone` errors cleanly: `production apply is not supported; use
   --clone to apply to a fresh staging assistant`. The only write is a
   `POST` creating a NEW assistant, never `PUT`/`PATCH` on the source.

2. **Refusal-first.** If the patch is the both-axes threshold funnel (the
   plan decided `do_not_tune_single_threshold`), apply REFUSES before
   doing anything and prints the exact recommendation:

   ```
   No config patch will be applied
   Reason: both missed real interruption and false stop on backchannel, one threshold cannot safely fix both
   Recommended: enable or add engagement-control / backchannel-aware turn detection
   ```

   The refusal is a FEATURE: a distinct, documented exit code (`3`) so a
   script can tell "refused by design" from a usage error. A single
   sensitivity threshold trades catching an interruption against holding
   through a backchannel; fixing it takes a discrimination fix, and every
   clone's patch reflects that.

3. **Opposite-risk required.** `--battery` must carry BOTH a yield
   fixture (an interruption the agent must stop for) AND a hold fixture
   (a backchannel it must keep the floor through), so apply can see the
   opposite risk before applying. Point `--battery` at your fixtures
   directory (with a `scenarios/` folder, as `hotato fixture promote`
   writes) or a folder of run-envelope / scenario JSONs.

4. **Gated side effect.** The default is a dry run: it prints the clone
   and patch, offline. Only `--yes` with credentials reaches the
   platform. The create call is the only networked step: read the source
   config (`GET`), apply the patch to a copy, create a NEW assistant
   (`POST`).

5. **Name required.** The staging clone must be named explicitly
   (`--name`); apply always uses the name you give it.

## What gets cloned

The REST-config stacks clone an assistant/agent through their own API,
read-only against the source and creating a NEW one:

| Stack | Read (source) | Create (new) | Name field |
| --- | --- | --- | --- |
| vapi | `GET https://api.vapi.ai/assistant/{id}` | `POST https://api.vapi.ai/assistant` | `name` |
| retell | `GET https://api.retellai.com/get-agent/{id}` | `POST https://api.retellai.com/create-agent` | `agent_name` |

The clone config is your source config with the patch deep-merged on top,
the new name, and server ids stripped: a fresh object, never an
overwrite of the source.

LiveKit and Pipecat keep turn-taking config in your agent source, so
`hotato patch` emits the exact source edit and apply points straight at
it. Twilio carries the audio but runs no turn-taking agent, so apply
points at the upstream stack instead.

## After the clone: prove it

The clone lets you prove the fix before touching production. Re-capture
the same battery through the SOURCE (`before/`) and the CLONE (`after/`),
then verify:

```
hotato verify --before before/ --after after/ --policy hotato.verify.yaml
```

`hotato verify` reports coincidence, not causation. Its policy is the
anti-bandaid gate: a fix passes only when every guardrail holds
(including `max_new_false_yields` on the hold fixtures) and every target
is met -- a patch that just makes the agent yield to everything gets
caught. See `docs/FIX-LOOP.md`.

Or run the whole thing (this gate, verify, an optional contracts
re-verify, and explain's attribution) as one fail-closed before/after
report: `hotato fix trial patch.json --name staging-refund-fix --before
before/ --after after/`. See [`docs/FIX-TRIAL.md`](FIX-TRIAL.md).

## Exit codes

- `0` the staging clone was rendered (dry run by default; with `--yes`
  and credentials the NEW staging assistant was created and patched, the
  source untouched).
- `2` usage error: no `--clone`, no `--name`, no opposite-risk battery (a
  yield and a hold fixture), a stack with no assistant to clone, a patch
  with no config change, or unreadable input.
- `3` principled refusal: the plan is the both-axes threshold funnel, so
  apply refuses a single-threshold patch by design -- the refusal is the
  feature.


================================================================================
FILE: docs/FIX-TRIAL.md
================================================================================

# fix trial: one before/after proof, composed, fail-closed

`hotato fix trial` is the fix ladder's last rung: it composes already-shipped,
already-guarded primitives into ONE before/after report that says whether a
change holds. No new scoring engine, no new networked path:

* **`hotato apply`'s offline gate** (`build_apply`, `clone=True`):
  refusal-first on the both-axes threshold funnel, requires an
  opposite-risk battery, clone-only. fix trial never calls
  `apply.create_clone` or `apply._http_json` -- no clone, no network call --
  so it carries apply's own clone-only, production-unmutatable guarantee by
  construction.
* **`hotato verify`'s battery-scale rollup**: scores the BEFORE run (the
  original failure evidence) against the AFTER run (re-captured through the
  clone you made with `hotato apply --clone --yes`) -- every paired
  fixture, not just the target failure. The "neighbouring cases" check.
* **`hotato contract verify`**, when `--contracts DIR` is given: another
  neighbouring-cases check, against labelled moments outside the battery.
* **`hotato explain`**, folded in as the report's attribution section: root
  cause of the original failure, reused exactly.

```
hotato patch fixplan.json --format json --out patch.json
hotato apply patch.json --clone --name staging-refund-fix --battery tests/hotato
# ... re-capture the battery through the source (before/) and the clone (after/) ...
hotato fix trial patch.json --name staging-refund-fix \
    --before before/ --after after/ --battery tests/hotato \
    --policy hotato.verify.yaml --out fix-trial.json --html fix-trial.html
```

## The verdict is fail-closed, never a soft pass

| Verdict | Exit | Fires when |
|---|---|---|
| `improved` | `0` | verify's claim is supported (>= `--min-n` previously-failing fixtures), at least one now passes, nothing regressed anywhere (including the hold/opposite-risk axis), no contract regressed, `--policy` passed if given, every before target/hold has an after counterpart, and every guarded fixture carries verifiable before/after audio identity (the provenance guard, below) |
| `regressed` | `1` | any fixture regressed, a contract regressed, or the policy failed |
| `inconclusive` | `1` | too few previously-failing fixtures to characterize, nothing that used to fail now passes, or a guarded fixture's audio identity is present but unverifiable (malformed/missing provenance, or well-formed but not recomputable because the audio was missing at trial time) |
| `refused` | `3` | the patch is the both-axes threshold funnel (apply's refusal-first gate fires before any before/after evidence is read); or the after set drops a required before fixture; or a guarded fixture's provenance doesn't match the audio on disk; or a guarded fixture's before/after audio is the SAME conversation (identical decoded PCM -- a re-score, not a recapture) |

**`inconclusive` is fail-closed, not a pass.** A low-n or zero-improvement
battery exits the same non-zero code as a regression, so CI never treats
"we could not tell" as green. A hold fixture that flips from passing to
failing is `compare.classify_pair`'s `"regressed"` result whichever axis
it's on -- there is no separate, weaker gate for the opposite-risk axis.

## Refusal is a correct output, not an error

If the patch is `do_not_tune_single_threshold` (the battery missed a real
interruption AND false-stopped on a backchannel), fix trial refuses before
reading `--before` / `--after` / `--contracts` at all, prints the canon
recommendation, and exits `3` -- the same code `hotato apply` uses for the
same refusal, so a script that already branches on apply's refusal code
recognizes fix trial's too.

```
No config patch will be applied
Reason: both missed real interruption and false stop on backchannel, one threshold cannot safely fix both
Recommended: enable or add engagement-control / backchannel-aware turn detection
```

## Fresh-capture provenance guard: a re-score is never a fix

`hotato apply`'s clone-only gate and `hotato verify`'s battery-scale rollup
answer "did the numbers move." Neither asks whether the AFTER evidence was
re-captured, or is the SAME recording the BEFORE run scored, just re-scored
under a looser threshold. That gap is exploitable: run the same fixture
twice with different `--max-time-to-yield` / `--max-talk-over` bounds and
you get a convincing "improved" verdict with no code, config, or model
change behind it.

Every run envelope records an `audio_provenance` block per event: a
streamed sha256 of the raw file bytes and of the decoded PCM samples (plus
sample rate and frame count), computed at capture time by `hotato run` /
`hotato capture`. `fix trial` does not trust the string -- it VERIFIES the
identity for every GUARDED fixture: the fail->pass targets AND the
still-passing holds (a frozen hold is a re-score too). The guard
recomputes what it can from disk and states exactly what it verified:

| What the guard finds | Why | Verdict effect |
|---|---|---|
| Well-formed, freshly distinct decoded-PCM identity, recomputed from disk and matching | a verified fresh recapture | proceeds -- eligible for `improved` |
| Identical decoded PCM before vs. after | the after run re-scored the SAME conversation (a header-only edit or trailing-byte append can't hide it -- the check is on samples, not container bytes) | `refused` (exit `3`) |
| Recorded digest does NOT match the audio on disk | provenance was hand-edited, or the audio was swapped after capture | `refused` (exit `3`) |
| A required before fixture (target or hold) missing from the after set | a cherry-picked, incomplete comparison | `refused` (exit `3`) |
| Malformed block (non-hex digest, absurd sample rate/frame count, or a top-level digest inconsistent with the per-side digests) | an unvalidated assertion, not a distinct recording | `inconclusive` (exit `1`) |
| Provenance block missing on either side | an older or hand-built envelope; identity is UNKNOWN | `inconclusive` (exit `1`) |
| Well-formed identity hotato could NOT recompute (audio not present) | asserted, not proven | `inconclusive` (exit `1`) |

A provenance-guard refusal is NOT the apply-gate refusal: it fires AFTER
`verify` / `contract verify` / `explain` already ran, so -- unlike the
both-axes refusal, which reads no evidence at all -- the full report
(verify's proof, the contract rollup, the provenance identities, the
attribution) still renders below the refusal banner. Every refusal path
exits the SAME code `3`; `refusal_kind` in the JSON output
(`"threshold_funnel"`, `"incomplete_after"`, `"recompute_mismatch"`,
`"same_audio_recapture"`) tells them apart for a script that wants to.

```
No fix will be certified from re-scored audio
Reason: 1 fixture(s) this verdict rests on (f1) have identical before/after decoded PCM: the after run re-scored the SAME conversation the before run scored, just against a different threshold or scorer config
Recommended: recapture the fixture(s) through the applied clone (hotato apply --clone --yes) and re-run hotato fix trial against the new after evidence
```

Every rendered report (text, `--format json`, `--html`) shows the short
digest and verified status for every guarded fixture, before and after --
a reader never has to take "fresh capture" on faith. The effective
`--min-n` is echoed in every surface too, so a lowered floor is always
visible. The report's own conclusion states plainly what a passing check
proves: that the fresh take passed the same human-labeled contract, not
that the change caused it (hotato reports coincidence, never causation,
throughout).

The text and HTML renders of this section also print, verbatim, wherever it
appears (an `improved` verdict, or a `refused`/`inconclusive` one the guard
itself downgraded): *"Provenance caution: this proves the specific fresh
capture scored above, at the revision it was captured from. It does not
certify a later deploy or every future call, and it does not re-run itself;
recapture again after the next change."* See
[`docs/RECAPTURE.md`](RECAPTURE.md#claim-language-what-each-kind-of-evidence-lets-you-accurately-say)
for the fuller claim-language table this line is drawn from.

## What this does not stop

This is an offline tool: a user who controls every input can lie to
themselves. Recomputing identity from the audio (above) makes the specific
forgeries an external red-team demonstrated against a prior build --
hand-written envelopes, a flipped header byte, a re-scored recording, a
cherry-picked after set -- impossible or loud. None of the following is a
bug the guard missed; each sits outside what an offline recompute over
supplied files can ever establish:

* **Fabricated inputs are still yours to fabricate.** Hand fix trial a
  fresh recording of a call that never happened, or one that does not
  match the bug you are claiming to fix, and the guard verifies the audio
  identity and still reaches `improved`. It checks that the bytes it
  scored are what they claim to be, never that the stimulus itself is
  real.
* **A contract's `MANIFEST.sha256.json` is integrity, not authenticity.**
  It proves the archive agrees with itself after packing, not who approved
  the policy inside it. Loosen a `.hotato` bundle's policy (raise
  `max_talk_over_sec`, say) before `contract pack`, and `contract verify`
  on the repacked bundle still passes -- it re-checks the archive against
  itself, not against an external record of what the policy was supposed
  to be. Only a trusted signature over the manifest closes this; none is
  implemented today.
* **A resample, re-encode, or gain change of the SAME call still changes
  the decoded PCM.** The freshness check above is exactly "is the decoded
  PCM different." A deliberately transcoded copy of the identical
  recording (resampled, gain-adjusted, round-tripped through a lossy
  codec) decodes to different samples, so it reads as a distinct capture
  -- it is not; it is the same call in a different container. A known,
  undetected residual of a PCM-identity check, not a claim the guard
  breaks.
* **Signatures are not implemented.** Nothing here is cryptographically
  signed; a sha256 digest is a checksum, not an attestation of who
  produced it.

A green fix trial does not prove the audio was freshly captured for the
scenario claimed, that the same policy and labels were used throughout,
that any omitted fixture was safe to omit, that the named revision or
clone existed, that the patch was applied to it, or that the deployed
agent improved. The verdict states exactly what it recomputed and
verified; everything else sits outside what an offline tool can promise.

## Flags

- **`PATCH_JSON`** -- a `hotato patch` artifact.
- **`--name NAME`** -- name of the staging clone this trial is proving
  (required for a non-refused patch; the same `--name` `hotato apply`
  takes -- fix trial evaluates the SAME clone-only gate, it never creates a
  clone itself).
- **`--before RUN.json|DIR`** -- the OLD run envelope(s): the original
  failure evidence. Also the default opposite-risk `--battery`, and the
  attribution source, when those are omitted.
- **`--after RUN.json|DIR`** -- the NEW run envelope(s), re-captured
  through the staging clone.
- **`--battery DIR`** -- the opposite-risk battery apply's gate checks
  (BOTH a yield and a hold fixture); defaults to `--before`.
- **`--contracts DIR`** -- also re-verify a directory of hotato contracts;
  any contract regression fails the trial.
- **`--policy hotato.verify.yaml`** -- gate verify's rollup (see
  [`docs/FIX-LOOP.md`](FIX-LOOP.md)); a violation fails the trial.
- **`--min-n N`** -- minimum previously-failing fixtures needed to support
  the claim (default 3).
- **`--out PATH`** -- also write the full proof JSON.
- **`--html PATH`** -- also write a self-contained before/after HTML
  report.

## Output

Every surface shows the apply receipt beside the verdict: fix trial calls
`apply.build_apply`, never `apply.create_clone`, so `apply_dry_run` is
`True` and `apply_created` / `apply_applies_change` are `False` on every
run, including an `improved` one -- a green verdict never means "and the
change was applied." Text prints `apply: dry_run=True created=False
applies_change=False` plus a plain-English line; JSON carries the same
fields (`apply_dry_run` / `apply_created` / `apply_applies_change` /
`apply_receipt_note`) at the top level next to `verdict`; HTML renders them
as header pills. Proving the change reached the clone or agent is `hotato
apply --clone --yes`'s job, recorded in its own receipt -- fix trial only
proves what the before/after evidence shows once that's already happened.

* **text** (default): the apply receipt, the verdict, verify's own
  rendered proof, the contract verify rollup when `--contracts` was given,
  and the attribution section (one `hotato explain` render per file under
  `--before`). When the trial's own verdict is not `improved` (a
  provenance, completeness, contract, or policy issue downgraded it),
  verify's nested `CLAIM` line is tagged `CLAIM (SUPERSEDED BY {VERDICT})`
  with a one-line restatement, whenever that sub-claim would otherwise
  read "supported" -- a fix-trial verdict of `regressed` / `refused` /
  `inconclusive` can still contain a verify claim that, read alone, looks
  like a pass; the parent verdict controls.
* **`--format json`**: the full machine shape, schema `hotato.fix_trial.v1`.
  Every sub-result is the nested result the underlying command already
  produces (`apply`, `verify`, `contract_verify`, `attribution`), not a
  re-derived summary. `apply_dry_run` / `apply_created` /
  `apply_applies_change` / `apply_receipt_note` sit at the top level next
  to `verdict`, not only inside the nested `apply` object.
* **`--html PATH`**: a self-contained report, reusing the same house style
  the other HTML reports use: the apply-receipt pills and note in the
  header, the verdict chip, the verify proof table (with the same
  superseded-claim label as text, when it applies), the contract-verify
  rollup, and the attribution cards.

## Exit codes

- **0** -- `improved`.
- **1** -- fail-closed: `regressed` or `inconclusive`.
- **2** -- usage error: the same gates `hotato apply` enforces (no
  `--name`, no opposite-risk battery, a stack with no clone target, a
  patch with no concrete change) or `hotato verify` / `contract verify`
  already enforce (no fixtures pair, an invalid `--policy`, a `--contracts`
  dir with no contracts, unreadable input).
- **3** -- principled refusal: the both-axes threshold funnel. Shared with
  `hotato apply`'s refusal code.

## What this is not

fix trial does not create, patch, or delete anything on your platform; it
never calls `apply.create_clone`. It does not prove authorization,
identity, compliance, or policy safety. It reports coincidence, never
causation -- every underlying `verify` and `contract verify` number keeps
that rule. See [`FIX-LOOP.md`](FIX-LOOP.md) for the manual steps this
composes, and [`APPLY.md`](APPLY.md) / [`EXPLAIN.md`](EXPLAIN.md) for the
primitives themselves.


================================================================================
FILE: docs/AUTOPSY.md
================================================================================

# `hotato autopsy <recording>`: one call in, the incidents out

Drop one call recording in with zero config and get the incident list:
barge-in, talk-over, dead air, latency spikes, each with a timestamp and
the measured magnitude, plus one self-contained HTML report.

```bash
hotato autopsy call.wav
```

```
hotato autopsy: call.wav  (12.0s, 2 channels, stereo)
  [CRITICAL] BARGE-IN       t=2.99s  overlap=1.96s  agent did not go silent within 3.0s
      the caller took the floor and the agent kept talking over them for 1.96 s without going quiet within 3.0 s
  [WARNING]  DEAD AIR       t=6.65s  trailing silence=1.94s  no caller energy within 0.50s
      the agent went quiet for 1.94 s with no caller energy nearby
  2 incidents: 1 critical, 1 warning
  report: hotato-output/autopsy-apx-cc33f46fad58.html
  pin: apx-cc33f46fad58  (incidents apx-cc33f46fad58#1..#2)
```

WAV reads natively; mp3/m4a convert through `ffmpeg` when it is on PATH
(when it is not, the one-line message names the install command).
Everything runs offline; no audio leaves the machine. `hotato autopsy`
with no recording prints the quick start on the bundled rendered example.

## Quick start from your platform (Vapi, Retell, Bland, Synthflow, Millis)

You never touch a WAV: one command pulls your recent calls straight from
the platform's own API, checks every recording, and writes the health
report led by the Voice Stability Score.

```bash
export VAPI_API_KEY=YOUR_KEY          # or: hotato connect vapi
hotato vapi health
```

```
hotato vapi health: pulled 38 of 38 listed calls (0 skipped, last 7d) -> hotato-output/vapi-calls
hotato scan: vapi-calls  (38 recordings: 38 analyzed, 0 refused)
  Voice Stability Score: 74/100  (38 dual-channel calls; policy 9412d82c1328)
  health: 28 of 38 dual-channel calls had no critical incidents (74%)
  ...
```

The entries share one implementation and the exact same flags:

```bash
hotato vapi health                    # VAPI_API_KEY / hotato connect vapi
hotato retell health --call-id ID     # RETELL_API_KEY / hotato connect retell
hotato bland health                   # BLAND_API_KEY / hotato connect bland
hotato synthflow health               # SYNTHFLOW_API_KEY / hotato connect synthflow
hotato millis health                  # MILLIS_API_KEY / hotato connect millis
```

| Flag | Meaning |
| :-- | :-- |
| `--last WINDOW` | how far back to pull (e.g. `7d`, `12h`, `2w`; default `7d`) |
| `--limit N` | maximum calls to pull (default 100) |
| `--dir PATH` | download directory (default `hotato-output/<stack>-calls`) |
| `--output PATH` | also write the HTML health report to PATH (the content-addressed report under `hotato-output/` is written either way) |
| `--call-id ID` | check this call id (repeatable; skips the list step). Required for Retell, which has no verified list-recent-calls endpoint -- hotato never guesses one |
| `--api-key KEY` | vendor API key (else the `hotato connect` store, else the stack's env var) |
| `--format text\|json` | output format (default text) |

Credentials resolve exactly as `hotato pull` does -- an explicit flag,
then the `hotato connect` store, then the stack's env var (`VAPI_API_KEY`
/ `RETELL_API_KEY` / `BLAND_API_KEY` / `SYNTHFLOW_API_KEY` /
`MILLIS_API_KEY`) -- and a missing key is one actionable line. Vapi and
Retell fetch the separated two-channel recording; Bland, Synthflow, and
Millis export one mixed channel, so each of their calls runs the
measured-confidence mono path below (silence timing measured from the
mixed channel; talk-over attribution comes from a two-channel recording
-- the scope line states this once per run). Mono calls report into the
best-effort mono observations block with their own counts and never
enter the Voice Stability denominator, so the mono stacks' reports carry
observations without a stability score. The analysis runs on this
machine; recordings download straight from the platform and go nowhere
else.

A window with no calls, a pull in which every recording failed to fetch,
or a pulled set with zero analyzable calls refuses with the reason (exit
2) -- no score is reported over zero calls.

## Stereo: the deterministic path

A two-channel recording (caller on one channel, agent on the other) runs
the existing whole-call scanner ([`hotato scan`](../src/hotato/scan.py))
unchanged: the same walk, the same measured numbers, byte-for-byte. The
same file produces byte-identical CLI text and a byte-identical report on
every run -- the report is even named by content (see the autopsy id
below), so the path is stable too.

The scanner's candidate kinds map to the incident vocabulary:

| Incident | From | CRITICAL when |
| :-- | :-- | :-- |
| `BARGE-IN` | the caller became active while the agent was talking (`overlap_while_agent_talking`) | the agent kept talking over the caller past the 1.0 s prompt-yield ceiling, or never went quiet in the search window |
| `TALK-OVER` | the agent started a fresh utterance over the caller (`agent_start_during_caller`) | the overlap exceeds the same 1.0 s ceiling |
| `DEAD AIR` | a response gap of 5 s or more (`long_response_gap`), or the agent stopping with no caller energy nearby (`agent_stop_no_caller`) | the gap is 5 s or more |
| `LATENCY SPIKE` | a response gap between 2 s and 5 s (`long_response_gap`) | never (a warning) |
| `ECHO SUSPECTED` | the caller channel tracks the agent's own audio (`echo_correlated_activity`) | never (a caveat) |

Each incident block carries the severity, the timestamp, the measured
magnitudes (`candidate_detail`), and the scanner's plain-English sentence
(`candidate_plain_english`) -- the same measured numbers, restated once.

## Mono: best-effort, confidence-scored

A one-channel (mixed) recording is analyzed best-effort with the same
energy VAD. One mixed channel measures silence timing -- dead air and
latency gaps -- and every mono finding carries a **measured confidence**
with its derivation printed beside it: how far the gap's mean energy sits
below the speech-activity threshold the VAD measured for this recording
(a 20 dB margin or more scores 1.00). Talk-over and barge-in attribution
comes from a two-channel recording, where the caller and the agent are
physically separated; that functional scope is stated once, on one line,
in the output. A mono gap says everything stopped, not who stopped --
nothing is guessed and no confidence is invented.

The stricter commands keep their bar: `run`, `scan`, `trust`, and the
contract path still refuse mono as NOT SCORABLE. Autopsy is discovery;
the CI gate stays deterministic and dual-channel.

An unreadable input -- a text file, a truncated header, a non-audio blob
-- is refused with the reason (exit 2), never scored.

## The report

One self-contained HTML file under `./hotato-output/`, in the
[`hotato report`](REPORTS.md) house style: the per-channel energy
waveform with one labeled marker per incident, then a card per incident
with the same measured numbers the CLI printed. Zero external requests
and zero scripts; the page renders the same bytes for the same recording
on every run.

## The autopsy id

`apx-` + the first 12 hex chars of the sha256 of the input file's bytes:
content-derived, so the same recording gets the same id on any machine,
whatever the file is named (an mp3 hashes its own bytes, so the id is
independent of the local ffmpeg build). Incidents are addressed as
`<autopsy-id>#<rank>` -- the `pin:` line prints both -- and the report is
`hotato-output/autopsy-<id>.html`.

## The persisted envelope

Every autopsy also writes `hotato-output/autopsy-<id>.json` next to the
HTML: the machine-readable result envelope -- the source path, the mode,
and the incidents with their onset, kind, and (on the stereo path) the
underlying scan candidate kind. Like the report it is content-addressed
and deterministic: the same recording writes the same bytes on every run.
The envelope carries the measured facts only; est. cost figures live on
the rendered surfaces (they exist only under `--cost-config`), never in
the stored envelope. This is the offline store `hotato pin` resolves an
`apx-...#N` ref from without re-running the analysis.

## `hotato scan <directory>`: the folder health report

Point `hotato scan` at a folder and it runs the autopsy engine over every
recording in it -- stereo through the deterministic scanner, mono
best-effort, exactly the rules above; an unreadable file is listed as
refused with the reason, never skipped silently:

```bash
hotato scan ./calls
```

```
hotato scan: calls  (5 recordings: 4 analyzed, 1 refused)
  Voice Stability Score: 25/100  (4 dual-channel calls; policy 9412d82c1328)
  SMALL SAMPLE: 4 dual-channel calls, under the 20-call bar
  health: 1 of 4 dual-channel calls had no critical incidents (25%)
  ...
  evidence coverage (measured from this run):
    dual-channel timing: 4 calls -- deterministic two-channel timing walk
    refused: 1 file -- unreadable as call audio; every file listed with its reason, never scored
  ...
  worst calls (critical count, then worst measured magnitude):
     1. late-followup.wav  apx-6836885bd877  1 critical, 0 warning  worst 5.14s gap
  ...
  report:   hotato-output/scan-scn-46a5af300d6b.html
  envelope: hotato-output/scan-scn-46a5af300d6b.json
```

The HEALTH headline is a measured share -- **dual-channel calls** with
zero critical incidents over dual-channel calls analyzed. The **Voice
Stability Score** is that same share, times 100: `round(share x 100)`,
nothing else (the machine field is `critical_free_call_rate`). The share
line prints directly beneath the score as its formula, the eligible
sample size and the analysis-policy sha print beside the score, a
`SMALL SAMPLE` label renders under 20 dual-channel calls, and the HTML
report carries a one-line "How this is calculated" note pointing at the
share line. A mono call **never enters the denominator**: mono-analyzed
calls report into the *Best-effort mono observations* block with their
own counts (measured silence timing from one mixed channel; talk-over
and barge-in attribution comes from a two-channel recording). With zero
dual-channel calls no score renders and the report states why. There is
deliberately **no blended quality score anywhere**: one blended number
hides exactly the distinction the tool exists to draw (see
[METHODOLOGY.md](../METHODOLOGY.md)), so the branded number restates the
measured share -- no weights, no other arithmetic. The share sits beside
the **evidence coverage** block (per-lane measured counts from what the
run actually had -- dual-channel timing, mono best-effort, refused with
reasons; a lane whose evidence was absent from the run never renders as
assessed), a per-category breakdown (counts plus the worst measured
magnitude in each category), and the worst-calls ranking, each call
linking to its own per-call autopsy report, generated alongside with its
envelope -- so `hotato pin` works straight from a folder scan.

The scan is deterministic end to end: the same directory with the same
flags produces byte-identical CLI text and a byte-identical HTML report,
and the outputs are content-addressed (`scn-` + 12 hex over the sorted
file-content manifest plus the analysis flags). The summary envelope
(`scan-<id>.json`) stores the aggregate with a `recorded_at` provenance
stamp written once, when that content is first seen; a re-run of
unchanged content resolves to the same file and leaves it untouched.

**Trend.** When prior summary envelopes for the same directory sit in the
output dir, the report renders a run-over-run strip from them -- each
prior run's share and critical count under its stored provenance
timestamp, with the current run beside them. The page stays
byte-identical given the same directory and the same prior-run store.

**Recurrence states.** An incident kind present in the current run that
also appears in stored prior runs of the same directory prints a
recurrence line, in the CLI text and in the report, each line carrying a
measured state:

```
RECURRING: DEAD AIR in 7 of 38 calls this run (23 in the stored window). Also present in 3 prior run(s): 2026-07-08T09:00:00Z, 2026-07-15T09:00:00Z, 2026-07-22T09:00:00Z.
```

The states, all derived from stored facts: `observed` (1-2 calls carry
the kind across this run plus the stored prior runs), `RECURRING` (3+),
`RECURRING, LOW SAMPLE` (3+ but the eligible dual-channel sample is
under 20 calls), and `ELEVATED` (this run and the most recent comparable
prior run -- same analysis policy, same evidence lanes, 20+ eligible
dual-channel calls in both -- have Wilson 95% intervals on the kind's
per-call rate that do not overlap, this run higher). Every count and
date is a measured aggregate or a stored envelope's `recorded_at`
provenance -- never extrapolation -- so the same directory and the same
prior-run store always print the same lines. The platform health
commands run the same aggregate over their download directory, so
re-running `hotato vapi health` week over week builds the store the
recurrence lines read from.

`--cost-config` renders est. cost totals across the analyzed calls, from
your own per-incident figures, exactly as above. The single-recording
mode is unchanged: `hotato scan --stereo call.wav` lists one recording's
candidate turn-taking moments, byte-for-byte as before.

## `hotato pin <autopsy-ref>`: incident to contract

`hotato pin` turns one autopsy incident into a portable `.hotato` failure
contract through the existing contract machinery
(`hotato contract create` on the recording at the incident's onset -- no
separate minting logic):

```bash
hotato pin apx-cc33f46fad58        # the call's top critical incident
hotato pin apx-cc33f46fad58#1      # one specific incident
```

Resolution is offline, from the persisted envelope under
`./hotato-output/` (or `--from DIR`). Before delegating, pin re-hashes
the source recording: the CURRENT bytes must still hash to the pinned
autopsy id, so a file that changed on disk refuses rather than binding a
different call.

The incident kind maps to the contract's expect decision: BARGE-IN and
TALK-OVER -- the floor-holding events the yield/hold contract vocabulary
expresses -- default to `--expect yield` (the caller held the floor), and
`--expect hold` records the human's call that the agent was right to keep
talking. A mono-derived incident refuses (contracts require the
two-channel deterministic path); DEAD AIR and LATENCY SPIKE are
silence-timing measurements with no caller onset to pin, and ECHO
SUSPECTED is a caveat, so those refuse with the reason too. Every refusal
-- malformed ref, unknown id, rank out of range, missing or changed
source, mono -- exits 2 and leaves no artifact.

Success prints the bundle dir and the CI step:

```bash
hotato prove --contracts contracts
```

## est. cost, only from your figures

Cost lines render only when you supply your own per-incident figures;
hotato ships no default dollar amount:

```bash
hotato autopsy call.wav --cost-config costs.json
```

```json
{"currency": "USD",
 "per_incident": {"dead-air": 3.0, "barge-in": 2.0,
                  "talk-over": 2.0, "latency-spike": 1.0}}
```

Each priced incident gets an `est. cost` line naming the kind the figure
came from, and the summary totals them. With no config, no figure
appears anywhere.

## The bundled examples

Three deterministic rendered example calls live in `examples/autopsy/`
(seeded renderer, seed = `sha256(id)`, byte-identical on any machine) --
rendered demonstrations of the failure patterns, one each:

```bash
hotato autopsy examples/autopsy/audio/autopsy-01-barge-in-say-do.example.wav
hotato autopsy examples/autopsy/audio/autopsy-02-latency-dead-air.example.wav
hotato autopsy examples/autopsy/audio/autopsy-03-talk-over.example.wav
```

## From an incident to a regression gate

An incident worth keeping fixed graduates in one step:

```bash
hotato pin apx-cc33f46fad58#1                # incident -> portable contract
hotato prove --contracts contracts           # the CI gate
```

The labeled-review path does the same through `hotato investigate`:

```bash
hotato investigate call.wav                  # ranked candidates + the label command
hotato investigate label .hotato/investigate-state.json#1 --expect yield
hotato contract verify contracts/ --junit hotato.xml   # the CI gate
```

## Scope and method

Deterministic energy measurement over time: per-frame RMS, a transparent
activity threshold, and the timing walk between the tracks. Timing and
floor-holding, not intent or transcription -- the scanner cannot know
whether a caller sound was "mhm" or "stop", and no accuracy percentage
appears anywhere. See [METHODOLOGY.md](../METHODOLOGY.md).


================================================================================
FILE: docs/ANALYZE.md
================================================================================

# `hotato analyze <folder>`: drop a folder, hear the bug

Point it at a folder of dual-channel call recordings and it finds the
worst turn-taking moments across all of them, ranks them, and lets you
hear each one.

```bash
hotato analyze ./recordings
```

Writes one self-contained, offline HTML dashboard (`hotato-analyze.html`
by default) and opens it. A bare folder as the first argument routes here
too: `hotato ./recordings`.

## What it does

For every `.wav` under the folder (walked recursively, sorted), `analyze`:

1. runs the same whole-call scanner as [`hotato scan`](../src/hotato/scan.py):
   walks caller/agent activity across the call, label-free, and emits
   candidate timing moments (`overlap_while_agent_talking`,
   `agent_start_during_caller`, `long_response_gap`,
   `agent_stop_no_caller`, `echo_correlated_activity`), each with a
   timestamp and a measured number;
2. aggregates the candidates across every call and ranks them by salience
   (overlap seconds / gap seconds / echo coherence), so the worst moments
   float to the top.

Then it emits three things.

### 1. A ranked dashboard

One card per top moment, in the [`hotato report`](REPORTS.md) house
style: the call file, timestamp, candidate kind, measured number, and a
to-scale caller/agent timeline (activity spans, the talk-over band, the
onset marker, and a yield marker where the agent went silent).

### 2. The hear-the-bug player

For the top `--audio-top` moments (default 8), audio around the moment
embeds inline as base64 WAV, so the page stays self-contained. Press play
and a **playhead** sweeps the timeline in lockstep: you hear the
talk-over or dead-air land exactly where the chart marks it. It tracks
playback under `prefers-reduced-motion` too.

Only the top moments carry audio; the rest show the timeline. That, plus
`--pre`/`--post` and a total audio budget, keeps the page a manageable
size.

### 3. JSON for agents

```bash
hotato analyze ./recordings --format json
```

Prints the ranked candidates and their metadata (source file, timestamp,
kind, salience, measured durations, audio/timeline window) to stdout,
capped by `--top`. `--out FILE` also writes the full ranked JSON.

## Framing

Each result is a **measured candidate timing moment**: a timestamp and a
number, not a verdict on intent. Energy sounds the same whether a caller
said "mhm" or "stop," so you decide and label the moments that matter:

```bash
hotato fixture create --stereo <call>.wav --onset <t> \
    --expect yield|hold --id found-moment-001 --out tests/hotato
```

Non-dual-channel or unreadable files land in a "Skipped files" section
with their reason (a mono mix can't attribute talk-over), and the run
continues.

## Flags

| flag | default | meaning |
| --- | --- | --- |
| `FOLDER` | (required) | directory of dual-channel WAVs, walked recursively |
| `--top` | 25 | ranked moments shown in the dashboard / stdout JSON (0 = all) |
| `--audio-top` | 8 | top moments that get the embedded hear-the-bug player |
| `--pre` | 2.0 | seconds of audio/timeline kept before each moment |
| `--post` | 4.0 | seconds of audio/timeline kept after each moment |
| `--min-gap` | 2.0 | minimum response gap (seconds) to surface as a candidate |
| `--caller-channel` / `--agent-channel` | 0 / 1 | channel assignment |
| `--format` | `html` | `html` (dashboard) or `json` (ranked candidates) |
| `--out` | `hotato-analyze.html` | where to write the dashboard, or the JSON |
| `--no-open` | off | do not launch a browser for the dashboard |

## Exit codes

- **0**: ran -- candidate moments listed across the folder, possibly zero.
- **2**: usage error (the path is not a folder) or an IO error reading it.

Everything, audio included, stays on your machine.


================================================================================
FILE: docs/TRUST.md
================================================================================

# `hotato trust --stereo call.wav`: is this recording even scorable?

The input-health check: inspect one recording and learn whether the audio is
good enough to score, before you scan or run it. `trust` catches a bad
export -- mono file, silent channel, swapped channel map, hot capture -- up
front, before it becomes a confident-looking but meaningless verdict
downstream.

```bash
hotato trust --stereo your-call.wav
```

```text
hotato trust: your-call.wav
  recording: 32.0s, 44100 Hz, 2 channels
  caller (ch0): 3.32s speech, first at 3.27s, peak -2.1 dBFS
  agent  (ch1): 23.30s speech, first at 0.45s, peak -0.9 dBFS
  leading silence: 0.45s
  crosstalk: coherence 0.057 (low) at 0.39s lag
  scorability: separated tracks yes, caller activity yes, agent activity yes
  => eligible for scan
```

Exit `0` means eligible for scan; exit `2` means NOT SCORABLE (or a usage
error, or an unreadable file). `trust` composes straight into a shell gate:

```bash
hotato trust --stereo call.wav && hotato scan --stereo call.wav
```

## What it checks

By convention the caller is channel 0 and the agent channel 1 (override with
`--caller-channel` / `--agent-channel`). `trust` reports INPUT health only:

| Signal | What it means |
|---|---|
| per-channel activity | how much speech each channel carries, and when each first speaks |
| possible channel swap | a heuristic flag -- if the channel mapped as caller holds the floor far longer than the one mapped as agent (the reverse of the usual pattern, since an assistant answers in paragraphs), the channels may be reversed |
| sample rate and duration | the basic recording facts |
| clipping | per-channel peak level (dBFS) and the fraction of samples at full scale, so a too-hot capture is visible |
| leading silence | dead air before the first speech on either channel |
| crosstalk risk | cross-channel echo coherence -- is the caller channel carrying a delayed copy of the agent's own audio (echo bleed, missing echo cancellation)? |
| cross-channel leakage (`crosstalk_risk.leakage_db`) | how many dB below the source a consistent, delayed COPY of one channel shows up on the other. Whole-clip coherence (above) is one best-lag cosine over the entire envelope, diluted by unrelated activity elsewhere in the call, so bleed loud enough to corrupt a timing verdict can sit under that bar and go unflagged. Leakage measures differently: the per-frame level ratio of the copy, constant across every frame the source speaks, so it catches that regime. At or above `-40 dB` (the level at which red-teaming flipped a verdict with symmetric bleed) the copy counts as the other party's activity: flagged, and the recommendation drops off `eligible for scan` |
| low signal level | when even the loudest channel peaks below `-30 dBFS`, turn timing can be under-measured downstream -- a warning, never a not-scorable condition |
| scorability | the three things a score needs -- separated tracks, enough caller activity, enough agent activity |
| recommendation | `eligible for scan`; `scan with caution` (scorable, but a loud cross-channel leak may corrupt the scan's timing); or `NOT SCORABLE`, with the specific reason and the next step |

## Scope: one question, and it stops there

`trust` answers one question -- is this audio good enough to score? -- and
stops there: no `yield`/`hold`, no `pass`/`fail`, no `did_yield`, no
talk-over number, no intent label, no turn-taking verdict. Eligible for scan
means the input is clean, not that the agent behaved well; finding agent
bugs is [`hotato scan`](../src/hotato/scan.py)'s and `hotato run`'s job.

## Not scorable

Three input defects make a recording unscorable. Each is reported with
`scorable: false`, a plain reason, and the next step -- and exits `2`:

- **mono**: a single channel can't separate caller from agent. Next step:
  export a dual-channel recording (the gold reference), or score it via the
  opt-in diarization front-end ("Mono via `--diarize`" below);
- **identical channels**: a mono recording duplicated into two channels --
  two channels, but not separated. Same fix as mono;
- **a silent required channel**: for example `caller channel has no
  detected speech`. Next step: verify channel mapping or export dual-channel
  again.

Clipping, leading silence, crosstalk risk, cross-channel leakage, low signal
level, and a possible channel swap are **warnings**: they surface as
signal, not as a scorability change -- except a loud leak, which alone also
downgrades the recommendation to `scan with caution` (above). `trust` adds
signal and discloses limits; the not-scorable boundary stays fixed.

## Mono via `--diarize`: is this mono file confidently separable?

A mono file is not scorable by default (above). The opt-in `[diarize]`
front-end changes the question: `hotato trust --stereo call.wav --diarize`
reports whether the mono is confidently SEPARABLE into caller/agent -- still
no turn-taking verdict. It runs the selected diarizer (`--diarizer
pyannote|sortformer|pyannoteai`, default local `pyannote`), then reports a
`scorability.separation` sub-block and a confidence **tier**:

- **high**: confidently separable -- score it with `hotato run --mono
  call.wav --diarize` for a diarized-mono verdict. Exit `0`.
- **low**: separable but only indicative -- voices close, overlap elevated,
  or the mapping balanced. `hotato run --mono ... --diarize` still scores
  it, but the verdict is stamped `indicative_only` and no SLA gate fires.
  Exit `0`.
- **refuse**: not confidently separable (not two clean parties, a
  near-silent speaker, extreme overlap, voices too similar). Not scorable,
  exit `2`; next step: record dual-channel.

The six signals behind the tier -- speaker count, per-speaker activity, mean
segmentation posterior, embedding cluster-separation margin, overlap ratio,
segment churn -- are in the `separation.signals` block. A missing extra,
token, or model raises a clean error (exit 2), never a raw-mono guess.
Dual-channel stays the gold reference; a diarized-mono verdict stays
distinct from it. Full front-end: `docs/DIARIZE.md`.

## JSON for agents

`--format json` emits one machine-parseable report. Branch on `scorable`; on
a defect, read `not_scorable_reason` and `next_step`. `exit_code` mirrors
the process exit.

```bash
hotato trust --stereo call.wav --format json
```

```json
{
  "tool": "hotato",
  "kind": "input-health",
  "schema_version": "1",
  "source": "call.wav",
  "recording": {
    "sample_rate": 44100,
    "duration_sec": 32.0,
    "channels": 2,
    "clipping": {
      "caller": {
        "peak": 0.7906, "peak_dbfs": -2.1,
        "clipped_fraction": 0.0, "clipped": false
      },
      "agent": {
        "peak": 0.9016, "peak_dbfs": -0.9,
        "clipped_fraction": 0.0, "clipped": false
      }
    },
    "leading_silence_sec": 0.45
  },
  "channels": {
    "caller": {
      "channel": 0, "active_sec": 3.32, "first_speech_sec": 3.27,
      "has_speech": true, "enough_activity": true
    },
    "agent": {
      "channel": 1, "active_sec": 23.3, "first_speech_sec": 0.45,
      "has_speech": true, "enough_activity": true
    },
    "possible_swap": false,
    "swap_reason": null
  },
  "crosstalk_risk": {
    "coherence": 0.057, "lag_sec": 0.39, "suspected": false,
    "leakage_db": null, "leakage_direction": null
  },
  "scorability": {
    "separated_tracks": true, "enough_caller_activity": true,
    "enough_agent_activity": true
  },
  "warnings": [],
  "scorable": true,
  "recommendation": "eligible for scan",
  "not_scorable_reason": null,
  "next_step": null,
  "exit_code": 0
}
```

Everything runs offline, on your machine, reusing hotato's existing
primitives -- the hardened WAV reader, reference framing, energy VAD, and
cross-channel echo coherence. It reports input-health signals only, never an
accuracy percentage.


================================================================================
FILE: docs/CONNECT.md
================================================================================

# Connect, pull, sweep: score every call across your stack

`hotato sweep` connects once and surfaces every turn-taking problem across
your calls, in three steps you can also run alone:

1. **`hotato connect <stack>`** -- store a stack's credentials once, locally.
2. **`hotato pull`** -- bulk-fetch your recent recordings into a folder.
3. **`hotato sweep`** -- pull, then run zero-config `analyze` and write one
   offline dashboard of the ranked turn-taking moments.

Everything scores offline; the only network call is the direct recording
download from vendor to machine. Your audio and keys stay between you and
the vendor.

## 1. Connect once

```
hotato connect vapi --api-key <key>
```

Runs a live auth check (lists one recent call), then writes the
credentials to `~/.hotato/connections.json` (mode `0600`, dir `0700`). The
key never prints and reaches only the vendor's API. Credentials also fall
back to the environment:

```
VAPI_API_KEY=<key> hotato connect vapi
```

Per-stack credentials:

| Stack | Flags (or env vars) |
| --- | --- |
| vapi | `--api-key` / `VAPI_API_KEY` |
| retell | `--api-key` / `RETELL_API_KEY` |
| twilio | `--account-sid` `--auth-token` / `TWILIO_ACCOUNT_SID` `TWILIO_AUTH_TOKEN` |
| bland | `--api-key` / `BLAND_API_KEY` |
| elevenlabs | `--api-key` / `ELEVENLABS_API_KEY` |
| synthflow | `--api-key` / `SYNTHFLOW_API_KEY` (+ `--model-id` / `SYNTHFLOW_MODEL_ID` to list) |
| millis | `--api-key` / `MILLIS_API_KEY` (+ `--base-url` for the EU region) |
| cartesia | `--api-key` / `CARTESIA_API_KEY` (+ `--agent-id` / `CARTESIA_AGENT_ID` to list) |

Retell has no list endpoint, so `connect retell` stores the key and
validates it on first pull. `--no-verify` skips the live check for any
stack. LiveKit and Pipecat capture in your own infra: connect with `hotato
setup --stack livekit|pipecat` instead.

Once connected, `--stack` and credential flags are optional for `pull` and
`sweep` when exactly one stack is connected.

## 2. Pull recent recordings

```
hotato pull --stack vapi --since 7d --limit 50
hotato pull                                   # the only connected stack
```

Lists recent recordings via the vendor's verified endpoint, then downloads
each into `hotato-pull-<stack>/` (override `--out DIR`). `--since` accepts
`7d`, `12h`, `30m`, `2w`. A recording that fails to fetch is a clean skip
with its reason; the pull continues. A pull in which **every** listed
recording failed to fetch exits 2 -- an outage, not a completed pull -- so
a cron or CI wrapper reading only the exit code sees it red.

**Retell has no verified list endpoint.** Pull it from explicit ids
instead:

```
hotato pull --stack retell --call-id c1 --call-id c2
```

For Twilio, `--call-id` values are Recording SIDs (`RE...`).

**Mono / mixed stacks** (bland, elevenlabs, synthflow, millis, cartesia)
produce one combined recording with no per-party channel; attributing
talk-over needs `--allow-mono`, and results stay indicative only:

```
hotato pull --stack bland --allow-mono --limit 20
```

## 3. Sweep = pull + analyze

```
hotato sweep --stack vapi --since 7d
hotato sweep                                  # the only connected stack
```

Pulls (as above) into `hotato-sweep-<stack>/` (override `--dir`), runs the
same zero-config `analyze`, and writes one self-contained, offline HTML
dashboard (`hotato-sweep-<stack>.html`, override `--out`): the ranked
candidate moments across every call, with a hear-the-bug player on the top
ones. `--format json` emits the candidates plus a pull summary instead.

Dual-channel stacks get separated scoring. Mono/mixed stacks need
`--allow-mono`; without per-party separation their calls land in the
dashboard's Skipped section.

Candidates are MEASURED timing moments you review and label with `hotato
fixture create`: a timestamp and a number, not a verdict on intent.

## What is and isn't pullable

See [`docs/ADAPTER-STATUS.md`](ADAPTER-STATUS.md) for the full map: which
stacks auto-pull dual-channel, which are mono-only, which capture in your
own infra, and which endpoint each uses.

## After the first catch

You have seen a catch on a recorded call. The second move is driving one
against your live agent on demand, fed into this same pull -> score
pipeline: [`docs/DRIVE-A-CALL.md`](DRIVE-A-CALL.md).


================================================================================
FILE: docs/ADAPTER-STATUS.md
================================================================================

# Adapter status

Which stack Hotato pulls recordings from, which endpoint it calls, and how
it separates caller/agent channels for each one -- every entry checked
verbatim against the vendor's live documentation on the date shown.

Terms:

- **auto-pull**: Hotato fetches the recording itself with your API key
  (`hotato capture` / `hotato pull` / `hotato sweep`).
- **capture-in-your-infra**: there is no vendor recording API; Hotato prints
  the recording config (`hotato setup`) and scores the file your deployment
  writes.
- **mono / mixed**: a single combined channel. Attributing overlap to
  caller or agent needs separated tracks, so Hotato scores a combined
  channel only behind an explicit `--allow-mono` / `HOTATO_ALLOW_MONO=1`
  opt-in, labeled indicative only. Separated turn-taking analysis needs a
  dual-channel file.

Each adapter ships only the endpoints confirmed verbatim in the spec;
where it marks a list-calls endpoint or channel layout **unconfirmed /
none**, Hotato falls back to explicit call ids and documents the gap here.

## Dual-channel: auto-pull, separated scoring

One entry per stack, verified 2026-07-07 unless noted otherwise. Each
auto-pulls a recording with per-party channels, so Hotato scores full
separated turn-taking without `--allow-mono`.

- **Vapi**
  - List recent calls: `GET https://api.vapi.ai/call` (params `limit`, `createdAtGt/Lt`) → JSON array of Call objects (`id`, `createdAt`)
  - Fetch recording: `GET /call/{id}` → `artifact.recording.stereoUrl` (current); deprecated fallbacks `artifact.stereoRecordingUrl`, `call.stereoRecordingUrl`
  - Channel basis: `stereoUrl` is a distinct 2-channel file (customer ch0, assistant ch1)
- **Twilio**
  - List recent calls: `GET .../Accounts/{Sid}/Recordings.json` (params `PageSize`, `DateCreatedAfter/Before`, `callSid`) → `recordings[].sid`
  - Fetch recording: `GET .../Recordings/{RE...}.wav?RequestedChannels=2` (HTTP Basic)
  - Channel basis: dual-channel only if the recording was created `RecordingChannels=dual`; 400 → clean stop or `--allow-mono` → `RequestedChannels=1`. Two-party order: ch0 = caller, ch1 = agent; conference: ch0 = first participant
- **Retell**
  - List recent calls: **none confirmed** -- the spec marks list-calls unconfirmed; do not fabricate one. Pull from explicit `--call-id`
  - Fetch recording: `GET https://api.retellai.com/v2/get-call/{id}` (Bearer) → `scrubbed_recording_multi_channel_url` preferred, then `recording_multi_channel_url`; plain mono `recording_url` rejected unless `--allow-mono`
  - Channel basis: per-party channels on the `*_multi_channel_url` fields
- **LiveKit**
  - List recent calls: **capture-in-your-infra** -- `ListEgress` is an RPC method name only; no REST list. `hotato setup --stack livekit`
  - Fetch recording: two audio-only Track egresses (one per party); recording location in `egress_info.file_results[].location`
  - Channel basis: separated by running one Track egress per participant
- **Pipecat**
  - List recent calls: **capture-in-your-infra** -- Pipecat Cloud session-list has no recording field; OSS has no list at all
  - Fetch recording: `AudioBufferProcessor(num_channels=2)` in-pipeline (user left, bot right); you write the WAV
  - Channel basis: 2-channel in-pipeline

## Mono / mixed: auto-pull, `--allow-mono`, indicative only

Each has a verified list + fetch, but the vendor produces a single combined
recording with **no documented per-party channel**. Hotato scores it
indicative-only, behind `--allow-mono`. One entry per stack, verified
2026-07-07.

- **Bland AI**
  - List recent calls: `GET https://api.bland.ai/v1/calls` → `calls[].call_id`
  - Fetch recording: `GET /v1/calls/{id}` → `recording_url`
  - Mono basis (verbatim): no stereo/channel field in the recording or call-details schema
- **ElevenLabs Conversational AI**
  - List recent calls: `GET https://api.elevenlabs.io/v1/convai/conversations` (params `page_size`, `call_start_after_unix`) → `conversations[].conversation_id`
  - Fetch recording: `GET /v1/convai/conversations/{id}/audio` → combined audio
  - Mono basis (verbatim): docs state the audio "does not include separate caller/agent channels, only a combined full conversation MP3"
- **Synthflow**
  - List recent calls: `GET https://api.synthflow.ai/v2/calls?model_id=…` (params `limit`, `from_date` epoch-ms) → `response.response.calls[].call_id`
  - Fetch recording: `GET /v2/calls/{id}` → `response.response.calls[0].recording_url` (a Twilio Recordings URL)
  - Mono basis (verbatim): Synthflow documents no dual-channel option; `recording_url` is a bare Twilio URL
- **Millis AI**
  - List recent calls: `GET {base}/call-logs` (US `api-west`, EU `api-eu-west`; param `limit`) → `histories[].session_id`
  - Fetch recording: `GET /call-logs/{session_id}` → `recording.recording_url`
  - Mono basis (verbatim): schema exposes only `recording_url`; `CallSettings` has only a boolean `enable_recording`
- **Cartesia (Line)**
  - List recent calls: `GET https://api.cartesia.ai/agents/calls?agent_id=…` (param `limit`; needs `Cartesia-Version` header) → `data[].id`
  - Fetch recording: `GET /agents/calls/{id}/audio` → `audio/wav`
  - Mono basis (verbatim): dual-vs-mono **unconfirmed** in the docs; treated as mono until a live channel-count check proves otherwise

`--stack synthflow` needs `--model-id` (its list endpoint requires `model_id`);
`--stack cartesia` needs `--agent-id`; `--stack millis` accepts `--base-url` for
the EU region. Single-call `capture`/`pull` by explicit id needs none of these.

## No vendor recording to pull

- **Deepgram Voice Agent API** (2026-07-07, confirmed absent) -- real-time
  WebSocket (`wss://agent.deepgram.com/v1/agent/converse`), no REST
  list-calls, fetch-recording endpoint, or recording-ready webhook: the call
  is never stored vendor-side. Capture it in your own infra instead (the
  LiveKit/Pipecat pattern above).
- **PlayAI (formerly Play.ht)** (2026-07-07, dead endpoints verified) -- the
  conversational-agent product is retired: `play.ai` DNS does not resolve,
  `docs.play.ai` returns `DEPLOYMENT_NOT_FOUND`. The only live domain,
  `docs.play.ht`, is TTS-only, no calls/recordings API.

## Needs credentials or a live probe before shipping

Documented weakly or behind a login/SPA, the spec could not verify a
recording field-path or channel layout for these, so they ship as
fallbacks (explicit call ids or the vendor's own webhook URL), not full
adapters -- listed here so the supported set stays exact.

- **Regal.ai**: no list-calls, no REST fetch-recording endpoint. The
  recording arrives only via the `call.recording.available` webhook's
  `properties.recording_link`, a literal Twilio Recordings URL -- feed it
  through Hotato's existing Twilio path, the one shipped route for
  Regal-sourced recordings.
- **Thoughtly**: OpenAPI types responses as an untyped `GenericResponse.data`
  blob; no recording-URL field path could be confirmed live.
- **Sindarin**: no Sindarin-hosted recording endpoint found; `record` is
  submitted with the caller's own Twilio creds, so the recording lands in
  the customer's own Twilio account (use the Twilio path).
- **xAI Grok Voice Agent Builder**: OpenAI-Realtime-compatible; no documented
  list-calls or fetch-recording endpoint (marketing mentions in-product
  recording, but no confirmable API contract).
- **Cartesia dual-channel**: the `/agents/calls/{id}/audio` WAV's channel count
  was not confirmable in docs; treated as mono here pending a live check.
- **Twilio dual-channel edge behaviour**: the exact 400-on-unavailable and
  conference channel-ordering guarantees were carried forward from prior notes,
  not re-verified verbatim this pass.
- **Synthflow / Regal Twilio-URL dual-channel trick**: because their
  `recording_url` is a Twilio URL, `?RequestedChannels=2` *might* yield stereo,
  but neither vendor documents or guarantees it; unconfirmed until tested against
  a live account.

Also researched, login-gated or SPA-only docs or transcript-only APIs: Daily,
Ultravox, Hume EVI, Telnyx, Infobip, OpenAI Realtime, Amazon Connect, Parloa,
Sierra, Decagon, PolyAI, Voiceflow, Cognigy, Dialogflow CX, Genesys Cloud,
NICE CXone. The integration spec carries the per-platform facts and gaps.

## What holds across every adapter

Every adapter validates that a dual-channel file has one party per channel
(2 channels) before producing separated talk-over numbers; mono stacks
score degraded, only behind `--allow-mono`. Every live fetch/list path is
stdlib-only (`urllib`); stack SDKs import lazily, only inside your own
infra. Credentials from `hotato connect` sit locally at
`~/.hotato/connections.json` (mode 0600), going straight from your machine
to the vendor's API -- Hotato stays out of that exchange. Scoring runs
offline; the only network call is the recording download itself.


================================================================================
FILE: docs/STARTER.md
================================================================================

# The starter kit: `hotato init starter`

The fastest way to add hotato to an existing voice-agent repo: one command
scaffolds the CI gate, a stack-tuned config file, and the three directories
the rest of the docs assume already exist.

```bash
hotato init starter --stack vapi --out .
```

`--stack` is one of `vapi`, `retell`, `twilio`, `livekit`, `pipecat` --
every stack hotato has a shipped connector for
([`ADAPTER-STATUS.md`](ADAPTER-STATUS.md)). `--out` is usually `.`, the
repo root. Generation runs offline -- nothing to connect.

## What it writes

```
HOTATO.md                                # what was added, next steps (read this first)
hotato.yaml                              # config skeleton, tuned for --stack
.gitignore                               # excludes local/pulled recordings;
                                          #   keeps pinned fixture/contract clips committed
.github/workflows/hotato-contracts.yml   # the CI gate
fixtures/
  README.md
  scenarios/.gitkeep                     # -> hotato fixture create --out fixtures
  audio/.gitkeep
contracts/
  README.md
  .gitkeep                               # -> hotato contract create --out contracts
reports/
  README.md
  .gitkeep                               # local/CI scratch: doctor/report/sweep output
```

Each file writes only when it doesn't already exist (`--force` to
overwrite), whole or not at all. Names are namespaced away from your own
files (`HOTATO.md`, not `README.md`; `hotato-contracts.yml`, not
`hotato.yml`), so a first run drops in cleanly next to files you already
have.

## Two input paths, chosen by `--stack`

**Auto-pull** (`vapi`, `retell`, `twilio`): hotato fetches the recording
itself once you connect a key. `hotato.yaml`'s `credentials.env` names the
exact variable(s) (`VAPI_API_KEY`; `RETELL_API_KEY`; `TWILIO_ACCOUNT_SID`
+ `TWILIO_AUTH_TOKEN`), the same ones `hotato connect <stack>` reads.
`recording.access` is `auto-pull`.

**Capture-in-your-infra** (`livekit`, `pipecat`): capture happens inside
your own deployment, so no credentials are needed. `credentials.env` is
`[]`, `recording.access` is `capture-in-your-infra`; `hotato setup --stack
<stack>` prints the two-track capture scaffold, and you point `hotato
contract create --stereo` at the WAV your deployment writes.

## LiveKit and Pipecat runbook

LiveKit and Pipecat are the two stacks where capture and turn-taking
config live in your own code, ahead of any vendor API. Runbook for both,
capture through CI.

### LiveKit

1. **Capture.** Two audio-only Track egresses, one per participant --
   RoomComposite mixes both parties into one channel and can't attribute
   overlap. `hotato setup --stack livekit` prints the scaffold (Python
   `livekit-api`, `TrackEgressRequest` + `DirectFileOutput`); a
   ready-to-copy version lives at `adapters/livekit_capture.py`.
2. **Find the turn-taking config.** On `AgentSession(turn_handling=
   TurnHandlingOptions(...))`: `turn_detection`
   (`inference.TurnDetector()` / `"realtime_llm"` / `"vad"` / `"stt"` /
   `"manual"`), `endpointing` (`min_delay`, `max_delay`), and
   `interruption` (`enabled`, `mode`, `min_duration`, `min_words`,
   `false_interruption_timeout`, `resume_false_interruption`). Check what
   an agent file runs, statically:
   `hotato inspect --stack livekit --config agent.py`.
3. **Score it.**
   `hotato capture --stack livekit --caller caller.wav --agent agent.wav --onset <sec> --expect yield`
   (convert the egress output to WAV first: `ffmpeg -i caller.ogg caller.wav`).
4. **Fixture, contract, CI.** Same as every stack -- see "Turn your first
   bad call into a contract" below.

### Pipecat

1. **Capture.** A 2-channel `AudioBufferProcessor` in-pipeline (channel 0
   = user/caller, channel 1 = bot/agent) -- don't mix down to one channel.
   `hotato setup --stack pipecat` prints the scaffold; a ready-to-copy
   version lives at `adapters/pipecat_capture.py`.
2. **Find the turn-taking config.** On `PipelineTask`'s user-turn
   strategies: start (`VADUserTurnStartStrategy`,
   `TranscriptionUserTurnStartStrategy`,
   `MinWordsUserTurnStartStrategy(min_words=...)`,
   `KrispVivaIPUserTurnStartStrategy(...)`) and stop
   (`SpeechTimeoutUserTurnStopStrategy(user_speech_timeout=...)`,
   `TurnAnalyzerUserTurnStopStrategy(turn_analyzer=...)`) --
   `MinWordsInterruptionStrategy` is deprecated since pipecat 0.0.99 in
   favor of `MinWordsUserTurnStartStrategy`. Check what a bot file runs,
   statically: `hotato inspect --stack pipecat --config bot.py`.
3. **Score it.**
   `hotato capture --stack pipecat --stereo captured.wav --expect yield`
   (write the WAV from `AudioBufferProcessor`'s `on_audio_data` handler
   first).
4. **Fixture, contract, CI.** Same as every stack -- see "Turn your first
   bad call into a contract" below.

Both APIs move; `hotato setup` and `hotato inspect` print the
verified-against date. Full field-level detail:
[`ADAPTER-STATUS.md`](ADAPTER-STATUS.md) (capture) and
[`FIX-PLANS.md`](FIX-PLANS.md) (inspect, Level 1 of the fix ladder).

## The CI gate

`.github/workflows/hotato-contracts.yml` runs on push, pull request, and
weekly. Two guarded steps that pass clean, as a no-op, until you add a
first contract or fixture (a fresh scaffold's normal starting state):

```bash
hotato contract verify contracts --junit hotato.xml --format json > contracts-verify.json
hotato run --scenarios fixtures/scenarios --audio fixtures/audio --format json > fixtures-run.json
```

The JUnit file publishes as a build artifact on every run (`always()`),
whether the gate passed, failed, or had nothing to check yet.

The three auto-pull stacks also get a `weekly-sweep` job: a passive,
candidate-only sweep of recent calls (`hotato sweep --stack <stack>`),
ranked by salience for you to review and label. It ships disabled (`if:
false`) -- flip it to `true` once the stack's credential env var(s) are
repo secrets (Settings -> Secrets and variables -> Actions). A live pull
runs only on your say-so, made once, in your own CI config.
`livekit`/`pipecat` skip this job; capture already happens in your own
deployment.

## Turn your first bad call into a contract

```bash
# auto-pull stacks
hotato connect vapi --api-key <key>
hotato sweep --stack vapi --out hotato-sweep.html
# open hotato-sweep.html, pick a candidate moment, then:
hotato contract create --from-candidate hotato-sweep.json#1 \
    --expect yield --id refund-cutoff-001 --out contracts

# capture-in-your-infra stacks
hotato setup --stack livekit
# once your deployment writes a two-channel WAV:
hotato contract create --stereo call.wav --onset 42.18 \
    --expect yield --id refund-cutoff-001 --out contracts
```

Commit the resulting `contracts/refund-cutoff-001.hotato/` directory. The
next push runs it through the CI gate above.

**A contract bundle contains call audio** (`audio/event.wav`). If this
repo is or could become public, commit a sanitized fixture (synthetic or
consent-cleared) and keep customer contracts in a private repository or
controlled artifact storage. See [`CONTRACTS.md`](CONTRACTS.md).

## Read more

- The bundle layout and the create/verify/inspect/pack/unpack commands:
  [`CONTRACTS.md`](CONTRACTS.md)
- The underlying fixture primitive, one bad call to a CI gate in five steps:
  [`BAD-CALL-TO-CI.md`](BAD-CALL-TO-CI.md)
- Per-stack connector support, verified against the vendor's live docs:
  [`ADAPTER-STATUS.md`](ADAPTER-STATUS.md)
- The connect-once bulk pull-and-analyze recipe: [`CONNECT.md`](CONNECT.md)
- An agent adding hotato to a repo end to end: [`../AGENTS.md`](../AGENTS.md)


================================================================================
FILE: docs/TRACE.md
================================================================================

# Voice traces: coincidence, not root cause

A voice trace is a timeline of discrete voice-pipeline events
(caller/agent audio activity, TTS cancel/stop, ASR partials, tool calls,
...): "the agent talked over the caller" becomes "evidence suggests TTS
cancellation lagged: cancel requested at 42.40s, audio stopped at 43.60s."
See [`docs/OTEL.md`](OTEL.md) for the ingest source formats.

Three commands:

```bash
hotato trace ingest --otel traces.jsonl --out voice_trace.jsonl
hotato trace attach contracts/refund-cutoff-001.hotato --trace voice_trace.jsonl
hotato trace export contracts/refund-cutoff-001.hotato --format otel --out otel.jsonl
```

## Schema

`voice_trace.jsonl` is JSONL (one meta line, then one line per span -- the
same convention `evidence/frames.jsonl` uses), validating against
`hotato.voice_trace.v1` (`src/hotato/schema/voice_trace.v1.json`) once
reassembled by `hotato.trace.load_voice_trace_jsonl`:

```json
{
  "schema": "hotato.voice_trace.v1",
  "call_id": null,
  "deployment": {
    "stack": "vapi",
    "agent_id": null,
    "git_sha": "deadbeef",
    "config_hash": "sha256:..."
  },
  "spans": [
    {"type": "agent_audio_active", "start_sec": 0.0, "end_sec": 2.9},
    {"type": "tts_cancel_requested", "time_sec": 2.6},
    {"type": "tts_audio_stopped", "time_sec": 2.9},
    {
      "type": "asr_partial",
      "start_sec": 2.4,
      "end_sec": 2.95,
      "text_redacted": true
    },
    {
      "type": "tool_call",
      "start_sec": 1.1,
      "end_sec": 1.42,
      "name": "lookup_order",
      "latency_ms": 320
    }
  ],
  "source": {"format": "otel-jsonl-bridge", "input_span_count": 6}
}
```

A span carries an open `type` string; common ones are
`caller_audio_active`, `agent_audio_active`, `tts_cancel_requested`,
`tts_audio_stopped`, `asr_partial`, `tool_call`, `llm_first_token`,
`handoff`. An unrecognized type passes through unchanged -- additive,
forward-compatible with a span type this release doesn't name. One time
shape per span: an interval carries `start_sec`/`end_sec`, a point event
carries `time_sec`.

## Redaction by default

`call_id` and `deployment.agent_id` are dropped (`null`) unless
`--include-identifiers` is passed at ingest. An `asr_partial` span's
transcript text is dropped (`text_redacted: true`, no `text` key) unless
`--include-text` is passed. `deployment.stack` / `git_sha` / `config_hash`
are kept by default -- not treated as identifiers.

## Ingest

```bash
hotato trace ingest --otel traces.jsonl --out voice_trace.jsonl
hotato trace ingest --otel export.json --out voice_trace.jsonl \
    --stack vapi --include-identifiers --include-text
```

`--otel FILE` accepts a standard OTel JSON export (a document with a
top-level `resourceSpans` array) or hotato's own OTel bridge JSONL (see
[`docs/OTEL.md`](OTEL.md)). `--stack` / `--call-id` / `--agent-id` /
`--git-sha` / `--config-hash` override or fill in the source's resource
attributes. Refused (exit 2, nothing written) for an unreadable file, an
empty file, or a source with zero spans.

## Attach

```bash
hotato trace attach contracts/refund-cutoff-001.hotato --trace voice_trace.jsonl
```

Copies the trace into `<bundle>/traces/voice_trace.jsonl` and re-renders
`evidence/timeline.html` with the trace's events as an additional row,
aligned to the same [0, duration] scale as the existing caller/agent
timeline. This reads the bundle's own `evidence/frames.jsonl` and
`contract.json` back in, reusing the scored evidence as-is: attaching a
trace needs no diarization extra and no re-scoring. On a diarized-mono
bundle (no frame-level evidence), the base timeline says so plainly, and
the trace row still renders on its own scale.

`contract.json` records the attachment (additive, schema-safe: `trace:
{attached, path, span_count, attached_at, source_format}`). Refused (exit
2) for a missing bundle, a trace file that doesn't validate as
`hotato.voice_trace.v1` JSONL, or an already-attached trace without
`--force`.

### Report wording

When a `tts_cancel_requested` / `tts_audio_stopped` pair is present, the
timeline states the delta plainly:

> Evidence suggests TTS cancellation delay: cancel requested at 2.60s,
> audio stopped at 2.90s (delta 0.30s).
> Hotato does not prove root cause.
> Unknowns: no client-side playout trace was attached.

The last line always appears in this release: a client-side playout trace
(when the caller's device stopped rendering audio, not when the server
issued the stop) sits outside what this release's span types collect, so
the gap is always named.

## Export

```bash
hotato trace export contracts/refund-cutoff-001.hotato --format otel --out otel.jsonl
```

Writes the bundle's attached trace back out as hotato's OTel bridge JSONL
-- the exact shape `trace ingest` reads, so `ingest -> attach -> export ->
ingest` round-trips identical spans. `--format` is claimed by the export
format here (only `otel` is supported today); pass `--json` for the
machine summary instead of the default text line. Refused (exit 2) when
the bundle has no attached trace, or `--out` exists without `--force`.

## What a voice trace adds

Timing correlation only: a pattern of events lined up with the contract's
timing measurement, named plainly. Authorization, identity, compliance,
policy safety, intent, and root cause stay outside its claims; a
client-side playout moment is named only when one was attached.

## Read more

- Source formats and the OTel bridge JSONL shape:
  [`docs/OTEL.md`](OTEL.md)
- Failure contracts a trace attaches to: [`docs/CONTRACTS.md`](CONTRACTS.md)


================================================================================
FILE: docs/OTEL.md
================================================================================

# OTel ingest: two source shapes

`hotato trace ingest --otel FILE` turns an OTel trace into hotato's own
`hotato.voice_trace.v1` spans -- reading a file you already have (an
exported trace, or a script's own event log) offline, once, and
translating `name`, `startTimeUnixNano`/`endTimeUnixNano`, `attributes`,
and span `events` into hotato's span shape. It recognizes two input
shapes; `source.format` records which one it used (`"otel-json"` or
`"otel-jsonl-bridge"`).

## 1. Standard OTel JSON export (`otel-json`)

A single JSON document with a top-level `resourceSpans` array (the shape a
standard OTel exporter/collector writes):

```json
{
  "resourceSpans": [{
    "resource": {"attributes": [
      {"key": "service.name", "value": {"stringValue": "vapi"}},
      {"key": "git_sha", "value": {"stringValue": "cafebabe"}}
    ]},
    "scopeSpans": [{"spans": [
      {
        "name": "agent_audio_active",
        "startTimeUnixNano": "1000000000",
        "endTimeUnixNano": "4400000000"
      },
      {
        "name": "tts.cancel_requested",
        "startTimeUnixNano": "2600000000",
        "events": []
      }
    ]}]
  }]
}
```

- Resource attributes flatten into a plain dict: `service.name` (or a custom
  `stack` attribute) becomes `deployment.stack`; `git_sha` / `config_hash`,
  when present, fill the matching deployment fields. Only the FIRST
  `resourceSpans` entry supplies deployment metadata -- every entry's spans
  are still walked.
- Timestamps convert to seconds relative to the EARLIEST timestamp in the
  file, matching the audio-relative-seconds convention every other hotato
  timestamp uses.
- A span's own `name` maps to a hotato span `type` via a small documented
  table (`caller_audio_active`, `agent_audio_active`,
  `tts.cancel_requested` -> `tts_cancel_requested`,
  `tts.audio_stopped` -> `tts_audio_stopped`, `asr.partial` -> `asr_partial`,
  `llm.first_token` -> `llm_first_token`); an unmapped name passes through
  unchanged.
- Span `events` -- OTel's point-in-time markers nested inside a span, e.g.
  a `tts.cancel_requested` marker inside a broader `tts_playback` span --
  flatten into their own point events the same way top-level spans do.
- A `tool_call`-mapped span takes its `name` field from the `tool.name` /
  `gen_ai.tool.name` attribute; a `latency_ms` attribute is used when
  present, otherwise computed from the span's own start/end.
- An `asr_partial`-mapped span's `text` / `asr.transcript.partial` attribute
  is captured and redacted by default (`text_redacted: true`); pass
  `--include-text` to keep it in the output.

## 2. Hotato's OTel bridge JSONL (`otel-jsonl-bridge`)

The shape for a script or test fixture that skips a full OTel exporter: one
JSON object per line (or one bare JSON array of them). Each line is a span
or a meta/resource line. A span line uses `type` directly; `name` is
reserved for `tool_call`'s own tool name, distinct from the span kind:

```
{"type": "caller_audio_active", "start_sec": 2.40, "end_sec": 4.10}
{"type": "agent_audio_active", "start_sec": 0.00, "end_sec": 2.90}
{"type": "tts_cancel_requested", "time_sec": 2.60}
{"type": "tts_audio_stopped", "time_sec": 2.90}
{"type": "asr_partial", "start_sec": 2.40, "end_sec": 2.95, "text": "wait, I need a refund"}
{"type": "tool_call", "start_sec": 1.10, "end_sec": 1.42, "name": "lookup_order", "latency_ms": 320}
```

An optional meta/resource line (no `type` key) attaches deployment
metadata and a call id:

```
{"call_id": "demo-call-001", "deployment": {"stack": "vapi", "agent_id": "agent-demo-1", "git_sha": "deadbeefcafe", "config_hash": "sha256:0f1e2d3c"}}
```

`--call-id` / `--stack` / `--agent-id` / `--git-sha` / `--config-hash` on
`trace ingest` override whatever a meta line (or a standard export's
resource) provided. `hotato trace export` writes exactly this bridge
shape, so `ingest -> attach -> export -> ingest` round-trips the same
spans; the shipped test fixture (`tests/data/otel/demo-trace.otel.jsonl`)
uses it.

## Which one to use

Already running an OTel collector or SDK? Point `--otel` at that trace
export directly -- the standard-export path reads it, best-effort. Wiring a
quick script (a webhook handler, a log-line scraper) outside a full OTel
pipeline? Write the bridge JSONL directly instead -- same information,
already flat.


================================================================================
FILE: docs/latency-waterfall.md
================================================================================

# Why is my voice agent's perceived latency worse than its dashboard?

Because a dashboard averages each pipeline component on its own while the
caller feels the sum of every hop on one specific turn: hotato renders a
per-hop latency waterfall (STT, LLM, tool call, TTS, transport) derived from
the trace spans of the exact call being scored, next to that call's timing
verdict, so the slow hop on the turn that went wrong is a row in a table
instead of a hunch.

## From a trace to the waterfall

Two commands. First normalize a trace into `hotato.voice_trace.v1`; the input
is a standard OTel JSON export or hotato's documented bridge JSONL (both
shapes: [`docs/OTEL.md`](OTEL.md)), and the repo ships a worked example at
`tests/data/otel/demo-trace.otel.jsonl`:

```console
$ hotato trace ingest --otel tests/data/otel/demo-trace.otel.jsonl --out voice_trace.jsonl
ingested voice trace: voice_trace.jsonl
  format:  otel-jsonl-bridge
  spans:   6
  stack:   vapi
  types:   agent_audio_active, asr_partial, caller_audio_active, tool_call, tts_audio_stopped, tts_cancel_requested
```

Then render the report for the recording with the trace alongside (the
two-channel example recording here ships in the package):

```console
$ hotato report --stereo src/hotato/data/audio/01-hard-interruption.example.wav \
    --trace voice_trace.jsonl --format md --out report.md --no-fail
wrote markdown report (1 events) to report.md
```

`--format md` writes Markdown; the default self-contained HTML report carries
the same block.

## Read the waterfall

The report's forensic section is derived entirely from evidence already in the
report: the timing verdict, the attached voice trace's span timestamps, and
the evaluated assertions. This is the block the commands above produce,
verbatim:

```markdown
### Per-hop latency waterfall

| hop | latency | derived from |
| --- | --- | --- |
| STT (speech to text) | 550 ms | asr_partial recognition window (first start to last end) |
| LLM (first token) | not captured | no llm_first_token span in the trace |
| Tool call | 320 ms | sum of 1 tool_call latency_ms |
| TTS (speech out) | 300 ms | tts_audio_stopped minus tts_cancel_requested (cancellation lag) |
| Transport (backend HTTP) | not captured | no http_exchange spans in the trace |
```

Each row names the spans it was derived from, so every number is traceable to
a timestamp you supplied. A hop with no spans reads `not captured`, never
estimated: the example trace carries no `llm_first_token` and no
`http_exchange` spans, so those rows say so. A pipeline that emits all five
span kinds gets all five hops measured.

The trace stays context, not a score. The report states it in place: the
spans are rendered alongside the timing measurement, and `did_yield`,
talk-over, time to yield, and the PASS/FAIL verdict are unaffected by
anything in the trace.

## Pin it to a failure

The same trace file attaches to a pinned failure so the waterfall's source
spans travel with the evidence:

```bash
hotato trace attach contracts/demo-missed-interruption.hotato --trace voice_trace.jsonl
```

That writes the trace into the contract bundle and re-renders its evidence
timeline with an aligned trace row. The full trace path (ingest, attach,
export, redaction defaults) is [`docs/TRACE.md`](TRACE.md); the report
surfaces in depth are [`docs/REPORTS.md`](REPORTS.md).


================================================================================
FILE: docs/REPORTS.md
================================================================================

# Reports: doctor, report, team, export

Four surfaces, one scorer: reproducible timing measurements read straight
from the envelope, with the method exposed at every layer.

## `hotato doctor`: the 5-minute path

One command: pass a recording and it scores that; run it bare and it runs
the bundled self-test battery. Either way it writes a self-contained HTML
report and opens it in your browser -- on a headless box, it prints the
path instead.

```bash
uvx hotato doctor --stereo call.wav     # score your call, open the report
uvx hotato doctor                       # self-test fallback, same flow
uvx hotato doctor --demo --no-open --out report.html
```

Wraps the scorer and report, runs offline end to end. Exit codes match
`run`: `0` all pass, `1` a regression (`--no-fail` forces `0`), `2`
usage/IO error or a not-scorable recording -- meaning it lacks a moment to
measure (the caller channel is silent, or the agent wasn't talking when
the caller started), reported plainly instead of a verdict.

**Your own recording gets its audio embedded in the report by default.**
Scoring a call with `--stereo` / `--caller`+`--agent` writes the scored
audio into the HTML file as a base64 data URI, so hearing the moment next
to its timeline works offline (the bundled self-test fallback stays
unembedded). Sharing, mailing, or posting the resulting `report.html`
shares the raw recording -- treat it with the same care.

## `hotato report`: the visual report

One self-contained file -- inline CSS, inline SVG, zero external requests
-- that opens by double-click and survives being mailed around.

```bash
uvx hotato report --stereo call.wav --out report.html
uvx hotato report --suite barge-in --out selftest.html
uvx hotato report --stereo call.wav --format md --out report.md
uvx hotato report --stereo call.wav --embed-audio --out report.html  # opt-in: embed the audio too
```

`--embed-audio` embeds the exact scored audio (base64, under a size cap)
so the report is a fully self-contained, hearable artifact. Same caution:
a report built with `--embed-audio` (or `hotato doctor` on your own
recording, which sets it by default) carries the call audio inside the
HTML -- give it the same care as the raw recording before posting it
publicly or attaching it to a public issue/PR.

Per event it draws a to-scale caller/agent activity timeline from the
frame data -- overlap shaded, caller-onset and yield markers, measured
talk-over seconds, expected vs. measured, and a PASS or FAIL chip.

Once the page has at least three event cards, an analytics rollup follows,
computed from the same measurements (fewer events skips it):

| Chart | Shows |
|---|---|
| Time-to-yield distribution | One dot per measured yield, with mean, median, and p90 (definitions in `METHODOLOGY.md`) |
| Talk-over histogram | Per-event seconds, bucketed on a fixed grid |
| Failure clustering by fix class | A batch of failures reads as "these five share one config setting" instead of five separate mysteries |

Every timeline carries a collapsible **frame inspector**: the full frame
dump behind that event as a table (`t_sec`, per-channel dBFS, active
flags, thresholds) -- any pixel on the page can be re-derived by hand.
The `ScoreConfig` thresholds sit in one collapsed "Thresholds used" panel
at the end, reproducible without stamping the parameter table above every
render.

### Voice-trace context with `--trace`

Pass `--trace voice_trace.jsonl` (written by `hotato trace ingest`;
format: [`TRACE.md`](TRACE.md)) to attach a voice trace as **context**
next to the timing:

```bash
hotato trace ingest --otel spans.otel.jsonl --out voice_trace.jsonl
hotato report --stereo call.wav --trace voice_trace.jsonl --out report.html
```

The report gains one collapsed "Trace (context, not a score)" section:
the trace's discrete voice-pipeline events -- TTS cancel/stop, ASR
partials, tool calls -- as a mono span table. It stays scoped to context:
`did_yield`, `talk_over_sec`, `seconds_to_yield`, and the PASS/FAIL
verdict come from the scorer alone; the trace folds into the envelope as
an additive `trace_context` key. Without `--trace`, the report is
byte-identical to one built before the flag existed. Redaction carries
through: a span ingested without `--include-text` shows `[redacted]`
instead of its text, so a shared report carries only what the trace
already chose to keep. Same `trace=` parameter on `build_report_html` /
`build_report_md`.

### Reliability in the scorecard (pass@1 / pass@k / pass^k)

When a report carries an `assert.v1` envelope with dimension-tagged
results, the "Deterministic" shelf renders as a per-dimension
**scorecard** (outcome / policy / conversation / speech / reliability).
**Reliability** is pass^k's home (definitions: [`SIMULATE.md`](SIMULATE.md)),
rendering the repetition data you thread in:

```python
from hotato import report, simulate

summary = simulate.run_matrix(scenario, conversation_test=ct)   # a matrix aggregate
html, _ = report.build_report_html(stereo="call.wav",
                                   assertions=env, reliability=summary)
```

`reliability=` accepts a `simulate.run_matrix` summary, a bare
`simulate.reliability()` dict, or a `{"aggregate": <reliability dict>, "origin":
...}` wrapper. The dimension shows each number labeled, tabular mono:
**pass@1**, **pass@k**, **pass^k**, `n`, `k`, `passes`, a **Wilson 95% CI**
on pass@1, a **per-variation-cell** breakdown when the summary carries
one, and a **SIMULATOR_INVALID** bucket for broken fixtures, excluded
from `n`. pass^k stays its own number, its own lane -- no `overall_score`
field to blend into. Runs from simulation are labeled
**origin=simulated**, scoped apart from production reliability.

`hotato test run --repetitions N` (`N > 1`) computes this aggregate over
the N deterministic runs and threads it into `report.{html,md}`
automatically. With no repetition data, the dimension shows the
empty-state -- "not measured: no repeated runs in this report" --
byte-identical to a report built without the parameter.

### Regression deltas with `--base`

Save an envelope, then compare a later run against it:

```bash
hotato run --suite barge-in --format json > base.json
hotato report --suite barge-in --base base.json --out report.html
```

The report renders per-scenario talk-over and time-to-yield deltas, worse
and better marks clearly flagged. The same `--base` flag drives
`scripts/pr_comment.py`'s CI comment (`docs/CI.md`).

### PDF

The page ships print CSS: print it from any browser and the interactive
parts collapse into a clean paper layout -- print-to-PDF is the PDF
export.

## `hotato team`: the trend view

Aggregates a directory of run envelopes into one trend.

```bash
hotato run --suite barge-in --format json > runs/001.json
# ... more runs over days or branches ...
hotato team runs/ --html team.html --out agg.json
```

Reports: run count; mean/median/p90 talk-over and time-to-yield, pooled
across all events; mean/median/p90/p95 response gap (dead air before the
agent speaks), pooled the same way; pass rate per run; the most common
failure class; and a pass-rate trend line in the HTML page. `--order
mtime` (default) orders by file time; `--order name` uses the filename,
so a numeric prefix acts as an explicit index.

`--max-response-gap SECONDS` turns the pooled p95 response gap into a
latency SLA: the run exits `1` exactly when p95 exceeds the bound, the
same pass/fail contract as a talk-over or time-to-yield regression
(`--no-fail` always exits `0`). Percentile definitions: `METHODOLOGY.md`;
pooling shape: `dist_summary` in `src/hotato/_stats.py`.

Fewer than two runs is stated plainly, and exits `0`; a trend line renders
once there are enough points to mean something.

## `hotato export`: research-grade CSVs

Scores a recording (or the bundled battery) exactly like `hotato run`, and
writes three files into a directory:

```bash
uvx hotato export --stereo call.wav --out research/
uvx hotato export --suite barge-in --out research/
```

| File | Contents |
|---|---|
| `events.csv` | One row per scored event, every measured signal plus verdict |
| `frames.csv` | One row per VAD frame, the evidence behind every number |
| `envelope.json` | The standard machine envelope, unchanged |

Column meanings live in comment lines at the top of each CSV, so the
files are self-describing in a notebook or stats package months later. An
empty cell means "not derivable", distinct from a zero. Stdlib only,
offline.

`export` also prints mean/median/p90/p95 response gap pooled across the
exported events, and accepts the same `--max-response-gap SECONDS`
latency SLA gate as `team` (exit `1` when pooled p95 exceeds the bound).
A plain export with no `--max-response-gap` writes a byte-identical
`envelope.json` -- pooled numbers and the gate live only in the printed
summary and returned manifest, never in the CSVs or envelope file.


================================================================================
FILE: docs/ASSERTIONS.md
================================================================================

# Assertions (`assert.v1`): deterministic, typed, each scored on its own lane

`hotato.assert_` checks a call's transcript, ingested trace, and timing
against a fixed set of typed assertions, each kind scored on its own lane by
construction. Every kind is a regex, checksum, or span/dict lookup --
deterministic end to end, no model call in the loop.

```python
from hotato import assert_ as A

ctx = A.build_context(
    transcript_path="transcript.json",   # or transcript=[...] turns already in hand
    trace_path="voice_trace.jsonl",      # or spans=[...] already in hand
    timing=envelope,                     # optional: a run's envelope.v1 events
)
env = A.run_assertions_from_file("assertions.yaml", ctx)
print(env["exit_code"])   # 0 pass, 1 fail
```

Python API: [`docs/API.md`](API.md) covers the scoring envelope this
complements; the schema is `src/hotato/schema/assert.v1.json`.

Runnable ground truth:
[`examples/reference-agent`](../examples/reference-agent) exercises these
kinds end to end: a 375-run offline suite (25 scenarios, 5 caller
behaviours, 3 audio environments) whose `tool_result`, `state`, `sequence`,
`handoff`, and `tool_error` assertions surface four seeded agent defects,
deterministically. Method: the "Say-do verification methodology" section of
[`METHODOLOGY.md`](../METHODOLOGY.md).

## The core deterministic kinds

Every kind is deterministic, so every result carries `deterministic: true`,
including `INCONCLUSIVE`. The five below are the original core; the full
vocabulary is under [the whole `kind` vocabulary](#the-whole-kind-vocabulary).

| Kind | Checks | Reads |
| --- | --- | --- |
| `phrase` | a regex is present (or, in `absent` mode, never present), with an optional `role` filter and `position` (`first`/`last`/`any`) | transcript text |
| `pii` | deterministic detectors (`ssn`, `card_luhn` with full Luhn validation, `email`, `phone`) find nothing, `mode: must_not_leak` | transcript text |
| `policy` | a named, versioned, offline rule pack's banned-language and required-disclosure rules | transcript text |
| `tool_call` | a tool was (or was not) called, with an optional argument subset, a count bound, a required order across tools, or a "never before" ordering constraint | ingested `voice_trace.v1` spans only |
| `outcome` | task success as `all_of`/`any_of` a list of the sub-predicates above (`tool_called`, `phrase`, `field_present`), reported as a `met`/`of` fraction | whichever context each sub-predicate needs |

`tool_call` checks only the ingested trace (`hotato trace ingest`,
[`docs/TRACE.md`](TRACE.md)) -- that's the evidence a tool ran; an agent's own
words claiming it ran don't count. `pii` surfaces only a `[REDACTED]`
transcript artifact plus hit metadata (detector name, turn index, role) --
never the matched text.

### The whole `kind` vocabulary

The full `assert.v1` `kind` vocabulary is **20 deterministic kinds**, all
`deterministic: true`, all model-free. Beyond the five core kinds:
`tool_result` and `tool_error` (a tool's returned value or raised error, from
the trace), `http_result` (a recorded HTTP exchange span's method, URL,
status, and response subset, from the trace), `state` and `state_change` (a
state adapter's snapshot or
transition -- see [STATE-ADAPTERS.md](STATE-ADAPTERS.md)), `handoff`, `dtmf`,
`termination`, `latency`, `timing_contract`, `entity_accuracy`, `sequence`,
`count`, `formula` (a boolean composite over other assertions' results in
the same run, below), and `order` (a transcript ordering check, below). Two
more, `human_rubric` and `judge_rubric`, belong to the SEPARATE
model-judged rubric lane ([RUBRIC.md](RUBRIC.md)); inside a raw `assert.v1`
document they resolve to a deterministic `INCONCLUSIVE`, so no model runs here
and the guarantee holds.

### `http_result`: a recorded HTTP exchange, checked from the trace

`http_result` reads `http_exchange` spans from the ingested
`hotato.voice_trace.v1` trace -- a recorded request/response report carrying
`method`, `url`, `status_code`, and `response`. Evaluation is a pure span
lookup: hotato never performs the request, so a rerun is deterministic and
offline, and it shares the Authority-1 wall with `tool_result` /
`tool_error` -- an agent's spoken claim about a request can never satisfy it.

```yaml
version: 1
assertions:
  - id: refund-posted
    kind: http_result
    method: POST                      # matched case-insensitively
    url_matches: "/v1/refunds$"       # a regex searched against the span's url
    status: 201                       # one status, or a list: [200, 201]
    response_subset: {status: ok}     # optional: fields the response must carry
```

The span's `method` and `url` select the exchange; `status` and
`response_subset` then judge it. `PASS` carries the grounding `span_ids`;
`FAIL` distinguishes "no exchange matched the method and URL" from "the
exchange matched but its status/response did not"; a context with no trace at
all reports `INCONCLUSIVE`, never a guess. A malformed assertion (missing
`method`, an invalid `url_matches` regex, a `status` outside 100-599) is a
usage error caught up front, before any assertion runs.

### `formula`: a composite verdict over other assertions

`formula` combines OTHER named assertions' results from the same run into one
verdict. Its `expr` is a boolean expression over assertion ids -- `and`, `or`,
`not`, parentheses -- plus a weighted-sum comparison
(`0.6*a + 0.4*b >= 0.5`, where a bare id weighs 1). A referenced `PASS` is
true and a `FAIL` is false; the weighted form sums the weights of the
referenced assertions that passed and compares the total with the threshold.

```yaml
version: 1
assertions:
  - id: refund-issued
    kind: tool_call
    name: issue_refund
  - id: confirmation-said
    kind: phrase
    regex: "confirmation number"
    role: agent
  - id: identity-verified
    kind: tool_call
    name: verify_identity
  - id: happy-path
    kind: formula
    expr: "refund-issued and (confirmation-said or identity-verified)"
  - id: mostly-healthy
    kind: formula
    expr: "0.5*refund-issued + 0.3*confirmation-said + 0.2*identity-verified >= 0.7"
```

The expression is parsed by a small recursive-descent parser -- never
`eval()` -- and the whole reference graph is checked up front, before any
assertion runs: an unknown id, a self-reference, or a reference cycle
(including one through another `formula` or a `when:`) is a usage error
(`ValueError`, exit `2`). References evaluate first whatever the document
order (a `formula` may reference another `formula`), and `results` are always
emitted in document order, byte-stable.

A referenced `INCONCLUSIVE` makes the formula `INCONCLUSIVE`, with a reason
naming the reference -- absent input propagates; a composite never guesses.
A determinate result carries `refs` (the referenced ids), `met`/`of` (how many
referenced assertions passed), and, on `FAIL`, a reason listing which
references passed and which did not.

### `order`: one phrase must come before another

`order` checks the sequence of two phrases in the transcript: the FIRST turn
matching the `before` regex must precede (be strictly earlier than) the FIRST
turn matching `after`. It measures where two phrases first appear, not why they
appear. It reads only transcript text, so it needs a `--transcript`.

```yaml
version: 1
assertions:
  - id: verify-before-account-details
    kind: order
    before: "verify your identity|date of birth|last four"
    after: "your account number is|current balance"
    role: agent            # optional: match only this speaker's turns
    case_sensitive: false  # optional, default false
```

`PASS` carries `before_turn` and `after_turn` (the zero-based turn indices the
two phrases first matched at). If either phrase never matches there is no
ordering constraint to violate, so the result is a vacuous `PASS` carrying
`vacuous: true`. A context with no transcript reports `INCONCLUSIVE`, never a
guess. The phrases present but in the wrong order is a `FAIL`. A malformed
assertion (a missing or invalid `before`/`after` regex, a non-string `role`) is
a usage error caught up front, before any assertion runs.

### `when:`: conditional assertions

Any assertion -- any kind -- may carry an optional `when:` precondition: one
assertion id, or a list of ids. Unless every referenced assertion `PASS`ed,
the assertion is SKIPPED: an `INCONCLUSIVE` result with `skipped: true` and a
reason naming the unmet reference(s), because the check itself never ran --
there is no verdict to report.

```yaml
version: 1
assertions:
  - id: refund-issued
    kind: tool_call
    name: issue_refund
  - id: refund-amount-correct
    kind: tool_result
    name: issue_refund
    result_subset: {status: ok}
    when: refund-issued          # or a list: [refund-issued, other-id]
```

A referenced `FAIL` and a referenced `INCONCLUSIVE` both skip -- the
precondition demands a `PASS`. `when:` references obey the same up-front
unknown-name and cycle refusal as `formula`, and the referenced assertions
always evaluate first. Under `inconclusive_policy: fail`/`refuse` a skip gates
like any other `INCONCLUSIVE`, so a suite that must not stay green on skipped
checks can opt in.

## Context: transcript, trace, timing

`build_context` assembles the three inputs an assertion run needs, each built
from hotato's existing primitives:

- **transcript**: `hotato.transcribe` (the opt-in `[transcribe]` extra,
  faster-whisper) produces one, or pass `--transcript FILE` / `transcript_path=`
  with a JSON file -- a plain array of `{role, text, start, end}` turns, or
  the `{"segments": [...]}` shape `hotato.transcribe` and the MCP surface
  write.
- **trace**: `hotato trace ingest --otel FILE --out voice_trace.jsonl`
  ([`docs/TRACE.md`](TRACE.md), [`docs/OTEL.md`](OTEL.md)) produces the
  `hotato.voice_trace.v1` spans `tool_call` reads (`name`, `arguments`, and --
  when the source trace carries them -- `result`/`error`) and `http_result`
  reads (`method`, `url`, `status_code`, `response`).
- **timing**: a scoring run's own envelope (`hotato run --format json`,
  [`docs/API.md`](API.md)) passed straight through as read-only context for
  `outcome`'s `field_present` sub-predicate. Nothing here recomputes it.

Context you never supply stays `None`, distinct from a supplied `[]` or `{}`
that happens to be empty. An assertion whose required input is absent reports
`INCONCLUSIVE`. `tool_call` with `spans=[]` (a trace was ingested with zero
spans) is a `FAIL`, distinct from `tool_call` with no trace at all, which
reports `INCONCLUSIVE`.

## `assertions.yaml`

A small, dependency-free YAML subset (block mappings/sequences, flow
`[...]`/`{...}`, quoted or bare scalars, `#` comments) -- or valid JSON,
accepted directly. Hotato parses this subset itself, so the core stays zero
third-party dependency either way.

```yaml
version: 1
assertions:
  - id: refund-confirmed
    kind: outcome
    all_of: [{tool_called: issue_refund}, {phrase: "confirmation number", role: agent}]
  - id: tool-order
    kind: tool_call
    require_order: [verify_identity, lookup_account, issue_refund]
    never_before: {tool: issue_refund, until: verify_identity}
  - id: disclosure
    kind: phrase
    regex: "recorded for quality"
    role: agent
    position: first
  - id: no-ssn-leak
    kind: pii
    detectors: [ssn, card_luhn]
    mode: must_not_leak
```

Every assertion needs a unique `id` and a recognized `kind`. Kind-specific
fields are validated (bad regex, unknown detector, missing required field,
unsupported `version`) up front -- a malformed file is caught whole, before
partial results exist.

## The deterministic/judge split

This is the entire point of the module: structural, not a convention someone
can quietly break.

- Every result carries `kind` and `deterministic: true` -- true on every
  deterministic kind and every status, including `INCONCLUSIVE`, itself a
  deterministic read of missing required input.
- The envelope's `summary` **splits** `deterministic` (`{pass, fail,
  inconclusive}`) from `judge` (`{pass, fail}`), each in its own count -- the
  schema (`src/hotato/schema/assert.v1.json`) enforces this with
  `"overall_score": false` and a `not: {required: [overall_score]}` on the
  summary object.
- `judge` -- an LLM-scored rubric kind -- stays structurally quarantined
  from the deterministic count, so a model-scored result can never blend
  in. `summary.judge` reports `{"pass": 0, "fail": 0}`; `summary.note`
  states how many judge-scored assertions ran.
- Same inputs, same file, same result, every time: `run_assertions` is
  byte-stable across repeated calls on identical input -- no wall-clock
  timestamp or random id in the mix.

The report (below) renders this as two visually separate shelves, not one
number -- visible on the page, not just in the JSON.

## The report: two shelves, each counted on its own

`hotato.report.build_report_html` / `build_report_md` accept an optional
`assertions=` parameter: an already-evaluated `assert.v1` envelope (build one
with `run_assertions` / `run_assertions_from_file` / `run_assertions_from_yaml`
above). Like `base` (a previous run envelope) and `transcript` (an
already-produced ASR artifact), the report purely renders whatever result
it's handed.

```python
from hotato import assert_ as A, report

env = A.run_assertions_from_file("assertions.yaml", ctx)
html, _ = report.build_report_html(suite="barge-in", assertions=env)
```

When present, it adds one "Assertions" section:

- **The headline.** Two counts side by side, each scored on its own:
  `N deterministic pass / M fail  K judge-scored (advisory)`.
- **Deterministic** (audio / timing / transcript / trace derived): one
  PER-DIMENSION TYPED card per result -- a kind tag, the `deterministic` flag,
  the PASS/FAIL/INCONCLUSIVE chip, and that result's kind-specific fields (a
  `pii` card's hit detail and redacted transcript, a `policy` card's matched
  rules and pack name, a `tool_call` card's grounding span ids, an `outcome`
  card's met/of fraction).
- **Model-assisted (advisory, quarantined)**: stays empty here by design,
  with a note pointing at the model-judged rubric lane
  ([RUBRIC.md](RUBRIC.md)) where that scoring runs.

`assertions=None` (the default) is byte-identical to a report built before this
parameter existed.

## `inconclusive_policy`: making missing input gate CI

By default an `INCONCLUSIVE` result -- a check whose required input was
absent -- leaves the exit code unaffected, the right default for an
exploratory run. `inconclusive_policy` lets a suite gate on that instead, so a
transcript or trace that never arrived fails loudly instead of leaving the
suite silently green:

| value | how `INCONCLUSIVE` gates | `exit_code` |
| --- | --- | --- |
| `report` (default) | reports missing input and leaves the gate unchanged | `1` if any `FAIL`, else `0` |
| `fail` | gates exactly like a `FAIL` | `1` if any `FAIL` **or** `INCONCLUSIVE`, else `0` |
| `refuse` | refuses to return a verdict at all | `2` if any `INCONCLUSIVE`, else `1` if any `FAIL`, else `0` |

**`refuse` precedence.** Under `refuse`, an `INCONCLUSIVE` result exits `2`
*even if another assertion also `FAIL`ed* -- the exit-2 refusal takes precedence
over the FAIL. A run that cannot fully see its inputs withholds its verdict.

The **default is `report`**, so a suite that sets nothing gates exactly as
before this field existed -- fully backward-compatible. **CI and compliance
suites should set `fail` or `refuse`**, so a missing transcript or trace fails
loudly.

Set it as an optional top-level key:

```yaml
version: 1
inconclusive_policy: fail   # or: refuse | report (default)
assertions:
  - id: disclosure
    kind: phrase
    regex: "recorded for quality"
    role: agent
```

or override the document's key from the CLI (the flag wins):

```bash
hotato assert run --assertions assertions.yaml --transcript call.json \
    --inconclusive-policy refuse
```

or from Python (an explicit argument overrides the document's key; absent both,
`report` applies):

```python
env = A.run_assertions_from_file("assertions.yaml", ctx, inconclusive_policy="fail")
```

A bad value (anything but `report`/`fail`/`refuse`) -- in the document or
passed explicitly -- is a usage error (`ValueError`, exit `2`), raised during
validation before any assertion runs. The envelope always carries the applied
`inconclusive_policy`, stated with the counts in `summary.note`.

## Mapping to a CI gate

Same exit-code convention as every hotato command:

| Exit | Meaning |
| ---: | --- |
| `0` | every assertion passed (under `report`, an `INCONCLUSIVE` reports missing input and leaves the exit at `0`; under `fail` it gates like `FAIL`; under `refuse` it exits `2`) |
| `1` | at least one deterministic status is `FAIL` (or, under `fail`, `INCONCLUSIVE`) |
| `2` | a refusal under `refuse` (an `INCONCLUSIVE`, taking precedence over a `FAIL`), or a malformed file / bad input, raised before any assertion runs |

```bash
python3 - <<'PY'
from hotato import assert_ as A
import sys

ctx = A.build_context(transcript_path="transcript.json", trace_path="voice_trace.jsonl")
env = A.run_assertions_from_file("assertions.yaml", ctx)
print(env["summary"]["note"])
sys.exit(env["exit_code"])
PY
```

A gate on `assert.v1` and a gate on the timing scorer's `exit_code`
(`hotato run` / `hotato verify`, [`docs/CI.md`](CI.md)) are two different,
composable guarantees: one gates turn-taking timing, the other gates
transcript/trace content. Run both; neither exit code substitutes for the
other's.


================================================================================
FILE: docs/CONVERSATION-TEST.md
================================================================================

# Conversation tests (`hotato test run`): one file, one call, a per-dimension scorecard

A **conversation test** (`hotato.conversation-test.v1`) is one file defining
one testable conversation: agent, simulated caller, environment, two
SEPARATE assertion lanes, and an explicit success condition. `hotato test
run` evaluates a supplied call against it and produces a **conversation
artifact** (`hotato.conversation.v1`, evidence bound by sha256) plus a
**per-dimension scorecard**. Success is a boolean
over a small closed vocabulary of named conditions; every dimension counts
on its own, including in `--format json`.

This is the single documented end-to-end conversation-QA workflow:

```
hotato scenario init   ->   hotato test run   ->   hotato conversation verify
   (author a file)          (evaluate a call)        (digest-check the artifact)
```

## 1. Author a conversation test

```bash
hotato scenario init refund-flow --agent support-v3 --out refund.yaml
```

The starter carries two lanes, tagged checks across the five report
dimensions, and a boolean `success`:

```yaml
kind: hotato.conversation-test
version: 1
id: refund-flow
agent: support-v3
caller:
  persona: a customer whose order arrived damaged and wants a refund
  goal: get a refund for order A-1001
  facts: {order_id: A-1001}
assertions:
  # DETERMINISTIC lane: pure, offline, no model. Each result is TAGGED with one
  # of the five report dimensions (a grouping key, never a weight).
  deterministic:
    - id: disclosure-said
      kind: phrase
      regex: "recorded for quality"
      role: agent
      dimension: policy
    - id: refund-tool-called
      kind: tool_call
      name: issue_refund
      dimension: outcome
    - id: lookup-then-refund
      kind: sequence
      steps: [{tool: lookup_order}, {tool: issue_refund}]
      dimension: conversation
    - id: refund-latency
      kind: latency
      tool: issue_refund
      max_ms: 1500
      dimension: speech
  # RUBRIC (model-judged) lane lands in Phase 3. In Phase 1 each is INCONCLUSIVE
  # (no model runs); it never counts toward the deterministic result or exit code.
  rubric:
    - id: was-empathetic
      kind: judge_rubric
      dimension: conversation
# inconclusive_policy: fail  # CI/compliance suites should set fail or refuse
success:
  # Success is the conjunction of these named conditions -- never a score.
  required: [all_deterministic_assertions_pass, no_rubric_failure]
  report_dimensions: [outcome, policy, conversation, speech, reliability]
```

Validate one file or a whole directory (exit 2 on any malformed file):

```bash
hotato scenario validate refund.yaml
hotato scenario validate ./scenarios --format json
```

The closed vocabularies are enforced structurally
(`hotato.conversation_test`): `success.required` is drawn from
`all_deterministic_assertions_pass`, `no_deterministic_fail`,
`no_rubric_failure`, `no_inconclusive`; `dimension` is one of `outcome`,
`policy`, `conversation`, `speech`, `reliability`; an `overall_score` key
anywhere is rejected.

## 2. Evaluate a call

`hotato test run` takes the file plus whatever evidence you have: a scored
recording (`--audio`), an ingested trace (`--trace`, see
[`docs/TRACE.md`](TRACE.md)), a transcript (`--transcript`), a post-call
state sandbox (`--state`, Authority 2). Each supplied piece feeds the
assertions; each absent one leaves the checks that need it `INCONCLUSIVE`.

```bash
hotato test run refund.yaml --agent support-v3 \
    --audio call.wav \
    --trace voice_trace.jsonl \
    --transcript call.transcript.json \
    --out ./conv-artifact --format html
```

Flow: load and validate the file -> build the evaluation context from the
supplied transcript / trace / state / (scored) timing -> evaluate the
DETERMINISTIC lane (rubric quarantined -> `INCONCLUSIVE`) -> evaluate
`success.required` over the results -> bind evidence into a
`hotato.conversation.v1` artifact -> render the unified report +
per-dimension scorecard. Summary printed to stdout:

```
hotato test run: refund-flow (agent support-v3) -- exit_code=0
inconclusive_policy: report
success: PASS  (required: all_deterministic_assertions_pass, no_rubric_failure)
  [ok] all_deterministic_assertions_pass
  [ok] no_rubric_failure
per-dimension (grouped view; never blended):
  outcome       1 pass / 0 fail / 0 inconclusive
  policy        1 pass / 0 fail / 0 inconclusive
  conversation  1 pass / 0 fail / 0 inconclusive
  speech        1 pass / 0 fail / 0 inconclusive
  reliability   0 pass / 0 fail / 0 inconclusive
reliability over 1 repeated run(s) of the deterministic lane on the same supplied real recording; pass^k == pass@1 because the deterministic replay is byte-identical (zero run-to-run variance)
rubric lane: 1 assertion(s) INCONCLUSIVE (quarantined, Phase 3 -- no model ran)
deterministic: 4 pass, 0 fail, 0 inconclusive
judge: 0 pass, 0 fail (no judge/rubric kind is built in this release)
```

`--format`:

| format | stdout | writes into `--out` |
| --- | --- | --- |
| `text` (default) | the per-dimension summary | the conversation artifact |
| `html` / `md` | the summary (+ where files landed) | the artifact **and** `report.{html,md}` (needs `--audio`) |
| `json` | the full machine result | the conversation artifact |

### Exit code

The exit code honors the file's `inconclusive_policy` exactly as
`assert run` does, rising to non-zero when a `success.required` condition
fails:

| `inconclusive_policy` | an `INCONCLUSIVE` (missing-input) result | a `FAIL` |
| --- | --- | --- |
| `report` (default) | does not gate (exit 0) | exit 1 |
| `fail` | gates like a FAIL (exit 1) | exit 1 |
| `refuse` | withholds the verdict (exit 2, precedence) | exit 1 |

A `success.required` failure makes an otherwise-passing run non-zero; a
refuse (exit 2) is never downgraded.

### Reliability and repetitions

`--repetitions N` runs the deterministic lane N times and reports per-run
results, run count, and a reliability aggregate: **pass@1** (single-run
pass rate), **pass@k** (>=1 of k passed), **pass^k** (all k passed), plus a
Wilson 95% CI. Every run scores the same recording, so the deterministic
lane has zero variance and `pass^k == pass@1`. With `N > 1` the aggregate
feeds the report's Reliability dimension (`--format html/md`); with none,
it shows the empty-state ("not measured: no repeated runs in this
report"). pass^k stays its own number, separate from every other
dimension and from `overall_score`.

## 3. Verify the artifact

The conversation artifact directory binds each child by sha256:

```
conv-artifact/
  conversation.json      # the manifest (origin real|simulated + bound digests)
  audio/call.wav         # the recording (single dual-channel form)
  transcript.json
  trace.jsonl
  timing.json            # the scored envelope
  assertions.json        # the evaluated assert.v1 envelope
  report.html            # the unified report + per-dimension scorecard
```

`hotato conversation verify` re-hashes every bound child and REFUSES
(exit 2) on any digest mismatch or missing file:

```bash
hotato conversation verify ./conv-artifact
# conversation refund-flow: VERIFIED
#   verified: assertions, audio, timing, trace, transcript
#   all 5 bound artifact(s) re-hashed to their recorded digest
```

`origin.kind` is `real` by default (a supplied recording is evaluated
as-is) and `simulated` only when the test file carries a `simulator`
block, which must declare its `model_id` / `scenario_id` / `seed`;
synthetic and live origins are never conflated.

## The deterministic/judge split (structural)

* **Each dimension scored on its own.** Success is a boolean conjunction
  of named conditions; the scorecard groups results by dimension, each
  with its own counts.
* **Two separate lanes.** Deterministic checks (regex / checksum / span /
  state lookup, no model) run; the model-judged rubric lane stays
  quarantined until Phase 3: `INCONCLUSIVE`, out of the deterministic
  summary and exit code.
* **Grounded in authority.** `tool_result` / `tool_error` (Authority 1)
  read the ingested trace spans; `state` / `state_change` (Authority 2)
  query a post-call state adapter; the trace and the adapter are the
  evidence a tool ran or a state changed, deterministic and model-free.
* **Missing input is `INCONCLUSIVE`,** never a guessed pass or fail.

## The bundled end-to-end example

`tests/data/conversation/` ships one worked call: a conversation-test file,
a transcript, and a `voice_trace.v1` trace, evaluated against the bundled
recording `01-hard-interruption.example.wav`.
`tests/test_test_run_cli.py`
(`test_bundled_call_one_file_end_to_end_scorecard_and_artifact`) drives it:
one call evaluated for outcome / policy / timing / transcript-facts /
tool-behaviour from one file, producing an artifact that `conversation
verify` passes and a five-dimension scorecard.

## See also

* [`docs/ASSERTIONS.md`](ASSERTIONS.md): deterministic assertion kinds.
* [`docs/TRACE.md`](TRACE.md): ingesting a `voice_trace.v1` trace.
* [`docs/REPORTS.md`](REPORTS.md): the unified report and scorecard.


================================================================================
FILE: docs/RUBRIC.md
================================================================================

# Rubric evaluation: the model-judged lane (`rubric.v1`)

`hotato rubric` scores a call against a **user-authored rubric** with a
pinned **local** model, structurally separate from the deterministic
`assert.v1` wall: every result carries `deterministic: false` with full
provenance, on its own report shelf, apart from any overall number.

- `assert.v1`'s kinds stay `deterministic: {const: true}` forever. A model
  verdict lives in its own `rubric.v1` lane.
- The default judge is a **local Ollama** model (`http://localhost:11434`),
  reached with only the stdlib. Zero egress; a hosted judge is opt-in.
- **Advisory by default**: deterministic checks set the gate, and a
  rubric FAIL is reported while the model's read stays advisory. `--gate`
  (or `--gate-judge` on `test run`) opts into gating: a rubric FAIL exits 1,
  and a judge that could not run -- backend down, or an empty/unparseable
  response even after the repair retry -- is an `ERROR` that exits 2 with no
  FAIL beside it, so "fix the agent" and "fix the judge" stay separable by
  exit code alone.
- Missing or insufficient evidence resolves to `INCONCLUSIVE`, always
  labeled plainly -- as does a well-formed `inconclusive` verdict the model
  returned. A `human_rubric` stays `INCONCLUSIVE` and human-required, by
  design.

## The rubric object

```yaml
version: 1
rubrics:
  - id: acknowledged-frustration
    kind: judge_rubric            # or human_rubric (human-scored only)
    dimension: conversation       # optional report-dimension tag
    criterion: "Did the agent acknowledge the caller's frustration before proposing a fix?"
    evidence: [transcript]        # transcript and/or tool_trace; absent -> INCONCLUSIVE
    examples: {pass: "...", fail: "..."}      # optional
    evaluation:
      model: qwen2.5vl:3b         # optional; else --judge-model / the default
      repetitions: 1              # N model calls, aggregated
      aggregation: unanimous_or_inconclusive
      confidence_required: 0.85
    review:
      human_required_on: [fail, disagreement, confidence_below_threshold]
```

## Run it

```bash
# advisory (exit 0 regardless of verdicts)
hotato rubric run --rubrics rubrics.yaml --transcript call.json --trace trace.jsonl

# opt into CI gating: a rubric FAIL exits 1; a judge ERROR with no FAIL exits 2
hotato rubric run --rubrics rubrics.yaml --transcript call.json --gate

# re-query the model and DIFF against the cached verdict (surface drift)
hotato rubric run --rubrics rubrics.yaml --transcript call.json --no-cache
```

Inside a conversation-test, author the rubric in the `assertions.rubric`
lane; `hotato test run` scores it inline (advisory; `--gate-judge` to gate),
and the unified report shows a populated **Model-assisted (advisory)**
shelf beside the deterministic one.

## Result provenance (`rubric.v1`)

Every result records: the pinned `model` + its content `model_digest`,
`provider`, `prompt_id` / `prompt_version` / `prompt_sha256`,
`temperature: 0`, `input_sha256`, `cache_key`, `cached`, raw
`votes`, `disagreement`, `confidence`, and `citations` to the exact
transcript turns / trace events; the `rubric.v1` schema rejects an
`overall_score` key.

## Reproducibility (stated precisely)

**What's guaranteed is replay.** Every verdict is content-addressed by
`sha256(provider:model + prompt_sha256 + input_sha256)` and cached, so a
cache hit reproduces the same `verdict_sha256` every time. A fresh model
call is a separate question,
checked explicitly with `--no-cache`, which re-queries the model and
**diffs** the result against the cached verdict, surfacing any drift.
`--sign` optionally signs a cached verdict as a "judge-record" (Ed25519
`human` tier via the `[sign]` extra, else HMAC `human-shared`), keeping a
stored verdict provably unmutated.

## Egress

The default local judge stays on the box. A **hosted** judge
(`--judge-provider hosted --judge-endpoint URL`) or a **non-local**
`--judge-endpoint` sends the transcript off-box and requires
`--judge-egress-opt-in` to proceed (exit 2 without it). See
[`docs/EGRESS.md`](EGRESS.md) and [`docs/THREAT-MODEL.md`](THREAT-MODEL.md).

## Calibration: measured credibility

```bash
hotato rubric calibrate --labeled ./labeled --out agreement.json
```

Each `*.json` item is `{rubric, transcript, trace?, label: pass|fail|inconclusive,
split?: train|held_out}`. Human labels are **mandatory**: humans author
the labels; the model is scored *against* them. The command computes
**agreement** (held-out items where the model verdict equals the human
label) and **selective accuracy** (agreement restricted to items where the
model committed to a verdict), writing a reproducible artifact of raw
counts, method, and provenance. Re-running on the same corpus with the same
model reproduces the same split and verdicts: the numbers travel with
their method and provenance built in.


================================================================================
FILE: docs/SIMULATE.md
================================================================================

# `hotato simulate`: a scenario to a deterministic, labelled conversation

`hotato simulate` renders a **scenario** (`hotato.scenario.v1`) through a
deterministic scripted caller into one or more **conversation artifacts**
(`hotato.conversation.v1`), each labelled `origin=simulated`. Fully
offline, scripted-caller only -- the deterministic input side of a
simulation: the ground truth a caller holds, the caller's scripted turns,
the environment, and an optional variation matrix.

Use it to build the regression foundation you want before any generative
caller: a fixed `(scenario, seed)` reproduces the transcript byte-for-byte
(content-hashed), so a run is a stable, reusable fixture.

## Quickstart (zero-install with uvx)

```bash
# run zero-install with uvx (or: pipx install hotato)
uvx hotato --help

# 1. Write a minimal scenario simulate accepts as-is:
hotato simulate --init demo.scenario.json

# 2. Render it into a labelled origin=simulated conversation:
hotato simulate demo.scenario.json --out ./sim

# 3. (optional) digest-verify the produced artifact:
hotato conversation verify ./sim
```

Prefer zero files? The package ships a minimal scenario you can run
directly, no file on disk:

```bash
hotato simulate --example --out ./sim
```

Step 2 prints:

```
wrote 1 simulated artifact(s) under ./sim/
hotato simulate: demo -- 1 run(s), origin=simulated (never real)
  run 1: seed=0 77f1eb9e6994 sim=ok  -> ./sim
reliability: pass@1=1.000 pass@k=1.000 pass^k=1.000 (n=1)
```

`./sim` holds a `conversation.json` plus its bound transcript and
`voice_trace.jsonl`, all `origin.kind=simulated`.

## The curated persona pack (seeded, byte-reproducible)

The package ships a curated persona/scenario library, `hotato-voice-personas`,
covering the common voice-agent test cases: a missed barge-in, a backchannel
that is not a floor-take, long silence / dead air, a mid-utterance pause before
an over-eager reply, a caller talking over the agent, and pacing variations (a
fast interrupter, a slow speaker). Each entry is a real `hotato.scenario.v1`
the deterministic caller renders, and each pins a fixed `seed`, so a fixed
`(scenario, seed)` renders **byte-identical every run** across machines and CI:
`conversation.json`, `transcript.json`, and `trace.jsonl` all match, because the
manifest `created_at` defaults to a reproducible instant (SOURCE_DATE_EPOCH-style,
never the wall clock). Pass `--created-at` (or set `$SOURCE_DATE_EPOCH`) to pin a
real timestamp when you want one. Run a common test case by name, no file
authoring:

```bash
# list the pack (add --format json for the machine-readable index):
hotato simulate --list

# run one pack scenario by name into a labelled origin=simulated conversation:
hotato simulate barge-in-missed --out ./sim
```

`hotato simulate --list` prints:

```
hotato simulate pack: hotato-voice-personas -- 7 curated scenario(s)
each renders a deterministic origin=simulated caller (never real); a fixed (scenario, seed) is byte-identical
  backchannel-not-floor-take  [should_hold] seed=202  Backchannel that is not a floor-take
      The caller drops short acknowledgements (mm-hmm, right) while listening. Run it to see whether your agent holds the floor instead of treating a backchannel as an interruption.
  barge-in-missed             [should_yield] seed=101  Missed barge-in (hard interruption)
      The caller takes the floor mid-sentence at a fixed offset. Run it to see whether your agent stops its own speech and listens.
  ...
run one with: hotato simulate <name> --out ./sim
```

The name resolves to a scenario bundled with the package; a scenario file on
disk always wins, so a local file is never shadowed by a pack name. A pack
scenario is deterministic INPUT that renders `origin=simulated`, labelled as
such; it is scripted-caller stimulus, not production audio, and it scores
nothing on its own -- scoring is the separate assert layer's job over the
produced artifact. The pack lives in
[`src/hotato/data/simulate/pack/`](../src/hotato/data/simulate/pack/) with its
own `README.md` and `manifest.json`.

> **Two different `scenario` concepts, one name.** `hotato simulate`
> consumes a `hotato.scenario.v1` doc -- author one with `hotato simulate
> --init`. `hotato scenario init` writes a separate file, a
> `hotato.conversation-test.v1`, that `hotato test run` consumes -- see
> [CONVERSATION-TEST.md](CONVERSATION-TEST.md). Feed a conversation-test
> file to `simulate` and you get an actionable error pointing back to
> `--init`.

## What `--init` writes

`hotato simulate --init demo.scenario.json` writes a minimal, valid
`hotato.scenario.v1` doc -- a starter you edit for your own agent, shaping
the caller turns to match your call. The scenario id derives from the
filename stem.

```json
{
  "kind": "hotato.scenario",
  "version": 1,
  "id": "demo",
  "goal": { "type": "get_refund", "target": "order A-1001" },
  "facts": { "order_id": "A-1001" },
  "caller": {
    "script": [
      { "say": "Hi, my order A-1001 arrived damaged and I would like a refund." },
      { "say": "Yes, please refund it to my card." }
    ],
    "behavior": { "backchannels": { "probability": 0.0 } }
  },
  "environment": { "locale": "en-US", "route": "phone" },
  "seed": 0
}
```

The caller's script holds only the caller's own turns -- a `say` is the
caller speaking. The schema enforces this structurally: there's no field
for the agent's words, so a scenario stays scoped to the caller's side by
construction.

Required fields: `kind`, `version`, `id`, `goal` (`type` + `target`), and
`caller.script` (at least one `say`). The full schema, including
`variation_matrix` and the optional deterministic `agent_mock`, is
[`schema/scenario.v1.json`](../src/hotato/schema/scenario.v1.json).

## The invariants, enforced structurally

- **`origin=simulated` on every produced conversation**, kept apart from
  real calls. `write_artifact` only writes artifacts whose origin is
  simulated.
- **A bad rendering is `SIMULATOR_INVALID`, kept distinct from an agent
  PASS/FAIL.** The simulator only decides whether the produced
  conversation faithfully renders its scenario; scoring is the separate
  assert layer's job.
- **A seeded replay is byte-identical** -- there's no model in this path.
  A fixed `(scenario, seed)` produces the same transcript bytes every
  time; different seeds differ only where the scenario allows it
  (probabilistic backchannels).
- **Each dimension scores on its own lane**, enforced in both the schema
  (which rejects an `overall_score` key) and the code path.

## Reliability: pass@1 / pass@k / pass^k

`simulate` reports Reliability as its own dimension, scored on its own
lane:

- `pass@1` -- the fraction of runs that rendered faithfully.
- `pass@k` -- at least one pass across `k` runs.
- `pass^k` -- all `k` pass.

For the scripted deterministic caller, `pass^k == pass@1`: a seeded replay
is byte-identical, so every run has the same outcome. Variance shows up
only where the scenario introduces it. `--repetitions N` expands the
variation matrix so Reliability is measured over `N` runs.

## Simulate many scenarios in parallel (`--matrix`)

```bash
# expand the scenario's FULL variation matrix (locale x speaking_rate x noise x
# behavior x repetitions), render + validate each in a bounded pool:
hotato simulate --matrix demo.scenario.json --out ./matrix

# score each produced conversation against a conversation-test's DETERMINISTIC
# assertions; SIMULATOR_INVALID runs are bucketed separately, kept distinct
# from a PASS/FAIL:
hotato simulate --matrix demo.scenario.json \
    --conversation-test refund.test.yaml --parallel 8 --format json
```

The summary stays byte-identical no matter the worker count. Every result
is attributed to its own variation cell, each scored on its own lane.

## Drive the scripted caller at YOUR chat agent (`--chat URL`)

`--chat` drives the same scripted caller turn plan against your own chat
agent over HTTP and writes a timestamped transcript
[`hotato investigate --transcript`](INVESTIGATE.md) scores:

```bash
hotato simulate demo.scenario.json --chat http://127.0.0.1:8080/chat
hotato investigate --transcript .hotato/chat-transcript.json
```

The whole wire contract is one POST per scripted turn:

```
request:   POST <URL>   {"conversation_id": "<id>", "turn_index": 0, "text": "<caller turn>"}
response:  200          {"text": "<agent reply>"}
```

Extra response keys are ignored; a non-200, a redirect, a non-JSON body, or
a missing `text` is a loud exit-2 error naming the contract. Local by
default: a host off `localhost`/`127.0.0.1` is refused before any request
unless you pass `--egress-opt-in` (the same explicit gate the hosted
diarizer and hosted judge carry).

What lands in the transcript, each labelled for what it is:

- agent replies **verbatim**, with per-turn reply latency **measured** as the
  HTTP round trip -- that measured latency is exactly the response gap
  `investigate --transcript` scores;
- caller/agent turn spans from the scenario's deterministic pacing model
  (the same word-count/speaking-rate constants the offline renderer uses),
  so the caller side derives from `(scenario, seed)`, never a wall clock;
- `origin.kind = "simulated"` provenance: the caller side is the scripted
  simulator's, and the agent text is your agent's own replies
  (`agent_replies: "live-chat-http"`).

`--out DIR` names the transcript's directory (default `.hotato/`, written
as `chat-transcript.json`).

## Dynamic variables and branches (matrix expansion)

Two optional scenario blocks widen a single scenario into a whole family of
deterministic cells, expanded through the same `--matrix` runner. Both are
gated and additive: a scenario with neither expands byte-for-byte as before,
and each produced cell is still `origin=simulated`, seeded from a stable
hash, and scored on its own lane.

### `variables`: templated caller lines

`variables` maps a name to a list of values. Each `{name}` reference in a
caller `say` line is substituted with a value, and every declared variable
is a cross-product axis of the matrix:

```json
{
  "kind": "hotato.scenario",
  "version": 1,
  "id": "refund-cities",
  "goal": { "type": "get_refund", "target": "an order" },
  "caller": {
    "script": [
      { "say": "Hi, I'm calling from {city} about order {order_id}." },
      { "say": "Please refund {order_id} to my card." }
    ],
    "behavior": { "backchannels": { "probability": 0.0 } }
  },
  "variables": {
    "city": ["Austin", "Denver"],
    "order_id": ["A-1001", "A-2002"]
  }
}
```

`hotato simulate --matrix refund-cities.scenario.json` expands to `2 x 2 = 4`
cells (crossed with any `variation_matrix` dimensions), each with the values
substituted in and a distinct, stable derived seed. A `{name}` reference with
no matching entry under `variables` is refused up front (exit 2).

### `branches`: enumerate every path

`branches` declares a caller decision tree (or DAG). `root` names the entry
node; each node under `nodes` carries its caller line(s) as `say` and its
successor node names as `next` (a node with no `next` is a leaf). The runner
enumerates **every root-to-leaf path** deterministically and appends each
path's lines to the base caller script, one cell per path, each stamped with
a `path` label:

```json
{
  "kind": "hotato.scenario",
  "version": 1,
  "id": "refund-tree",
  "goal": { "type": "get_refund", "target": "order A-1001" },
  "facts": { "order_id": "A-1001" },
  "caller": {
    "script": [ { "say": "Hi, order A-1001 arrived damaged." } ],
    "behavior": { "backchannels": { "probability": 0.0 } }
  },
  "branches": {
    "root": "ask",
    "nodes": {
      "ask": { "say": "Can I get a refund?", "next": ["insist", "accept"] },
      "insist": { "say": ["No, I want the money back.", "Refund it, please."] },
      "accept": { "say": "A store credit is fine." }
    }
  }
}
```

Here two paths (`ask>insist`, `ask>accept`) expand to two cells. A `next`
that names an undefined node (an unknown node) or any cycle (a `next` that
points back to an ancestor on the same path) is refused up front (exit 2);
root-to-leaf paths must be finite. A shared child reached by two distinct
parents (a diamond) is not a cycle -- it yields one path per route.

`variables` and `branches` compose: with both present, the runner produces
one cell per `(variable binding, root-to-leaf path, variation-matrix cell)`,
and variable substitution applies to the branch lines too.

## Exit codes

| Exit | Meaning |
|---|---|
| `0` | every produced conversation is `origin=simulated`, validated as a faithful rendering (and, under `--matrix --conversation-test`, every scored aggregate passed); with `--chat`, every scripted turn was driven and the transcript written |
| `1` | at least one simulation was `SIMULATOR_INVALID` -- a broken fixture, distinct from an agent PASS/FAIL -- or, under `--matrix --conversation-test`, a scored aggregate FAILed |
| `2` | usage error / unusable input: a malformed or unreadable scenario/conversation-test file -- including an unbound `{name}` template, an unknown branch node, or a branch cycle -- (or, under `--matrix --conversation-test` with `inconclusive_policy refuse`, a withheld verdict); a non-local `--chat` URL without `--egress-opt-in`, an unreachable chat agent, or a reply off the `--chat` contract |

## Related

- [CONVERSATION-TEST.md](CONVERSATION-TEST.md) -- `hotato scenario init` /
  `hotato test run`: score a real call against a conversation-test.
- [`schema/scenario.v1.json`](../src/hotato/schema/scenario.v1.json) -- the
  full scenario schema.


================================================================================
FILE: docs/SUITE-RUN.md
================================================================================

# `hotato suite run`: execute a suite of conversation-tests

`hotato suite run` loads a `hotato.suite` file (a named set of
conversation-test refs, each scored on its own), resolves each ref relative
to the suite file, executes every test, and emits a per-dimension +
reliability report, also recording the run into the fleet registry so
`hotato serve` can browse it.

A conversation-test is one file defining one testable conversation; see
[CONVERSATION-TEST.md](CONVERSATION-TEST.md). A suite (JSON or YAML) groups
those files under a `suite_id`, a `purpose`, an `inconclusive_policy`, and a
`required_for_release` flag.

## How a test runs

- A test that declares a `scenario` runs **offline** through the
  deterministic scripted-caller simulator: its variation matrix expands
  into concrete runs, each rendered, validated, and scored against the
  test's deterministic lane. Authority-1 tool spans and the Authority-2
  mock-state sandbox come from the scenario's `agent_mock`, fully simulated
  and offline (Authority 2 is grounded state; see
  [STATE-ADAPTERS.md](STATE-ADAPTERS.md)).
- A test with no scenario is evaluated once against an empty context: every
  input-dependent check reports INCONCLUSIVE, never a guessed pass/fail.

## Every dimension scored on its own

The report carries per-dimension counts (`outcome`, `policy`,
`conversation`, `speech`, `reliability`) plus a reliability aggregate:
`pass@1`, `pass@k`, `pass^k`. Per-dimension counts group results across
tests; reliability stands as its own dimension over every valid simulated
run. Each dimension keeps its own count and verdict, exposed as-is,
including in `--format json`; the schemas reject an `overall_score` key.

A `SIMULATOR_INVALID` run flags a broken fixture, bucketed separately and
reported by test id and run id, kept out of every dimension and the
reliability aggregate.

## The suite's policy is authoritative

The suite's `inconclusive_policy` is the effective policy for every test it
names, regardless of a test's own policy. `inconclusive_policy: fail` makes
an INCONCLUSIVE (absent required input) FAIL the gate; `refuse` withholds
the verdict (exit `2`, precedence).

The suite exit code is the **worst** test outcome under that policy
(`refuse` > `fail` > `pass`), raised to at least `1` by any
`SIMULATOR_INVALID` run.

## Run it

```bash
hotato suite run examples/reference-agent/suite.json \
    --agent reference-agent-v1 \
    --release reference-agent-v1 \
    --out ./suite-out
```

`--agent ID` (required) is recorded on every conversation and the release.
`--release ID` names the release runs are recorded under (default
`<agent>@<suite_id>`) -- keep it stable per release for
[`hotato release compare`](RELEASE-COMPARE.md). `--out DIR`
writes the per-test simulated conversation artifacts plus
`suite-report.md`, `suite-report.html`, and `suite-run.json`. `--parallel N`
caps worker threads per scenario's matrix; the worker count never changes
the byte-identical result. `--junit PATH` also writes a JUnit XML report
for a CI test widget (one `<testsuite>` per dimension, one `<testcase>`
per test; a FAILed dimension is a `<failure>` with the measured reason, an
INCONCLUSIVE dimension or a `SIMULATOR_INVALID` run is an `<error>`, never
a silent pass); see [CI.md](CI.md).

Runs are recorded into the fleet registry (`--registry`, default
`~/.hotato/fleet`; `--workspace`/`-w`, default `default`) as Release / Suite
/ Scenario / Run / Conversation / Evaluation rows, so `hotato serve -w
<workspace>` renders them (see [WORKSPACE.md](WORKSPACE.md));
`--no-registry` reports only.

## Worked example: the reference agent

`examples/reference-agent/` ships a full suite: 25 realistic voice-agent
jobs x 5 caller behaviours x 3 audio environments = 375 offline simulated
runs, scored across the outcome / policy / conversation / speech
dimensions. Its `suite.json` has `inconclusive_policy: fail` and
`required_for_release: true`.

One command regenerates the files, runs the 375-run suite offline, and
records a browsable workspace:

```bash
cd examples/reference-agent
make reference
# then browse:
hotato serve --workspace reference --registry ./.workspace
```

The summary printed to stdout (counts here are schematic):

```
hotato suite run: reference-agent-suite (...) -- agent reference-agent-v1, release reference-agent-v1 -- exit_code=0
inconclusive_policy: fail  required_for_release: True
tests: 25  (25 pass, 0 fail, 0 refuse)
runs: 375 (375 valid, 0 SIMULATOR_INVALID -- broken fixtures, never an agent PASS/FAIL), origin=simulated
per-dimension (grouped across tests; never blended):
  outcome       ... pass / ... fail / ... inconclusive
  policy        ... pass / ... fail / ... inconclusive
  conversation  ... pass / ... fail / ... inconclusive
  speech        ... pass / ... fail / ... inconclusive
  reliability   ... pass / ... fail / ... inconclusive
reliability [origin=simulated]: pass@1=... pass@k=... pass^k=... (n=...)
per-test:
  [pass   ] refund-damaged-order            runs=15  ...
  ...
```

Origin is always labelled `simulated`, keeping a simulator's replay
reliability distinct from production reliability.

## Exit codes

- **`0`**: every test in the suite passed under its `inconclusive_policy`
  (no run `SIMULATOR_INVALID`).
- **`1`**: at least one test FAILed (a `success.required` condition
  failed, a deterministic assertion FAILed, or, under
  `inconclusive_policy fail`, an INCONCLUSIVE gated), or a run was
  `SIMULATOR_INVALID`.
- **`2`**: under `inconclusive_policy refuse` a scored INCONCLUSIVE
  withheld the verdict (precedence over a FAIL); or a usage error /
  unusable input: a malformed suite / conversation-test / scenario file, or
  an unresolvable test/scenario ref.

## See also

- [CONVERSATION-TEST.md](CONVERSATION-TEST.md): the conversation-test file each suite ref points at.
- [STATE-ADAPTERS.md](STATE-ADAPTERS.md): Authority 2 state grounding for `state` / `state_change` checks.
- [RELEASE-COMPARE.md](RELEASE-COMPARE.md): diff two releases recorded by `suite run`.
- [WORKSPACE.md](WORKSPACE.md): browse the recorded runs with `hotato serve`.
- [SUITES.md](SUITES.md): the `corpus/suites/` scenario-audio suites for the `hotato run` scoring path (a distinct mechanism).


================================================================================
FILE: docs/RELEASE-COMPARE.md
================================================================================

# `hotato release compare`: diff two releases, per dimension

`hotato release compare BASELINE CANDIDATE` reads two releases from the fleet
registry and reports what moved between them -- `BASELINE` first, `CANDIDATE`
second. It reports movement only; gating a release on that movement is a
separate step (a `required_for_release` suite).

Each release is a snapshot recorded by [`hotato suite run`](SUITE-RUN.md).
Run the same suite twice under two stable `--release` ids in the same
workspace and registry, then diff them.

## What it reports

- **Per-dimension counts for each side, plus the delta.** Each dimension
  (`outcome`, `policy`, `conversation`, `speech`, `reliability`) keeps its
  own three counts (pass / fail / inconclusive) and its own delta -- every
  dimension scored on its own lane.
- **New failures** -- a `scenario x dimension` that PASSED on the baseline
  and FAILs on the candidate. **Fixed-since** -- the reverse. Both diff
  **only** where BOTH releases ran the same `scenario x dimension`; a
  scenario only one side ran counts as new coverage, separate from a
  regression.
- **Per-scenario status changes** -- every `scenario x dimension` whose
  status differs between the two, diffed only where both sides have a
  comparable result.

The releases' pinned digests are surfaced, so the reader knows exactly
which two snapshots were compared, every time.

## Empty state

A side with no runs is stated plainly:

- a release id absent from the workspace is reported as unregistered;
- a release that is registered but has **zero runs** is reported as an
  empty state.

When no `scenario x dimension` result is comparable across both releases,
the report says so: new-failures / fixed-since needs a scenario BOTH
releases ran.

## Compare

```bash
hotato release compare reference-agent-v1 reference-agent-rc2
```

`--workspace`/`-w ID` selects the fleet-registry workspace (default `default`);
`--registry PATH` sets the registry home (default `~/.hotato/fleet`);
`--format {text,json}` picks the output.

## Worked example: two reference-agent releases

Record two releases of the reference agent into the same registry and
workspace, then diff them. The reference example writes to
`examples/reference-agent/.workspace`, workspace `reference`:

```bash
cd examples/reference-agent

# record the baseline release
python run_reference.py --release reference-agent-v1

# edit the scenario / agent_mock fixtures (or point at a changed agent),
# regenerate, and record the candidate release
python run_reference.py --generate --release reference-agent-rc2

# diff them (same workspace + registry both runs recorded into)
hotato release compare reference-agent-v1 reference-agent-rc2 \
    --workspace reference --registry ./.workspace
```

Two runs over identical fixtures record byte-identical results, so the diff
shows all-zero deltas and no status changes. Edit the fixtures between the
two runs to see movement.

The text output follows this shape (counts here are illustrative placeholders):

```
hotato release compare: baseline reference-agent-v1 -> candidate reference-agent-rc2  (workspace reference)
baseline runs=375 conv=375 eval=...  candidate runs=375 conv=375 eval=...
per-dimension counts (baseline -> candidate, delta; never blended):
  outcome       ...P/...F/...I  ->  ...P/...F/...I   (dP +.., dF -.., dI +..)
  policy        ...P/...F/...I  ->  ...P/...F/...I   (dP +.., dF -.., dI +..)
  conversation  ...P/...F/...I  ->  ...P/...F/...I   (dP +.., dF -.., dI +..)
  speech        ...P/...F/...I  ->  ...P/...F/...I   (dP +.., dF -.., dI +..)
  reliability   ...P/...F/...I  ->  ...P/...F/...I   (dP +.., dF -.., dI +..)
new failures (PASS on baseline -> FAIL on candidate): N
  - <scenario> / <dimension>
fixed since (FAIL on baseline -> PASS on candidate): N
  + <scenario> / <dimension>
all per-scenario status changes:
  ~ <scenario> / <dimension>: PASS -> FAIL
```

## Exit codes

- **`0`** -- the two releases were compared (per-dimension deltas and
  new-failures / fixed-since printed; a side with no runs is reported as an
  empty state, exit stays `0`).
- **`2`** -- a usage error or an unreadable registry `--registry`.

## See also

- [SUITE-RUN.md](SUITE-RUN.md) -- records the releases this command diffs.
- [CONVERSATION-TEST.md](CONVERSATION-TEST.md) -- the per-dimension scorecard model.
- [WORKSPACE.md](WORKSPACE.md) -- browse the releases and their runs with `hotato serve`.


================================================================================
FILE: docs/WORKSPACE.md
================================================================================

# The team workspace: `hotato serve` and `hotato console`

A self-hosted, local web app for a team's voice-agent conversation QA:
the live call feed (with pin-to-contract on every scored moment), suite
health, failure clusters, failure records, release readiness, plus the
scenario-matrix and conversation-inspector drill-ins.
Stdlib-only (`http.server` + `sqlite3`) -- no framework, no build step,
nothing that phones home. It serves the same fleet registry and evidence
store the CLI writes, reading that data directly instead of passing a
database file around.

```
hotato serve --workspace default
```

On first start it prints where it is listening, the bearer token, and the URL to
open:

```
hotato serve: workspace 'default'
  registry:  /home/you/.hotato/fleet
  listening: http://127.0.0.1:8321
  token:     Ab3xQ_p1…                 (generated, stored 0600 at …/serve/default/token)
  open:      http://127.0.0.1:8321/?token=Ab3xQ_p1…
  audit log: /home/you/.hotato/fleet/serve/default/audit.jsonl   (append-only)
  writes:    one route -- pin-to-contract (POST /calls/<id>/pin, CSRF-fenced, audited); every view reads with SELECTs. No telemetry, no external calls.
```

Open the `open:` URL in a browser; the server sets a session cookie and
redirects to strip the token from the address bar (see [Auth](#auth)).

## Flags

| flag | default | meaning |
|---|---|---|
| `--workspace`, `-w` | `default` | workspace id to serve |
| `--host` | `127.0.0.1` | bind address (see [Binding](#binding-127001-by-default)) |
| `--port` | `8321` | listen port |
| `--registry` | `~/.hotato/fleet` | registry home directory |
| `--production-db` | none | read session manifests and alerts from this separate Hotato production SQLite database in `/health` (mode=ro; see below) |
| `--score-production` | off | with `--production-db`: score completed sessions in the background into a `console.sqlite3` sidecar beside the evidence database (see below) |
| `--rebuild-scores` | off | with `--production-db`: deterministically regenerate the entire `console.sqlite3` sidecar from the evidence database, then exit |
| `--token` | none | supply the bearer token yourself |
| `--token-file` | none | read the bearer token from a file (first line) |

Exit codes: `0` clean shutdown (Ctrl-C); `2` usage error (unusable registry or
token, or the port was unavailable).

## `hotato console`: the call console in one command

```
hotato console --production-db .hotato/production.sqlite3
```

One command, one process, one browser tab: `serve` with the production
evidence database wired (read-only, `mode=ro`), the score-on-arrival worker
on, and the printed URL landing on the live call feed at `/calls`. Every
serve behavior is identical -- loopback bind by default, bearer-token auth on
every request, the append-only audit log, a `?format=json` mirror on every
view -- and `--workspace`, `--host`, `--port`, `--registry`, `--token`,
`--token-file`, and `--no-open` work exactly as they do on `serve`. The same
exit codes apply.

## The views

Every view has a machine mirror at `?format=json` (same auth, same data) so
agents and scripts can drive the workspace without scraping HTML.

The nav reads as one product: **Calls · Suite health · Failure clusters ·
Failure records · Release readiness**; the scenario matrix, one call, one
conversation, and one record are drill-ins.

| View | URL | Shows |
|---|---|---|
| **Calls** | `/calls` | The console feed over scored production calls, with the "contracts protecting this agent" count in its header (see [The call feed](#the-call-feed-calls)). |
| **One call** | `/calls/<id>` | One call's derived score record: per-dimension observations, ranked candidate moments **each with a pin-to-contract form**, the timing waterfall, evidence lanes, and the audio path as recorded. |
| **Suite health** | `/health` | Your CI suite's history from the fleet registry: ingest counts, evaluated coverage, and per-dimension failure rate over time, **separated for real and simulated** conversations. Sparse days/dimensions read *not enough history* rather than a misleading point. No single combined quality score -- each dimension keeps its own number. (`/production` serves the same view for URL compatibility.) |
| **Failure clusters** | `/clusters` | Failed evaluations and assertions grouped by **observable signature** (dimension + assertion kind + reason-class), with counts and drill-through into the inspector -- it groups what was observed; the cause stays yours to determine. |
| **Failure records** | `/records` | The read-only Failure Record viewer over `hotato.failure-record.v1`. |
| **Release readiness** | `/` | Pre-ship home screen: per-release rollup of suites/runs/evaluations -- required-suite completion, scenario/run counts, **failures by dimension** (outcome / policy / conversation / speech / reliability), inconclusive count, real-vs-simulated split, and **new-vs-fixed since the previous release**. Small samples flagged (`low sample, N=3`), never smoothed. |
| **Scenario matrix** | `/scenarios` | Rows are scenarios, columns are the current and previous release, with a per-dimension status and **reliability** (`pass^k` where a scenario has repetitions). Filterable by `agent`, `release`, `suite`, `status`. |
| **Conversation inspector** | `/conversation/<id>` | One conversation: evidence manifest, transcript, trace spans, per-dimension evaluations with rationale and citations (deterministic checks and model-judged/advisory results in **separate lanes**), reviewer decisions. Every digest links to the raw evidence (`/evidence/<digest>`); redacted transcript segments and trace spans render `[redacted]`, in both HTML and JSON. |

### Optional production-evidence bridge

The fleet registry and the production event store have different storage
authorities. Pointing the workspace at both is explicit:

```bash
hotato serve --workspace default \
  --production-db .hotato/production.sqlite3
```

`/health` then adds a separate **Production evidence plane** section with
bounded session manifests, current alerts, event-source identifiers, every
evidence lane's availability and authority, and the required lanes still
missing. The JSON mirror exposes the same projection under
`production_evidence`.

The bridge opens the selected database with SQLite `mode=ro` for each
request. It never constructs the writer-side `ProductionStore`, never selects
the event `payload_json` column, and never imports a production row into the
fleet registry. Production counts therefore stay outside `ingested_total`,
the real/simulated buckets, and release trends. The production schema does not
carry a fleet workspace id, so the UI states `workspace_scope =
not_encoded_by_production_schema` instead of silently assigning those sessions
to the workspace being served.

### Score-on-arrival (`--score-production`)

```bash
hotato serve --workspace default \
  --production-db .hotato/production.sqlite3 --score-production
```

A background worker in the same process polls the evidence database (same
`mode=ro` read-only discipline) for sessions that reached
`COMPLETE`/`QUIESCENT` and scores each one with the deterministic scorer over
the session's recorded two-channel audio (the path named by the
`media.asset.available` event's `data.path`). Bind and auth are unchanged;
the server gains no new routes or write endpoints from this flag.

Each session becomes one durable record in `console.sqlite3` beside the
evidence database:

- **`SCORED`** -- per-dimension observations (candidate counts and worst
  measured magnitude per scan kind, never blended), the ranked candidate
  moments, and one plain-English failure-reason sentence built only from
  measured numbers;
- **`NOT_SCORABLE`** -- the scorer's refusal with its reason (audio lane
  unavailable, no recorded path, a one-channel or unreadable recording);
- **`ERROR`** -- a scorer crash or persist failure on that session, with its
  reason; the worker records it and continues to the next session.

Every record carries the scorer version and a config hash, and every timing
figure derives from evidence event timestamps: per-hop latency rows keep the
reporting event's declared `authority`, turn spans and the end-to-end figure
are labeled `derived:event_timestamps`, and reported turn fields
(`yield_latency_ms`, `overlap_ms`, `duration_ms`) stay in a separate
`reported` block. Sessions are scored one at a time and a record is claimed
only after its sidecar write commits.

The sidecar is derived data -- the evidence database stays the only
authority. `--rebuild-scores` regenerates the whole sidecar from the evidence
database and exits; the same evidence database always rebuilds to identical
content (the one wall-clock column, `created_at`, is excluded from the
canonical comparison). A sidecar written by a different schema version is
refused with that rebuild instruction.

### The call feed (`/calls`)

`/calls` renders the score sidecar joined read-only with the evidence
database's session metadata, newest arrival first. Each row shows the
evidence-clock timestamp beside the arrival stamp (each labeled with its
clock), the evidence-derived duration, the session state, the score state,
the worst dimension with the measured failure-reason sentence, per-call
hop-latency p50/p95 with the declared authorities behind it, and
evidence-lane completeness (missing required lanes named). `SCORED`,
`NOT_SCORABLE` (with its reason), and `ERROR` are all first-class rows: a
refused or crashed session is shown, never hidden and never rendered as OK.

Filters are query params -- `state` (session state), `scorability`
(`SCORED`/`NOT_SCORABLE`/`ERROR`), and a `since`/`until` window (epoch
seconds or RFC3339); a malformed filter or cursor is a 400, never a silently
dropped filter. Pagination is keyset (`cursor`, `limit`) -- every page is one
bounded query, and the trends strip above the table reports call volume, the
score-state split, the candidate-moment share, and per-kind hop-latency
p50/p95 (nearest-rank) over the whole filtered window; hop kinds keep their
own percentiles and state their authorities.

The feed's JSON mirror carries an `ETag` and answers a matching
`If-None-Match` with a body-less 304. The page's small inline script polls
that mirror every few seconds, shows an "updated Ns ago" indicator, and
re-renders the rows only when the ETag changes -- no external code, and with
JavaScript off the page is complete as served (reload for the latest).

The feed header shows **"N contracts protecting this agent"** -- a read-only
`COUNT(*)` over the fleet registry's existing `contracts` table for the serve
workspace, the same table `hotato fleet` registration and the pin route
write through.

`/calls/<id>` is one call: per-dimension observations (candidate counts and
worst measured magnitude per scan kind, each on its own), the ranked
candidate moments with their measured magnitudes and plain-English timing
sentences, the timing waterfall (turn spans with derived durations kept apart
from event-reported values; every hop row with its timestamp, latency, and
declared authority), scorer version + config hash, the session's evidence
lanes, a link to the production evidence plane, and the local audio path
exactly as the evidence recorded it -- the recording stays on this machine.
Where the session has finalized (`COMPLETE`/`DEGRADED`), the view also shows
the exact `hotato production export-regression <id> --out DIR --db FILE`
command that exports it as an offline-verifiable regression candidate.

### Pin-to-contract (`POST /calls/<id>/pin`)

Each top-ranked candidate moment on a SCORED call carries a small form:
choose `expect yield` or `expect hold` and pin. The POST delegates to the
same fleet machinery the CLI drives -- ingest the recorded audio
(content-addressed), re-scan it, then
`fleet contract create --from-candidate` semantics
(`FleetAPI.contract_from_candidate`): the human label, the sealed portable
`.hotato` contract bundle, and the registry registration land in one atomic
step, under agent id `production` in the serve workspace. On success the
page shows the contract id and bundle path (with JavaScript on, inline; with
it off, as a result page), and the feed's contract count moves.

A pin **refuses with its reason -- HTTP 4xx, no artifact** -- exactly where
the CLI would: a NOT_SCORABLE/ERROR call, a candidate reference that does
not exist, a recording that changed on disk since scoring (the re-scan must
reproduce the chosen moment's onset), a stale page (the form binds the score
record's `evidence_sha256`; a rebuilt sidecar refuses), or a trust-preflight
refusal. The mint + label + registration step is atomic, so a refused pin
leaves no contract, no label, and no registry row.

The route is the server's one write action and is fenced three ways: the
same bearer/cookie auth as every view (an unauthenticated POST is 401), the
session cookie's `SameSite=Strict`, and a same-origin check -- a
cookie-authenticated POST must carry an `Origin` (or `Referer`) header
matching the request's own `Host`, so a forged cross-site form is refused
403 before any handler runs; a request presenting the bearer secret itself
needs no origin header. Every attempt, accepted or refused, appends one
line to the audit log.

## Auth

Every request is authenticated against one shared **bearer token**:

- **Browser:** open `/?token=<token>` once; the server mints an in-memory,
  HttpOnly session cookie and redirects to remove the token from the URL.
- **Agent / API / `curl`:** send `Authorization: Bearer <token>`.

The token is compared in constant time (`hmac.compare_digest`). Without
`--token`/`--token-file`, one is generated with `secrets.token_urlsafe` on
first start and stored `0600` at `<registry>/serve/<workspace>/token`, so
a restart keeps the same URL. Sessions live only in memory, never
persisted, never cross-tenant. The session cookie is `HttpOnly` +
`SameSite=Strict`, and the one write route adds a same-origin check on
cookie-authenticated POSTs (see
[Pin-to-contract](#pin-to-contract-post-callsidpin)).

## Audit log

Every request appends one JSONL line to
`<registry>/serve/<workspace>/audit.jsonl` (created `0600`):

```json
{"ts":"2026-07-12T18:04:11Z","who":"Ab3xQ_p1…","method":"GET","path":"/scenarios","query":"status=FAIL","status":200,"remote":"127.0.0.1"}
```

`who` is a token/session **prefix**, never the secret; the `token`
parameter is stripped from the recorded query. The audit log is the only
file the serve layer itself writes; a pin-to-contract POST additionally
writes its contract bundle and registry rows through the fleet machinery,
and each attempt is one audit line.

## Binding 127.0.0.1 by default

The server binds loopback (`127.0.0.1`) unless you pass `--host`. A
non-loopback bind (e.g. `--host 0.0.0.0`, to reach the workspace from
another machine) prints a prominent warning -- it exposes the workspace to
your local network. Token auth still applies; an SSH tunnel or a reverse
proxy you control is the tighter choice over binding a wide interface.

## Zero egress

The server only opens a **listening** socket: no outbound connection, and
nothing it imports phones home -- audio, traces, and evaluations stay on
the machine. A test whitelists loopback and fails if any view attempts an
external connection, backed by a threat-model row.


================================================================================
FILE: docs/GUARDIAN-FLEET.md
================================================================================

# Hotato Guardian / Fleet

Guardian/Fleet runs Hotato's capture → scan → trust → label → contract →
trial primitives as one continuous workflow inside a private, self-hosted
control plane: a hardened evidence kernel with a fleet view on top. Every
production deploy stays a human call.

## The evidence kernel

### An evidence vector, scored dimension by dimension
Every artifact carries a machine-readable **evidence vector**
(`hotato/evidence.py`, schema `evidence_vector.v1.json`): each dimension
scored on its own, public tier set to the *weakest* tier any required
dimension allows -- a minimum over an inspectable lattice. Tiers, ascending:

| tier | name | meaning |
|---|---|---|
| 0 | none | no usable evidence for a positive claim |
| 1 | asserted | envelope-only; not recomputed from audio |
| 2 | measured | recomputed from audio, one recording, input clean |
| 3 | paired | manifest-bound before/after, both sides recomputed under one pinned policy |
| 4 | attested | paired + signed policy/contract + runner-attested capture |

Dimensions: `score_integrity`, `audio_identity`, `policy_integrity`,
`fixture_set_integrity`, `input_health`, `channel_mapping`,
`label_authority`, `pairing_integrity`, `capture_origin`. One weak dimension
pulls the whole tier down; no renderer may raise it.

### Trial manifest (`hotato/manifest.py`, `trial_manifest.v1.json`)
An immutable pin created *before* an experiment, from the battery: the
scorer config hash, one policy (applied to both sides), the complete
fixture universe, and each fixture's expectation, onset, and
scripted-stimulus identity. Before and after must reference the same
`manifest_hash`; a changed policy, scorer, label, onset, or fixture set
refuses the comparison.

### Recompute-from-audio (`hotato/recompute.py`)
The proof gate re-derives every verdict from the on-disk audio under the
manifest, checks each stored `verdict.passed` against the recomputed
result, and refuses:

- **verdict tampering** -- a stored verdict that disagrees with the recomputed one;
- **same-audio re-encode** -- before and after decode to the same PCM;
- **dropped fixtures** -- either side omits a pinned fixture;
- **unrelated audio** -- the after-side caller stimulus does not match the pinned one and no capture receipt binds it.

A green *paired* proof additionally requires evidence tier >= paired.

### Capture receipt (`hotato/receipt.py`, `capture_receipt.v1.json`)
A fresh-recapture claim needs more than distinct PCM: a capture runner
emits a receipt binding the recording to a trial, agent, call id,
timestamps, and decoded PCM; an HMAC signature makes it *machine-verified*.
A manual distinct WAV with no receipt is *operator-asserted* -- the
receipt is what promotes it to machine-verified fresh recapture.

### Contract authenticity (`hotato/attest.py`)
Contracts embed a canonical digest over schema + label + policy + audio +
scorer version, which `contract verify`/`unpack` recompute: a bundle
edited after creation (a loosened policy re-pack, say) reports **tampered**
and fails; a matching but unsigned bundle is *"unsigned, internally
consistent evidence"*; a valid HMAC signature promotes it to *authenticated*.

### Boundary sensitivity
Every event exposes `onset_frame_index`, `onset_effective_sec`,
`decision_margin_sec`, `decision_margin_hops`, and `boundary_sensitive`: a
result within one hop of flipping is flagged, so it never reads with the
confidence of a result a hundred milliseconds inside the limit.

### Trust headline
Any verdict-changing warning (low signal, possible channel swap,
VAD-relevant leakage) sets the headline to `scan with caution`.
`input_health` is an explicit three-state field (`clean` / `caution` /
`not_scorable`); leakage is judged both against a fixed -40 dB bar and
dynamically against the receiving channel's own VAD gate.

## Fleet control plane (local mode)

Local mode is zero-dependency: SQLite metadata plus a content-addressed
artifact directory, both stdlib.

- **Registry** (`hotato/fleet/registry.py`) -- workspace-scoped rows for
  agents, deployments, calls, recordings, candidates, labels, contracts,
  trials, decisions. Agents scale per workspace with no product-level cap;
  every row carries `workspace_id`, scoping its paths and call ids.
- **Artifact store** (`hotato/fleet/store.py`) -- content-addressed blobs
  with dedup and lineage; audio is stored separately from any UI (privacy
  reversal).
- **Job queue** (`hotato/fleet/jobs.py`) -- leased jobs with deterministic
  idempotency keys, heartbeats, retries, and dead-lettering, so duplicate
  webhooks and worker crashes converge on one logical result.
- **Guardian API** (`hotato/fleet/api.py`) -- ingest → discover (trust
  preflight + scan) → human review + label → manifest-bound before/after
  experiment that **recommends a change for you to apply**, and refuses
  forged / same-audio / incomplete trials.

### CLI

```
hotato fleet init -w acme
hotato fleet agent add -w acme --agent-id support-bot --stack vapi --assistant-id asst_123
hotato fleet ingest -w acme --agent support-bot call.wav
hotato fleet discover -w acme --agent support-bot call.wav
hotato fleet review -w acme
hotato fleet label <candidate-id> --decision yield --reviewer you
hotato fleet experiment run -w acme --agent support-bot --trial-id t1 \
    --battery before/run.json --before before/ --after after/
hotato fleet status -w acme
```

Live clone/recapture/canary run against a connected stack with credentials
and a tested rollback; every release recommends the change and keeps
routing production traffic a manual, human-gated step.

## Synthetic perturbations (`hotato/synth.py`)
Deterministic acoustic transforms of real fixtures -- resample, gain, noise,
leakage, channel invert, silence, onset offset, clip -- each carrying its
parent hash, recipe, seed, and an explicit synthetic designation. Synthetic
and real stay on separate axes: only a real recapture raises the confidence
tier.

## What fleet mode ships today
Evidence stays per-dimension, never folded into a single fleet-wide score,
and every contract needs human approval before it exists. Today's live
provider adapters are Vapi and Retell (clone→apply→recapture); production
deploys stay the same manual, credentialed step described above.


================================================================================
FILE: docs/START.md
================================================================================

# Start here: the guided first run

`hotato start --demo` runs the whole hotato loop on bundled data, in two
acts. Act one (timing): it sweeps the two demo recordings (the same calls
`hotato demo` scores) and builds and verifies one failure contract from
them. Act two (say-do): it runs one conversation check over a bundled
scripted conversation -- the agent says the refund was sent; the trace and
the post-call state show it was not issued. It writes every artifact you
need to see both flows and prints the exact next commands.

```bash
hotato start --demo
```

## What it writes

Into the current directory (or `--dir DIR`):

| File | What it is |
|---|---|
| `hotato-sweep.json` | Every candidate timing moment across the two demo calls, ranked. An `analyze` result -- `FILE#N` off it drives `fixture promote` / `card` unchanged. |
| `hotato-sweep.html` | Self-contained dashboard (embedded audio, no external assets), opens in any browser. |
| `hotato-no-single-threshold.svg` | Threshold-funnel card: the demo battery misses an interruption **and** false-stops on a backchannel, so no single dial fixes both. See `docs/CARDS.md`. |
| `contracts/demo-missed-interruption.hotato/` | One failure contract, built from the sweep's missed-interruption candidate with `--expect yield`, verified on the spot. See `docs/CONTRACTS.md`. |
| `saydo/` | Act two's conversation: `transcript.json`, `trace.jsonl`, `state.json`, `test.json`, and the evaluated `test-run.json`. See below. |

Everything runs offline: demo, analyze, card, contract, and say-do steps
all run on packaged audio, packaged conversation files, and local files
alone.

## The demo failure contract

`start --demo` runs the whole loop once, on a bundled recording -- not
just a ranked list of candidates:

```bash
hotato contract create --from-candidate hotato-sweep.json#2 --expect yield \
    --id demo-missed-interruption --out contracts
hotato contract verify contracts/
```

Candidate #2 in the bundled sweep is the missed-interruption call: the
agent talked over the caller instead of yielding. Scored against
`--expect yield`, it fails -- so `start --demo` prints:

```
verified contract: FAIL as expected -- the demo call missed the
interruption; a CI gate on this contract catches any change to the evidence
or policy -- catching the AGENT regressing requires a fresh recapture (see
docs/RECAPTURE.md)
(start --demo itself exits 0 because setup succeeded; run the next command to
see the contract's CI exit 1: hotato contract verify contracts/)
```

`hotato contract verify contracts/` re-scores that same audio and reports
the same regression (exit `1`); a CI job wired to it fails the same way if
the evidence or policy ever changes. Telling whether a currently deployed
agent still has the bug takes a fresh recapture:
[`docs/RECAPTURE.md`](RECAPTURE.md). Two-lane breakdown:
[`docs/CONTRACTS.md`](CONTRACTS.md#two-lanes-what-verify-proves-depends-on-which-recording-you-feed-it).
Run it yourself:

```bash
hotato contract verify contracts/
```

## Act two: the say-do check

Timing is act one -- how the call sounded. Act two checks what the agent
did. The demo bundles one scripted say-do conversation (`saydo/`,
mirroring the reference agent's `refund-claimed-not-issued` job in
`examples/reference-agent`): the caller asks for a refund, the agent looks
the order up, and then says the refund was sent -- while the trace carries
no `issue_refund` tool span and the post-call state's `refund_status`
stays `"none"`. Every packaged file is verified against the sha256 its
manifest records before use.

`start --demo` evaluates that conversation through the same machinery
`hotato test run` drives -- a `phrase` assertion holds the agent's claim
(it passes), and the `tool_result` + `state` assertions read the trace
(Authority 1) and the post-call state (Authority 2) -- so it prints:

```
say-do check:    FAIL, by design: the agent said the refund was sent;
                 the trace shows no such tool call succeeded (no
                 issue_refund span), and the order's post-call
                 refund_status stayed "none".
```

Tool and state evidence decide the outcome, never the agent's words. The
evaluated result lands at `saydo/test-run.json`; render it as the say-do
card (`docs/CARDS.md`) with:

```bash
hotato card saydo/test-run.json --out saydo-card.svg
```

Replay the check as the CI gate it is (exit `1`, by design):

```bash
hotato test run saydo/test.json --agent demo-agent \
    --transcript saydo/transcript.json --trace saydo/trace.jsonl \
    --state saydo/state.json
```

The whole conversation-QA loop behind it -- suites, scenarios, the 375-run
reference agent -- lives in [`docs/CONVERSATION-TEST.md`](CONVERSATION-TEST.md)
and [`docs/SUITE-RUN.md`](SUITE-RUN.md).

## What it prints

The exact next commands, ready to copy:

1. **Save a candidate as a permanent regression test** -- you choose the
   label, hotato leaves that judgment to you:

   ```bash
   hotato fixture promote hotato-sweep.json#1 --expect <yield|hold> \
       --id my-first-fixture --out tests/hotato
   ```

2. **Run your fixtures in CI** (exits non-zero on a regression):

   ```bash
   hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio
   ```

3. **Render a PR card** from any candidate:

   ```bash
   hotato card hotato-sweep.json#1 --out candidate.svg
   ```

4. **Re-verify the demo failure contract in CI** (or build your own from any
   candidate with `hotato contract create`):

   ```bash
   hotato contract verify contracts/
   ```

5. **Replay the say-do gate** (exit `1`, by design) and **render the
   say-do card**:

   ```bash
   hotato test run saydo/test.json --agent demo-agent \
       --transcript saydo/transcript.json --trace saydo/trace.jsonl \
       --state saydo/state.json
   hotato card saydo/test-run.json --out saydo-card.svg
   ```

## Other modes

`--stack`, `--folder`, and `--stereo` each hand you the shipped command
for the job:

| Flag | Command | Does |
|---|---|---|
| `--stack` | `hotato sweep --stack <stack>` | Connect once, sweep your own calls |
| `--folder` | `hotato analyze <folder>` | Scan a directory of recordings |
| `--stereo` | `hotato run --stereo <call.wav>` | Score one dual-channel call |

## Output and exit codes

`--format json` emits a machine object (the files written, the next
commands, a `contract` block with the demo contract's id, expect,
scorable, passed, and `verified_fail_as_expected`, and a `saydo` block
with the say-do check's test id, exit code, claim assertion, and evidence
assertions with their share-safe public reasons) for an agent to drive.

- **0**: the guided first run completed -- including when `--stack`,
  `--folder`, or `--stereo` printed the shipped command to use instead.
  This holds even though the demo contract FAILS its policy and the
  say-do check FAILS its test: `start --demo` finishing is separate from
  those gates passing, which `hotato contract verify contracts/` and
  `hotato test run saydo/test.json ...` each report (exit `1`, by
  design).
- **2**: usage error -- no mode given, or `--dir` is not a directory.


================================================================================
FILE: docs/FULL-DUPLEX.md
================================================================================

# Full-duplex turn-taking: scoring the moment both sides speak at once

In a full-duplex conversation the agent hears while it speaks, so overlap is
a normal part of the audio: a caller barges in mid-sentence and both voices
are live at the same time. The question a transcript cannot answer is what
happened next, in seconds: did the agent drop its turn when the caller took
the floor, how fast, and how long did both keep talking at once. hotato
measures exactly that from the two channels of the recording -- `did_yield`,
`seconds_to_yield`, `talk_over_sec` -- the same timing physics whether the
agent runs half- or full-duplex.

`examples/full-duplex/` is the runnable pair for that claim: two scripted
two-channel scenarios, byte-identical on every render, that differ in one
variable only -- what the agent's channel does after the barge-in.

| id | behaviour | outcome |
|---|---|---|
| `fdx-01-barge-in-clean-yield` | caller takes the floor at 2.00 s; the agent stops inside the bounds | overlap, then a clean yield: PASS |
| `fdx-02-barge-in-talk-over` | same barge-in; the agent keeps transmitting through 2.75 s of simultaneous speech | sustained talk-over: FAIL |

Both scenarios declare the same expectation (`yield: true`,
`max_time_to_yield_sec: 1.00`, `max_talk_over_sec: 1.00`) and the same
caller onset. The PASS case still contains measured simultaneous speech:
overlap itself is not the failure, holding the floor through it is.

## Render the pair (deterministic)

```bash
python examples/render_examples.py
```

The per-channel seed is `sha256(scenario_id)`, so two renders are
byte-identical on any machine; `tests/test_examples_full_duplex.py` renders
twice and diffs to pin it. The audio is synthetic, energy-shaped noise --
a scripted fixture, never a recorded person.

## Run both: one PASS verdict, one FAIL verdict

```bash
hotato run --scenarios examples/full-duplex/scenarios --audio examples/full-duplex/audio
```

```
hotato [suite] stack=generic offline=True
  1/2 events pass  (failed=1)
  [PASS] fdx-01-barge-in-clean-yield: did_yield=True seconds_to_yield=0.75s talk_over=0.75s
  [FAIL] fdx-02-barge-in-talk-over: did_yield=True seconds_to_yield=2.75s talk_over=2.75s
         fix[config]: Slow yield: the agent stopped, but too late
            knob: endpointing / VAD min-silence-duration
            move: lower the min-silence and hangover so the agent goes quiet sooner after the caller takes the floor
  exit_code=1
```

The battery exits 1 because it carries one caught regression. In the JSON
envelope (`--format json`) the FAIL verdict states its reasons exactly:

```
yielded in 2.75s, slower than the 1.00s bound
talked over the caller for 2.75s, more than the 1.00s bound
```

Note what the FAIL is not: in `fdx-02` the agent DID yield eventually
(`did_yield=True`), unlike `funnel-demo`'s `fd-01`, where the agent never
stops inside the search window. The regression here is the measured seconds
of talk-over on the way to the floor transfer -- the full-duplex failure
mode, invisible to a text-level eval.

## Score either recording on its own

The same verdicts, one file at a time, with the bounds on the command line:

```bash
hotato run --stereo examples/full-duplex/audio/fdx-01-barge-in-clean-yield.example.wav \
    --onset 2.0 --expect yield --max-time-to-yield 1.0 --max-talk-over 1.0   # exit 0
hotato run --stereo examples/full-duplex/audio/fdx-02-barge-in-talk-over.example.wav \
    --onset 2.0 --expect yield --max-time-to-yield 1.0 --max-talk-over 1.0   # exit 1
```

## Investigate the failure

`hotato investigate` ranks the caught moment without any label given up
front:

```bash
hotato investigate examples/full-duplex/audio/fdx-02-barge-in-talk-over.example.wav
```

```
hotato investigate [run 1]: fdx-02-barge-in-talk-over.example.wav
  capture origin: frozen regression clip (examples/full-duplex/scenarios/fdx-02-barge-in-talk-over.json)
    this recording is a previously-created hotato fixture clip (fdx-02-barge-in-talk-over.json), not a live call: a pinned regression, not fresh evidence
  input health: eligible for scan
  verdict path: eligible (a labeled event here can carry a real yield/hold verdict)
  most likely failure (top-ranked candidate):
    [1] t=1.99s overlap_while_agent_talking  overlap_sec=2.76
  next: label it (use --expect hold instead if the agent was right to keep talking):
    hotato investigate label '.hotato/investigate-state.json#1' --expect yield
  state remembered at: .hotato/investigate-state.json
```

The origin line says out loud that this is a fixture clip, not a live call.
On your own recording the same command prints the same ranked moments and
the one `investigate label` next step that pins the catch as a CI contract
-- the core loop in [AGENTS.md](../AGENTS.md) from step 2 on.

## Exit codes

| Exit | Meaning |
|---|---|
| `0` | every scorable event passed its declared bounds |
| `1` | a scorable event regressed (here: the talk-over bound) |
| `2` | usage error or unusable input (e.g. a mono export: NOT SCORABLE) |

## Related

- [`examples/README.md`](../examples/README.md) -- every additive fixture
  set, including this pair, and the deterministic renderer.
- [INVESTIGATE.md](INVESTIGATE.md) -- one recording to ranked candidate
  moments and a signed contract.
- [CONTRACTS.md](CONTRACTS.md) -- the CI gate that re-runs the stored
  evidence deterministically.
- [SIMULATE.md](SIMULATE.md) -- the scripted-caller renderer for
  transcript/trace fixtures (`origin=simulated`), the no-audio side of the
  same regression foundation.


================================================================================
FILE: docs/INVESTIGATE.md
================================================================================

# `hotato investigate`: one recording in, ranked candidate moments out

Point it at one recording -- a local dual-channel WAV, or a live pull by
call id -- and it authenticates the source, runs the input-health gate,
scans for candidate turn-taking moments, and prints the exact command to
turn each one into a signed, CI-ready contract.

```bash
hotato investigate label <candidate_ref> --expect yield|hold
```

The one decision that stays with you: which candidate is a bug, and whether
the agent should have *yielded* (stopped) or *held* (kept talking through a
backchannel, noise, or ack). `"mhm"` and `"stop"` can sound identical by
energy, so the label is yours.

## Scan one recording

```bash
hotato investigate call.wav                        # a local WAV
hotato investigate --stack vapi --call-id abc123    # or pull live
```

Give it a local `SOURCE` path, or `--stack`/`--call-id`, never both. Stacks:
`vapi`, `twilio`, `retell`, `bland`, `elevenlabs`, `synthflow`, `millis`,
`cartesia`. LiveKit and Pipecat capture in your own infra: run `hotato setup
--stack <name>` and pass the resulting WAV as `SOURCE`. `--allow-mono`
accepts a mono/mixed recording, but a summed channel can't score separated
candidates.

Flags: `--min-gap` (min. gap in seconds to surface, default `2.0`), `--top`
(candidates shown, default `10`, `0` = all), `--caller-channel` /
`--agent-channel`, `--state PATH` (default `.hotato/investigate-state.json`).

### What it reports

```
hotato investigate [run 1]: call.wav
  capture origin: operator-asserted local file (call.wav)
  input health: <trust recommendation>
  verdict path: eligible (a labeled event here can carry a yield/hold verdict)
  N candidate moment(s) (showing N):
    [1] t=42.18s <kind>  <durations>
        label: hotato investigate label .hotato/investigate-state.json#1 --expect yield|hold
  state remembered at: .hotato/investigate-state.json
```

## Capture origin: tracked every run

Every run tags how the audio arrived: `frozen_regression` (a pinned hotato
fixture, replayed exactly), `provider_pulled` (fetched live from the
stack's own recording API -- for the stronger, signed claim see
[RECAPTURE.md](RECAPTURE.md)), or `operator_asserted_local` (a local WAV,
taken at face value).

## The verdict gate

Trust runs in contract mode, the same crosstalk/leakage bar `hotato contract
create` checks. Two outcomes:

- **NOT SCORABLE** (mono, identical channels, a silent required channel): no
  candidates scanned, the reason named, exit `2`. Fix the input and re-run.
- **Verdict path REFUSED** (a suspected channel swap or crosstalk/leakage):
  candidates still surface; a labeled event carries a verdict once you fix
  the crosstalk or confirm the mapping with `--confirm-channels`. See
  [TRUST.md](TRUST.md).

Scan runs whenever the input is scorable, so candidates appear even when the
verdict path is refused.

## State and candidate refs

State persists to `.hotato/investigate-state.json` (run-numbered, atomically
written) in the same shape `hotato analyze` / `hotato sweep` use, so it's a
valid `FILE#N` candidate ref for `hotato fixture promote` and `hotato
contract create --from-candidate`. `#1` is the top-ranked candidate.

## Label a candidate into a contract

Your `--expect` goes straight to `hotato contract create --from-candidate`,
which mints a signed label-record bound to the exact decoded audio when a
signing key is configured. Without one, `label_authority` floors at
`asserted`.

```bash
hotato investigate label .hotato/investigate-state.json#1 \
    --expect yield \
    --max-talk-over 0.6 --max-time-to-yield 1.0 \
    --reviewer "$USER" --out contracts
```

Writes `contracts/<id>.hotato/`. The id defaults to a slug from the source,
onset, and label; `--id` names it, `--force` overwrites. `--pre`/`--post`
(defaults `2.0`/`6.0`) set the clip window before/after the onset;
`--no-clip` keeps the full recording. If the verdict path was refused, add
`--confirm-channels` to carry it through `contract verify`. Source
basenames are redacted by default; `--include-identifiers` keeps them.

A contract, not a bare fixture, is the point: it carries the trust block,
the CI policy, and the exact `hotato contract verify` command -- the
handoff into [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md) and
[CONTRACTS.md](CONTRACTS.md).

Scanning, the gate, and labeling all run offline; audio stays on your
machine unless you pull it from your own stack
([THREAT-MODEL.md](THREAT-MODEL.md)).

## Exit codes

`hotato investigate`:

- **`0`** -- candidate-eligible: trust + scan ran, candidates (if any)
  persisted. A verdict may still be refused; see `verdict_status`.
- **`2`** -- usage error (no `SOURCE`/`--stack`+`--call-id`, both given, a
  bad flag, a missing credential, an unreadable state file), or NOT
  SCORABLE.

`hotato investigate label`:

- **`0`** -- a signed, CI-ready contract was written.
- **`2`** -- usage error (a bad `--expect`, a bad candidate ref, an
  unresolved source, an existing contract without `--force`), or not
  scorable.

## See also

- [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md) -- the bad-call to CI-gate flow this feeds.
- [CONTRACTS.md](CONTRACTS.md) -- the failure-contract bundle.
- [TRUST.md](TRUST.md) -- the input-health scorability gate.
- [RECAPTURE.md](RECAPTURE.md) -- the stronger, signed fresh-recapture claim.
- [CONVERSATION-TEST.md](CONVERSATION-TEST.md), [SUITE-RUN.md](SUITE-RUN.md) -- the conversation-QA path.


================================================================================
FILE: docs/CONTRACTS.md
================================================================================

# Failure contracts

A failure contract packages ONE call moment into a portable, private
bundle: audio, frame-level timing evidence, an input-health report, a
PR card, a CI pass/fail policy, and the commands to replay and
re-verify it. It is the CI object -- `hotato contract verify` re-scores
a directory of contracts and exits non-zero when one regresses.

A contract's label always comes from a human (`label_source` is frozen
to `"human"`). Hotato measures whether the recorded timing matched that
label; `contract verify` re-measures the SAME recording later and
reports pass/fail.

> **A contract bundle contains call audio (`audio/event.wav`).** Do not
> commit a raw customer contract to a public repository; treat it like
> any recording of a caller. Use sanitized fixtures (synthetic or
> consent-cleared) for anything public, and keep customer contracts in a
> private repository or controlled artifact storage. `--include-identifiers`
> also writes a source basename and candidate ref into `contract.json`
> and the card; leave it off by default. This covers metadata only --
> full audio-handling rules are in
> [`SECURITY.md`](../SECURITY.md#audio-handling).

## Two lanes: same command, two different proofs

| | Lane A -- frozen recording | Lane B -- fresh recapture |
| --- | --- | --- |
| **Re-scores** | the SAME `audio/event.wav` the contract was created from | a new recording of the SAME stimulus against your CURRENT agent |
| **A pass proves** | the evidence, policy, and scorer are still intact and still agree with the human label | the CURRENT agent's behavior on that stimulus still matches the label |
| **Leaves open** | whether the deployed agent's behavior has changed since | nothing extra -- this lane speaks to the live agent |
| **Runs** | every push, in the shipped `ci/github_action.yml` (`contract verify contracts/`) | only when you recapture by hand or on a schedule -- see [`docs/RECAPTURE.md`](RECAPTURE.md) |

A frozen-recording pass is necessary but not sufficient: the recording
never changes, so it can only fail if someone edits the bundle's audio
or policy. Confirming the CURRENT agent still yields correctly takes
re-running the same stimulus and re-verifying the fresh capture, by
hand ([`docs/RECAPTURE.md`](RECAPTURE.md)). Run the frozen gate on
every push to catch evidence/policy drift for free, and recapture
periodically (or after an agent change) to catch what the frozen
recording cannot.

## The bundle

`hotato contract create` writes one self-contained directory, `<id>.hotato/`:

```
refund-cutoff-001.hotato/
  contract.json                      # the contract itself (schema hotato.contract.v1)
  audio/event.wav                    # the (clipped) two-channel recording, or the mono file
  evidence/
    frames.jsonl                     # per-frame timing evidence behind every measurement
    timeline.html                    # the to-scale caller/agent timeline, self-contained
    trust.json                       # the input-health (trust doctor) report
    card.svg                         # a self-contained 1200x630 SVG card (redacted by default)
  traces/                            # empty until `hotato trace attach` (see docs/TRACE.md)
  source/
    call_metadata.json               # redacted-by-default: stack, category, expect
    stack_config_snapshot.json       # placeholder until populated by hand
  policy/verify.yaml                 # the SAME subset `hotato verify --policy` reads
  reports/
    initial.html                     # the full scored report at creation time
    after.html                       # placeholder until a fix is re-captured and verified
  provenance.json                    # who/when/how this contract was created
  ci/
    github-action.yml                # a weekly + on-push CI scaffold
    junit.xml                        # this ONE contract's JUnit result at creation time
```

Every path above is also recorded, machine-readable, in
`contract.json["bundle"]["paths"]`.

## Create

From a candidate a sweep or scan already surfaced:

```bash
hotato sweep --demo --format json > hotato-sweep.json
hotato contract create --from-candidate hotato-sweep.json#1 \
    --expect yield --id refund-cutoff-001 --out contracts
```

From a raw two-channel recording you already have:

```bash
hotato contract create --stereo bad-call.wav --onset 42.18 \
    --expect yield --id refund-cutoff-001 --out contracts \
    --max-talk-over 0.6 --max-time-to-yield 1.0
```

Both forms wrap the SAME round-trip guarantee `hotato fixture create`
gives: the moment is scored immediately, and a not-scorable input (the
agent silent at the onset, an unreadable file, a bad channel map) is
refused with a clear reason and exit code 2 -- no bundle is written. A
single-channel (mono) recording passed as `--stereo` is rejected the
same way `fixture create` rejects it: caller and agent cannot be told
apart on one channel.

`--caller FILE --agent FILE` (two mono WAVs) is a third input form,
scored and clipped identically.

### Redaction

By default the bundle and card hide a candidate ref and a source
recording's basename. Pass `--include-identifiers` to show them (in
`source/call_metadata.json`, `contract.json`, and `evidence/card.svg`).

### The opt-in diarized-mono path

A single-channel recording can still become a contract, through the
SAME quality-gated diarizer front-end `hotato run --mono --diarize`
uses:

```bash
hotato contract create --mono call.wav --diarize \
    --expect yield --id refund-cutoff-002 --out contracts
```

This keeps an indicative-only verdict in its tier: a `low`-confidence
separation tier carries `measurement.indicative_only: true` into
`contract.json` and every renderer; a `refuse` tier (not two clean
parties, extreme overlap, unstable segmentation, voices too similar) is
refused exactly like a plain mono file, with the specific reason. This
evidence lives in `evidence/trust.json`'s separation confidence tier:
`evidence/timeline.html` points there instead of the frame-level
to-scale timeline (`evidence/frames.jsonl`) a dual-channel contract
carries.

## Verify

```bash
hotato contract verify contracts/
hotato contract verify contracts/ --format json --junit contracts-junit.xml
hotato contract verify contracts/refund-cutoff-001.hotato --html verify.html
```

`DIR` is a contracts directory (every `*.hotato` subdirectory carrying a
`contract.json`) or one bundle directly. For each contract, `verify`
re-scores the SAME bundled audio against the SAME policy in its own
`contract.json` -- this is what changes after an engine upgrade, a
threshold change, or a re-captured recording swapped into
`audio/event.wav` -- and reports pass/fail per contract and overall.

Exit codes are the CI contract:

| Exit | Meaning |
| ---: | --- |
| `0` | every contract passes |
| `1` | at least one regressed (or is no longer scorable), or an embedded assertion deterministically FAILed (below -- it still counts toward this exit) |
| `2` | usage error, empty directory, or corrupt `contract.json` |

`--junit` writes one `<testcase>` per contract; the shipped
`ci/github_action.yml` scaffold runs this on push, on PR, and weekly,
and publishes the JUnit file as an artifact.

Every text and HTML render of `verify` also prints, verbatim: *"This
result re-measures stored evidence. It does not test the current
agent."* A green CI run here means the evidence, policy, and scorer are
still intact; checking today's deployed agent takes the fresh-recapture
lane above. See
[`docs/RECAPTURE.md`](RECAPTURE.md#claim-language-what-each-kind-of-evidence-lets-you-accurately-say)
for exactly what each kind of evidence lets you claim.

### Embedded assertions (optional)

A contract can carry its own `assertions` block (schema
`hotato.contract.v1`; see `schema/contract.v1.json`) -- the same
`{version, assertions}` document `hotato assert` reads from
`assertions.yaml`. When present, `contract verify` evaluates it through
the SAME `assert.v1` engine and reports it as a per-contract
`assertions` field, separate from the timing pass/fail in
`summary`/`passed`. Pass `--transcript FILE` (a plain JSON array of
`{role, text, start, end}` turns, or hotato's own `{"segments": [...]}`
shape) to give transcript context to `phrase`/`pii`/`policy`
assertions; `tool_call` assertions read the bundle's own attached trace
(`hotato trace attach`) if one exists. Missing context reports
`INCONCLUSIVE`. A deterministic FAIL contributes to the batch's nonzero
exit code exactly like a timing regression; the batch result also
carries a separate `assertions_failed` count. See
[`schema/assert.v1.json`](../src/hotato/schema/assert.v1.json) for the
result shape and `hotato.assert_` for the five deterministic kinds.

## Inspect

```bash
hotato contract inspect contracts/refund-cutoff-001.hotato
hotato contract inspect contracts/refund-cutoff-001.hotato/contract.json --format json
```

## Pack / unpack

A bundle directory travels as one file:

```bash
hotato contract pack contracts/refund-cutoff-001.hotato
# -> contracts/refund-cutoff-001.hotato.pack (deterministic; a MANIFEST.sha256.json
#    of every member travels inside it)

hotato contract unpack contracts/refund-cutoff-001.hotato.pack \
    --out contracts/refund-cutoff-001.hotato
# every member is verified against the packed sha256 manifest; any mismatch
# (a corrupt or tampered archive) is refused (exit 2) and nothing partial is
# left behind
```

Packing the SAME bundle directory twice produces byte-identical
archives (sorted member order, fixed timestamps, and every other value
written into each member's ZipInfo, including its `create_system`
byte): a contract's pack is a pure function of its bundle contents,
deterministic for a fixed hotato version. Byte-identical re-runs are
verified in CI on Linux x86_64, Python 3.10, 3.11, and 3.12 -- see
[VALIDATION.md](VALIDATION.md) Job 1.

### Security: unpack treats an archive as hostile input

A `.hotato` archive travels between teams, so `contract unpack`
verifies everything about it before trusting a single byte. Before or
during extraction it refuses (exit 2) on any of the following, writing
only inside a scratch temp directory removed on any failure:

* path traversal (`..`), absolute paths, and Windows-style backslash /
  drive-letter paths (`C:\...`) in a member name;
* symlink members and encrypted members;
* duplicate member names;
* any member not declared in its own `MANIFEST.sha256.json`;
* more members than a legitimate bundle could plausibly need;
* a declared or measured decompressed size past the cap (default 512
  MiB; set `HOTATO_CONTRACT_MAX_UNPACK_BYTES` or pass `--max-bytes` to
  raise it for a trusted archive) -- checked against the bytes measured
  live during extraction, beyond what the archive's own (untrusted)
  size metadata claims;
* a single member whose compression ratio is far beyond anything a
  legitimate bundle member produces (a zip-bomb signal), even under the
  total-bytes cap.

The sha256-per-member check (above) still runs on top of all of this: a
member can pass every hardening check and still be refused for not
matching the packed manifest.

## CI

The shipped `ci/github_action.yml` is the minimal wiring:

```bash
uvx hotato contract verify contracts/ --junit contracts-junit.xml \
    --format json > contracts-verify.json
```

Gate a pull request or a scheduled job on the process exit code; do not
parse stdout to decide pass or fail.

## What a contract proves, precisely

Hotato proves timing behavior against this explicit contract -- not
authorization, identity, compliance, or policy safety. `verify` reports
coincidence, not causation: a passing re-verify after a config change
coincides with the change, distinct from a controlled experiment. See
`docs/VALIDATION.md` and `docs/THREAT-MODEL.md`.

## Read more

- Proving the CURRENT agent, not just the frozen recording:
  [`docs/RECAPTURE.md`](RECAPTURE.md)
- The underlying regression-fixture primitive `contract create` wraps:
  [`docs/BAD-CALL-TO-CI.md`](BAD-CALL-TO-CI.md)
- Battery-scale before/after proof: [`docs/FIX-LOOP.md`](FIX-LOOP.md)
- Input-health (trust doctor): [`docs/TRUST.md`](TRUST.md) ·
  [`docs/TRUST-MATRIX.md`](TRUST-MATRIX.md)
- Diarized-mono scoring: [`docs/DIARIZE.md`](DIARIZE.md)
- PR cards (one measured moment as an image): [`docs/CARDS.md`](CARDS.md)
- Attaching a voice trace (observability bridge): [`docs/TRACE.md`](TRACE.md) ·
  [`docs/OTEL.md`](OTEL.md)
- Audio-handling rules (what raw audio may contain, redaction limits,
  distribution): [`SECURITY.md`](../SECURITY.md#audio-handling)


================================================================================
FILE: docs/COUNTEREXAMPLES.md
================================================================================

# Counterexamples: reduce one scripted failure to a verified repro

`hotato counterexample` takes one failing deterministic scripted scenario,
deletes inputs that are unnecessary to that exact failure, and writes a local
`.hotato-repro` directory.

```text
scenario + conversation test + target assertion
                    |
                    v
       hotato counterexample compile
                    |
                    v
  private runnable capsule + deletion certificate
```

Every accepted deletion is evaluated through Hotato's own scripted simulator
and deterministic assertion engine. A candidate is retained only when the
same assertion still contains the source-selected failure branch, and
therefore keeps the same failure fingerprint.

The v1 boundary is narrow:

- input is one local `hotato.scenario` plus one local
  `hotato.conversation-test`;
- the target is one assertion in `assertions.deterministic`;
- the scripted simulator supplies the transcript, trace, mock tools, and mock
  state;
- transformations only delete units named by `hotato.reducers.v1`;
- model-judged targets, provider sessions, captured recordings, external
  timing bundles, DTMF targets, and custom policy-pack paths are refused.

The scripted proof lane covers 15 deterministic assertion kinds through 48
closed, schema-coupled failure branches: `phrase`, `pii`, `policy`,
`tool_call`, `outcome`, `tool_result`, `tool_error`, `state`, `state_change`,
`handoff`, `termination`, `latency`, `entity_accuracy`, `sequence`, and
`count`.

The compiler does not invoke a voice agent, provider, model, TTS service, STT
service, command oracle, or network endpoint: its result describes the
scripted fixture under the recorded Hotato evaluator source identity, not
the behavior of a deployed agent.

## Run the example

The repository example: three caller turns, two mock tools, two state
records. The selected assertion fails because order `A-1` remains `pending`
instead of `posted`.

```bash
sh examples/counterexample/run.sh
```

The script compiles a private capsule in a new temporary directory, verifies
the proof, reproduces the reduced fixture, checks predicate semantics, and
exports a non-runnable share-safe projection, leaving both directories in
place and printing their paths.

Run the steps directly:

```bash
hotato counterexample compile \
  --scenario examples/counterexample/refund-not-posted.scenario.json \
  --test examples/counterexample/refund-not-posted.test.json \
  --target refund-posted \
  --workspace examples/counterexample \
  --out /tmp/refund-not-posted.hotato-repro

hotato counterexample verify /tmp/refund-not-posted.hotato-repro
hotato counterexample reproduce /tmp/refund-not-posted.hotato-repro
```

`--out` must name a path that does not exist. Compilation builds a sibling
temporary directory and promotes it with a no-replace primitive: Linux,
macOS, and Windows have explicit implementations, and an unsupported
platform refuses rather than falling back to an overwriting move.

## Source and target selection

### One base scenario

The compiler evaluates the scenario document passed to `--scenario` directly,
without expanding `variation_matrix` or selecting a matrix row. `capsule.json`
records:

```json
{
  "scenario_selection": {
    "mode": "base-scenario",
    "variation_matrix_applied": false
  }
}
```

`--seed` selects the scripted replay seed, defaulting to `scenario.seed` or
`0` when the scenario has none. A seed does not select a variation-matrix
combination.

To minimize one generated variation, materialize it as its own complete
scenario document, then pass the concrete file to `compile`. Leaving a matrix
in the source only preserves its bytes in source provenance; the reducer can
delete the matrix when it has no effect on the base-scenario failure.

### One deterministic assertion

`--target` must match exactly one assertion ID in `assertions.deterministic`.
Compilation confirms twice that the selected assertion fails with identical
result, content, and trace identities, and refuses a passing, inconclusive,
advisory, missing, duplicated, or unsupported target.

### Exact failure branch identity

A deterministic `FAIL` can contain more than one reason: the oracle maps them
into a closed set of payload-free failure atoms, sorts and de-duplicates them
canonically, and selects the first as the preservation anchor. The private
capsule records the selected atom and the complete ordered source atom set;
a candidate is `PRESERVED` only when that atom is still present in its own
atom set.

An atom carries only the discriminator needed to keep failure branches apart:
for example, an assertion index, detector, policy rule/type, state field, or
entity key. Example: a wrong tool argument gives `tool-argument-value-mismatch`
(plus its key); delete the value and the branch becomes
`tool-argument-field-missing`; delete the call and it becomes `tool-missing`.
Both are `DRIFTED`, though the assertion still reports `FAIL`. The same
missing-versus-value care extends to other kinds:

| Atom type | Distinguishes |
| --- | --- |
| results / termination | missing vs. wrong value |
| required-order / sequence | present-but-out-of-order vs. absent step |
| latency | declared mock measurement vs. simulator default |

Payload values, transcript text, and broad diagnostic counts are never part
of an atom.

The fingerprint binds the test and assertion identity, assertion bytes,
kind, dimension, deterministic authority, required `FAIL` status, and
selected source atom -- not every diagnostic field in the assertion result,
so irrelevant evidence stays reducible without the preserved failure
collapsing into a different branch.

The emitted `input/conversation-test.json` is a canonical projection
containing only the selected deterministic assertion: other source
assertions and the rubric lane sit outside the preservation oracle, and
their removal is normalization, not evidence those checks passed.

### Workspace boundary

`--workspace` is the directory both input files must resolve within, as
regular local paths without symlink escape. Omitted, it defaults to the
common parent of the scenario and test files; set it explicitly in
automation so the filesystem boundary stays visible in review.

## What the compiler deletes

`hotato.reducers.v1` is a closed, versioned, deletion-only algebra over the
scripted scenario, able to attempt removing:

- variation-matrix, facts, and environment entries;
- caller behavior and interruption declarations;
- caller turns while retaining at least one turn;
- mock tool calls, optional tool fields, and nested argument/result entries;
- handoff and termination records;
- mock-state resources, rows, snapshots, and nested leaves;
- optional additive scenario fields.

It cannot add content, replace a scalar, reorder a list, rewrite speech, or
change the target assertion. `certificate.json` records the digest-bound
delete-only transform for every accepted step; `reduction.jsonl` records
every candidate attempt.

## The 1-minimal claim

A capsule with:

```json
{"minimality": {"status": "one_minimal"}}
```

supports this exact statement:

> The fixture is 1-minimal under `hotato.reducers.v1` with the recorded
> observation-scope freezes.

After hierarchical reduction, the compiler repeatedly tries every remaining
single-unit deletion, reaching `one_minimal` only when none of those
deletions preserves the selected source failure branch. `verify` recomputes
the single-unit check rather than trusting `minimality.json`.

This is a local claim over the named deletion algebra -- not a global minimum,
a root-cause determination, or a semantic-equivalence claim. A remaining
deletion can make the target `PASS`, change the selected failure branch,
invalidate the scenario, or make required evidence unavailable; each such
outcome means that deletion did not preserve the recorded exact failure
identity, and is recorded as such in `minimality.json`.

## Budgets

`--budget` limits uncached candidate evaluations during reduction:

```bash
hotato counterexample compile ... --budget 512
```

| | |
| --- | --- |
| default | `512` |
| minimum | `1` |
| hard maximum | `100000` |

Cache hits do not consume the budget, nor do the two source and two final
qualification executions (reported separately).

When the budget ends before the final deletion pass completes, the compiler
still emits a capsule if the reduced fixture preserves the exact failure on
both final executions: status `budget_exhausted`, claim limited to failure
preservation, no 1-minimal claim.

Compilation exits `0` for `one_minimal` and `1` for `budget_exhausted`,
writing a complete capsule either way; automation can gate on the process
status alone, without parsing a nested field. `verify` can check the
integrity and preserved-failure evidence of a `budget_exhausted` capsule, but
does not upgrade it to `one_minimal`.

## Capsule contents

A private runnable capsule contains:

```text
case.hotato-repro/
  capsule.json
  oracle.json
  certificate.json
  minimality.json
  reduction.jsonl
  source/
    scenario.json
    scenario.original
    conversation-test.json
    conversation-test.original
  input/
    scenario.json
    conversation-test.json
  expected/assertion-result.json
  report.md
  report.html
  card.svg
  reproduce.sh
  predicate.sh
  MANIFEST.sha256.json
```

`MANIFEST.sha256.json` binds every other file; the capsule ID binds the
proof-relevant documents and their digests too. Strict verification rejects
missing, changed, extra, symlinked, or unsafe members, and independently
re-renders `report.md`, `report.html`, `card.svg`, `reproduce.sh`, and
`predicate.sh` before accepting their bytes.

The Markdown, HTML, and SVG reports summarize identifiers, proof status, and
reduction counts -- bound, reproducible projections of the capsule that do
not replace verification.

### Resource and local-filesystem boundary

Verification refuses a capsule before replay when it exceeds any of these
limits:

| Limit | Cap |
| --- | --- |
| files | 1,024 |
| directories | 4,096 |
| directory levels | 64 |
| one member | 64 MiB |
| all members combined | 256 MiB |

Each source JSON file is limited to 16 MiB and 96 JSON levels. The scripted
scenario also caps caller turns and mock tools at 10,000 each.

Counterexample compilation and replay add a narrower proof-lane budget:

| Item | Cap |
| --- | --- |
| selected assertion (canonical JSON) | 256 KiB |
| candidate scenario (canonical JSON) | 2 MiB |
| rendered transcript text (UTF-8) | 256 KiB |
| one deterministic assertion result (canonical JSON) | 2 MiB |
| `hits` / `matched_rules` evidence rows | 10,000 |
| deletion steps per accepted proof chain | 512 |
| deletion operations per step | 10,000 |
| remaining units in a completed minimality proof | 512 |
| proof-lane regex length | 1,024 UTF-8 bytes |

Each proof-lane regex also belongs to a closed, fixed-width replay subset;
groups, alternation, backreferences, and variable quantifiers are refused.
The assertion and regex byte bounds run before the general assertion
validator can invoke Python's regex parser.

The selected structured failure branch is still the preservation identity;
these limits just bound the work needed to establish it. A candidate that
crosses an execution limit is `UNRESOLVED` with `resource_limit_exceeded`,
never treated as preserved or absent. These counterexample compiler/replay
limits do not narrow the regex or result surface of Hotato's deterministic
evaluator outside the proof lane.

The capsule directory must stay unchanged for the duration of `verify`,
`reproduce`, `inspect`, or `export`: persistent mutation, a root
replacement, and symlink or special-file substitution are all detected. A
privileged local process able to swap bytes between reads and restore them
before the final manifest pass sits outside this version's proof snapshot --
copy an untrusted capsule into a private, non-writable workspace before
verification.

### What the journal proves

`certificate.json` and the `PRESERVED` rows in `reduction.jsonl` carry the
accepted delete-only chain; strict verification reconstructs those transforms
and reruns the final single-unit deletion inventory for a `one_minimal`
claim. `ABSENT`, `DRIFTED`, and `UNRESOLVED` journal rows are diagnostic reduction
history only; their candidate payloads sit outside the accepted proof chain,
unreplayed by `verify`.

## Verify, reproduce, and inspect

These commands answer different questions.

| Command | Question | Evaluator rule | Work performed |
|---|---|---|---|
| `verify` | Is this the same intact proof produced by the recorded evaluator source? | Recorded package version and evaluator source digest must match the installed implementation. | Replays source and final cases twice, reconstructs every accepted deletion, checks the selected source failure branch and expected result, re-renders derived artifacts, and recomputes claimed single-unit minimality. |
| `reproduce` | Does the reduced fixture still produce the selected source failure branch under the installed evaluator? | Evaluator drift is allowed and reported. | Checks capsule integrity and the delete-only chain, then runs the reduced fixture twice. It does not reassert historical intermediate verdicts or renew the original minimality proof under changed code. |
| `inspect` | What does the intact capsule claim? | No evaluator execution. | Checks the closed member inventory, source/oracle/artifact bindings, canonical human files, manifest, and capsule schema, then prints target, reduction, minimality, preservation, and profile metadata. |

Use `verify` for proof audit and artifact integrity against the recorded
Hotato package version and evaluator source closure. The evaluator digest
does not bind the Python interpreter build, OS, CPU, or native libraries;
strict replay still compares the recorded result, content, and trace hashes,
so a runtime difference that changes evaluator behavior fails verification.
Use `reproduce` to evaluate the frozen reduced fixture after Hotato
evaluator or simulator code changes.

`reproduce` returning `0` means the failure reproduced -- the inverse of a
conventional quality gate. Use `predicate` or the generated `predicate.sh`
when the desired shell result is “failure present means bad.”

## Private and share-safe forms

Compilation writes `private-runnable-v1`: complete source and reduced
scenario/test inputs, including scripted speech and mock tool/state
payloads. Treat the directory as sensitive even when its origin is simulated.
The compiler applies restrictive POSIX modes where supported, but permissions
do not establish publication rights or de-identification.

Create a payload-free projection only after the private capsule verifies:

```bash
hotato counterexample export /tmp/refund-not-posted.hotato-repro \
  --profile share-safe-v1 \
  --out /tmp/refund-not-posted.share-safe
```

The exported directory has one closed inventory: `capsule.json`, `report.md`,
`report.html`, `card.svg`, `README.md`, and `MANIFEST.sha256.json`. Extra,
missing, renamed, symlinked, or special-file members are refused even when a
manifest declares them. `inspect` re-renders every human-facing file and
requires byte-for-byte equality with its canonical rendering, so rebinding
modified report text into a new manifest does not make the projection valid.

Those canonical renderer bytes are part of the capsule v1 exchange contract.
A compatible implementation must retain the v1 renderer; changing its output
requires a versioned renderer/profile or a capsule format version bump.

The projection omits runnable inputs, scenario and assertion bodies,
transcript or audio content, tool payloads, state values, credentials,
provider identifiers, absolute paths, source-derived reducer paths, and
per-candidate digests: private single-unit rows become an aggregate count by
outcome instead.

The share target and canonical reports expose the selected failure code
(such as `tool-argument-value-mismatch` or `state-field-value-mismatch`), so
the failure category stays readable without a payload. Atom discriminators
(a field, key, rule, detector, or index) stay private.
`failure_atom_digest` binds the complete selected atom without publishing
those values.

`share-safe-v1` is an engineering access boundary, not an anonymity claim:
capsule, source, assertion, evaluator, fingerprint, and failure-atom digests
remain correlators, and low-entropy or externally known source material can
be tested against them. Keep a projection under the same review,
artifact-retention, and disclosure controls as engineering metadata.

The projection cannot reproduce the failure. Keep the private capsule when a
reviewer, CI job, or coding agent must execute the fixture.

## Exit codes

### `compile`

| Exit | Meaning |
|---:|---|
| `0` | An atomic private capsule was written and earned `one_minimal`. |
| `1` | An atomic private capsule was written with the exact failure preserved, but the candidate budget ended before one-minimality was proved. |
| `2` | Refused: input, target, workspace, output, replay, or proof safety requirements were not met. No completed destination is left. |

### `verify`

| Exit | Meaning |
|---:|---|
| `0` | Integrity, provenance, source/final replay, accepted deletion chain, exact failure identity, and the stated minimality level verified. |
| `1` | The capsule is intact, but the target failure or claimed single-unit minimality no longer reproduces. |
| `2` | No proof verdict: malformed, tampered, unsafe, evaluator-incompatible, unsupported, or inconclusive capsule. |

### `reproduce`

| Exit | Meaning |
|---:|---|
| `0` | The selected source failure branch reproduced twice under the installed evaluator. |
| `1` | The capsule is intact, but that exact failure is absent. |
| `2` | No reproduction verdict: malformed, tampered, unsafe, unsupported, disagreeing, or inconclusive capsule. |

`inspect` and `export` return `0` on success and `2` on refusal.

### `predicate`

`hotato counterexample predicate DIR` and `DIR/predicate.sh` map reproduction
to `git bisect run` semantics:

| Exit | Meaning |
|---:|---|
| `1` | Bad: the exact target failure is present. |
| `0` | Good: the exact target failure is absent. |
| `125` | Skip: the fixture is unusable, unsafe, unsupported, disagreeing, or inconclusive at this revision. |

## CI and evaluator bisect boundaries

### Recorded-evaluator proof audit

Use `verify` when CI installs the Hotato evaluator recorded in the capsule:

```bash
hotato counterexample verify tests/repros/refund-not-posted.hotato-repro
```

This detects capsule mutation and evaluator-source drift: a changed evaluator
digest is a refusal (`2`), since a different implementation cannot silently
validate the historical proof (what the digest binds is above).

### Current-evaluator regression

Use the generated predicate when a reproduced failure should fail the job:

```bash
tests/repros/refund-not-posted.hotato-repro/predicate.sh
```

On Windows, run the helpers through the interpreter -- `sh predicate.sh`,
`sh reproduce.sh`, `git bisect run sh /absolute/path/to/predicate.sh` -- the
capsule digests bind the same helper bytes on every platform.

The predicate executes the frozen reduced mock scenario; it does not call an
application's current voice agent, prompt, tools, provider, or telephony
path. V1 gates Hotato evaluator/simulator behavior over the capsule, not an
end-to-end agent release gate.

### `git bisect run`

```bash
git bisect start BAD_REVISION GOOD_REVISION
git bisect run /absolute/path/to/refund-not-posted.hotato-repro/predicate.sh
```

This locates the revision where Hotato evaluator or scripted-simulator
behavior begins reproducing the recorded failure. The capsule retains its own
source and reduced inputs, so external scenario/test file changes play no
part in the predicate. A revision that cannot load or evaluate the capsule
is skipped with `125` -- as with `git bisect run` generally, skipped
revisions near the transition can leave a candidate range rather than one
first bad revision.

Bisecting agent implementations, hosted models, provider configurations, or
prompt revisions needs a separately defined adapter, outside counterexample
v1, that runs those revisions and emits the evidence an assertion consumes.

## Refusals worth designing around

The compiler fails closed when:

- the target is in the rubric lane or does not fail twice identically;
- required evidence is missing and the target is inconclusive;
- a `timing_contract` depends on an external bundle;
- a DTMF assertion requests trace evidence the scripted simulator cannot emit;
- an `http_result` assertion requests captured exchange evidence the scripted
  simulator cannot emit;
- an outcome predicate reads an external timing field;
- a latency assertion reads an external timing field or a span type other than
  a scripted `tool_call`;
- a keyed tool argument/result target uses an empty key, or an entity reference
  uses an empty key or a null expected value;
- a policy assertion uses an external `pack_path`;
- an input escapes the workspace or traverses a symlink;
- the output path already exists;
- source or final replays disagree;
- a capsule member, manifest, certificate, or evaluator provenance is invalid.

Refusal is evidence the compiler cannot make its promised statement for that
input; it never becomes a passing or failing agent verdict.

## Machine output

Add `--format json` to `compile`, `verify`, `reproduce`, `inspect`, or `export`.
The JSON carries the command kind, exit code, capsule/failure identities, and
the command-specific proof or reduction fields. `predicate` communicates only
through its process exit.

See the complete runnable fixture in
[`examples/counterexample/`](../examples/counterexample/).


================================================================================
FILE: docs/PYTEST.md
================================================================================

# Pytest: the fixture and the gate

Install hotato and the plugin registers itself: standard `pytest11` entry
point, zero `conftest.py`, zero imports. It adds one fixture and one
opt-in session gate, both scoring with the CLI's engine and returning the
same envelope every hotato surface emits.

```bash
# zero-install with uvx, or: pipx install hotato
uvx hotato --help
```

## The `hotato_score` fixture

Score a recording or a suite inside any test and assert on the envelope --
the same inputs the CLI takes:

```python
def test_call_yields(hotato_score):
    env = hotato_score(stereo="call.wav", expect="yield")
    assert env["exit_code"] == 0
    assert all(e["verdict"]["passed"] for e in env["events"])

def test_split_channels(hotato_score):
    env = hotato_score(caller="caller.wav", agent="agent.wav", expect="yield")
    assert env["events"][0]["verdict"]["passed"]

def test_selftest_battery(hotato_score):
    env = hotato_score(suite="barge-in")
    assert env["exit_code"] == 0
```

The envelope is the standard machine shape (`schema_version` "1"): per
event, a `verdict` (`passed`, `did_yield`, `seconds_to_yield`,
`talk_over_sec`), a `signals` bus, a `fix` on every failure, and `limits`.
Assert on whatever your test cares about.

## The `--hotato-suite` session gate

Opt in and the battery runs after your tests, printing a summary and
failing the whole session (exit 1) on a regression:

```bash
pytest --hotato-suite                    # default suite: barge-in
```

The bundled battery self-tests the harness. Point the same flag at your own
labelled scenario and audio directories to gate on your own agent:

```bash
pytest --hotato-suite \
  --hotato-suite-scenarios corpus/suites/gold/scenarios \
  --hotato-suite-audio corpus/suites/gold/audio
```

Any directory in the same scenario shape works, including your own
labelled calls -- `docs/SUITES.md` covers the bundled tiers,
`docs/SUBMITTING.md` covers building fixtures from your own recordings.

## Two ready-made gates

- This flag rides on a test run you already have: turn-taking checks run
  with every `pytest` invocation, locally and in CI.
- The GitHub workflow (`docs/CI.md`) is the other ready-made gate -- it
  scores every pull request and posts a sticky results comment. Use either,
  or both.
- The plugin runs offline with deterministic, self-contained fixtures, so
  results stay stable run to run.


================================================================================
FILE: docs/SUITES.md
================================================================================

# Corpus suites: tiered, deterministic, explicit about what they prove

`corpus/suites/` ships four labelled scenario suites (112 scenarios), each
synthetic shaped noise rendered deterministically from its own labelled
timings (seed `sha256(scenario_id)`), the ground truth. Synthetic audio is
the floor: these suites prove the scorer runs end to end and catch
regressions; validity on your system comes from your own labelled calls
([`docs/SUBMITTING.md`](SUBMITTING.md)).

## The four suites

| Suite | Scenarios | Conditions | Expected exit |
| --- | --- | --- | --- |
| `silver` | 40 | clean, 16 kHz, default noise floor | `0` |
| `silver-defects` | 16 | clean conditions, deliberate defect renders | `1` |
| `gold` | 40 | hard conditions: noise floors, 8 kHz, gain extremes, echo, edge timings, endurance | `0` |
| `gold-defects` | 16 | hard-condition defect renders, plus two labelled capture-defect cases | `1` |

Scenario families: hard interruptions (onset, speed, duration, resume),
backchannels, short acknowledgements like "mhm" the agent should talk
through (position, density, repeats, length), double-talk, one-word
interruptions, stutter onsets, multi-turn exchanges, resume-then-reinterrupt,
and latency prompts.
`corpus/suites/manifest.json` is the machine-readable inventory: per suite,
family and category breakdown, sample rates, and expected exit code.

## Defect suites fail by design

Every `silver-defects` / `gold-defects` scenario is rendered to fail on its
labelled axis: an agent that keeps talking through a true interruption,
stops for a backchannel, or misses its latency budget. A defect suite that
exits `1` is the scorer catching what it claims to; one that exits `0`
would be a bug: the negative control the positive suites need to mean
anything.

## Run a suite

A suite is a scenarios directory plus an audio directory:

```bash
hotato run --suite barge-in \
  --scenarios corpus/suites/gold/scenarios \
  --audio corpus/suites/gold/audio
```

The same directories plug into every other surface: `hotato report ... --out
report.html` for the visual report, `hotato export ... --out research/` for
CSVs, and `pytest --hotato-suite --hotato-suite-scenarios ... --hotato-suite-audio ...`
for the session gate ([`docs/PYTEST.md`](PYTEST.md)).

## Deterministic builder

The suites are generator-built end to end, and the generator is the proof:

```bash
python3 corpus/suites/build_suites.py           # rebuild in place
python3 corpus/suites/build_suites.py --check   # regenerate to a temp dir, byte-compare
```

`--check` regenerating byte-identical output is the reproducibility
guarantee: the audio on disk exactly matches its labelled timings,
deterministic for a fixed hotato version (byte-identical re-runs verified
in CI on Linux x86_64, Python 3.10, 3.11, 3.12; see
`.github/workflows/tests.yml`, job `determinism`) -- usable as a CI drift
gate.

## Additive scenario classes

`corpus/classes/` ships four deterministic classes on top of the suites
above, built the same way (synthetic shaped noise, seed
`sha256(scenario_id)`, `--check` byte-compares a rebuild):

- `mid-utterance-pause`: a multi-second thinking gap mid-turn, distinct
  from a true turn end.
- `backchannel-multilingual`: short non-English acknowledgement tokens;
  Hotato's energy-based VAD treats them like English ones.
- `noise-hold`: sustained background presence, distinct from a brief
  backchannel, the agent should hold through.
- `telephony-degraded`: an existing gold scenario re-rendered through
  G.711 mu-law plus a fixed packet-loss schedule, proving the verdict
  stable across codec degradation.

Kept separate because `mid-utterance-pause` needs a non-default
`turn_end_silence_sec` beyond the generic suite tests. Full per-class
detail: `corpus/classes/README.md`.

## Against a live stack

Running scenario audio through a live voice stack and scoring what comes
back: [`docs/BENCHMARK-STACKS.md`](BENCHMARK-STACKS.md). Measurement-error
methodology for the scorer itself: [`docs/BENCHMARK.md`](BENCHMARK.md).

## Contribute scenarios

New synthetic families are welcome through the normal PR path
(`CONTRIBUTING.md`). The highest-value contribution stays a consented,
labelled call from your own traffic: walkthrough at
[`docs/SUBMITTING.md`](SUBMITTING.md); intake via the
[corpus-submission issue form](https://github.com/attenlabs/hotato/issues/new?template=corpus_submission.yml).


================================================================================
FILE: docs/CI.md
================================================================================

# CI: gate a PR on turn-taking

hotato scores turn-taking timing from call recordings, so a pull request
carries a running score and fails when the agent gets slower to stop
talking for an interrupting caller. The workflow runs offline with zero
extra dependencies.

## One command to the same record CI renders

`hotato start --demo` runs the whole loop offline and leaves the canonical
share-safe Failure Record under `hotato-failure-record/` --
`failure-record.{json,md,html,svg}`, the same primitive this Action renders
per non-passing unit (the `records` output below). The demo prints its
evidence-specific headline, the Markdown and SVG share paths, and the
one-command verifier:

```
Conversation failed: Agent did not yield; measured talk-over was 2.66 s.

  Share in a PR:      hotato-failure-record/failure-record.md
  Share as an image:  hotato-failure-record/failure-record.svg
  Verify the record:  uvx --from hotato==1.16.0 hotato record verify hotato-failure-record/failure-record.json
```

Preview it locally, then scaffold the durable gate into your own repository
in one command:

```bash
hotato init starter --stack generic --out .
# stack-tuned instead:  --stack vapi, retell, twilio, livekit, pipecat
```

That writes a CI workflow verifying `contracts/` and re-scoring `fixtures/`
on every push and pull request (a no-op until your first one lands), plus a
stack-tuned `hotato.yaml`, `contracts/`, and `fixtures/`.

## The root Action: run a committed suite from any repository

The repository root ships a composite GitHub Action: a repository with no
hotato source can run a committed suite, conversation test, or contract
verification and gate on hotato's exit status. The default run is
offline -- it runs the pinned Action revision itself off PYTHONPATH (no
pip, no package index), installs no model, no ASR, no Node tool, calls no
external judge, and reads no secret.

Composite Action since v1.4.0. Adopt the current release (v1.16.0), pinned
by its full commit SHA; resolve the tag to its SHA first:

```bash
git ls-remote https://github.com/attenlabs/hotato refs/tags/v1.16.0
```

Then commit this workflow (replace the `attenlabs/hotato` pin with the SHA
the command printed):

```yaml
name: conversation QA

on:
  pull_request:

permissions:
  contents: read

jobs:
  conversation-qa:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
      - id: hotato
        # Pin by full commit SHA (immutable); the comment names the release.
        uses: attenlabs/hotato@<full-commit-sha>  # v1.16.0
        with:
          suite: tests/voice/qa.suite.json
          agent: support-agent
      # Artifact upload is an explicit consumer step, pinned by full commit
      # SHA; the Action itself never uploads, comments, or notifies.
      - if: always()
        uses: actions/upload-artifact@834a144ee995460fba8ed112a2fc961b36a5ec5a # v4.3.6
        with:
          name: hotato-conversation-qa
          path: ${{ steps.hotato.outputs.output }}
          if-no-files-found: error
```

`permissions: contents: read` is sufficient. The Action never posts a
comment, uploads an artifact, or sends a notification -- retention and
publication stay under your workflow's control.

### Inputs

Exactly one of `suite`, `test`, `contracts` selects what runs (a
workspace-relative committed path); `agent` names the agent under test.
Everything else is optional:

| Input | Meaning |
|---|---|
| `release` | Defaults to the commit SHA |
| `output` | Defaults to `.hotato/results` |
| `parallel` | Suite worker cap; never changes result bytes |
| `transcript` / `trace` / `state` | Evidence files for a test run |
| `gate-advisory` | Test runs only; passes `--gate-judge` so the model-judged rubric lane gates |
| `hotato-version` | Which install to use -- see below |

`hotato-version` pins the install to one of three forms:

| Value | Effect |
|---|---|
| `action` (default) | Installs the pinned Action revision itself, `--no-deps`, no package-index egress -- exactly the revision your workflow pinned |
| an exact version, e.g. `1.16.0` | `pip install --no-deps hotato==1.16.0` |
| `preinstalled` | Skips installation (hotato is already on the runner) |

A range or `latest` is refused, so the pin always names one exact
revision. Suite policy lives in the committed suite file -- the Action
never overrides a suite's `inconclusive_policy`.

### Outputs and the exit contract

The step exit IS hotato's exit code, so the job gates without any extra
step:

| Exit | Meaning |
|---|---|
| 0 | every deterministic check passed under the committed policy |
| 1 | a deterministic check or a `success.required` condition failed |
| 2 | the verdict was withheld (refuse policy), or a usage/validation error |

The machine JSON stays primary:

| Output | Meaning |
|---|---|
| `output` | Results directory |
| `suite-result` | The machine result JSON path |
| `summary` | The rendered Markdown |
| `records` | Failure Record directory when produced, else empty |
| `exit-code` | As a string |
| `status` | `pass`, `fail`, `inconclusive`, or `error`, read from the machine result -- presentation only |
| `hotato-version` | The executed package version |

The five-lane summary (Outcome, Policy, Conversation, Speech, Reliability)
appends to the job page on pass AND failure, with the reproduce command
and evaluated check ids. A lane with no evaluated check renders NOT_RUN; a
lane whose checks lack required evidence renders INCONCLUSIVE, never
PASS. The model-judged rubric lane reports in its own advisory section
with `gate enabled: true|false` and changes the exit only when
`gate-advisory: true` was set; with no local judge reachable it reports
ERROR instead of guessing.

The conformance fixture:
[`tests/fixtures/action-consumer/`](../tests/fixtures/action-consumer/);
`.github/workflows/tests.yml` runs the same consumer shape against the
local checkout on every pull request (job `action-smoke`).

## Drop it in

Copy [`.github/workflows/hotato.yml`](../.github/workflows/hotato.yml) into
your repository at the same path. That is the whole setup. On the next
pull request it will:

- install the package with `pip install .`
- score the bundled barge-in suite to a JSON envelope
- render a Markdown summary with [`scripts/pr_comment.py`](../scripts/pr_comment.py)
- post or update one sticky comment (found by a hidden marker, so it stays
  a single comment across runs)
- fail the job on any regression

The sticky comment shows a pass/fail line, a per scenario table (expect,
yielded, time to yield, talk over, result), and a short regressions
section.

The job needs `pull-requests: write` to post the comment -- the workflow
already requests it; if your org restricts the default `GITHUB_TOKEN`,
grant that scope.

## Point it at your own recordings

The bundled suite is a self-test: it scores frozen synthetic fixtures,
proving the harness works. The strongest gate for your agent is a suite of
your OWN bad moments, pinned with `hotato fixture create` and run with
`hotato run --scenarios DIR --audio DIR`; full loop:
[BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md). Or replace one step, `Score
turn-taking (head)`, with your own capture and score:

1. play each corpus `*.caller.wav` into your agent
2. record your agent's reply
3. score the pair, caller on channel 0 and your agent on channel 1:

```bash
hotato run --stereo your_call.wav --expect yield --format json --no-fail > head.json
```

The envelope shape and exit codes are identical, so render, comment, and
gate steps stay exactly as they are. Keep `--no-fail` on the score step so
the comment still posts on a regression -- the true pass/fail lives in the
envelope's `exit_code`, which the `Fail on regression` step reads.

## Deltas against the base branch

When the workflow can install and score the base branch, it runs the same
suite there in an isolated venv as the baseline. Any scenario where
overlap grew (talk-over up) or the agent stopped later (time to yield up)
lists under Regressions with the delta. Best effort: when it can't run,
the comment falls back to the current pass/fail table and the gate still
holds.

## Gate timing drift against a saved baseline

`hotato baseline check` turns those deltas into a hard gate. Commit a
tolerance file naming how much each timing dimension may rise, save a
baseline run envelope, and the check exits 1 the moment a candidate run
drifts beyond a tolerance:

```yaml
# tolerances.yaml -- how much increase each dimension may absorb
response_gap_sec: "+10%"     # percent of the baseline mean
seconds_to_yield: "+0.05"    # absolute seconds
```

```bash
hotato run --suite barge-in --format json --no-fail > baseline.json   # once, on main
hotato run --suite barge-in --format json --no-fail > candidate.json  # per PR
hotato baseline check tolerances.yaml baseline.json candidate.json --junit drift.xml
```

Dimensions: `seconds_to_yield`, `talk_over_sec`, `response_gap_sec`,
`premature_start_sec`; each side's value is the pooled mean across its
scorable events. Every dimension is lower-is-better timing, so the gate is
one-sided: an improvement always passes, and a dimension with no
measurements on a side refuses (exit 2) rather than passing silently.
`--format json` emits the machine envelope with the per-dimension deltas;
`--junit drift.xml` feeds the same CI test widgets as `hotato contract
verify --junit`. Exit codes: 0 within tolerance, 1 drift beyond tolerance,
2 usage error.

## Feed the CI test widget: `--junit`

`hotato suite run --junit suite.xml` and `hotato prove --junit proof.xml`
write the same JUnit XML the contract and baseline gates emit, so one
upload feeds GitLab's `artifacts.reports.junit`, Jenkins `junit`, Azure
`PublishTestResults@2`, or CircleCI `store_test_results`:

```bash
hotato suite run ci.suite.yaml --agent support-v3 --junit suite.xml
hotato prove --contracts contracts/ --gauntlet --junit proof.xml
```

The file mirrors each command's grouping: one `<testsuite>` per dimension
(suite run) or per evidence lane (prove), one `<testcase>` per test or
lane. A scored FAIL is a `<failure>` carrying the measured reason; an
INCONCLUSIVE dimension, a `SIMULATOR_INVALID` run, or a refused lane is an
`<error>` with the refusal reason preserved -- never a `<failure>` and
never a silent pass, so the dashboard shows red exactly when the exit code
gates. The XML carries no timestamp and renders byte-identically on a
repeat run.

## Render a comment yourself

`scripts/pr_comment.py` is stdlib only and reads the same JSON the CLI
emits, so you can preview the comment locally:

```bash
hotato run --suite barge-in --format json | python3 scripts/pr_comment.py
```

Add `--base base.json` to include deltas against a saved baseline run.

## The other ready-made gate: pytest

If your CI already runs pytest, one flag adds the same regression gate:
`pytest --hotato-suite` scores the battery after your tests and fails the
session on a regression. Point it at your own labelled sets with
`--hotato-suite-scenarios` and `--hotato-suite-audio`. Details:
[`PYTEST.md`](PYTEST.md). For richer artifacts on a gate failure, `hotato
report` renders the visual report and `hotato team` tracks the trend
([`REPORTS.md`](REPORTS.md)).

## GitLab CI, Jenkins, Azure Pipelines, CircleCI: `hotato init ci`

The gate is exit-code driven, so it runs in any CI. `hotato init ci`
writes the one canonical config the chosen system reads:

```bash
hotato init ci --system gitlab     # .gitlab-ci.yml
hotato init ci --system jenkins    # Jenkinsfile
hotato init ci --system azure      # azure-pipelines.yml
hotato init ci --system circleci   # .circleci/config.yml
```

Every generated config does the same four things: install the pinned
hotato release (the version that generated the file), verify `contracts/`
with `hotato contract verify`, re-score `fixtures/` with `hotato run`, and
publish the JSON reports plus the JUnit file. A regression exits non-zero
and fails the pipeline; an empty `contracts/` or `fixtures/` directory is
a normal starting state, and each gate stays a no-op until the first one
lands.

- **GitLab CI**: one `hotato` job on `python:3.12`;
  `artifacts.reports.junit` feeds the merge request test widget, and the
  JSON reports upload with `when: always`.
- **Jenkins**: a declarative pipeline on a `python:3.12` docker agent
  (swap in `agent any` on a controller with Python 3.10+); `junit` and
  `archiveArtifacts` publish in the `post { always }` block.
- **Azure Pipelines**: `UsePythonVersion@0` plus script steps;
  `PublishTestResults@2` feeds the Tests tab and `PublishBuildArtifacts@1`
  keeps the JSON reports, both on `always()`.
- **CircleCI**: a `cimg/python:3.12` job wired into a workflow;
  `store_test_results` feeds the Tests tab and `store_artifacts` keeps the
  JSON reports.

`--out DIR` writes into another directory (default `.`, the repo root
where each system looks for its config); `--force` overwrites an existing
file. The pinned version is the hotato that generated the file; bump it in
one place when you upgrade.


================================================================================
FILE: docs/MCP.md
================================================================================

# MCP: the hotato server

hotato ships an MCP server, `hotato-mcp`, over stdio, with fifteen tools:
one scoring tool, `voice_eval_run` (returns the identical JSON envelope,
`schema_version` "1", the CLI emits); three proof-preserving counterexample
tools that compile and check offline regression capsules; and eleven fleet
tools -- eight read/verify/propose over a local fleet workspace, three
clone-scoped actions that recompute and hand the deploy decision back to
you. Everything, including audio, runs and stays on your machine.

Every tool response carries a uniform control envelope: four keys, pure
reads included, so an autonomous caller parses one shape: `evidence_status`
(or null for a pure read with no verdict), `refusal_reason` (or null),
`artifact_digests` (a list, or `[]`), and `pending_irreversible_action`
(the exact human-gated action still pending, e.g. deployment approval, or
null).

## Run it (zero-install)

```bash
uvx --from "hotato[mcp]" hotato-mcp
```

**Common mistake:** `uvx hotato-mcp` (no `--from`) FAILS -- uv looks for a
package literally named `hotato-mcp` on PyPI, which does not exist; the
console script lives inside the `hotato` distribution's `mcp` extra. Hit
that "package not found" error? Retry with `--from "hotato[mcp]"` exactly
as above. Already have hotato installed? `python -m hotato.mcp_server`
works too, no extra syntax needed.

## Add it to a client

In a project checkout, one command registers hotato with every agent config
surface the project already carries, including the `.mcp.json` server entry
when that file exists:

```bash
hotato init --agents
```

Every block below is copy-paste exact; only the outer key (client-specific)
differs -- all three use the same command and args.

### Claude Desktop

Edit `claude_desktop_config.json` (Settings -> Developer -> Edit Config):

```json
{
  "mcpServers": {
    "hotato": {
      "command": "uvx",
      "args": ["--from", "hotato[mcp]", "hotato-mcp"]
    }
  }
}
```

### Cursor

Project-scoped: `.cursor/mcp.json` in your repo root. User-scoped:
`~/.cursor/mcp.json`.

```json
{
  "mcpServers": {
    "hotato": {
      "command": "uvx",
      "args": ["--from", "hotato[mcp]", "hotato-mcp"]
    }
  }
}
```

### Codex CLI

Edit `~/.codex/config.toml`:

```toml
[mcp_servers.hotato]
command = "uvx"
args = ["--from", "hotato[mcp]", "hotato-mcp"]
```

## The scoring tool: `voice_eval_run`

Same two input modes as the CLI, all parameters optional beyond your chosen
mode:

| Parameter | Type | Default | Meaning |
| --- | --- | --- | --- |
| `stereo` | str | None | two-channel WAV path |
| `caller` | str | None | mono caller WAV (with `agent`) |
| `agent` | str | None | mono agent WAV (with `caller`) |
| `suite` | str | None | `"barge-in"` to run the bundled battery |
| `stack` | str | `"generic"` | livekit, pipecat, vapi, or generic |
| `expect` | str | `"yield"` | `"yield"` or `"hold"` |
| `onset_sec` | float | None | caller onset hint |
| `caller_channel` | int | 0 | caller channel index |
| `agent_channel` | int | 1 | agent channel index |
| `max_talk_over_sec` | float | None | pass threshold |
| `max_time_to_yield_sec` | float | None | pass threshold |
| `report_path` | str | None | also write the HTML report here; the envelope then carries `report_path` (absolute) |

Pass exactly one input mode: `stereo`, OR `caller` + `agent` together, OR
`suite`. Mixing modes (or passing none) returns a structured, parseable
error with a stable `error_code` and message; see below.

## The counterexample tools

These three tools call the same stdlib-only core as
`hotato counterexample ...`. `HOTATO_MCP_INPUT_DIR` confines scenario, test,
and capsule reads; `HOTATO_MCP_REPORT_DIR` confines new capsule writes; with
neither set, reads and writes stay under the working directory or the OS
temp directory. Compilation never overwrites an existing path.

| Tool | Scope | Does |
| --- | --- | --- |
| `counterexample_compile` | local write | reduces one failing scripted `hotato.scenario` + conversation-test target into a new private `.hotato-repro` capsule |
| `counterexample_verify` | read-only | audits source bytes, evaluator provenance, the replayed delete-only chain, the source-selected structured failure branch, and a completed local-minimality claim |
| `counterexample_reproduce` | read-only | runs only the reduced fixture under the current evaluator, permitting evaluator-version drift while still requiring the source-selected structured failure branch |

`counterexample_compile` takes `scenario_path`, `test_path`, `target`,
`out_dir`, optional `budget` (candidate evaluations), and optional `seed`.
The compiler selects the base scenario at that seed and does not expand
`variation_matrix`. `counterexample_verify` and `counterexample_reproduce`
each take one `path`.

The evidence status is `asserted`: this path operates on a deterministic
scripted simulation, not measured call audio. Every response still carries
the uniform control envelope; no tool publishes a capsule or promotes it
into a corpus.

## Output

On success: the identical envelope the CLI emits for `--format json`, schema
at [`https://hotato.dev/schema/envelope.v1.json`](https://hotato.dev/schema/envelope.v1.json)
(`schema_version` "1", additive-only -- new keys may appear, the documented
core never changes).

Every expected failure (missing / mono / mismatched / not-found file,
unknown suite, ambiguous input mode, or a well-formed input with no scorable
event) comes back as a structured error object, `ok: false`, with a stable
`error_code` and actionable `message` -- schema at
[`https://hotato.dev/schema/error.v1.json`](https://hotato.dev/schema/error.v1.json).
The MCP tool and CLI share this error shape: a caller, human or agent,
parses one contract across both surfaces.

## The fleet tools

Eleven tools drive the local, self-hosted fleet control plane
([`GUARDIAN-FLEET.md`](GUARDIAN-FLEET.md)): they read and reason over a
workspace, surfacing findings for a human to label and deploy.

| Tool | Scope | Does |
| --- | --- | --- |
| `fleet_status` | read-only | workspace counts + job-queue stats |
| `candidate_list` | read-only | top candidate moments awaiting a human label |
| `candidate_inspect` | read-only | one candidate's detail: onset, measured components (severity/input_health/recurrence/novelty/covered_by_contract), and trust findings |
| `contract_list` | read-only | contracts in a workspace |
| `trial_explain` | read-only | a recorded trial's verdict, evidence tier, and any pending human-gated action |
| `experiment_status` | read-only | a trial's current verdict, evidence tier, recommendation, and manifest hash |
| `artifact_verify` | read-only | verifies a contract bundle's authenticity + evidence on its own terms |
| `experiment_propose` | read-only | a bounded variant set with expected effects, ready for a human to clone, apply, or deploy |
| `experiment_create` | clone-scoped | precommit a trial manifest from a committed battery BEFORE any capture; capture and deploy stay separate, later steps |
| `experiment_run` | clone-scoped | recompute a before/after trial offline, entirely within the clone, and record a recommendation |
| `clone_cleanup` | clone-scoped | delete a STAGING clone an experiment created, scoped entirely to that clone |

## More

- Python API and the shared error contract: [`API.md`](API.md)
- What the envelope measures and the scope/ceiling: the top-level
  [`README.md`](../README.md) and [`METHODOLOGY.md`](../METHODOLOGY.md)
- Machine-readable index of every command (CLI and MCP): [`llms.txt`](../llms.txt)


================================================================================
FILE: docs/API.md
================================================================================

# Python API reference

Every function the CLI, pytest plugin, and MCP server use is a plain
Python function. The core is stdlib-only, offline, deterministic: same
audio and config, same envelope, every time. Import and score:

```python
from hotato.core import run_single, run_suite

env = run_single(stereo="call.wav", expect="yield")
env = run_suite()  # bundled 8-scenario battery
```

The top-level package re-exports the essentials (`from hotato import
run_single, run_suite, LIMITS, SUITE_ID, __version__`); every scoring
function takes keyword arguments only.

## The envelope

`run_single` and `run_suite` return the same machine-readable dict
(schema: `src/hotato/schema/envelope.v1.json`):

```python
{
  "tool": "hotato",
  "schema_version": "1",
  "mode": "single" | "suite",
  "stack": "generic",            # normalized stack label
  "offline": True,
  "engine": {"name", "version", "upstream"},
  "limits": {...},               # scope and ceiling, hotato.core.LIMITS
  "summary": {"events", "passed", "failed", "regression"},
                                 # plus additive "not_scorable" (count) when
                                 # at least one event could not be judged
  "events": [...],               # one dict per scored event, below
  "fix_map": [...],              # one entry per failing event with a fix
  "funnel": {...} | None,        # systemic pointer, fires only when both axes fail
  "exit_code": 0 | 1,            # 1 when any scorable event failed; the CLI
                                 # process exits 2 for a single recording that
                                 # is not scorable (see Exit codes below)
  "suite": "barge-in",           # run_suite only
}
```

Each event:

```python
{
  "event_id": str,               # file basename or scenario id
  "scenario_id": str | None,
  "title": str | None,
  "category": str | None,        # e.g. "should_yield"
  "expected_yield": bool,
  "verdict": {
    "passed": bool,
    "did_yield": bool,
    "seconds_to_yield": float | None,
    "talk_over_sec": float,
    "reasons": [str],            # failure reasons, empty on pass
  },
  "measurements": {
    "caller_onset_sec": float,
    "agent_talking_at_onset": bool,
    "hop_sec": float,
    "notes": str,
  },
  "signals": {                   # namespaced signal bus, additive
    "barge_in": {"did_yield", "time_to_yield_sec", "talk_over_sec"},
    "latency": {"response_gap_sec", "premature_start_sec"},
    "echo": {                    # every event; cross-channel coherence,
                                 # deterministic, computed in hotato's own
                                 # layer (hotato/echo.py), never _engine
      "coherence", "lag_sec", "echo_suspected",
    },
    "resume": {                  # only on events where the agent yielded
      "resumed", "resume_gap_sec", "restart_suspected",
    },
  },
  "fix": None | {                # set on failing events
    "fix_class": "config" | "engagement-control",
    "title": str, "detail": str,
    "knob": str | None, "pointer": str | None,
  },
}
```

An event without enough input to judge (a silent caller channel with no
onset label, or a should-yield expectation with the agent silent at onset)
carries `"scorable": False` and a plain `"not_scorable_reason"` instead. It
sits outside `passed`/`failed`, the fix router, and the funnel -- an input
problem, not a verdict -- so envelopes for valid recordings stay
byte-identical.

## hotato.core

### run_single

```python
run_single(
    *,
    stereo: str | None = None,        # two-channel WAV path
    caller: str | None = None,        # mono WAV path (with agent=)
    agent: str | None = None,         # mono WAV path (with caller=)
    caller_channel: int = 0,
    agent_channel: int = 1,
    onset_sec: float | None = None,   # caller onset hint, seconds from start
    expect: str = "yield",            # "yield" or "hold" (backchannel)
    stack: str | None = None,         # livekit | pipecat | vapi | generic
    max_talk_over_sec: float | None = None,
    max_time_to_yield_sec: float | None = None,
    echo_gate: bool = False,          # hold an echo-suspected yield out of
                                      # the verdict (scorable: false) instead
                                      # of counting it as a clean pass
    cfg: ScoreConfig | None = None,
) -> dict
```

Scores one recording and returns the envelope: provide `stereo`, or both
`caller` and `agent`. `expect="hold"` means the caller's speech is a
backchannel (a short acknowledgement like "mhm") that a correct agent
talks through. Both `max_*` thresholds tighten the pass criteria;
`echo_gate` is opt-in and off by default, though `signals.echo` always
computes. A malformed or truncated WAV raises a clean `ValueError` naming
the ffmpeg fix.

### run_suite

```python
run_suite(
    *,
    suite: str = "barge-in",          # the only suite id (SUITE_ID)
    stack: str | None = None,
    scenarios_dir: str | None = None, # your scenario JSON labels
    audio_dir: str | None = None,     # your recordings, <scenario-id><suffix>
    suffix: str = ".example.wav",
    caller_channel: int = 0,
    agent_channel: int = 1,
    echo_gate: bool = False,          # see run_single
    cfg: ScoreConfig | None = None,
) -> dict
```

Runs a labelled battery and returns the envelope with a `suite` key. The
bundled 8-scenario battery ships inside the package, zero external files;
point `scenarios_dir`/`audio_dir` at your own labelled set instead (e.g.
`corpus/suites/gold/scenarios` and `.../audio`). Suite audio must be
two-channel.

### dump_frames_for_input

```python
dump_frames_for_input(
    *,
    stereo: str | None = None,
    caller: str | None = None,
    agent: str | None = None,
    caller_channel: int = 0,
    agent_channel: int = 1,
    onset_sec: float | None = None,
    cfg: ScoreConfig | None = None,
) -> dict
```

The per-frame evidence behind every reported number: each channel's dBFS, VAD
activity, threshold, and noise floor, plus a self-describing `config` block.
Every signal it reports is re-derivable by hand from this dump.

### LIMITS and SUITE_ID

`hotato.core.LIMITS` is the scope dict embedded in every envelope: method,
ceiling, best input, and boundaries. `SUITE_ID` is `"barge-in"`.

### ScoreConfig

Every threshold is an exposed parameter:

```python
from hotato._engine.score import ScoreConfig
from hotato._engine.vad import VADParams

cfg = ScoreConfig(
    frame_ms=20.0, hop_ms=10.0,
    yield_hangover_sec=0.20,       # agent quiet this long = yielded
    max_search_sec=3.0,            # yield search window after onset
    caller_proximity_sec=0.5,
    turn_end_silence_sec=0.20,
    premature_tolerance_sec=0.05,
    onset_min_run_sec=0.05,
    agent_onset_lookback_sec=0.10,
    caller_vad=VADParams(),        # rel_db=15.0, abs_gate_db=-60.0,
    agent_vad=VADParams(),         # hangover_sec=0.15, noise_percentile=0.10,
)                                  # dyn_margin_db=22.0, backend="energy"
```

`VADParams.backend` is `"energy"` (the deterministic reference behind every
published number) or `"neural"` (an optional Silero VAD cross-check via
`pip install 'hotato[neural]'`; without the extra, a clean
`BackendUnavailable` names the missing dependency).

## hotato.report

Self-contained visual reports scored from the same measurements. All three
take the full scoring parameter set from `run_single`/`run_suite` above, as
keyword arguments.

```python
build_report_html(*, base: dict | None = None,
                  base_label: str | None = None, **kwargs) -> (str, dict)
build_report_md(*, base: dict | None = None,
                base_label: str | None = None, **kwargs) -> (str, dict)
write_report(path: str, fmt: str = "html", **kwargs) -> dict
```

`build_report_html` scores the input and returns `(html, envelope)`: one
self-contained file with inline CSS and SVG, per-event timelines, analytics,
a frame inspector, and print CSS for PDF. `build_report_md` mirrors it as
Markdown tables. `write_report` builds in `fmt` (`"html"` or `"md"`), writes
to `path`, and returns the envelope. Pass `base` (a prior envelope, e.g.
loaded from `hotato run --format json`) to render per-scenario regression
deltas; `base_label` names it on the page.

```python
from hotato.report import write_report

env = write_report("report.html", suite="barge-in", stack="livekit")
```

## hotato.aggregate

Team mode: many run envelopes, one trend view.

```python
load_run_dir(dirpath: str, order: str = "mtime") -> dict
    # {"runs": [{"file", "path", "mtime", "env"}], "skipped": [...], "order"}
    # order: "mtime" (oldest first) or "name" (numeric prefix = explicit index)

aggregate_runs(runs: list, order: str = "mtime",
               skipped: list | None = None) -> dict
    # team envelope: kind "team-aggregate", runs, events_total,
    # talk_over_sec / seconds_to_yield distribution summaries (mean/median/p90),
    # pass_rate {latest, first, mean, direction}, pass_rate_over_time,
    # failure_classes, most_common_failure_class, skipped, exit_code 0.
    # Raises ValueError with fewer than 2 runs; each trend point corresponds
    # to one input run.

build_team_section_html(agg: dict) -> str   # embeddable section
build_team_page_html(agg: dict) -> str      # full self-contained page
```

```python
from hotato.aggregate import load_run_dir, aggregate_runs, build_team_page_html

loaded = load_run_dir("runs/")
agg = aggregate_runs(loaded["runs"], order=loaded["order"],
                     skipped=loaded["skipped"])
html = build_team_page_html(agg)
```

## hotato.export

Research-grade flat files from the same scorer.

```python
run_export(
    *,
    out_dir: str,
    # plus the full scoring parameter set: stereo, caller, agent,
    # caller_channel, agent_channel, onset_sec, expect, stack, suite,
    # scenarios_dir, audio_dir, suffix, max_talk_over_sec,
    # max_time_to_yield_sec, cfg
) -> dict   # {"env", "events_rows", "frames_rows", "paths"}
```

Writes `events.csv` (one row per event, columns in
`hotato.export.EVENT_COLUMNS`), `frames.csv` (one row per VAD frame, columns
in `FRAME_COLUMNS`), and `envelope.json` into `out_dir` (created if
missing); `#` comment lines at the top of each CSV document the columns.
An empty cell means no value for that input; every filled cell is a direct
measurement.

## hotato.stackbench

Identical scenarios, your stack, comparable result files: every number is a
measurement of the recordings you provide.

### run_stackbench

```python
run_stackbench(
    *,
    stack: str,                       # one of BENCH_STACKS:
                                      # vapi | twilio | livekit | pipecat | generic
    recordings_dir: str,              # <scenario-id><suffix> WAVs
    scenarios_dir: str | None = None, # default: bundled battery
    suffix: str | None = None,        # default: auto-detected
                                      # (.wav, .stereo.wav, .example.wav)
    caller_channel: int = 0,
    agent_channel: int = 1,
    cfg: ScoreConfig | None = None,
) -> dict
```

Returns a result dict (`kind: "stack-benchmark"`) with the envelope fields
plus `config`, `scenarios {total, captured, not_captured}`, and `provenance`.
Scoring is unchanged `run_suite`; a scenario with no matching recording
lands under `not_captured`, out of the scoring and failure counts. The
timestamp derives from input file mtimes, so the same inputs always
reproduce the same file.

### load_result, compare_results, render_comparison_md

```python
load_result(path: str) -> dict
    # loads and validates one result JSON; anything else is a clean ValueError

compare_results(inputs: Sequence[tuple[str, dict]]) -> dict
    # inputs: (path, loaded_result) pairs, at least two.
    # Compares the intersection of scenarios scored in EVERY input;
    # the rest is listed under "skipped". Deltas are signed differences
    # against the FIRST input. Returns kind "stack-benchmark-comparison"
    # with inputs, compared, skipped, per_scenario, medians.

render_comparison_md(cmp_env: dict) -> str
    # the comparison as Markdown tables
```

```python
from hotato.stackbench import load_result, compare_results, render_comparison_md

cmp = compare_results([(p, load_result(p)) for p in ("a.json", "b.json")])
print(render_comparison_md(cmp))
```

## Pytest fixture

Installs automatically via the `pytest11` entry point (or load it explicitly
with `-p hotato.pytest_plugin`). Sits inert until your test calls it.

```python
def test_call_yields(hotato_score):
    env = hotato_score(stereo="call.wav", expect="yield")
    assert env["summary"]["regression"] is False
    assert env["events"][0]["verdict"]["seconds_to_yield"] < 1.0
```

`hotato_score(**kwargs)` takes the same keyword arguments as `run_single`;
pass `suite="barge-in"` (plus `run_suite` keywords) to score a battery
instead. It returns the envelope -- write your own assertions against it.

Session gate flags: `pytest --hotato-suite` runs the battery after your tests
and fails the session (exit 1) on a regression; `--hotato-suite-scenarios DIR`
and `--hotato-suite-audio DIR` point it at your own labelled set. Detail:
`docs/PYTEST.md`.

## `hotato.counterexample`

The counterexample API reduces one failing deterministic scripted scenario,
then emits a content-addressed replay capsule and a replayable deletion proof.
It never loads a provider adapter, model, network client, or subprocess.

```python
from hotato.counterexample import (
    compile_counterexample,
    verify_counterexample,
    reproduce_counterexample,
    inspect_counterexample,
    export_counterexample,
    predicate_counterexample,
)

compiled = compile_counterexample(
    "refund.scenario.json",
    "refund.test.json",
    target="refund-posted",
    out_dir="refund-posted.hotato-repro",
    workspace=".",
    budget=512,
)
assert compiled["exit_code"] in (0, 1)
```

`compile_counterexample` returns exit `0` only after the final
unit-deletion pass earns `one_minimal`. Exit `1` means the failure was
preserved and a runnable capsule written, but the evaluation budget ran
out before the proof completed. A refusal raises `CounterexampleRefusal`,
leaving no destination directory. Input parsing raises `ValueError` for
malformed or schema-invalid JSON/YAML-subset, `OSError` for I/O
failures -- both mapped by the CLI to its handled-error, exit-code
contract.

```python
verify_counterexample("refund-posted.hotato-repro")
reproduce_counterexample("refund-posted.hotato-repro")
inspect_counterexample("refund-posted.hotato-repro")
export_counterexample(
    "refund-posted.hotato-repro",
    out_dir="refund-posted.share",
)
predicate_counterexample("refund-posted.hotato-repro")
```

- `verify_counterexample` requires the recorded package version and evaluator
  source digest, then independently replays the source, the accepted
  delete-only chain, the final case, derived artifacts, and any completed
  one-minimal claim. The digest identifies shipped evaluator source, not
  interpreter or platform identity; replay hashes catch behavior changes.
- `reproduce_counterexample` permits evaluator drift and checks the reduced
  case twice for the source-selected structured failure branch. It targets
  Hotato evaluator/scenario regressions; it does not execute a deployed
  voice agent.
- `inspect_counterexample` verifies the closed member inventory, bound source,
  oracle and artifacts, and canonical human files without executing the scenario.
- `export_counterexample` first verifies the private capsule, then writes a
  non-runnable projection with content-bearing inputs omitted. Hashes remain
  correlators and may still be sensitive.
- `predicate_counterexample` returns `1` when the failure remains, `0` when it
  is absent, and `125` when the result cannot be used by `git bisect run`.

The v1 source is the base scripted scenario at the selected seed. A
`variation_matrix` is recorded as unapplied and may be removed during
reduction; compile a concrete scenario when a specific expanded run matters.
Full scope, artifacts, commands, and claim ceiling:
[`COUNTEREXAMPLES.md`](COUNTEREXAMPLES.md).

## MCP tool

`hotato-mcp` (or `python -m hotato.mcp_server`) speaks MCP over stdio: one
scoring tool (`voice_eval_run`, returning the identical envelope the CLI
emits), three counterexample tools, and eleven fleet tools -- full list and
scope in [`MCP.md`](MCP.md). Install: `uvx --from "hotato[mcp]" hotato-mcp`.

The parameters below are `voice_eval_run`'s, all optional:

| Parameter | Type | Default | Meaning |
| --- | --- | --- | --- |
| `stereo` | str | None | two-channel WAV path |
| `caller` | str | None | mono caller WAV (with `agent`) |
| `agent` | str | None | mono agent WAV (with `caller`) |
| `suite` | str | None | `"barge-in"` to run the bundled battery |
| `stack` | str | `"generic"` | livekit, pipecat, vapi, or generic |
| `expect` | str | `"yield"` | `"yield"` or `"hold"` |
| `onset_sec` | float | None | caller onset hint |
| `caller_channel` | int | 0 | caller channel index |
| `agent_channel` | int | 1 | agent channel index |
| `max_talk_over_sec` | float | None | pass threshold |
| `max_time_to_yield_sec` | float | None | pass threshold |
| `report_path` | str | None | also write the HTML report here; the envelope then carries `report_path` (absolute) |

## Exit codes and errors

Envelopes carry `exit_code`: 0 means every scorable event passed, 1 a
regression. `hotato.core.process_exit_code(env)` maps the envelope to the
CLI process exit: a single-recording run whose event isn't scorable
(silent caller with no onset label, or agent silent at onset; reason in
`not_scorable_reason`) exits 2 -- an input problem, not a verdict. Suite
runs count such events in `summary.not_scorable`, keeping their 0/1
semantics. Malformed input (bad WAV, out-of-range channel, negative onset,
unknown suite or stack) raises `ValueError`, surfaced as exit code 2: only
a file the scorer can fully read gets scored.


================================================================================
FILE: docs/SDK.md
================================================================================

# Python SDK (`hotato.sdk`)

`hotato.sdk` is a typed facade over the same functions the CLI runs. It imports the internal code paths directly, so there is no subprocess. Every result is a frozen dataclass whose fields are the keys of the JSON the CLI emits with `--format json`.

## Install

`pip install hotato` ships the SDK and its `py.typed` marker, so a type checker reads the package as typed. The scoring core is stdlib-only. Transcription uses the optional `hotato[transcribe]` extra.

## Run a suite

```python
from hotato.sdk import run_suite

result = run_suite()                  # the bundled labelled battery, zero files
print(result.passed, result.failed)   # process pass (exit 0) and failure count
for event in result.events:
    print(event.event_id, event.passed, event.seconds_to_yield, event.talk_over_sec)
```

Point `run_suite` at your own labelled set with `run_suite(scenarios="scenarios/", audio="audio/")`.

## Verify contracts

```python
from hotato.sdk import verify_contracts

result = verify_contracts("contracts/")
print(result.passed, result.summary["failed"])
for c in result.results:
    print(c.id, c.passed, c.authenticity)   # authenticity passes through verbatim
```

A regressed or tampered contract sets `passed` to `False` with `exit_code` 1. It returns as data; it does not raise. A missing or corrupt bundle raises `ValueError`.

## Compile a counterexample

```python
from hotato.sdk import compile_counterexample, verify_counterexample

compiled = compile_counterexample(
    "case.scenario.json", "case.test.json",
    target="pii-email", out="case.hotato-repro")
print(compiled.minimality, compiled.output)

verified = verify_counterexample("case.hotato-repro")
print(verified.status, verified.passed)
```

A deterministic refusal raises `CounterexampleRefusal`, which carries a stable `.code`.

## The JSON contract

The dataclass fields are the CLI's JSON keys. `SuiteResult`, `ContractVerifyResult`, `InvestigateResult`, `CounterexampleResult`, and `CounterexampleVerifyResult` mirror `hotato run`, `hotato contract verify`, `hotato investigate`, and `hotato counterexample`. A `SuiteResult.passed` is `exit_code == 0`, and `failed` is the count of failed scorable events. A NOT-SCORABLE or INCONCLUSIVE result keeps its own status; nothing is blended into a single number.

## Errors

Bad input raises `ValueError` and its subclasses. A missing optional extra raises `BackendUnavailable`. A counterexample refusal raises `CounterexampleRefusal`. All of these sit inside `hotato.errors.HANDLED`, the same set the CLI and MCP surfaces map to a structured error.

## Transcription

```python
from hotato.sdk import transcribe, build_transcript_cache

cache, _warning = build_transcript_cache()
transcript = transcribe("call.wav", cache=cache)
print(transcript.text)
```

A transcript is context beside the timing score. It leaves `did_yield`, `seconds_to_yield`, and `talk_over_sec` unchanged. For the cache-hit flag and the cache key, call `transcribe_cached` and read its `CachedTranscribeResult`.


================================================================================
FILE: docs/SUBMITTING.md
================================================================================

# Submitting a recording

The full path from call to merged corpus entry: record, label, validate,
submit. One consented, de-identified, human-labeled call sharpens this eval.

The scorer measures speech energy over time. Your label supplies the
meaning: which onset was a true bid for the floor, and whether a
well-behaved agent should yield to it. Energy isn't intent -- the label is
the ground truth, and it's what you contribute.

## 1. Record dual-channel

Caller on one channel, agent on the other, separated at capture -- two legs
of a SIP bridge, or two fully separate streams. This makes overlap a fact
of the recording, exact to the sample. Save a WAV at the call's native
sample rate (8000 Hz telephony, 16000 Hz wideband).

Consent, PII, and PHI rules live in
[`CORPUS-GOVERNANCE.md`](CORPUS-GOVERNANCE.md): documented consent from
every audible party, identifiers redacted with same-duration tone or
silence, no PHI. Read it before you record -- it has a reusable release
paragraph.

## 2. Label it

The label is a JSON file next to your WAV: the bundled scenario shape, plus
provenance and attestation. Spec: [`corpus/label.schema.json`](../corpus/label.schema.json).
Example: [`corpus/examples/sample-contribution.json`](../corpus/examples/sample-contribution.json).

Minimal example:

```json
{
  "id": "call-021-address-change-interrupt",
  "title": "Caller interrupts the shipping recap to change the address",
  "category": "should_yield",
  "source_type": "real-call",
  "audio": "call-021-address-change-interrupt.wav",
  "channels": { "caller_channel": 0, "agent_channel": 1 },
  "sample_rate": 8000,
  "duration_sec": 9.2,
  "caller_onset_sec": 3.15,
  "expected": {
    "yield": true,
    "max_time_to_yield_sec": 0.7,
    "max_talk_over_sec": 0.8
  },
  "license": "MIT",
  "attestation": {
    "contributor": "Your Name <you@example.com>",
    "consent_on_file": true,
    "pii_removed": true,
    "no_phi": true,
    "right_to_release_mit": true
  }
}
```

For a hold case (the caller backchannels "mm-hm" and a good agent keeps
talking), set `category` to `should_not_yield`, `expected.yield` to `false`,
and both `max_*` bounds to `null`.

Recording a role-play instead of a live call? The repeatable recipe -- script
shape, the consent/PII/attestation checklist mapped to the schema fields,
dual-channel capture, and the defect-performed-on-purpose labeling -- is
[`RFC-ROLEPLAY-FIXTURES.md`](RFC-ROLEPLAY-FIXTURES.md).

Label only what you can defend by hand: `caller_onset_sec` is required,
`reference_render` segment timings are a bonus if you have them. The
harness reports error only for the signals you supply.

## 3. Validate locally

```bash
python3 corpus/validate.py your_label.json
```

The WAV resolves next to the label via its `audio` field. If it lives
elsewhere, pass the audio path as a second argument:

```bash
python3 corpus/validate.py your_label.json your_recording.wav
```

PASS (exit 0) means the pair conforms: fields well-typed, category and
expected bounds consistent, timings in range, attestation affirmed, audio a
readable WAV with two or more channels matching the label. A human still
reviews consent and PII before anything merges.

## 4. Submit

Two doors, one intake:

- **Issue form.** Open
  ["Corpus submission: labeled recording"](https://github.com/attenlabs/hotato/issues/new?template=corpus_submission.yml).
  Attach the WAV as a .zip (GitHub rejects a raw .wav) or link it, and
  paste your label JSON in the description.
- **Pull request.** Add the label and WAV under [`corpus/`](../corpus/) --
  e.g. `corpus/call-021-address-change-interrupt.json` plus its `.wav`.
  State the source type in the PR body and confirm the attestation, per
  [`CONTRIBUTING.md`](../CONTRIBUTING.md).

## What maintainers do at intake

1. **Validate.** Re-run `corpus/validate.py` on the pair; check the label
   against [`corpus/label.schema.json`](../corpus/label.schema.json).
2. **Dedupe.** Hash the audio and compare call details against existing
   clips -- every clip earns its place.
3. **Normalize.** Slug the `id`, align filenames, confirm the channel map
   and declared sample rate. Any audio edit leaves the label's timing
   untouched.
4. **Add to a suite.** Register the clip so the benchmark harness scores it
   and reports millisecond error distributions and the yield confusion
   matrix ([`BENCHMARK.md`](BENCHMARK.md)).
5. **Credit.** `attestation.contributor` is the credit of record, and
   contributors are named in the changelog when their clip lands.

Removal requests are honored in the next release, per
[`CORPUS-GOVERNANCE.md`](CORPUS-GOVERNANCE.md).


================================================================================
FILE: docs/CORPUS-GOVERNANCE.md
================================================================================

# Corpus governance

The rulebook for contributing labeled voice-agent calls to hotato's corpus:
why they matter, consent, PII, and what gets published. See "Validity
metrics" at the end for what a report's numbers mean.

---

## Why real recordings matter

The synthetic fixtures in `src/hotato/data/` are a deliberately conservative
floor: rendered from known segment timings, so ground truth is exact and the
eval runs offline in seconds. They're the regression backbone -- proof the
scorer behaves as specified.

Production validity is a different question. Synthetic audio has clean
onsets, controlled overlap, and no room acoustics or codec artifacts beyond
what we inject. A scorer that looks flawless on synthetic fixtures still
needs proving on an 8 kHz call with echo bleed, cross-talk, and a caller
trailing off mid-word.

A small, carefully labeled corpus of real calls closes that gap: the
difference between "the tool does what the spec says" and "the tool measures
what happens on a phone line." The corpus stays small and high-integrity on
purpose -- every clip is consented, de-identified, and human-labeled. Ten
trustworthy calls beat ten thousand scraped ones.

---

## Scope and boundaries

- The corpus measures the **audio timing of turn-taking**: did the agent
  stop when the caller interrupted (yield), how long both talked at once
  (talk-over), and did the agent talk through a short "mhm" (backchannel
  handling).
- Labels are about **events in time**: did the agent stop, and when did the
  caller start.
- Everything here redistributes under the project's MIT license. A clip
  enters the corpus only if it can be released under MIT with consent.

---

## Consent / release template

Every party audible on a recording agrees to its inclusion, in a form you
can produce on request. Below: a short, reusable paragraph. Send it to the
caller and/or agent-operator, keep the signed or written reply on file, and
fill in the brackets.

> **Recording release, hotato open test corpus**
>
> I, [name or role], took part in the audio recording made on [date] and described
> as [short description]. I understand this recording will be used as a test
> fixture in *hotato*, an open-source project, and grant a perpetual, worldwide,
> royalty-free right to store, redistribute, and publish the recording and its
> derived timing labels as part of that project's MIT-licensed corpus. I confirm
> that the recording contains no confidential, personal, health, or account
> information, or that any such information has been removed or replaced before
> submission. I understand I can request removal of the recording from future
> releases at any time by contacting the maintainers.
>
> Signed / affirmed: [name], [date], [contact]

Notes:

- Get consent from **all** identifiable parties: the caller, and whoever
  runs the agent side, whether that's a person or a proprietary system
  whose owner must agree.
- Consent is **affirmative and documented**. A generic call-center "this
  call may be recorded" notice doesn't clear the bar for redistribution
  under an open license.
- Honor removal requests in the next release.

---

## PII policy

Audio is de-identified before it is committed, in order of preference:
**do not capture it, then remove it, then reject the clip.**

Strip or redact:

- **Names** of any individual (caller, agent, third parties mentioned).
- **Phone numbers**, extensions, and any dialed digits (DTMF tones included).
- **Postal and email addresses.**
- **Account identifiers**: customer numbers, order numbers, card numbers,
  policy numbers, ticket IDs, anything that keys back to a person or account.
- Any other free-text detail that, alone or combined, re-identifies someone.

Strongly preferred practice:

- Use **synthetic or role-played content**: actors reading a script hit the
  same turn-taking dynamics (interruption, correction, backchannel) with
  zero PII risk and a live call's acoustics.
- **PHI stays out of scope.** It disqualifies a clip regardless of consent.

Redaction guidance:

- Replace spoken PII with a tone or silence of the **same duration**, so the
  turn-taking timing survives -- cutting samples out shifts every onset
  after the edit and corrupts the labels.
- Redact **all channels**. A name muted on the mixed channel but audible on
  the caller channel is still present.
- Keep a private note of what was redacted and when (timestamps). The
  removed content itself is never committed.

---

## Data-handling policy

- **Self-hosted.** `hotato` reads and writes local files, offline;
  recordings stay on your machine. Contributing is a deliberate, manual
  act: committing a de-identified, consented clip.
- **Storage.** Corpus audio lives in the repository next to its scenario
  JSON, under the project's MIT terms. Keep working copies on your own
  machine until cleared for release -- never a third-party cloud bucket,
  shared drive, or chat tool.
- **Only the final clip ships.** The committed clip is redacted and
  consented; securing or destroying the un-redacted original is your own
  call.
- **Contributor attestation.** Each real-audio PR carries a short statement
  affirming: (a) consent is on file for every party, (b) PII is removed per
  the policy above, (c) no PHI, and (d) you have the right to release it
  under MIT -- part of the merge record.

---

## Validity metrics: what we publish

The corpus exists to quantify measurement quality, which shapes how results
are reported. We publish:

- **Per-signal measurement error, in milliseconds.** For each labeled call,
  we compare measured event times to human labels and report the error
  distribution per signal:
  - `time_to_yield`: signed and absolute error in ms between measured and
    labeled floor-yield.
  - `talk_over`: error in ms on measured overlap duration.
  - onset agreement: error in ms between measured caller onset and
    `caller_onset_sec`.

  We report these as distributions (median, spread, worst case), so a reader
  sees the shape of the error.
- **A `did_yield` confusion matrix.** Against the human `category`
  (`should_yield` / `should_not_yield`) label, we report four cells: correct
  yields, correct holds, false yields (yielded on a backchannel), and
  missed yields (talked over a real interruption). The two error cells are
  what operators feel, so we surface them directly.

The distribution and the four-cell matrix are the report: each failure mode
scored on its own, so the trade-off between them stays visible. The scorer
works on speech energy over time, so validity means timing agreement plus
decision agreement.

### Reading a corpus report

A healthy result: tight millisecond error distributions on `time_to_yield`
and `talk_over`, onset agreement within a small ms band, and a confusion
matrix whose off-diagonal (false-yield / missed-yield) cells are small and
individually inspectable. A poor result: wide error spread or a heavy
off-diagonal -- pointing you toward a concrete fix in the call.

Where a call fails, the fix map names candidate remedies, including a
learned engagement-control layer at that level. Audio alone is the weaker
modality for that fix class; the corpus measures how it holds up on the
room it was recorded in.


================================================================================
FILE: docs/BENCH-SPEC.md
================================================================================

# hotato bench: specification v0.1

hotato bench is a versioned freeze of the scenario batteries this repository
already ships, a scoring protocol over them, and a verify command that
re-executes a result and compares hashes. It is a benchmark you can hold in
your hand: a fixed set of labelled recordings, a fixed scorer, and a result
file whose numbers anyone can recompute from the same bytes. It is not a
leaderboard.

Run it:

```bash
hotato bench run --out bench-bundled.json          # the packaged battery
hotato bench run --suite gold --out bench-gold.json
hotato bench verify bench-gold.json                # re-execute + hash-compare
```

## Neutral harness

hotato is the measuring instrument, never a party to the comparison. This
repository publishes exactly two kinds of rows:

- rows measured on hotato's own reference renders (the frozen batteries
  below, scored as shipped);
- rows measured on fully open-source baseline agents, named and pinned by
  commit.

Rows for any vendor stack are adopter-published: the adopter who operates
that stack runs the bench, publishes the result in their own repository
under their own provenance block, and owns the claim. This repository never
authors a comparative row for a named vendor, and a pull request adding one
is declined.

## Scope: the frozen set

Bench v0.1 freezes the four tiered batteries under `corpus/suites/`
(112 scenarios; inventory in `corpus/suites/manifest.json`) plus the
packaged 8-scenario battery that installs with the package:

| Battery | Tier | Scenarios | Conditions | Suite content hash |
| --- | --- | --- | --- | --- |
| `bundled` | (package) | 8 | the packaged barge-in battery; runs from any install | `sha256:a02a4550c65fc0c8c897c523dac14d92bfb70b58cef6c8b1b70c59e2b6af8f5f` |
| `silver` | silver | 40 | clean, 16 kHz, default noise floor | `sha256:9ce2d70e9499880ca8b692796c4c7329dba8c15e6db579e2aa8ae7e8816e6c97` |
| `silver-defects` | silver | 16 | clean conditions, deliberate defect renders | `sha256:db948076fe9f617cbf1d901ef552a56dbb4858a4b2b7cd0f46e4b9244e079713` |
| `gold` | gold | 40 | noise floors, 8 kHz telephony, gain extremes, echo, edge timings, endurance | `sha256:f143781dbd1d09d3ca1bb1776ee0e2b2cf3c995f73e134aebedeee06575a307b` |
| `gold-defects` | gold | 16 | hard-condition defect renders, including two labelled capture-defect cases | `sha256:d1fd0bebb98650d51d6672861b4b29c9cbf3c061c46ef5ecd0bf0712388c2ea2` |

Every scenario is synthetic and says so: deterministic shaped noise rendered
from the exact segment timings in its own JSON (seed = `sha256(scenario_id)`),
so the timings are the ground truth and a fresh render is byte-identical to
the committed battery (`python3 corpus/suites/build_suites.py --check` is the
proof, and CI runs it). The defect batteries fail by design: they are the
negative control that makes the passing batteries mean something
(`docs/SUITES.md`).

A suite content hash pins the exact files a run consumes: one line per file,
`<relative_name>\0<file_sha256_hex>\n` over the sorted scenario JSONs and
their `<id>.example.wav` recordings, hashed with sha256
(`hotato.bench.suite_content_hash`).

## Versioning

- **Bench version** (`bench_version`, semver, currently `0.1`): the frozen
  set and the protocol together. Any change to any pinned file, to the audio
  suffix, to the hashed body shape, or to the hashing rules is a new bench
  version. The table above is the v0.1 pin list; a row change and a version
  bump land in the same commit.
- **Result schema** (`schema_version`, currently `1`): the shape of the
  result document. Additive-only within a major; a consumer must ignore
  unknown fields.
- **Scorer identity**: every result embeds the vendored engine identity
  (`engine`) and the full default `ScoreConfig` snapshot (`config`), so a
  result names the exact scorer and thresholds that produced it.

## Scoring protocol

Bench rows measure talk over on the hangover smoothed activity tracks;
relative to raw energy labels the end bias bound is `hangover_sec` plus one
hop (see docs/BENCHMARK.md, Quantization). This convention is part of the
frozen v0.1 protocol; changing it is a new bench version and a new engine
identity.

One result per battery, three measurements, side by side and never blended:

- **Pass counts** (`pass_counts`): scenarios, passed, failed, and
  not-scorable counts from the standard envelope
  (`hotato.core.run_suite` under the default shipped config). Results group
  by the battery's tier; a tier is a grouping label, never a weight.
- **Measurement-error distributions** (`error_stats_ms`): per signal
  (caller onset, time to yield, response gap), the n / median / mean / max /
  min of `|measured - rendered|` in milliseconds, against each scenario's
  own `reference_render` timings (`hotato.benchmark.run_benchmark`). A
  signal with no rendered ground truth reports `n: 0` and nulls, never a
  fabricated reference.
- **Confusion cells** (`confusion`): the four `did_yield` cells
  (correct_yield, missed_yield, false_yield, correct_hold) plus their
  off-diagonal sum.

There is no blended score. No field anywhere in a bench result aggregates
across these measurements, and no accuracy percentage appears; collapsing
the error distribution and the confusion matrix into one figure hides
exactly the missed-yield / false-yield trade-off the bench exists to
surface. This is code-enforced: the repo-wide `overall_score` rejection
(`hotato.errors.reject_overall_score`) applies to every nested section of a
bench result, and `bench verify` refuses a result that smuggles one in.

## Reproducibility

Byte-for-byte means: two executions of `bench run` on the same frozen
battery under the same package version produce result bodies whose
canonical JSON bytes are identical, so their sha256 addresses are equal.
Canonical JSON here is the form the rest of the repository already uses
(`hotato.manifest.canonical_json`): sorted keys, compact separators, ASCII,
finite numbers only. The result carries no wall-clock field, so its address
is stable across re-runs.

The whole path is offline: the batteries are local files (or packaged
data), the scorer is the vendored stdlib-only engine, and no code path in
`bench run` or `bench verify` touches the network. A machine with the
package and the frozen batteries can produce and check every number in a
result with no account, no key, and no service.

## Submission model: adopter-published rows

An adopter publishes a row by committing the evidence, not by sending a
score:

1. run the bench in their own CI: `hotato bench run --suite <name> --out
   results/<name>.json`;
2. gate the row on re-execution in the same pipeline: `hotato bench verify
   results/<name>.json` (exit 0 required);
3. commit the result files and a provenance block to their own repository,
   and render whatever badge they choose from their own CI status.

The provenance block published next to the rows carries, at minimum:

| Field | Content |
| --- | --- |
| `bench_version` | the bench semver the rows were measured under |
| `suite` / `suite_content_hash` | each battery name and its pinned hash, from the result |
| `result_content_hash` | each result's canonical sha256 address |
| `engine` | the vendored scoring-engine identity from the result |
| `config` | the `ScoreConfig` snapshot from the result |
| `runner` | OS, Python version, and CI system the rows were produced on |
| `commit` | the adopter repository commit the rows were produced from |
| `published_by` | who stands behind the row |

A row is the adopter's claim, verified by the adopter's own re-execution.
This repository links to adopter rows as published; it does not restate,
rank, or merge them.

## Anti-gaming

- **Verification is re-execution.** `bench verify` re-runs the pinned
  battery through the same code path `bench run` uses and compares the two
  canonical result addresses. The verdict is a hash comparison of two
  executions on the verifier's machine; no judge, no model, no service
  scores anything.
- **Edits are caught twice.** A result edited in place fails its own
  embedded `content_hash` and is refused. A result whose body and hash were
  both rewritten passes the integrity check and is then caught by
  re-execution, because the recomputed body hashes differently.
- **The frozen set is pinned.** A verify against a battery whose content
  hash differs from the one the result pins is withheld, not guessed: the
  comparison only ever runs like against like.
- **The batteries are public, and the pin makes that safe to say.** A stack
  tuned to the published scenarios still has to produce its numbers through
  the pinned scorer at the pinned config, and its provenance block names
  the exact config that produced every number.
- **The held-out check is a manual maintainer spot check.** A maintainer
  re-renders the frozen batteries from the generator, re-executes a
  published row by hand on maintainer hardware, and hash-compares. There is
  no hosted scoring server to game: every verdict, including the spot
  check, is a local re-execution.


================================================================================
FILE: docs/BENCHMARK-STACKS.md
================================================================================

# Benchmarking voice stacks with hotato

`hotato benchmark` runs one fixed scenario battery through your configured
voice stack, scoring every stack against the same scenarios, labels, and
thresholds -- so result files line up directly, stack against stack.

Every number comes from the recordings you capture, scored the same way
for every stack -- a benchmark result describes your setup, on the day you
captured it.

## 1. Pick the scenario set

The bundled 8-scenario barge-in battery is the default. Any scenarios
directory works the same way, including the larger tiered suites:

```
corpus/suites/gold/scenarios
corpus/suites/silver/scenarios
```

Each scenario JSON carries the caller transcript, the caller onset timing,
and the pass thresholds; the corpus suites also ship each scenario's
caller stimulus as `audio/<id>.caller.wav`.

## 2. Capture the battery through your stack

For every scenario, drive the caller side through your stack and record
the call dual-channel (caller on channel 0, agent on channel 1):

- Corpus suites: play the shipped `<id>.caller.wav` stimulus into your
  stack. For the bundled battery, reproduce the caller side from each
  scenario's transcript and `caller_onset_sec`.
- `hotato setup --stack livekit` or `--stack pipecat` prints the exact
  dual-channel recording scaffold for your infra.
- `hotato capture --stack vapi --call-id <id>` or
  `hotato capture --stack twilio --recording-sid RE...` pulls the finished
  dual-channel recording for you.

Name each recording by its scenario id, one directory per stack:

```
captures/livekit/01-hard-interruption.wav
captures/livekit/02-backchannel-mhm.wav
...
captures/vapi/01-hard-interruption.wav
```

## 3. Run the benchmark

```bash
hotato benchmark --stack livekit --recordings captures/livekit --out livekit.json
hotato benchmark --stack vapi    --recordings captures/vapi    --out vapi.json
```

Each result JSON records the stack, the scenario set, every measured
signal (the same events `hotato run` produces), the exact scoring
thresholds, and a provenance block: who ran it, on which files, with each
file's mtime. The timestamp comes from the input files, not the wall
clock, so the same inputs reproduce the same result.

Scenarios without a matching recording land under `not_captured`, kept out
of both the scoring and the failure count. Exit code is 0 when the run
completes; add `--fail-on-regression` to exit 1 when a scored event fails
its thresholds.

## 4. Compare stacks

```bash
hotato benchmark compare livekit.json vapi.json
```

Side by side, per scenario: yielded, talk-over seconds, and time-to-yield
seconds for each input, with signed deltas against the first file, plus
summary medians. When the files cover different scenario sets, only the
intersection is compared; the rest is listed as skipped, with the files
each was missing from.

## What the numbers are

Reproducible timing measurements of the recordings you provided, scored by
the same engine as `hotato run`, with every threshold exposed in the
result. They show how your configuration of a stack handled the battery
you captured.


================================================================================
FILE: docs/BENCHMARK.md
================================================================================

# Benchmark methodology

`hotato` ships a reproducible measurement-error harness,
`src/hotato/benchmark.py`. Given labelled dual-channel recordings, it
reports how far the scorer's measured event times land from the rendered
or hand-labelled ground truth, and how its yield/hold decisions line up
with the human labels. That is the whole output.

```bash
PYTHONPATH=src python3 -m hotato.benchmark
```

This runs on every synthetic fixture in the checkout (the bundled battery,
the `examples/` reference set, and the deliberately-bad `funnel-demo` set),
prints a markdown table, and writes `benchmark-report/measurement-error.json`
and `benchmark-report/measurement-error.md`.

The harness is a standalone measurement tool you run against fixtures; the
`hotato` CLI scores a single call or the battery.

**Scope.** This benchmark measures the timing signals: per-signal
measurement error against rendered or hand-labelled ground truth, and the
yield/hold decision. Say-do verification is scored on its own deterministic
lane, evaluated against recorded trace spans; its methodology is the
"Say-do verification methodology" section of
[`METHODOLOGY.md`](../METHODOLOGY.md), with
[`examples/reference-agent`](../examples/reference-agent) as the runnable
worked example.

---

## What it measures

For every `(recording, label)` pair it reports two things.

### 1. Per-signal measurement error, in milliseconds

For each timing signal it computes `|measured - rendered|` in ms, where
`rendered` is either the exact value the fixture was rendered from (its
`reference_render` block) or a contributor's hand label:

- **caller onset**
  - Measured: onset the VAD detects (the scorer is given no onset label).
  - Rendered: start of the first caller segment.
- **time to yield**
  - Measured: seconds from caller onset to the agent going quiet.
  - Rendered: the agent's in-progress turn end minus the caller onset.
- **response gap**
  - Measured: endpointing dead-air from the caller's turn end to the
    agent's next onset.
  - Rendered: the fixture's rendered response gap.

Errors are reported as a distribution: median, mean, worst case, best case,
and n. A signal scores only where a reference exists and the scorer
produced a value; missing either yields a `-`. (The echo-of-agent fixture
has no independent caller speech, so its onset/yield error is a gap by
construction.)

Onset is measured in **detect mode** (the scorer gets no onset hint), so it
tests the onset detector directly. Yield, talk-over, response gap, and
`did_yield` are measured in **label mode** (the scorer is given the human
`caller_onset_sec`, as the shipped battery runs) -- those are the numbers a
user sees.

### 2. A `did_yield` confusion matrix

Against the `should_yield` / `should_not_yield` label (each scenario's
`expected.yield`), it reports the four cells:

|  | measured **did_yield** | measured **held floor** |
|---|---|---|
| **should_yield** | correct yield | **missed yield** |
| **should_not_yield** | **false yield** | correct hold |

The two off-diagonal cells -- missed yields and false yields -- are the
failures an operator feels, so the report surfaces them directly.

---

## Why milliseconds and a matrix

The report pairs a per-signal error distribution with a four-cell confusion
matrix and keeps the cells separate: a missed yield and a false yield are
different failures with different fixes, so averaging them into one number
would hide which one you have. `docs/CORPUS-GOVERNANCE.md` ("Validity
metrics") enforces the same rule for corpora; the benchmark applies it to
the tooling itself.

The reported error is what the default shipped config measures, so it is
the number you get. On the synthetic fixtures, the yield error equals the
exposed VAD hangover, and the onset/gap error is one frame hop -- both
documented `ScoreConfig` parameters. Set the hangover to zero
(`caller_vad.hangover_sec = agent_vad.hangover_sec = 0`) and every signal
collapses to within one hop of ground truth; the test suite asserts this,
so the claim is checkable against the code.

The scorer reads speech energy over time, so that is what the harness
reports: timing and decisions.

---

## Quantization: every reported time has a resolution floor

Every timing signal the scorer reports is quantized to the frame hop
(`ScoreConfig.hop_ms`, default `10.0` ms) plus, for yield/talk-over, the VAD
hangover (`caller_vad.hangover_sec` / `agent_vad.hangover_sec`, default
`0.15` s): the measured value can land up to one hop off the true event,
purely from where that event falls inside a 10 ms frame, before hangover is
even counted. This sub-frame-phase rounding is deterministic -- the same
one-hop collapse the section above already pins down (hangover zero ->
every signal within one hop of ground truth).

Against a label placed at the raw end of speech energy, the measured end of
an overlap or yield sits late by at most `hangover_sec` plus one hop
(0.16 s at defaults) and the measured start sits early by at most one frame
(0.02 s at defaults). The bias is deterministic and one sided. Setting
`caller_vad.hangover_sec` and `agent_vad.hangover_sec` to 0 removes the
hangover term and leaves frame quantization; on recorded speech a zero
hangover can fragment one utterance at intra word dips, which lowers
measured overlap. Measured case: a two channel recording of two human
speakers with a hand labeled 0.420 s overlap at the raw speech edges
measures `talk_over_sec = 0.590` at defaults and `0.420` at hangover zero,
the bound holding exactly (end +0.155 s, start -0.005 s).

The consequence for a `--max-time-to-yield` (or any) policy bound: a
physically identical yield event, shifted by a few milliseconds of
sub-frame phase with nothing else changed, can cross an exact bound purely
from quantization. Reproduced case: a 250 ms yield event against a 400 ms
bound flips PASS/FAIL as the event's sub-frame phase sweeps through 3, 6,
12, and 16 ms offsets, the underlying event unchanged -- the measured value
moves by exactly one hop (10 ms) at each transition, no further. Phase
alone can flip a bound set within one hop of the true value either way, on
that recording.

**Read policy bounds accordingly: a margin under one hop (10 ms default)
from the true value sits inside the scorer's quantization noise.** `hotato`
surfaces that margin plainly (see `docs/FIX-PLANS.md`'s no-single-threshold
rule, extended here to quantization); set bounds at least one hop from any
value you need to hold.

---

## Noise floor and the verdict cliff

Below a measurable per-channel SNR the verdict flips rather than degrades.
The mechanism is one line of the energy VAD: the speech threshold is capped
at the channel's loudest frame minus `dyn_margin_db` (22 dB), so once the
noise floor climbs inside that margin every frame reads active, agent
activity never ends, and a correct yield scores as a false 3.0 s talk-over.

Measured on the reference fixture (`01-hard-interruption`, seeded noise
added to both channels): with uniform noise the yield verdict flips between
19 and 18 dB per-channel SNR; with babble-shaped noise, between 21 and
20 dB. The hardest shipped pass tier (the gold noise family) bottoms out at
a 23.8 dB noise floor, about 5 dB above the cliff.

The opt-in scorability gate covers the band below: `hotato run
--snr-gate-db` (bare flag = 22.0, which equals `dyn_margin_db`, the
geometric constant of the cliff, not a tuned number) estimates each
channel's stationary SNR deterministically and refuses to score
(not-scorable, exit 2, reason `low-snr`) when either estimate falls below
the floor, instead of emitting the false talk-over. A gated run carries the
per-channel `snr_estimate` block on the event.

The estimate certifies a stationary noise floor. Strongly non-stationary
noise (babble) can flip a verdict while estimating above the floor: the
babble flip sits 2 dB above the uniform flip on the curve above. Read a
gated pass as "the stationary floor clears the margin", and treat heavy
competing speech as its own capture problem.

---

## Reproducing it

```bash
# render the synthetic fixtures deterministically (sha256-seeded; byte-identical
# for a fixed hotato version -- verified in CI on Linux x86_64, Python 3.10,
# 3.11, and 3.12 -- see .github/workflows/tests.yml), then run the harness over them
python3 examples/render_examples.py
PYTHONPATH=src python3 -m hotato.benchmark
```

The report carries no wall-clock timestamp, and the render is
deterministic, so two runs on the same code produce byte-identical
artifacts. The JSON carries a `config` snapshot of every threshold behind
the numbers, so a reader can re-derive any value. `tests/test_benchmark.py`
pins all of it: the harness runs, the confusion matrix matches known
rendered behaviour, and every ms-error sits within its known,
config-derived tolerance.

The synthetic fixtures are a floor: deterministic rendered audio with exact
known timings, showing the scorer behaves as specified and guarding
against regressions. See `examples/README.md` and `docs/CORPUS-GOVERNANCE.md`.

---

## Extending to your own recordings (bring your own labelled data)

The synthetic floor shows the scorer does what the spec says. A labelled
recording from a live phone line shows what it measures under production
conditions. The harness runs on your own labelled data with no code
change:

```bash
PYTHONPATH=src python3 -m hotato.benchmark \
  --scenarios path/to/your/scenarios \
  --audio     path/to/your/audio \
  --name      my-corpus \
  --out       my-benchmark-report
```

- `--scenarios` is a directory of scenario JSON labels, same shape as
  `src/hotato/data/scenarios/*.json`. Each needs an `id`, an `expected.yield`
  label, and whatever reference timings you can defend: a `reference_render`
  with segment timings (as the synthetic fixtures carry), or at minimum a
  hand-labelled `caller_onset_sec` plus the true event times to score error
  against. Supply a reference only where you have ground truth -- the
  harness reports error for those signals and leaves the rest blank.
- `--audio` is a directory of dual-channel `<id>.example.wav` recordings (caller on
  channel 0, agent on channel 1). Record two physically separated channels whenever
  you can: that is what makes overlap ground-truthable.

When `--scenarios` is given, only that set is scored -- a clean report for
your corpus alone.

The path to a number from your own recordings is manual and consented:
bring your own labelled audio, governed by `docs/CORPUS-GOVERNANCE.md`
(consent, PII, data-handling) and the pipeline in `corpus/` (a labelling
schema and a validator). Synthetic fixtures keep their synthetic label.


================================================================================
FILE: docs/CALLER-LOAD.md
================================================================================

# Caller-program load runner

`hotato.caller_load` executes the same bounded caller program many times without
discarding the evidence from any started invocation. It is the load plane for
Hotato's caller engine, rather than a shortcut that treats an HTTP response or
a completed call as a quality pass.

The runner supports two workload models:

- **closed concurrency** keeps up to `concurrency` caller invocations active
  until an exact call count has been attempted;
- **open arrival** schedules starts against a monotonic arrival clock. When the
  target rate exceeds `max_in_flight`, the scheduled start is recorded as
  `DROPPED`. The scheduler does not wait and silently shift that call later.

Every started invocation runs in its own operating-system process and writes a
normal `hotato.caller-package.v1` package. The aggregate package binds each
child to the workload-plan hash, stage ID, stage model, and invocation index.
This lets an offline verifier detect a child copied from another stage or run.

## Minimal plan

```json
{
  "schema": "hotato.caller-load-plan.v1",
  "id": "refund-release-candidate",
  "caller_plan": {
    "schema": "hotato.caller-plan.v1",
    "id": "refund-caller",
    "mode": "scripted",
    "start": "ask",
    "nodes": [
      {"id": "ask", "type": "say", "text": "Please refund order 42.", "next": "done"},
      {"id": "done", "type": "hangup", "reason": "scenario_complete"}
    ],
    "limits": {"max_duration_ms": 30000, "max_cost_microusd": 0}
  },
  "stages": [
    {"id": "warm", "model": "closed", "concurrency": 4, "calls": 20},
    {
      "id": "arrival-spike",
      "model": "open",
      "arrival_rate_per_second": 10,
      "duration_seconds": 30,
      "max_in_flight": 50
    }
  ],
  "safety": {
    "max_calls": 320,
    "max_concurrency": 50,
    "max_call_duration_ms": 30000,
    "max_start_delay_ms": 250,
    "max_cost_per_call_microusd": 0,
    "max_cost_microusd": 0,
    "stop_file": "./STOP-HOTATO-CALLER-LOAD"
  },
  "slos": {
    "max_dropped_start_rate": 0.01,
    "min_completion_rate": 0.99,
    "max_blocked_error_rate": 0.01,
    "min_child_verification_rate": 1.0,
    "min_evidence_complete_rate": 0.99,
    "max_p95_scheduling_delay_ms": 100
  }
}
```

Open stages gain a normalized, read-only `calls` field equal to the number of
scheduled arrival slots. The stored `workload-plan.json` contains that
normalized value.

## Adapter boundary

The runner does not put credentials in the workload plan. Supply credentials
through a factory closure or a secret manager. Every API call declares its
session-factory transport scope; Hotato cannot infer what arbitrary Python
factory code opens.

```python
from hotato import caller_load
from my_project.sessions import new_session

plan = caller_load.load_plan("caller-load.json")

def session_factory(run_context):
    # run_context is safe to attach to transport receipts and traces.
    # It contains child_id, workload_plan_sha256, stage_id, stage_model,
    # invocation_index, and expected_session_boundary. Fetch credentials
    # outside the plan.
    return new_session(
        target_url="ws://127.0.0.1:9000/voice",
        token_from_secret_manager=True,
        evidence_labels=run_context,
    )

run = caller_load.run_caller_load(
    plan,
    "./hotato-caller-load-run",
    session_factory,
    execution_scope="local",
)
raise SystemExit(run.exit_code)
```

`model_factory(context)` and `tts_factory(context)` use the same contract and
are optional. A `generative` caller plan requires `model_factory`. Each factory
runs inside the child process. On platforms whose multiprocessing runtime uses
`spawn`, factories must be pickleable module-level callables.

Session and model credentials can still leak if an adapter deliberately writes
them into its `evidence()` output. Adapter authors must return receipts and
content hashes, never bearer tokens, authorization headers, or signed URLs.

## Safety contract

Validation happens before a child process starts:

- scheduled calls cannot exceed `safety.max_calls`;
- stage concurrency cannot exceed `safety.max_concurrency`;
- the supervisor terminates a worker after `max_call_duration_ms` and writes a
  separate verified `ERROR` child package for that invocation;
- caller-plan duration and model-cost ceilings are narrowed to the workload
  ceilings for each child;
- `scheduled_calls × max_cost_per_call_microusd` must fit inside
  `max_cost_microusd` before execution;
- the stop file prevents every subsequent scheduled start. Those rows are
  `STOPPED`, not silently omitted, and the aggregate status is `INCONCLUSIVE`.

The cost bound depends on model adapters reporting usage correctly to the
caller engine. A provider with unknown or unbounded spend must not be assigned
a zero cost ceiling. On timeout, the worker attempts `session.hangup()` and an
optional `session.close()` before the supervisor escalates from termination to
a process kill. Carrier/provider billing and teardown remain external evidence;
an adapter that ignores teardown cannot be made cost-bounded by a local process
limit alone.

## Coordinated-omission evidence

Each scheduled row records:

- its fixed `scheduled_offset_ms`;
- observed `scheduling_delay_ms`;
- `STARTED`, `DROPPED`, or `STOPPED` disposition;
- `MAX_IN_FLIGHT` or `START_DELAY` for a dropped open-arrival start.

Open-arrival capacity pressure is therefore visible as dropped work. The
scheduler does not convert a 10 calls/second workload into a slower closed loop
whose latency distribution looks healthier.

## Separate result lanes

The aggregate and every stage report these values independently:

1. `scheduled`, `started`, `dropped_starts`, and `stopped_before_start`;
2. caller status counts: `completed`, `hung_up`, `blocked`, and `error`;
3. caller child-package `verified` and `unverified` counts;
4. delivered-evidence states: `present`, `missing`, `unsupported`, and
   `unobservable`;
5. remote session-endpoint bindings: `matched`, `missing`, `mismatch`, and
   `not_required`;
6. scheduling-delay and invocation-duration distributions;
7. reported model cost in micro-USD.

`COMPLETED` and `HUNG_UP` mean the caller program reached a terminal state.
They do not mean the target agent produced the right outcome. `PRESENT`
delivery evidence means the session explicitly reported target-bound evidence;
it is also not an outcome verdict. The schema fixes both statements and fixes
`blended_score` to `null`.

## Offline verification

```python
from hotato.caller_load import verify_caller_load

verification = verify_caller_load("./hotato-caller-load-run")
assert verification["ok"], verification["mismatches"]
```

Verification performs no network, model, TTS, session, or provider call. It:

- uses bounded no-follow reads and refuses FIFOs and symlinks;
- verifies the full aggregate file manifest and exact package layout;
- verifies every started child with `caller.verify_package`;
- rebuilds the expected child caller plan and checks its workload/stage/index
  identity;
- recomputes child status, exit code, evidence state, and reported cost from the
  child package;
- recomputes aggregate and per-stage metrics, every SLO, aggregate status, and
  exit code;
- rejects rehashed status/exit forgeries, child swaps, and extra files.

Exit codes are `0` for `PASS`, `1` for `FAIL`, and `2` for `INCONCLUSIVE`.
Only declared SLOs gate `PASS`/`FAIL`; the runner never invents a blended score
or an undeclared quality threshold.

`delivery_evidence=PRESENT` has a narrow contract. Credit can come from either
an exact `session_boundary.delivery_evidence` receipt or an exact
`hotato.delivered-audio.v1` custom child event. Both forms require an authority
of `target_boundary`, `target_participant_reported`, or `carrier_boundary`, the
workload and child IDs from the supplied run context, and both
`submitted_sha256` and `delivered_sha256`. The submitted digest must match PCM
emitted by that child; every emitted PCM digest must be covered. The event form
is documented in `CALLER-SIDECAR-PROTOCOL.md`. A malformed claim, generic
digest, configured impairment hash, SDK submission receipt, partial receipt
set, or summary state cannot earn delivery credit. Those cases remain
`MISSING` or `UNOBSERVABLE`. This is reporting-boundary evidence, not a packet
trace or proof about a later uninstrumented hop.

Remote sidecar load is default-deny. `--allow-remote` is accepted only when
the normalized workload plan contains an exact credential-free `wss://`
endpoint, a remote call ceiling covering the schedule, the fixed
`I_ACCEPT_REMOTE_CALL_SIDE_EFFECTS_AND_UNOBSERVABLE_EXTERNAL_COST`
acknowledgement, and `external_cost_state=UNOBSERVABLE`. Queries and embedded
credentials are refused. The result separates the plan-declared endpoint
digest from the runtime-configured endpoint digest. Every started child also
must return the same `connected_endpoint_sha256` from its successfully
constructed session boundary. `metrics.session_endpoint_binding` exposes
matched, missing, and mismatched children. Any missing or mismatch makes the
aggregate `INCONCLUSIVE` with exit code 2, even when quality SLOs pass; the
offline verifier recomputes that state from each child. The result records
external provider cost as `null`/`UNOBSERVABLE`; `actual_cost_microusd` and its
bound cover caller-model-reported cost only. A remote provider bill never
silently appears as zero.

## Package layout

```text
hotato-caller-load-run/
├── workload-plan.json
├── observations.jsonl
├── result.json
├── package-manifest.json
└── children/
    └── <child-id>/
        ├── caller-plan.json
        ├── caller-result.json
        ├── package-manifest.json
        └── artifacts/...
```

The aggregate manifest hashes every byte-bearing file, including each child
manifest. Child packages retain the caller engine's model, TTS, text, PCM,
event, session-boundary, and authority evidence.

## Scope boundary

This module exercises caller programs against a supplied session adapter. It
does not provision SIP trunks, carrier capacity, media relays, speech models,
or production alerting. Those components remain independently observable so a
provider lifecycle response cannot substitute for target-delivered media or a
target outcome.


================================================================================
FILE: docs/CALLER-SIDECAR-PROTOCOL.md
================================================================================

# Caller sidecar protocol

`WebSocketCallerSession` is the concrete boundary between Hotato's bounded
caller program and a Pipecat, LiveKit, SIP, or provider adapter. The WebSocket
subprotocol is `hotato.caller.v1`. Loopback is the default; remote egress is an
explicit option and requires `wss://`.

The client sends a `hotato.caller-session.v1` hello with a random nonce and the
complete operation list. The sidecar returns a nonce-bound `ready` message,
adapter name/version, and one of `SUPPORTED`, `UNSUPPORTED`, or `UNOBSERVABLE`
for every operation. An omitted capability refuses the session.

Commands are canonical JSON envelopes with a contiguous sequence. PCM16LE is
announced by a `send_audio` command containing byte count and SHA-256, followed
by one binary message: the four bytes `HTC1`, a big-endian 32-bit command
sequence, and the exact PCM bytes.

Every command is synchronous at the protocol boundary. After accepting the
complete command (including the binary frame for `send_audio`), the sidecar
must return the matching sequence and command:

```json
{
  "schema": "hotato.caller-session.v1",
  "type": "command_result",
  "sequence": 4,
  "command": "send_audio",
  "status": "completed",
  "receipt": {
    "accepted_bytes": 6400,
    "accepted_sha256": "sha256:..."
  }
}
```

Status is `completed`, `unsupported`, or `error`. An absent, mismatched, or
failed result stops the caller program. Events that arrive while a command is
pending are retained in order for the next `receive` node; any still queued at
the terminal node are drained into `caller-result.json.events`. A command result
establishes that the sidecar accepted the operation. It does not establish
that a carrier or target agent received the media; that stronger claim needs a
target-boundary event carrying the delivered stream hash.

For caller-load delivery credit, the target or carrier boundary emits this
exact custom event (the caller engine adds `sequence` and `event_sha256`):

```json
{
  "schema": "hotato.caller-session.v1",
  "type": "event",
  "event": {
    "kind": "custom",
    "custom_type": "hotato.delivered-audio.v1",
    "authority": "target_boundary",
    "submitted_sha256": "sha256:<64 lowercase hex>",
    "delivered_sha256": "sha256:<64 lowercase hex>",
    "workload_child_id": "<run_context.child_id>",
    "workload_plan_sha256": "<run_context.workload_plan_sha256>"
  }
}
```

`authority` is exactly `target_boundary`, `target_participant_reported`, or
`carrier_boundary`. The submitted digest must identify PCM emitted by that
child. Every emitted PCM digest needs a valid receipt before the aggregate
reports delivery evidence `PRESENT`. Unknown fields, malformed identities,
replayed child/workload bindings, or partial coverage remain `MISSING`.
`delivered_sha256` identifies bytes observed by the reporting boundary; it is
not a packet trace or evidence about an uninstrumented downstream hop.

Incoming data is a JSON event envelope:

```json
{
  "schema": "hotato.caller-session.v1",
  "type": "event",
  "event": {"kind": "transcript", "text": "How can I help?"}
}
```

The caller engine accepts bounded transcript, tool-result, state-snapshot,
DTMF, lifecycle, transfer, hold, timing, timeout, and custom events. Sidecar
events are adapter evidence. A configured impairment receives delivery credit
only when the target boundary emits evidence for the bytes it received.

Handshake headers may carry credentials, but values are resolved inside the
worker and never serialized into multiprocessing factories, control messages,
or caller packages. Plaintext `ws://` is accepted only for loopback targets.
The strict WebSocket client refuses
redirects, extensions, remote addresses unless opted in, oversized messages,
invalid framing, and malformed JSON.

After the nonce-bound ready exchange, the session records
`connected_endpoint_sha256`, computed from the normalized credential-free
WebSocket URL. This binds remote load children to the configured endpoint. It
does not prove media delivery.


================================================================================
FILE: docs/CARDS.md
================================================================================

# Cards: one measured moment, rendered for the pull request

`hotato card` turns a machine result into a self-contained SVG for a pull
request or an issue. The card names the measured timing moment: a
reproducible measurement, not a verdict about intent, with no accuracy
number and no blended score anywhere on it. The PR-native artifact the
shipped Action lands automatically is the
[Failure Record](#the-failure-record-the-artifact-that-lands-in-the-pull-request)
below; a card is the single-moment image you attach yourself.

```bash
hotato card INPUT[#REF] --out card.svg
```

Everything runs locally. The SVG is a pure function of the input JSON alone
(the same input renders the same bytes forever), with every color inlined
and no font, image, stylesheet, script, or link to fetch. It renders
wherever the PR renders; no CDN or asset host required.

## The cards, auto-detected

The input's kind decides the card; you do not pick.

| Input | Card |
|---|---|
| a `sweep`/`analyze` candidate ref, `FILE#N`, that is a talk-over moment | **talk-over candidate** |
| a `sweep`/`analyze` candidate ref, `FILE#N`, that is a false-stop moment | **false-stop candidate** |
| a fix plan whose `decision` is `do_not_tune_single_threshold` | **threshold funnel** (the hero) |
| a supported `hotato verify` before/after rollup that improved | **paired comparison** |
| a `hotato contract create` contract (kind `voice-turn-taking-contract`) | **failure contract** |
| a `hotato test run` result (kind `hotato.test-run`) whose tool/state evidence failed a declared outcome | **say-do failure** |

`#N` is the same 1-based rank the sweep report and dashboard show, and the
same ref `hotato fixture promote` takes -- a card and a fixture speak of
the exact same moment.

### A. Talk-over candidate

An `overlap_while_agent_talking` (or `agent_start_during_caller`) moment:
the agent kept the floor while the caller was speaking. The card leads with
the measured overlap in seconds and closes with "Hotato reports timing
candidates, not intent."

```bash
hotato sweep --demo --format json > hotato-sweep.json
hotato card hotato-sweep.json#3 --out talk-over.svg
```

### B. False-stop candidate

An `agent_stop_no_caller` moment: the agent went quiet with no caller
nearby to explain the drop. The card leads with the measured trailing
silence.

```bash
hotato card hotato-sweep.json#1 --out false-stop.svg
```

### C. Threshold funnel (the hero)

The plan the both-axes case produces: the battery missed an interruption
**and** false-stopped on a backchannel, so satisfying both needs more than
a single sensitivity dial. The card states Hotato refused threshold
tuning and names the fix class (`engagement-control`) -- the card the
project leads with.

```bash
hotato demo --format json > demo.json
hotato plan demo.json --out fix-plan.json
hotato card fix-plan.json --out no-single-threshold.svg
```

Only a `do_not_tune_single_threshold` plan renders this card; any other
plan is a clean exit-2 usage error (not one of the card kinds).

### D. Paired comparison

A supported `hotato verify` before/after rollup where at least one
previously-failing fixture now passes and no hold/backchannel fixture
regressed. The card reads "PAIRED FRESH-RECAPTURE IMPROVED" only when the
recapture is runner-attested and "PAIRED (OPERATOR-ASSERTED)" otherwise,
never "verified" or "fix verified", and closes with "Hotato reports
coincidence, not causation." A verify result that doesn't support that
claim (too few previously-failing fixtures, nothing now passing, or a
regressed hold fixture) is refused with exit 2.

```bash
hotato card verify.json --out comparison.svg
```

### E. Failure contract

A committed contract's card: the expected behavior (`yield` or `hold`), the
measured seconds, and PASSED / FAILED / NOT SCORABLE. The footer reads "A
human labeled this contract; Hotato measured the timing."

```bash
hotato card contracts/demo-missed-interruption.hotato/contract.json --out contract.svg
```

### F. Say-do failure

A `hotato test run` result (`--format json`, kind `hotato.test-run`) whose
deterministic lane failed a tool/state evidence assertion (`tool_result`,
`tool_call`, `tool_error`, `http_result`, `state`, `state_change`): the
conversation claims an outcome the trace (Authority 1) or the post-call
state (Authority 2) does not back. The card renders the claim vs the
evidence -- the failing assertion's id and kind, its span refs when the
evaluator recorded any, and its share-safe `public_reason` (built from
allowlisted structured fields only, never transcript text, a tool payload,
or a state value). The failing outcome-tagged evidence assertion leads;
the footer reads "Tool and state evidence decide the outcome, never the
agent's words." A test-run result with no failing tool/state evidence
assertion is refused with exit 2.

```bash
hotato start --demo   # act two writes saydo/test-run.json
hotato card saydo/test-run.json --out saydo-card.svg

# or from any of your own runs:
hotato test run refund.yaml --agent support-v3 --trace voice_trace.jsonl \
    --transcript call.transcript.json --format json > test-run.json
hotato card test-run.json --out saydo-card.svg
```

## Redaction: identifiers stay hidden by default

A card is a public image, so identifiers stay hidden by default: call id,
filesystem path (only a basename is ever shown), and vendor recording name
are omitted. A pulled recording named `STACK__ID.wav`
carries the call id inside its name; that name shows only under
`--include-identifiers`.

```bash
# shows the source recording's basename on a candidate card
hotato card hotato-sweep.json#1 --out card.svg --include-identifiers
```

## The Failure Record: the artifact that lands in the pull request

A card is one moment's image you attach yourself. The **Failure Record**
is the PR-native trust artifact: the shipped GitHub Action renders one per
non-passing unit ([`docs/CI.md`](CI.md)), so it arrives in the pull
request with the run that produced it. Each record is a lane-structured
projection of one failed, inconclusive, or errored result:

- one evidence-specific headline;
- the five lanes (outcome, policy, conversation, speech, reliability),
  each with its own status and never a blended or overall score;
- the reproduce command (`reproduction.argv`, rendered in the Markdown,
  HTML, and SVG forms) plus the pinned one-command verifier
  (`hotato record verify failure-record.json`);
- content-addressed evidence digests and a privacy profile: no audio,
  transcript body, tool payload, state value, or absolute path, so it
  attaches to a PR as-is.

`hotato start --demo` emits one automatically under
`hotato-failure-record/` --
`failure-record.{json,md,html,svg}` -- alongside the sweep and the demo
contract, and prints the record paths plus the one-command verifier.
Render one from any result yourself with `hotato record render`.

## Output and exit codes

Without `--out`, the SVG goes to stdout, so you can pipe it. With `--out`
it is written there atomically.

- **0**: the SVG card was rendered (to `--out`, or to stdout).
- **2**: usage error, unreadable input, a bad candidate ref, or an input
  not a fix plan / verify result / contract / test-run result / sweep
  candidate.

## Regenerating the committed cards

Four commit-ready examples live under `docs/assets/cards/`
(`no-single-threshold-card.svg`, `talk-over-card.svg`,
`false-stop-card.svg`, `say-do-card.svg`), rendered from the bundled demo
data. Regenerate them with:

```bash
PYTHONPATH=src python3 scripts/render_card_assets.py
```


================================================================================
FILE: docs/COMPARE.md
================================================================================

# Compare: hotato vs hosted platforms

Local-first testing and observability for AI agents, next to the hosted
platforms that run the same jobs as a service. The short version: the same
lifecycle, three structural differences.

## Property by property

The hosted platforms (LangSmith, Langfuse, Phoenix, LangWatch, and the
voice-specific tools) are capable products: many support deterministic code
evaluators, datasets, experiments, human review, and self-hosting. This table
compares default properties, not a caricature, and it is honest about where a
managed platform is the stronger fit.

| Property | hotato default | Typical managed deployment |
|---|---|---|
| Raw evidence location | your local filesystem or VPC | the vendor's service |
| Price at scale | free and MIT, any volume | metered per seat, run, or event |
| Deterministic timing lane (turn-taking, say-do) | built in | varies by platform |
| Content-addressed contracts + release proofs | built in | platform-specific |
| Offline verification, no service dependency | built in | often service-dependent |
| Model-judge lane | separate and advisory by default | commonly integrated into the score |
| Hosted parallel execution at scale | you operate it | managed by the vendor |
| Persistent trace search, datasets, prompt management | limited | commonly built in |
| Team collaboration, roles, review queues | local workspace | mature hosted collaboration |

## Where hotato is the clear pick

- **Your data stays yours.** Recordings, traces, prompts, and backend state
  never leave your machine, and a contract or proof stays verifiable if you
  stop using hotato, if the vendor changes, or if the service is down.
- **Verdicts you can gate on.** The deterministic lanes score the same input
  the same way on every machine, byte for byte, so an exit code can block a
  merge. A model-judged score that varies run to run cannot.
- **Free at any volume.** No per-seat, per-run, or per-event meter anywhere.

## Where a managed platform may fit better

A hosted platform is often the faster path if you need mature multi-user
collaboration, a persistent searchable trace explorer with saved views,
built-in dataset and prompt management, or vendor-operated execution at high
concurrency. hotato is deliberately local and CLI-first; those surfaces are
limited today. The two compose: point a hosted platform's OTel export at
`hotato trace ingest` and keep the deterministic test and release-evidence
layer on your own machine.

## The layer hotato is not

A runtime voice layer (an orchestration platform's endpointing, a turn
detector, barge-in suppression) acts *during* the call. hotato runs *after*,
from the recording and the trace: it measures what happened, pins what must not
happen again, and proves a candidate against it. Run the runtime layer that
makes your median call good; run hotato so the failure a caller hit last week
stays fixed on every push. The frozen moment a runtime layer got wrong is
exactly what `hotato investigate label` pins and `hotato prove` re-verifies.

## What the scoring claims, precisely

- Timing and say-do, not intent: energy over time, tool spans, and post-call
  state decide verdicts; no output claims emotion or meaning.
- Two channels or a timestamped transcript in; a mono or mixed export is
  refused as NOT SCORABLE, never guessed at.
- Five dimensions (outcome, policy, conversation, speech, reliability), each
  reported on its own evidence; there is no blended score anywhere.
- Deterministic checks stay separate from the model-judged rubric lane, which
  is advisory and local.
- Determinism is verified in CI: byte-identical re-runs on Linux x86_64,
  Python 3.10, 3.11, and 3.12 ([VALIDATION.md](VALIDATION.md) Job 1).

## Every example is scoped to one run

A hotato example that shows a call failing on a named stack is labelled a
**provider-default** run: one assistant, one configuration, one date, one
scripted caller, on that provider's out-of-the-box interruption settings. It
demonstrates the threshold funnel scoped to that one run. Any stack, tuned, can
pass the same fixtures. hotato publishes fixtures and reproduction steps: a
record you can run yourself, not a scoreboard.

The full loop, command by command: [`docs/LIFECYCLE.md`](LIFECYCLE.md).


================================================================================
FILE: docs/DIARIZE.md
================================================================================

# `hotato run --mono call.wav --diarize`: score a single-channel recording

Hotato's gold reference is **two-channel** audio: caller on one channel,
agent on the other, no separation needed. That path, and every published
number, stays unchanged. A **mono** (single, mixed) recording fails by
default: both voices sum into one waveform, so the scorer cannot attribute
energy to a speaker and rejects it as not scorable.

The opt-in `[diarize]` front-end widens that coverage: it runs an
off-the-shelf **speaker diarizer** over the mono to recover *who was active
when*, reconstructs caller/agent tracks, and hands them to the **existing**
scorer -- so a mono call becomes scorable. The result is **quality-gated**
and tiered per file: above the confidence bar it is a `diarized-mono`
verdict; below it, indicative only with no SLA gate; non-separable, it is
refused.

**Turn-timing reconstruction, from anonymous labels.** Hotato scores
*timing* -- who was active when, and overlap -- and a diarizer's turn
timestamps reconstruct that directly: anonymous `SPEAKER_00` / `SPEAKER_01`
labels per turn, handed to the scorer as boundaries. The audio stays mixed;
the labels carry only channel timing, never a name, a voice-print match, or
an identity.

The companion opt-in extra is [`docs/TRANSCRIBE.md`](TRANSCRIBE.md): it
attaches plain-text, timestamped words next to the score, computed strictly
after scoring and never fed back into it.

## Quickstart

```bash
# Install the default (local, offline) diarizer extra, plus a Hugging Face token
# with the gated model conditions accepted (one-time download, then offline):
pip install 'hotato[diarize]'
export HUGGINGFACE_TOKEN=hf_...            # accept the model card conditions first

# Check first whether the mono is confidently separable (no scoring):
hotato trust --stereo call.wav --diarize            # -> high / low / refuse tier

# Score the mono:
hotato run --mono call.wav --diarize --format json  # diarized-mono verdict
```

Without the extra (or the token/model), `--diarize` exits `2` with a clean
error -- it always requires the diarizer to run before scoring.

## Backends and extras

A pluggable backend seam, mirroring the neural-VAD seam: pick with
`--diarizer`, install only the extra you select. `pyannote` is the
accessible local default, chosen from the downstream benchmark; for
best-in-class telephone-audio accuracy, pick `sortformer` (local) or
`pyannoteai` (hosted).

- **`pyannote`** (default)
  - Extra: `[diarize]`
  - Where it runs: local, CPU-viable, offline
  - Notes: richest confidence signals (posterior + embedding margin); gated
    HF weights
- **`sortformer`**
  - Extra: `[diarize-sortformer]`
  - Where it runs: local, GPU-leaning
  - Notes: best self-hostable on 2-speaker telephone; EEND (no embedding
    margin)
- **`pyannoteai`**
  - Extra: `[diarize-hosted]`
  - Where it runs: HOSTED (audio leaves the machine)
  - Notes: best absolute accuracy; requires `--egress-opt-in`

```toml
# default; needs system ffmpeg
diarize = ["pyannote.audio>=4.0", "torch>=2.8", "torchaudio>=2.8", "numpy>=1.21"]

# best self-hostable, GPU
diarize-sortformer = ["nemo-toolkit[asr]>=2.7", "torch>=2.8", "numpy>=1.21"]

# hosted, egress opt-in
diarize-hosted = ["pyannoteai-sdk>=0.3"]
```

The `[diarize]` path raises the effective Python floor to **>=3.10**
(pyannote 4.x); the stdlib core stays >=3.9 -- this only constrains the
optional path.

### Model licenses (log per FTO note)

Off-the-shelf diarizers are an integration, orthogonal to any Hotato IP
claim; licenses are logged here and carried in the score envelope's
`diarization.licenses` block:

- `pyannote-audio`: **MIT** (code)
- `speaker-diarization-community-1` weights: **CC-BY-4.0** (attribution required)
- `segmentation-3.0` weights: **MIT**; `wespeaker` embedding: **CC-BY-4.0**
- `torch`: **BSD-3-Clause**; `torchaudio`: **BSD-2-Clause**
- `nemo-toolkit`: **Apache-2.0**
- Sortformer **streaming v2**: **CC-BY-4.0** (the offline v1 is **CC-BY-NC**,
  non-commercial, kept out of the shipped extras)
- pyannoteAI hosted: proprietary terms (verify before enabling egress)

For a permissive weights license, prefer `speaker-diarization-3.1` (MIT
weights) over community-1 (CC-BY-4.0) via `HOTATO_DIARIZE_MODEL`.

## The confidence gate (the core safeguard)

Aggregate diarization error is a corpus statistic; the gate is **per file,
at runtime, with no ground truth**. Six signals feed a
`separation_confidence` in `[0, 1]` and one of three tiers:

| signal | flags low quality when |
|---|---|
| speaker count == 2 | != 2: not two clean parties (1 = couldn't separate; 3+ = extra voices / mis-cluster) -> refuse |
| both speakers >= 0.30s activity | a near-silent "speaker" is a spurious split of one party -> refuse |
| mean segmentation posterior | near-chance: the model is unsure who is speaking (uncalibrated; a relative signal) |
| embedding cluster margin (pyannote only) | small: two voices too similar to attribute confidently |
| overlap ratio in a sane band | extreme: heavy crosstalk / collapsed turns -> talk-over unreliable |
| segment churn (short turns/sec) | high: a jittery, unstable timeline -> noisy timing |

**Tiers:**

- **high** -- scores normally, tagged `source: "diarized-mono"` and
  `confidence_tier: "high"`, distinct at every step from a dual-channel
  verdict.
- **low** -- scores with `indicative_only: true`, reading "indicative only,
  reconstructed from single-channel diarization." The pass/fail SLA gate
  (`--max-talk-over` / `--max-time-to-yield`) stays off at this tier.
- **refuse** -- `scorable: false`, a reason naming the failed signal, exit
  `2` (same as today's mono rejection).

The thresholds are provisional and **uncalibrated**, pinned by the
downstream verdict-agreement benchmark rather than asserted as accuracy, and
exposed as constants in `hotato/diarize.py`.

## Caller vs agent assignment (proposed, stated, overridable)

A diarizer returns anonymous `SPEAKER_00` / `SPEAKER_01`; Hotato needs
caller/agent. The mapping is proposed, stated as an assumption, and
overridable:

- **default proposal** -- the floor-dominance heuristic (`trust`'s
  possible-swap band): an agent usually holds the floor longer, so the
  higher-talk-time speaker is proposed as agent. A balanced (ambiguous)
  floor time is broken by who-speaks-first and flagged `balanced: true`,
  downgrading the verdict to indicative until confirmed.
- **override** -- `--caller-speaker SPEAKER_00 --agent-speaker SPEAKER_01`;
  when both are given, no heuristic runs.

The chosen mapping and its basis are emitted in
`diarization.speaker_map: {caller, agent, basis, balanced, confidence}`.

## Echo / crosstalk is N/A on this path

The two reconstructed tracks are slices of **one physical microphone**, so
`signals.echo` / crosstalk coherence carries no real signal (it reads
trivially high on overlap) and is marked `applicable: false` on this path --
`--echo-gate` only means something across two physically separate channels.

## Limits (what stays indicative or refused)

- **very similar voices** (same gender/pitch, or one person on both ends) →
  low embedding margin → low/refuse.
- **heavy crosstalk / echo bleed** → overlap balloons → low/refuse.
- **>2 speakers** (a supervisor, hold music with vocals) → refuse.
- **deep sustained overlap** and **sub-second boundary precision** run
  weaker than dual-channel and are stamped as such, never hidden.
- **balanced speaker map** → mapping uncertain → indicative until
  confirmed.

A de-risk spike, with a *perfect* diarizer, confirmed the
masked-reconstruction path inflates sub-second talk-over by ~0.1-0.36s and
can bridge a short backchannel gap -- an error intrinsic to single-channel
masking that no diarizer quality removes. That is why elevated-overlap and
short-yield cases land in the fragile `low` zone. The recommended follow-up
is direct diarization-timeline injection, skipping the reconstruction
re-VAD; either way, the gate above governs the error budget.

## JSON shape (agents)

A scored diarized-mono envelope is the SAME envelope as any single run, plus
a `diarization` provenance block and, when the tier is not high,
`indicative_only: true` on the event:

```json
{
  "mode": "single",
  "diarization": {
    "source": "diarized-mono",
    "backend": "pyannote",
    "model": "pyannote/speaker-diarization-community-1",
    "num_speakers": 2,
    "speaker_map": {
      "caller": "SPEAKER_00",
      "agent": "SPEAKER_01",
      "basis": "floor-dominance",
      "balanced": false
    },
    "separation_confidence": 0.86,
    "confidence_tier": "high",
    "overlap_ratio": 0.14,
    "licenses": {"pyannote-audio": "MIT (code)", "...": "..."}
  },
  "events": [{
    "verdict": {
      "did_yield": true,
      "seconds_to_yield": 0.5,
      "talk_over_sec": 0.5,
      "reasons": []
    },
    "diarization": { "...": "same block" },
    "scorability": {
      "separation": {
        "confidence_tier": "high",
        "separation_confidence": 0.86,
        "signals": {"...": "..."}
      }
    },
    "signals": {
      "echo": {"applicable": false, "reason": "single physical channel ..."}
    }
  }]
}
```

Branch on `diarization.confidence_tier`: an event carrying
`indicative_only: true` stays indicative-only, distinct from a confident
dual-channel verdict. A refused file is `scorable: false` with
`not_scorable_reason` and exit `2`.


================================================================================
FILE: docs/DRIVE-A-CALL.md
================================================================================

# Drive-a-call: originate a call against a live agent, then score it

`hotato capture` scores a call you already ran. Drive-a-call places the
call: it dials a live voice agent, waits for it to finish, and feeds the
recording into the same pull -> score pipeline. The agent's side is live
and unscripted, so it scores unchanged.

## The second move, after a first catch on captured calls

You have seen a catch on a recorded call -- a pulled recording
([`docs/CONNECT.md`](CONNECT.md)), a sweep candidate, or a trace
assertion. Drive-a-call is the next move: instead of waiting for
production traffic to reproduce the moment, place a call against your
live agent on demand and score the recording the same way. It reaches the
provider's API and bills one outbound phone call per run, so the
captured-call paths stay the cheap default and this is the deliberate
follow-up.

## The caller side runs from a script

The agent's half is unscripted. The CALLER's half runs from a script, and
the origin records that plainly as `origin.caller`:

- **Twilio** renders your `scenario.v1` caller script into TwiML: one
  `<Say>` per `say`-turn, `<Pause>` between them. `<Say>` speaks at
  **fixed offsets** on its own clock, not the agent's -- a deterministic
  driver built to catch a regression in a scripted sequence, not a live
  back-and-forth. A turn's reactive `when_agent_asks` / `after` label still
  speaks unconditionally, in order. `origin.caller = "scripted-twiml"`.
- **Vapi** originates the call FROM the assistant under test (a staging
  clone) TO a customer number: the assistant is the party being measured,
  whoever answers is the other side. `origin.caller =
  "assistant-originated"`.

Either way `origin.kind = "real"`, tagged with `origin.provider` and
`origin.provider_call_id` (the provider's own call id) -- the invariant-5
axis: every driven call states exactly how it was produced.

## The endpoints implemented

- **Twilio**
  - Originate: `POST /2010-04-01/Accounts/{sid}/Calls.json` with
    `To`/`From`/`Twiml` + `Record=true`, `RecordingChannels=dual` (the REST
    equivalent of `<Dial record="record-from-answer-dual">`).
  - Poll until: `GET .../Calls/{CallSid}.json` -> `status == completed`.
  - Then pull: `GET .../Recordings.json?CallSid=...` -> `RecordingSid` ->
    existing `capture_twilio` (`?RequestedChannels=2`).
- **Vapi**
  - Originate: `POST https://api.vapi.ai/call`
    `{assistantId, phoneNumberId, customer:{number}}`.
  - Poll until: `GET /call/{id}` -> `status == ended`.
  - Then pull: existing `capture_vapi` -> `artifact.recording.stereoUrl`.

Only `POST` (create) and `GET` (poll + pull) are issued: a provider config
stays untouched. Vapi drives the call FROM a staging CLONE, so production
stays untouched too.

A non-`completed` Twilio call (busy / failed / no-answer / canceled) has no
recording to score, so it raises a clear error.

## Credentials and the egress opt-in (both required)

Placing a call reaches the provider's REST API and bills a phone call, so
`run_scenario` requires BOTH before it dials:

1. **Credentials** -- `VAPI_API_KEY`, or `TWILIO_ACCOUNT_SID` +
   `TWILIO_AUTH_TOKEN` (via `hotato connect` or the environment).
2. **An explicit egress opt-in** -- set `HOTATO_DRIVE_OPT_IN=1`, or pass
   `egress_opt_in: true` on the scenario. Without it, `run_scenario` raises
   the same clean, structured refusal a hosted op has always given, and
   dials nothing.

Drive parameters ride on the scenario (inline or under a `drive:` block) or
the environment:

- Twilio: `to_number` (the agent's number, `HOTATO_DRIVE_TO_NUMBER`),
  `from_number` (your Twilio number, `HOTATO_DRIVE_FROM_NUMBER`).
- Vapi: `phone_number_id` (`VAPI_PHONE_NUMBER_ID`), `customer_number`
  (`HOTATO_DRIVE_CUSTOMER_NUMBER`).

The recording download reuses `capture`'s validated, SSRF-safe path. See
[`docs/EGRESS.md`](EGRESS.md) and [`docs/THREAT-MODEL.md`](THREAT-MODEL.md)
for the per-command rows.

## What it costs, and what it proves

It bills one outbound phone call per scenario run -- even the deterministic
Twilio caller is billed. The recording captures a live conversation;
whether the agent "passed," or the scripted caller's timing matched a live
one, is the separate assert layer's job.

## Retell / LiveKit / Pipecat

Retell calls are captured after the fact with `hotato pull`. LiveKit and
Pipecat calls are captured inside your own infra: point your own capture
path at the recording, and hotato scores whatever file it writes.


================================================================================
FILE: docs/EGRESS.md
================================================================================

# Egress: what talks to the network, command by command

Every `urllib`/`http`/`subprocess`-to-a-network-tool call site in
`src/hotato/`, mapped to the CLI command that reaches it -- derived straight
from the code. Every command under "Fully local" runs entirely on your
machine: no `urllib` import, no socket, no networked subprocess.

## Fully local -- everything runs on your machine

`run`, `report`, `doctor`, `team`, `export`, `benchmark`, `compare` (the
batch result-comparison command, `stackbench.py`), `demo`, `start`, `card`,
`diagnose`, `plan`, `explain`, `fixture create`, `fixture promote`,
`contract create` (stereo / caller+agent / local-file mono), `contract
verify`, `contract inspect`, `contract pack`, `contract unpack`, `trace
attach`, `trace export`, `scan`, `trust`, `analyze`, `patch`, `verify`, `fix
trial`, `loop`, `describe`, `init starter`, `init webhook` (the scaffold
generator -- the server it scaffolds is a local listener, see below),
`setup`, and every `counterexample` subcommand (`compile`, `verify`,
`reproduce`, `inspect`, `export`, `predicate`). Counterexample compilation
uses the local scripted simulator and deterministic assertion engine; it never
executes a provider adapter, judge, download, or arbitrary command.

These read and write only local files (recordings, scenarios, `hotato.yaml`,
connection files). `connect` also belongs here: it validates and stores
credentials at `~/.hotato/connections.json` (mode `0600`), no network round
trip (`src/hotato/connections.py`: "nothing in this module makes a network
call").

`rubric run`, `rubric calibrate`, and the `test run` rubric lane also belong
here **on the default path**: the judge is a **LOCAL Ollama** daemon
(default `http://localhost:11434`), and the verdict cache
(`~/.hotato/rubric-cache`) is local too. Two paths send a rubric command to
the network instead -- both explicit, both in the extras table below: a
non-local Ollama endpoint, or `--judge-provider hosted`.

## Reaches your configured vendor -- only when the command's job is to

| Command | Reaches | When | Code |
|---|---|---|---|
| `capture --stack vapi\|retell\|twilio` | the stack's REST API | always (fetches the one call you named) | `capture.py`: `_http_get`/`_http_get_json`/`_download` via `urllib.request` |
| `capture --stack vapi\|retell\|twilio --demo` | nothing | `--demo` scores a bundled reference file instead | `capture.py` |
| `capture --stack livekit\|pipecat` | nothing from Hotato | the recording is produced by YOUR infra; Hotato only scores the local file `setup` pointed you at | `capture.py` |
| `pull` | the stack's list + download endpoints | always (bulk-fetches recent recordings) | `capture.py` (same `_http_get`/`_download` path `capture` uses, looped) |
| `sweep --stack <stack>` | same as `pull` | whenever `--demo` is NOT passed | `capture.py` via the pull path |
| `sweep --demo` | nothing | sweeps the two bundled demo recordings | `analyze.py` on packaged audio |
| `inspect --stack vapi\|retell` | the stack's assistant-config read endpoint (GET only, never a write) | always | `inspectcfg.py`: `_http_get_json`, `inspect_vapi`, `inspect_retell` |
| `inspect --stack livekit\|pipecat` | nothing | parses YOUR local config file with `ast`, no network | `inspectcfg.py`: `inspect_livekit_file`, `inspect_pipecat_file` |
| `ingest` (webhook worker) | inbound only, from the stack you configured to call your webhook; outbound only if the event's `recording_url` needs a download | per event, and only for the fetch step | `ingest.py`: `_resolve_recording` reuses `capture.py`'s fetch/download; payloads are always treated as data, never instructions |
| `apply` (default, no `--yes`) | nothing | dry run -- prints the staging clone it WOULD create | `apply.py`: `build_apply` returns `dry_run: True`, never calls the networked function |
| `apply --clone --yes` | the stack's REST API (`vapi`, `retell` only) | only with `--yes` and credentials | `apply.py`: `create_clone` / `_http_json` is "the only networked function" in the module (its own docstring) -- reads the source config via GET, then POSTs to create a NEW staging assistant. Never PUTs/PATCHes the source. |
| `issue create` (default, no `--yes`) | nothing | dry run -- prints the issue body and the `gh` command it would run | `issuecmd.py` |
| `issue create --yes` | GitHub, through your local `gh` CLI's existing auth | only with `--yes` | `issuecmd.py`: `subprocess.run(["gh", "issue", "create", ...])` |
| `pr create` (default, no `--yes`) | nothing | dry run -- prints the PR body and the `git`/`gh` argv it would run | `prcmd.py` |
| `pr create --yes` | Git remote + GitHub, through your local `gh` CLI's existing auth | only with `--yes` | `prcmd.py`: `subprocess.run([...git...])` then `subprocess.run(["gh", "pr", "create", ...])` |
| `test run --state F` (state-config `adapter: http`) | your configured system-of-record REST API | per `state`/`state_change` assertion, and ONLY when the config sets `egress_opt_in: true`. What leaves: the mapped filter VALUES for that one query (in the URL or a JSON body); never audio, transcript, or the config | `state_adapter.py`: `HttpStateAdapter.query` via `urllib.request` (credential-safe redirects reused from `capture.py`) |
| `test run --state F` (state-config `adapter: sql` + a `dsn`) | your database, over the network via the driver you install | per assertion, and ONLY with `egress_opt_in: true`. A parameterized read-only SELECT; filter VALUES are bound as data, never interpolated. A local `sqlite_path` opens no socket | `state_adapter.py`: `SqlStateAdapter.query` (stdlib `sqlite3`, or a caller-installed DBAPI driver) |
| `simulate --chat URL` | YOUR chat agent at the URL you name -- `localhost`/`127.0.0.1` only by default | one POST per scripted caller turn (`{conversation_id, turn_index, text}` -> `{text}`); a non-local host is refused (exit 2) BEFORE any request unless you pass `--egress-opt-in`, and redirects are refused outright, so a local URL never bounces off-box. What leaves: the scenario's caller turn text, nothing else | `chat_sim.py`: `check_chat_url` / `_post_turn` via `urllib.request` |

`load_state_adapter` requires `egress_opt_in: true` before an `http`
adapter or a `sql` adapter over a `dsn` connects -- checked before any
connection, on the CLI's exit-2 usage-error path. The default `--state`
adapters (the local mock JSON/SQLite sandbox and a local `sqlite_path` SQL
DB) run entirely on your machine and need no opt-in. Full config format:
[`docs/STATE-ADAPTERS.md`](STATE-ADAPTERS.md).

Notify surfaces (Slack, GitHub) fire only through credentials you
configured (`gh`, a Slack token), and only for actions you invoked.

## Optional extras that add a hosted call

| Extra / flag | Adds | Gate |
|---|---|---|
| `hotato[neural]` | nothing network -- a local Silero VAD cross-check model, run offline | N/A |
| `hotato[transcribe]` (`run --transcribe`) | nothing at inference -- a local `faster-whisper` ASR pass over the same recording, fully offline once the model is cached. First use of an uncached model name downloads its weights once (like installing any pip package with model weights); every run after opens no socket. Context only, kept separate from the score | N/A |
| `hotato[livekit]` / `hotato[pipecat]` | nothing from Hotato directly -- these SDKs run YOUR live capture infra; Hotato scores the file that infra writes | N/A |
| `--diarizer pyannoteai` (`contract create --mono --diarize`, `run --mono --diarize`) | uploads the mono audio to `pyannote.ai` for diarization | requires `--egress-opt-in` (exit 2 without it); the default diarizer (`pyannote`, local) stays offline. See `diarize.py`: `build_pyannoteai_backend` |
| `hotato[judge]` (`rubric run`, `rubric calibrate`, `test run` rubric lane) | nothing on the default path -- the judge is a LOCAL Ollama model (`http://localhost:11434`), reached with only the stdlib (`urllib`). The transcript stays on your machine | N/A on the default (local) path. `rubric.py`: `OllamaJudge` |
| `--judge-provider hosted --judge-endpoint URL` (any rubric command) | sends the transcript + rubric criterion to a hosted OpenAI/Anthropic-compatible `/chat/completions` endpoint you name | requires `--judge-egress-opt-in` (exit 2 without it); the default local judge stays on your machine. `rubric.py`: `HostedJudge` (raises `EgressRefused` without the flag) |
| A non-local `--judge-endpoint` (e.g. a remote Ollama host) | reaches that host | same gate -- requires `--judge-egress-opt-in`. `localhost`/`127.0.0.1`/`::1` skip it. `rubric.py`: `OllamaJudge._is_local_endpoint` |
| `--notify URL` (`sweep`, `fleet run`) | POSTs one JSON summary -- counts, top candidate moments (id, kind, timing numbers only), local artifact paths, plus a `text` line for Slack incoming webhooks. Audio, credentials, and transcript text stay out of it | off by default; fires only with an explicit, repeatable `--notify URL`. A non-http(s) scheme is refused (exit 2) before any network attempt. Once sent, delivery is fail-open -- a down or slow webhook logs one stderr warning and the run keeps going. See `notify.py`: `post_notification` |
| `--notify URL` (`contract verify`) | POSTs one JSON run-summary when the gate finishes -- kind `contract-verify`, the pass/fail counts (`passed`/`failed`/`tampered`/`refused`/`assertions_failed`, reported as separate fields, never blended), the top FAILING contracts' ids + measured timing (`did_yield`/`seconds_to_yield`/`talk_over_sec`), plus a `text` line for Slack. Audio, credentials, transcript text, and file/bundle paths all stay out of it (this payload carries no artifact paths, unlike sweep's) | off by default; fires on every verify (pass or fail) only with an explicit, repeatable `--notify URL`. A non-http(s) scheme is refused (exit 2) before the re-score. Delivery is fail-open -- a down webhook never raises and never changes the verify exit code. See `notify.py`: `contract_verify_payload` / `post_notification` |

## The one credential-safety detail worth knowing

`capture.py` installs a process-wide `urllib` redirect handler
(`_CredentialSafeRedirectHandler`) so a 3xx redirect from a vendor API keeps
your Authorization header and API key scoped to the original host. Applies
to every fetch path above that goes through `capture.py`'s
`_http_get`/`_download`.

## Read more

- Posture summary and reporting a vulnerability: [`SECURITY.md`](../SECURITY.md)
- Command-by-command threat model: [`docs/THREAT-MODEL.md`](THREAT-MODEL.md)
- Ingest's untrusted-payload handling: [`docs/INGEST.md`](INGEST.md)

## Drive-a-call: originates a real call, so it is double-gated

`run_scenario` (the fleet experiment step; `src/hotato/drive.py`)
ORIGINATES a real, billable call against a live agent and pulls its
recording -- the one path here that both reaches a vendor AND places an
outbound call, so it carries a gate on top of the usual credential
requirement.

- **`run_scenario` (Vapi adapter)**
  - Reaches: `POST https://api.vapi.ai/call`, then polls `GET /call/{id}`,
    then the existing `capture_vapi` download.
  - When: only when driven.
  - Gate: requires real credentials AND an explicit egress opt-in
    (`HOTATO_DRIVE_OPT_IN=1` or `egress_opt_in: true` on the scenario),
    both present before dialing. Creates and reads only (GET/POST); the
    call is driven from the staging CLONE, keeping production untouched.
- **`run_scenario` (Twilio adapter)**
  - Reaches: `POST .../Calls.json`, then polls `GET .../Calls/{sid}.json` +
    `GET .../Recordings.json`, then the existing `capture_twilio` download.
  - When: only when driven.
  - Gate: same double gate. TwiML `<Say>` is a fixed-timeline scripted
    caller; recording is dual-channel via
    `Record=true`/`RecordingChannels=dual`. GET/POST only, so config stays
    untouched.

The recording download reuses `capture.py`'s validated fetch (scheme
allowlist, default-deny SSRF, cross-host credential strip, atomic write); a
local test recording server on `127.0.0.1` needs
`HOTATO_ALLOW_PRIVATE_URLS=1` like every other download. Full walkthrough:
[`docs/DRIVE-A-CALL.md`](DRIVE-A-CALL.md).


================================================================================
FILE: docs/EVIDENCE-PACK.md
================================================================================

# The evidence pack

Hotato's proof is reproducible: commands, recorded audio, and
deterministic, byte-stable verdicts you rerun on your own machine.

The standard every artifact here is held to, and why it's ranked the way
it is, lives in [evidence/README.md](evidence/README.md) -- read that
first when deciding how much to trust any single piece below.

## What's in the pack

- **Bundled demo** -- two bundled calls an agent fails (a synthesized
  talk-over and a recorded provider-default call), scored offline in under a
  minute, one command. `hotato demo` ·
  [START.md](START.md)
- **Recorded provider-default battery** -- 12 scripted calls against a
  live voice agent on default settings; a missed interruption and a false
  stop fail in the same run.
  [`corpus/vapi-defaults/README.md`](../corpus/vapi-defaults/README.md)
- **Determinism check** -- the same recording produces the same numbers
  every run. CI runs it on Linux, macOS, and Windows; Linux is green
  today. [VALIDATION.md](VALIDATION.md) Job 1
- **Not-scorable gallery** -- eight input conditions, and the exact
  verdict each produces, including three hard refusals.
  [GALLERY.md](GALLERY.md) · [TRUST-GALLERY.md](TRUST-GALLERY.md)
- **Trust contract** -- the input-condition table the gallery
  demonstrates. [TRUST-MATRIX.md](TRUST-MATRIX.md)
- **Validation scope** -- the three jobs Hotato is measured on, and where
  that measurement stops. [VALIDATION.md](VALIDATION.md)
- **Where Hotato fits next to a QA platform** -- a named-vendor routing
  guide. [COMPARE.md](COMPARE.md)
- **Case studies** -- recorded-audio write-ups, each with a repro command
  and a scope section on exactly what that run proved.
  [`case-studies/`](case-studies/README.md)
- **PR cards** -- self-contained SVGs of a candidate, a
  threshold-funnel finding, or a verify result. [CARDS.md](CARDS.md) ·
  [`assets/cards/`](assets/cards/)
- **Launch-bar status** -- the checklist scoring where each launch item
  stands. [evidence/validation-plan.md](evidence/validation-plan.md)

## Check it yourself

Every artifact above is re-derivable from a command:

```bash
hotato demo --no-open --format text          # the two bundled failures
hotato run --stereo FILE.wav --expect yield  # diff two runs, get nothing
hotato trust --stereo FILE.wav               # the refusal contract, live
```

Every command runs offline, on your machine -- reproduce it yourself
instead of taking the claim on faith.

## What counts as evidence here

A command, a recording, a verdict you check yourself -- ranked by the
standard in [evidence/README.md](evidence/README.md).


================================================================================
FILE: docs/EXPLAIN.md
================================================================================

# explain: root-cause-by-layer, composed from what already exists

`hotato explain` reads a finished result and produces a root-cause
attribution: which layer likely failed, whether a fix is safe to try, the
opposite-risk tradeoff, and a plain next command. It's compositional, not
a new scoring engine: it reframes `hotato diagnose`'s per-event findings
and `hotato plan`'s policy gate (a mapped knob, one bounded step, a
passing opposite-risk fixture, config-only-safe), plus, for a contract
bundle, its own trust report, policy bounds, and an attached voice trace
when present.

When the evidence can't support picking ONE root cause, explain states
the reason plainly.

```bash
hotato run --suite barge-in --format json > result.json
hotato explain result.json
```

## Three input shapes, auto-detected

| Input | Example | Behavior |
| --- | --- | --- |
| Run envelope | `hotato explain result.json` | Full diagnose + policy-gate attribution, per failing event. |
| Sweep/analyze candidate ref | `hotato explain hotato-sweep.json#1` | ALWAYS refused: a candidate carries no human label. |
| Contract bundle directory | `hotato explain contracts/refund-cutoff-001.hotato` | Attributed from the contract's own measurement and policy bounds, or refused when it needs a signal the bundle lacks. |

## The attribution shape

Layer-general by design: `turn_taking` is the only populated layer today,
with room for the next (asr, tool, policy, latency, handoff, ...) without
a version bump.

```
{"event_id":         the failing event, or null for a battery-level finding,
 "failure_layer":    "turn_taking",
 "type":             missed_real_interruption | false_stop_on_backchannel |
                     false_stop_on_ambient_noise | slow_yield |
                     excess_talk_over | endpointing_miss | threshold_funnel,
 "turn_taking_layer": interruption_detection | endpointing,
 "confidence":       high | medium | low,
 "fixability":       safe_to_patch | needs_human | insufficient_evidence |
                     do_not_patch,
 "opposite_risk":    the tradeoff a fix on this axis trades against,
 "evidence_for":     measured fields and notes that support this attribution,
 "evidence_against":  the caveats against treating it as settled,
 "unknowns":         explicit gaps in the evidence (never silently assumed),
 "safe_next_action": one concrete next command, never an auto-apply}
```

`fixability` reuses the SAME gate `hotato plan` already enforces:

* `safe_to_patch` -- the failure maps to one setting AND the battery
  already has a passing opposite-risk fixture on that axis (the same
  coverage `hotato plan` requires before proposing a change).
* `insufficient_evidence` -- the failure maps to one setting but the
  opposite-risk fixture is missing, so a change couldn't be verified; or
  a contract bundle's policy-bound comparison found a violation with no
  companion moment to verify against.
* `needs_human` -- an audio-path problem (echo bleed).
* `do_not_patch` -- the threshold funnel: the battery missed an
  interruption it should have caught AND false-stopped on a backchannel.
  No single sensitivity threshold fixes both; the fix class is
  engagement-control. This adds one COMPOSITE battery-level attribution
  (`event_id: null`, `type: threshold_funnel`) alongside the two
  per-event attributions it explains.

## Refusals: a precise account of the gap

A refusal states exactly which gap in the evidence blocks attributing ONE
root cause:

* **a not-scorable event** -- an input problem.
* **an ambiguous slow yield** (`unknown_root_cause`, no passing
  opposite-risk fixture) -- TTS buffering, transport latency, and VAD
  smoothing are indistinguishable from one recording.
* **an echo-tagged false stop** -- the agent most likely heard its own
  TTS bleed, an audio-path problem.
* **a sweep/analyze candidate ref** -- ALWAYS refused (no human label);
  explain prints the promote command for BOTH labels and lets a human
  choose.
* **a contract's false stop with no disambiguating `candidate_kind`** --
  a `contract.json` carries a narrower signal set than a run envelope's
  `diagnose`, so a false stop on `hold` could be a
  backchannel-discrimination miss, ambient noise, or echo bleed. Explain
  names the ambiguity and points at `hotato run --dump-frames` / `hotato
  diagnose` on the original envelope for a definitive read.

```
{"event_id":         the refused event, or null,
 "reason":           why the evidence cannot support one attribution,
 "evidence_for":     what IS known,
 "unknowns":         the specific gap,
 "safe_next_action": one concrete next command}
```

## Contract bundles: attributed from the bundle's OWN fields, bounded

A `contract.json` carries a narrower field set than a run envelope's raw
`reasons` text, so a contract-bundle attribution is narrower than a full
`diagnose`:

* a missed interruption (`expect: yield`, `did_yield: false`) is
  unambiguous and always attributed;
* a "yielded but still failed its policy" contract is attributed by
  comparing the MEASURED `seconds_to_yield` / `talk_over_sec` against the
  contract's OWN `policy.pass_conditions` bounds -- a bound comparison,
  never a guess;
* a false stop on `hold` is attributed as echo ONLY when the contract's
  `source.candidate_kind` says so (created `--from-candidate` an
  `echo_correlated_activity` moment); otherwise it's refused (see above);
* a single bundle covers one moment only, so a contract-bundle
  attribution caps at `insufficient_evidence` until a companion contract
  or fixture on the other axis exists.

A voice trace, when attached (`hotato trace attach`), folds its findings
(`traces/voice_trace.jsonl`) into `evidence_for`; without one, explain
adds an explicit unknown -- TTS-cancellation lag, transport latency, and
VAD smoothing stay indistinguishable from timing alone. See
[`TRACE.md`](TRACE.md).

## Output

| Format | Gives you |
| --- | --- |
| text (default) | a plain summary, per attribution and per refusal |
| `--format json` | the full machine shape, schema `hotato.explain.v1` (`src/hotato/schema/explain.v1.json`) |
| `--html PATH` | a self-contained report, in the same house style (`hotato.report`'s CSS and escaping) the other HTML reports use |

## Exit codes

| Code | Meaning |
|---|---|
| 0 | explained: nothing attributable (no failing or ambiguous events) |
| 1 | explained: at least one attribution or refusal was produced |
| 2 | usage error or unusable input (a bad candidate ref, a file that is not a hotato result, or an unreadable contract bundle) |

## What explain measures

Every attribution here is evidence-based: a measurement scoped to root
cause, with its `evidence_against` and `unknowns` stated on every record.
`explain` is read-only, like `diagnose` and `plan` -- applying a change
stays a human decision in your own stack. See
[`FIX-PLANS.md`](FIX-PLANS.md) and [`FIX-LOOP.md`](FIX-LOOP.md) for the
guarded ladder from a plan to a proven fix.


================================================================================
FILE: docs/GALLERY.md
================================================================================

# Gallery

Every image and worked example Hotato ships, in one place, each one
reproducible with the command shown beside it.

## Cards: findings as PR images

Four self-contained SVGs: offline, byte-stable from the same input, every
font, image, and script inlined. Full spec: [CARDS.md](CARDS.md).

- ![threshold funnel](assets/cards/no-single-threshold-card.svg)
  **The hero.** A battery where a missed interruption and a false stop fail
  together -- Hotato names the failure class (engagement-control) instead
  of picking one sensitivity dial.
  Regenerate: `hotato card fix-plan.json --out card.svg`
- ![talk-over candidate](assets/cards/talk-over-card.svg)
  A measured talk-over moment: the agent kept the floor while the caller
  was speaking.
  Regenerate: `hotato card hotato-sweep.json#N --out card.svg`
- ![false-stop candidate](assets/cards/false-stop-card.svg)
  A measured false-stop moment: the agent went quiet with no caller nearby
  to explain it.
  Regenerate: `hotato card hotato-sweep.json#N --out card.svg`
- ![say-do failure](assets/cards/say-do-card.svg)
  A say-do failure: the conversation claims an outcome the trace and
  post-call state do not back.
  Regenerate: `hotato card saydo/test-run.json --out card.svg`

## The bundled demo, rendered

`hotato demo` writes a self-contained dashboard with embedded audio and a
hear-the-bug playhead. The report and the terminal walkthrough that produced
it:

- Report: [`assets/hotato-demo-report.png`](assets/hotato-demo-report.png) ·
  [`assets/hotato-demo-report.html`](assets/hotato-demo-report.html)
- Terminal walkthrough: [`assets/hotato-demo.gif`](assets/hotato-demo.gif)

## Trust gallery: eight input conditions, eight verdicts

The full worked set, with verbatim CLI output for every row of the
[trust matrix](TRUST-MATRIX.md), lives in [TRUST-GALLERY.md](TRUST-GALLERY.md).
Summary:

| # | Condition | Verdict |
|---|---|---|
| 1 | Clean dual-channel | Scores, full confidence |
| 2 | Silent caller channel | `NOT SCORABLE`, exit 2 |
| 3 | Silent agent channel | `NOT SCORABLE`, exit 2 |
| 4 | Swapped channels | Scores, with a swap warning |
| 5 | Crosstalk / echo bleed | Scores, at lower confidence |
| 6 | Mono (mixed channel) | `NOT SCORABLE` by default, exit 2; two opt-in escapes, both indicative only |
| 7 | Backchannel candidate | Surfaced as a candidate; you label it |
| 8 | Noisy false-positive candidate | Surfaced, with the reason it is likely spurious |

## Case studies

Recorded-audio write-ups, each following
[case-study-TEMPLATE.md](evidence/case-study-TEMPLATE.md), each carrying a
mandatory "What Hotato did not prove" scope section:

- [`case-studies/vapi-01-hard-interruption.md`](case-studies/vapi-01-hard-interruption.md)

The full index and the launch target (3 external or semi-external studies, 5
consented fixtures, 1 before/after, 1 public PR) is in
[`case-studies/README.md`](case-studies/README.md), tracked against the gap in
[evidence/validation-plan.md](evidence/validation-plan.md).

## Where this fits

This page is the visual survey. For what counts as evidence and how it's
ranked, see [EVIDENCE-PACK.md](EVIDENCE-PACK.md) and
[evidence/README.md](evidence/README.md).


================================================================================
FILE: docs/GENERATIVE-CALLER.md
================================================================================

# Bounded generative caller

`hotato.caller` runs caller-side conversation programs against a media or
signaling session supplied by an adapter. It supports scripts, state-reactive
graphs, allow-listed model proposals, and model-free frozen replay. The output
is a content-addressed evidence package.

The caller has participant authority only. It cannot determine whether the
agent completed a task, mutate backend state, write tool results, or produce a
test verdict. Feed the captured call, tool trace, and state evidence into the
separate Hotato exercise/evaluation path after the caller finishes.

## Execution modes

| Mode | Model use | Intended use |
|---|---:|---|
| `scripted` | forbidden | fixed regression inputs |
| `hybrid` | only at `generate` nodes | deterministic flow with bounded caller variation |
| `generative` | required | state-reactive caller programs with validated proposals |
| `frozen_replay` | forbidden | replay the text/PCM and signaling actions bound by a prior package |

Frozen replay repeats caller outputs. It does not claim to reproduce the
agent, carrier, network, or timing scheduler. When a source action contains
PCM, replay sends those PCM16LE bytes and refuses a transport without
`send_audio`. When the source action contains text only, the text is replayed;
any downstream speech synthesis remains outside the evidence boundary.

## Plan

Plans use `hotato.caller-plan.v1`. Every loop and resource has a hard bound.
This example listens for a refund request, observes a successful tool event,
then allows a model to propose only caller speech.

```json
{
  "schema": "hotato.caller-plan.v1",
  "id": "refund-follow-up",
  "mode": "hybrid",
  "start": "greeting",
  "initial_state": {"journey": "refund"},
  "limits": {
    "max_steps": 40,
    "max_turns": 8,
    "max_duration_ms": 120000,
    "max_model_calls": 2,
    "max_tokens": 2000,
    "max_cost_microusd": 0
  },
  "nodes": [
    {
      "id": "greeting",
      "type": "say",
      "text": "I need help with a refund.",
      "next": "agent-reply"
    },
    {
      "id": "agent-reply",
      "type": "listen",
      "timeout_ms": 10000,
      "max_events": 8,
      "until": {"event": "tool_result", "tool": "refund", "status": "success"},
      "next": "follow-up",
      "on_timeout": "leave"
    },
    {
      "id": "follow-up",
      "type": "generate",
      "prompt": "Ask for the refund confirmation identifier in one sentence.",
      "allowed_actions": ["say"],
      "next": "leave"
    },
    {"id": "leave", "type": "hangup", "reason": "scenario_complete"}
  ]
}
```

The code API is intentionally adapter-injected:

```python
from hotato.caller import load_plan, run_caller

result = run_caller(
    load_plan("refund.caller.json"),
    session=my_session,
    output_dir="artifacts/refund-caller",
    model=my_caller_model,
    tts=my_tts,
)
assert result.verification["ok"]
```

## Graph nodes

| Node | Effect |
|---|---|
| `say` | sends fixed participant text, or bound PCM when a TTS adapter is present |
| `generate` | asks a model for one allow-listed action and validates its exact shape |
| `listen` | consumes bounded session events until a trigger matches or times out |
| `wait` | advances the adapter's caller schedule |
| `dtmf` | sends declared DTMF digits |
| `silence` | emits a declared silence interval through the adapter |
| `impairment` | asks the adapter to apply a named/configured media profile |
| `expect` | branches on already observed events or caller working state |
| `set_state` | updates caller working state; outcome/verdict roots are forbidden |
| `branch` | takes the first matching trigger, with an optional default |
| `repeat_bounded` | visits a target at most `max_iterations` times |
| `transfer_expect` | requires observable evidence of a completed transfer |
| `hangup` | ends the caller leg with a recorded reason |

Every run also enforces `max_steps`, `max_visits_per_node`, `max_events`,
`max_event_chars`,
`max_wait_ms`, model request/response character limits, text and PCM limits,
token limits, and a micro-USD spend ceiling. The default spend ceiling is zero.
A hosted model adapter must report integer usage and an operator must raise the
ceiling explicitly. Missing usage is a blocked run.

## Triggers

Triggers are deterministic predicates over normalized session events. They can
be combined with `all`, `any`, and `not`.

```json
{
  "any": [
    {"event": "transcript", "text_regex": "refund"},
    {"event": "transcript", "text_regex": "chargeback"}
  ]
}
```

```json
{"event": "tool_result", "tool": "refund", "status": "success"}
```

```json
{"event": "state_snapshot", "path": "subscription.status", "equals": "cancelled"}
```

```json
{"event": "timing", "metric": "agent_latency_ms", "gte": 800}
```

The normalized event kinds are `transcript`, `tool_result`, `state_snapshot`,
`dtmf`, `lifecycle`, `transfer`, `hold`, `timing`, `timeout`, and `custom`.
An adapter may put fields at the top level or under `data`. Each accepted event
receives a canonical SHA-256 identity in the result.

`text_regex` uses a 512-character safe subset: literals, escaped characters,
`.`, character classes, boundary `^`/`$`, and bounded flat repeats. Groups,
alternation, lookaround, backreferences, nested repetition, more than one
unbounded repeat, unanchored unbounded repeats, and repeat sets exceeding the
fixed path budget are refused when the plan is loaded. Regex searches inspect
at most 1,024 characters; a
larger candidate text ends the caller run with the stable
`REGEX_SEARCH_TEXT_LIMIT` `ERROR`. Use `any` for alternatives, as above.

## Session contract

The engine does not implement SIP, RTP, WebRTC, telephony, STT, or TTS. A
`CallerSession` owns those operations:

```python
class CallerSession:
    def capabilities(self): ...
    def send_text(self, text, metadata): ...
    def send_audio(self, pcm_s16le, sample_rate_hz, metadata): ...
    def receive(self, timeout_ms): ...
    def send_dtmf(self, digits): ...
    def wait(self, duration_ms): ...
    def silence(self, duration_ms): ...
    def set_impairment(self, profile): ...
    def hangup(self, reason): ...
```

Capabilities use exactly three states:

- `SUPPORTED`: the requested operation can be executed and observed at this
  boundary.
- `UNSUPPORTED`: the adapter declares that it cannot perform the operation.
- `UNOBSERVABLE`: the operation may occur, but the adapter cannot supply the
  evidence required by the node.

Missing capabilities are treated as `UNSUPPORTED`. A transfer expectation with
`UNOBSERVABLE` evidence produces `CAPABILITY_UNOBSERVABLE`; it is never recoded
as a failed transfer.

Concrete adapters should run on top of maintained media substrates. Examples
include a Pipecat transport for streaming caller audio, a LiveKit room/SIP
sidecar for WebRTC or SIP legs, and a SIPp process for fixed high-rate SIP
fixtures. The caller engine controls the program and evidence contract while
those projects own their protocols.

Two optional concrete adapters cover common local paths without changing that
separation: `LiveKitCallerSession` joins a LiveKit room through the Python RTC
SDK, and `PiperCallerTTS` invokes a local Piper model with bounded,
digest-bound provenance. See [Direct LiveKit caller session](LIVEKIT-CALLER-SESSION.md)
and [Local Piper caller speech](PIPER-CALLER-TTS.md). Neither adapter claims
PSTN delivery, carrier behavior, or speech quality from successful local
execution.

## Model contract and authority boundary

A `CallerModel.propose(request)` response has six required fields:

```json
{
  "proposal": {"action": "say", "text": "What is the confirmation number?"},
  "raw": "provider response or parsed response text",
  "provider": "local-provider-name",
  "model": "model-identifier",
  "parameters": {"temperature": 0, "seed": 7},
  "usage": {"input_tokens": 120, "output_tokens": 12, "cost_microusd": 0}
}
```

The only model-proposable actions are `say`, `dtmf`, `silence`, and `hangup`.
Each `generate` node narrows that set with `allowed_actions`. The proposal must
contain the exact fields for its action. Extra fields are refused. `set_state`,
tool results, backend state, assertions, outcomes, and verdicts cannot be model
proposals.

The model request and response are saved separately and bound by SHA-256. The
run records provider, model identifier, parameters, usage, and proposal. This
is provenance, not a claim that hosted inference is repeatable.

### Local Ollama adapter

`OllamaCallerModel` is the built-in zero-dependency model adapter. It accepts
loopback HTTP endpoints only, refuses redirects, requests one JSON object, and
requires Ollama's input/output token counts.

```python
from hotato.caller import OllamaCallerModel

model = OllamaCallerModel(
    "your-local-caller-model",
    endpoint="http://127.0.0.1:11434",
    temperature=0,
    seed=7,
)
```

The adapter records a zero API charge because the daemon is local. Hardware,
energy, and operator costs are not represented by `cost_microusd`. A hosted
model remains an injected `CallerModel` implementation and must report its
metered charge.

## TTS contract

`CallerTTS.synthesize(text)` returns:

```python
{
    "pcm_s16le": pcm_bytes,
    "sample_rate_hz": 16000,
    "provider": "local-tts",
    "model": "model-id",
    "voice": "voice-id",
    "settings": {"speed": 1.0},
}
```

All fields are required. The run writes the exact PCM16LE bytes and the UTF-8
source text as separate artifacts and records both hashes. No text-only
fallback occurs when a TTS adapter is configured and `send_audio` is absent.

## Evidence package

Each output directory contains:

```text
caller-plan.json
caller-result.json
package-manifest.json
artifacts/
  text/             # exact UTF-8 participant utterances
  audio/            # exact PCM16LE submitted at the caller session boundary
  model-request/    # canonical request per model call
  model/            # canonical response/provenance per model call
```

`package-manifest.json` binds the exact file set, byte sizes, and SHA-256
digests. `verify_package()` rejects missing, changed, symlinked, path-escaping,
and unlisted files. Frozen replay verifies the complete source package before
performing any session action.

`caller-result.json` keeps these lanes separate:

- `actions`: participant outputs and signaling requests;
- `events`: normalized observations supplied by the session adapter;
- `model_calls`: model request/response provenance and usage;
- `actor_state`: scenario-owned working memory;
- `authority`: fixed declarations that outcome is unevaluated and no verdict
  was produced.

`COMPLETED` and `HUNG_UP` exit zero. Capability gaps, invalid model/TTS output,
and resource limits return a verifiable `BLOCKED` package. Adapter exceptions
return a verifiable `ERROR` package. Neither state is converted into an agent
quality verdict.

## Frozen replay

```json
{
  "schema": "hotato.caller-plan.v1",
  "id": "refund-follow-up-replay",
  "mode": "frozen_replay",
  "frozen_package": "artifacts/refund-caller"
}
```

No model or TTS adapter is called. The replay result names the verified source
package ID. Tampering with source text, PCM, model material, plan, or result
blocks replay before a session operation occurs.

## Remaining acceptance work for a concrete transport

This engine does not establish that a particular adapter delivers configured
audio or signaling to a target. Before publishing transport claims, execute a
common fixture through the adapter and retain target-boundary captures for PCM,
DTMF, hold, transfer, disconnect, and impairment behavior. Report unsupported
and unobservable operations as such. Do not infer carrier behavior from a
successful local method call.


================================================================================
FILE: docs/LIFECYCLE.md
================================================================================

# The hotato lifecycle

Turn production failures into portable tests, run candidate releases against
them, and carry evidence with every release. That is the whole product; this
page maps every command onto it.

The loop has five steps you drive, and one that feeds itself:

```text
OBSERVE -> CATCH -> PIN -> TEST -> PROVE
   ^                                  |
   +---------- production ----------- +
```

Eight nouns carry the loop: a **stack** (the platform your agent runs on), a
**recording** (one two-channel call), a **trace** (the call's pipeline events,
from your OTel spans), a **candidate** (a moment hotato flagged for judgment),
a **label** (your one-word verdict on it), a **contract** (the labeled moment,
packaged as a byte-reproducible check), a **suite** (a named set of tests run
together), and a **proof** (every evidence lane composed into one fail-closed
release verdict).

## 1. Observe

Your calls and traces, on your machine. Nothing leaves it.

| What | Command |
| :-- | :-- |
| Store a stack credential once | `hotato connect --stack vapi` |
| Pull recent recordings in bulk | `hotato pull --stack vapi --since 7d` |
| Fetch and score one call | `hotato capture --stack vapi --call-id ID` |
| Capture OTel spans from any process | `hotato observe capture -- <command>` |
| Ingest an OTel export as a voice trace | `hotato trace ingest --otel spans.json` |
| Run the durable production evidence store | `hotato production serve` |
| Score a webhook's completed call | `hotato ingest --stack vapi --event call.json` |

Cost, latency percentiles, and a self-contained dashboard come straight off the
captured traces: `hotato observe cost`, `hotato observe percentiles`,
`hotato observe report`.

## 2. Catch

Deterministic scoring finds what text evals miss: talk-over, dead air, slow
yields, a tool the agent claimed it ran. Five dimensions (outcome, policy,
conversation, speech, reliability), never one blended score, and every number
reproduces byte for byte.

| What | Command |
| :-- | :-- |
| One recording, zero config, the incident list | `hotato autopsy call.wav` |
| One recording, ranked candidate moments | `hotato investigate call.wav` |
| Every recent call on a stack, one sweep | `hotato sweep --stack vapi --since 7d` |
| A folder of recordings, ranked dashboard | `hotato analyze recordings/` |
| Is this recording scorable at all | `hotato trust --stereo call.wav` |
| Why a failing event failed, by layer | `hotato explain result.json` |
| Cluster failures across many runs | `hotato diagnose --fleet runs/` |

A transcript works when there is no audio (`hotato investigate --transcript
t.json`), and a chat agent is driven directly (`hotato simulate scenario.json
--chat URL`). The scorer measures timing and say-do, not intent; a mono or
mixed export is refused as NOT SCORABLE, never guessed at.

## 3. Pin

You make the one human decision the loop needs: is this candidate a failure the
agent must never repeat? Your label turns it into a contract: clipped audio,
frame evidence, the policy it was judged under, and a content address, packaged
portable. It re-verifies on any machine, with or without hotato's help.

| What | Command |
| :-- | :-- |
| Label a candidate, mint the contract | `hotato investigate label STATE#1 --expect yield` |
| Pin a moment from a sweep | `hotato fixture promote SWEEP.json#2 --expect hold` |
| File the confirm-or-ignore decision as a GitHub issue | `hotato issue create sweep.json` |
| Build a contract directly from a moment | `hotato contract create call.wav --onset 2.0 --expect yield` |
| Review and label at fleet scale | `hotato fleet review` / `hotato fleet label` |

## 4. Test

Run candidates against everything you have pinned, plus scripted callers,
personas, robustness batteries, and the bundled stress suite. Simulation is
deterministic: the same scenario and seed render the same bytes on every run.

| What | Command |
| :-- | :-- |
| Render a scenario into a scored conversation | `hotato simulate scenario.json --out ./sim` |
| Drive a live call at a candidate agent | `hotato drive contracts/ID.hotato --stack vapi` |
| Stress one recording across noise and loss | `hotato battery robustness call.wav` |
| The bundled turn-taking stress suite | `hotato gauntlet` |
| A suite of conversation tests, per-dimension | `hotato suite run suite.json` |
| Before/after across a whole battery | `hotato verify --before old/ --after new/` |

## 5. Prove

One command composes every evidence lane you have into one fail-closed,
content-addressed proof: contracts re-verified, suites re-run, before/after
movement measured, the stress suite cleared. The proof headlines its claim
scope, exactly what the evidence establishes: contracts alone re-measure stored
evidence (Captured Evidence), a suite or the stress suite establishes a Test
Suite ran, and a before/after run reaches Candidate Revision only when you bind
the candidate identity (`--candidate-config-hash`, `--provider`). CI gates on
the exit code, and the receipt stays verifiable anywhere.

```bash
hotato prove --contracts contracts/ --before before/ --after after/ --out .hotato/proofs/v42/
```

`pass` means every lane passed. Any failure fails the proof; a lane that could
not support its claim is `inconclusive`, and CI never reads "could not tell" as
green.

The Candidate Revision binding is measured, not asserted. `hotato candidate
hash --provider vapi --assistant <id>` fetches the candidate's configuration,
canonicalizes it (dropping volatile fields like timestamps and ids), and prints
its content hash. Re-run `hotato candidate verify --provider vapi --assistant
<id> --expect <hash>` after the before/after calls: it refuses (exit 1) if the
configuration drifted mid-run, so a Candidate Revision proof cannot survive a
swapped candidate. The end to end flow:

```bash
hotato candidate hash --provider vapi --assistant asst_123    # -> sha256:...
hotato drive contracts/ID.hotato --stack vapi --assistant asst_123 --out after/
hotato candidate verify --provider vapi --assistant asst_123 --expect sha256:...  # exit 1 = drifted, void
hotato prove --before before/ --after after/ --candidate-config-hash sha256:... --provider vapi
```

The deeper machinery is there when you need it: `hotato fix trial` binds a
specific patch to its before/after evidence, `hotato apply --clone` stages a
candidate without touching production, and `hotato record render` turns any
failure into a share-safe card.

## ...and production feeds the next loop

The deploy is not the end of the loop; it is the next input. The production
evidence plane watches completed sessions, raises alerts on evidence gaps, and
exports any session as an offline-verifiable regression candidate, which lands
back in step 2.

| What | Command |
| :-- | :-- |
| Continuous evidence intake | `hotato production serve` / `production ingest` |
| Alert transitions on session evidence | `hotato production alerts` |
| A session back into the loop as a test | `hotato production export-regression SESSION` |
| Canary a config change, roll it back | `hotato fleet canary start` / `rollback` |
| Team trends over releases | `hotato team` / `hotato release compare` |

## Where the other commands sit

`hotato start`, `demo`, `doctor`, and `init --auto` are onboarding: they walk
the loop on bundled calls before you spend a minute of your own. `hotato bench`
and `hotato gauntlet badge` prove the scorer itself. `hotato describe` emits
the machine-readable manifest a coding agent drives the loop with, and the MCP
server (`hotato-mcp`) exposes the same loop over stdio.


================================================================================
FILE: docs/LIVEKIT-CALLER-SESSION.md
================================================================================

# Direct LiveKit caller session

`hotato.livekit_session.LiveKitCallerSession` is an optional transport for the
bounded scripted, hybrid, generative, and frozen-replay caller engine. It joins
a LiveKit room as a participant, publishes PCM16LE through one audio track,
receives the target participant's audio and transcription events, and submits
SIP DTMF through the official Python RTC SDK.

This adapter removes the custom WebSocket sidecar requirement for a LiveKit
agent. It does not implement SIP, a carrier, STT, TTS, packet impairment, or a
telephony transfer service. A PSTN test still needs LiveKit SIP and a configured
trunk. LiveKit documents `publish_dtmf` as publishing a SIP DTMF message; a
successful SDK call is not evidence that a downstream carrier honored it.

Official transport references:

- [LiveKit Python SDK](https://github.com/livekit/python-sdks)
- [Python RTC API](https://docs.livekit.io/reference/python/livekit/rtc/index.html)
- [AudioSource queue and playout contract](https://docs.livekit.io/reference/python/livekit/rtc/audio_source.html)
- [AudioStream bounded-capacity option](https://docs.livekit.io/reference/python/livekit/rtc/audio_stream.html)

## Install and run

```bash
pip install 'hotato[livekit]'
```

Mint a short-lived room token with only the permissions needed by the caller
participant. Keep token generation outside the test plan and result package.

The CLI can join the room directly and synthesize caller turns with a local
Piper model. The token enters through an environment variable or regular file,
never a command-line value:

```bash
export LIVEKIT_CALLER_TOKEN="$(mint-short-lived-room-token)"

hotato caller run scenarios/refund-barge-in.json \
  --livekit-url wss://livekit.example.test \
  --livekit-target-identity agent-under-test \
  --livekit-token-env LIVEKIT_CALLER_TOKEN \
  --livekit-evidence-topic hotato.evidence.v1 \
  --piper-model models/en_US-lessac-medium.onnx \
  --piper-config models/en_US-lessac-medium.onnx.json \
  --allow-remote \
  --out artifacts/refund-barge-in
```

`--livekit-token-file` is the file-backed alternative. `--target-ws` remains
the mutually exclusive sidecar transport. The CLI derives the LiveKit audio
rate from `audio.sample_rate` in the Piper config; an explicit
`--livekit-sample-rate` must match it.

```python
import os

from hotato.caller import load_plan, run_caller
from hotato.livekit_session import LiveKitCallerSession

# Your TTS adapter must return Hotato's documented PCM + provenance mapping.
from my_test_adapters import LocalTTS

session = LiveKitCallerSession(
    os.environ["LIVEKIT_URL"],
    os.environ["LIVEKIT_CALLER_TOKEN"],
    target_identity="agent-under-test",
    allow_remote=True,                  # explicit network-egress decision
    sample_rate_hz=48_000,              # TTS output must match exactly
    evidence_topic="hotato.evidence.v1",
)

try:
    run = run_caller(
        load_plan("scenarios/refund-barge-in.json"),
        session,
        "artifacts/refund-barge-in",
        tts=LocalTTS(sample_rate_hz=48_000),
    )
    print(run.result["status"], session.media_summary())
finally:
    session.close()
```

`send_text` is `UNSUPPORTED`: a LiveKit data message is not spoken caller
audio. Supply a TTS adapter so caller `say` and model-generated `say` actions
produce PCM. Sample-rate conversion is also explicit; the transport refuses a
TTS result whose sample rate differs from the configured `AudioSource`.

## Evidence semantics

The adapter records each layer without promoting one into another:

| Event | What it establishes | Authority |
|---|---|---|
| `audio_submitted` | `AudioSource.capture_frame` accepted every frame and `wait_for_playout` completed | local LiveKit SDK |
| `received_audio_frame` | decoded PCM reached the Hotato caller participant | local LiveKit receiver |
| `delivered_audio_receipt` | a cooperating target participant reported the bytes at its named boundary | target participant reported |
| `dtmf_submitted` | the local participant's `publish_dtmf` calls completed | local LiveKit SDK |
| native transcription | LiveKit emitted a transcription event for the target identity | LiveKit room event |

Every received audio frame includes its SHA-256 and a rolling SHA-256 over the
ordered PCM stream. `media_summary()` returns only digests and counters. Raw PCM
remains in the caller engine's content-addressed package when the scenario and
TTS lane write it there.

`evidence()` binds that digest-only media summary and the five evidence
capability states into `caller-result.json.session_boundary`. It contains no
room token, endpoint query, raw audio, transcript text, or target payload.

`sdk_playout_complete` does not establish target, SIP, PSTN, or carrier
delivery. `evidence_capabilities()["outgoing_audio_delivery"]` remains
`UNOBSERVABLE` until a valid, session-bound target receipt arrives.

## Optional target evidence channel

Set `evidence_topic` only when the target participant implements this strict
envelope on reliable LiveKit data messages:

```json
{
  "schema": "hotato.livekit-evidence.v1",
  "session_id": "<session id from the control message>",
  "sequence": 1,
  "kind": "audio_receipt",
  "payload": {
    "submission_sequence": 1,
    "submitted_sha256": "sha256:<announced caller PCM digest>",
    "delivered_sha256": "sha256:<PCM digest measured at the target boundary>",
    "delivered_bytes": 19200,
    "sample_rate_hz": 48000,
    "channels": 1,
    "boundary": "agent-input-after-decode"
  }
}
```

The default control topic is `hotato.control.v1`. The adapter sends
`session_started` and `audio_submission` messages only to `target_identity`.
The target copies the session id and announced submission digest into its
receipt, then supplies its separately measured delivered digest. Receipt
sequence must be contiguous. Duplicate, reordered, wrong-session, malformed,
oversized, and wrong-submission receipts become
`target_evidence_rejected` events; their payload is represented by a digest,
not copied into the result.

The same evidence envelope accepts `transcript`, `tool_result`,
`state_snapshot`, `transfer`, `hold`, `timing`, and `custom`. These remain
`target_participant_reported`. They can supply evidence to Hotato's later
deterministic assertions; this transport never produces a pass/fail verdict.

## Resource and failure behavior

- Remote egress is refused unless `allow_remote=True`; loopback remains the
  default.
- The URL refuses embedded credentials, query strings, and fragments.
- Before publishing the caller track, the driver applies LiveKit track
  subscription permissions that allow only `target_identity`. Control and
  evidence data messages are also addressed only to that identity.
- The short-lived token is passed to the SDK during connection and never copied
  into the facade, control messages, event queue, `repr`, or media summary. The
  SDK necessarily retains connection credentials while its room is connected.
- PCM per send, PCM per session, per-send duration, event count, remote track
  count, control message size, frame duration, connection time, operation time,
  and close time are bounded.
- Caller media operations are serialized. A concurrent audio/DTMF call is
  refused instead of interleaving frames from two scenario actions.
- The SDK's remote `AudioStream` uses a nonzero capacity. The SDK's behavior at
  that internal capacity boundary remains transport behavior to measure. The
  separate Hotato event queue is bounded and its overflow fails the run instead
  of silently dropping evidence.
- Driver errors expose the operation and exception type without copying SDK
  messages that may contain endpoint details.
- The adapter has no hidden retry. Load scheduling and retry policy belong to
  the load controller, where attempts remain observable.

## Acceptance gate before a transport claim

The unit suite validates protocol mapping against a fake SDK. Publish a LiveKit
transport result only after an environment-specific run captures:

1. pinned LiveKit server, SIP server, Python SDK, agent, STT, TTS, and carrier
   versions;
2. the caller package and rolling received-stream digest;
3. a target-boundary receipt, if target delivery is claimed;
4. SIP participant state and carrier evidence for PSTN claims;
5. network capture and egress destinations;
6. disconnect, reconnect, event-overflow, token-expiry, and media-timeout cases.

Until that acceptance run exists, describe this as an implemented optional
LiveKit caller transport with fake-SDK contract tests, not as a measured
carrier-grade path.


================================================================================
FILE: docs/LOAD-AND-RECOVERY.md
================================================================================

# Load and recovery with portable evidence

Hotato schedules calls in either a closed concurrency model or an open arrival
model. It preserves one verifiable child package per started call. Provider
completion, delivered-media evidence, tool/state evidence, scheduling loss,
and recovery remain separate measurements.

```json
{
  "schema": "hotato.load-plan.v2",
  "id": "refund-release",
  "call": {
    "schema": "hotato.telephony-call.v1",
    "id": "base",
    "provider": "vapi",
    "to": "+15551234567",
    "agent_id": "agent-id",
    "phone_number_id": "phone-id"
  },
  "stages": [
    {"name": "warm", "phase": "warmup", "model": "closed", "concurrency": 2, "calls": 10},
    {"name": "spike", "phase": "spike", "model": "open", "arrival_rate_per_second": 4, "duration_seconds": 30, "max_in_flight": 40},
    {"name": "recover", "phase": "recovery", "model": "closed", "concurrency": 4, "calls": 20}
  ],
  "safety": {
    "max_calls": 150,
    "estimated_cost_per_call_usd": 0.05,
    "max_estimated_cost_usd": 7.50,
    "allowed_destinations": ["+1555"],
    "stop_file": ".hotato/STOP"
  },
  "slos": {
    "max_dropped_start_rate": 0.01,
    "max_p95_scheduling_delay_seconds": 0.25,
    "min_lifecycle_completion_rate": 0.99,
    "min_evidence_complete_rate": 0.99
  }
}
```

An open stage does not queue work when its generator is saturated. It records a
`generator_saturated` dropped start so coordinated omission remains visible.
Closed stages keep a fixed number of calls in flight and start a replacement
only after one finishes.

The safety block is evaluated before a provider call. Twilio, Vapi, and Retell
workloads require a v2 plan with a positive per-call estimate, a positive total
budget, and a non-empty destination allowlist. Omitting any one refuses the
workload before the first provider request. Destination prefixes, a hard call
ceiling, and an operator stop file bound execution. Hotato performs zero
automatic create retries because the provider-neutral interface has no shared
idempotency guarantee.

Each `calls/*/manifest.json` binds redacted lifecycle receipts and any evidence
export to the provider, provider-namespaced call pseudonym, normalized call
digest, and normalized workload digest. The four evidence states are
`PRESENT`, `MISSING`, `UNSUPPORTED`, and `UNOBSERVABLE`. `PRESENT` requires a
content digest and an explicit authority for every lane; a bare string cannot
satisfy an evidence SLO. `min_evidence_complete_rate` is stricter than artifact
presence: all three non-lifecycle lanes must be `PRESENT` and carry either
`measured` or `signed_attestation` authority, with
`eligible_for_execution_claim=true`. Provider-, sidecar-, target-, and
unverified reports remain portable evidence but cannot satisfy that execution
evidence SLO. A provider-reported completed call can satisfy lifecycle
completion while evidence completeness remains zero.

`hotato load telephony verify DIR` hashes every child and recomputes per-stage and aggregate
metrics, recovery windows, SLO rows, the HTML report, overall status, and exit
code from `observations.jsonl` plus the bound, non-secret
`verification-plan.json`. It rejects missing/duplicate observations, swapped
child references, extra artifacts, invalid evidence states, and a summary that
has merely been rehashed after its verdict changed. No network or model is
required.

The package hashes establish internal identity and consistency. They do not
authenticate who produced a directory: a party able to replace every artifact
can construct a different, internally consistent package. Preserve the package
in an access-controlled artifact store or attach an external organization
signature before using publisher identity as a trust decision.

Fault schedules require a call controller that exposes the named injection.
Unsupported injection produces an explicit call error; it is never simulated
silently. Recovery is measured from the end of the declared fault window to the
first later successful lifecycle observation and stays separate from quality.


================================================================================
FILE: docs/OBSERVE.md
================================================================================

# observe: LLM/voice observability, derived on your machine

`hotato observe` reads the observability an LLM or voice agent already emits
as OpenTelemetry spans, on your machine, from files you already have. One
command group, four subcommands, all built on the same
`hotato.voice_trace.v1` spans [`docs/TRACE.md`](TRACE.md) /
[`docs/OTEL.md`](OTEL.md) describe:

```bash
hotato observe capture --out voice_trace.jsonl -- python agent.py
hotato observe cost voice_trace.jsonl --prices starter
hotato observe percentiles traces/ --html latency.html
hotato observe report traces/ --out observe.html --prices starter
```

Every number is derived locally: token counts read from your spans, latency
from their timestamps, USD from a price table you keep. hotato opens no
socket, runs no listener, and makes no network call of its own.

## capture: a local file sink, no account

`hotato observe capture -- <command...>` runs the child process with a local
file sink wired through its environment, then ingests whatever the child
wrote:

- `HOTATO_OTEL_FILE` -- the file path hotato names for the spans.
- `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` -- a
  `file://` URL pointing at that same path (not a host:port, so there is
  nothing for an exporter to dial out to).
- `OTEL_EXPORTER_OTLP_PROTOCOL=http/json` -- the plain-text OTLP encoding.

A cooperating process writes its spans to that file: either as a standard
OTel JSON export, or as hotato's own OTel bridge JSONL written directly to
`$HOTATO_OTEL_FILE` (the shape in [`docs/OTEL.md`](OTEL.md)). On the child's
exit, hotato ingests the file through the same `trace ingest` path into
`--out` and prints a one-screen summary: span count, per-hop latency, and
token totals. If the child wrote no spans, capture refuses (exit 2) and
leaves nothing behind. The child's own network is the child's business;
hotato adds none.

```bash
hotato observe capture --out run.jsonl -- python agent.py
# observe capture: 7 spans -> run.jsonl
#   child exit: 0
#   per-hop latency:
#     STT (speech to text)      : 550.0 ms
#     LLM (first token)         : 250.0 ms
#     Tool call                 : 320.0 ms
#   tokens:
#     input 1200  output 340  cached not captured  reasoning not captured
```

## cost: tokens are facts, USD is a local estimate

`hotato observe cost <voice_trace.jsonl>` sums each span's LLM token usage
per model and in total. Token attributes resolve by first-match alias:

| category  | attributes (first match wins)                                       |
|-----------|---------------------------------------------------------------------|
| input     | `gen_ai.usage.input_tokens`, `gen_ai.usage.prompt_tokens`, `input_tokens`, `prompt_tokens` |
| output    | `gen_ai.usage.output_tokens`, `gen_ai.usage.completion_tokens`, `output_tokens`, `completion_tokens` |
| cached    | `gen_ai.usage.cached_tokens`, `gen_ai.usage.cache_read_input_tokens`, `cached_tokens` |
| reasoning | `gen_ai.usage.reasoning_tokens`, `reasoning_tokens`                  |

The model is `gen_ai.response.model` (else `gen_ai.request.model`,
`gen_ai.model`, `llm.model_name`, `model`).

Tokens are facts read from the spans. A category no span reported reads
**not captured** (null, plus a count of the spans that lacked it), never 0.

`--prices FILE` adds an estimated USD cost from a **local** per-model $/1M
table (`--prices starter` reads the bundled `src/hotato/data/prices.yaml`).
Copy that file, set the rates from your own agreements, and pass it. USD is
labeled "estimated from `<table>`": the tokens are measured, the dollars are
your arithmetic over your table. A model with no row in the table is
**unpriced** (its cost reads null), never a guessed rate; a category with no
rate in a row is simply not billed.

```yaml
# your-rates.yaml (USD per 1,000,000 tokens)
table: my-contract
per_tokens: 1000000
models:
  gpt-4o:
    input: 2.5
    output: 10.0
    cached: 1.25
```

## percentiles: nearest-rank over a folder

`hotato observe percentiles DIR` reports p50 / p90 / p99 of each per-hop
latency (STT, LLM, tool, TTS, transport) and of end-to-end latency over every
readable voice trace in `DIR`, by the nearest-rank method
(`hotato._stats.nearest_rank`): with the measured values sorted ascending and
n of them, `rank = ceil(q * n)` and the percentile is the value at that rank.
Every percentile is therefore an observed measurement, re-derivable by hand.

A trace that did not capture a hop is **excluded** from that hop's
percentiles, and the excluded count is shown, so an uncaptured hop is never
counted as 0. `--format json` and `--html PATH` render the same numbers as a
machine envelope and a self-contained panel.

## report: one self-contained HTML page

`hotato observe report DIR --out observe.html` writes one self-contained HTML
page (inline CSS and SVG, no external request) summarizing trace and span
counts, per-hop latency and its percentiles, per-model token totals and an
estimated USD line (with `--prices`), and links to the slowest traces. It
opens offline and embeds every value it shows.

## Determinism

The same inputs render byte-identical text, JSON, and HTML. No artifact
embeds a wall clock. `hotato observe` reads files and writes files; it holds
no state and reaches nowhere.


================================================================================
FILE: docs/PIPER-CALLER-TTS.md
================================================================================

# Local Piper caller speech

`hotato.piper_tts.PiperCallerTTS` turns caller `say` actions into mono PCM16LE
with a local Piper executable. It is an optional adapter around the Piper CLI,
not a bundled speech model and not an audio-quality claim.

## CLI

Supply both the ONNX model and its matching config:

```bash
hotato caller run scenarios/appointment.json \
  --target-ws ws://127.0.0.1:8765/caller \
  --piper-model models/en_US-lessac-medium.onnx \
  --piper-config models/en_US-lessac-medium.onnx.json \
  --out artifacts/appointment
```

The same flags provide spoken input for the direct LiveKit caller session. The
config must declare `audio.sample_rate`. Hotato publishes that rate without an
implicit resample; an incompatible transport rate is refused.

Useful bounds are explicit:

```text
--piper-command PATH             default: piper
--piper-timeout SEC              default: 60
--piper-max-output-bytes BYTES   default: 33554432
--piper-voice LABEL              default: default
```

The voice label is descriptive provenance. It does not select a Piper speaker;
use a single-speaker model or configure speaker selection in a separately
reviewed adapter before claiming multi-speaker coverage.

## Execution boundary

For each CLI run, the adapter:

1. refuses model/config symlinks, FIFOs, devices, oversized files, and files
   changed while they are staged;
2. copies the exact model and config bytes once into a mode-`0600` private
   temporary directory;
3. invokes the resolved Piper executable with an argument vector and
   `shell=False`;
4. supplies a minimal environment instead of forwarding operator secrets;
5. bounds input text, PCM stdout, diagnostic stderr, execution time, staged
   model/config bytes, and executable bytes;
6. accepts only non-empty, even-length `--output_raw` bytes; and
7. deletes the private staging directory when the caller run closes.

No diagnostic text enters the package. Successful synthesis records the staged
model/config SHA-256 values, the resolved executable SHA-256 observed before
and after execution, sample rate, mono PCM16LE encoding, diagnostic byte count,
and diagnostic digest. Model/config paths, the private temporary path,
environment values, and Piper stderr contents are excluded.

The resulting PCM is content-addressed again by the caller engine before it is
submitted to the transport. Piper success establishes local synthesis only.
`sdk_playout_complete`, target delivery, SIP delivery, and carrier delivery are
separate evidence boundaries.

## Acceptance gate

The hermetic suite exercises argument-vector invocation, bounded output,
timeout, symlink refusal, private-environment behavior, PCM structure, cleanup,
and provenance. Before publishing a voice-quality or performance result, pin
the Piper release, executable digest, model license and digest, config digest,
hardware, input corpus, and the metric and adjudication method used.


================================================================================
FILE: docs/PRODUCTION-MONITORING.md
================================================================================

# Production monitoring: durable evidence into a regression candidate

Hotato's production plane accepts bounded call events, preserves ambiguity, and
turns a finalized session into an offline-verifiable regression candidate. It
is a single-process, self-hosted evidence assembler backed by SQLite WAL. Put a
durable OpenTelemetry Collector or message queue in front when a deployment
needs ingress buffering across process restarts. The included
[`deploy/control-plane`](../deploy/control-plane/README.md) bundle does this
with Collector Contrib and a local persistent queue. That queue survives a
process restart; it does not replicate data across hosts.

The implementation lives in `hotato.production`. It has no runtime dependency
beyond Python's standard library.

The gateway accepts OTLP/HTTP JSON traces at the standard `/v1/traces` path
and at `/v1/otlp/traces`. Protobuf input is refused rather than guessed.
Authentication is required on this boundary. The deployed Collector accepts
standard OTLP/gRPC and OTLP/HTTP protobuf or JSON on loopback, converts trace
batches to JSON, and authenticates the private hop to this gateway.
After its database transaction commits, `/v1/traces` returns the standard empty
`ExportTraceServiceResponse` JSON object (`{}`). The Hotato-specific
`/v1/otlp/traces` compatibility path returns the richer per-event receipt.

## The write contract

The gateway follows this order for every request:

1. bound `Content-Length` before reading;
2. authenticate the exact body bytes with bearer token or HMAC;
3. parse and validate the event;
4. enter an SQLite `BEGIN IMMEDIATE` transaction;
5. persist the event, session update, and audit-chain entry;
6. commit with `PRAGMA synchronous=FULL`;
7. return a receipt containing `"durability": "committed"`.

A storage error returns `503`. A validated event never receives a success ACK
before its transaction commits. The HTTP server admits a bounded number of
workers and returns `503` with `Retry-After: 1` when capacity is full. There is
no hidden in-memory queue.

## Start the gateway

For an operated service, declare the maintenance loop as data instead of
depending on an operator to remember three separate commands:

```json
{
  "schema": "hotato.production-maintenance.v1",
  "interval_seconds": 30,
  "quiescence_seconds": 30,
  "required_lanes": [
    "participant_audio",
    "transcript",
    "model_trace",
    "tool_calls",
    "backend_state"
  ],
  "alert_rules": [
    {"id": "degraded-session", "condition": "degraded"},
    {"id": "event-conflict", "condition": "conflict"},
    {"id": "missing-audio", "condition": "missing_audio"}
  ],
  "retention_seconds": 2592000
}
```

Run one inspectable cycle:

```bash
hotato production maintain maintenance.json \
  --db .hotato/production.sqlite3 --format json
```

Or run the authenticated gateway and maintenance loop together while keeping
the bearer value out of argv and the container environment:

```bash
hotato production serve \
  --db .hotato/production.sqlite3 \
  --token-file /run/secrets/hotato_production_token \
  --maintenance-policy maintenance.json
```

Each cycle finalizes eligible sessions first, evaluates persisted alert state
second, and enforces retention last. A cycle failure is retained in supervisor
status and retried at the next interval; it never receives a success ACK from
the ingest gateway because ingest durability is a separate transaction. The
supervisor does not create assertions, score calls, export regression
candidates, or send notifications.

The lower-level Python gateway remains available:

```python
from hotato.production import ProductionGateway, ProductionStore

store = ProductionStore(".hotato/production.sqlite")
gateway = ProductionGateway(
    store,
    token="replace-with-at-least-16-characters",
    hmac_secret="replace-with-at-least-32-characters",
    host="127.0.0.1",
    port=8099,
    max_workers=16,
)

try:
    gateway.thread.join()
finally:
    gateway.close()
    store.close()
```

Binding to a non-loopback address exposes an authenticated HTTP service. Place
TLS termination in front of it and rotate credentials through the surrounding
secret manager.

## Event envelope

Every event conforms to
[`production-event.v1.json`](../src/hotato/schema/production-event.v1.json).
The envelope is CloudEvents-shaped and adds required execution authority:

```json
{
  "specversion": "1.0",
  "id": "tool-result-00017",
  "source": "livekit-sidecar",
  "type": "tool.result",
  "subject": "call-01J2Q1N4WQ",
  "time": "2026-07-17T12:00:00.123Z",
  "sequence": 17,
  "traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
  "data": {
    "availability": "available",
    "tool": "cancel_subscription",
    "result": {"status": "failed"}
  },
  "authority": {
    "kind": "adapter_reported",
    "eligible_for_execution_claim": false
  }
}
```

Supported event types are enumerated in the schema. Unknown types and unknown
top-level fields are refused. Each event is capped at 8 MiB.

### Availability and authority remain separate

Evidence availability is one of:

- `available`: the event carries or references evidence;
- `unavailable`: the producer expected the evidence and could not obtain it;
- `unsupported`: the producer cannot provide that evidence lane;
- `missing`: no event has yet described the lane.

Authority is one of:

- `measured`;
- `signed_attestation`;
- `provider_export`;
- `adapter_reported`;
- `submitted`.

Only `measured` and `signed_attestation` may set
`eligible_for_execution_claim: true`. Request authentication establishes who
submitted the envelope. It never upgrades an adapter or provider export into an
independent execution measurement.

The session manifest reports five independent evidence lanes:

- participant audio;
- transcript;
- model trace;
- tool calls;
- backend state.

No lane substitutes for another. The manifest has no blended score.

## Authentication

### Bearer

```bash
curl -sS http://127.0.0.1:8099/v1/events \
  -H 'Authorization: Bearer replace-with-at-least-16-characters' \
  -H 'Content-Type: application/json' \
  --data-binary @event.json
```

### HMAC

HMAC signs the exact bytes sent in the request:

```text
hex(HMAC-SHA256(secret, decimal_unix_timestamp + "." + raw_body))
```

Send the result as `X-Hotato-Signature: v1=<hex>` and the timestamp as
`X-Hotato-Timestamp`. The default acceptance window is 300 seconds. Event
identity deduplication makes an accepted replay idempotent.

```python
import hashlib, hmac, json, time, urllib.request

secret = b"replace-with-at-least-32-characters"
raw = json.dumps(event, separators=(",", ":")).encode()
timestamp = str(int(time.time()))
signature = hmac.new(secret, timestamp.encode() + b"." + raw, hashlib.sha256).hexdigest()
request = urllib.request.Request(
    "http://127.0.0.1:8099/v1/events",
    data=raw,
    headers={
        "Content-Type": "application/json",
        "X-Hotato-Timestamp": timestamp,
        "X-Hotato-Signature": "v1=" + signature,
    },
)
urllib.request.urlopen(request).read()
```

## Duplicate, conflict, and ordering behavior

Event identity is `(source, id)`.

| Arrival | Stored result | Session effect |
|---|---|---|
| unseen identity | `stored` | event count and evidence update |
| same identity, same canonical bytes | `duplicate` | duplicate counter increments |
| same identity, different canonical bytes | HTTP `409` | original event remains; conflict is recorded; session becomes `DEGRADED` |
| sequence lower than or equal to the source cursor | `out_of_order` | event remains stored; out-of-order counter increments |
| no sequence | `stored` | unsequenced counter increments; finalization is `DEGRADED` |

Sequence cursors are scoped to `(subject, source)`, so independent producers may
each begin at sequence zero. Arrival order remains in the event export. Hotato
does not discard or silently reorder late observations.

An event arriving after session finalization remains stored and changes the
manifest to `DEGRADED` with `late_event_after_finalization`. The previously
exported candidate remains content-addressed and unchanged.

## Finalization

`session.ended` moves a session to `QUIESCENT`. Call `finalize()` after the
declared quiescence period:

```python
manifests = store.finalize(quiescence_seconds=30)
```

By default, a session becomes `COMPLETE` only when exactly one
`session.started` and one `session.ended` event were observed, all five evidence
lanes are available, no identity conflict occurred, and no out-of-order event
was observed. Every event must also carry a sequence from its source. Lifecycle
and unsequenced counts remain explicit in the manifest. A fixed
workflow that does not produce every evidence lane can declare its required
subset at finalization:

```python
manifests = store.finalize(
    quiescence_seconds=30,
    required_lanes=("participant_audio", "transcript", "backend_state"),
)
```

The selected list is persisted in `required_evidence_lanes`; it cannot disappear
from the exported manifest. Every other finalized session is `DEGRADED`; the
manifest identifies each unavailable or missing lane and the finalization
reason.

Finalization measures evidence completeness. It does not decide whether the
agent passed a product assertion.

## OpenTelemetry trace bridge

`POST /v1/traces` and `POST /v1/otlp/traces` accept bounded OTLP/HTTP JSON with
the standard `resourceSpans -> scopeSpans -> spans` shape. Use
`X-Hotato-Source` to name the sidecar. The former returns `{}` for OTLP client
compatibility after commit; the latter returns Hotato's normalized result and
durability fields. Each span must carry one of these attributes:

- `hotato.session_id`;
- `session.id`;
- `conversation.id`;
- `call.id`.

Optional attributes:

- `hotato.event_type`: one supported production event type;
- `hotato.sequence`: integer ordering within that source;
- `hotato.evidence.availability`: `available`, `unavailable`, or `unsupported`.

The bridge preserves trace ID, span ID, parent span ID, resource attributes,
scope, status, and nanosecond timestamps. It assigns `adapter_reported`
authority. A caller may use `normalize_otlp_json()` directly and select a
different authority, subject to the same structural validation.

The OTLP endpoint validates the complete span set before opening its transaction
and commits the normalized batch atomically. Identity conflicts are returned as
per-event results while preserving the original event bytes.

### Included Collector boundary

The control-plane deployment renders these local-only inputs:

```text
OTLP/gRPC          127.0.0.1:4317
OTLP/HTTP   http://127.0.0.1:4318
```

The Collector uses the stable `otlp_http` exporter name, JSON encoding, and no
compression for its private hop to `127.0.0.1:8432/v1/traces`. The Hotato bearer
credential exists only in the OTel-specific mode-`0400` config volume. It is
absent from the Collector environment, process arguments, and bootstrap
manifest.

Exporter requests are staged in an fsyncing `file_storage` queue before
delivery. The fixed queue has a capacity of 10,000 requests and retries an
enqueued request without an elapsed-time cutoff. This is bounded, single-host
buffering:

- queue units are exporter requests, not spans, calls, or bytes;
- the storage volume has no application-level byte quota;
- storage compacts at startup and after its allocated database exceeds 100 MiB
  and later drains below 10 MiB; compaction itself needs temporary disk
  headroom;
- when the queue is full, the disk is full, or storage returns an I/O error,
  enqueue can fail and telemetry can be dropped;
- Collector acceptance is not a Hotato SQLite commit receipt;
- deleting or losing the host volume loses any request that has not drained.

The Collector exposes its own metrics only on
`http://127.0.0.1:8888/metrics`. Monitor
`otelcol_exporter_queue_size`, `otelcol_exporter_queue_capacity`,
`otelcol_exporter_enqueue_failed_spans`,
`otelcol_exporter_send_failed_spans`, receiver-refused spans, process restarts,
and disk space. Treat any nonzero enqueue-failure delta as an evidence-loss
incident. Run restart, gateway-outage, queue-capacity, and disk-exhaustion drills
against the exact pinned image digest before qualifying a deployment.

The rendered settings follow the pinned Collector `v0.153.0` contracts for the
[OTLP HTTP exporter](https://github.com/open-telemetry/opentelemetry-collector/blob/v0.153.0/exporter/otlphttpexporter/README.md),
[exporter helper persistent queue](https://github.com/open-telemetry/opentelemetry-collector/blob/v0.153.0/exporter/exporterhelper/README.md),
and Contrib
[file storage extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.153.0/extension/storage/filestorage/README.md).

This endpoint is a trace normalizer. The included Collector pipeline also wires
traces only; application metrics and logs require a separately declared
pipeline and backend. The Prometheus endpoint above contains the Collector's
own operational metrics, not application telemetry.

## Alerts

The local alert engine persists each transition and an audit-chain entry:

```python
changes = store.evaluate_alerts([
    {"id": "evidence-complete", "condition": "incomplete_evidence"},
    {"id": "event-conflict", "condition": "conflict"},
])
```

Supported conditions:

- `degraded`;
- `missing_audio`;
- `missing_tool_evidence`;
- `incomplete_evidence`;
- `conflict`;
- `out_of_order`;
- `unsequenced`.

Transitions use `FIRING` and `RESOLVED`. A resolved alert that fires again gets
a new generation and opened timestamp. Stable alert states produce no duplicate
transition.

## Prometheus

`GET /metrics` requires bearer authentication. Event, duplicate, conflict, and
ordering-anomaly counters carry no labels. Session `status` and alert `state`
use fixed enumerations. Session IDs, rule IDs, providers, phone numbers, and
trace IDs never become labels, preventing unbounded series cardinality.
Counters live in a dedicated SQLite table and remain monotonic when retention
deletes event/session rows.

## Promote a production failure

Finalize the session, then export it:

```python
result = store.export_regression_candidate(
    "call-01J2Q1N4WQ",
    "fixtures/candidates/call-01J2Q1N4WQ",
)
```

The exporter writes to an owned staging directory, fsyncs the files, verifies
all declared byte counts and SHA-256 digests, and atomically renames the
directory into place. It refuses to overwrite an existing path.

The output contains:

```text
call-01J2Q1N4WQ/
├── candidate.json
└── events.jsonl
```

`candidate.json` conforms to
[`production-regression-candidate.v1.json`](../src/hotato/schema/production-regression-candidate.v1.json).
Its status is `CANDIDATE`: production evidence alone does not invent an expected
outcome or become a CI gate. A reviewer authors the assertion and chooses which
evidence may enter a shareable fixture.

Verify on a clean, offline machine:

```python
from hotato.production import verify_regression_candidate

result = verify_regression_candidate("fixtures/candidates/call-01J2Q1N4WQ")
assert result["valid"]
```

Payload persistence is default-deny at ingest. Each event type has a narrow
allowlist for structural values such as evidence availability, validated
content digests, timing counters, and bounded status enums. Every other scalar,
object, or array is replaced as a whole by `redacted: true`, its canonical
`byte_count`, and no value-derived digest. Omitting the digest prevents an
offline dictionary attack against low-entropy redacted values. An allowlisted
field with the wrong value type is reduced to the same descriptor. This
preserves portable monitoring structure without assuming an unfamiliar field
is safe.
Field names and the event envelope remain visible, so identifiers still need
review before a candidate is shared. A caller can explicitly use
`store.ingest(event, redact_payloads=False)` inside its own boundary; the
session manifest and candidate then declare `payload_storage`/`payloads` as
`unredacted` or `mixed` instead of claiming redaction. Review remains required
before sharing because envelope identifiers and allowlisted structural metadata
may still be organization-specific.

## Audit and retention

Verify the append-only audit hash chain:

```python
assert store.verify_audit_chain()["valid"]
```

Audit targets are deterministic SHA-256 digests from their first write, so the
raw provider identifier does not remain in the audit row after retention.
These digests are integrity identifiers, not an anonymity guarantee: a
low-entropy identifier can be guessed offline. Chain verification returns the
first invalid sequence if a row changes and compares the computed head/count
with a local metadata checkpoint to detect tail truncation. An administrator
who can rewrite the entire database can rewrite both the chain and its local
checkpoint; export the returned head digest to an external append-only system
when that threat is in scope.

Delete one session:

```python
receipt = store.delete_session("call-01J2Q1N4WQ")
```

Apply a local retention window to finalized sessions:

```python
receipts = store.enforce_retention(retention_seconds=30 * 24 * 60 * 60)
```

Deletion removes events, conflicts, sequence cursors, session state, alerts, and
their transitions. A pseudonymous deletion receipt and the pseudonymous audit
chain remain. The receipt records the pre-deletion manifest digest and deleted
event count. Database backups and upstream collectors need their own retention
policy; this process cannot delete copies outside its store.

## Operational boundary

The included implementation has hermetic tests for authentication, exact-byte
HMAC verification, commit visibility from a second connection, duplicate and
conflict behavior, source-scoped ordering, evidence authority, finalization,
late events, alert generations, bounded-label metrics, OTLP correlation,
candidate tamper detection, audit tamper detection, and retention receipts.

Those tests establish the local contract. They do not establish a public
throughput, recovery-time, multi-region, or availability number. Measure those
properties in the intended deployment with a frozen load schedule and publish
the environment and raw results before making an operational claim.


================================================================================
FILE: docs/README.md
================================================================================

# hotato docs

Find what broke in your agent calls. Pin it so it never ships again. Every
production failure becomes a portable test, every candidate runs against it,
and every release carries evidence. This index maps every doc to the step it
belongs to.

New here? Run `hotato autopsy ./call.wav` on one recording
(**[AUTOPSY.md](AUTOPSY.md)**), or start with
**[GETTING-STARTED.md](GETTING-STARTED.md)**. The whole loop on one page:
**[LIFECYCLE.md](LIFECYCLE.md)**.

## Getting started

- [GETTING-STARTED.md](GETTING-STARTED.md) - one path from first touch to a CI gate
- [START.md](START.md) - guided first run on the bundled demo data
- [STARTER.md](STARTER.md) - `hotato init starter` scaffolds a CI gate and config
- [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md) - turn one bad call into a CI gate
- [WHY.md](WHY.md) - four timing failures a text-level eval cannot see

## Observe

- [OBSERVE.md](OBSERVE.md) - LLM and voice observability from your OpenTelemetry spans, locally
- [TRACE.md](TRACE.md) - voice traces: the pipeline-event timeline
- [OTEL.md](OTEL.md) - ingest OTel traces into the `voice_trace` span format
- [latency-waterfall.md](latency-waterfall.md) - per-hop latency waterfall from a scored call
- [PRODUCTION-MONITORING.md](PRODUCTION-MONITORING.md) - turn production call events into offline regression candidates

## Evaluate

- [AUTOPSY.md](AUTOPSY.md) - one recording, zero config: the incident list and report; the folder health report (`scan DIR`) and `pin` to a contract
- [INVESTIGATE.md](INVESTIGATE.md) - one recording in, ranked candidate moments out
- [ANALYZE.md](ANALYZE.md) - drop a folder, rank and hear the worst moments
- [ASSERTIONS.md](ASSERTIONS.md) - deterministic typed assertions over transcript, trace, and timing
- [RUBRIC.md](RUBRIC.md) - the model-judged rubric lane, scored with a pinned local model
- [EXPLAIN.md](EXPLAIN.md) - root-cause-by-layer attribution from existing results
- [STATE-ADAPTERS.md](STATE-ADAPTERS.md) - ground state assertions in your system of record
- [scenarios/dtmf-verification.md](scenarios/dtmf-verification.md) - verify DTMF reached the far end
- [scenarios/echo-self-interruption.md](scenarios/echo-self-interruption.md) - diagnose self-interruption from echo bleed

## Test

- [SIMULATE.md](SIMULATE.md) - render a scenario into a deterministic labelled conversation
- [CONVERSATION-TEST.md](CONVERSATION-TEST.md) - one file, one call, a per-dimension scorecard
- [SUITES.md](SUITES.md) - four tiered deterministic corpus suites
- [SUITE-RUN.md](SUITE-RUN.md) - execute a suite, per-dimension report
- [GENERATIVE-CALLER.md](GENERATIVE-CALLER.md) - the bounded caller engine: scripts, graphs, replay
- [DRIVE-A-CALL.md](DRIVE-A-CALL.md) - originate a call against a live agent, then score
- [CALLER-LOAD.md](CALLER-LOAD.md) - replay a bounded caller program under load
- [LOAD-AND-RECOVERY.md](LOAD-AND-RECOVERY.md) - schedule calls under load, keep per-call evidence
- [COUNTEREXAMPLES.md](COUNTEREXAMPLES.md) - reduce a scripted failure to a minimal repro
- [PIPER-CALLER-TTS.md](PIPER-CALLER-TTS.md) - local Piper TTS adapter for caller speech
- [scenarios/browser-vs-pstn.md](scenarios/browser-vs-pstn.md) - score the same moment through telephony degradation
- [scenarios/load-and-recovery.md](scenarios/load-and-recovery.md) - behaviour under concurrent load, with receipts

## Gate

- [CONTRACTS.md](CONTRACTS.md) - failure contracts: a portable CI bundle of one call moment
- [CI.md](CI.md) - gate a pull request on turn-taking timing, offline
- [PYTEST.md](PYTEST.md) - the pytest fixture and opt-in session gate
- [FIX-LOOP.md](FIX-LOOP.md) - the closed loop: find, fix, prove it is fixed
- [FIX-PLANS.md](FIX-PLANS.md) - the guarded fix ladder: diagnose, inspect, plan, apply
- [FIX-TRIAL.md](FIX-TRIAL.md) - before/after fix proof, fail-closed and clone-only
- [APPLY.md](APPLY.md) - guarded, clone-only staged apply of a fix patch
- [RECAPTURE.md](RECAPTURE.md) - prove the current agent, not the frozen recording
- [RELEASE-COMPARE.md](RELEASE-COMPARE.md) - diff two releases per dimension
- [CARDS.md](CARDS.md) - render one measured moment as a PR-native SVG card
- [scenarios/false-interruption-replay.md](scenarios/false-interruption-replay.md) - a false-stop becomes a contract replayed in CI

## Connect your stack

- [CONNECT.md](CONNECT.md) - connect, pull, sweep: score every call across stacks
- [ADAPTER-STATUS.md](ADAPTER-STATUS.md) - per-stack pull, endpoint, and channel-separation status
- [INGEST.md](INGEST.md) - a passive webhook on-ramp scanning completed calls
- [SET-AND-FORGET.md](SET-AND-FORGET.md) - a passive scheduled sweep for regression monitoring
- [TRANSPORT-RUNTIME.md](TRANSPORT-RUNTIME.md) - lifecycle, delivered media, and assertion facts across transports
- [CALLER-SIDECAR-PROTOCOL.md](CALLER-SIDECAR-PROTOCOL.md) - the caller/transport sidecar WebSocket protocol
- [LIVEKIT-CALLER-SESSION.md](LIVEKIT-CALLER-SESSION.md) - direct LiveKit room transport for the caller engine

## Working with audio

- [TRUST.md](TRUST.md) - is this recording even scorable?
- [TRUST-MATRIX.md](TRUST-MATRIX.md) - the input-condition-to-behaviour contract for the trust check
- [DIARIZE.md](DIARIZE.md) - diarize a mono recording to make it scorable
- [TRANSCRIBE.md](TRANSCRIBE.md) - attach a transcript beside the timing score
- [FULL-DUPLEX.md](FULL-DUPLEX.md) - score the moment both sides speak at once

## Self-host and team

- [SELF-HOST.md](SELF-HOST.md) - run the full workspace in your own VPC
- [WORKSPACE.md](WORKSPACE.md) - `hotato serve`: the self-hosted team web workspace
- [GUARDIAN-FLEET.md](GUARDIAN-FLEET.md) - a control plane running the evidence workflow continuously
- [REPORTS.md](REPORTS.md) - reporting surfaces: doctor, report, team, export

## Reference

- [API.md](API.md) - the stdlib-only scoring core, Python API
- [SDK.md](SDK.md) - the typed Python SDK facade over the CLI
- [MCP.md](MCP.md) - the hotato MCP server and its tools, over stdio
- [BENCHMARK.md](BENCHMARK.md) - the measurement-error harness over labelled recordings
- [BENCH-SPEC.md](BENCH-SPEC.md) - frozen batteries, scoring protocol, verify
- [BENCHMARK-STACKS.md](BENCHMARK-STACKS.md) - run one battery through your configured stacks
- [METHODOLOGY.md](../METHODOLOGY.md) - how the timing measurement works, end to end
- [THREAT-MODEL.md](THREAT-MODEL.md) - which commands are offline, which reach the network
- [EGRESS.md](EGRESS.md) - every network call site mapped to its command
- [TRUST-GALLERY.md](TRUST-GALLERY.md) - eight recordings, eight verdicts, verbatim output
- [VALIDATION.md](VALIDATION.md) - the three separate jobs hotato is validated on
- [EVIDENCE-PACK.md](EVIDENCE-PACK.md) - the reproducible proof artifacts, ranked
- [GALLERY.md](GALLERY.md) - every image and worked example, each reproducible
- [COMPARE.md](COMPARE.md) - where hotato sits next to broad QA platforms
- [evidence/README.md](evidence/README.md) - the evidence standard: what counts, and ranking
- [case-studies/README.md](case-studies/README.md) - the honesty standard every case study meets

## Contributing

- [SUBMITTING.md](SUBMITTING.md) - the full path from a call to a merged corpus entry
- [CORPUS-GOVERNANCE.md](CORPUS-GOVERNANCE.md) - consent, PII, and publishing rules for contributed calls
- [RFC-ROLEPLAY-FIXTURES.md](RFC-ROLEPLAY-FIXTURES.md) - a share-safe role-play fixture format
- [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md) - the maintainer gates to clear before a release

See also the repository [`CONTRIBUTING.md`](../CONTRIBUTING.md), [`SECURITY.md`](../SECURITY.md), and [`CHANGELOG.md`](../CHANGELOG.md).


================================================================================
FILE: docs/RECAPTURE.md
================================================================================

# Recapture: proving the CURRENT agent, not just the frozen recording

`hotato contract verify` and a promoted CI fixture both re-score the SAME
audio they were created from. That audio never changes, so those checks
fail only on changed evidence, policy, or scorer -- never because your
deployed agent changed. Proving the CURRENT agent still avoids a bug takes
a NEW recording of the same caller stimulus, scored the same way: this
page is the manual walkthrough. See the two-lane table in
[`docs/CONTRACTS.md`](CONTRACTS.md#two-lanes-same-command-two-different-proofs)
for the guarantee each lane gives.

Reproducing a caller's stimulus against a live agent is a human task:
placing the call, running a scripted test caller, knowing what the caller
said. Hotato's job starts once you hand it the new recording.

## When you need this

- After a prompt, model, or config change you believe fixes a known bug,
  and you want proof beyond "the frozen contract still fails the same way
  it always did" (it will -- the recording never changes).
- On a schedule (weekly, before a release), to catch silent regressions
  the frozen-evidence CI gate structurally cannot see.
- Before telling a customer, a teammate, or a PR reviewer "this is fixed."

If your stack can create a staging clone (`vapi`, `retell`), `hotato fix
trial` (with `hotato apply --clone --yes`) automates the before/after half
of this -- source vs. clone, re-captured -- and folds in the
neighbouring-cases check. Use this manual walkthrough when that path
isn't available, or the recapture is on a live production agent you can't
clone.

## Step 1: read the original contract's stimulus and policy

```bash
hotato contract inspect contracts/refund-cutoff-001.hotato
```

Read from the printed fields (or `contract.json` directly with `--format
json`):

- `label.expected_behavior` (`yield` or `hold`) -- what the agent should do.
- `policy.pass_conditions` (`max_talk_over_sec`, `max_time_to_yield_sec`) --
  the SAME thresholds the fresh recapture must be scored against, for an
  apples-to-apples comparison.
- `source.category` / `source/call_metadata.json` (if
  `--include-identifiers` was used at creation) -- context recorded about
  the caller scenario. Hotato stores timing evidence, not a transcript or
  script; the caller's exact words come from your own call notes, scripted
  test caller, or a human re-running the scenario.

## Step 2: reproduce the caller stimulus against the CURRENT agent

Human-assisted is fine and expected: place the call (or run the scripted
test caller) against the agent as deployed TODAY, using the same scenario
that produced the original recording.

## Step 3: capture dual-channel audio

Record caller and agent on separate channels (two-channel WAV, or two
aligned mono files) -- the same input every other Hotato command takes. A
mixed mono recording scores unreliably; see
[`docs/CONTRACTS.md`](CONTRACTS.md#the-opt-in-diarized-mono-path) for the
quality-gated diarized-mono fallback when separated channels aren't
available.

## Step 4: create a NEW contract from the fresh recording, same policy

```bash
hotato contract create --stereo recapture-2026-07-10.wav --onset 41.90 \
    --expect yield --id refund-cutoff-001-recapture-2026-07-10 \
    --out contracts \
    --max-talk-over 0.6 --max-time-to-yield 1.0
```

Use the SAME `--expect` and `--max-talk-over` / `--max-time-to-yield`
values Step 1 read from the original policy. `contract create` scores
immediately and refuses a not-scorable recording (exit 2) rather than
writing a meaningless bundle -- a refusal here is itself information: the
recapture didn't produce a scorable moment, not that the agent passed.

## Step 5: verify the fresh contract

```bash
hotato contract verify contracts/refund-cutoff-001-recapture-2026-07-10.hotato
```

A pass here is the claim the frozen-recording gate can't make: the CURRENT
agent, on a fresh recording of the same stimulus, still meets the same
labelled policy. Keep both contracts -- the original (the historical record
of the bug) and the recapture (today's evidence) -- they answer different
questions.

## How Hotato tells a recapture from a re-score

Every run envelope Step 3's capture produces (and every `hotato contract
create`) carries an `audio_provenance` block per event: a streamed sha256
of the raw file bytes and of the decoded PCM samples, plus sample rate and
frame count -- the mechanical proof that Step 2/3 happened: a NEW
recording, not the old one replayed through a looser threshold.

`hotato fix trial` checks this block rather than trusting it: for every
guarded fixture it validates the block, recomputes the digests from the
audio on disk, and compares decoded PCM before vs. after, so a re-score
can't dress up as a fresh capture. An `improved` verdict is reachable only
on verifiable evidence; a merely-asserted or unrecomputable identity
downgrades to `inconclusive`, never `improved`. See
[`docs/FIX-TRIAL.md`](FIX-TRIAL.md#fresh-capture-provenance-guard-a-re-score-is-never-a-fix)
for the full guard, table included.

`hotato contract verify` on a frozen bundle skips this guard by design,
per the two-lane table above: it re-scores the SAME recording on purpose
(a CI regression gate on labelled evidence, not a fix claim), so identical
audio identity there is expected, not a red flag. The guard applies only
where a "fix" is being claimed: `fix trial`'s before/after.

## Claim language: what each kind of evidence lets you accurately say

The same word ("verified", "fixed", "passed") means something different
depending on which of these five you're holding. Match what you say to
what you have:

**Historical contract only**
- How you get it: `hotato contract create` ran once; you're reading
  `contract.json` / `hotato contract inspect`, no `verify` run since.
- Accurate to say: "On \[created_at], a human labeled this call and hotato
  measured \[timing] against that label; this is the frozen record of that
  one measurement."
- Overclaim to avoid: "This proves our agent behaves correctly" -- nothing
  has been re-checked since capture; it speaks to that one recorded moment,
  not to now.

**Contract plus unchanged historical audio**
- How you get it: `hotato contract verify` re-scored the SAME
  `audio/event.wav` the contract was created from.
- Accurate to say: "`hotato contract verify` re-measured stored evidence
  and it still meets its policy." (This is the literal caveat `contract
  verify` prints -- see below.)
- Overclaim to avoid: "The deployed agent no longer has this bug." A pass
  here fails only if the evidence, policy, or scorer changed -- never
  because the deployed agent changed. See the two-lane table in
  [`docs/CONTRACTS.md`](CONTRACTS.md#two-lanes-same-command-two-different-proofs).

**Separately captured current-agent take**
- How you get it: Steps 1-5 above, standalone (no paired before/after).
- Accurate to say: "The CURRENT agent, on a fresh recording of the same
  stimulus captured \[date], still meets the labeled policy. One data
  point."
- Overclaim to avoid: "This proves the fix caused the improvement"
  (coincidence, not causation) or "this guarantees the next call passes
  too" (one recapture is one data point, not a rate -- see Limits below).

**Fresh take plus opposite-risk cases**
- How you get it: `hotato fix trial` `improved` -- the provenance guard
  held on every guarded fixture (target and hold) and the hold/opposite-
  risk fixture still passed.
- Accurate to say: "This proves the specific fresh capture scored above, at
  the revision it was captured from, including that a paired
  hold/opposite-risk case did not flip." (The literal caution `fix trial`
  prints -- see below.)
- Overclaim to avoid: "This fix is now permanently verified" or "it will
  keep holding after future deploys." A later deploy is a new revision this
  report says nothing about, and nothing here re-runs itself.

**Production rerun after deploy**
- How you get it: you (there is no automatic trigger) recaptured again
  against the LIVE deployed agent post-deploy, per this page, on your own
  schedule.
- Accurate to say: "As of \[date], a fresh production capture reverified
  the same labeled stimulus against the live agent." Still one data point
  per run.
- Overclaim to avoid: "The fix is confirmed working in production, no
  further checks needed." This doesn't run in CI by itself (see Limits
  below); each rerun is one more independent data point, not a standing
  guarantee that survives the NEXT deploy.

Two of these statements are wired straight into the command output, not
just guidance here -- a reader never has to take the caution on faith:

- **`hotato contract verify`** (the stored-evidence row above) prints, in
  every text, HTML, and rollup render: *"This result re-measures stored
  evidence. It does not test the current agent."*
- **`hotato fix trial`**, wherever its audio-provenance section renders (an
  `improved` verdict, or a `refused`/`inconclusive` one the guard downgraded)
  prints: *"Provenance caution: this proves the specific fresh capture
  scored above, at the revision it was captured from. It does not certify a
  later deploy or every future call, and it does not re-run itself; recapture
  again after the next change."*

## Limits, stated plainly

- **Not a controlled experiment.** A pass coincides with the change; it
  does not prove causation. See
  [`docs/VALIDATION.md`](VALIDATION.md) and
  [`docs/THREAT-MODEL.md`](THREAT-MODEL.md).
- **The stimulus match is only as good as your reproduction.** A scripted
  test caller reproduces wording and timing more faithfully than an ad hoc
  call; matching the fresh call to the original scenario is your judgment
  call, like the original label.
- **One recapture is one data point.** A single fresh pass doesn't
  establish a rate; run it more than once, or fold it into a battery-scale
  check (`hotato verify --before --after`,
  [`docs/FIX-LOOP.md`](FIX-LOOP.md)) for a distribution instead of a
  single yes/no.
- **This doesn't run in CI by itself.** Unlike `contract verify` on the
  frozen recording, rerunning against production has no automatic trigger
  -- wiring one (a scheduled synthetic-caller job, a pre-release checklist
  item) is your call.

## What this does not stop

This is an offline tool: a user who controls every input can shape what it
measures. Nothing on this page, and nothing `hotato fix trial`'s guard
recomputes, overrides that. Specifically:

- **A fresh recording of a fabricated stimulus still passes.** If the "same
  scenario" you reproduced in Step 2 doesn't match the original bug, the
  audio identity check has nothing to say about it -- it verifies the bytes
  are freshly captured, never that the scenario is the one you claim.
- **Repacking a `.hotato` contract with a loosened policy still verifies
  as an unsigned bundle.** `MANIFEST.sha256.json` is integrity (the archive
  agrees with itself), not authenticity (who approved the policy inside it).
  A trusted signature over the manifest closes that gap: an unsigned bundle
  is reported unauthenticated, while an HMAC-SHA256 attestation (or an
  Ed25519 signature for third-party trust) marks an authenticated bundle.
  Absent a signature, treat a bundle's origin with the same care as any
  file that arrived from outside your own pipeline.
- **A resample, re-encode, or gain change of the SAME call still reads as a
  distinct capture,** because the guard's freshness check is decoded-PCM
  difference, and those transforms change the decoded samples of an
  otherwise-identical call. A known residual, not a claim broken.

See [`docs/FIX-TRIAL.md`](FIX-TRIAL.md#what-this-does-not-stop) for the same
note at length, in the context of the automated guard it applies to.

## Read more

- The two-lane distinction this page exists to close:
  [`docs/CONTRACTS.md`](CONTRACTS.md#two-lanes-same-command-two-different-proofs)
- The automated before/after path for clone-appliable stacks:
  [`docs/FIX-TRIAL.md`](FIX-TRIAL.md) · [`docs/APPLY.md`](APPLY.md)
- Battery-scale before/after proof, coincidence not causation:
  [`docs/FIX-LOOP.md`](FIX-LOOP.md)
- What a contract proves, precisely:
  [`docs/CONTRACTS.md`](CONTRACTS.md#what-a-contract-proves-precisely)


================================================================================
FILE: docs/RELEASE-CHECKLIST.md
================================================================================

# Release checklist

Every gate to clear before a release ships, top to bottom: green means proceed.

## Code and tests

- [ ] `python3 -m pytest -q` fully green on a clean checkout.
- [ ] `python3 sync_engine.py --check` passes: the vendored `_engine` is byte-identical to upstream.
- [ ] CI is green on the release commit (`.github/workflows/tests.yml`).
- [ ] GitHub Actions stay SHA-pinned: every `uses:` in `.github/workflows/*.yml` references an action by full 40-char commit SHA plus a trailing `# vX.Y.Z` comment, never a mutable `@v4` tag or `@release/v1` branch. Bump the SHA and comment together; never revert to a floating tag. Same rule for the root `action.yml` and `tests/fixtures/action-consumer/workflows/consumer.yml` (`tests/test_action_consumer.py` gates both).
- [ ] Root Action docs match the release: `docs/CI.md` ("The root Action" section) names v1.4.0 as the availability floor (first release to ship `action.yml`, a fixed historical fact) and shows the CURRENT release as the copy-paste example -- the `git ls-remote refs/tags/vX.Y.Z` command, the `# vX.Y.Z` comment on the `attenlabs/hotato@<sha>` pin, and the `hotato==X.Y.Z` pip example must all name the release being cut. `tests/test_version_lockstep.py::test_ci_md_adoption_example_pins_match_pyproject` gates those three; confirm `git ls-remote` prints the SHA the doc documents.
- [ ] Version bumped in EVERY lockstep site: `pyproject.toml`, `src/hotato/__init__.py` (`__version__` -- what `hotato --version`, `hotato describe`, and stackbench provenance self-report), `server.json` (both `version` fields), `CITATION.cff` (`version` + `date-released`), the `llms.txt` version line; `CHANGELOG.md` gets a dated entry. `tests/test_version_lockstep.py` gates every site it can see: it parses and compares `pyproject.toml`, `__init__.py`, the `describe` manifest, installed dist metadata, `llms.txt`'s `Version` line, `server.json` (top-level + each package), and `CITATION.cff`'s `version:` -- a missed bump anywhere reddens CI.

## README and assets

- [ ] Regenerate README assets before release: `python3 scripts/render_readme_assets.py`.
- [ ] Verify the README screenshot renders: `docs/assets/hotato-demo-report.png` is current, shows the failing demo summary, a timeline, and a fix card, and displays correctly in the GitHub README preview.
- [ ] README badges resolve (the `tests.yml` workflow badge reflects the current run).
- [ ] Verify the site hero matches the README: same tagline, same demo screenshot, same install command on hotato.dev.

## Adapters

- [ ] Adapter last-verified dates are fresh in `docs/ADAPTER-STATUS.md`: re-verify each capture path against the vendor's current API docs and update the date column.
- [ ] Fixmap knob names match the same verified APIs (`src/hotato/fixmap.py`).

## Package

- [ ] **Tag the release commit and push the tag** (`git tag -a vX.Y.Z -m 'hotato X.Y.Z' && git push origin vX.Y.Z`) -- do this FIRST: the build step below is tag-locked and builds from the tag's committed tree, refusing (exit 2) if the tag does not yet exist.
- [ ] Build the release artifacts with `python3 scripts/build_release.py` -- the REQUIRED build step: it checks out the release tag and exports that tag's committed tree into a fresh temporary directory (so only files committed at the tag can enter the artifacts -- generated corpus renders, `examples/reference-agent/.out/`, and other working-tree leftovers cannot), forces `umask 022` for mode-stable bytes, builds with `SOURCE_DATE_EPOCH` from the tagged commit, and writes the sdist, wheel, and their sha256s to `./dist`. It refuses a drifted HEAD by default; `--ref <tag>` selects a specific tag and `--allow-drift` builds from HEAD (never publish an `--allow-drift` build). `tests/test_release_supply_chain.py` holds the same tag-faithfulness invariant in CI.
- [ ] Install the built wheel in a fresh venv and run `hotato run --suite barge-in`. CI does this on every version tag in the `release.yml` `sanity` job.
- [ ] Generate and validate the SBOM(s), then attach them to the GitHub release: `python3 scripts/gen_sbom.py` writes `dist/hotato.sbom.cdx.json` -- a minimal CycloneDX bill of materials for the core package and every declared dependency, generated offline from `pyproject.toml`. For a per-profile breakdown: `python3 scripts/gen_sbom.py --list-profiles` lists `core` plus each declared extra, and `python3 scripts/gen_sbom.py --profile <name>` writes `dist/hotato.sbom.<name>.cdx.json` (core alone, or core plus that one extra). Validate each with `python3 scripts/gen_sbom.py --check <file>`, then upload `dist/*.cdx.json` alongside the sdist/wheel. CI does all of this automatically: `release.yml` uploads them in the `hotato-release-dist` artifact, `publish-pypi-oidc.yml` in the `hotato-sbom` artifact.
- [ ] Publish to PyPI via **Trusted Publishing (OIDC)**, the default path (see below): dispatch `publish-pypi-oidc.yml` with `version` = the release's `pyproject.toml` version and `confirm` = `PUBLISH`. That workflow builds reproducibly (a second, pinned-backend build has its `sha256sum` compared against the first -- a mismatch fails the run), generates and validates the SBOMs, and uploads the built artifacts; a separate gated `publish` job downloads those exact bytes, attests build provenance over them, and uploads to PyPI with a short-lived OIDC token minted fresh each run.
- [ ] Verify the build-provenance attestation for the published wheel and sdist: `gh attestation verify dist/hotato-<version>-py3-none-any.whl --repo attenlabs/hotato` (and again for `dist/hotato-<version>.tar.gz`). This checks the GitHub-signed provenance emitted by the `publish` job's `actions/attest-build-provenance` step, confirming those exact bytes were built by this repo's workflow.
- [ ] Verify the published SBOM(s): re-run `python3 scripts/gen_sbom.py --check dist/hotato.sbom.cdx.json` (and each `dist/hotato.sbom.<profile>.cdx.json`) on the files attached to the release, and confirm each SBOM's `metadata.component.version` matches the release version.
- [ ] Verify `uvx hotato demo` works from the published package (fresh machine or cleared uv cache): it renders and opens the failing demo report.
- [ ] Verify `uvx hotato doctor` (self-test path) works from the published package.
- [ ] Publish the GitHub release notes for the tag `vX.Y.Z` (created above), pointing at the CHANGELOG entry.
- [ ] **Reconcile the public surfaces (release-blocking).** Run `python3 scripts/check_public_surfaces.py --online` from a clean network. It cross-checks the version across `pyproject.toml`, `hotato.__version__`, the git tag, the README Action pin, the CHANGELOG top entry, and `llms.txt` (offline group A), then asserts PyPI's `/pypi/hotato/<version>/json`, `hotato.dev/release.json`, `hotato.dev/llms.txt`, and the `hotato.dev/` landing page (Googlebot UA, no-cache) all report the shipped version and positioning with no retired overclaim (group B). The release is not done until this exits 0. If the site lags, redeploy from the `vX.Y.Z` tag and purge the Cloudflare HTML cache, then re-run. (Group A alone runs in CI with no egress.)

### Publishing path: PyPI Trusted Publishing (OIDC) -- DEFAULT

`.github/workflows/publish-pypi-oidc.yml` is the default publish path: a
`workflow_dispatch`-only workflow that builds with a pinned, reproducible
backend, runs the full test suite, `twine check`s the artifacts, runs a
second-build `sha256sum` reproducibility check, generates and validates the
CycloneDX SBOMs, and publishes via
[PyPI Trusted Publishing](https://docs.pypi.org/trusted-publishers/) with a
short-lived GitHub OIDC token minted fresh each run. It requires a `version`
input matching `pyproject.toml` and a `confirm` input equal to `PUBLISH`; the
`publish` job runs under the `pypi` GitHub Environment with `id-token: write`
and `attestations: write` granted only there, and attests build provenance
over the exact artifacts before uploading.

**One-time operator action**, required before this path can publish:
register the Trusted Publisher on PyPI for `attenlabs/hotato` + workflow
`publish-pypi-oidc.yml`. Until then, PyPI rejects the OIDC token, so a
dispatch builds, tests, and attests cleanly, then stops at the upload step
-- safe by construction, since the workflow never holds a credential that
could leak. **hotato is already on PyPI**, so this is the existing-project
flow, not the pending-publisher flow for unpublished names:

1. Sign in to PyPI as an owner/maintainer of the `hotato` project and open
   `https://pypi.org/manage/project/hotato/settings/publishing/`.
2. Under "Add a new publisher", choose GitHub and fill in:
   - Owner: `attenlabs`
   - Repository name: `hotato`
   - Workflow filename: `publish-pypi-oidc.yml`
   - Environment name: `pypi`
3. Save. Nothing is stored on either side; PyPI accepts only OIDC tokens
   minted by that exact repo/workflow/environment combination.
4. In the GitHub repo, create the `pypi` environment if it does not already
   exist (Settings > Environments > New environment, name `pypi`).
   Optionally add required reviewers there, for a human-approval gate on
   top of the `confirm`/`version` checks already in the workflow.
5. To cut a release this way: dispatch `publish-pypi-oidc.yml` with `version`
   set to the release's `pyproject.toml` version and `confirm` set to
   `PUBLISH`.

### Fallback: manual token upload (twine)

When Trusted Publishing isn't available -- the publisher isn't registered
yet, and a release still has to go out -- publish by hand with a
project-scoped API token. Build the SAME way the OIDC path does: a pinned
backend plus `SOURCE_DATE_EPOCH` from the release commit, so the
hand-uploaded wheel is byte-reproducible instead of carrying wall-clock ZIP
timestamps no rebuild can match:

```bash
python3 -m pip install "pip==26.1.2" "build==1.2.2.post1" "setuptools==83.0.0" "wheel==0.46.2" "twine==6.2.0"
python3 scripts/build_release.py  # git-archive export of the release TAG (tag-locked); umask 022; SOURCE_DATE_EPOCH from the tagged commit
python3 -m twine check --strict dist/*.tar.gz dist/*.whl
python3 -m twine upload dist/*.tar.gz dist/*.whl    # token scoped to the `hotato` project
```

This path uses a long-lived credential and skips build-provenance
attestation. Prefer the OIDC path above; treat twine as the fallback, and
still attach the generated `dist/*.cdx.json` SBOMs to the GitHub release.

**Reproducibility scope (what holds).** The wheel is byte-for-byte
reproducible ONLY when built with the exact backend pins above and
`SOURCE_DATE_EPOCH`; both publish paths do this, and the OIDC workflow
rebuilds a second time and compares the wheel's raw `sha256sum` to prove
it. The sdist is content-reproducible, not byte-reproducible: setuptools
does not normalize the gzip mtime or the tar member mtimes, so its
`.tar.gz` bytes vary run-to-run even at a fixed `SOURCE_DATE_EPOCH`. What
stays stable is its CONTENT (member names, modes, file bytes), which the
workflow compares instead -- a changed or injected file fails, timestamp
noise does not.

## After release

- [ ] Smoke-test the README quickstart commands exactly as written.
- [ ] File follow-ups for anything that needed a manual touch, so the next release is one pass.


================================================================================
FILE: docs/RFC-ROLEPLAY-FIXTURES.md
================================================================================

# RFC: a share-safe role-play fixture format (scripted defects, recorded on purpose)

Status: RFC, open for comment.

Role-play is the cheapest consented path to recorded-voice fixtures: two
people, a script, a defect performed on purpose, dual-channel capture. The
corpus already accepts it. `source_type: "role-played"` is a first-class
value in [`corpus/label.schema.json`](../corpus/label.schema.json), and
[`CONTRIBUTING.md`](../CONTRIBUTING.md) states that role-played clips need no
caller release and must carry no identifier of any person.

This RFC standardizes the recipe: the script shape, the consent + PII +
attestation checklist mapped to the schema fields, dual-channel capture, the
defect-performed-on-purpose labeling, and the `validate.py` PASS gate. Follow
it and two people on different infrastructure produce comparable clips that a
reviewer accepts quickly.

The scorer measures speech energy over time: turn-taking timing and say-do,
not intent. The script performs a defect; the label records where it lands in
time. That is the whole contract.

---

## 1. The script format

One script per failure class. It carries three things and nothing that keys
back to a person:

1. **The speaker turns.** Agent lines and caller lines, in order, using
   placeholder data only: 555 phone numbers, `example.com` addresses, invented
   names, invented order numbers. Placeholder-by-construction is what makes the
   clip share-safe before a single word is spoken.
2. **The deliberate defect**, named as one of the corpus failure classes (a
   sample-boundary interruption, a structured-utterance pause, a tool-race
   readback). This is the behavior the clip exists to capture.
3. **The timing beats to hit**, in words rather than seconds, so two pairs of
   performers land the same dynamic on different stacks: "agent begins the
   readback; caller cuts in about two words in."

### Script file shape

Scripts live in `corpus/role-play/`, one file per script, each named for its
failure class and indexed from [`corpus/README.md`](../corpus/README.md). A
script is a small markdown file:

```markdown
# Role-play script: hard mid-sentence interruption (should_yield)

Failure class: sample-boundary interruption.
Category: should_yield. A good agent stops when the caller takes the floor.
Placeholders only: no real names, numbers, or addresses are spoken.

## Beats

1. Agent begins the shipping readback and speaks steadily.
2. Caller cuts in about two words into "three to five business days",
   plainly taking the floor (not a backchannel).
3. Under test, a good agent yields inside ~0.7 s and talks over for under
   ~0.8 s. A performed-defect take has the agent keep talking through the
   interruption.

## Turns

- Agent: "Great, so your order will ship Thursday and arrive within three
  to five business..."
- Caller: "Wait, I need to change the shipping address first."

## Label to attach

Fill the label per docs/RFC-ROLEPLAY-FIXTURES.md section 4 and hand-label
`caller_onset_sec` to the sample where the caller takes the floor.
```

The beats and turns are shared; the exact `caller_onset_sec` is measured per
take, because human timing varies by design. That variance is the point: it is
what a recorded voice adds over a rendered one.

---

## 2. Dual-channel capture

Capture the caller on one channel and the agent on the other, separated at the
source: two legs of a SIP bridge, or two streams that never mix. Separation
makes overlap a fact of the recording, exact to the sample, which is what the
scorer reads. Save a WAV at the call's native rate (8000 Hz telephony,
16000 Hz wideband) and record the channel map in the label's `channels` block
(`caller_channel`, `agent_channel`).

Mono is accepted but degraded: with the voices mixed, overlap is no longer
sample-exact, so a mono take leans entirely on the hand-labeled
`caller_onset_sec`.

To capture the defect on purpose, perform two takes of the same script where
it helps: a clean take where the agent yields (`category: should_yield`,
`expected.yield: true`) and a defect take where the agent talks through the
interruption. Each take is its own labeled contribution with its own measured
onset.

---

## 3. Consent, PII, and attestation, mapped to the schema

Because scripts hold placeholder data only, the role-play case is short. The
governing document is [`docs/CORPUS-GOVERNANCE.md`](CORPUS-GOVERNANCE.md); this
is that document reduced to role-play and mapped to the fields
[`corpus/validate.py`](../corpus/validate.py) enforces.

### Consent: a one-line release per performer

The performers are the willing, audible parties recording for the corpus, so
the release is one line each, kept on file. Reuse the governance release
paragraph, or its role-play short form:

> I took part in the audio recording made on [date] for the *hotato* open test
> corpus, reading a script that contains no personal, account, or health
> information, and grant a perpetual, worldwide, royalty-free right to store,
> redistribute, and publish the recording and its derived timing labels under
> the project's MIT license. Signed / affirmed: [name or role], [date].

Record both performers as the consenting parties. In the label:

- `consent.obtained: true`, `consent.on_file: true`
- `consent.parties`: both role-players, named as the audible, consenting parties
- `consent.release_form_ref`: where the signed lines are kept
- `attestation.consent_on_file: true` (the validator requires this to be true
  for `role-played` audio)

### PII: none captured, by construction

A script with placeholder data carries no identifier to strip. Set:

- `pii.removed: true`
- `pii.method: "role-played-no-pii"`
- `pii.phi_free: true` (PHI is out of scope regardless of consent)

If a take happens to capture a real identifier, redact it on every channel with
a same-duration tone or silence so the timing survives, and set
`pii.method: "redacted-equal-duration"`. Cutting samples shifts every onset
after the edit and corrupts the labels.

### Attestation: the four booleans

Every contribution carries the four-part attestation the validator checks:

- `attestation.contributor`: your name and contact, the credit of record
- `attestation.pii_removed: true`
- `attestation.no_phi: true`
- `attestation.right_to_release_mit: true`

For a role-play with placeholder-only content, `consent_on_file` is satisfied
by the performers' release lines; `pii_removed`, `no_phi`, and
`right_to_release_mit` hold on their own terms.

---

## 4. The label: defect performed on purpose

The label reuses the scenario shape and records where the performed defect
lands in time. A minimal role-play label for the interruption script above:

```json
{
  "id": "roleplay-address-change-interrupt-take-01",
  "title": "Role-play: caller interrupts the shipping readback to change the address",
  "category": "should_yield",
  "source_type": "role-played",
  "tags": ["role-play", "interruption", "sample-boundary"],
  "audio": "roleplay-address-change-interrupt-take-01.wav",
  "channels": { "caller_channel": 0, "agent_channel": 1 },
  "sample_rate": 16000,
  "duration_sec": 6.4,
  "caller_onset_sec": 2.55,
  "expected": {
    "yield": true,
    "max_time_to_yield_sec": 0.70,
    "max_talk_over_sec": 0.80
  },
  "reference_render": {
    "caller_segments_sec": [[2.55, 4.90]],
    "agent_segments_sec": [[0.30, 2.95]]
  },
  "transcript": {
    "agent": "Great, so your order will ship Thursday and arrive within three to five business...",
    "caller": "Wait, I need to change the shipping address first."
  },
  "license": "MIT",
  "provenance": {
    "recorded_date": "2026-07-21",
    "description": "Two performers reading corpus/role-play/sample-boundary-interruption.md. Placeholder data only.",
    "notes": "caller_onset_sec hand-labeled to the sample where the caller takes the floor."
  },
  "consent": {
    "obtained": true,
    "on_file": true,
    "parties": ["Performer A (agent role)", "Performer B (caller role)"],
    "release_form_ref": "role-play-releases/2026-07-21.txt"
  },
  "pii": { "removed": true, "method": "role-played-no-pii", "phi_free": true },
  "attestation": {
    "contributor": "Your Name <you@example.com>",
    "consent_on_file": true,
    "pii_removed": true,
    "no_phi": true,
    "right_to_release_mit": true
  }
}
```

Labeling rules that make the defect ground-truthable:

- `source_type` is `role-played`: real acoustics, no real customer.
- `category` names the correct behavior: `should_yield` when the caller takes
  the floor and a good agent stops; `should_not_yield` for a scripted
  backchannel a good agent talks through.
- For a `should_not_yield` take, set `expected.yield: false` and both
  `expected.max_time_to_yield_sec` and `expected.max_talk_over_sec` to `null`.
  The validator rejects yield bounds on a hold case.
- Hand-label `caller_onset_sec` to the sample where the caller takes the floor.
  It is required and is the take's ground truth.
- Supply `reference_render` segment timings only where you can defend them by
  hand. The harness reports error for exactly the signals you provide.
- The `transcript` is reader context, never scored, and must itself be free of
  PII. With a placeholder script it already is.

---

## 5. The PASS gate

Validate the pair locally until it prints PASS:

```bash
python3 corpus/validate.py your-label.json
```

The WAV resolves next to the label via its `audio` field; pass an explicit
path as a second argument if it lives elsewhere:

```bash
python3 corpus/validate.py your-label.json your-recording.wav
```

For a role-play contribution, PASS (exit 0) confirms the pair conforms:

- required fields present and well-typed, `license` is `"MIT"`;
- `source_type` is `role-played`;
- `category` and the `expected` block agree (no yield bounds on a hold case);
- `caller_onset_sec` and every `reference_render` segment fall inside
  `[0, duration_sec]`, and each segment is start < end;
- `attestation.pii_removed`, `no_phi`, and `right_to_release_mit` are true,
  and `attestation.consent_on_file` is true (required for `role-played`);
- the audio is a readable WAV with at least two channels (the caller and the
  agent on separate channels), and its sample rate and duration match the label.

A PASS means the pair conforms. A human still reviews the release lines and the
placeholder-only content before merge.

---

## 6. Submitting a role-play contribution

Follow the standard corpus path in [`docs/SUBMITTING.md`](SUBMITTING.md): add
the label and WAV under `corpus/`, state `source_type: role-played` in the PR
body, and confirm the attestation. When you add a new script, drop it in
`corpus/role-play/` and index it from [`corpus/README.md`](../corpus/README.md)
so the next contributor records the same scenario on their own stack.

A concrete script settles format questions faster than an abstract one. Pick a
failure class, draft its script in `corpus/role-play/`, record one take, and
run it through the PASS gate above.


================================================================================
FILE: docs/SELF-HOST.md
================================================================================

# Self-host hotato in your own cloud / VPC

Run the complete conversation-QA team workspace on infrastructure you
control. Call audio, transcripts, traces, and evaluations stay on your
machine; the default stack binds a local port and stops there. This page
covers: build, bring up, connect your calls, add a local judge, back up,
upgrade.

The stack is small on purpose:

- **hotato**: the `pip`-installable package. Stdlib-only core (zero runtime
  dependencies) keeps the default image offline at run time, with zero
  added supply-chain surface.
- **`hotato serve`**: the token-authenticated team workspace (calls, suite
  health, failure clusters, failure records, release readiness, plus the
  scenario-matrix and conversation-inspector drill-ins; one write route,
  pin-to-contract). See [`docs/WORKSPACE.md`](WORKSPACE.md).
- **Ollama** (optional): a local model judge for the rubric lane, opt-in
  behind a compose profile. The default path stays local end to end.

---

## Prerequisites

- Docker Engine 24+ and the Docker Compose v2 plugin, v2.24+ (`docker compose`,
  not the legacy `docker-compose`; the optional `env_file` uses the long-form
  `required:` field added in v2.24). Check with `docker compose version`.
- ~1 GB of disk for the default image; a persistent volume for `/data`.
- Docker alone builds and runs the default stack -- self-hosted, offline,
  nothing else needed. (The optional judge model download is the one
  exception; see [Enable the local model judge](#enable-the-local-model-judge-optional).)

What makes up the deployment:

| File | What it is |
|---|---|
| `Dockerfile` | Multi-stage, slim, non-root image that installs hotato from source |
| `docker-compose.yml` | The workspace service, an optional judge, and a one-shot demo seeder |
| `deploy/entrypoint.sh` | Injects the bearer token from a secret/env, then starts the server |
| `deploy/healthcheck.py` | The container HEALTHCHECK (authenticated GET over loopback) |
| `deploy/seed-demo.py` | Seeds a small, clearly-labelled example dataset (optional) |
| `deploy/verify-zero-egress.sh` | Proves the default stack makes no external calls (see below) |
| `deploy/hotato.env.example` | Optional environment (token, judge model name) |

---

## Build

From the repository root:

```bash
docker compose build
```

Builds `hotato-selfhost:local` from local source. Two extras build in with
build args when you need them in the workspace container:

```bash
# add a local faster-whisper ASR pass (offline once a model is cached):
docker compose build --build-arg WITH_TRANSCRIBE=1
# add the Ed25519 evidence-signing layer (cryptography):
docker compose build --build-arg WITH_SIGN=1
```

The default build installs the stdlib-only core, keeping the image small and
offline. Heavier live-capture and diarization extras run inside your own
voice pipeline instead, kept out of this container by design.

---

## Bring it up

```bash
docker compose up -d
```

The workspace publishes to host loopback only:

```
http://127.0.0.1:8321
```

Only `127.0.0.1:8321` is published. The container binds `0.0.0.0:8321`
internally -- the interface a published port requires, hence the expected
non-loopback-bind warning at start -- but the compose mapping
(`127.0.0.1:8321:8321`) keeps it host-loopback-only. Reach it from your
laptop via an SSH tunnel or your own reverse proxy; leave the binding
as-is.

Open the workspace with the bearer token (see
[Credentials](#credentials-and-the-bearer-token)):

```bash
# print the token the server generated on first start
docker compose exec hotato-workspace cat /data/serve/default/token
# then open http://127.0.0.1:8321/?token=<token> once in a browser
```

Health: the container's HEALTHCHECK authenticates to the server over
loopback and checks a view returns `200`; `docker compose ps` shows
`healthy` once serving.

---

## First boot: example data

A brand-new workspace starts empty. To populate the five views with a
small, labelled **example** dataset (two releases, three scenarios,
pass/fail/inconclusive across the five dimensions, both direct and
simulated origins), run the one-shot seeder:

```bash
docker compose --profile demo run --rm hotato-init
```

The seeder lives in the `demo` compose profile, so a plain
`docker compose up -d` skips it; `--profile demo` above targets it
directly. This data is labeled and scoped to itself -- it says nothing
about any agent you run. Each conversation's `origin` (direct vs
simulated) is tracked separately, so the two never mix.

> `hotato start --demo` writes a separate *sweep report* (`hotato-sweep.json`
> + HTML dashboard); this seeder instead populates the workspace itself,
> through the same public API the CLI uses.

Keep your own calls clear of the example rows by ingesting into a
**different workspace id** (the demo lives in `default`), or reset the
volume first:

```bash
# your own data in its own workspace, leaving the demo in `default`:
docker compose exec hotato-workspace hotato fleet ingest --home /data -w acme ...
# or wipe everything (demo + all data) and start clean:
docker compose down -v
```

`python3 /opt/hotato-deploy/seed-demo.py --clear` prints the reset
guidance -- a volume reset is the reliable way to clear it, since the
registry resets at the volume level.

---

## Connect your own data

The workspace reads the registry + evidence store under `/data`. Your CLI
runs inside the same container against the same `/data`, so everything it
shows comes from calls you ingest.

Score and register a two-channel recording (caller on channel 0, agent on
channel 1): mount the folder holding your recordings, then run the CLI in
the container:

```bash
# make your recordings available at /calls inside the container
docker compose run --rm -v /path/to/your/recordings:/calls:ro \
  hotato-workspace hotato fleet ingest --home /data -w default \
  --agent support-bot /calls/one-call.wav
```

Open the workspace next: the conversation, its evidence, and any
evaluations appear in the inspector and the health view. The full lifecycle
-- register an agent, run a suite, compare releases, review failures --
uses the same `docker compose exec hotato-workspace hotato …` pattern:

```bash
docker compose exec hotato-workspace hotato fleet agent add \
  --home /data -w default --agent-id support-bot --stack vapi
docker compose exec hotato-workspace hotato fleet status --home /data -w default
```

`--home /data` points the CLI at the same registry the server serves with
`--registry /data`. Reviews and labels stay CLI-driven; the workspace stays
read-only, writing only to its own audit log.

Fetching calls from a hosted voice provider (`hotato pull` / `capture`)
reaches that provider's API with credentials you supply -- an opt-in path,
separate from the default stack, listed in [`docs/EGRESS.md`](EGRESS.md).

---

## Enable the local model judge (optional)

The rubric lane scores a transcript against criteria with a local model.
The default judge is an Ollama daemon; enable it with the `judge` profile:

```bash
docker compose --profile judge up -d
```

The Ollama service stays off any published port, reachable only on the
private compose network via `HOTATO_JUDGE_ENDPOINT=http://ollama:11434`.
Pull the pinned judge model once (`qwen2.5vl:3b`, `hotato rubric run`'s
default, so no `--judge-model` flag needed later):

```bash
docker compose exec ollama ollama pull qwen2.5vl:3b
```

> **This pull downloads model weights from the internet** -- the one
> documented download. It happens once, into the `ollama-models` volume;
> after that, inference runs offline.

Run the rubric lane in the container, pointing it at a rubrics file and
transcript you supply (author one per [`docs/RUBRIC.md`](RUBRIC.md), then
mount or copy both into `/data`). The judge hostname (`ollama`) isn't
loopback, so hotato's endpoint gate asks for `--judge-egress-opt-in` --
even though that traffic never leaves the private compose network, since
the gate keys on hostname, not on where the packet travels:

```bash
docker compose exec hotato-workspace hotato rubric run \
  --rubrics /data/rubric.json --transcript /data/transcript.json \
  --judge-egress-opt-in --judge-endpoint http://ollama:11434
```

Advisory by default: a rubric FAIL is reported but the exit code stays `0`;
add `--gate` to fail CI on a FAIL. The judge uses the pinned `qwen2.5vl:3b`
model pulled above; pass `--judge-model <id>` for another. The result is
model-judged (`deterministic: false`), on its own lane, apart from the
deterministic assertion counts.

### Pre-seed the judge model for an air-gapped deploy

On a connected machine, pull the model into a named volume, then move that
volume (or its backing directory) to the air-gapped host:

```bash
# on a connected machine
docker volume create ollama-models
docker run --rm -v ollama-models:/root/.ollama ollama/ollama:latest \
  sh -c "ollama serve & sleep 5 && ollama pull qwen2.5vl:3b"
# back up the volume and restore it as `hotato_ollama-models` on the target
docker run --rm -v ollama-models:/data -v "$PWD":/backup alpine \
  tar czf /backup/ollama-models.tgz -C /data .
```

With the model already in the volume, the air-gapped stack runs the judge
with no download.

---

## Credentials and the bearer token

Every request to the workspace is authenticated against one shared bearer
token, compared in constant time. Three ways to provide it (precedence high
to low):

1. **A Docker secret** (best): mount a secret file at
   `/run/secrets/hotato_token`; the entrypoint passes it as `--token-file`,
   keeping the token off the command line. Example `docker-compose.yml`
   addition:

   ```yaml
   services:
     hotato-workspace:
       secrets:
         - hotato_token
   secrets:
     hotato_token:
       file: ./deploy/hotato_token.txt   # chmod 0600
   ```

2. **An env var**: set `HOTATO_SERVE_TOKEN` in `deploy/hotato.env` (copied
   from `deploy/hotato.env.example`); the entrypoint writes it to a `0600`
   file in the container and passes `--token-file`, keeping it off the
   command line.

3. **Generated**: set nothing and the server generates a token with
   `secrets.token_urlsafe` on first start, stored `0600` at
   `/data/serve/default/token`; read it with
   `docker compose exec hotato-workspace cat /data/serve/default/token`.

The token file and audit log are written owner-only (`0600`). Keep
`deploy/hotato.env` and any token file `0600` on the host too, and out of
version control.

---

## Backup and restore

Everything the workspace needs lives in the `/data` volume: the registry
(SQLite), the content-addressed evidence store, the serve token, and the
audit log. Back up that one volume:

```bash
# back up hotato_hotato-data to ./hotato-data-backup.tgz
docker run --rm -v hotato_hotato-data:/data -v "$PWD":/backup alpine \
  tar czf /backup/hotato-data-backup.tgz -C /data .

# restore into a fresh volume
docker volume create hotato_hotato-data
docker run --rm -v hotato_hotato-data:/data -v "$PWD":/backup alpine \
  sh -c "cd /data && tar xzf /backup/hotato-data-backup.tgz"
```

Because the evidence store is content-addressed (sha256), a restored
artifact is the same bytes that produced the original verdict; any digest
mismatch is caught and surfaced explicitly.

---

## Upgrade

The image installs hotato from the source in this repository, so upgrading
is a rebuild:

```bash
git pull                       # or check out the release tag you want
docker compose build --pull    # rebuild the image
docker compose up -d           # recreate the workspace container
```

The `/data` volume is untouched by a rebuild, so your registry and evidence
survive the upgrade; the registry re-asserts its (idempotent) schema on
open. Back up `/data` before a major upgrade, like any stateful service.

---

## Zero-migration promise

The `/data` registry and content-addressed evidence store use the same
schemas the managed cloud uses. Conversation artifacts, conversation tests,
and dashboards move between self-hosted and cloud without changing a line.
Self-host is the same platform, on your own infrastructure -- the whole QA
platform, no hosted login required.

---

## Air-gapped deployment

The default stack (the workspace alone) runs offline at run time, so it
works on an air-gapped host once the image is present: bring the image over
instead of pulling it on the target:

```bash
# on a connected machine
docker compose build
docker save hotato-selfhost:local | gzip > hotato-selfhost.tar.gz
# move the tarball to the air-gapped host, then:
gunzip -c hotato-selfhost.tar.gz | docker load
docker compose up -d
```

This is air-gapped once its one-time steps are done: the judge profile
needs the model pull above, unless you pre-seed the `ollama-models` volume.
With the image loaded (and the model volume pre-seeded, if you use the
judge), the whole stack runs fully offline.

---

## What "stays offline" covers

- This covers the default stack's run-time behaviour: the workspace server
  binds a listening socket and stops there, imports nothing that phones
  home, and keeps audio, traces, and evaluations on the machine. The
  workspace's own writes stay local too: its append-only audit log, plus
  the contract bundles and registry rows a pin-to-contract POST creates
  through the fleet machinery.
- The default workspace runs on a normal Docker bridge so its port can
  publish; the guarantee rests on the server's own behaviour. Verify it
  directly on your own machine, independent of any firewall:

  ```bash
  ./deploy/verify-zero-egress.sh
  ```

  It (1) confirms only `127.0.0.1:8321` is published, (2) runs the same
  image on an `internal` Docker network with egress physically removed and
  shows the workspace still answers a view with `200` -- proof the server
  needs nothing external -- and (3) lists ESTABLISHED connections inside the
  running container and confirms every one stays internal.

- **Every opt-in path that reaches the network is named explicitly.** The
  local judge talks to the in-stack Ollama over the private network (plus the
  one-time model pull); `hotato pull` / `capture` fetch calls from a
  provider you configure; a hosted judge or the pyannoteAI diarizer send
  data off-box only behind an explicit `--judge-egress-opt-in` /
  `--egress-opt-in` flag. All enumerated, command by command, in
  [`docs/EGRESS.md`](EGRESS.md) and [`docs/THREAT-MODEL.md`](THREAT-MODEL.md).

A capability that lacks its input for a given run returns INCONCLUSIVE --
plain and consistent, whether you self-host or not.

---

## See also

- [`docs/WORKSPACE.md`](WORKSPACE.md): the five views, auth, and the audit log
- [`docs/EGRESS.md`](EGRESS.md): every network call site, command by command
- [`docs/THREAT-MODEL.md`](THREAT-MODEL.md): the offline / opt-in split and the
  workspace's threat-model row
- [`SECURITY.md`](../SECURITY.md): posture summary and reporting a vulnerability


================================================================================
FILE: docs/SET-AND-FORGET.md
================================================================================

# Set and forget: passive turn-taking regression monitoring

`hotato sweep` turns from a command you remember to run into a job that
runs on schedule and asks for your attention only when it finds something
worth acting on.

Every command below is shipped (verify with `hotato <command> --help`).
Run the whole loop now with `--demo` -- no credentials needed -- before
pointing it at your own stack.

## The loop

```
connect (once) -> sweep (on a schedule) -> read the dashboard
   -> promote confirmed candidates into fixtures -> hotato run gates CI
```

Sweeping only reads: it lists candidate timing moments for you to judge
and label. `hotato fixture create` / `hotato fixture promote` turn a
candidate into a permanent test.

## 1. Connect once

```bash
hotato connect vapi --api-key <key>
# or: VAPI_API_KEY=<key> hotato connect vapi
```

Runs a lightweight live auth check (skip with `--no-verify`) and stores
the credential in `~/.hotato/connections.json`, file mode `0600`. The key
travels only to the vendor's own API, kept out of hotato's hands. Full
per-stack credential table: [`CONNECT.md`](CONNECT.md). LiveKit and
Pipecat are capture-in-your-infra: use `hotato setup --stack
livekit|pipecat` instead.

Once connected, `--stack` and credential flags are optional on `pull` /
`sweep` if exactly one stack is connected.

## 2. Sweep on a schedule

Run the same command your terminal runs interactively, from cron or CI:

```bash
hotato sweep --stack vapi --since 7d --format json > hotato-sweep.json
hotato sweep --stack vapi --since 7d --out hotato-sweep.html --no-open
```

| Flag | Does |
|---|---|
| `--format json` | Machine-readable candidate list `fixture promote` reads -- redirect to a file every time (`--out` writes the HTML dashboard; capture stdout for the JSON) |
| `--out FILE.html --no-open` | Writes the shareable dashboard without popping a browser -- for when nothing is watching the screen |
| `--since 7d` | Scopes the pull to recent calls; a nightly job narrows this to `--since 1d` so it only re-scores what's new |

A crontab entry, 03:00 daily:

```
0 3 * * * cd /path/to/repo && hotato sweep --stack vapi --since 1d --format json > "reports/sweep-$(date +\%F).json" && hotato sweep --stack vapi --since 1d --out "reports/sweep-$(date +\%F).html" --no-open
```

A runnable version of this, plus the CI half of the loop:
[`examples/set-and-forget/`](../examples/set-and-forget/README.md).

No stack connected? Everything above works with `--demo` instead of
`--stack ... --since ...` -- credential-free, on two bundled calls:

```bash
hotato sweep --demo --format json > hotato-sweep.json
hotato sweep --demo --out hotato-sweep.html --no-open
```

## 3. Read the report

| Output | What it is |
|---|---|
| HTML (default `--format`) | One self-contained file: every candidate across every swept call, ranked by salience, hear-the-bug audio embedded for the top `--audio-top` (default 8). Calls that couldn't be scored (mono/mixed without `--allow-mono`, an unreadable file) list under Skipped with the reason. |
| JSON (`--format json`) | The same list as structured data: source recording, timestamp (`t_sec`), kind (`agent_stop_no_caller`, `overlap_while_agent_talking`, ...), salience score -- a fact for you to judge, per candidate: yielded, or held? |

## 4. Promote a confirmed bug into a fixture

Once you've listened to a candidate and decided what should have happened,
`hotato fixture promote` turns it into a permanent regression test -- no
`--stereo` / `--onset` needed, since the candidate ref already carries the
recording and the moment:

```bash
hotato fixture promote hotato-sweep.json#3 --expect yield \
    --id refund-cutoff-001 --out tests/hotato
```

`CANDIDATE_REF` is `FILE#N` (Nth candidate, ranked order) or
`FILE#CALL:N` (Nth candidate from one call, by file or call id) -- e.g.
`hotato-sweep.json#3` or `hotato-sweep.json#call_abc123:2`. `--expect
yield`: the agent should have stopped. `--expect hold`: it should have
kept talking through a backchannel. Scored immediately -- an unscorable
candidate is refused (exit 2).

This is the loop's most important step: a suspicious call becomes a fact
CI enforces forever, not something you noticed once and forgot. (`hotato
fixture create --stereo ... --onset ...` does the same from a raw
recording and a timestamp you already know.)

## 5. Gate CI on your fixtures

Every promoted fixture lives under `--out DIR` (`tests/hotato/scenarios` +
`tests/hotato/audio` above) and scores with the same command, laptop or
CI:

```bash
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio --format json
```

Exit `1` pins a regression; `0` means every fixture passes. Wire it into a
GitHub Action (drop-in workflow: [`.github/workflows/hotato.yml`](../.github/workflows/hotato.yml),
guide: [`docs/CI.md`](CI.md)) or an existing pytest run with one flag
(`pytest --hotato-suite --hotato-suite-scenarios tests/hotato/scenarios
--hotato-suite-audio tests/hotato/audio`, see [`docs/PYTEST.md`](PYTEST.md)).
[`examples/set-and-forget/`](../examples/set-and-forget/README.md) has the
complete cron + CI pairing.

## Worked example (zero setup, verified end to end)

Every command above works right now on the bundled demo calls -- watch a
failure get pinned before wiring anything to your own stack:

```bash
hotato sweep --demo --format json > hotato-sweep.json
hotato fixture promote hotato-sweep.json#2 --expect yield \
    --id demo-missed-interruption --out tests/hotato
hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio
# exit code 1: the demo agent never yielded for an interruption -- pinned
```

That last `run` also prints the fix card (fix class, config knob,
direction to move it): the failed fixture is a labelled bad-agent moment
from the bundled demo battery.

## What you control in this loop

- Labeling, fixture creation, and threshold tuning are each a command you
  run, for a candidate you listened to.
- `sweep` and `ingest` (the webhook-driven version of this loop, see
  [`docs/INGEST.md`](INGEST.md)) both report candidates as timing facts;
  fixtures scored with `hotato run` produce the pass/fail verdict.
- This runs as processes you control: cron, CI, and a webhook handler all
  shell out to the same CLI, on a schedule that's yours to own.


================================================================================
FILE: docs/STATE-ADAPTERS.md
================================================================================

# State adapters: ground a `state` assertion in your system of record

`state` and `state_change` assertions are **Authority 2**: post-call state
verification -- querying a **system of record** after the call and
comparing it against what the agent claimed ("I issued the refund"). A
state adapter is the small, pluggable seam that runs that query.

`hotato test run --state FILE` loads a state adapter from `FILE`. Three
adapters ship, and one method backs all three:

```
query(resource, **filters) -> dict | None
```

Return the record for `resource` matching every `filters` key, or `None`
when the system of record was read and holds no such record.

## The three-outcome boundary

A `state` query lands on exactly three outcomes -- the distinction is the
whole point:

- **The record is present and matches** -> `PASS`.
- **The system of record was read and holds no matching record** (or a
  field does not match) -> a grounded `FAIL`. The agent claimed something
  the record does not confirm.
- **The system of record could not be reached or read** (network error,
  timeout, a 5xx, a non-JSON response, a DB error) -> `INCONCLUSIVE`, with a
  reason every time.

A `state` assertion is satisfied by the query result alone: a lookup plus a
dict comparison, deterministic, no model in the loop.

## The three adapters

### 1. `mock`: the local sandbox (default, offline)

A JSON or SQLite fixture: `{resource: rows}`. Runs offline, byte-stable,
ready with no opt-in. Use it for regression suites and for a captured
before/after snapshot (`state_change` reads a `{"before": [...], "after":
[...]}` shape); `--state some_sandbox.json` and `--state some.sqlite3` both
select it.

### 2. `http`: a REST system of record

Queries your API over stdlib `urllib`. A **resource map** turns a query into
one request:

```json
{
  "adapter": "http",
  "egress_opt_in": true,
  "base_url": "https://api.yourcompany.com/v1",
  "auth": { "type": "bearer", "token_env": "RECORDS_API_TOKEN" },
  "timeout": 30,
  "resources": {
    "appointment": {
      "path_template": "/patients/{patient_id}/appointment",
      "method": "GET",
      "params_map": { "status": "appt_status" },
      "response_pointer": "data/appointment"
    }
  }
}
```

`query("appointment", patient_id="P1", status="booked")` fills `{patient_id}`
into the path (URL-encoded), sends the remaining filters as query params (GET)
or a JSON body (POST) under their `params_map` wire names, then extracts the
record dict at `response_pointer`.

- `response_pointer` is a JSON-pointer-ish path (`/`- or `.`-separated; digits
  index a list; empty means the whole body is the record). A pointer that
  resolves to nothing on a well-formed response means "no such record" ->
  `None`.
- `method` is `GET` (default) or `POST`.
- **HTTPS is required by default.** A plain `http://` base URL is refused
  unless you set `"allow_http": true` (only for a trusted local endpoint);
  otherwise a state query would send filter values and the auth header in
  cleartext.
- A 404 means the record does not exist -> `None` -> grounded FAIL. Every
  other non-2xx, and any network/timeout/non-JSON failure, is INCONCLUSIVE
  (`query` raises `StateAdapterError`; the assertion engine turns it into an
  INCONCLUSIVE result, with the structured cause on the adapter's
  `last_error`).

### 3. `sql`: a SQL system of record

Queries a database with a **parameterized, read-only** SELECT.

```json
{
  "adapter": "sql",
  "sqlite_path": "records.sqlite3",
  "resources": {
    "refund": {
      "query": "SELECT order_id, status, amount FROM refunds WHERE order_id = ? AND status = ?",
      "params_order": ["order_id", "status"]
    }
  }
}
```

`query("refund", order_id="O1", status="issued")` binds `[order_id, status]`
as the statement's parameters and returns the first row as a `{column:
value}` dict, or `None` when no row matches.

- Connection source (exactly one): `sqlite_path` (a local file DB, fully
  local, no egress); `dsn` + `driver` (a DBAPI module to import and
  `connect`, e.g. `psycopg2` for PostgreSQL -- a network DB needing
  `egress_opt_in: true`; imported lazily, never a hard dependency); or a
  caller-supplied `connection` object (Python API only).
- Use the placeholder style your driver expects (`?` for sqlite3, `%s` for
  psycopg2). Hotato only passes the bound value sequence through.
- A DB/driver error resolves to INCONCLUSIVE -- a query never guesses a FAIL.

## Injection safety and read-only discipline

Enforced by construction and covered in `tests/test_state_adapters_real.py`:

- **Injection-safe.** Filter values are always bound as query parameters,
  never string-interpolated into the SQL: a filter value carrying SQL
  (`"O1'; DROP TABLE refunds; --"`) is treated as a literal, matching no
  row, table untouched.
- **Read-only.** Every mapped SQL query is validated at construction (and
  re-checked before each execute): it must begin with `SELECT` (or a `WITH
  ... SELECT` CTE), carry no second statement after a `;`, and contain no
  data-modifying keyword, so a `DELETE`/`UPDATE`/`DROP`/... mapped query is
  rejected.

## Egress opt-in

The `http` adapter, and a `sql` adapter over a `dsn`, are **network paths**
(see [`docs/EGRESS.md`](EGRESS.md) and
[`docs/THREAT-MODEL.md`](THREAT-MODEL.md)). `load_state_adapter` **refuses**
them unless the config sets `"egress_opt_in": true` -- the same opt-in
`--egress-opt-in` applies to the hosted diarizer and `--notify`. Without
it, you get a clear usage error (the CLI's exit-2 path) before any
connection is made. A local `sqlite_path` SQL DB and the mock sandbox run
entirely on your machine, opening no socket, no opt-in needed.

When an `http` / non-local-`sql` query does fire, only the mapped filter
VALUES leave the machine (in the URL, query string, or JSON body). Audio,
transcript, and the config file itself stay on your machine.

## Credentials

Adapters take environment-variable **names** for credentials, keeping the
config file free of secrets and shareable. Supply the secret at run time
from the environment, e.g. `source` a `0600` file that exports it:

- `bearer`: `{ "type": "bearer", "token_env": "RECORDS_API_TOKEN" }`
- `basic`: `{ "type": "basic", "username_env": "DB_USER", "password_env": "DB_PASS" }`
  (`username` may be inline since it is not a secret).
- `header`: `{ "type": "header", "headers": { "X-Api-Key": { "env": "API_KEY" }, "X-Tenant": { "value": "acme" } } }`,
  where each header value is either `{ "env": NAME }` (a secret from the
  environment) or `{ "value": LITERAL }` (a non-secret constant).

A missing credential env var fails fast at construction, naming the
variable in the message, never the value. Credential values stay out of
logs and `last_error`.

## `state_change` and before/after snapshots

`state_change` reads a `before` and `after` snapshot to measure a delta.
Only the **mock** adapter provides both (from a captured `{"before": ...,
"after": ...}` fixture, or `<table>__before` / `<table>__after` SQLite
tables). The `http` and `sql` adapters answer point-in-time `state`
queries -- a live API/DB exposes only "now" -- so their `before` snapshot
reports absent, grounding a `state_change` against them at INCONCLUSIVE
instead of guessing "no change."

## See also

- [`docs/ASSERTIONS.md`](ASSERTIONS.md): the `state` / `state_change`
  assertion kinds.
- [`docs/EGRESS.md`](EGRESS.md), [`docs/THREAT-MODEL.md`](THREAT-MODEL.md):
  the network rows.
- `src/hotato/state_adapter.py`: the adapters; `tests/test_state_adapters_real.py`: the tests.


================================================================================
FILE: docs/THREAT-MODEL.md
================================================================================

# Threat model

Your call recordings stay on your machine. Every network action is one you
name. This page draws the line: which commands are offline-only, which
reach the network, and what holds true either way.

## Three guarantees

1. **You apply every production change yourself.** `plan` and `patch`
   produce a proposal; `apply` dry-runs by default on a fresh staging
   clone. No command pushes to production on its own.
2. **Recordings stay on your machine.** Audio moves only from your stack to
   your disk, only when you run `pull`/`capture`/`sweep`. The one opt-in
   exception: the hosted `--diarizer pyannoteai` backend, gated by
   `--egress-opt-in`.
3. **A webhook payload is data, never code.** `ingest` reads a
   completed-call notification, extracts a recording reference, and scans
   it. Payload fields are read, never executed or used to choose what
   runs.

## Core: fully offline, on your machine

These commands read and write only the local files you point them at. They
open no sockets -- this is the whole scoring, analysis, and fixture
surface.

| Command | What it does |
|---|---|
| `run` | score a local WAV (or the bundled self-test battery) |
| `scan` | list candidate moments in a local recording |
| `trust` | input-health check on a local recording |
| `report` | render a self-contained HTML report from local run data |
| `fixture` | create or promote a local moment into a regression fixture |
| `compare` | score a before/after pair of local takes |
| `verify` | roll up before/after run envelopes on disk |
| `diagnose` | explain a finished local run envelope (read-only) |
| `plan` | combine a diagnosis and config into a fix-plan JSON |
| `patch` | render a fix plan into a paste-ready patch; never applies it |
| `demo` | run the packaged two-call battery, zero extra files |
| `team`, `export`, `benchmark` | aggregate or export local run data |
| `setup`, `describe`, `init` | print recording config, emit the CLI manifest, scaffold local integration files |
| `rubric run`, `rubric calibrate` | score a local transcript against a rubric with a **local** model (Ollama on `localhost`); opens no off-box socket by default, and the verdict cache is local too. Reaches the network only with an explicit hosted or non-local judge (see below) |
| `test run` (rubric lane) | same local judge as `rubric run`, run inline on a test's `assertions.rubric` lane. Local by default |
| `counterexample compile\|verify\|reproduce\|inspect\|export\|predicate` | read local scenario/test/capsule files, write a new local capsule or share-safe projection. Loads no provider adapter, model, subprocess, or network client |

Default retention is local-only: reports, envelopes, and exports land
exactly where you point them.

### Counterexample capsule boundary

A private `.hotato-repro` holds the source scenario and test, reduced
fixture, target assertion result, and proof journal -- treat it as
sensitive source material. The compiler creates it with owner-only
permissions where the host supports POSIX, never overwrites an existing
destination, and promotes a sibling staging directory only once every file
and hash exists.

The verifier rejects: path traversal; symlinked members and directories;
special files; undeclared files; oversized or deep inputs; digest
mismatches; broken deletion chains; evaluator-source drift on the strict
proof path; and a minimality claim that admits a preserving unit deletion.
Preflight caps a capsule at 1,024 files, 4,096 directories, 64 directory
levels, 64 MiB per member, and 256 MiB total. `reproduce` is a separate,
current-evaluator check: it permits evaluator drift, but still validates
capsule integrity and the source-to-final delete-only chain.

Replay also bounds proof-specific work: 256 KiB per selected assertion, 2
MiB per canonical candidate scenario, 256 KiB of rendered transcript text, 2
MiB per assertion result, 10,000 evidence rows, 512 accepted proof-chain
steps, 10,000 deletion operations per accepted step, and 512 remaining
deletion units in a completed minimality proof. Accepted steps require
fresh oracle evaluations -- cached preserved journal rows can't inflate the
chain. Proof regexes use a closed, fixed-width 1,024-byte subset that
refuses groups, alternation, backreferences, and variable quantifiers.
These limits apply to counterexample compilation and replay, not to global
evaluator limits. The selected-assertion and proof-regex byte checks run
before the general assertion validator can invoke Python's regex parser.

Each profile has an exact member allowlist; a manifest can't authorize a
new file outside it. For `share-safe-v1`, the only members are
`capsule.json`, `report.md`, `report.html`, `card.svg`, `README.md`, and
`MANIFEST.sha256.json` -- each human-facing file is reconstructed
canonically and compared byte for byte during inspection. This stops a
modified report, README, or card from being accepted just because its
replacement digest was written into a new manifest.

The canonical share renderer's bytes are frozen as part of the capsule v1
exchange contract. A future renderer must retain v1 output, or use a
versioned profile/format, rather than silently changing bytes under an
existing proof.

The evaluator digest binds the shipped Python source closure and package
version, not the interpreter build, host platform, CPU, or native
libraries. Strict replay compares the recorded result, content, and trace
hashes, so a runtime difference that changes behavior is rejected. Accepted
transforms and the final single-unit inventory are proof-bearing;
non-`PRESERVED` journal rows are diagnostic history, not independently
replayed.

The capsule directory must stay unchanged during a command. Persistent
mutation, root replacement, and symlink or special-file substitution are
all detected. A privileged local process that can swap and restore bytes
between individual reads sits outside the v1 proof snapshot -- move
untrusted capsules into a private, non-writable workspace before
verification.

`counterexample export` derives a non-runnable projection after strict
private verification. It omits scenario/test bodies, transcript content,
tool payloads, state values, provider identifiers, reducer paths, and
per-candidate digests; private minimality rows are reduced to aggregate
outcome counts. The projection exposes the payload-free selected failure
code for review while keeping its field/key/rule/detector/index
discriminator private. It still contains identifiers and hashes that
remain correlators -- low-entropy or externally known source material can
be tested against them. `share-safe-v1` belongs inside the team's normal
engineering-artifact access boundary; it does not claim anonymous public
disclosure.

## Network: only when you explicitly request it

These commands reach outside the machine, and only because reaching out is
their job. Each requires you to name a stack, a repository, or a webhook
you configured.

| Command | Network surface | Notes |
|---|---|---|
| `connect` | none at connect time | stores a stack's credentials locally at `0600`. Setup for the pull path; no audio moves |
| `pull` | your voice stack's API | bulk-fetch recent recordings from a stack **you** connected, into a local folder |
| `capture` | your voice stack's API | fetch and score one call from your stack |
| `sweep` | your voice stack's API | `pull` + analyze in one command |
| `inspect` | your voice stack's API | read (never write) the current turn-taking config. Read-only |
| `ingest` | a webhook endpoint you host | worker that scans each completed call. Payloads are data, not instructions (guarantee 3) |
| `issue` | GitHub, via your local `gh` | file a sweep's candidates as an issue. Uses your existing `gh` auth |
| `pr` | GitHub, via your local `gh` | open a PR adding promoted fixtures. Uses your existing `gh` auth |
| `apply` | a git clone you point it at | applies a patch to a fresh **staging** clone only, never the source. Dry-run by default; refuses a both-axes threshold funnel |
| `--diarizer pyannoteai` | hosted diarizer backend | the only audio path that can send audio off-box, and only with `--egress-opt-in`. The default diarizer is local |
| `--judge-provider hosted` / non-local `--judge-endpoint` (any rubric command) | a hosted or remote model host you name | sends the transcript + rubric criterion off-box for judging. Refused (exit 2) unless `--judge-egress-opt-in`. The default judge is a local Ollama model that stays on the box. See [`docs/EGRESS.md`](EGRESS.md) and [`docs/RUBRIC.md`](RUBRIC.md) |
| `test run --state` http adapter | your system-of-record's REST API | only when the state-config names `adapter: http`, and only with `egress_opt_in: true` in that config. Sends the mapped filter VALUES for one `state`/`state_change` query -- audio, transcript, and the config itself stay local. See [`docs/STATE-ADAPTERS.md`](STATE-ADAPTERS.md) |
| `test run --state` sql adapter over a `dsn` | your database, over the network | only when the state-config names `adapter: sql` with a `dsn`, and only with `egress_opt_in: true`. A parameterized, read-only SELECT with the mapped filter values bound as data. A local `sqlite_path` opens no socket |

The state adapters read a post-call **system of record** to ground an
Authority-2 `state` assertion (did the refund/appointment get written?). A
record the system of record can read, and does not hold, is a grounded
FAIL; a system of record Hotato could **not** reach or read (network error,
timeout, 5xx, non-JSON) is INCONCLUSIVE, never a guessed verdict.
Credentials come from environment-variable **names** in the config, keeping
the secret itself out of it; the HTTP adapter refuses a plain-`http://`
base URL unless `allow_http: true` is set for a trusted local endpoint. The
default `--state` path -- the local mock sandbox (a JSON/SQLite fixture)
and a local `sqlite_path` SQL DB -- runs entirely on-machine, no opt-in
needed.

Notify surfaces (Slack, GitHub) fire only through credentials you
configured (`gh`, a Slack token), and only for actions you invoked.

## Network trust: proxies and TLS

Every networked command above (`pull`, `capture`, `sweep`, `inspect`,
`apply`, and the credential probe in `connect`) makes its HTTP calls
through Python's standard `urllib`, which honors the `HTTP_PROXY` /
`HTTPS_PROXY` / `NO_PROXY` convention -- the same one `curl`, `pip`, `git`,
and `docker` follow. That's deliberate: it lets Hotato work behind a
corporate proxy with no extra flags. It also means an environment variable
set for the Hotato process controls where its outbound credentialed
requests go -- a trust decision worth stating plainly.

Two things bound how far that ambient trust reaches:

- **TLS certificate validation is always enforced.** Every credentialed
  base URL in `capture.py` and `apply.py` is hardcoded `https://`
  (`api.vapi.ai`, `api.retellai.com`, `api.twilio.com`, and the other
  supported stacks). A proxy set via `HTTP_PROXY`/`HTTPS_PROXY` can see the
  `CONNECT` target host and port, and can refuse or stall the connection (a
  denial of service) -- but reading or rewriting the `Authorization` header
  or the response body also requires a certificate the machine's own trust
  store already accepts for that vendor's domain: a much larger compromise
  (a rogue trusted root CA already installed), outside `HTTP_PROXY`'s reach
  and Hotato's control.
- **The threat prerequisite is already a compromised local environment.**
  An attacker able to set environment variables for the Hotato process can
  already read `VAPI_API_KEY` / `RETELL_API_KEY` / etc. straight out of the
  environment, or read the `0600` credentials file `connect` wrote -- both
  easier than standing up a TLS-valid proxy.

If you don't trust the ambient proxy environment a command will run in, set
`HOTATO_NO_PROXY=1` to make Hotato's HTTP opener ignore `HTTP_PROXY` /
`HTTPS_PROXY` for that run, or unset those variables first. The default is
unchanged: proxy env vars are honored, matching curl/pip/git, so a
corporate-proxy setup keeps working with no configuration.

## Hotato's containment guarantees

The three guarantees above, restated as the boundary they draw: recordings
stay local unless you opt in; a webhook payload is read as data and never
executed (guarantee 3); and the furthest a config change goes on its own is
a patch you apply yourself, to a staging clone.

## Verifying the posture

Confirm the posture yourself, without trusting this page:

- **Self-hosted, nothing to audit but the standard library.** `pip install
  hotato` pulls zero runtime dependencies; the only network code path is
  the one you explicitly invoke.
- **Credentials at `0600`.** `connect` writes stack credentials locally
  with owner-only permissions; check with `ls -l` on your credentials path.
- **Local-only retention.** Reports, envelopes, and exports exist exactly
  where you wrote them.
- **Egress is opt-in.** The only off-box audio path is `--diarizer
  pyannoteai`, gated by `--egress-opt-in` -- audio stays on the machine
  until you set it.

## Drive-a-call (`run_scenario`): placing a live call

`run_scenario` (`src/hotato/drive.py`, wired into the Vapi and Twilio
adapters) places an outbound call against a live agent and pulls the
recording. Its threat surface, and the controls on it:

- **A billable, outbound call always requires explicit opt-in.**
  `run_scenario` places the call only when BOTH live credentials AND an
  explicit egress opt-in (`HOTATO_DRIVE_OPT_IN=1` or `egress_opt_in: true`
  on the scenario) are present. Absent either, it raises a clean structured
  refusal and dials nothing -- the same opt-in posture as `--allow-mono` /
  `HOTATO_ALLOW_PRIVATE_URLS` / `--egress-opt-in`.
- **Production config stays untouched.** The surface is create-and-read
  only: the only verbs issued are `POST` (create the call) and `GET` (poll
  status, list the recording, download it) -- no `PUT`/`PATCH`/`DELETE`
  path to alter an assistant, phone number, or any other provider resource
  in place. For Vapi the call originates FROM the staging clone the
  experiment created, keeping the production source untouched.
- **The caller side is labelled precisely.** The produced conversation
  carries `origin.kind == "real"` with the provider and its call id, and
  `origin.caller` (`scripted-twiml` for the fixed-timeline Twilio caller,
  `assistant-originated` for a Vapi call the assistant placed) -- stating
  exactly what happened, with no claim that a human placed the call or that
  the scripted caller reacted to the agent.
- **The recording download inherits every capture-side control.** The
  vendor URL (`stereoUrl` / `RecordingSid` media) flows through the same
  validated download as `capture`: http(s)-only scheme allowlist,
  default-deny SSRF (loopback/private/metadata refused unless
  `HOTATO_ALLOW_PRIVATE_URLS=1`), cross-host `Authorization`-strip on
  redirect, and atomic write. The API key is never attached to a
  pre-signed media URL from the vendor's JSON response.

See [`docs/DRIVE-A-CALL.md`](DRIVE-A-CALL.md) and the drive-a-call rows in
[`docs/EGRESS.md`](EGRESS.md).

## Team workspace (`hotato serve`): a new local listening socket

`hotato serve` (`src/hotato/serve/`, wired into the CLI) opens a new local
HTTP listening socket to serve the conversation-QA views over the
fleet registry and evidence store. Its threat surface, and the controls on
it:

- **Localhost-default bind.** The server binds `127.0.0.1` unless you
  explicitly pass `--host`. A non-loopback bind (e.g. `--host 0.0.0.0`)
  always prints a prominent warning at start, and always requires that
  explicit flag.
- **Token auth on every request.** A shared bearer token authenticates
  every request, compared in constant time (`hmac.compare_digest`). It's
  either operator-supplied (`--token` / `--token-file`) or generated with
  `secrets.token_urlsafe` and stored `0600` under the per-workspace state
  dir. Browsers bootstrap an in-memory, HttpOnly session cookie from the
  printed `/?token=…` URL, then the token is stripped from the address bar
  via a redirect; it stays out of every response body. An unauthenticated
  request gets `401` before it's routed anywhere.
- **Reads everywhere; one fenced write route.** Every view issues only
  `SELECT`s against the registry and reads evidence blobs by digest. The
  single write endpoint, `POST /calls/<id>/pin`, delegates to the same
  fleet label/contract machinery the CLI drives, and is fenced beyond the
  bearer/cookie auth above: the session cookie is `SameSite=Strict`, and a
  cookie-authenticated POST must carry a same-origin `Origin`/`Referer`
  header matching the request's own `Host` -- a forged cross-site form is
  refused `403` before any handler runs (a bearer-authenticated request
  needs no origin header; a cross-site attacker cannot set one). The
  delegated mint is atomic, so a refused pin (4xx with reason) leaves no
  artifact. The serve layer's own file stays the append-only audit log
  (`…/serve/<workspace>/audit.jsonl`, `0600`), recording who
  (token/session prefix, the secret stays out of it), what (method + path,
  token stripped from the query), when, and the response status of every
  request -- pin attempts included, accepted or refused.
- **Zero egress.** The server only binds a listening socket: no outbound
  connection, nothing that phones home. Audio, traces, and evaluations
  stay on the machine. A test allowlists loopback and fails if any view
  attempts an external connection.
- **Evidence renders as data, never as executable content.** The raw
  evidence endpoint (`/evidence/<digest>`) serves blobs as `text/plain`
  with `X-Content-Type-Options: nosniff` (pages also set
  `X-Frame-Options: DENY`, `Referrer-Policy: no-referrer`), keeping a
  crafted evidence blob inert in the viewer. Redacted transcript/trace
  content is scrubbed at the data layer, so both the HTML and the JSON
  mirror render only the `[redacted]` placeholder.
- **Workspace id sanitized against path traversal.** The workspace id is
  used verbatim only in parameterized SQL; as a state-directory name it is
  sanitized to stay inside the registry home.

See [`docs/WORKSPACE.md`](WORKSPACE.md).

Security policy and reporting: [SECURITY.md](../SECURITY.md).


================================================================================
FILE: docs/TRANSCRIBE.md
================================================================================

# `hotato run --stereo call.wav --transcribe`: read a transcript next to the score

Hotato's gold reference is a deterministic **energy VAD**: `did_yield`,
`talk_over_sec`, and `seconds_to_yield` come from frame energy alone, and
every published number is computed that way. The opt-in `[transcribe]`
extra runs an off-the-shelf speech-to-text model over the same recording
and returns plain text with per-segment timestamps -- so a reader, human or
agent, sees WHAT was said next to WHEN the timing engine says it was said.

That is the entire feature: a transcript is attached as **context** beside
the score, computed strictly after scoring is final, and never feeds back
into it.

## What stays unchanged

- **The timing score stays grounded in the audio.** `did_yield` /
  `talk_over_sec` / `seconds_to_yield` / every other field computes
  identically with or without `--transcribe`; adding it produces
  byte-identical timing numbers, pinned by a test
  (`tests/test_transcribe.py`).
- **Plain text, not a quality judgment.** The transcript reports what was
  said, not how well: plain text and timestamps only -- no WER, accuracy,
  or quality number, anywhere in hotato.
- **Timestamps only, no speaker attribution.** No caller/agent labels --
  anonymous speaker separation lives in the separate
  [`docs/DIARIZE.md`](DIARIZE.md) extra.

## Quickstart

```bash
# Install the opt-in extra (local, offline; no gated token, unlike [diarize]):
pip install 'hotato[transcribe]'

# Score a call and attach a transcript next to the verdict:
hotato run --stereo call.wav --transcribe --format json
```

With the extra absent, `--transcribe` fails loud: a clean error, exit `2`,
never a silent skip.

## Usage

```bash
# base.en, device auto-detected:
hotato run --stereo call.wav --transcribe

# pick a model + device:
hotato run --stereo call.wav --transcribe --transcribe-model small.en \
  --transcribe-device cpu

# also works on the diarized-mono path:
hotato run --mono call.wav --diarize --transcribe
```

`--transcribe` needs a **single** audio file: `--stereo`, or `--mono` when
scoring through the `--diarize` front-end (it transcribes the same mono
file the diarizer separated). Two separate `--caller`/`--agent` files
raise a clean usage error naming `--stereo`, rather than guessing which
channel to transcribe or silently dropping one. Combining `--transcribe`
with the self-test battery (`--suite` / `--scenarios`+`--audio`) also raises
a clean usage error, not a silent no-op, so CI never mistakes "flag ignored"
for "flag applied."

## Models and device

- Default model: `base.en` (English-only, the fastest useful size).
  Override with `--transcribe-model NAME` -- any name or local path
  `faster-whisper`/CTranslate2 accepts.
- Default device: `auto` -- `cuda` if CTranslate2 reports a CUDA device,
  else `cpu`. `compute_type` defaults to `float16` on GPU and `int8` on CPU
  (CTranslate2's documented fast defaults per device, not an accuracy
  claim); both overridable.
- Every choice used is stamped in the output (`transcript.model` /
  `.device` / `.compute_type` / `.language`), so a report is reproducible.

## Backend and extra

- **`[transcribe]`**
  - Brings in: `faster-whisper` (a CTranslate2 re-implementation of
    OpenAI's Whisper).
  - Where it runs: local, CPU-viable, GPU-optional, fully offline once the
    model is cached.

```toml
transcribe = ["faster-whisper>=1.0"]
```

This extra keeps the core's Python floor at `>=3.9` (unlike `[diarize]`),
since `faster-whisper`/CTranslate2 support that range -- no model card to
accept, either.

### Model licenses (log per FTO note)

Using an off-the-shelf ASR model is integration, orthogonal to any Hotato
IP claim; licenses are logged here:

- `faster-whisper` -- **MIT** (code)
- `CTranslate2` -- **MIT** (the inference runtime `faster-whisper` runs on)
- Whisper model weights -- **MIT** (OpenAI)

## The invariant: context beside the score, grounded in the audio

A transcript is computed strictly **after** the score is final, over the
same audio the score already used, and attached on a new top-level key of
the envelope -- nothing already there is read or rewritten:

- `events`, `verdict`, `measurements`, and `signals` are the exact same
  values with or without `--transcribe`.
- The energy VAD's frame-level activity stays the ground truth for timing.
  A transcript word boundary is a separate thing -- ASR timestamps are a
  model estimate, read alongside it, never substituted in.
- A missing/broken `[transcribe]` extra raises a clean, actionable error --
  exactly like the `[neural]` and `[diarize]` seams: never a bare
  `ImportError`, a silent skip, or a fallback to a different backend.
- `hotato.transcribe` and the vendored `_engine` import nothing from each
  other, keeping the transcript path fully separate from the scorer.

## JSON shape (agents)

A `--transcribe` run is the same envelope as any single run, plus one
additive top-level `transcript` block. `events[0].verdict` is identical to
a run without it:

```json
{
  "mode": "single",
  "events": [{
    "verdict": {
      "did_yield": true,
      "seconds_to_yield": 0.42,
      "talk_over_sec": 0.18,
      "reasons": []
    }
  }],
  "transcript": {
    "text": "I need to check on my refund -- hold on, let me pull that up for you.",
    "segments": [
      {"start": 0.10, "end": 2.30, "text": "I need to check on my refund"},
      {"start": 2.60, "end": 4.95, "text": "hold on, let me pull that up for you."}
    ],
    "model": "base.en",
    "device": "cpu",
    "compute_type": "int8",
    "language": "en"
  }
}
```

Branch on the presence of `transcript`: present only when `--transcribe`
was passed, additive only -- the meaning of everything under `events` stays
the same. On the diarized-mono path, `transcript` still attaches even when
the diarization confidence gate refuses or downgrades the verdict: ASR runs
independent of diarization.

## Egress

Fully local at inference time: once the named model is cached, transcribing
opens no socket. The **first** run of a new model downloads its weights
from its public host (the same one-time fetch as any pip package with model
weights); every run after that is offline. See [`docs/EGRESS.md`](EGRESS.md)
for the full per-command network table.


================================================================================
FILE: docs/TRANSPORT-RUNTIME.md
================================================================================

# Transport runtime

Hotato separates three facts that voice-test systems often collapse:

1. a provider accepted or completed a call lifecycle operation;
2. a media/signalling runtime delivered bytes or events at an agent boundary;
3. the resulting conversation satisfied outcome, policy, timing, and reliability assertions.

The first fact comes from `TelephonyClient`. The second comes from a
`ConversationSession` implementation. The third comes from Hotato's evidence and scoring
lanes. A lifecycle status never substitutes for delivered-media evidence or task outcome.

## Runtime contracts

`hotato.call_runtime.CallController` is the lifecycle boundary:

```text
capabilities(provider)
create(spec) -> handle
get(provider, call_id) -> handle
wait(handle) -> terminal handle
cancel(handle) -> handle
export(handle, output_dir) -> redacted receipt path
cleanup(handle, export_path) -> local deletion receipt
```

`hotato.call_runtime.ConversationSession` is the duplex media/signalling boundary:

```text
capabilities()
connect()
events()
send_audio(pcm_s16le, sample_rate_hz, channels)
send_dtmf(digits)
hold(enabled)
transfer(destination, warm=False)
hangup()
close()
```

These are Python protocols. A provider SDK, local WebSocket harness, or separately operated
sidecar can implement them without being imported into Hotato's zero-dependency core.

Every capability has one explicit state:

| State | Meaning |
|---|---|
| `SUPPORTED` | The selected implementation executes the operation and returns the evidence its contract requires. |
| `UNSUPPORTED` | The selected implementation cannot execute the operation. |
| `UNOBSERVABLE` | The operation may occur elsewhere, but this boundary cannot return the evidence required to credit it. |

Unknown support does not become a failed quality verdict. `require_capability()` refuses both
`UNSUPPORTED` and `UNOBSERVABLE` before an operation starts.

## Normalized call events

Every media/signalling implementation emits `hotato.call-event.v1` records. Records form an
append-only SHA-256 chain per `(run_id, call_id)`:

```json
{
  "schema": "hotato.call-event.v1",
  "event_id": "agent-audio-0001",
  "run_id": "run-2026-07-17-001",
  "call_id": "call-01",
  "leg_id": "agent",
  "sequence": 0,
  "source": "livekit-sidecar",
  "kind": "audio.delivered",
  "observed_monotonic_ns": 321000000,
  "source_timestamp": "2026-07-17T12:00:00.321Z",
  "trace_id": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
  "payload": {"sample_rate_hz": 16000, "channels": 1, "bytes": 640},
  "raw_sha256": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
  "trust": "measured",
  "previous_event_hash": null,
  "event_hash": "sha256:<canonical-event-body-hash>"
}
```

`normalize_call_event()` rejects unknown fields, non-canonical JSON values, oversized events,
invalid digests, sequence gaps, clock reversal, run/call changes, and a mismatched claimed hash.
`AppendOnlyCallLog` rejects duplicate event IDs and verifies the entire chain.

`raw_sha256` hashes the bytes observed at the stated boundary. It does not claim those bytes
were heard by a participant. `trust` identifies the authority:

- `measured` for bytes or state measured at the boundary;
- `provider_reported` for provider lifecycle/API statements;
- `sidecar_reported` for an external runtime's report;
- `derived` for a deterministic transform over retained inputs;
- `model_reported` for a model output;
- `operator_attested` for an operator statement;
- `unverified` when no stronger authority is available.

## Lifecycle controller matrix

`hotato.telephony.TelephonyClient` controls provider lifecycle APIs. It does not transport media.

| Operation | local | Twilio | Vapi | Retell |
|---|---:|---:|---:|---:|
| create | supported | supported | supported | supported |
| status endpoint | unsupported | supported | supported | supported |
| wait for terminal status | supported | supported | supported | supported |
| cancel | supported | supported | unsupported | unsupported |
| redacted Hotato receipt export | supported | supported | supported | supported |
| delete matching local export | supported | supported | supported | supported |
| delete provider history | unsupported | unsupported | unsupported | unsupported |
| media / DTMF / hold / transfer | unobservable | unobservable | unobservable | unobservable |

Credentials come only from environment variables:

| Provider | Variables |
|---|---|
| Twilio | `TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN` |
| Vapi | `VAPI_API_KEY` |
| Retell | `RETELL_API_KEY` |

Receipts retain request field names and body byte counts. The provider response
uses a fixed lifecycle allowlist; every omitted response is bound only by its
canonical byte count and SHA-256. Free-form reason strings, identifiers,
telephone numbers, transcripts, recording URLs, messages, metadata,
authorization fields, keys, tokens, secrets, and passwords are never copied
from a provider response. Portable receipts and exports retain a provider-
domain-separated call-ID digest instead of the provider call ID. Export uses exclusive
file creation. Cleanup deletes only an export whose provider, call-ID digest,
and lifecycle receipt ID match the supplied in-memory handle. It never deletes
provider history.

## Media and signalling sidecars

Hotato composes established transport runtimes instead of implementing RTP, SIP, codec
negotiation, NAT traversal, or WebRTC congestion control.

### LiveKit SIP

`livekit_sip_contract(endpoint)` declares the evidence boundary for a LiveKit Server + LiveKit
SIP deployment. DTMF, hold, and transfer begin as `UNOBSERVABLE`; a session adapter promotes each
operation to `SUPPORTED` only when it returns the corresponding room/participant/SIP events and
delivered-audio digest. The expected evidence set is:

- room and participant events;
- SIP lifecycle/status events;
- delivered PCM or encoded-stream digest at the agent input boundary;
- per-leg event timestamps and trace correlation.

### Pipecat

`pipecat_media_contract(endpoint)` declares a streaming media boundary. The generic contract
requires pipeline events and a delivered-audio digest. DTMF, hold, and transfer are unsupported
until a concrete transport adapter defines and evidences them.

### SIPp

`SippSubprocessAdapter` executes a caller-supplied SIPp XML scenario for SIP-path and load probes.
SIPp owns SIP/RTP behavior. Hotato applies a static scenario policy, fixes the
subprocess argv and environment, and records receipts. The adapter does not
provide an operating-system sandbox.

```python
from hotato.call_runtime import SippSubprocessAdapter

receipt = SippSubprocessAdapter().run(
    {
        "target": "127.0.0.1:5060",
        "scenario_path": "fixtures/inbound.xml",
        "calls": 20,
        "rate_per_second": 2,
        "timeout_seconds": 180,
    },
    "artifacts/sipp-run-01",
)
```

The adapter:

- uses an argv array with `shell=False`;
- accepts no free-form extra arguments;
- validates the target and binds a regular, non-symlink XML file to its opened inode;
- parses XML with DTDs, entity declarations, external entities, and processing
  instructions disabled;
- defaults to a safe scenario profile that rejects every `<exec>` action,
  scenario file attribute, unsafe path attribute, and SIPp `[file ...]` host-file keyword;
- rejects `<setdest>` in the safe profile so scenario XML cannot bypass the
  resolved target and remote-IP policy;
- bounds calls, rate, scenario bytes, timeout, stdout, and stderr;
- supplies only `PATH`, `HOME`, and locale variables to the child;
- writes data artifacts with exclusive creation and mode `0600`, and stages
  the executable copy as mode `0700`;
- hashes the target instead of writing it into the receipt;
- reports process exit as `PASS` or `FAIL` without converting it into conversation quality.

Non-loopback SIP is default-deny. A remote spec must set `allow_remote=true`,
list the resolved destination IP in `remote_ip_allowlist`, cap the run with
`max_remote_calls`, and supply
`I_ACCEPT_REMOTE_SIP_SIDE_EFFECTS_AND_UNOBSERVABLE_EXTERNAL_COST`. The adapter
resolves once, passes the selected allowlisted IP to SIPp, and binds that
destination into the receipt. External SIP/carrier cost remains `null` and
`UNOBSERVABLE`. The built-in subprocess path stages and hashes the SIPp
executable; injected test runners report executable identity as unobservable.

The remote acknowledgement authorizes network side effects only. It does not
authorize scenario-level host commands, host-file reads, or destination
redirection.

### Trusted SIPp scenarios

SIPp XML is executable input. Its `<exec command="...">` action invokes a host
shell, while media actions and scenario keywords can read host files. Those
features are denied by default. A deployment that has reviewed the scenario
and accepts running it without an OS sandbox can opt into the unrestricted
scenario profile with a separate fixed acknowledgement:

```python
from hotato.call_runtime import SIPP_TRUSTED_SCENARIO_ACKNOWLEDGEMENT

receipt = SippSubprocessAdapter().run(
    {
        "target": "127.0.0.1:5060",
        "scenario_path": "fixtures/reviewed-media.xml",
        "trusted_scenario_acknowledgement": SIPP_TRUSTED_SCENARIO_ACKNOWLEDGEMENT,
    },
    "artifacts/sipp-reviewed-01",
)
```

The fixed phrase is
`I_ACCEPT_SIPP_SCENARIO_HOST_COMMAND_AND_FILE_ACCESS_WITHOUT_OS_SANDBOX`.
Trusted mode still refuses DTDs, entity declarations, external entities,
processing instructions, malformed XML, excessive nesting, and excessive
element count. Destination redirection remains denied in every profile so the
scenario cannot bypass the resolved and allowlisted target. The receipt records
`trusted_host_access`, `operator_attested`, and
`os_process_sandbox: ABSENT`; it does not claim that
the scenario was confined. Run trusted scenarios only in an independently
isolated container or virtual machine with the minimum filesystem and network
access required.

RTP playback, DTMF, hold, and transfer encoded inside XML remain `UNOBSERVABLE` until target-side
events establish delivery. Warm transfer is unsupported by the generic SIPp adapter because it
does not orchestrate a verified second live leg.

## Minimum acceptance evidence

Promote a session operation from `UNOBSERVABLE` to `SUPPORTED` only after a hermetic adapter test
and a live-path fixture produce all applicable artifacts:

- normalized hash-chained events with no sequence gaps;
- target-boundary audio digest and channel/sample format;
- provider or SIP lifecycle events;
- DTMF digit plus sender/receiver evidence;
- hold start/end plus media behavior during hold;
- transfer initiation, destination-leg connection, source-leg disposition, and warm/cold mode;
- bounded stdout, stderr, network destination inventory, and process exit;
- teardown evidence for every created room, call, process, and local artifact.

Keep live-path results scoped to the provider version, region, codec, adapter commit, and fixture
that produced them. No controller capability or provider documentation establishes a universal
carrier-path result.


================================================================================
FILE: docs/TRUST-GALLERY.md
================================================================================

# Trust gallery: eight recordings, eight verdicts

Eight named input conditions, each with the `hotato` output it produces.
Every block below is verbatim CLI output from hotato 1.16.0, offline. The rows
correspond to the [trust matrix](TRUST-MATRIX.md).

The clean and echo cases use the bundled examples
`01-hard-interruption.example.wav` and `07-echo-bleed.example.wav` (shipped in
`src/hotato/data/audio/`); the backchannel case uses `02-backchannel-mhm.example.wav`.
The four defect cases (silent caller, silent agent, swap, mono) are
deterministic synthetic two-channel fixtures pinned in `tests/test_trust.py`,
each isolating one input defect, so every verdict reproduces byte-for-byte
from the shipped builders.

---

## 1. Safe stereo: the good case

Clean two-channel export, caller on ch0, agent on ch1. `trust` clears it.

```text
$ hotato trust --stereo 01-hard-interruption.example.wav
hotato trust: 01-hard-interruption.example.wav
  recording: 6.0s, 16000 Hz, 2 channels
  caller (ch0): 2.46s speech, first at 2.39s, peak -4.4 dBFS
  agent  (ch1): 2.71s speech, first at 0.19s, peak -4.4 dBFS
  leading silence: 0.19s
  crosstalk: coherence 0.306 (low) at 0.5s lag
  scorability: separated tracks yes, caller activity yes, agent activity yes
  => eligible for scan
```

**Verdict:** `eligible for scan`, exit 0. Score it -- the only condition
where a full turn-taking verdict is trustworthy without a caveat.

---

## 2. Silent caller: nothing to measure against

The caller channel carries no speech (a dead leg, a wrong export, a muted line).

```text
$ hotato trust --stereo silent-caller.wav
hotato trust: silent-caller.wav
  recording: 6.0s, 16000 Hz, 2 channels
  caller (ch0): 0.00s speech, -, peak -120.0 dBFS
  agent  (ch1): 5.76s speech, first at 0.19s, peak -9.1 dBFS
  leading silence: 0.19s
  crosstalk: coherence 0.0 (low) at 0.0s lag
  scorability: separated tracks yes, caller activity no, agent activity yes
  => NOT SCORABLE: caller channel has no detected speech
     next step: verify channel mapping or export dual-channel again
```

**Verdict:** `NOT SCORABLE`, exit 2. A yield is a response to a caller --
with no caller, there is nothing to be late to. Hotato names the gap instead
of printing a hollow pass.

---

## 3. Silent agent: no floor to interrupt

The agent channel is empty. There is no floor for the caller to barge into.

```text
$ hotato trust --stereo silent-agent.wav
hotato trust: silent-agent.wav
  recording: 6.0s, 16000 Hz, 2 channels
  caller (ch0): 2.16s speech, first at 0.99s, peak -9.1 dBFS
  agent  (ch1): 0.00s speech, -, peak -120.0 dBFS
  leading silence: 0.99s
  crosstalk: coherence 0.0 (low) at 0.0s lag
  possible channel swap: channel 0 (mapped as caller) holds the floor 2.16s vs 0.0s on channel 1 (mapped as agent); an agent usually holds the floor longer, so the caller/agent channels may be reversed
  scorability: separated tracks yes, caller activity yes, agent activity no
  => NOT SCORABLE: agent channel has no detected speech
     next step: verify channel mapping or export dual-channel again
```

**Verdict:** `NOT SCORABLE`, exit 2. Hotato also flags a possible channel
swap here: a caller-only recording looks like a mis-mapped agent. Two
independent checks point at the same fix -- re-check the export.

---

## 4. Swapped channels: candidate-eligible, verdict pending confirmation

Both channels carry speech, but the long floor-holder is on the channel
mapped as the caller. `trust` keeps the input candidate-eligible and holds
the verdict until the mapping is confirmed.

```text
$ hotato trust --stereo swapped-warn.wav
hotato trust: swapped-warn.wav
  recording: 6.0s, 16000 Hz, 2 channels
  caller (ch0): 5.76s speech, first at 0.19s, peak -9.1 dBFS
  agent  (ch1): 0.66s speech, first at 1.99s, peak -9.1 dBFS
  leading silence: 0.19s
  crosstalk: coherence 0.31 (low) at 0.5s lag
  possible channel swap: channel 0 (mapped as caller) holds the floor 5.76s vs 0.66s on channel 1 (mapped as agent); an agent usually holds the floor longer, so the caller/agent channels may be reversed
  scorability: separated tracks yes, caller activity yes, agent activity yes
  => scan with caution: channel 0 (mapped as caller) holds the floor 5.76s vs 0.66s on channel 1 (mapped as agent); an agent usually holds the floor longer, so the caller/agent channels may be reversed
  [!] not verdict-eligible (scan mode): channel mapping unconfirmed: suspected swap/crosstalk; confirm mapping or supply provider metadata
```

**Verdict:** `scan with caution`, exit 0: candidate-eligible, verdict
eligibility waiting on your confirmation. A swap would invert every yield
into a hold, so Hotato surfaces candidates for review and holds the
turn-taking verdict until the mapping is confirmed. Set
`--caller-channel` / `--agent-channel`, or supply provider metadata, to
restore verdict eligibility.

---

## 5. Crosstalk / echo bleed: candidate-eligible, verdict pending confirmation

The caller channel carries a delayed copy of the agent's own audio. Coherence
pegs at 1.0 and the measured leakage is -9.1 dB.

```text
$ hotato trust --stereo 07-echo-bleed.example.wav
hotato trust: 07-echo-bleed.example.wav
  recording: 6.0s, 16000 Hz, 2 channels
  caller (ch0): 5.69s speech, first at 0.31s, peak -13.5 dBFS
  agent  (ch1): 5.81s speech, first at 0.19s, peak -4.4 dBFS
  leading silence: 0.19s
  crosstalk: coherence 1.0 (HIGH) at 0.12s lag
  leakage: -9.1 dB (agent->caller, HIGH)
  scorability: separated tracks yes, caller activity yes, agent activity yes
  => scan with caution: cross-channel leakage: agent audio on the caller channel sits at -9.1 dB, a consistent delayed copy (lag 0.12s); leaked audio at this level can be counted as the other party's activity, so a downstream timing measurement may be wrong even though the tracks are separated
  [!] not verdict-eligible (scan mode): channel mapping unconfirmed: suspected swap/crosstalk; confirm mapping or supply provider metadata
```

**Verdict:** `scan with caution`, exit 0: candidate-eligible, verdict
eligibility waiting on confirmation. The "caller" activity may be leaked
TTS, so Hotato still surfaces candidates for review (example 8's scan names
the specific moment) and holds the turn-taking verdict until the mapping is
confirmed.

---

## 6. Mono: refused by default, tiered under `--diarize`

A single channel. The caller and the agent are mixed into one track.

```text
$ hotato trust --stereo mono-mixed.wav
hotato trust: mono-mixed.wav
  recording: 6.0s, 16000 Hz, 1 channel
  scorability: separated tracks no, caller activity no, agent activity no
  => NOT SCORABLE: the recording has a single channel, so the caller and the agent cannot be told apart
     next step: export a dual-channel recording with the caller on one channel and the agent on the other
```

**Verdict:** `NOT SCORABLE`, exit 2 -- the gold-standard refusal. Two
opt-in escapes exist, both indicative only: `--allow-mono` on `capture` /
`pull` / `sweep` accepts a mono-only stack in degraded mode (talk-over
unattributable, no SLA gate), and the `--diarize` separation front-end
reports a `high` / `low` / `refuse` tier, with `hotato run --mono call.wav
--diarize` stamping the verdict `indicative_only` at `low`. Neither stands
in for dual-channel. See [TRUST-MATRIX.md](TRUST-MATRIX.md) and
[DIARIZE.md](DIARIZE.md).

---

## 7. Backchannel candidate: surfaced, yours to label

A scan of a call where the caller says "mhm" over the agent. `scan` lists the
overlaps as candidates and hands the intent decision to you.

```text
$ hotato scan --stereo 02-backchannel-mhm.example.wav --top 5
hotato scan: 02-backchannel-mhm.example.wav  (6.0s, 3 candidate moments)
Candidates are timing events. You decide the expected behavior; label with: hotato fixture create --onset <t> --expect yield|hold
  [ 1] t=2.09s  overlap_while_agent_talking  overlap=1.58s  agent did not go silent within 3.0s
  [ 2] t=3.19s  overlap_while_agent_talking  overlap=1.07s  agent did not go silent within 3.0s
  [ 3] t=4.29s  overlap_while_agent_talking  overlap=0.56s  agent did not go silent within 3.0s
```

**Verdict:** three candidates, exit 0. Whether "agent did not go silent" is
correct (held the floor through a backchannel, good) or a bug (talked over a
caller trying to take the floor) is **your** call. `scan` reports the
timing; you label `hold` or `yield` -- the intent behind a caller sound is
always yours to decide.

---

## 8. Noisy false positive: a candidate that is really the agent

A scan of the echo-bleed call. One candidate is an overlap fact; the
second is Hotato flagging that the "caller" activity is leaked TTS.

```text
$ hotato scan --stereo 07-echo-bleed.example.wav --top 5
hotato scan: 07-echo-bleed.example.wav  (6.0s, 2 candidate moments)
Candidates are timing events. You decide the expected behavior; label with: hotato fixture create --onset <t> --expect yield|hold
  [ 1] t=0.31s  overlap_while_agent_talking  overlap=3.00s  agent did not go silent within 3.0s
  [ 2] t=0.31s  echo_correlated_activity     WARNING likely agent echo: coherence=1.00 at lag 0.12s  (caller channel looks like leaked TTS; a yield here may be the agent hearing itself)
```

**Verdict:** two candidates, exit 0. Candidate 1 looks like a bad
talk-over. Candidate 2 is Hotato telling you the overlap is probably the
agent hearing its own audio, not the caller interrupting. This is candidate
discovery's failure mode: the net is wide, and Hotato labels a
likely-spurious candidate instead of hiding it. You label it `hold` (or fix
the echo capture) and it never becomes a fixture.

---

## How to read the gallery

The eight cases fall into three buckets:

- **Score it** (1): clean stereo, verdict-eligible, full confidence.
- **Candidate-eligible, verdict pending confirmation** (4, 5): swap and
  crosstalk keep `scan` surfacing candidates (exit 0), with the turn-taking
  verdict waiting on your confirmation of the channel mapping. Example 8 is
  that same echo call under `scan`, naming the specific leaked moment.
- **Refuse** (2, 3, 6): silent caller, silent agent, and mono are not
  scorable; Hotato names the reason and the next step, and exits 2.

Case 7 is the everyday case: a candidate with no defect, waiting for your
label. The goal: any Hotato verdict you act on has already survived this
gauntlet.


================================================================================
FILE: docs/TRUST-MATRIX.md
================================================================================

# Trust matrix: what Hotato does for each input condition

Before Hotato scores a recording, `hotato trust` checks the audio and
decides whether a score would mean anything -- catching a bad export before
it becomes a confident-looking but hollow verdict. This page is the exact
contract: input condition on the left, Hotato's behavior on the right. It
covers input health only; a turn-taking verdict is `scan`'s and `run`'s job.

Worked examples with CLI output for every row are in
[TRUST-GALLERY.md](TRUST-GALLERY.md).

## The contract

| Input condition | `trust` says | Exit | Downstream scoring |
|---|---|---|---|
| Clean dual-channel (stereo) | `eligible for scan` | 0 | **Full scoring.** `scan`, `run`, `compare`, `verify` all run normally. |
| Silent caller channel | `NOT SCORABLE: caller channel has no detected speech` | 2 | **Refused.** No caller to measure a yield against. |
| Silent agent channel | `NOT SCORABLE: agent channel has no detected speech` | 2 | **Refused.** No agent floor to interrupt. |
| Channel swap risk | `eligible for scan` **plus** `possible channel swap` warning | 0 | Scores -- but confirm the mapping first (`--caller-channel` / `--agent-channel`); a swap silently inverts every yield/hold. |
| High crosstalk / echo bleed | `eligible for scan` **plus** high `crosstalk: coherence` warning | 0 | Scores at **lower confidence.** `scan` tags the moment `echo_correlated_activity`; a "yield" there may be the agent hearing itself. |
| Mixed mono (single channel) | `NOT SCORABLE: single channel, caller and agent cannot be told apart` | 2 | **Refused by default.** Export dual-channel, or use one of the two opt-in escapes below. |
| Mono, opt-in `--allow-mono` | Accepted in **degraded mode**, results **indicative only** | 0 | Used by `capture` / `pull` / `sweep` for a mono-only stack. Talk-over can't be attributed, so no SLA gate fires; dual-channel stays the gold reference. |
| Mono, opt-in `--diarize` | Separability **tier**: `high` / `low` / `refuse` | 0 (high/low), 2 (refuse) | The separation front-end. `hotato run --mono call.wav --diarize` scores it; `low` stamps the verdict `indicative_only`. Dual-channel stays the gold reference. |
| Short backchannel ("mhm") overlap | Outside `trust`'s scope; `scan` lists it as a **candidate** | 0 | **Candidate only, human labels.** `trust`/`scan` report the timing; you always label `yield` vs `hold`. |
| Noisy / false-positive candidate | `trust` warns (clipping, crosstalk, hot capture); `scan` still lists the candidate | 0 | Surfaced **with** the warning, as a candidate for you to inspect and label `hold`. |

**Reading the two axes.** `trust` answers one question: is this audio good
enough to score? The exit code is the machine contract: `0` means eligible
for scan (maybe with warnings), `2` means not scorable, with the reason and
the next step. Warnings (swap, crosstalk, clipping, leading silence) inform
the read; only the three hard refusals -- mono, identical channels, a
silent required channel -- change scorability.

## Why refuse instead of guessing

A less careful tool would still print a number here. A mono file "scored"
by guessing which speaker is which produces a verdict indistinguishable
from a properly-scored one -- and it is worthless. A swapped-channel file
scored without the warning inverts caller and agent, so "the agent yielded"
becomes "the caller yielded" with no visible sign. Hotato treats these as
input defects, reports them, and stops: a green or red build always means
something.

## Two ways past a mono refusal

Mono is refused by default: one channel cannot separate the two parties.
Two opt-in escapes exist, both indicative only.

**1. `--allow-mono` (degraded mode).** On `capture`, `pull`, and `sweep`,
the `--allow-mono` flag (or `HOTATO_ALLOW_MONO=1`) accepts a mono-only
recording from a mono stack. Nothing is separated, so talk-over can't be
attributed to caller or agent: the result is explicitly indicative and no
SLA gate fires. Use it when a stack only exports a mono mix and a rough
signal still beats none.

**2. `--diarize` (separation front-end).** The opt-in `[diarize]` extra
(`pip install 'hotato[diarize]'`) runs a local diarizer and reports a
confidence tier before you score:

- **high**: confidently separable. `hotato run --mono call.wav --diarize`
  gives a diarized-mono verdict. Exit 0.
- **low**: separable but only indicative (voices close, overlap elevated).
  The verdict is stamped `indicative_only`; no SLA gate fires. Exit 0.
- **refuse**: not confidently two clean parties. Not scorable, exit 2;
  record dual-channel.

The dual-channel path stays the gold reference -- neither a degraded-mono
nor a diarized-mono verdict is promoted to equivalence with it. Full
front-end: [DIARIZE.md](DIARIZE.md).

## Composing the gate

Because `trust` exits `2` on any unscorable input, it drops straight into a
shell gate ahead of a scan or a scheduled sweep:

```bash
hotato trust --stereo call.wav && hotato scan --stereo call.wav
```

For agents, `hotato trust --stereo call.wav --format json` emits one
machine-parseable report. Branch on `scorable`; on a defect, read
`not_scorable_reason` and `next_step`. Details: [TRUST.md](TRUST.md).


================================================================================
FILE: docs/VALIDATION.md
================================================================================

# What Hotato validates

A turn handoff is not one number. Hotato is validated on **three separate
jobs**, each with its own question and its own reported output. Judge each
on its own terms when you decide whether to trust it.

Everything below runs offline against recordings you control. Every threshold
is exposed and every frame is inspectable (`hotato run --dump-frames`).

---

## Job 1: timing reproducibility

**The question:** given the same recording and the same reference config,
does Hotato produce the same timing measurements every run? (Deterministic
for a fixed hotato version; byte-identical re-runs are verified in CI on
Linux x86_64, Python 3.10-3.12 -- `.github/workflows/tests.yml`, job
`pytest`. Deterministic scoring produces the same digest on Ubuntu, macOS,
and Windows in CI: jobs `portability` and `determinism` run the double-run
check on each OS and compare digests across all three.)

**What is reported.** Per scored event: `did_yield` (true/false),
`seconds_to_yield`, and `talk_over_sec`, plus the exact thresholds used
(`max_talk_over`, `max_time_to_yield`) and the frame grid behind them. No
learned weights, no sampling, no RNG: the energy VAD and the reference framing
are deterministic, so the numbers are byte-stable.

**How to check it.** Score the same file twice and diff the output.

```text
$ hotato run --stereo 01-hard-interruption.example.wav --expect yield
hotato [single] stack=generic offline=True
  1/1 events pass  (failed=0)
  [PASS] 01-hard-interruption.example.wav: did_yield=True seconds_to_yield=0.51s talk_over=0.51s
  exit_code=0

$ hotato run --stereo 01-hard-interruption.example.wav --expect yield
  [PASS] 01-hard-interruption.example.wav: did_yield=True seconds_to_yield=0.51s talk_over=0.51s
```

`seconds_to_yield=0.51s` and `talk_over=0.51s` are identical across runs. This
is the property a regression test needs: under a fixed hotato version and the
same pinned audio, channel map, onset, label, and scoring config, a changed
result means one of those pinned inputs changed, not that the scorer drifted.

**What this job establishes.** That the measurement is stable and re-derivable
by hand from [`METHODOLOGY.md`](../METHODOLOGY.md). It is a claim about
stability, not that 0.51s is the absolute "true" yield latency or that the
reference thresholds fit your product.

---

## Job 2: candidate-discovery usefulness

**The question:** when Hotato scans a whole recording, does it surface the
moments a human reviewer would want to look at, ranked by salience?

**What is reported.** A ranked list of candidate turn-taking moments as **timing
facts only**: overlap onsets (caller became active while the agent was talking,
with the overlap length and whether the agent went silent), agent starts during
caller activity, and long response gaps. Each candidate is a timestamp and a
measurement -- the timing fact, reported without a verdict or an intent label.

```text
$ hotato scan --stereo 02-backchannel-mhm.example.wav --top 5
hotato scan: 02-backchannel-mhm.example.wav  (6.0s, 3 candidate moments)
Candidates are timing events. You decide the expected behavior; label with: hotato fixture create --onset <t> --expect yield|hold
  [ 1] t=2.09s  overlap_while_agent_talking  overlap=1.58s  agent did not go silent within 3.0s
  [ 2] t=3.19s  overlap_while_agent_talking  overlap=1.07s  agent did not go silent within 3.0s
  [ 3] t=4.29s  overlap_while_agent_talking  overlap=0.56s  agent did not go silent within 3.0s
```

The usefulness bar is **recall of human-notable moments at a workable
candidate count**, not precision against a ground-truth intent label (no
such label exists at scan time, by design). A candidate that turns out to be
a harmless backchannel is simply one you label `hold` and move on. The
validation artifact is the [trust gallery](TRUST-GALLERY.md): it includes a
deliberate false positive, showing what an unhelpful candidate looks like
and why Hotato still surfaces it.

**What this job establishes.** That scan widens the net for you to make the
call. It is a claim about recall, not that every candidate is a bug or that a
quiet region is clean.

---

## Job 3: contract verification

**The question:** once you have labelled a moment's expected behavior
(`yield` = stop for the caller, `hold` = keep the floor through a backchannel),
does Hotato's PASS/FAIL verdict agree with that label on the audio, against an
explicit, portable, CI-enforced policy?

Today this job runs on a fixture (`hotato fixture create` / `hotato run`): a
labelled recording plus an explicit threshold policy, scored the same way on
every CI machine. Deterministic scoring produces the same digest on Ubuntu,
macOS, and Windows (jobs `portability` and `determinism`); the broader test
suite is verified on Linux (see Job 1). The portable
contract bundle (`hotato contract create` / `hotato contract verify` --
audio, timing evidence, trace evidence, label, policy, and a CI command, all
in one artifact) carries this job forward; only the artifact
changes, not the verdict's shape.

**What is reported.** Per fixture: the verdict (`PASS`/`FAIL`), the measured
signals behind it, and the named fix class when the failure maps cleanly to a
config family. Agreement is checked against **your** label, not against an
opaque key.

```text
$ hotato demo --no-open --format text
hotato demo: bundled calls an agent fails
hotato [suite] stack=generic offline=True
  0/2 events pass  (failed=2)
  [FAIL] fd-01-missed-interruption: did_yield=False seconds_to_yield=n/a talk_over=2.65s
         fix[config]: Missed interruption: the agent kept talking over the caller
  [FAIL] fd-02-backchannel-yielded: did_yield=True seconds_to_yield=0.34s talk_over=0.32s
         fix[engagement-control]: False barge-in: a backchannel was treated as a bid for the floor
  note: no single sensitivity threshold satisfies this battery
  exit_code=1
```

Both labelled failures are caught, and the battery-level note reports the
disagreement plainly: a missed interruption and a false stop failing in the
same run means no single threshold satisfies both. Reporting that
disagreement, instead of inventing a fix, is part of the validated behavior.

**What this job establishes.** That the verdict follows the audio and the
label consistently. It is a claim about consistency, not that the label was
correct (you own the label) or that a passing fixture means the agent is good
in general.

---

## What Hotato measures, and where it stops

Read this as the scope of the claim, stated once, plainly.

- **Timing, not semantic intent.** Hotato measures timing; whether a caller
  sound meant "stop" or "mhm, go on" is a label you supply.
- **A likely layer, not certainty.** A slow yield can be TTS buffering,
  transport, or VAD. `diagnose` names a likely layer and stays
  `unknown_root_cause` when one recording can't separate them. A voice trace
  (once the trace layer ships) narrows the candidates further, short of
  proof.
- **Timing, not task success.** Whether the call booked the appointment,
  resolved the ticket, or satisfied the caller is a QA platform's job (see
  [COMPARE.md](COMPARE.md)).
- **A demonstration, not a vendor ranking.** Hotato scores calls, not
  platforms. A provider-default example demonstrates the threshold funnel on
  one assistant, one config, one date, one scripted caller.
- **Timing, not tone.** Sentiment, satisfaction, and CSAT sit outside what
  Hotato measures.
- **Reproducible timing measurements, with the method exposed.** The three
  jobs above are the whole claim -- deliberately no headline percentage.

The validation plan for the launch battery (external testers, consented
fixtures, before/after) lives in
[docs/evidence/validation-plan.md](evidence/validation-plan.md).


================================================================================
FILE: src/hotato/schema/counterexample-oracle.v1.json
================================================================================

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://hotato.dev/schema/counterexample-oracle.v1.json",
  "title": "Hotato deterministic counterexample oracle",
  "type": "object",
  "additionalProperties": false,
  "required": ["kind", "version", "authority", "ci_gate_eligible", "target", "observation_scope"],
  "properties": {
    "kind": {"const": "hotato.counterexample-oracle.v1"},
    "version": {"const": 1},
    "authority": {"const": "deterministic"},
    "ci_gate_eligible": {"const": true},
    "target": {"$ref": "#/definitions/target"},
    "observation_scope": {"$ref": "#/definitions/observation_scope"}
  },
  "definitions": {
    "digest": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"},
    "nonnegative_integer": {"type": "integer", "minimum": 0},
    "dimension": {
      "type": ["string", "null"],
      "enum": ["outcome", "policy", "conversation", "speech", "reliability", null]
    },
    "assertion_kind": {
      "enum": [
        "phrase", "pii", "policy", "tool_call", "outcome", "tool_result",
        "tool_error", "state", "state_change", "handoff",
        "termination", "latency", "entity_accuracy", "sequence", "count"
      ]
    },
    "failure_atom": {
      "$comment": "Closed payload-free branch identity; kind coupling is enforced by the target schema.",
      "oneOf": [
        {
          "type": "object", "additionalProperties": false,
          "required": ["code"],
          "properties": {
            "code": {"enum": [
              "forbidden-match", "no-qualifying-turns", "required-match-missing",
              "tool-missing", "tool-arguments-missing", "tool-count-below",
              "tool-count-above", "never-before-boundary-missing",
              "never-before-order-violation",
              "no-predicate-met", "result-missing", "unexpected-tool-error",
              "tool-error-missing", "tool-error-pattern-mismatch", "state-record-missing",
              "unexpected-handoff", "handoff-missing", "handoff-target-mismatch",
              "unexpected-termination", "termination-missing",
              "latency-declared-threshold-exceeded", "latency-default-threshold-exceeded",
              "count-below", "count-above"
            ]}
          }
        },
        {
          "type": "object", "additionalProperties": false,
          "required": ["code", "index"],
          "properties": {
            "code": {"enum": ["order-step-absent", "order-step-out-of-order", "predicate-unmet", "sequence-step-absent", "sequence-step-out-of-order"]},
            "index": {"$ref": "#/definitions/nonnegative_integer"}
          }
        },
        {
          "type": "object", "additionalProperties": false,
          "required": ["code", "detector"],
          "properties": {
            "code": {"const": "pii-detected"},
            "detector": {"enum": ["ssn", "card_luhn", "email", "phone"]}
          }
        },
        {
          "type": "object", "additionalProperties": false,
          "required": ["code", "rule", "type"],
          "properties": {
            "code": {"const": "policy-violation"},
            "rule": {"type": "string", "minLength": 1},
            "type": {"enum": ["banned", "required_disclosure_missing"]}
          }
        },
        {
          "type": "object", "additionalProperties": false,
          "required": ["code", "field"],
          "properties": {
            "code": {"enum": ["state-field-missing", "state-field-value-mismatch", "before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged", "termination-attribute-missing", "termination-attribute-value-mismatch"]},
            "field": {"type": "string", "minLength": 1}
          }
        },
        {
          "type": "object", "additionalProperties": false,
          "required": ["code", "key"],
          "properties": {
            "code": {"enum": ["entity-missing", "entity-value-mismatch", "tool-argument-field-missing", "tool-argument-value-mismatch", "result-field-missing", "result-field-value-mismatch"]},
            "key": {"type": "string", "minLength": 1}
          }
        }
      ]
    },
    "target_kind_atom_coupling": {
      "allOf": [
        {
          "if": {"properties": {"kind": {"const": "phrase"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_phrase"}
        },
        {
          "if": {"properties": {"kind": {"const": "pii"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_pii"}
        },
        {
          "if": {"properties": {"kind": {"const": "policy"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_policy"}
        },
        {
          "if": {"properties": {"kind": {"const": "tool_call"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_tool_call"}
        },
        {
          "if": {"properties": {"kind": {"const": "outcome"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_outcome"}
        },
        {
          "if": {"properties": {"kind": {"const": "tool_result"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_tool_result"}
        },
        {
          "if": {"properties": {"kind": {"const": "tool_error"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_tool_error"}
        },
        {
          "if": {"properties": {"kind": {"const": "state"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_state"}
        },
        {
          "if": {"properties": {"kind": {"const": "state_change"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_state_change"}
        },
        {
          "if": {"properties": {"kind": {"const": "handoff"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_handoff"}
        },
        {
          "if": {"properties": {"kind": {"const": "termination"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_termination"}
        },
        {
          "if": {"properties": {"kind": {"const": "latency"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_latency"}
        },
        {
          "if": {"properties": {"kind": {"const": "entity_accuracy"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_entity_accuracy"}
        },
        {
          "if": {"properties": {"kind": {"const": "sequence"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_sequence"}
        },
        {
          "if": {"properties": {"kind": {"const": "count"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_count"}
        }
      ]
    },
    "target_atom_codes_phrase": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["forbidden-match", "no-qualifying-turns", "required-match-missing"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["forbidden-match", "no-qualifying-turns", "required-match-missing"]}}}}
      }
    },
    "target_atom_codes_pii": {
      "properties": {
        "failure_atom": {"properties": {"code": {"const": "pii-detected"}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"const": "pii-detected"}}}}
      }
    },
    "target_atom_codes_policy": {
      "properties": {
        "failure_atom": {"properties": {"code": {"const": "policy-violation"}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"const": "policy-violation"}}}}
      }
    },
    "target_atom_codes_tool_call": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["tool-missing", "tool-arguments-missing", "tool-argument-field-missing", "tool-argument-value-mismatch", "tool-count-below", "tool-count-above", "order-step-absent", "order-step-out-of-order", "never-before-boundary-missing", "never-before-order-violation"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["tool-missing", "tool-arguments-missing", "tool-argument-field-missing", "tool-argument-value-mismatch", "tool-count-below", "tool-count-above", "order-step-absent", "order-step-out-of-order", "never-before-boundary-missing", "never-before-order-violation"]}}}}
      }
    },
    "target_atom_codes_outcome": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["predicate-unmet", "no-predicate-met"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["predicate-unmet", "no-predicate-met"]}}}}
      }
    },
    "target_atom_codes_tool_result": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["tool-missing", "result-missing", "result-field-missing", "result-field-value-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["tool-missing", "result-missing", "result-field-missing", "result-field-value-mismatch"]}}}}
      }
    },
    "target_atom_codes_tool_error": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["tool-missing", "unexpected-tool-error", "tool-error-missing", "tool-error-pattern-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["tool-missing", "unexpected-tool-error", "tool-error-missing", "tool-error-pattern-mismatch"]}}}}
      }
    },
    "target_atom_codes_state": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["state-record-missing", "state-field-missing", "state-field-value-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["state-record-missing", "state-field-missing", "state-field-value-mismatch"]}}}}
      }
    },
    "target_atom_codes_state_change": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged"]}}}}
      }
    },
    "target_atom_codes_handoff": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["unexpected-handoff", "handoff-missing", "handoff-target-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["unexpected-handoff", "handoff-missing", "handoff-target-mismatch"]}}}}
      }
    },
    "target_atom_codes_termination": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["unexpected-termination", "termination-missing", "termination-attribute-missing", "termination-attribute-value-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["unexpected-termination", "termination-missing", "termination-attribute-missing", "termination-attribute-value-mismatch"]}}}}
      }
    },
    "target_atom_codes_latency": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["latency-declared-threshold-exceeded", "latency-default-threshold-exceeded"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["latency-declared-threshold-exceeded", "latency-default-threshold-exceeded"]}}}}
      }
    },
    "target_atom_codes_entity_accuracy": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["entity-missing", "entity-value-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["entity-missing", "entity-value-mismatch"]}}}}
      }
    },
    "target_atom_codes_sequence": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["sequence-step-absent", "sequence-step-out-of-order"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["sequence-step-absent", "sequence-step-out-of-order"]}}}}
      }
    },
    "target_atom_codes_count": {
      "properties": {
        "failure_atom": {"properties": {"code": {"enum": ["count-below", "count-above"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["count-below", "count-above"]}}}}
      }
    },
    "target": {
      "$comment": "Runtime validation also requires canonical UTF-8 atom ordering, de-duplication, selected atom first, and a matching fingerprint.",
      "type": "object", "additionalProperties": false,
      "required": [
        "test_id", "assertion_digest", "assertion_id", "kind", "dimension",
        "authority", "required_status", "failure_atom", "source_failure_atoms",
        "fingerprint"
      ],
      "properties": {
        "test_id": {"type": "string", "minLength": 1},
        "assertion_digest": {"$ref": "#/definitions/digest"},
        "assertion_id": {"type": "string", "minLength": 1},
        "kind": {"$ref": "#/definitions/assertion_kind"},
        "dimension": {"$ref": "#/definitions/dimension"},
        "authority": {"const": "deterministic"}, "required_status": {"const": "FAIL"},
        "failure_atom": {"$ref": "#/definitions/failure_atom"},
        "source_failure_atoms": {
          "type": "array", "minItems": 1, "uniqueItems": true,
          "items": {"$ref": "#/definitions/failure_atom"}
        },
        "fingerprint": {"$ref": "#/definitions/digest"}
      },
      "allOf": [{"$ref": "#/definitions/target_kind_atom_coupling"}]
    },
    "observation_scope": {
      "type": "object", "additionalProperties": false,
      "required": ["frozen_components", "rule", "minimum_caller_turns", "transforms"],
      "properties": {
        "frozen_components": {
          "type": "array", "uniqueItems": true,
          "items": {"enum": ["script", "tools", "state", "handoff", "termination"]}
        },
        "rule": {
          "const": "candidate is a complete schema-valid scripted session and must preserve the source-selected structured failure branch"
        },
        "minimum_caller_turns": {"const": 1},
        "transforms": {"const": "deletion-only"}
      }
    }
  }
}


================================================================================
FILE: src/hotato/schema/reduction-certificate.v1.json
================================================================================

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://hotato.dev/schema/reduction-certificate.v1.json",
  "title": "Hotato counterexample reduction certificate",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "kind", "version", "algorithm", "reducer_set", "source_scenario_digest",
    "final_scenario_digest", "source_test_digest", "final_test_digest",
    "failure_fingerprint", "accepted_steps", "journal_sha256", "budget",
    "candidate_evaluations", "cache_hits", "termination"
  ],
  "properties": {
    "kind": {"const": "hotato.reduction-certificate.v1"},
    "version": {"const": 1},
    "algorithm": {"const": "hierarchical-ddmin"},
    "reducer_set": {"const": "hotato.reducers.v1"},
    "source_scenario_digest": {"$ref": "#/definitions/digest"},
    "final_scenario_digest": {"$ref": "#/definitions/digest"},
    "source_test_digest": {"$ref": "#/definitions/digest"},
    "final_test_digest": {"$ref": "#/definitions/digest"},
    "failure_fingerprint": {"$ref": "#/definitions/digest"},
    "accepted_steps": {
      "type": "array", "maxItems": 512,
      "items": {"$ref": "#/definitions/accepted_step"}
    },
    "journal_sha256": {"$ref": "#/definitions/digest"},
    "budget": {"type": "integer", "minimum": 1, "maximum": 100000},
    "candidate_evaluations": {"type": "integer", "minimum": 0, "maximum": 100000},
    "cache_hits": {"type": "integer", "minimum": 0},
    "termination": {"enum": ["one_minimal", "budget_exhausted"]}
  },
  "definitions": {
    "digest": {
      "type": "string",
      "pattern": "^sha256:[0-9a-f]{64}$"
    },
    "raw_digest": {
      "type": "string",
      "pattern": "^[0-9a-f]{64}$"
    },
    "path_segment": {
      "oneOf": [
        {"type": "string"},
        {"type": "integer", "minimum": 0}
      ]
    },
    "deletion_operation": {
      "type": "object",
      "additionalProperties": false,
      "required": ["op", "path", "removed_digest"],
      "properties": {
        "op": {"const": "remove"},
        "path": {
          "type": "array", "minItems": 1, "maxItems": 96,
          "items": {"$ref": "#/definitions/path_segment"}
        },
        "removed_digest": {"$ref": "#/definitions/digest"}
      }
    },
    "deletion_transform": {
      "type": "object",
      "additionalProperties": false,
      "required": ["kind", "operations"],
      "properties": {
        "kind": {"const": "hotato.delete-only.v1"},
        "operations": {
          "type": "array", "minItems": 1, "maxItems": 10000,
          "uniqueItems": true,
          "items": {"$ref": "#/definitions/deletion_operation"}
        }
      }
    },
    "remove_field": {
      "type": "object",
      "additionalProperties": false,
      "required": ["kind", "phase", "path"],
      "properties": {
        "kind": {"const": "remove-field"},
        "phase": {"type": "string", "minLength": 1},
        "path": {"type": "string", "minLength": 1}
      }
    },
    "remove_path_set": {
      "type": "object",
      "additionalProperties": false,
      "required": ["kind", "phase", "paths"],
      "properties": {
        "kind": {"const": "remove-path-set"},
        "phase": {"type": "string", "minLength": 1},
        "paths": {
          "type": "array", "minItems": 1, "maxItems": 10000,
          "uniqueItems": true,
          "items": {"type": "string", "minLength": 1}
        }
      }
    },
    "component": {
      "enum": [
        "top-level-optional", "goal-optional", "caller-optional",
        "variation_matrix", "facts", "environment", "interruptions", "behavior",
        "script", "script-field", "tools", "tool-field", "handoff-field", "handoff",
        "termination-field", "termination", "state", "agent-mock-optional", "agent_mock"
      ]
    },
    "remove_single_unit": {
      "type": "object",
      "additionalProperties": false,
      "required": ["kind", "phase", "path", "component"],
      "properties": {
        "kind": {"const": "remove-single-unit"},
        "phase": {"enum": ["one-minimality", "minimality-proof-restart"]},
        "path": {"type": "string", "minLength": 1},
        "component": {"$ref": "#/definitions/component"}
      }
    },
    "reducer_operation": {
      "oneOf": [
        {"$ref": "#/definitions/remove_field"},
        {"$ref": "#/definitions/remove_path_set"},
        {"$ref": "#/definitions/remove_single_unit"}
      ]
    },
    "accepted_step": {
      "type": "object",
      "additionalProperties": false,
      "required": [
        "step", "parent_digest", "child_digest", "operation", "transform",
        "oracle_result_digest", "failure_atom_digest"
      ],
      "properties": {
        "step": {"type": "integer", "minimum": 1, "maximum": 512},
        "parent_digest": {"$ref": "#/definitions/raw_digest"},
        "child_digest": {"$ref": "#/definitions/raw_digest"},
        "operation": {"$ref": "#/definitions/reducer_operation"},
        "transform": {"$ref": "#/definitions/deletion_transform"},
        "oracle_result_digest": {"$ref": "#/definitions/digest"},
        "failure_atom_digest": {"$ref": "#/definitions/digest"}
      }
    }
  }
}


================================================================================
FILE: src/hotato/schema/counterexample.v1.json
================================================================================

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://hotato.dev/schema/counterexample.v1.json",
  "title": "Hotato proof-preserving counterexample capsule",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "kind", "version", "counterexample_id", "source", "target",
    "reduction", "minimality", "preservation", "privacy", "provenance"
  ],
  "properties": {
    "kind": {"const": "hotato.counterexample.v1"},
    "version": {"const": 1},
    "counterexample_id": {"$ref": "#/definitions/digest"},
    "source": {},
    "target": {},
    "oracle": {"$ref": "#/definitions/oracle_reference"},
    "artifacts": {"$ref": "#/definitions/artifacts"},
    "artifact_digests": {"$ref": "#/definitions/artifact_digests"},
    "reduction": {"$ref": "#/definitions/reduction"},
    "minimality": {},
    "preservation": {"$ref": "#/definitions/preservation"},
    "privacy": {},
    "provenance": {}
  },
  "allOf": [
    {
      "oneOf": [
        {
          "required": ["oracle", "artifacts", "artifact_digests"],
          "properties": {
            "source": {"$ref": "#/definitions/private_source"},
            "target": {"$ref": "#/definitions/private_target"},
            "minimality": {"$ref": "#/definitions/private_minimality"},
            "privacy": {"$ref": "#/definitions/private_privacy"},
            "provenance": {"$ref": "#/definitions/private_provenance"}
          }
        },
        {
          "not": {
            "anyOf": [
              {"required": ["oracle"]}, {"required": ["artifacts"]},
              {"required": ["artifact_digests"]}
            ]
          },
          "properties": {
            "source": {"$ref": "#/definitions/share_source"},
            "target": {"$ref": "#/definitions/share_target"},
            "minimality": {"$ref": "#/definitions/share_minimality"},
            "privacy": {"$ref": "#/definitions/share_privacy"},
            "provenance": {"$ref": "#/definitions/share_provenance"}
          }
        }
      ]
    },
    {
      "oneOf": [
        {
          "properties": {
            "reduction": {"properties": {"termination": {"const": "one_minimal"}}},
            "minimality": {"properties": {"status": {"const": "one_minimal"}}}
          }
        },
        {
          "properties": {
            "reduction": {"properties": {"termination": {"const": "budget_exhausted"}}},
            "minimality": {"properties": {"status": {"const": "budget_exhausted"}}}
          }
        }
      ]
    }
  ],
  "definitions": {
    "digest": {
      "type": "string",
      "pattern": "^sha256:[0-9a-f]{64}$"
    },
    "raw_digest": {
      "type": "string",
      "pattern": "^[0-9a-f]{64}$"
    },
    "dimension": {
      "type": ["string", "null"],
      "enum": ["outcome", "policy", "conversation", "speech", "reliability", null]
    },
    "assertion_kind": {
      "enum": [
        "phrase", "pii", "policy", "tool_call", "outcome", "tool_result",
        "tool_error", "state", "state_change", "handoff",
        "termination", "latency", "entity_accuracy", "sequence", "count"
      ]
    },
    "nonnegative_integer": {"type": "integer", "minimum": 0},
    "failure_atom": {
      "$comment": "Closed payload-free branch identity; kind coupling is enforced by the target schema.",
      "oneOf": [
        {
          "type": "object", "additionalProperties": false,
          "required": ["code"],
          "properties": {
            "code": {"enum": [
              "forbidden-match", "no-qualifying-turns", "required-match-missing",
              "tool-missing", "tool-arguments-missing", "tool-count-below",
              "tool-count-above", "never-before-boundary-missing",
              "never-before-order-violation",
              "no-predicate-met", "result-missing", "unexpected-tool-error",
              "tool-error-missing", "tool-error-pattern-mismatch", "state-record-missing",
              "unexpected-handoff", "handoff-missing", "handoff-target-mismatch",
              "unexpected-termination", "termination-missing",
              "latency-declared-threshold-exceeded", "latency-default-threshold-exceeded",
              "count-below", "count-above"
            ]}
          }
        },
        {
          "type": "object", "additionalProperties": false,
          "required": ["code", "index"],
          "properties": {
            "code": {"enum": ["order-step-absent", "order-step-out-of-order", "predicate-unmet", "sequence-step-absent", "sequence-step-out-of-order"]},
            "index": {"$ref": "#/definitions/nonnegative_integer"}
          }
        },
        {
          "type": "object", "additionalProperties": false,
          "required": ["code", "detector"],
          "properties": {
            "code": {"const": "pii-detected"},
            "detector": {"enum": ["ssn", "card_luhn", "email", "phone"]}
          }
        },
        {
          "type": "object", "additionalProperties": false,
          "required": ["code", "rule", "type"],
          "properties": {
            "code": {"const": "policy-violation"},
            "rule": {"type": "string", "minLength": 1},
            "type": {"enum": ["banned", "required_disclosure_missing"]}
          }
        },
        {
          "type": "object", "additionalProperties": false,
          "required": ["code", "field"],
          "properties": {
            "code": {"enum": ["state-field-missing", "state-field-value-mismatch", "before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged", "termination-attribute-missing", "termination-attribute-value-mismatch"]},
            "field": {"type": "string", "minLength": 1}
          }
        },
        {
          "type": "object", "additionalProperties": false,
          "required": ["code", "key"],
          "properties": {
            "code": {"enum": ["entity-missing", "entity-value-mismatch", "tool-argument-field-missing", "tool-argument-value-mismatch", "result-field-missing", "result-field-value-mismatch"]},
            "key": {"type": "string", "minLength": 1}
          }
        }
      ]
    },
    "target_kind_atom_coupling": {
      "allOf": [
        {
          "if": {"properties": {"kind": {"const": "phrase"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_phrase"}
        },
        {
          "if": {"properties": {"kind": {"const": "pii"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_pii"}
        },
        {
          "if": {"properties": {"kind": {"const": "policy"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_policy"}
        },
        {
          "if": {"properties": {"kind": {"const": "tool_call"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_tool_call"}
        },
        {
          "if": {"properties": {"kind": {"const": "outcome"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_outcome"}
        },
        {
          "if": {"properties": {"kind": {"const": "tool_result"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_tool_result"}
        },
        {
          "if": {"properties": {"kind": {"const": "tool_error"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_tool_error"}
        },
        {
          "if": {"properties": {"kind": {"const": "state"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_state"}
        },
        {
          "if": {"properties": {"kind": {"const": "state_change"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_state_change"}
        },
        {
          "if": {"properties": {"kind": {"const": "handoff"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_handoff"}
        },
        {
          "if": {"properties": {"kind": {"const": "termination"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_termination"}
        },
        {
          "if": {"properties": {"kind": {"const": "latency"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_latency"}
        },
        {
          "if": {"properties": {"kind": {"const": "entity_accuracy"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_entity_accuracy"}
        },
        {
          "if": {"properties": {"kind": {"const": "sequence"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_sequence"}
        },
        {
          "if": {"properties": {"kind": {"const": "count"}}, "required": ["kind"]},
          "then": {"$ref": "#/definitions/target_atom_codes_count"}
        }
      ]
    },
    "target_atom_codes_phrase": {
      "properties": {
        "failure_code": {"enum": ["forbidden-match", "no-qualifying-turns", "required-match-missing"]},
        "failure_atom": {"properties": {"code": {"enum": ["forbidden-match", "no-qualifying-turns", "required-match-missing"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["forbidden-match", "no-qualifying-turns", "required-match-missing"]}}}}
      }
    },
    "target_atom_codes_pii": {
      "properties": {
        "failure_code": {"const": "pii-detected"},
        "failure_atom": {"properties": {"code": {"const": "pii-detected"}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"const": "pii-detected"}}}}
      }
    },
    "target_atom_codes_policy": {
      "properties": {
        "failure_code": {"const": "policy-violation"},
        "failure_atom": {"properties": {"code": {"const": "policy-violation"}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"const": "policy-violation"}}}}
      }
    },
    "target_atom_codes_tool_call": {
      "properties": {
        "failure_code": {"enum": ["tool-missing", "tool-arguments-missing", "tool-argument-field-missing", "tool-argument-value-mismatch", "tool-count-below", "tool-count-above", "order-step-absent", "order-step-out-of-order", "never-before-boundary-missing", "never-before-order-violation"]},
        "failure_atom": {"properties": {"code": {"enum": ["tool-missing", "tool-arguments-missing", "tool-argument-field-missing", "tool-argument-value-mismatch", "tool-count-below", "tool-count-above", "order-step-absent", "order-step-out-of-order", "never-before-boundary-missing", "never-before-order-violation"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["tool-missing", "tool-arguments-missing", "tool-argument-field-missing", "tool-argument-value-mismatch", "tool-count-below", "tool-count-above", "order-step-absent", "order-step-out-of-order", "never-before-boundary-missing", "never-before-order-violation"]}}}}
      }
    },
    "target_atom_codes_outcome": {
      "properties": {
        "failure_code": {"enum": ["predicate-unmet", "no-predicate-met"]},
        "failure_atom": {"properties": {"code": {"enum": ["predicate-unmet", "no-predicate-met"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["predicate-unmet", "no-predicate-met"]}}}}
      }
    },
    "target_atom_codes_tool_result": {
      "properties": {
        "failure_code": {"enum": ["tool-missing", "result-missing", "result-field-missing", "result-field-value-mismatch"]},
        "failure_atom": {"properties": {"code": {"enum": ["tool-missing", "result-missing", "result-field-missing", "result-field-value-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["tool-missing", "result-missing", "result-field-missing", "result-field-value-mismatch"]}}}}
      }
    },
    "target_atom_codes_tool_error": {
      "properties": {
        "failure_code": {"enum": ["tool-missing", "unexpected-tool-error", "tool-error-missing", "tool-error-pattern-mismatch"]},
        "failure_atom": {"properties": {"code": {"enum": ["tool-missing", "unexpected-tool-error", "tool-error-missing", "tool-error-pattern-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["tool-missing", "unexpected-tool-error", "tool-error-missing", "tool-error-pattern-mismatch"]}}}}
      }
    },
    "target_atom_codes_state": {
      "properties": {
        "failure_code": {"enum": ["state-record-missing", "state-field-missing", "state-field-value-mismatch"]},
        "failure_atom": {"properties": {"code": {"enum": ["state-record-missing", "state-field-missing", "state-field-value-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["state-record-missing", "state-field-missing", "state-field-value-mismatch"]}}}}
      }
    },
    "target_atom_codes_state_change": {
      "properties": {
        "failure_code": {"enum": ["before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged"]},
        "failure_atom": {"properties": {"code": {"enum": ["before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged"]}}}}
      }
    },
    "target_atom_codes_handoff": {
      "properties": {
        "failure_code": {"enum": ["unexpected-handoff", "handoff-missing", "handoff-target-mismatch"]},
        "failure_atom": {"properties": {"code": {"enum": ["unexpected-handoff", "handoff-missing", "handoff-target-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["unexpected-handoff", "handoff-missing", "handoff-target-mismatch"]}}}}
      }
    },
    "target_atom_codes_termination": {
      "properties": {
        "failure_code": {"enum": ["unexpected-termination", "termination-missing", "termination-attribute-missing", "termination-attribute-value-mismatch"]},
        "failure_atom": {"properties": {"code": {"enum": ["unexpected-termination", "termination-missing", "termination-attribute-missing", "termination-attribute-value-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["unexpected-termination", "termination-missing", "termination-attribute-missing", "termination-attribute-value-mismatch"]}}}}
      }
    },
    "target_atom_codes_latency": {
      "properties": {
        "failure_code": {"enum": ["latency-declared-threshold-exceeded", "latency-default-threshold-exceeded"]},
        "failure_atom": {"properties": {"code": {"enum": ["latency-declared-threshold-exceeded", "latency-default-threshold-exceeded"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["latency-declared-threshold-exceeded", "latency-default-threshold-exceeded"]}}}}
      }
    },
    "target_atom_codes_entity_accuracy": {
      "properties": {
        "failure_code": {"enum": ["entity-missing", "entity-value-mismatch"]},
        "failure_atom": {"properties": {"code": {"enum": ["entity-missing", "entity-value-mismatch"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["entity-missing", "entity-value-mismatch"]}}}}
      }
    },
    "target_atom_codes_sequence": {
      "properties": {
        "failure_code": {"enum": ["sequence-step-absent", "sequence-step-out-of-order"]},
        "failure_atom": {"properties": {"code": {"enum": ["sequence-step-absent", "sequence-step-out-of-order"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["sequence-step-absent", "sequence-step-out-of-order"]}}}}
      }
    },
    "target_atom_codes_count": {
      "properties": {
        "failure_code": {"enum": ["count-below", "count-above"]},
        "failure_atom": {"properties": {"code": {"enum": ["count-below", "count-above"]}}},
        "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["count-below", "count-above"]}}}}
      }
    },
    "private_target": {
      "$comment": "Runtime validation also requires canonical UTF-8 atom ordering, de-duplication, selected atom first, and a matching fingerprint.",
      "type": "object", "additionalProperties": false,
      "required": [
        "test_id", "assertion_digest", "assertion_id", "kind", "dimension",
        "authority", "required_status", "failure_atom", "source_failure_atoms",
        "fingerprint"
      ],
      "properties": {
        "test_id": {"type": "string", "minLength": 1},
        "assertion_digest": {"$ref": "#/definitions/digest"},
        "assertion_id": {"type": "string", "minLength": 1},
        "kind": {"$ref": "#/definitions/assertion_kind"},
        "dimension": {"$ref": "#/definitions/dimension"},
        "authority": {"const": "deterministic"},
        "required_status": {"const": "FAIL"},
        "failure_atom": {"$ref": "#/definitions/failure_atom"},
        "source_failure_atoms": {
          "type": "array", "minItems": 1, "uniqueItems": true,
          "items": {"$ref": "#/definitions/failure_atom"}
        },
        "fingerprint": {"$ref": "#/definitions/digest"}
      },
      "allOf": [{"$ref": "#/definitions/target_kind_atom_coupling"}]
    },
    "share_target": {
      "$comment": "failure_code is the selected private atom code; failure_atom_digest binds the complete private atom.",
      "type": "object", "additionalProperties": false,
      "required": ["assertion_ref", "kind", "dimension", "authority", "required_status", "fingerprint", "failure_code", "failure_atom_digest"],
      "properties": {
        "assertion_ref": {"$ref": "#/definitions/digest"},
        "kind": {"$ref": "#/definitions/assertion_kind"},
        "dimension": {"$ref": "#/definitions/dimension"},
        "authority": {"const": "deterministic"},
        "required_status": {"const": "FAIL"},
        "fingerprint": {"$ref": "#/definitions/digest"},
        "failure_code": {"type": "string", "minLength": 1},
        "failure_atom_digest": {"$ref": "#/definitions/digest"}
      },
      "allOf": [{"$ref": "#/definitions/target_kind_atom_coupling"}]
    },
    "private_source": {
      "type": "object", "additionalProperties": false,
      "required": ["scenario_file_sha256", "test_file_sha256", "scenario_digest", "test_digest"],
      "properties": {
        "scenario_file_sha256": {"$ref": "#/definitions/digest"},
        "test_file_sha256": {"$ref": "#/definitions/digest"},
        "scenario_digest": {"$ref": "#/definitions/digest"},
        "test_digest": {"$ref": "#/definitions/digest"}
      }
    },
    "share_source": {
      "type": "object", "additionalProperties": false,
      "required": ["scenario_digest", "test_digest"],
      "properties": {
        "scenario_digest": {"$ref": "#/definitions/digest"},
        "test_digest": {"$ref": "#/definitions/digest"}
      }
    },
    "oracle_reference": {
      "type": "object", "additionalProperties": false,
      "required": ["path", "digest"],
      "properties": {"path": {"const": "oracle.json"}, "digest": {"$ref": "#/definitions/digest"}}
    },
    "artifacts": {
      "type": "object", "additionalProperties": false,
      "required": [
        "source_scenario", "source_conversation_test", "source_scenario_file",
        "source_conversation_test_file", "scenario", "conversation_test",
        "expected_result", "certificate", "journal", "minimality",
        "report_markdown", "report_html", "share_card",
        "reproduce_script", "predicate_script"
      ],
      "properties": {
        "source_scenario": {"const": "source/scenario.json"},
        "source_conversation_test": {"const": "source/conversation-test.json"},
        "source_scenario_file": {"const": "source/scenario.original"},
        "source_conversation_test_file": {"const": "source/conversation-test.original"},
        "scenario": {"const": "input/scenario.json"},
        "conversation_test": {"const": "input/conversation-test.json"},
        "expected_result": {"const": "expected/assertion-result.json"},
        "certificate": {"const": "certificate.json"},
        "journal": {"const": "reduction.jsonl"},
        "minimality": {"const": "minimality.json"},
        "report_markdown": {"const": "report.md"},
        "report_html": {"const": "report.html"},
        "share_card": {"const": "card.svg"},
        "reproduce_script": {"const": "reproduce.sh"},
        "predicate_script": {"const": "predicate.sh"}
      }
    },
    "artifact_digests": {
      "type": "object", "additionalProperties": false,
      "required": [
        "source_scenario", "source_conversation_test", "source_scenario_file",
        "source_conversation_test_file", "scenario", "conversation_test",
        "expected_result", "certificate", "journal", "minimality",
        "report_markdown", "report_html", "share_card",
        "reproduce_script", "predicate_script"
      ],
      "properties": {
        "source_scenario": {"$ref": "#/definitions/digest"},
        "source_conversation_test": {"$ref": "#/definitions/digest"},
        "source_scenario_file": {"$ref": "#/definitions/digest"},
        "source_conversation_test_file": {"$ref": "#/definitions/digest"},
        "scenario": {"$ref": "#/definitions/digest"},
        "conversation_test": {"$ref": "#/definitions/digest"},
        "expected_result": {"$ref": "#/definitions/digest"},
        "certificate": {"$ref": "#/definitions/digest"},
        "journal": {"$ref": "#/definitions/digest"},
        "minimality": {"$ref": "#/definitions/digest"},
        "report_markdown": {"$ref": "#/definitions/digest"},
        "report_html": {"$ref": "#/definitions/digest"},
        "share_card": {"$ref": "#/definitions/digest"},
        "reproduce_script": {"$ref": "#/definitions/digest"},
        "predicate_script": {"$ref": "#/definitions/digest"}
      }
    },
    "stats": {
      "type": "object", "additionalProperties": false,
      "required": ["bytes", "turns", "tools", "state_leaves", "transcript_segments", "trace_spans"],
      "properties": {
        "bytes": {"type": "integer", "minimum": 1},
        "turns": {"type": "integer", "minimum": 1, "maximum": 10000},
        "tools": {"type": "integer", "minimum": 0, "maximum": 10000},
        "state_leaves": {"$ref": "#/definitions/nonnegative_integer"},
        "transcript_segments": {"$ref": "#/definitions/nonnegative_integer"},
        "trace_spans": {"$ref": "#/definitions/nonnegative_integer"}
      }
    },
    "reduction": {
      "type": "object", "additionalProperties": false,
      "required": [
        "algorithm", "reducer_set", "initial", "final", "attempts",
        "candidate_evaluations", "qualification_evaluations", "total_evaluations",
        "accepted", "cache_hits", "budget", "termination"
      ],
      "properties": {
        "algorithm": {"const": "hierarchical-ddmin"},
        "reducer_set": {"const": "hotato.reducers.v1"},
        "initial": {"$ref": "#/definitions/stats"},
        "final": {"$ref": "#/definitions/stats"},
        "attempts": {"$ref": "#/definitions/nonnegative_integer"},
        "candidate_evaluations": {"type": "integer", "minimum": 0, "maximum": 100000},
        "qualification_evaluations": {"const": 4},
        "total_evaluations": {"type": "integer", "minimum": 4, "maximum": 100004},
        "accepted": {"type": "integer", "minimum": 0, "maximum": 512},
        "cache_hits": {"$ref": "#/definitions/nonnegative_integer"},
        "budget": {"type": "integer", "minimum": 1, "maximum": 100000},
        "termination": {"enum": ["one_minimal", "budget_exhausted"]}
      }
    },
    "minimality_check": {
      "type": "object", "additionalProperties": false,
      "required": ["path", "component", "outcome", "code", "candidate_digest"],
      "properties": {
        "path": {"type": "string", "minLength": 1},
        "component": {"enum": [
          "top-level-optional", "goal-optional", "caller-optional",
          "variation_matrix", "facts", "environment", "interruptions",
          "behavior", "script", "script-field", "tools", "tool-field",
          "handoff-field", "handoff", "termination-field", "termination",
          "state", "agent-mock-optional", "agent_mock"
        ]},
        "outcome": {"enum": ["PRESERVED", "ABSENT", "DRIFTED", "UNRESOLVED"]},
        "code": {"enum": [
          null, "budget_exhausted", "simulator_invalid",
          "assertion_contract_invalid", "assertion_inconclusive",
          "target_failed", "target_absent", "candidate_invalid",
          "failure_identity_drift", "failure_atom_unavailable",
          "resource_limit_exceeded"
        ]},
        "candidate_digest": {"$ref": "#/definitions/raw_digest"}
      }
    },
    "private_minimality": {
      "type": "object", "additionalProperties": false,
      "required": ["status", "reducer_set", "claim", "remaining_unit_checks", "frozen_components"],
      "properties": {
        "status": {"enum": ["one_minimal", "budget_exhausted"]},
        "reducer_set": {"const": "hotato.reducers.v1"},
        "claim": {"type": "string", "minLength": 1},
        "remaining_unit_checks": {"type": "array", "maxItems": 512, "items": {"$ref": "#/definitions/minimality_check"}},
        "frozen_components": {
          "type": "array", "uniqueItems": true,
          "items": {"enum": ["script", "tools", "state", "handoff", "termination"]}
        }
      },
      "allOf": [
        {
          "if": {"properties": {"status": {"const": "one_minimal"}}},
          "then": {"properties": {"claim": {"const": "1-minimal under hotato.reducers.v1 with the recorded observation-scope freezes"}}}
        },
        {
          "if": {"properties": {"status": {"const": "budget_exhausted"}}},
          "then": {"properties": {"claim": {"const": "failure preserved; minimality incomplete because the candidate budget ended"}}}
        }
      ]
    },
    "share_minimality": {
      "type": "object", "additionalProperties": false,
      "required": ["status", "reducer_set", "claim", "check_summary", "frozen_components"],
      "properties": {
        "status": {"enum": ["one_minimal", "budget_exhausted"]},
        "reducer_set": {"const": "hotato.reducers.v1"},
        "claim": {"type": "string", "minLength": 1},
        "check_summary": {
          "type": "object", "additionalProperties": false,
          "required": ["count", "outcomes"],
          "properties": {
            "count": {"type": "integer", "minimum": 0, "maximum": 512},
            "outcomes": {
              "type": "object", "additionalProperties": false,
              "required": ["PRESERVED", "ABSENT", "DRIFTED", "UNRESOLVED"],
              "properties": {
                "PRESERVED": {"$ref": "#/definitions/nonnegative_integer"},
                "ABSENT": {"$ref": "#/definitions/nonnegative_integer"},
                "DRIFTED": {"$ref": "#/definitions/nonnegative_integer"},
                "UNRESOLVED": {"$ref": "#/definitions/nonnegative_integer"}
              }
            }
          }
        },
        "frozen_components": {
          "type": "array", "uniqueItems": true,
          "items": {"enum": ["script", "tools", "state", "handoff", "termination"]}
        }
      },
      "allOf": [
        {
          "if": {"properties": {"status": {"const": "one_minimal"}}},
          "then": {"properties": {"claim": {"const": "1-minimal under hotato.reducers.v1 with the recorded observation-scope freezes"}}}
        },
        {
          "if": {"properties": {"status": {"const": "budget_exhausted"}}},
          "then": {"properties": {"claim": {"const": "failure preserved; minimality incomplete because the candidate budget ended"}}}
        }
      ]
    },
    "preservation": {
      "type": "object", "additionalProperties": false,
      "required": [
        "source_executions", "source_matching_failures", "final_executions",
        "final_matching_failures", "source_result_digests", "final_result_digests",
        "source_content_hashes", "final_content_hashes", "source_trace_digests",
        "final_trace_digests", "same_failure_fingerprint"
      ],
      "properties": {
        "source_executions": {"const": 2},
        "source_matching_failures": {"const": 2},
        "final_executions": {"const": 2},
        "final_matching_failures": {"const": 2},
        "source_result_digests": {
          "type": "array", "minItems": 2, "maxItems": 2,
          "items": {"$ref": "#/definitions/digest"}
        },
        "final_result_digests": {
          "type": "array", "minItems": 2, "maxItems": 2,
          "items": {"$ref": "#/definitions/digest"}
        },
        "source_content_hashes": {
          "type": "array", "minItems": 2, "maxItems": 2,
          "items": {"$ref": "#/definitions/raw_digest"}
        },
        "final_content_hashes": {
          "type": "array", "minItems": 2, "maxItems": 2,
          "items": {"$ref": "#/definitions/raw_digest"}
        },
        "source_trace_digests": {
          "type": "array", "minItems": 2, "maxItems": 2,
          "items": {"$ref": "#/definitions/digest"}
        },
        "final_trace_digests": {
          "type": "array", "minItems": 2, "maxItems": 2,
          "items": {"$ref": "#/definitions/digest"}
        },
        "same_failure_fingerprint": {"const": true}
      }
    },
    "private_privacy": {
      "type": "object", "additionalProperties": false,
      "required": ["profile", "content_included", "network_egress", "hashes_are_correlators"],
      "properties": {
        "profile": {"const": "private-runnable-v1"},
        "content_included": {"const": ["scenario", "conversation_test", "assertion_result"]},
        "network_egress": {"const": false},
        "hashes_are_correlators": {"const": true}
      }
    },
    "share_privacy": {
      "type": "object", "additionalProperties": false,
      "required": ["profile", "content_included", "omitted", "runnable", "hashes_are_correlators"],
      "properties": {
        "profile": {"const": "share-safe-v1"},
        "content_included": {"const": []},
        "omitted": {
          "const": [
            "audio", "transcript_body", "scenario_body", "assertion_body",
            "tool_payload", "state_value", "credentials", "absolute_paths",
            "provider_identifiers", "deletion_paths"
          ]
        },
        "runnable": {"const": false},
        "hashes_are_correlators": {"const": true}
      }
    },
    "evaluator_digest": {"$ref": "#/definitions/digest"},
    "version_string": {
      "type": "string",
      "pattern": "^[0-9]+(?:\\.[0-9]+){1,3}(?:[A-Za-z0-9._+-]*)?$"
    },
    "scenario_selection": {
      "type": "object", "additionalProperties": false,
      "required": ["mode", "seed", "variation_matrix_applied"],
      "properties": {
        "mode": {"const": "base-scenario"},
        "seed": {"type": "integer", "minimum": 0},
        "variation_matrix_applied": {"const": false}
      }
    },
    "private_provenance": {
      "type": "object", "additionalProperties": false,
      "required": ["hotato_version", "evaluator_digest", "seed", "scenario_selection"],
      "properties": {
        "hotato_version": {"$ref": "#/definitions/version_string"},
        "evaluator_digest": {"$ref": "#/definitions/evaluator_digest"},
        "seed": {"type": "integer", "minimum": 0},
        "scenario_selection": {"$ref": "#/definitions/scenario_selection"}
      }
    },
    "share_provenance": {
      "type": "object", "additionalProperties": false,
      "required": ["hotato_version", "evaluator_digest", "scenario_selection"],
      "properties": {
        "hotato_version": {"$ref": "#/definitions/version_string"},
        "evaluator_digest": {"$ref": "#/definitions/evaluator_digest"},
        "scenario_selection": {"$ref": "#/definitions/scenario_selection"}
      }
    }
  }
}


================================================================================
FILE: src/hotato/schema/envelope.v1.json
================================================================================

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://hotato.dev/schema/envelope.v1.json",
  "title": "hotato result envelope, schema_version 1",
  "description": "The single machine-readable shape emitted by the CLI, the one MCP tool, and the suite runner. Additive-only: new signal dimensions and sub-verdicts may appear as extra keys without a version bump, so consumers must ignore unknown fields. The stable core below is frozen for schema_version 1.",
  "type": "object",
  "required": ["tool", "schema_version", "mode", "stack", "offline", "engine", "limits", "summary", "events", "fix_map", "funnel", "exit_code"],
  "additionalProperties": true,
  "properties": {
    "tool": { "const": "hotato" },
    "schema_version": { "const": "1" },
    "mode": { "enum": ["single", "suite"] },
    "suite": { "type": "string" },
    "stack": { "type": "string" },
    "offline": { "const": true },
    "exit_code": { "enum": [0, 1] },
    "engine": {
      "type": "object",
      "required": ["name", "version", "upstream"],
      "additionalProperties": true,
      "properties": {
        "name": { "type": "string" },
        "version": { "type": "string" },
        "upstream": { "type": "string" }
      }
    },
    "limits": {
      "type": "object",
      "required": ["method", "accuracy_claim", "ceiling", "best_input", "does_not_do", "scope", "offline"],
      "additionalProperties": true,
      "properties": {
        "method": { "type": "string" },
        "accuracy_claim": { "type": "null", "description": "HONESTY INVARIANT: always null. This tool never emits an accuracy percentage." },
        "ceiling": { "type": "string" },
        "best_input": { "type": "string" },
        "does_not_do": { "type": "array", "items": { "type": "string" } },
        "scope": { "type": "string" }
      }
    },
    "summary": {
      "type": "object",
      "required": ["events", "passed", "failed", "regression"],
      "additionalProperties": true,
      "properties": {
        "events": { "type": "integer", "minimum": 0 },
        "passed": { "type": "integer", "minimum": 0 },
        "failed": { "type": "integer", "minimum": 0 },
        "regression": { "type": "boolean" },
        "not_scorable": { "type": "integer", "minimum": 1, "description": "Additive: count of events that could not be judged (malformed input, see each event's not_scorable_reason). Present ONLY when at least one event is not scorable; a run of valid recordings emits the exact pre-existing summary shape. Not-scorable events count in events but in neither passed nor failed." }
      }
    },
    "events": {
      "type": "array",
      "items": { "$ref": "#/definitions/event" }
    },
    "fix_map": {
      "type": "array",
      "items": { "$ref": "#/definitions/fix_map_entry" }
    },
    "funnel": {
      "oneOf": [
        { "type": "null" },
        {
          "type": "object",
          "required": ["reason", "pointer"],
          "additionalProperties": true,
          "properties": {
            "reason": { "type": "string" },
            "pointer": { "$ref": "#/definitions/engagement_pointer" }
          }
        }
      ]
    }
  },
  "definitions": {
    "event": {
      "type": "object",
      "required": ["event_id", "expected_yield", "verdict", "measurements"],
      "additionalProperties": true,
      "dependencies": {
        "scorable": ["not_scorable_reason"],
        "not_scorable_reason": ["scorable"]
      },
      "properties": {
        "event_id": { "type": "string" },
        "scorable": { "const": false, "description": "Additive: present ONLY on events that cannot be judged (and then always false); scorable events omit the key entirely, so every envelope for a valid recording is byte-identical to schema_version 1 before this key existed. A not-scorable event is excluded from passed/failed counting and can never carry a normal pass or fail." },
        "not_scorable_reason": { "type": "string", "description": "Plain-language reason the event cannot be judged (for example no detectable caller speech with no onset provided, or the agent silent at the caller onset under a should-yield expectation). Present exactly when scorable is present." },
        "scenario_id": { "type": ["string", "null"] },
        "title": { "type": ["string", "null"] },
        "category": { "type": ["string", "null"] },
        "expected_yield": { "type": "boolean" },
        "verdict": {
          "type": "object",
          "required": ["passed", "did_yield", "seconds_to_yield", "talk_over_sec", "reasons"],
          "additionalProperties": true,
          "properties": {
            "passed": { "type": "boolean" },
            "did_yield": { "type": "boolean" },
            "seconds_to_yield": { "type": ["number", "null"] },
            "talk_over_sec": { "type": "number" },
            "reasons": { "type": "array", "items": { "type": "string" } }
          }
        },
        "measurements": {
          "type": "object",
          "additionalProperties": true,
          "description": "Timing measurements behind the verdict. The core keys (caller_onset_sec, agent_talking_at_onset, hop_sec, notes) are frozen for schema_version 1. The boundary-sensitivity keys below are ADDITIVE (documented here, never in required): they expose where a near-threshold result sat on the frame grid and how far it was from flipping. A reader built before they existed simply ignores them.",
          "properties": {
            "caller_onset_sec": { "type": ["number", "null"] },
            "agent_talking_at_onset": { "type": "boolean" },
            "hop_sec": { "type": "number" },
            "notes": { "type": "string" },
            "onset_requested_sec": { "type": ["number", "null"], "description": "Additive: the caller_onset_sec the caller requested for this event, or null when the onset was auto-detected rather than supplied." },
            "onset_frame_index": { "type": ["integer", "null"], "description": "Additive: the integer hop index the onset was snapped to (null on a not-scorable placeholder)." },
            "onset_effective_sec": { "type": ["number", "null"], "description": "Additive: the quantized onset time actually used, equal to onset_frame_index * hop_sec (unrounded, so the equality holds exactly). Null on a not-scorable placeholder." },
            "yield_frame_index": { "type": ["integer", "null"], "description": "Additive: the hop index of the counted yield, or null when the agent did not yield (or the event was not scorable)." },
            "decision_margin_sec": { "type": ["number", "null"], "description": "Additive: signed slack in seconds from the tightest binding pass/fail threshold to the measured value (positive = inside the pass boundary, magnitude = how much slack; negative = over the line). Null when no numeric bound applies (pure yield/hold) or the event was not scorable." },
            "decision_margin_hops": { "type": ["integer", "null"], "description": "Additive: decision_margin_sec expressed in whole frame hops, round(decision_margin_sec / hop_sec). Null when decision_margin_sec is null." },
            "boundary_sensitive": { "type": "boolean", "description": "Additive: true when the result sits within one hop of flipping the verdict (abs(decision_margin_hops) <= 1). False when comfortably inside the limit, not derivable, or not scorable." }
          }
        },
        "signals": {
          "type": "object",
          "additionalProperties": true,
          "description": "Namespaced signal bus. barge_in is present whenever scoring ran; other dimensions (latency, and future overlap/resume/backchannel) are additive.",
          "properties": {
            "barge_in": {
              "type": "object",
              "required": ["did_yield", "time_to_yield_sec", "talk_over_sec"],
              "additionalProperties": true
            },
            "latency": {
              "type": "object",
              "additionalProperties": true,
              "properties": {
                "response_gap_sec": { "type": ["number", "null"] },
                "premature_start_sec": { "type": ["number", "null"] }
              }
            },
            "resume": {
              "type": "object",
              "description": "Post-yield behaviour on the agent VAD track; present only when the agent yielded. resumed: did the agent start speaking again within the window after the yield. resume_gap_sec: seconds from the yield to that fresh onset (null when it did not resume). restart_suspected: the longest post-resume run is unusually long, a timing proxy for re-answering from the top (a transcript repeat is out of scope).",
              "additionalProperties": true,
              "properties": {
                "resumed": { "type": "boolean" },
                "resume_gap_sec": { "type": ["number", "null"] },
                "restart_suspected": { "type": "boolean" }
              }
            }
          }
        },
        "fix": {
          "oneOf": [ { "type": "null" }, { "$ref": "#/definitions/fix" } ]
        },
        "audio_provenance": { "$ref": "#/definitions/audio_provenance" }
      }
    },
    "audio_provenance": {
      "type": "object",
      "description": "Additive: identity of the exact audio bytes this event was scored from (a streamed sha256 of the raw file, never a full in-memory decode) plus the sample rate and frame count read from the WAV header. Present on every event scored from a real audio file. Absent (never fabricated, never guessed) on an event with no file to hash -- for example a not-scorable / missing-audio placeholder, or an envelope from before this field existed. A reader (in particular hotato fix trial's before/after recapture guard) MUST treat absence as UNKNOWN provenance, never as proof that a fresh capture happened.",
      "required": ["schema_version", "sha256", "sides"],
      "additionalProperties": true,
      "properties": {
        "schema_version": { "type": "string" },
        "sha256": { "type": "string", "description": "The event's combined audio identity: one file's own sha256 for a single-file input (stereo/mono), or an order-stable combination of both sides' sha256 for a caller+agent dual-mono input." },
        "sides": {
          "type": "array",
          "items": {
            "type": "object",
            "required": ["role", "path", "sha256", "sample_rate", "num_samples"],
            "additionalProperties": true,
            "properties": {
              "role": { "type": "string" },
              "path": { "type": "string" },
              "sha256": { "type": "string", "description": "Streamed sha256 of this side's raw file bytes -- container identity. Changes on any byte anywhere in the file, including header-only metadata edits, a trailing-byte append, or a lossless repack/transcode that never touches a sample value." },
              "pcm_sha256": { "type": "string", "description": "OPTIONAL, additive: streamed sha256 of this side's DECODED PCM sample bytes only (the WAV data chunk, read the same way the scorer decodes it -- fixed width, little-endian, channel-interleaved, bounded by the header's own frame count). Identity of the SAMPLES, not the container: a header-only edit or a trailing-byte append leaves this unchanged even though sha256 above differs; an edit to a sample value changes both. Absent on an envelope from before this field existed -- a reader MUST treat absence as UNKNOWN, never as proof either way." },
              "sample_rate": { "type": "integer" },
              "num_samples": { "type": "integer" },
              "duration_sec": { "type": ["number", "null"] }
            }
          }
        }
      }
    },
    "fix": {
      "type": "object",
      "required": ["fix_class", "title", "detail", "knob", "pointer"],
      "additionalProperties": true,
      "properties": {
        "fix_class": { "enum": ["config", "engagement-control"] },
        "title": { "type": "string" },
        "detail": { "type": "string" },
        "knob": {
          "oneOf": [
            { "type": "null" },
            {
              "type": "object",
              "required": ["stack", "parameter", "direction", "trade_off"],
              "additionalProperties": true
            }
          ]
        },
        "pointer": {
          "oneOf": [ { "type": "null" }, { "$ref": "#/definitions/engagement_pointer" } ]
        }
      }
    },
    "fix_map_entry": {
      "type": "object",
      "required": ["event_id", "fix_class", "title", "detail", "knob", "pointer"],
      "additionalProperties": true,
      "properties": {
        "event_id": { "type": "string" },
        "scenario_id": { "type": ["string", "null"] },
        "fix_class": { "enum": ["config", "engagement-control"] },
        "title": { "type": "string" },
        "detail": { "type": "string" }
      }
    },
    "engagement_pointer": {
      "type": "object",
      "description": "VENDOR-NEUTRAL pointer for the engagement-control fix class. Names the problem class and the KIND of fix (a learned engagement-control / addressee-detection layer); names no vendor, no product, and nothing you can adopt/license/buy, and carries no numbers.",
      "required": ["layer", "what", "honest_scope"],
      "additionalProperties": true,
      "properties": {
        "layer": { "type": "string" },
        "what": { "type": "string" },
        "honest_scope": { "type": "string" }
      }
    }
  }
}
