Metadata-Version: 2.5
Name: apresvous
Version: 0.2.0
Summary: Measure when a voice agent decides it is your turn, from outside the process
Project-URL: Homepage, https://github.com/brijmohan/voice-agent-latency-lab
Project-URL: Source, https://github.com/brijmohan/voice-agent-latency-lab
Author: Brij Mohan Lal Srivastava
License: Apache-2.0
License-File: LICENSE
Keywords: endpointing,latency,realtime,speech,turn-taking,voice-agents
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Multimedia :: Sound/Audio :: Analysis
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.12
Requires-Dist: websockets>=13
Provides-Extra: livekit
Requires-Dist: livekit-agents>=1.7.0; extra == 'livekit'
Requires-Dist: livekit-plugins-cartesia>=1.7.0; extra == 'livekit'
Requires-Dist: livekit-plugins-deepgram>=1.7.0; extra == 'livekit'
Requires-Dist: livekit-plugins-google>=1.7.0; extra == 'livekit'
Requires-Dist: livekit-plugins-silero>=1.7.0; extra == 'livekit'
Requires-Dist: livekit-plugins-turn-detector>=1.7.0; extra == 'livekit'
Requires-Dist: python-dotenv>=1.2.3; extra == 'livekit'
Description-Content-Type: text/markdown

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="docs/banner-dark.svg">
  <img alt="apresvous: measuring when a voice agent decides it is your turn. Median time to first audio 1.577 seconds over 115 turns." src="docs/banner-light.svg" width="100%">
</picture>

# voice-agent-latency-lab

**Where the 800ms goes in a cascaded voice agent, decomposed into four stages and reported
at the tail.**

The wire-level half of this repository is **`apresvous`**, after the phrase for yielding a
turn that a voice agent never says. The banner draws what it measures: the caller stops, and
the bracket is how long they wait. Both numbers on it are measured, not illustrative.

A caller finishes a sentence and waits. This measures that wait, splits it into the four
stages that produce it, and reports the distribution rather than the average.

```
TTFA = end_of_utterance_delay + llm_ttft + tts_ttfb
       └──────────┬──────────┘  └───┬───┘  └───┬───┘
        transcription + endpointing  LLM      TTS
```

The framework reports per component. A caller experiences per turn. Nothing bridges the
two, so this does.

---

## Results

LiveKit Agents 1.8.0, Deepgram nova-3, Gemini 3.5 Flash Lite (direct, not via gateway),
Cartesia sonic-3, Silero VAD, LiveKit audio turn detector v1. macOS, M3 Pro. English.
`endpointing_ms=25`. Console mode. 7 Sep 2026.

**n = 10 completed turns**, from one session. Not enough for P99. See *Honest limits*.

| Stage | P50 | P90 | max | share of P50 TTFA |
|---|---|---|---|---|
| `end_of_utterance_delay` | 0.582s | 2.501s | 2.501s | 48% |
| ├─ `transcription_delay` | 0.361s | 1.256s | 1.515s | 30% |
| └─ `endpointing_delay` | 0.269s | 1.112s | 2.253s | 22% |
| `llm_ttft` | 0.423s | 0.482s | 0.498s | 35% |
| `tts_ttfb` | 0.138s | 0.174s | 0.201s | 11% |
| **TTFA** | **1.201s** | **3.029s** | **3.043s** | |

The full session ledger, which is the part a percentile table normally hides: **12
`speech_id`s, 10 responded, 1 agent-initiated greeting, 1 speculative generation discarded,
0 unclassified.** The table above is the 10.

An earlier session on **1.7.0** (24 Aug) gave TTFA P50 1.530s, P90 3.145s, max 17.065s.
**Do not read the difference as an SDK improvement.** Different conversation, different time
of day, and the LLM is a hosted service whose speed is not being controlled here. Almost all
of the gap is `llm_ttft` moving 0.605s to 0.423s, which is the one stage this setup has no
claim over. Both sessions are kept because the *shape* is reproducible, not the absolute.

Stage percentiles are marginal distributions. They do not sum to the TTFA percentile,
because the worst turn for one stage is not the worst turn for another. Shares are computed
at P50 only.

### Four things that surprised me

**1. `transcription_delay` varies 7x on one speaker in one session** (0.211s to 1.515s),
at a fixed `endpointing_ms=25`. It is the first stage of the budget, almost nobody tunes it,
and it is the least stable. It is also not 25ms: `endpointing_ms` is one input to the
provider's finalisation decision, not the decision. The 1.7.0 session showed the same effect
at 4x, so the instability reproduces and its magnitude does not.

**2. The turn detector is acoustic, and near-identical utterances get opposite decisions.**
Measured on 1.7.0 and not re-run on 1.8.0, because `EOTInferenceMetrics` carries no
`speech_id` and so is not captured by the harness.

| Utterance | P(end of turn) | Decision |
|---|---|---|
| "Wait. Wait. Wait. Wait. Wait." | **0.915** | committed in 0.3s |
| "Wait. Wait. Wait. Wait." | **0.068** | held for 2.5s |

Same words, an order of magnitude apart. `inference.TurnDetector()` encodes audio directly
and scores prosody. No transcript-based model can produce that pair. It is also why an agent
feels inconsistent to its operator: the variable is one the speaker cannot hear themselves
changing.

**3. The tail is the provider retrying, not the model being slow.** On 1.7.0 one turn
recorded `ttft=16.350s` with `cancelled=False` and every other stage normal. It was a 504 and
a retry cycle under `max_retry=3, retry_interval=2.0, timeout=10.0`. Nothing in the metric
object says so; the only evidence was a WARNING line. **A latency table that cannot separate
"slow" from "retried" is measuring provider availability while claiming to measure model
latency.**

A single connection attempt cannot exceed its own per-attempt timeout, so `ttft` above it
proves at least one attempt was retried. The correlator will report that, but only if you
tell it the timeout you configured. It does not assume the SDK default, because the
deployment may not be using it. No retry occurred in the 1.8.0 session, which is why its max
is 3.043s rather than 17s: **that is the difference between a good night and a bad one on
someone else's API, not a property of the stack.**

**4. `end_of_utterance_delay` is bimodal, and neither mode is a measurement of the speaker.**
Seven of ten turns sat at 0.580s to 0.586s. Two sat at 2.5006s and 2.5011s, and the 1.7.0
session also produced one at 2.501s. A value that lands twice within half a millisecond
across two SDK versions is a ceiling, not a latency. It is **not** `max_delay`, which
defaults to 3.0s. I have not established what sets it, and it is on the roadmap rather than
in this table as a claim.

### The endpointing floor nobody tunes

`end_of_utterance_delay` pins to ~0.582s on 7 of 10 turns, and did the same at ~0.578s on
1.7.0. That floor is not `endpointing_ms` and not `min_delay`. It is Silero VAD's
`min_silence_duration=0.55`, because `min_delay` behaves as `max(VAD silence, min_delay)`.
The audio turn detector permits 0.25, so there is roughly **280ms available on every fast
turn**, behind a parameter that appears in neither of the two knobs people reach for.

Reproducing across two SDK versions is what makes this worth stating. One caveat that is not
yet resolved: 1.8.0 ships a `DynamicEndpointing` that *learns* `min_delay` from the caller's
own pausing behaviour with an exponential moving average, clamped by a fixed `max_delay`. If
that is the active path, the floor is not a constant within a session and this measurement
is the start of it rather than the whole of it. Whether it is the default here has not been
checked, and it is on the roadmap.

---

## The second stack, measured on the wire

`huggingface/speech-to-speech` v1.0.0, fully local on the same laptop, measured from outside
the process over the Realtime API it already exposes. No fork and no patch: the harness is an
external WebSocket client that streams recorded human speech and timestamps every server
event on arrival.

**n = 115 turns. TTFA P50 1.577s, P90 2.298s, P99 3.192s.**

The findings that do not appear in any total: **endpointing is the one stage the wire cannot
see**, because `speech_stopped` is emitted after the decision is already made;
`response.created` fires when the LLM *finishes*, so the wire cannot yield time-to-first-token
at any anchor; the endpointer held on **135 of 275 decisions** and each hold costs about
600ms; and **61% of turns arrive as more than one speech segment**, each revision restating
the whole utterance and costing LLM work that is then superseded.

Full write-up in [REPORT-speech-to-speech.md](REPORT-speech-to-speech.md).

**There is no comparison table between the two stacks, on purpose.** One runs three cloud
APIs, the other runs fully local on a laptop. That difference dominates any total, so a
shared table would say something about hosting while appearing to say something about
architecture. The two halves measure from different vantage points and the contribution is
what each one cannot see.

## Honest limits

What this does **not** measure, which is what makes the rest worth reading:

- **The network leg.** Metrics stop at publication. Transit and the receiver's adaptive
  jitter buffer are invisible, and the buffer grows on a bad network. Real caller-perceived
  latency is higher than every number here.
- **Console mode has no room.** No WebRTC, no Opus encode/decode, no SFU. Three tiers exist,
  each strictly worse: console, browser over WebRTC, phone. This is the most optimistic.
- **Not everything is local even so.** The turn detector runs on LiveKit Inference and the
  adaptive interruption detector calls `agent-gateway.livekit.cloud`. There is network in
  the turn-taking path, from Lille.
- **n=10.** P99 at n=10 is the maximum, not a percentile. P99 needs n≥100 per configuration,
  which is not reachable by hand. P90 is reported; P99 is not.
- **One speaker, one language, one session.** No claim about anyone else's voice.
- **The greeting is excluded.** Agent-initiated, so no end-of-utterance exists and TTFA is
  undefined. Excluded by construction, counted separately.

## Correlation, and why it is not a `GROUP BY`

`EOUMetrics`, `LLMMetrics` and `TTSMetrics` share a `speech_id`. `STTMetrics`, `VADMetrics`
and `EOTInferenceMetrics` do not, because STT and VAD run continuously over the input and
are not scoped to an utterance.

Six things break the naive join, all observed in a two-minute conversation:

1. Events arrive out of order. Verified independent across 200 shuffles and full reversal.
2. Nothing marks a turn finished. Completeness is defined as "all TTFA fields present".
3. STT and VAD cannot be joined at all.
4. Turns that never produce audio. Excluded from the table, and counted. Reproducible by
   interrupting inside the first word of a reply: one such turn recorded a 2.501s
   end-of-utterance and a healthy `ttft=0.577s` with `cancelled=False`, and played nothing.
   **Every latency metric for it looked fine.** A table that drops these silently reports
   good numbers for a caller who heard silence.
5. Agent-initiated turns have no end-of-utterance.
6. **`preemptive_generation` is on by default.** On 1.7.0, 21 `LLMMetrics` across 11 turns,
   21 distinct `speech_id`s, none cancelled: the LLM is dispatched speculatively on partial
   transcripts and 9 generations were discarded. Roughly 2x the LLM calls, and the bill. The
   Still on by default in 1.8.0, and **the multiplier is set by the caller, not by the
   config.** Two sessions on the same SDK with the same settings, differing only in how the
   caller spoke: 1.2 LLM calls per responded turn when speaking cleanly with full stops, and
   **2.4 when chunking sentences with rising prosody and interrupting**. A hesitant caller
   roughly doubles the LLM bill of a fluent one. If your users pause mid-sentence, and in
   most public-facing deployments they do, the config is not what determines the spend.

## The scenario

A narrow slot-filling task: booking a civil ceremony at a fictional city hall, with a fixed
list of available slots. Deliberately not a general assistant. A fixed script against a
bounded task is what makes turns comparable between runs; an open-domain agent produces
conversations of varying length and shape and the comparison stops meaning anything.

The prompt is written for voice, not chat: no markdown, no lists, no symbols, because every
character is spoken. An earlier version carried the voice-realism sections from the LiveKit
prompting guide, which emit SSML `<break>` tags into the text stream. Those appeared in 100%
of assistant turns, land in `TTSMetrics.characters_count`, and add real audio if the
provider honours them. Removed, because they contaminate the thing being measured.

## Run it

```bash
uv sync
cp .env.example .env.local          # fill in your keys
uv run python src/apresvous/agent.py console
uv run pytest                        # 51 tests, see docs/TESTING.md
uv run python tools/replay_log.py <logfile>
```

## What is next

Milestones and acceptance criteria are in [ROADMAP.md](ROADMAP.md).

- P90 at n≥100 per configuration, which needs an automated caller.
- The same decomposition against a second architecture that shares no event model.
- French. English STT mangles French proper nouns badly enough that the same agent is
  unusable for a francophone caller, which is fine for measurement and disqualifying for
  anything else. Quantifying that delta is its own result.

## Corrections

Claims made here that turned out to be wrong are recorded in [CORRECTIONS.md](CORRECTIONS.md)
rather than quietly edited.

## Licence

Apache-2.0.
