Metadata-Version: 2.5
Name: rtpbench
Version: 0.4.0
Summary: Wire-level latency and conformance tests for voice agents over SIP/RTP
Author-email: Juanjo Aroztegi <juanjo@aroztegi.com>
License-Expression: Apache-2.0
License-File: LICENSE
Requires-Python: >=3.14
Requires-Dist: cryptography>=50
Requires-Dist: rich>=15
Provides-Extra: vad
Requires-Dist: onnxruntime>=1.31; extra == 'vad'
Requires-Dist: webrtcvad-wheels>=2.0.14; extra == 'vad'
Description-Content-Type: text/markdown

# rtpbench

Wire-level latency and conformance tests for voice agents over SIP/RTP.

rtpbench calls a voice agent over SIP, captures the RTP on both ends, and measures turn latency,
barge-in and SIP behaviour from the packets themselves, on clean and impaired networks
(`tc netem`). Every measurement carries an error bar calibrated against a reference agent that
logs the truth, and every case is proven against that agent, passing and failing:
`rtpbench cases` runs that proof on your host.

Status: early; commands and result formats can still change between releases.

## What it found

Three open-source ways to put the OpenAI Realtime API on a SIP call, through 24 cases, with and
without a carrier in front, on a mock engine that holds the AI engine's share of each turn fixed:

| | [LiveKit](https://github.com/livekit/sip) | [AI Voice Connector CE](https://github.com/OpenSIPS/opensips-ai-voice-connector-ce) at b95f5af | Asterisk + [AVA](https://github.com/hkjarral/AVA-AI-Voice-Agent-for-Asterisk) |
| --- | --- | --- | --- |
| cases failed of 24 | 5, and 2 unsupported | 5, 3 unsupported and 1 inconclusive | 4, and 3 inconclusive |
| turn latency, clean, p50 | 1833 ms | 1511 ms | 1589 ms |
| its own media path per turn, p50 | 335 ms | 9 ms | 88 ms |
| stops for an interruption, clean, p50 | 306 ms | 24 ms | never |

The counts take every profile, direct and through the carrier, and AVA as it ships, over
AudioSocket. Run direct only, as
`target NAME --mock` does by default, LiveKit fails 4: `late-offer` fails only through the
carrier. All three pass the 3 cases for lost SIP messages, run direct on a clean network.
Unsupported is an optional feature declined as its RFC allows, with the call unharmed:
LiveKit refuses DTMF by SIP INFO (405), AI Voice Connector CE refuses it (501) and negotiates no
RFC 4733 events, and neither takes session timers. LiveKit does negotiate RFC 4733 events, but
the digit never reaches the engine: a failure.

- LiveKit keeps sending audio to a held caller in every kind of hold, AI Voice Connector CE when
  held with `c=0.0.0.0`, and both answer a re-INVITE that changes codec with the old codec. AVA
  sends RTP only while it speaks, so whether its media resumes after a hold cannot be judged,
  except over Asterisk's WebSocket channel, where Asterisk keeps sending to a held caller.
- LiveKit's extra latency is its own media path: 326 ms per turn more than AI Voice Connector
  CE's, against 322 ms between their turn latencies.
- LiveKit's RTP falls 20 ms behind its own timestamps 4 s into every call, so a receiver needs a
  21 ms jitter buffer for it on a clean network; AI Voice Connector CE's needs 1.1 ms.
- AI Voice Connector CE delivers the engine's speech sample for sample. LiveKit re-encodes it on
  the way, through its WebRTC room, and on a clean network stretches or cuts a pause by 20-40 ms
  in 6% of answers (10% in a second wave).
- AVA, as its golden configuration ships, cuts the last 0.6 s of speech from every answer, over
  each of its three transports, and over Asterisk's WebSocket channel its pacer also breaks every
  answer with 7-10 short silences. It never stops for an interruption: it keeps the caller's
  audio from the engine while it speaks. Over AudioSocket under packet loss, the engine sometimes
  got silence instead of the caller (a squelch AVA has fixed since), so 5 more cases flip now
  and then.
- AI Voice Connector CE's main branch fixed 4 of the 10 cases it failed in the baseline (a failed
  transfer no longer hangs up on the caller; compact headers, a bare `a=ssrc` and SIP over TCP
  are taken), and now declines a DTMF INFO with 501 where it left it unanswered. It still
  refuses an INVITE without SDP and answers no re-INVITE without one.

The details, commands, conditions and raw runs are in [the baseline](results/baseline),
[AI Voice Connector CE at b95f5af](results/aivc-b95f5af), [Asterisk + AVA](results/ava) and
[lost SIP messages](results/retransmit), and every call's timeline on
[the results site](https://jjaroztegi.github.io/rtpbench/).

Common ways of measuring turn latency have errors of their own: measured inside the agent, a turn
misses the network (80 ms on an LTE path); an energy rule on a recording reads turns long, by
87-110 ms at the median. Each method against the truth: [metrology](results/metrology).

## Quick start

Needs Linux, Docker 28 or later with Compose, and [uv](https://docs.astral.sh/uv/). WSL2 runs
it, but its clock can drift and step by far more than the error bars: read what `calibrate`
warns about first ([clock steps and rate](docs/calibration.md#clock-steps-and-rate)).

```sh
uvx rtpbench calibrate --profiles clean --turns 5     # how exactly this host times the wire
uvx 'rtpbench[vad]' cases                             # prove every case against the reference agent
uvx 'rtpbench[vad]' target livekit --mock             # a bundled target, no AI account needed
uvx 'rtpbench[vad]' target sip:agent@host             # an agent of your own, the same way
```

The first three take about 1, 17 and 11 minutes, after pulling about 0.4 GB of Docker images
(compressed) on first use; every target and the carrier take 1.5 GB
([images](docs/README.md#images)). The last needs an agent that answers SIP; [your own
agent](docs/targets.md#your-own-agent) says what it needs, such as `--network` for one in Docker on
this host.

Each run writes `results/<timestamp>/` with the captures, `summary.md` and, for cases,
`report.html`: a timeline of every call's SIP, speech and RTP. `rtpbench index` links every run
under `results/` from `results/index.html`; `rtpbench COMMAND --help` lists a command's options,
defaults and valid names.

## Documentation

- [Calibration](docs/calibration.md): error bars, clock steps, network profiles and their sources
- [Conformance cases](docs/cases.md): what each case checks
- [Targets (and adding one), your own agent, the mock engine and the carrier](docs/targets.md)
- [CI](docs/ci.md): pass or fail your agent's run against a policy

## Development

From a clone, `uv run rtpbench` (`uv run --extra vad rtpbench` for `[vad]`) runs the checkout's
code, as the docs show it:

```sh
uv sync
uv run ruff format && uv run ruff check --fix
uv run mypy
uv run pytest
```

## Licence

Apache-2.0, except `targets/aivc/opensips.cfg`, derived from AI Voice Connector CE's example and
under its GPL-3.0-or-later, and `targets/ava/ai-agent.yaml`, derived from AVA's golden OpenAI
configuration and under its MIT licence.
