<div align="center">
  <a href="https://github.com/mahimailabs/openrtc">
    <img src="assets/banner.png" alt="OpenRTC Banner" width="100%" />
  </a>
</div>

<br />

# openrtc-python

Run N LiveKit voice agents in one worker. Pay the model-load cost once.

*PyPI package name: [`openrtc`](https://pypi.org/project/openrtc/).*

<br />

<div align="center">
  <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a>
  <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.11+-blue.svg" alt="Python Version"></a>
  <a href="https://github.com/astral-sh/ruff"><img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json" alt="Ruff"></a>
  <a href="https://pypi.org/project/openrtc/"><img src="https://img.shields.io/pypi/v/openrtc.svg" alt="PyPI version"></a>
  <a href="https://codecov.io/gh/mahimairaja/openrtc-python"><img src="https://codecov.io/gh/mahimairaja/openrtc-python/graph/badge.svg?token=W7VQ5FGSA9" alt="codecov"></a>
  <a href="https://github.com/mahimairaja/openrtc-python/actions/workflows/test.yml"><img src="https://github.com/mahimairaja/openrtc-python/actions/workflows/test.yml/badge.svg" alt="CI"></a>
</div>

<br />

<details>
  <summary><b>Table of Contents</b></summary>
  <ol>
    <li><a href="#the-problem">The problem</a></li>
    <li><a href="#what-openrtc-does">What openrtc does</a></li>
    <li><a href="#installation">Installation</a></li>
    <li><a href="#quick-start-explicit-registration-with-add">Quick start: explicit registration with add()</a></li>
    <li><a href="#quick-start-one-python-file-per-agent-with-discover">Quick start: one Python file per agent with discover()</a></li>
    <li><a href="#memory-before-and-after">Memory: before and after</a></li>
    <li><a href="#routing">Routing</a></li>
    <li><a href="#greetings-and-session-options">Greetings and session options</a></li>
    <li><a href="#provider-configuration">Provider configuration</a></li>
    <li><a href="#cli">CLI</a></li>
    <li><a href="#public-api-at-a-glance">Public API at a glance</a></li>
    <li><a href="#project-structure">Project structure</a></li>
    <li><a href="#contributing">Contributing</a></li>
    <li><a href="#license">License</a></li>
  </ol>
</details>

<br />

## The problem

You already ship three voice agents with `livekit-agents`. Each agent is its own worker on the same VPS. Every worker process loads the same shared stack: Python runtime, Silero VAD, and the turn-detection model. You are not loading three different models. You are loading the same stack three times because the process boundary forces it. On a 1–2 GB instance, that shows up as duplicate resident set for every idle worker. You pay RAM for copies you do not need.

## What openrtc does

`openrtc` gives you one `AgentPool` in one worker: prewarm runs once, each incoming call still gets its own `AgentSession`, and you register multiple `Agent` subclasses on the pool so dispatch can pick one per session from metadata or fallbacks. This package does not replace your agent code. It does not sit between you and `livekit.agents.Agent`, `@function_tool`, `RunContext`, `on_enter`, `on_exit`, `llm_node`, `stt_node`, or `tts_node`. You keep your subclasses and tools as they are. You change how many workers you run, not how you write an agent.

## Installation

OpenRTC **requires Python 3.11 or newer**. The LiveKit Silero / turn-detector
plugins depend on `onnxruntime`, which does not ship supported wheels for
Python 3.10 in current releases—use 3.11+ to avoid install failures.

```bash
pip install openrtc
```

The base install pulls in `livekit-agents[openai,silero,turn-detector]` so shared prewarm has the plugins it expects. The package ships a **PEP 561** `py.typed` marker for downstream type checkers.

With **uv** (recommended in [CONTRIBUTING.md](CONTRIBUTING.md)):

```bash
uv add openrtc
uv add "openrtc[cli]"
```

```bash
pip install 'openrtc[cli]'
```

Set the same variables you use for any LiveKit worker:

```bash
export LIVEKIT_URL=ws://localhost:7880
export LIVEKIT_API_KEY=devkey
export LIVEKIT_API_SECRET=secret
```

For OpenAI-backed plugins, set `OPENAI_API_KEY` as you already do.

## Quick start: explicit registration with `add()`

Use this when you want every agent registered in one place with explicit names and providers.

```python
from livekit.agents import Agent
from livekit.plugins import openai
from openrtc import AgentPool


class RestaurantAgent(Agent):
    def __init__(self) -> None:
        super().__init__(instructions="You help callers make restaurant bookings.")


class DentalAgent(Agent):
    def __init__(self) -> None:
        super().__init__(instructions="You help callers manage dental appointments.")


pool = AgentPool()
pool.add(
    "restaurant",
    RestaurantAgent,
    stt=openai.STT(model="gpt-4o-mini-transcribe"),
    llm=openai.responses.LLM(model="gpt-4.1-mini"),
    tts=openai.TTS(model="gpt-4o-mini-tts"),
    greeting="Welcome to reservations.",
)
pool.add(
    "dental",
    DentalAgent,
    stt=openai.STT(model="gpt-4o-mini-transcribe"),
    llm=openai.responses.LLM(model="gpt-4.1-mini"),
    tts=openai.TTS(model="gpt-4o-mini-tts"),
)
pool.run()
```

## Quick start: one Python file per agent with `discover()`

Use this when you prefer one module per agent and optional `@agent_config(...)` on each class.

Create a directory (for example `agents/`) and add one `.py` file per agent. Then:

```python
from pathlib import Path

from livekit.plugins import openai
from openrtc import AgentPool

pool = AgentPool(
    default_stt=openai.STT(model="gpt-4o-mini-transcribe"),
    default_llm=openai.responses.LLM(model="gpt-4.1-mini"),
    default_tts=openai.TTS(model="gpt-4o-mini-tts"),
)
pool.discover(Path("./agents"))
pool.run()
```

Example file `agents/restaurant.py`:

```python
from livekit.agents import Agent
from openrtc import agent_config


@agent_config(name="restaurant", greeting="Welcome to reservations.")
class RestaurantAgent(Agent):
    def __init__(self) -> None:
        super().__init__(instructions="You help callers make restaurant bookings.")
```

If a module has no `@agent_config`, the agent name defaults to the filename stem. STT, LLM, TTS, and greeting fall back to the pool defaults.

Discovered agents work with `livekit dev` and spawn-based workers on macOS. For `add()`, define agent classes at module scope so worker reload can import them.

## Migrating from livekit-agents

Already running one or more `livekit-agents` workers? Each is its own process that
loads the same VAD and turn-detector models. Collapse them into one `AgentPool`
worker without changing your agents.

**Before** (one worker per agent, N processes):

```python
# restaurant_worker.py  (plus a near-identical dental_worker.py, support_worker.py, ...)
from livekit import agents
from livekit.agents import Agent, AgentSession
from livekit.plugins import openai, silero


class RestaurantAgent(Agent):
    def __init__(self) -> None:
        super().__init__(instructions="You help callers book tables.")


async def entrypoint(ctx: agents.JobContext) -> None:
    session = AgentSession(
        stt=openai.STT(), llm=openai.LLM(), tts=openai.TTS(), vad=silero.VAD.load()
    )
    await session.start(agent=RestaurantAgent(), room=ctx.room)
    await ctx.connect()


if __name__ == "__main__":
    agents.cli.run_app(agents.WorkerOptions(entrypoint_fnc=entrypoint))
```

**After** (one worker, N agents, one shared prewarm):

```python
# worker.py
from livekit.agents import Agent
from livekit.plugins import openai
from openrtc import AgentPool


class RestaurantAgent(Agent):  # unchanged
    def __init__(self) -> None:
        super().__init__(instructions="You help callers book tables.")


class DentalAgent(Agent):  # unchanged
    def __init__(self) -> None:
        super().__init__(instructions="You help callers manage appointments.")


pool = AgentPool(default_stt=openai.STT(), default_llm=openai.LLM(), default_tts=openai.TTS())
pool.add("restaurant", RestaurantAgent)
pool.add("dental", DentalAgent)
pool.run()
```

Your `Agent` subclasses, tools, and provider objects are unchanged. You delete the
per-worker boilerplate (`entrypoint`, `AgentSession` wiring, `cli.run_app`) and
register the agents on one pool; OpenRTC owns prewarm, routing, and per-call
session construction. On the first run the worker logs the win, for example:

```text
OpenRTC: 2 agents in 1 worker (baseline ~410 MB). 2 separate livekit-agents
workers would cost ~820 MB; sharing one worker saves ~410 MB of idle baseline
(assumes equal per-worker baselines).
```

See [Routing](#routing) for how each incoming call resolves to one registered agent.

## Memory: before and after

Assume an illustrative **~400 MB** idle baseline per worker for the shared stack (VAD, turn detector, and similar). Your measured RSS will differ by provider, model, and OS.

| | Before openrtc | After openrtc |
| --- | --- | --- |
| Three workers, same stack | about **3 × 400 MB ≈ 1.2 GB** idle baseline (three loads) | — |
| One worker, three registered agents | — | about **one × 400 MB** idle baseline (one load) plus per-session overhead |

Exact numbers depend on your providers, concurrency, and call patterns. The win is not loading that stack once per agent worker.

## Isolation modes

`AgentPool` accepts an `isolation` argument that picks how each session
runs inside the worker. The v0.1 default is `"coroutine"`; pass
`isolation="process"` to opt back into the v0.0.x behavior:

```python
pool = AgentPool(
    isolation="coroutine",          # default in v0.1
    max_concurrent_sessions=50,     # backpressure threshold (coroutine only)
)
```

| Aspect | `coroutine` (default) | `process` |
| --- | --- | --- |
| Sessions per worker | Many (one `asyncio.Task` per session, shared `JobProcess`) | One (each session is its own subprocess via `livekit-agents` `ProcPool`) |
| Prewarm cost (VAD, turn detector) | Paid once per worker | Paid once per session subprocess |
| Crash isolation | Cooperative: an unhandled exception in one session is logged and marked FAILED; siblings continue. After `consecutive_failure_limit` (default 5) the worker calls `aclose()` so the platform restarts it. | Hard: each subprocess crashes independently; siblings unaffected. |
| Per-session memory cap | Not enforced (asyncio shares one process) | Enforced via `livekit-agents` `job_memory_limit_mb` |
| Backpressure | `current_load() = active / max_concurrent_sessions` reported as worker load; LiveKit dispatch routes elsewhere at `>= load_threshold` | `livekit-agents` default load math (CPU-based) |
| When to pick | High density on a single host; cost-sensitive deployments. | Regulatory/compliance requires hard process isolation; per-session memory caps required. |

### Density: coroutine vs process at 10/25/50/100 sessions

From the v0.1 stub-workload benchmark (`tests/benchmarks/density.py`,
results recorded at `docs/benchmarks/density-v0.1.md`):

| Sessions | `coroutine` peak RSS | `process` est. peak RSS | Within 4 GB budget |
| ---: | ---: | ---: | :---: |
|  10 |   165 MB | ~30 GB  | coroutine ✓ / process ✗ |
|  25 |   230 MB | ~75 GB  | coroutine ✓ / process ✗ |
|  50 |   367 MB | ~150 GB | coroutine ✓ / process ✗ |
| 100 |   617 MB | ~300 GB | coroutine ✓ / process ✗ |

The same harness scales cleanly to 200 sessions (1073 MB) and 500
sessions (1370 MB) without breaching the 4 GB budget — see the full
results doc for the headroom sweep.

**How the process column was estimated.** Each `livekit-agents`
subprocess loads the Silero VAD + turn-detector models per worker
process (~250-400 MB of model weights) plus the WebRTC peer connection
and Python runtime, settling at the **~3 GB per process** baseline
documented in `docs/audit-2026-05-02.md`. Multiply by `N` for the
total. We do not run process mode at N>2 in CI for this reason.

**Stub-workload caveat.** The benchmark allocates ~5 MB per session to
stress task scheduling, not a realistic ~60 MB/session WebRTC + LLM
footprint. Validate against the §8.4 real-LiveKit integration test
(which needs `docker compose -f docker-compose.test.yml up -d` and
`OPENAI_API_KEY`) before quoting a per-session memory number to your
operators.

### Throughput: steady-state event-loop p99

Memory density is only half the question. N sessions share one event loop and
one GIL, so the other half is whether the loop keeps up.
`tests/benchmarks/throughput.py` drives N concurrent sessions through the real
Silero VAD over synthetic 16 kHz PCM at 50 fps (the continuous on-loop CPU cost)
and measures event-loop p99 latency, separating the startup burst from steady
state.

```bash
uv run python tests/benchmarks/throughput.py --sessions 1,10,25,50,100
```

Sample sweep (Apple M-series laptop, `vad` workload, steady state):

| Sessions | steady-state loop p99 | peak RSS |
| ---: | ---: | ---: |
|   1 | 0.9 ms | 160 MB |
|  10 | 1.3 ms | 160 MB |
|  25 | 1.2 ms | 160 MB |
|  50 | 1.1 ms | 160 MB |
| 100 | 2.8 ms | 160 MB |

Steady-state VAD inference stays well under a 100 ms loop-latency budget to 100
sessions, with flat resident memory (the model loads once). The expensive,
bursty part is session *startup* (each `session.start()` plus greeting), which
the benchmark reports as a separate `startup_p99` column and which dominates
early-life latency. This workload models the continuous VAD path, not the full
STT/LLM/TTS orchestration, so read it as the on-loop-CPU ceiling rather than a
full-pipeline guarantee. Run it on your own hardware before quoting a
sessions-per-worker number.

### Prove it on your machine

The process column above is estimated. This script measures both models for
real on your laptop: it spawns one subprocess per session for the
process-per-session model, runs the same number of sessions as `asyncio`
tasks in a single process for the coroutine model, then prints the memory
used each way. No LiveKit server, no API keys, no model download.

```bash
uv run python examples/density_demo.py                 # 16 sessions
uv run python examples/density_demo.py --sessions 32   # the gap widens with N
uv run python examples/density_demo.py --sessions 50 --load-vad   # adds the shared Silero VAD model
```

Sample output (Apple M-series laptop, import-only mode):

```text
Hosting 16 concurrent voice sessions. Measuring resident memory.

  livekit-agents (process per session):     1861 MB total   ( 116.3 MB/session)
  OpenRTC coroutine pool (one process):      195 MB total   (  12.2 MB/session)

  OpenRTC uses 9.5x less memory for the same 16 sessions.
```

Your numbers vary by machine, and the ratio grows as you raise `--sessions`
(the coroutine pool pays the import cost once and amortizes it across every
session). This default mode counts only the `livekit-agents` import cost, so
it is a conservative lower bound: `--load-vad` adds the shared Silero VAD
model weights (paid once in the pool, once per process otherwise), and
`tests/benchmarks/density.py --sessions 50` proves the 50-sessions-under-4-GB
ceiling. The full script is [examples/density_demo.py](examples/density_demo.py).

## Routing

One process hosts several agent classes, so each session must resolve to a single registered name. `AgentPool` resolves the agent in this order:

1. `ctx.job.metadata["agent"]`
2. `ctx.job.metadata["demo"]`
3. `ctx.room.metadata["agent"]`
4. `ctx.room.metadata["demo"]`
5. room name prefix match, such as `restaurant-call-123`
6. the first registered agent

If metadata names an agent that is not registered, you get a `ValueError` instead of a silent fallback.

## Greetings and session options

You can pass a greeting and extra `AgentSession` options per registration.

```python
pool.add(
    "restaurant",
    RestaurantAgent,
    greeting="Welcome to reservations.",
    session_kwargs={"turn_handling": {"interruption": {"enabled": False}}},
    max_tool_steps=4,
    preemptive_generation=True,
)
```

Direct keyword arguments win over the same keys inside `session_kwargs`.

By default, OpenRTC sets explicit `turn_handling` with the multilingual turn detector and VAD-based interruption. To opt into adaptive interruption, pass `session_kwargs={"turn_handling": {"interruption": {"mode": "adaptive"}}}`.

## Provider configuration

Pass instantiated provider objects through to `livekit-agents` unchanged, for example:

- `openai.STT(model="gpt-4o-mini-transcribe")`
- `openai.responses.LLM(model="gpt-4.1-mini")`
- `openai.TTS(model="gpt-4o-mini-tts")`

If you pass strings such as `openai/gpt-4.1-mini`, OpenRTC leaves them as-is and the LiveKit runtime interprets them for your deployment.

## Session observers

Attach external telemetry to every session without subclassing or touching OpenRTC internals. Any object with two async methods is a `SessionObserver` (structural typing, no base class):

```python
from openrtc import AgentPool, SessionInfo, SessionOutcome


class LoggingObserver:
    async def on_session_start(self, info: SessionInfo, session: object) -> None:
        print(f"live: {info.agent_name} in {info.room_name}")

    async def on_session_end(self, info: SessionInfo, outcome: SessionOutcome) -> None:
        print(f"done: {info.agent_name} -> {outcome.status.value}")


pool = AgentPool(observers=[LoggingObserver()])  # or pool.add_observer(...)
```

`on_session_start` receives the live `AgentSession`, which is where you subscribe to its metrics. `on_session_end` receives the terminal outcome (`SUCCESS`, `FAILED`, or `CANCELLED`). Observer calls are isolated: a raising or slow observer is logged and skipped, never crashing the session. In `process` isolation mode an observer must be picklable, so build live resources lazily on the first `on_session_start`.

## CLI

Install `openrtc[cli]` to get `openrtc` on your PATH. Subcommands follow the LiveKit Agents CLI shape (`dev`, `start`, `console`, `connect`, `download-files`), plus `list`. For most commands you can pass the agents directory as the first path argument instead of `--agents-dir`.

**List what discovery would register** (defaults are string passthroughs for `livekit-agents`, not constructed provider objects):

```bash
openrtc list \
  ./agents \
  --default-stt openai/gpt-4o-mini-transcribe \
  --default-llm openai/gpt-4.1-mini \
  --default-tts openai/gpt-4o-mini-tts
```

**Run a production worker** (after exporting `LIVEKIT_*`):

```bash
openrtc start ./agents
```

**Run a development worker**:

```bash
openrtc dev ./agents
```

Same as ``openrtc dev --agents-dir ./agents``. The metrics JSONL file is **optional**: add a second path only when you want JSONL output (same as ``--metrics-jsonl``), e.g. ``openrtc dev ./agents ./openrtc-metrics.jsonl``.

Optional visibility: `--dashboard` prints a Rich summary in the terminal. `--metrics-json-file ./runtime.json` overwrites a JSON snapshot on each tick. Use that for scripts, dashboards, or CI. For a streaming feed, use `--metrics-jsonl ./openrtc-metrics.jsonl` on the worker: it appends one JSON object per line as metrics arrive. Tail it with `tail -f ./openrtc-metrics.jsonl` or pipe it through `jq`, and any log shipper or script can consume the JSON Lines.

Stable machine output: `openrtc list --json` and `--plain`. Combine `--resources` when you want footprint hints. OpenRTC-only flags are stripped before the handoff to LiveKit’s CLI parser.

Full flag lists live in [docs/cli.md](docs/cli.md).

## Public API at a glance

Everything openrtc exposes publicly is listed here. Anything else is internal and not treated as stable.

- `AgentPool`
- `AgentConfig`
- `AgentDiscoveryConfig`
- `agent_config(...)`
- `ProviderValue` — type alias for STT/LLM/TTS slot values (provider ID strings or LiveKit plugin instances)

`AgentPool(...)` constructor (all keyword-only, all optional):

- `default_stt`, `default_llm`, `default_tts`, `default_greeting` — pool-wide defaults applied when `add()` / `discover()` doesn't override them.
- `isolation: "coroutine" | "process"` (v0.1) — worker isolation mode. Default `"coroutine"` runs every session as an `asyncio.Task` in one worker; `"process"` keeps the v0.0.x one-subprocess-per-session behavior.
- `max_concurrent_sessions: int` (v0.1) — coroutine-mode backpressure threshold. Default `50`. The worker reports `load >= 1.0` to LiveKit dispatch once this many sessions are in flight; ignored under `isolation="process"`.
- `consecutive_failure_limit: int` (v0.1) — coroutine-mode supervisor threshold. Default `5`. After this many non-`SUCCESS` session terminations the worker calls `aclose()` so the deployment platform can restart it; ignored under `isolation="process"`.

On `AgentPool`:

- `add(...)`
- `discover(...)`
- `list_agents()`
- `get(name)`
- `remove(name)`
- `run()`
- `runtime_snapshot()`
- `drain_metrics_stream_events()` — for JSONL export paths (mainly CLI; rare in app code)
- `server`
- `isolation` (read-only property, v0.1)
- `max_concurrent_sessions` (read-only property, v0.1)
- `consecutive_failure_limit` (read-only property, v0.1)

## Project structure

```text
src/openrtc/
├── __init__.py
├── py.typed
├── core/                  # foundational, flat (pool, config, discovery, wiring)
│   ├── pool.py            # AgentPool facade
│   ├── config.py          # AgentConfig, AgentDiscoveryConfig, agent_config
│   ├── discovery.py       # file-system discovery helpers
│   ├── serialization.py   # spawn-safe config serialization
│   ├── turn_handling.py   # turn-detector integration
│   └── wiring.py          # AgentSession assembly helpers
├── routing/               # family: base_routing.py + variant siblings + resolver
│   ├── base_routing.py    # RoutingStrategy protocol
│   ├── metadata_routing.py
│   ├── room_prefix_routing.py
│   ├── default_routing.py
│   └── resolver.py        # selects active strategy
├── runtime/               # family: base_runtime.py + variant siblings + registry
│   ├── base_runtime.py    # RuntimeBackend protocol
│   ├── coroutine_runtime.py
│   ├── process_runtime.py
│   ├── coroutine_server.py
│   ├── prewarm.py         # shared prewarm helpers (non-variant)
│   ├── resources.py       # shared resource cache
│   ├── file_watcher.py    # hot-reload file watcher
│   └── registry.py        # selects active runtime
├── observability/         # family: base_observer.py + base_sink.py + concretes
│   ├── base_observer.py   # SessionObserver protocol
│   ├── base_sink.py       # MetricsSink protocol
│   ├── jsonl_sink.py      # JSONL metrics schema and writer
│   ├── metrics.py         # RuntimeMetricsStore, footprint helpers
│   ├── snapshot.py        # PoolRuntimeSnapshot dataclass
│   ├── resident_set.py    # RSS memory helpers
│   ├── savings.py         # cost-savings estimator
│   └── footprint.py       # per-session memory footprint
├── cli/                   # family: base_cli.py + variant siblings
│   ├── base_cli.py        # shared Typer args and parameter bundles
│   ├── main_cli.py        # top-level Typer app and subcommands
│   ├── dashboard_cli.py   # Rich dashboard and list output
│   ├── entry_cli.py       # lazy console entry / missing-extra hint
│   ├── livekit_cli.py     # LiveKit argv/env handoff, pool run
│   └── reporter_cli.py    # background metrics reporter thread
└── utils/                 # foundational, flat
    ├── types.py           # ProviderValue and related typing
    └── validation.py      # input validation helpers
```

- `core/pool.py`: `AgentPool` facade; discovery, routing, and session assembly all delegate to focused modules
- `cli/`: Typer/Rich CLI (`openrtc[cli]`)
- `observability/jsonl_sink.py`: JSONL metrics schema and writer

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md). CI runs **Ruff** and **mypy** on pull
requests alongside the test suite.

## License

MIT. See [LICENSE](LICENSE).
