# OpenRTC-Python v0.1 — Design Document

**Status:** Locked
**Target:** v0.1 release in 2 weeks (Phase 1 + Phase 2)
**Audience:** Implementation guide for Claude Code; reference for contributors

---

## 1. Summary

OpenRTC-Python v0.1 introduces a **coroutine-mode worker** that runs N concurrent voice agent sessions inside a single Python process, sharing prewarmed models and a single asyncio event loop. This delivers a 30–50× density improvement over livekit-agents' default process-per-job model (target: 50+ sessions per worker vs ~1 today, at ~60 MB per active session vs ~3 GB).

The change is **contained**. We subclass `livekit.agents.AgentServer` and replace its internal `proc_pool` with our own `CoroutinePool`. We keep livekit-agents' worker WebSocket state machine, the `AgentSession` voice loop, and the entire `livekit.plugins.*` ecosystem unchanged. The user-facing API is backward compatible — existing `AgentPool` code keeps working, and a single optional `isolation` parameter lets users opt back into process mode if they need hard crash isolation.

This is **Option B** from the strategic audit: bypass `proc_pool`, keep everything else.

---

## 2. Goals

1. **Density.** 50+ concurrent sessions per worker process at <4 GB total RSS.
2. **Drop-in upgrade.** Existing `AgentPool` users get the density win with zero code changes — coroutine mode is the default.
3. **Backward compat.** Process-isolation mode preserved for users who require hard crash isolation, exposed via `AgentPool(isolation="process")`.
4. **Same DX.** Agent classes, plugins, `@function_tool`, `@agent_config`, discovery, routing — all unchanged.
5. **Single Docker container.** Self-hostable with `docker compose up`. No external dependencies beyond LiveKit server.
6. **Horizontal scale path.** Multi-worker deployments work via LiveKit's existing dispatch/load balancing — no code change required to scale to 1000+ sessions across multiple workers.

## 3. Non-Goals

- **Multi-participant-per-room** (the `multi-user-transcriber` pattern). One session = one room = one caller. Defer to v0.2.
- **GPU optimization.** Coroutine mode incidentally enables shared GPU memory for local models. Not a v0.1 marketing claim. Validate post-launch.
- **Rust/PyO3 hot path.** Premature. Revisit if profiling shows a Python GIL ceiling around 200+ sessions/worker.
- **Replacing `AgentServer`'s worker WS state machine** (Option C). LiveKit maintains it; we reuse it.
- **Replacing `AgentSession`** or any `livekit.plugins.*` package.
- **Multi-tenancy, billing, hosted dashboard, plugin marketplace.** These belong to the OpenRTC platform repo and OpenRTC Cloud — not this package.

---

## 4. Architecture Decision

We chose Option B (custom `JobExecutor`) over Option A (THREAD mode) and Option C (replace `AgentServer`).

**Why not THREAD mode (Option A).** `JobExecutorType.THREAD` exists in livekit-agents but each thread still creates its own `JobProcess`, runs `setup_fnc` independently, and instantiates its own `rtc.Room`. Density gain is 5–10×, far below the 30× target. Repurposing it would require monkey-patching `setup_fnc` to share globals — at which point we're closer to B anyway.

**Why not replace `AgentServer` (Option C).** The worker WS state machine is ~600 LOC of register/heartbeat/load/drain/reconnect logic. LiveKit maintains it. Reimplementing buys nothing for v0.1 except weeks of work and ongoing protocol tracking. We pin to a specific livekit-agents minor and let them own it.

**Why Option B works.** The `JobExecutor` Protocol is the right abstraction layer. We implement it once with a coroutine-based runner, swap it into `AgentServer._proc_pool`, and gain full multi-session density while keeping every other livekit-agents subsystem (WS, dispatch, drain, plugins, voice loop) unchanged.

---

## 5. Public API

### 5.1 New parameter

```python
from openrtc import AgentPool

pool = AgentPool(
    isolation="coroutine",          # NEW. Default: "coroutine". Alt: "process"
    max_concurrent_sessions=50,     # NEW. Default: 50. Backpressure threshold.
)
```

### 5.2 Existing API — unchanged

```python
from livekit.agents import Agent
from livekit.plugins import openai
from openrtc import AgentPool

class RestaurantAgent(Agent):
    def __init__(self) -> None:
        super().__init__(instructions="You help callers make reservations.")

pool = AgentPool()  # defaults: isolation="coroutine", max_concurrent_sessions=50
pool.add(
    "restaurant",
    RestaurantAgent,
    stt=openai.STT(model="gpt-4o-mini-transcribe"),
    llm=openai.responses.LLM(model="gpt-4.1-mini"),
    tts=openai.TTS(model="gpt-4o-mini-tts"),
    greeting="Welcome to reservations.",
)
pool.run()
```

The 6-symbol public surface (`AgentPool`, `AgentConfig`, `AgentDiscoveryConfig`, `agent_config`, `ProviderValue`, `__version__`) is preserved.

### 5.3 CLI

```bash
openrtc dev ./examples/agents \
  --isolation coroutine \                  # NEW. Default: coroutine
  --max-concurrent-sessions 50 \           # NEW. Default: 50
  --default-stt openai/gpt-4o-mini-transcribe \
  --default-llm openai/gpt-4.1-mini \
  --default-tts openai/gpt-4o-mini-tts
```

### 5.4 Backward compatibility guarantees

| Concern | v0.0.17 behavior | v0.1 behavior |
|---|---|---|
| Default isolation | process (inherited from livekit-agents) | **coroutine** |
| `AgentPool()` with no args | works | works (new default kicks in) |
| `isolation="process"` | n/a | works exactly as v0.0.17 did |
| Agent class API | `class A(livekit.agents.Agent)` | identical |
| Plugin API | passthrough to `livekit.plugins.*` | identical |
| Routing rules | 6-tier resolution chain | identical |
| Metrics / JSONL / TUI | works | works |
| `@agent_config` decorator | works | works |
| Discovery (`pool.discover(path)`) | works | works |

The default change (process → coroutine) is a behavior change, but the user-visible contract (the API) does not break. Document prominently in the changelog and migration notes.

---

## 6. Internal Design

### 6.1 New modules

```
src/openrtc/
├── _coroutine_pool.py       # NEW. CoroutineJobExecutor + CoroutinePool.
├── _coroutine_server.py     # NEW. _CoroutineAgentServer subclass.
├── pool.py                  # Modified. Threads `isolation` parameter through.
├── cli_app.py               # Modified. Adds --isolation flag.
├── cli_params.py            # Modified. Adds isolation to SharedLiveKitWorkerOptions.
└── ...                      # Everything else unchanged.
```

### 6.2 `CoroutineJobExecutor`

Implements `livekit.agents.ipc.job_executor.JobExecutor` Protocol. One instance per active session.

**Responsibilities:**
- Hold a reference to the shared `JobProcess` (singleton across the worker)
- Hold a reference to the shared event loop
- On `launch_job(info)`: construct a `JobContext` referencing the shared `JobProcess`, then schedule the user's entrypoint as an `asyncio.Task` with a wrapping coroutine that handles errors and cleanup
- On `aclose()`: cancel the task, await it
- On `kill()`: cancel forcefully
- Report status (`starting`, `running`, `finished`, `failed`) to the pool
- Catch unhandled exceptions inside the task wrapper; log; mark status `failed`; do **not** propagate to the worker

**Key invariant:** an exception in one `CoroutineJobExecutor`'s task must never escape to crash siblings or the worker.

### 6.3 `CoroutinePool`

Mirrors the public surface that `AgentServer` expects from `proc_pool.ProcPool`. One instance per worker.

**Responsibilities:**
- `start()`: invoke the user's `setup_fnc` (prewarm) **once**, populating the singleton `JobProcess.userdata` with shared models (VAD, turn detector class, inference executor reference). All future jobs read from this same `userdata`.
- `launch_job(info)`: spawn a new `CoroutineJobExecutor` and start it. Track in active set.
- `current_load()`: returns `len(active) / max_concurrent_sessions`. Used by `AgentServer` for status updates to LiveKit (load-based dispatch).
- `aclose()`: cancel all active executors, await them, then await any cleanup hooks.
- Drain support: stop accepting new jobs but allow in-flight to finish gracefully on SIGTERM.

**The pool does not pre-warm executors.** Each executor is a thin wrapper around an asyncio task — they're cheap to create on demand.

### 6.4 `_CoroutineAgentServer`

A thin subclass of `livekit.agents.AgentServer` that swaps `_proc_pool` for our `CoroutinePool` after `__init__`.

```python
# Conceptual sketch — actual impl depends on AgentServer internals.
class _CoroutineAgentServer(AgentServer):
    def __init__(self, *args, max_concurrent_sessions: int = 50, **kwargs):
        super().__init__(*args, **kwargs)
        self._proc_pool = CoroutinePool(
            setup_fnc=self.setup_fnc,
            max_concurrent_sessions=max_concurrent_sessions,
            ...
        )
```

If `AgentServer` resists this swap, the fallback is monkey-patching `_proc_pool` after construction. Either way, the change is contained to one file.

### 6.5 `AgentPool` integration

`pool.py:284` (currently `self._server = AgentServer()`) becomes:

```python
if self._isolation == "coroutine":
    self._server = _CoroutineAgentServer(
        max_concurrent_sessions=self._max_concurrent_sessions,
    )
else:
    self._server = AgentServer()
```

The rest of `AgentPool` (`add`, `discover`, `run`, routing, metrics) works without modification because it talks to `_server` through public surfaces (`setup_fnc`, `rtc_session()`, `cli.run_app()`).

### 6.6 Prewarm refactor

Current `_prewarm_worker(proc: JobProcess, ...)` populates `proc.userdata["vad"]` and `proc.userdata["turn_detection_factory"]`. **No change required to this function.**

In coroutine mode, `JobProcess` is a singleton — all `CoroutineJobExecutor` instances pass the same `proc` reference into the `JobContext` they construct. So `ctx.proc.userdata["vad"]` returns the same shared VAD instance for every session. The entrypoint code (`_run_universal_session`) reads `ctx.proc.userdata["vad"]` and works unchanged.

In process mode, behavior is identical to v0.0.17 — each subprocess gets its own `proc` and its own prewarm.

### 6.7 InferenceExecutor

Keep as-is. livekit-agents spawns one `InferenceProcExecutor` subprocess per worker for turn detection inference; in coroutine mode, this single subprocess is shared across all sessions in the worker. No change needed.

### 6.8 Crash isolation semantics

**Coroutine mode contract:**
- An unhandled exception inside one session's task is caught by the executor wrapper, logged, and the session is marked `failed`.
- Other sessions in the same worker continue running.
- A worker-level supervisor (new in v0.1) tracks consecutive crashes; if N consecutive sessions fail (default: 5), the worker calls `aclose()` and exits — relying on the deployment platform (Docker, systemd, Kubernetes) to restart it. This bounds the blast radius of a systemic bug.

**Process mode contract:**
- Identical to v0.0.17 / livekit-agents default. One crash kills one process; siblings unaffected.

This must be documented prominently. Users with regulatory/compliance requirements for hard isolation should pick `isolation="process"`.

---

## 7. Phase Plan

### Phase 1 — Prove the density win (Week 1)

**Deliverables:**
- `_coroutine_pool.py` and `_coroutine_server.py` skeletons
- `CoroutineJobExecutor` and `CoroutinePool` implementing the Protocol
- `AgentPool(isolation="coroutine")` runs end-to-end against a real LiveKit server
- One sanity-check integration test: 1 coroutine-mode session completes a real call
- Density benchmark: 50 simulated jobs (`AgentServer.simulate_job`) running concurrently in one process; measure peak RSS

**Success gate:** density benchmark shows ≥ 50 concurrent sessions at ≤ 4 GB total RSS. If we can't hit this, stop and reassess.

### Phase 2 — Productionize (Week 2)

**Deliverables:**
- Per-job error isolation (unit + integration tests)
- Backpressure: `max_concurrent_sessions` enforced; `current_load()` reports correctly to LiveKit
- Graceful drain on SIGTERM (in-flight sessions complete; no new accepts)
- Worker-level supervisor: N consecutive failures triggers shutdown
- CLI: `--isolation` and `--max-concurrent-sessions` flags
- Integration tests against a real LiveKit server (exercises the `integration` pytest marker that's been declared but unused since v0.0.x)
- README updated with isolation modes, density benchmark numbers, when to use which
- `docs/concepts/architecture.md` updated with a coroutine-mode lifecycle diagram
- Migration note in `docs/changelog.md`
- CI job pinned to `livekit-agents~=1.5` exactly + a separate canary job that runs against the latest livekit-agents minor as it releases

**Success gate:** all acceptance criteria below pass. Tag v0.1.0. Publish to PyPI.

---

## 8. Acceptance Criteria

A v0.1 release ships when **all** of the following are true:

1. Existing test suite (124 tests) passes unchanged on `isolation="process"`.
2. New coroutine-mode test suite has ≥ 80% coverage of `_coroutine_pool.py` and `_coroutine_server.py`.
3. Density benchmark: 50 concurrent simulated sessions in one worker process, peak RSS ≤ 4 GB, no session-level errors.
4. Real-LiveKit integration test: 5 concurrent calls in one coroutine worker, all complete successfully with real STT/LLM/TTS.
5. Crash isolation test: a session that raises unhandled `RuntimeError` does not affect 4 other sessions running in the same worker.
6. Backpressure test: with `max_concurrent_sessions=10`, the 11th job is not accepted (LiveKit dispatch sees `load = 1.0` and routes elsewhere).
7. `isolation="process"` mode is verified to behave identically to v0.0.17.
8. Drain test: SIGTERM with 3 in-flight sessions waits for completion before worker exits.
9. CLI flags `--isolation` and `--max-concurrent-sessions` work and are documented.
10. README has a comparison table: process vs coroutine mode (memory, density, crash isolation, recommended use cases).
11. CI passes on Python 3.11, 3.12, 3.13.
12. Tagged release on GitHub; published to PyPI; `docs/changelog.md` updated.

---

## 9. Open Questions / Risks

### 9.1 `AgentServer` API stability

We are subclassing or patching internal-ish parts of livekit-agents (`_proc_pool` field, `JobExecutor` Protocol). LiveKit may break these across minor versions.

**Mitigation:** Pin `livekit-agents~=1.5` exactly in `pyproject.toml`. Add a CI canary job that runs the test suite against the latest livekit-agents release as it ships — early warning system.

### 9.2 `JobExecutor` Protocol completeness

The audit identified the surface of the Protocol but the exact method list and semantics need verification against the actual livekit-agents code at the pinned version. Phase 1 starts with reading `livekit/agents/ipc/job_executor.py` and writing a stub that satisfies the type checker.

### 9.3 Memory budget per session

Target is ~60 MB/session. Real WebRTC peer connections are 30–50 MB; audio buffers 5–10 MB; conversation state 5 MB. **Tight but feasible.** If Phase 1 benchmark shows we can't hit 50 sessions in 4 GB, fall back to a more conservative claim ("20+ sessions per worker, 10× improvement") and document the path to better density in v0.2.

### 9.4 Memory monitoring per session

In process mode, `ProcJobExecutor` enforces `job_memory_limit_mb`. In coroutine mode there's no separate process to measure. We can approximate via `psutil` and tracking conversation-state size, but **per-session memory caps are not enforced in v0.1**. Document this gap.

### 9.5 Git tag hygiene

`git tag --list` is empty locally despite `docs/changelog.md` running through v0.0.17. Confirm tags exist on the remote before tagging v0.1.0.

### 9.6 Integration test infrastructure

The `integration` pytest marker exists but no test uses it. Phase 2 needs to set up a real LiveKit dev server (containerized, in CI) and write the first real integration tests. This is its own non-trivial workstream — budget time accordingly.

---

## 10. Out of Scope (Future Work)

The following are explicitly **not** in v0.1. Listed here so they're tracked but not scope-creep candidates:

- **v0.2:** Multi-participant-per-room sessions; per-session memory limits; selective process mode for high-memory agents
- **v0.3:** Rust/PyO3 hot-path port for audio frame routing (only if Python ceiling is hit during real usage)
- **v0.4:** Plugin marketplace, custom plugin SDK
- **v1.0:** Stability commitment; semantic versioning; LTS branch
- **OpenRTC platform repo:** Docker compose stack assembling OpenRTC-Python + VoiceGateway + VoiceQuant + OpenOrca UI
- **OpenRTC Cloud:** Hosted/managed version with multi-tenancy, billing, dashboard

---

## 11. Appendix: File Change Summary

### New files
- `src/openrtc/_coroutine_pool.py` (~300 LOC estimated)
- `src/openrtc/_coroutine_server.py` (~50 LOC)
- `tests/test_coroutine_pool.py` (~400 LOC)
- `tests/test_integration_coroutine.py` (~200 LOC, marked `integration`)
- `docs/benchmarks/density-v0.1.md` (benchmark methodology + numbers)

### Modified files
- `src/openrtc/pool.py` — thread `isolation` and `max_concurrent_sessions` through `AgentPool.__init__`; conditional server construction at line 284 (~20 LOC delta)
- `src/openrtc/cli_params.py` — add `isolation`, `max_concurrent_sessions` to `SharedLiveKitWorkerOptions`
- `src/openrtc/cli_app.py` — wire CLI flags to `AgentPool` constructor
- `src/openrtc/cli_types.py` — add Typer Annotated aliases for new flags
- `pyproject.toml` — pin `livekit-agents~=1.5` exactly; bump version to `0.1.0`
- `README.md` — isolation modes section; benchmark table; migration note
- `docs/concepts/architecture.md` — coroutine-mode lifecycle
- `docs/changelog.md` — v0.1.0 entry with breaking-default note

### Dead code to remove (housekeeping, not strictly required for v0.1)
- `src/openrtc/_version.py` — never imported, leftover from pre-`hatch-vcs`
- `AgentPool._resolve_agent` / `AgentPool._handle_session` (`pool.py:483-500`) — dead methods
- `cli_app.py` `__all__` — strip underscore-prefixed symbols

### CI changes
- Add `livekit-canary` job: runs `pytest -m integration` against the latest `livekit-agents` release (allowed to fail; informational)
- Add `density-benchmark` job: runs the 50-session simulation; fails if peak RSS > 4 GB

---

## 12. Decision Log

| Decision | Rationale |
|---|---|
| Option B (custom `JobExecutor`) over A (THREAD) and C (replace `AgentServer`) | A's density gain too low (5–10×); C's effort too high (3–5 weeks). B hits 30–50× in 1–2 weeks. |
| Default `isolation="coroutine"` | The density win is the headline. Users with isolation requirements opt in. |
| Keep `isolation="process"` working | AGENTS.md preference for compat; users with regulatory needs preserved. |
| One session = one room = one caller | Multi-participant rooms are a separate problem shape; out of scope for v0.1. |
| Failed sessions don't crash siblings; restart-on-N-failures supervisor | Demanding zero blast radius means hand-auditing every async path — too much for v0.1. |
| Keep `InferenceExecutor` as-is (separate subprocess) | Moving inline adds GIL pressure; downgrading loses turn detection quality. |
| No GPU marketing in v0.1 | Real benefit, but unverified. Validate post-launch. |
| No Rust/PyO3 in v0.1 | Pure Python should hit 50+ sessions. Optimize the hot path only if profiling proves it's needed. |
| Pin `livekit-agents~=1.5` exactly | Subclassing internal-ish surfaces; need a stable target. Canary CI job watches for breakage. |
