Metadata-Version: 2.4
Name: harbor-voice
Version: 0.1.2
Summary: Evaluate voice agents on phone-call benchmarks, on harbor.
Keywords: voice agents,speech-to-speech,evaluation,benchmark,harbor
Author: amoh-ai
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Dist: harbor==0.25.0
Requires-Dist: numpy>=2.0
Requires-Dist: websockets>=13.0
Requires-Dist: litellm>=1.108.8
Requires-Dist: audioop-lts>=0.2 ; python_full_version >= '3.13'
Requires-Dist: aws-sdk-bedrock-runtime[awscrt]>=0.0.2 ; extra == 'aws'
Requires-Dist: boto3>=1.40 ; extra == 'aws'
Requires-Dist: harbor[daytona]==0.25.0 ; extra == 'daytona'
Requires-Dist: harbor[ec2]==0.25.0 ; extra == 'ec2'
Requires-Dist: harbor-voice-eva-adapter==0.1.2 ; extra == 'eva'
Requires-Dist: harbor[gke]==0.25.0 ; extra == 'gke'
Requires-Dist: google-auth[requests]>=2.0 ; extra == 'google'
Requires-Dist: harbor[langsmith]==0.25.0 ; extra == 'langsmith'
Requires-Dist: aiortc==1.15.0 ; extra == 'live'
Requires-Dist: harbor[modal]==0.25.0 ; extra == 'modal'
Requires-Dist: harbor[prime]==0.25.0 ; extra == 'prime'
Requires-Dist: harbor[sbx]==0.25.0 ; extra == 'sbx'
Requires-Dist: harbor-voice-tau-voice-adapter==0.1.2 ; extra == 'tau-voice'
Requires-Dist: harbor[vercel]==0.25.0 ; extra == 'vercel'
Requires-Python: >=3.12
Project-URL: Changelog, https://github.com/amoh-ai/harbor-voice/releases
Project-URL: Documentation, https://github.com/amoh-ai/harbor-voice/blob/main/docs/library.md
Project-URL: Homepage, https://github.com/amoh-ai/harbor-voice
Project-URL: Issues, https://github.com/amoh-ai/harbor-voice/issues
Project-URL: Repository, https://github.com/amoh-ai/harbor-voice
Provides-Extra: aws
Provides-Extra: daytona
Provides-Extra: ec2
Provides-Extra: eva
Provides-Extra: gke
Provides-Extra: google
Provides-Extra: langsmith
Provides-Extra: live
Provides-Extra: modal
Provides-Extra: prime
Provides-Extra: sbx
Provides-Extra: tau-voice
Provides-Extra: vercel
Description-Content-Type: text/markdown

# harbor-voice

**Evaluate voice agents on phone-call benchmarks, on
[harbor](https://github.com/harbor-framework/harbor).**

[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/amoh-ai/harbor-voice/blob/main/LICENSE)
[![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-blue)](https://github.com/amoh-ai/harbor-voice/blob/main/pyproject.toml)
[![Built on harbor](https://img.shields.io/badge/built%20on-harbor-105864)](https://harborframework.com/docs)

harbor-voice evaluates voice agents on simulated phone calls. You can use
harbor-voice to:

- Run voice benchmarks like [τ-Voice](https://github.com/amoh-ai/harbor-voice/blob/main/adapters/tau-voice/README.md) and
  [EVA](https://github.com/amoh-ai/harbor-voice/blob/main/adapters/eva/README.md) against speech-to-speech agents.
- Test agents under realistic callers: accents, background noise,
  interruptions.
- Run calls in parallel, locally or in the cloud.
- Keep every call: the recording, a trajectory and the grade.
- Add your own voice benchmark.

## Benchmarks

- **[tau-voice](https://github.com/amoh-ai/harbor-voice/blob/main/adapters/tau-voice/README.md)**: τ-Voice, the voice mode of
  Sierra's [tau3-bench](https://github.com/sierra-research/tau2-bench).
  Customer-service calls about airline, retail and telecom tasks.
- **[EVA](https://github.com/amoh-ai/harbor-voice/blob/main/adapters/eva/README.md)**: ServiceNow's
  [EVA](https://github.com/ServiceNow/eva). Enterprise calls in airline, IT
  service and medical HR.

Each adapter's README covers its tasks, agents, caller conditions and parity
with the original harness. Other agents run on the [library](https://github.com/amoh-ai/harbor-voice/blob/main/docs/library.md)'s built-in call
loop.

## Install

You need Python 3.12+ and Docker.

**From PyPI:**

```bash
pip install "harbor-voice[tau-voice,eva]"
```

The `tau-voice` and `eva` extras add each benchmark's commands; the library
alone is `pip install harbor-voice`. Each trial installs the same versions
into its environment.

**From source**, to convert tasks or change the code (needs
[uv](https://docs.astral.sh/uv/)):

```bash
git clone https://github.com/amoh-ai/harbor-voice
cd harbor-voice
uv sync   # harbor-voice, both adapters, all extras, dev tools
```

Each trial then installs harbor-voice and the adapter from these source
folders, so a change runs as it is. Prefix the commands below with `uv run`.

## Try it without API keys

The library has offline providers that need no keys and no network. A
two-turn call on them takes a few seconds and writes the recording, the
metrics and a validity report: see the example under
[The built-in call loop](https://github.com/amoh-ai/harbor-voice/blob/main/docs/library.md#the-built-in-call-loop).

## Run a benchmark

Each benchmark needs its API keys in your environment, and a one-time setup
of its callers in your ElevenLabs account: see the
[tau-voice](https://github.com/amoh-ai/harbor-voice/blob/main/adapters/tau-voice/README.md) and [EVA](https://github.com/amoh-ai/harbor-voice/blob/main/adapters/eva/README.md)
READMEs. Every call is a live session with paid APIs, so start with one call.
Published datasets run without converting anything:

```bash
tau-voice job-config --hub-dataset tau-voice-control@0.1.2 \
    --task-names airline-0-d9c07603 --trials 1 -n 1 \
    --model openai.live/gpt-live-1 --agent-config '{"backend_model": "gpt-6-luna"}' \
    -o one.yaml
harbor-voice jobs start -c one.yaml
```

Then a full job: τ-Voice's leaderboard condition, 3 trials per task with 4
calls at a time, or all of EVA:

```bash
tau-voice job-config --hub-dataset tau-voice-regular-test@0.1.2 \
    --trials 3 -n 4 --model openai.live/gpt-live-1 \
    --agent-config '{"backend_model": "gpt-6-luna"}' -o tau.yaml
eva-adapter job-config --hub-dataset eva@0.1.2 \
    --model openai.realtime/gpt-realtime-1.5 -o eva.yaml
harbor-voice jobs start -c tau.yaml
```

- **The agent.** `openai.live/gpt-live-1` is OpenAI Live: `gpt-live-1` speaks
  and listens, and the `backend_model` behind it reasons and calls tools. For
  OpenAI Realtime, use `--model openai.realtime/gpt-realtime-1.5` with no
  `--agent-config`.
- **Concurrency.** Calls run in real time, a few minutes each, so `-n` sets a
  job's wall time. Keep it within your providers' limits on concurrent
  sessions (ElevenLabs' especially), or calls are refused.

| Dataset | Tasks | What |
|---|---|---|
| `tau-voice-control` | 278 | every task, `control` caller |
| `tau-voice-regular` | 278 | every task, `regular` caller |
| `tau-voice-control-test` | 100 | tau3-bench's `test` split, `control` caller |
| `tau-voice-regular-test` | 100 | the `test` split, `regular` caller: the leaderboard's condition |
| `tau-voice-parity-control` | 40 | the parity sample, `control` caller |
| `tau-voice-parity-regular` | 40 | the parity sample, `regular` caller |
| `eva` | 213 | every task, clean audio |
| `eva-parity` | 60 | the parity sample, 20 per domain |

[`registry.json`](https://github.com/amoh-ai/harbor-voice/blob/main/registry.json) lists the datasets. Their tasks are in
[harbor-voice-datasets](https://github.com/amoh-ai/harbor-voice-datasets) and
their images on `ghcr.io/amoh-ai`, both pinned, so a version always runs the
same thing. Published versions: 0.1.0, 0.1.1 and 0.1.2, the latest. The
datasets repo's README says what changed in each and which images each pins.
From 0.1.1 the images are built for x86 and ARM; each adapter's README lists
the current tags. Without `@VERSION`, harbor picks the latest. For a condition or
subset that isn't published, convert the tasks yourself from a checkout: each
adapter's README shows how.

## Running on cloud

The orchestrator stays on your machine. The backend creates one environment
per trial and deletes it afterwards. Choose it with `job-config --env <type>`:

- **Daytona.** Set `DAYTONA_API_KEY` and pass `--env daytona`.
- **EC2.** harbor launches a VM per trial and reaches it over SSH; your
  account's settings go in the job config.
- **Modal, Docker Sandboxes, GKE, Prime Intellect, Vercel.** `--env modal`,
  `sbx`, `gke`, `prime` or `vercel`, with that service's sign-in.

Setup for each, and prebuilt images for faster startup:
[docs/cloud.md](https://github.com/amoh-ai/harbor-voice/blob/main/docs/cloud.md).

## What a run leaves behind

```
<trial>/
├── result.json             harbor's trial result: reward, timings, agent
├── agent/
│   ├── trajectory.json     the call, in ATIF-voice
│   └── call/
│       ├── outcome.json
│       └── <adapter>/      each attempt: the recording, the benchmark's own record and grade
└── verifier/               the reward (reward.txt for tau-voice, reward.json for EVA), result.json
```

## CLI

| Command | What it does |
|---|---|
| `harbor-voice jobs start` / `trials start` | harbor's runners, with the agent defaulting to harbor-voice's |
| `harbor-voice status <job>` | a running job: counts, reward so far, each trial's phase |
| `harbor-voice show <trajectory>` | summarise a call: turns, timeline, validity |
| `harbor-voice listen <trajectory> --out call.wav` | write a call's recording to a WAV |
| `tau-voice` | `job-config`, `voices`; from a checkout, converting tasks, `build-image`, and parity's `sample` and `compare` |
| `eva-adapter` | `job-config`, `agents`, `judges`; from a checkout, `convert`, `build-image`, and parity's `sample` and `compare` |

`harbor view`, harbor's viewer, shows a call as text. The library's own
commands are in [docs/library.md](https://github.com/amoh-ai/harbor-voice/blob/main/docs/library.md).

## Development

```
src/harbor_voice/     the library: providers, the call loop, ATIF-voice, metrics, the CLI
adapters/tau-voice/   the tau-voice adapter (its own package)
adapters/eva/         the EVA adapter (its own package)
tests/                the library's tests (unit/) and each adapter's (adapters/)
```

From a source install:

```bash
uv run pytest
uv run ruff check src adapters scripts tests && uv run ruff format src adapters scripts tests
uv run ty check
```

Tests that need Docker and the benchmarks' images skip without them; CI runs
the rest on Linux and Windows. To contribute, or to add a benchmark, see
[CONTRIBUTING.md](https://github.com/amoh-ai/harbor-voice/blob/main/CONTRIBUTING.md).

## License

Apache-2.0; see [LICENSE](https://github.com/amoh-ai/harbor-voice/blob/main/LICENSE). harbor-voice builds on harbor and
depends on it as a package. The adapters run tau2-bench and EVA in
containers at pinned commits and don't redistribute them, except the parts
copied into the repo, which are under the MIT License: see
[tau2-bench's](https://github.com/amoh-ai/harbor-voice/blob/main/adapters/tau-voice/src/tau_voice/tau2_audio/LICENSE) and
[EVA's](https://github.com/amoh-ai/harbor-voice/blob/main/adapters/eva/src/eva_adapter/vendored/LICENSE) license files.
