Metadata-Version: 2.4
Name: stt-eval-bench
Version: 0.1.1
Summary: Self-hostable STT evaluation toolkit: compare providers on WER, latency, cost, and streaming UX
Author: BrickerP
License: MIT
Project-URL: Homepage, https://github.com/BrickerP/stt-bench
Project-URL: Repository, https://github.com/BrickerP/stt-bench
Keywords: speech-to-text,benchmark,stt,evaluation,whisper,deepgram
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastapi>=0.115.0
Requires-Dist: uvicorn[standard]>=0.32.0
Requires-Dist: httpx>=0.27.0
Requires-Dist: pydantic>=2.9.0
Requires-Dist: pydantic-settings>=2.6.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: typer[all]>=0.12.0
Requires-Dist: rich>=13.9.0
Requires-Dist: openai>=1.54.0
Requires-Dist: deepgram-sdk>=3.8.0
Requires-Dist: jiwer>=3.0.0
Requires-Dist: sacrebleu>=2.4.0
Requires-Dist: aiofiles>=24.1.0
Requires-Dist: aiosqlite>=0.20.0
Requires-Dist: python-multipart>=0.0.12
Requires-Dist: sse-starlette>=2.1.0
Requires-Dist: pyyaml>=6.0.0
Requires-Dist: websockets>=13.0
Requires-Dist: soundfile>=0.12.0
Provides-Extra: dev
Requires-Dist: pytest>=8.3.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.24.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0.0; extra == "dev"
Requires-Dist: ruff>=0.7.0; extra == "dev"
Requires-Dist: mypy>=1.13.0; extra == "dev"
Dynamic: license-file

# STT Bench

[![CI](https://github.com/BrickerP/stt-bench/actions/workflows/ci.yml/badge.svg)](https://github.com/BrickerP/stt-bench/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)

**Self-hostable STT evaluation toolkit** for comparing speech-to-text providers on quality (WER/chrF), streaming latency, cost, and a dictation-oriented vibe score.

This is **not** a Whisper demo. It is a reproducible harness for builders choosing STT for dictation and voice-agent systems — CLI + local Docker service + PR-friendly reports.

PyPI package name: **`stt-eval-bench`** (the name `stt-bench` was already taken on PyPI). CLI command remains **`stt-bench`**.

## Quick start (2 minutes)

```bash
pip install stt-eval-bench

stt-bench benchmark run --provider mock --force
stt-bench report --format markdown
stt-bench results leaderboard
```

No API keys required for mock runs.

From source / editable install:

```bash
git clone https://github.com/BrickerP/stt-bench.git
cd stt-bench
pip install -e ".[dev]"
```

## What you get

| Surface | Description |
|---|---|
| **CLI** | `compare`, `listen`, `benchmark`, `report`, `results` |
| **Local service** | `stt-bench serve` or `docker compose up` — FastAPI + SQLite |
| **Providers** | `mock`, `whisper`, `deepgram`, `assemblyai` (pluggable) |
| **Reports** | Markdown suitable for PR comments |

## Self-host in 60s

```bash
docker compose up --build -d
curl -s http://127.0.0.1:8766/healthz
curl -s http://127.0.0.1:8766/version
```

Details: [docs/service.md](docs/service.md)

## With real providers

```bash
cp .env.example .env   # OPENAI_API_KEY / DEEPGRAM_API_KEY / ASSEMBLYAI_API_KEY
stt-bench listen 1089-134686-0000_noisy
stt-bench compare --all
```

## Dataset

31 samples in `data/manifest.json` (LibriSpeech subset + one noisy clip).

```bash
stt-bench samples
stt-bench dataset validate
```

## Metrics

WER · chrF · punctuation F1 · TTFT · RTF · cost · vibe score — see `stt-bench math`.

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) and [docs/providers.md](docs/providers.md).  
Maturity / Codex-for-OSS execution gate: [docs/OSS_MATURITY.md](docs/OSS_MATURITY.md).  
Architecture: [DESIGN.md](DESIGN.md).

## License

MIT — see [LICENSE](./LICENSE). LibriSpeech data: [CC BY 4.0](https://www.openslr.org/12/).
