Metadata-Version: 2.4
Name: calibrate-agent
Version: 0.0.14
Summary: An open-source evaluation framework for voice agents
Author-email: Aman Dalmia <aman.dalmia@artpark.in>
License-Expression: CC-BY-SA-4.0
Project-URL: Documentation, https://calibrate.artpark.ai/docs
Keywords: voice-agents,evaluation,benchmarking,stt,tts,llm,simulation,pipecat
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE.md
Requires-Dist: backoff>=2.2.1
Requires-Dist: elevenlabs>=2.31.0
Requires-Dist: evaluate>=0.4.6
Requires-Dist: instructor>=1.13.0
Requires-Dist: jiwer>=4.0.0
Requires-Dist: langfuse==3.3.0
Requires-Dist: matplotlib>=3.10.7
Requires-Dist: natsort>=8.4.0
Requires-Dist: onnxruntime>=1.23.2
Requires-Dist: openpyxl>=3.1.5
Requires-Dist: opentelemetry-api>=1.38.0
Requires-Dist: opentelemetry-exporter-otlp>=1.38.0
Requires-Dist: opentelemetry-sdk>=1.38.0
Requires-Dist: pipecat-ai-small-webrtc-prebuilt==1.0.0
Requires-Dist: pipecat-ai[cartesia,deepgram,google,groq,local-smart-turn-v3,openrouter,tracing,webrtc,websocket]==0.0.98
Requires-Dist: python-dotenv>=1.2.1
Requires-Dist: sarvamai>=0.1.21
Requires-Dist: simpleaudio>=1.0.4
Requires-Dist: smallestai>=4.0.1
Requires-Dist: sounddevice>=0.5.3
Requires-Dist: uvicorn>=0.38.0
Requires-Dist: whisper-normalizer>=0.1.12
Provides-Extra: dev
Requires-Dist: ipdb>=0.13.13; extra == "dev"
Requires-Dist: ipython>=8.37.0; extra == "dev"
Requires-Dist: pytest>=7.0; extra == "dev"
Dynamic: license-file

# Calibrate

[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/ARTPARK-SAHAI-ORG/calibrate)
[![Discord](https://img.shields.io/badge/Discord-Join%20Community-7289da?logo=discord&logoColor=white)](https://discord.gg/9dQB4AngK2)
[![WhatsApp](https://img.shields.io/badge/WhatsApp-Join%20Community-25D366?logo=whatsapp&logoColor=white)](https://chat.whatsapp.com/JygDNcZ943a3VmZDXYMg5Z)
[![CC BY-SA 4.0][cc-by-sa-shield]][cc-by-sa]

An open-source evaluation framework for voice agents.

With [Calibrate](https://calibrate.artpark.ai), you can move from slow, manual testing to a fast, automated, and repeatable testing process for your entire voice agent stack:

- `Speech to Text (STT)`: Benchmark multiple providers (Google, Sarvam, ElevenLabs and more) on your dataset across 10+ indic languages using metrics optimised for agentic use cases
- `Text to Speech (TTS)`: Benchmark generated speech by multiple providers automatically using an Audio LLM Judge across 10+ indic languages
- `Text to Text (LLMs)`: Evaluate the response quality and tool calling of your LLMs for multi-turn conversations and find the find LLM for your agent
- `Simulations`: Simulate realistic conversations using realistic user personas and scenarios to test failure modes for your agent including interruptions

Calibrate is built on top of [pipecat](https://github.com/pipecat-ai/pipecat), a framework for building voice agents.

## Installation

```bash
pip install calibrate-agent
```

## Usage

```bash
calibrate              # Interactive main menu
calibrate stt          # Benchmark STT providers
calibrate tts          # Benchmark TTS providers
calibrate llm          # Interactive LLM evaluation
calibrate simulations  # Interactive text or voice simulations
```

- [CLI Documentation](https://calibrate.artpark.ai/docs/cli/overview)

## License

This work is licensed under a
[Creative Commons Attribution-ShareAlike 4.0 International License][cc-by-sa].

[![CC BY-SA 4.0][cc-by-sa-image]][cc-by-sa]

[cc-by-sa]: http://creativecommons.org/licenses/by-sa/4.0/
[cc-by-sa-image]: https://licensebuttons.net/l/by-sa/4.0/88x31.png
[cc-by-sa-shield]: https://img.shields.io/badge/License-CC%20BY--SA%204.0-lightgrey.svg
