Metadata-Version: 2.4
Name: openagent-eval
Version: 0.1.0
Summary: Open-source CLI framework for evaluating RAG systems and AI Agents
Project-URL: Homepage, https://github.com/OpenAgentHQ/openagent-eval
Project-URL: Repository, https://github.com/OpenAgentHQ/openagent-eval
Project-URL: Changelog, https://github.com/OpenAgentHQ/openagent-eval/blob/main/CHANGELOG.md
Project-URL: Documentation, https://openagenthq.github.io/openagent-eval/
Author: OpenAgent Eval Contributors
License: Apache-2.0
License-File: LICENSE
Keywords: ai,cli,evaluation,llm,metrics,rag
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27.0
Requires-Dist: jinja2>=3.1.0
Requires-Dist: loguru>=0.7.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0.0
Requires-Dist: typer>=0.12.0
Provides-Extra: all
Requires-Dist: datasets>=2.0.0; extra == 'all'
Requires-Dist: deepeval>=1.0.0; extra == 'all'
Requires-Dist: elasticsearch>=8.0.0; extra == 'all'
Requires-Dist: evaluate>=0.4.0; extra == 'all'
Requires-Dist: faiss-cpu>=1.7.0; extra == 'all'
Requires-Dist: mypy>=1.10.0; extra == 'all'
Requires-Dist: pgvector>=0.2.0; extra == 'all'
Requires-Dist: pinecone-client>=2.2.0; extra == 'all'
Requires-Dist: psycopg[binary]>=3.1.0; extra == 'all'
Requires-Dist: pypdf>=4.0.0; extra == 'all'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'all'
Requires-Dist: pytest-cov>=5.0.0; extra == 'all'
Requires-Dist: pytest>=8.0.0; extra == 'all'
Requires-Dist: qdrant-client>=1.7.0; extra == 'all'
Requires-Dist: ragas>=0.1.0; extra == 'all'
Requires-Dist: rank-bm25>=0.2.0; extra == 'all'
Requires-Dist: ruff>=0.4.0; extra == 'all'
Requires-Dist: scikit-learn>=1.0.0; extra == 'all'
Requires-Dist: sentence-transformers>=2.0.0; extra == 'all'
Requires-Dist: weaviate-client>=4.5.0; extra == 'all'
Provides-Extra: bm25
Requires-Dist: rank-bm25>=0.2.0; extra == 'bm25'
Provides-Extra: datasets
Requires-Dist: datasets>=2.0.0; extra == 'datasets'
Provides-Extra: dev
Requires-Dist: mypy>=1.10.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.4.0; extra == 'dev'
Provides-Extra: elasticsearch
Requires-Dist: elasticsearch>=8.0.0; extra == 'elasticsearch'
Provides-Extra: evaluation
Requires-Dist: deepeval>=1.0.0; extra == 'evaluation'
Requires-Dist: evaluate>=0.4.0; extra == 'evaluation'
Requires-Dist: ragas>=0.1.0; extra == 'evaluation'
Requires-Dist: scikit-learn>=1.0.0; extra == 'evaluation'
Requires-Dist: sentence-transformers>=2.0.0; extra == 'evaluation'
Provides-Extra: faiss
Requires-Dist: faiss-cpu>=1.7.0; extra == 'faiss'
Provides-Extra: pdf
Requires-Dist: pypdf>=4.0.0; extra == 'pdf'
Provides-Extra: pgvector
Requires-Dist: pgvector>=0.2.0; extra == 'pgvector'
Requires-Dist: psycopg[binary]>=3.1.0; extra == 'pgvector'
Provides-Extra: pinecone
Requires-Dist: pinecone-client>=2.2.0; extra == 'pinecone'
Provides-Extra: qdrant
Requires-Dist: qdrant-client>=1.7.0; extra == 'qdrant'
Provides-Extra: weaviate
Requires-Dist: weaviate-client>=4.5.0; extra == 'weaviate'
Description-Content-Type: text/markdown

# OpenAgent Eval

**Open-source CLI framework for evaluating RAG systems and AI Agents.**

[![CI](https://github.com/OpenAgentHQ/openagent-eval/actions/workflows/ci.yml/badge.svg)](https://github.com/OpenAgentHQ/openagent-eval/actions/workflows/ci.yml)
[![Coverage](https://codecov.io/gh/OpenAgentHQ/openagent-eval/branch/main/graph/badge.svg)](https://codecov.io/gh/OpenAgentHQ/openagent-eval)
[![PyPI Version](https://img.shields.io/pypi/v/openagent-eval.svg)](https://pypi.org/project/openagent-eval/)
[![Python Versions](https://img.shields.io/pypi/pyversions/openagent-eval.svg)](https://pypi.org/project/openagent-eval/)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
[![CodeQL](https://github.com/OpenAgentHQ/openagent-eval/actions/workflows/codeql.yml/badge.svg)](https://github.com/OpenAgentHQ/openagent-eval/actions/workflows/codeql.yml)

---

## Overview

OpenAgent Eval is a local-first, developer-friendly evaluation framework that runs entirely from the command line. It helps developers measure quality, compare experiments, detect hallucinations, and identify retrieval failures in their RAG systems.

**Goal:** Become the `pytest` of AI evaluation.

---

## Features

- **Local-First** - No cloud services, dashboards, or authentication required
- **CLI + SDK** - Use via command line or import as a Python library
- **Framework Agnostic** - Works with any RAG implementation (LangChain, LlamaIndex, custom)
- **Plugin-Based** - Extend with custom metrics, providers, and report generators
- **Comprehensive Metrics** - Retrieval, generation, performance, and cost evaluation
- **Beautiful Reports** - Terminal, Markdown, HTML, and JSON output formats
- **Failure Analysis** - Identify why evaluations fail, not just that they failed

---

## Installation

```bash
pip install openagent-eval
```

For development:

```bash
git clone https://github.com/openagenthq/openagent-eval.git
cd openagent-eval
uv sync
```

---

## Quick Start

### 1. Initialize Configuration

```bash
oaeval init
```

This creates a `config.yaml` file with default settings.

### 2. Configure Your Setup

Edit `config.yaml`:

```yaml
dataset: data/questions.json

retriever:
  provider: chroma
  settings:
    collection_name: my_docs

llm:
  provider: openai
  model: gpt-4o

metrics:
  - faithfulness
  - answer_relevancy
  - hallucination
  - latency

output: terminal
output_dir: ./reports
```

### 3. Run Evaluation

```bash
oaeval run config.yaml
```

### 4. View Results

```bash
oaeval report latest
```

---

## CLI Commands

| Command | Description |
|---------|-------------|
| `oaeval init` | Create configuration file |
| `oaeval run <config>` | Run evaluation pipeline |
| `oaeval report <id>` | View evaluation reports |
| `oaeval compare <a> <b>` | Compare two experiments |
| `oaeval list` | List previous evaluations |
| `oaeval doctor` | Check environment and dependencies |

---

## SDK Usage

Use OpenAgent Eval as a Python library:

```python
from openagent_eval import Evaluator

evaluator = Evaluator(config_path="config.yaml")
result = evaluator.evaluate(dataset)

print(result.summary)
```

---

## Evaluation Categories

### Retrieval Metrics
- Context Precision
- Context Recall
- Recall@K / Precision@K
- Hit Rate
- Mean Reciprocal Rank (MRR)
- NDCG

### Generation Metrics
- Faithfulness (via Ragas)
- Answer Relevancy (via Ragas)
- Hallucination Detection (via DeepEval)
- Semantic Similarity
- Exact Match / F1 Score
- BLEU / ROUGE
- BERTScore

### Performance Metrics
- Embedding latency
- Retrieval latency
- LLM latency
- Total latency

### Cost Metrics
- Token counting (prompt, completion, total)
- Cost estimation per provider
- Total experiment cost

---

## Supported Providers

### LLM Providers
- OpenAI
- Google Gemini
- Anthropic
- Groq
- OpenRouter
- Ollama (local)

### Retriever Providers
- Chroma
- (More coming soon)

---

## Project Structure

```
openagent-eval/
├── openagent_eval/          # Main package
│   ├── cli/                 # CLI commands (Typer)
│   ├── config/              # Configuration system (Pydantic)
│   ├── core/                # Core orchestration
│   ├── datasets/            # Dataset loaders
│   ├── metrics/             # Evaluation metrics
│   ├── providers/           # LLM/Retriever adapters
│   ├── reports/             # Report generators
│   ├── plugins/             # Plugin system
│   └── exceptions/          # Custom exceptions
├── tests/                   # Test suite
├── pyproject.toml           # Project configuration
└── README.md
```

---

## Development

### Setup

```bash
# Clone repository
git clone https://github.com/openagenthq/openagent-eval.git
cd openagent-eval

# Install dependencies
uv sync

# Run tests
uv run pytest

# Run linter
uv run ruff check .

# Format code
uv run ruff format .
```

### Running Tests

```bash
# Run all tests
uv run pytest

# Run with coverage
uv run pytest --cov=openagent_eval

# Run specific test file
uv run pytest tests/unit/test_exceptions.py
```

---

## Contributing

Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

---

## Roadmap

### v1.0 (Current)
- RAG evaluation
- CLI + SDK interfaces
- Plugin architecture
- Multiple report formats

### v2.0 (Planned)
- AI Agent evaluation
- Tool-call evaluation
- Planning evaluation
- Memory evaluation
- Multi-agent evaluation

### v3.0 (Future)
- CI/CD integration
- GitHub Action
- Cloud synchronization
- Hosted evaluation platform

---

## License

Licensed under the Apache License, Version 2.0 - see [LICENSE](LICENSE) for details.

---

## Support

- **Documentation:** [docs.openagenthq.com](https://docs.openagenthq.com)
- **Issues:** [GitHub Issues](https://github.com/openagenthq/openagent-eval/issues)
- **Discussions:** [GitHub Discussions](https://github.com/openagenthq/openagent-eval/discussions)
