Metadata-Version: 2.5
Name: rag-eval-py
Version: 0.1.0
Summary: Score any RAG pipeline: retrieval precision + answer faithfulness
License: MIT
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: pydantic>=2.0
Requires-Dist: rich>=13.0
Requires-Dist: typer>=0.12
Provides-Extra: api
Requires-Dist: fastapi>=0.110; extra == 'api'
Requires-Dist: uvicorn>=0.29; extra == 'api'
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: judge
Requires-Dist: openai>=1.30; extra == 'judge'
Description-Content-Type: text/markdown

# rag-eval

> **Score any RAG pipeline on retrieval precision and answer faithfulness — zero framework lock-in, zero required API keys.**

[![CI](https://github.com/mustafaabadshah/rag-eval/actions/workflows/ci.yml/badge.svg)](https://github.com/mustafaabadshah/rag-eval/actions/workflows/ci.yml)
[![PyPI version](https://img.shields.io/pypi/v/rag-eval-py.svg)](https://pypi.org/project/rag-eval-py/)
[![Python versions](https://img.shields.io/pypi/pyversions/rag-eval-py.svg)](https://pypi.org/project/rag-eval-py/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

---

## Demo

<p align="center">
  <img src="assets/demo.svg" alt="rag-eval CLI Demo" width="820" />
</p>

---

## Install in 30 seconds

Install the core library and CLI:
```bash
pip install rag-eval-py
```

Optional extras:
```bash
# FastAPI server + uvicorn
pip install "rag-eval-py[api]"

# OpenAI LLM-as-a-judge backend
pip install "rag-eval-py[judge]"

# All extras
pip install "rag-eval-py[api,judge]"
```

---

## Quickstart

Evaluate samples programmatically in 10 lines of Python:

```python
from rag_eval.faithfulness import faithfulness
from rag_eval.retrieval import precision_at_k

# 1. Provide your RAG outputs
retrieved = ["Paris is the capital of France.", "Lyon is in France."]
golden = ["Paris is the capital of France."]
answer = "Paris is the capital of France."
context = ["Paris is the capital and largest city of France."]

# 2. Score retrieval and answer faithfulness
p_at_k = precision_at_k(retrieved, golden, k=1)
score, supported, unsupported = faithfulness(answer, context)

print(f"Retrieval Precision@1: {p_at_k:.2f}")  # 1.00
print(f"Answer Faithfulness:    {score:.2f}")  # 1.00
```

---

## How Scoring Works

`rag-eval` evaluates two independent axes of RAG performance:

### 1. Retrieval Quality: Precision@k
Measures whether the retriever fetched ground-truth chunks in its top-$k$ results:

$$\text{Precision@k} = \frac{|\text{set}(\text{retrieved}[:k]) \cap \text{set}(\text{golden\_documents})|}{k}$$

- Uses strict **set semantics**: duplicated golden documents never artificially inflate the score.
- If no retrieval data is supplied, this metric is omitted (`None`).

### 2. Answer Faithfulness: Claim Overlap
Evaluates whether the generator hallucinated information beyond the retrieved context:

1. **Claim Extraction**: The answer is decomposed into atomic claims by splitting on sentence punctuation (`[.!?]`) and contrasting clauses (`, but `, `; `, etc.).
2. **Stopword Filtering**: Extracts alphanumeric content words while stripping English stopwords (`"the"`, `"is"`, `"of"`, etc.).
3. **Threshold Check**: A claim is verified as supported if at least **80%** (`SUPPORT_THRESHOLD = 0.8`) of its content words appear in the context.
4. **Faithfulness Score**:
$$\text{Faithfulness} = \frac{|\text{supported claims}|}{|\text{total claims}|}$$

If the answer has no claims, the score defaults to `1.0`.

---

## JSONL Format Spec

Feed evaluation data in plain `.jsonl` files (one JSON object per line):

| Field | Type | Required? | Meaning |
| :--- | :--- | :--- | :--- |
| `question` | `string` | **Yes** | The user prompt or question passed to the RAG pipeline. |
| `answer` | `string` | **Yes** | The final answer generated by the LLM. |
| `context` | `list[string]` | **Yes** (min 1) | The actual text chunks injected into the LLM prompt context. |
| `golden_documents` | `list[string]` | No | Ground-truth reference chunks (required for retrieval scoring). |
| `retrieved` | `list[string]` | No | Chunks returned by retriever in ranked order. |
| `id` | `string` | No | Unique sample ID (auto-generates UUID4 if omitted). |

---

## CLI Reference

```bash
rag-eval eval [OPTIONS] DATA
```

| Option / Flag | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| `DATA` | `Path` | *Required* | Path to the evaluation `.jsonl` file. |
| `--json` | Flag | `False` | Render output report as a JSON string to stdout. |
| `--skip-errors` | Flag | `False` | Skip invalid lines and log warnings instead of aborting. |
| `--k` | `int` | `5` | Rank cutoff for Precision@k retrieval calculation. |
| `--claims-only` | Flag | `False` | Skip retrieval metrics and evaluate answer faithfulness only. |
| `--judge` | `string` | `None` | Optional LLM judge backend (`openai`). |
| `--help` | Flag | | Show help message and exit. |

---

## FastAPI Usage

Run the bundled evaluation microservice:

```bash
uvicorn examples.fastapi_server:app --reload --port 8000
```

### Endpoints

- **`GET /health`**  
  Returns `{"status": "ok"}`

- **`POST /evaluate`**  
  Accepts a JSON payload with `samples` and optional `k`:
  ```json
  {
    "samples": [
      {
        "question": "What is the capital of France?",
        "answer": "Paris is the capital of France.",
        "context": ["Paris is the capital and largest city of France."],
        "golden_documents": ["Paris is the capital and largest city of France."],
        "retrieved": ["Paris is the capital and largest city of France."],
        "id": "sample-1"
      }
    ],
    "k": 5
  }
  ```
  Returns `EvalReport` JSON with sample statistics and arithmetic means.

---

## Framework Adapters

`rag-eval` is 100% framework-agnostic. Pre-built adapters in [`examples/`](examples/) map outputs from popular frameworks without introducing hard dependencies:

- **Haystack 2.x**: [`examples/haystack_adapter.py`](examples/haystack_adapter.py) — maps pipeline outputs directly into `list[Sample]`.
- **LangChain**: [`examples/langchain_adapter.py`](examples/langchain_adapter.py) — maps LCEL and RetrievalQA datasets into `list[Sample]`.

---

## Limitations

- **Heuristic Claim Overlap**: The default zero-dependency `claim_overlap` method uses lexical content-word overlap ($0.8$ threshold). While extremely fast and completely free of API charges, it may miss semantic paraphrasing or complex negations.
- **When to Use `--judge openai`**: For mission-critical production evaluations requiring semantic nuance, pass `--judge openai` to verify claims using GPT-4o-mini or specify your preferred LLM judge.

---

## Contributing & License

Contributions are welcome! To run tests, linting, and type checking locally:

```bash
hatch run test
hatch run lint
hatch run typecheck
```

Released under the [MIT License](LICENSE). Copyright (c) 2026 Mustafa Abad Shah.
