Metadata-Version: 2.5
Name: ragjudge
Version: 0.1.0
Summary: Evals for RAG pipelines — typed, async, honest.
Project-URL: Homepage, https://github.com/learnwithsatyam/ragjudge
Project-URL: Repository, https://github.com/learnwithsatyam/ragjudge
Project-URL: Issues, https://github.com/learnwithsatyam/ragjudge/issues
Author: Satyam Shivhare (@satyamshivhare)
License: MIT
Keywords: ai,evals,evaluation,llm,rag,retrieval
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.5
Requires-Dist: rich>=13.7
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.34; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: openai
Requires-Dist: openai>=1.40; extra == 'openai'
Description-Content-Type: text/markdown

# ragjudge

**Typed, async evals for RAG pipelines** — the Python package. TypeScript port at [`packages/js`](../js).

## Install

```bash
pip install ragjudge              # core (pydantic + httpx + rich)
pip install "ragjudge[openai]"    # + OpenAI judge
pip install "ragjudge[anthropic]" # + Anthropic judge
```

Requires Python 3.10+. Strict-typed end to end.

## 60-second quickstart

```python
import asyncio
from ragjudge import (
    Sample, Suite, ContextRelevance, Faithfulness, AnswerRelevance,
)
from ragjudge.judges.openai import OpenAIJudge

samples = [
    Sample(
        question="What is the capital of France?",
        contexts=["Paris is the capital of France."],
        answer="Paris.",
    ),
]

suite = Suite(
    name="my-rag",
    metrics=[ContextRelevance(), Faithfulness(), AnswerRelevance()],
    judge=OpenAIJudge(model="gpt-4o-mini"),
)
report = asyncio.run(suite.run(samples))
print(f"{report.passed_count}/{report.total} passed")
```

## CLI

```bash
ragjudge run samples.jsonl \
  --judge openai \
  --model gpt-4o-mini \
  --metrics context_relevance,faithfulness,answer_relevance \
  --report-out report.json \
  --fail-under 0.8
```

Each line of `samples.jsonl` is a `Sample` — see `examples/samples.jsonl`.

## Metrics

| Metric              | What it catches                                | Threshold default |
| ------------------- | ---------------------------------------------- | ----------------- |
| `context_relevance` | Retriever pulling unrelated chunks             | 0.7               |
| `faithfulness`      | Hallucination — claims not grounded in context | 0.8               |
| `answer_relevance`  | Answer that misses the question                | 0.7               |
| `answer_correctness`| Semantic drift from a reference answer         | 0.7               |

## Custom judge

Anything matching this shape is a judge — `Judge` is a `runtime_checkable Protocol`, no inheritance needed:

```python
class MyJudge:
    async def judge(self, prompt: str, schema: dict) -> "JudgeResponse":
        ...
```

## Custom metric

```python
from ragjudge import Judge, Metric, Sample, Score

class MyMetric:
    name = "my_metric"
    threshold = 0.5

    async def score(self, sample: Sample, judge: Judge) -> Score:
        ...
```

Pass instances of it to `Suite(metrics=[...])`. That's it.

## License

MIT.
