Metadata-Version: 2.4
Name: evalreliability
Version: 0.0.1
Summary: Measure the reliability of LLM-as-judge evaluations
Keywords: llm-as-a-judge,inter-rater-reliability,intraclass-correlation,krippendorff-alpha,generalizability-theory,evaluation,psychometrics
Author: Shreyas Jagadeep Shete
Author-email: Shreyas Jagadeep Shete <shreyasshete@ieee.org>
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Typing :: Typed
Requires-Dist: numpy>=2.0
Requires-Dist: pandas>=2.2
Requires-Dist: scipy>=1.14
Requires-Dist: evalreliability[inspect,lmeval] ; extra == 'all'
Requires-Dist: inspect-ai>=0.3.263 ; extra == 'inspect'
Requires-Dist: lm-eval>=0.4.13 ; extra == 'lmeval'
Requires-Python: >=3.12
Project-URL: Homepage, https://github.com/shreyasshete/evalreliability
Project-URL: Repository, https://github.com/shreyasshete/evalreliability
Project-URL: Issues, https://github.com/shreyasshete/evalreliability/issues
Project-URL: Changelog, https://github.com/shreyasshete/evalreliability/blob/main/CHANGELOG.md
Provides-Extra: all
Provides-Extra: inspect
Provides-Extra: lmeval
Description-Content-Type: text/markdown

# evalreliability

**Measure the reliability of LLM-as-judge evaluations.**

[![tests](https://github.com/shreyasshete/evalreliability/actions/workflows/tests.yml/badge.svg)](https://github.com/shreyasshete/evalreliability/actions/workflows/tests.yml)
[![PyPI](https://img.shields.io/pypi/v/evalreliability.svg)](https://pypi.org/project/evalreliability/)
[![Python](https://img.shields.io/pypi/pyversions/evalreliability.svg)](https://pypi.org/project/evalreliability/)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)

---

Your eval score went from 3.80 to 4.10 after a prompt change. Did anything
improve?

Run the same judge on the same items twice and you may find it moves 0.68 on
its own. Then 0.30 was never a result. Most teams have never measured this,
because no tool in the eval stack reports it.

Other tools give you a judge score. This one gives you the error bar on that
score, and tells you which decisions it can carry.

## Install

```bash
pip install evalreliability
```

Core depends only on numpy, pandas and scipy. Harness integrations are extras:

```bash
pip install "evalreliability[inspect]"
```

## Status

Early. `0.0.1` ships the input contract and design checking. The measurement
verbs land next â€” see the roadmap below.

## Input

One long frame, one row per rating:

| item | rater | rater_type | run | dimension | score |
| --- | --- | --- | --- | --- | --- |
| doc_01::M3 | gpt-4 | judge | 0 | coherence | 4 |
| doc_01::M3 | claude | judge | 0 | coherence | 3 |
| doc_01::M3 | expert_0 | human | 0 | coherence | 4 |

`item`, `rater` and `score` are required. `rater_type`, `run` and `dimension`
are optional, and each one you supply unlocks another facet of the analysis.

## Design checking

Every reliability coefficient assumes a measurement design. Choose the wrong
one and the number is wrong in a way nothing downstream will flag. So the
design question gets answered from the data, first:

```python
import pandas as pd
from evalreliability import check_design

df = pd.read_parquet("ratings.parquet")

print(check_design(df, by="dimension")["coherence"])
```

```
coherence: crossed, 1600 items x 3 raters, 3 ratings per item
```

When a two-way model is not licensed, it says so and why:

```
relevance: nested, 400 items x 12 raters, 3 ratings per item
  two-way models are not licensed here (raters differ across items);
  use a one-way model
```

And it warns where precision will be poor regardless of how many items you
have:

```
coherence: crossed, 1600 items x 3 raters, 3 ratings per item
  warning: 3 raters gives 2 df on the rater term, so absolute-agreement
  intervals will be very wide
```

That last one matters more than it looks. Absolute-agreement reliability
depends on a rater variance component estimated from the number of raters, not
the number of items. Sixteen hundred items do nothing for it.

## Roadmap

| Release | Verb | The question it answers |
| --- | --- | --- |
| 0.0.1 | `check_design()` | What measurement design do I actually have? |
| 0.1 | `ceiling()` | How much do my raters agree, and what is the best score anyone could get? |
| 0.1 | `stability()` | How much does my judge wobble, and what is the smallest change it can detect? |
| 0.2 | `compare()` | How does my judge do against the reference, relative to the ceiling? |
| 0.3 | `plan()` | How many raters, runs and items do I need, and what will it cost? |

## What this does not do

It will not tell you whether a judge is **correct**. No tool can. There is no
ground truth for whether a summary is coherent â€” only what qualified people
think, and they disagree. In SummEval, three expert annotators reached a
Krippendorff's alpha of 0.41 before reconciliation.

So "correct" is not available as a target. What is available is how far a
judge sits from a human reference whose own noise you have measured, and how
much of your reported number is signal.

## Design principles

- **Correct over comprehensive.** Every coefficient is tested against an
  independent implementation â€” pingouin, krippendorff, GeneralizIT â€” rather
  than against itself.
- **The form follows the decision.** You say whether the score value carries
  your decision or only the ordering. The library derives which coefficient
  that implies and names it in the output. No dropdown of six options you have
  to already understand.
- **Small install.** A measurement library nobody installs measures nothing.
- **Say what is not supported.** Where a claim is not licensed by the data,
  the output says so rather than printing a number anyway.

## Citing

See [CITATION.cff](CITATION.cff). Each release is archived on Zenodo with a
DOI.

## License

Apache-2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE).
