Metadata-Version: 2.4
Name: evaltrust
Version: 0.1.0
Summary: An auditor for LLM evaluations — tells you whether you can trust your eval results.
Project-URL: Homepage, https://github.com/k-dickinson/evaltrust
Project-URL: Repository, https://github.com/k-dickinson/evaltrust
Project-URL: Issues, https://github.com/k-dickinson/evaltrust/issues
Project-URL: Changelog, https://github.com/k-dickinson/evaltrust/blob/main/CHANGELOG.md
Author: EvalTrust contributors
License: MIT
License-File: LICENSE
Keywords: audit,benchmark,eval,evaluation,llm,statistics
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Requires-Dist: numpy>=1.24
Requires-Dist: rich>=13.0
Requires-Dist: scipy>=1.11
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: statsmodels>=0.14; extra == 'dev'
Description-Content-Type: text/markdown

<h1 align="center">EvalTrust</h1>

<p align="center">
  <strong>An auditor for LLM evaluations.</strong><br>
  It doesn't tell you how good your model is — it tells you whether you can trust the evaluation you used to decide.
</p>

<p align="center">
  <a href="#installation">Install</a> ·
  <a href="#quick-start">Quick start</a> ·
  <a href="#what-it-checks">What it checks</a> ·
  <a href="docs/">Docs</a> ·
  <a href="CONTRIBUTING.md">Contributing</a>
</p>

<p align="center">
  <img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-blue.svg">
  <img alt="Python 3.10+" src="https://img.shields.io/badge/python-3.10%2B-blue.svg">
  <img alt="CI" src="https://github.com/k-dickinson/evaltrust/actions/workflows/ci.yml/badge.svg">
</p>

---

Teams spend real money evaluating models, then look at two numbers:

```
Model A: 84.7
Model B: 86.2   ->  ship B
```

That single comparison hides a dozen assumptions. Maybe the difference isn't
statistically significant. Maybe the sample is too small. Maybe another judge
disagrees, or the benchmark is already saturated. Most eval tools tell you *what*
your score is. **EvalTrust tells you whether you should believe it.**

It works like a financial audit: bookkeeping answers "what are the numbers?"; an
audit answers "can you trust them?" EvalTrust is the audit for evaluations. It runs
*after* your existing eval tool — it doesn't replace it.

## Example

```console
$ evaltrust audit gpt4_run.json claude_run.json
```

```
EvalTrust Audit
Comparing claude-3 vs gpt-4  · 150 examples · source: deepeval+deepeval
╭─ Verdict ────────────────────────────────────────────────────────────────────╮
│ Low Confidence                                                               │
│ The evidence does not support the conclusion. Do not ship on this result     │
│ as-is — resolve the issues below first.                                      │
╰──────────────────────────────────────────────────────────────────────────────╯
 ✗ Improvement is not statistically significant   Statistical Validity
 ⚠ 95% confidence interval overlaps zero          Statistical Validity
 ⚠ Effect size is negligible                      Statistical Validity
 ✓ Benchmark has headroom                         Benchmark Health

  ✗ Improvement is not statistically significant
    Why it matters   A raw gap means nothing until you rule out chance.
    How we detected  A paired permutation test over 150 examples gave p = 0.41.
    How to fix       Do not claim a winner yet. Collect more examples first.
```

Two runs at 71% and 74% — a three-point "win" that is actually noise. EvalTrust
catches it before it becomes a shipping decision.

## Installation

> **Note:** EvalTrust is not yet published to PyPI. Once it is, installation will be
> a single command:
>
> ```bash
> pip install evaltrust
> ```

Until then, install from source:

```bash
git clone https://github.com/k-dickinson/evaltrust
cd evaltrust
pip install -e .
```

## Quick start

<!-- TODO: expand once the packaged release and hosted docs are live. -->

1. Run your evaluation with whatever tool you already use (DeepEval, Promptfoo,
   LangSmith, OpenEvals, or a plain CSV).
2. Point EvalTrust at the output:

   ```bash
   # A file that already compares two or more models:
   evaltrust audit results.json

   # Two single-model runs (e.g. two DeepEval runs), paired by example id:
   evaltrust audit gpt4_run.json claude_run.json
   ```

3. Read the verdict. Fix what it flags. Re-run.

Useful flags:

| Flag | Effect |
|------|--------|
| `--strict` | Exit with a non-zero status on a Low-Confidence verdict (use it to gate CI). |
| `--model-a`, `--model-b` | Choose which two models to compare, or label the two files. |
| `--alpha` | Significance level (default `0.05`). |
| `--seed` | Seed for the resampling (results are deterministic; change only to stress-test). |

## What it checks

EvalTrust audits four pillars of trust and ends in one plain-language verdict —
**High**, **Moderate**, or **Low Confidence**. There is no arbitrary aggregate
score.

| Pillar | The question it answers |
|--------|-------------------------|
| **Statistical Validity** | Is the gap real, large enough to matter, and was the sample big enough to detect it? Paired permutation test, bootstrap confidence interval, Cohen's *d*, and power analysis. |
| **Benchmark Health** | Can the benchmark even separate these models, or is it saturated / flat? |
| **Repeatability** | If you reran the evaluation, would the winner stay the winner? Uses repeated-run data when the file contains it. |
| **Judge Reliability** | Would a different judge reach the same verdict? Uses multi-judge data when the file contains it. |

Every finding follows the same rule — **why it matters**, **how we detected it**,
and **how to fix it**. Checks that need extra data (repeated runs, multiple
judges) don't guess when it's missing; they tell you how to generate it.

See [`docs/checks.md`](docs/checks.md) for the methods and thresholds behind each
one.

## Supported inputs

You never write an EvalTrust-specific format. It reads what your tool already
produced and auto-detects the shape:

- **Promptfoo** results (several providers compared across test cases)
- **Nested JSON** — `{"models": [...], "examples": [{"id", "scores": {...}}]}`
- **Record lists** — JSON like `[{"id", "model", "score"}, ...]`
- **CSV** — long (`id,model,score`) or wide (`id,gpt,claude`)

Single-model tools (DeepEval, LangSmith, OpenEvals) evaluate one model per run,
so you pass two files and EvalTrust pairs them. Details in
[`docs/input-formats.md`](docs/input-formats.md).

## How it works

```
your eval output ──▶ auto-detect + adapter ──▶ canonical model ──▶ audit ──▶ verdict
```

Adapters map every format into one internal representation, so the statistics are
written once and work everywhere. Every statistical method is validated in the
test suite against an independent reference (`scipy` and `statsmodels`), and all
resampling is seeded, so the auditor is itself reproducible. See
[`docs/architecture.md`](docs/architecture.md).

## Roadmap

- **Now:** offline CLI, four pillars, terminal report.
- **Next:** dedicated adapters for more tools, a Python API (`evaltrust.audit(...)`),
  and an optional HTML report.
- **Later:** opt-in orchestration for the pillars that need to generate evidence
  (robustness perturbations, extra judges) and a provenance/reproducibility check.

## Contributing

Contributions are welcome — new format adapters and additional checks especially.
Start with [`CONTRIBUTING.md`](CONTRIBUTING.md).

## License

[MIT](LICENSE).
