Metadata-Version: 2.5
Name: opik-rigor
Version: 0.2.0
Summary: Statistical assertions and pinned-judge evaluation primitives for LLM test suites.
Project-URL: Homepage, https://github.com/ericwehmeyer/opik-rigor
Project-URL: Repository, https://github.com/ericwehmeyer/opik-rigor
Project-URL: Issues, https://github.com/ericwehmeyer/opik-rigor/issues
Author: E. Wehmeyer
License-Expression: MIT
License-File: LICENSE
Keywords: evaluation,llm,opik,statistics,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: numpy>=1.21
Requires-Dist: scipy>=1.10
Provides-Extra: dev
Requires-Dist: build>=1.0; extra == 'dev'
Requires-Dist: hatchling>=1.20; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: opik
Requires-Dist: opik<3,>=2.0; extra == 'opik'
Provides-Extra: pytest
Requires-Dist: pytest>=7.0; extra == 'pytest'
Provides-Extra: typecheck
Requires-Dist: mypy>=1.11; extra == 'typecheck'
Requires-Dist: pytest>=7.0; extra == 'typecheck'
Description-Content-Type: text/markdown

# opik-rigor

[![PyPI](https://img.shields.io/pypi/v/opik-rigor)](https://pypi.org/project/opik-rigor/)
[![CI](https://github.com/ericwehmeyer/opik-rigor/actions/workflows/ci.yml/badge.svg)](https://github.com/ericwehmeyer/opik-rigor/actions/workflows/ci.yml)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](https://github.com/ericwehmeyer/opik-rigor/blob/main/LICENSE)

Statistical assertions and pinned-judge evaluation primitives for LLM test suites.

```python
assert_pass_rate(result, min_rate=0.9)   # not: assert pass_rate >= 0.9
```

Two primitives, done properly, with an audit trail. Optional Opik integration.

---

## The problem

You have an eval. It calls a model 20 times, 18 pass, and your test asserts
`pass_rate >= 0.9`. It goes green and you ship.

That test told you almost nothing, for three separate reasons.

**You measured a stochastic system once.** 18/20 is a sample, not a property. The
same system on the same inputs gives you 17/20 tomorrow and the suite goes red
with nothing having changed. So the team adds a retry, or drops the bar to 0.85,
and now the gate is measuring the team's patience rather than the model.

**Your judge moved.** The model id was `claude-3-5-sonnet-latest`. The provider
re-pointed the alias in March. Every score you recorded before March is not
comparable to every score after, and nothing anywhere says so. Or somebody
improved the wording of the rubric, which is the same problem wearing different
clothes.

**Your failures and your outages are in the same bucket.** The provider 500ed
four times, your harness counted those as failures, and now a quality gate is
reporting an infrastructure incident. Nobody notices, because the number moved in
a direction that looks like a real regression.

rigor fixes exactly these three things and nothing else.

---

## The primitives

### 1. Statistical gates

An assertion that accounts for having sampled a stochastic system *n* times rather
than measured it once. `assert_pass_rate` compares the **one-sided Wilson lower
confidence bound** against your bar, never the observed rate.

The practical consequence is worth internalising before you use it:

| observed | n | 95% lower bound | `min_rate=0.9` |
|---|---|---|---|
| 90% | 20 | 0.7383 | **fails** |
| 90% | 200 | 0.8596 | **fails** |
| 90% | 1000 | 0.8833 | **fails** |
| 95% | 200 | 0.9181 | passes |

**You cannot pass a 90% gate by scoring 90%.** The bound approaches the observed
rate from below and never reaches it, so you need real headroom above the bar —
and how much headroom is exactly what *n* buys you. That is the whole idea. A gate
that let 18/20 through would be telling you a story about 20 coin flips.

Three gates ship:

- `assert_pass_rate(result, min_rate=...)` — Wilson lower bound vs a floor.
- `assert_score_distribution(result, min_mean=..., min_p10=..., max_stddev=...)` —
  each threshold independent and optional, every violation reported at once. A
  mean gate alone passes a system that is excellent four times in five and
  unusable the fifth time, which is the failure users actually notice.
- `assert_no_regression(current, baseline)` — Mann-Whitney U against a recorded
  baseline. Nonparametric because judge scores are ordinal and routinely
  multi-modal; a t-test there is testing an assumption the data does not meet.

The failure message *is* the statistical report. It distinguishes the two failures
that matter, in as many words: **you missed the bar** versus **you did not sample
enough to tell**.

Every gate takes a `SampleResult`, and there are two ways to get one:

- `sample(fn, n)` calls `fn` *n* times and records each run's value, outcome,
  duration and exception.
- `sample_of(values)` wraps results you already have in the same object — for
  gating a run you collected elsewhere, or feeding a stored baseline into the
  regression gate, without pretending to re-run either.

`sample_of` is not `sample_over`. `sample_over(items, fn)` — map a function across
a dataset — is a roadmap item below and **does not exist yet**; `sample_of` takes
finished values and no function at all. Two names one letter apart, meaning
opposite things, is exactly the kind of thing to read twice.

### 2. A pinned judge

`PinnedJudge` refuses to run against an aliased model id — at construction, not
after a week of wasted compute:

```
judge 'summariser' refuses unpinned model id 'claude-3-5-sonnet-latest'. It contains
the alias token 'latest', which names whatever the provider is serving today rather
than one fixed version. A pinned id names one immutable model version ... An alias
re-points over time, which silently invalidates every score recorded against it.
```

An id is pinned when it carries no alias token (`latest`, `newest`, `current`,
`stable`, `default`) and ends in a release designator — a release number
(`claude-opus-5`, `claude-opus-4-8`), a date stamp (`claude-haiku-4-5-20251001`,
`gpt-4o-2024-08-06`), or an explicit version (`-v1`, `-2.1.0`). The property being
checked is *immutability*, not spelling: a retired id that still names one fixed
set of weights is pinned, and an id ending in a *word* (`gpt-4o`, `mistral-large`)
is not, because a word names a kind of model and kinds get re-pointed. What no
string can tell you is a provider's policy, so the one place that needs vendor
knowledge — providers that publish `<family>-<number>` as a moving pointer — is a
single documented table in `pinning.py`, and its limits are written down there.

It hashes its rubric and raises when the rubric changes underneath a baseline
(`accept_rubric_change=True` acknowledges it and records both hashes). And it
parses the judge's response strictly: an unparseable response raises, and is
**never** converted into a failing verdict — missing data is not evidence of
failure, and folding it into the failure bucket biases your pass rate by exactly
the judge's own flakiness rate.

### 3. An evidence log you cannot edit

Everything above writes to an append-only JSONL log with a fixed envelope and
**no delete, truncate, or rotate method**. That absence is a feature, and a test
enforces it — an audit trail you can quietly edit is not an audit trail.

---

## Quickstart

Every code block below was executed in a clean virtualenv holding nothing but the
published `opik-rigor` **0.1.1** wheel from PyPI — the release this one supersedes
— and every message, number and hash below is that run's own output. Every block
that prints something was then re-executed unchanged against the 0.2.0 source tree
on 2026-08-14 and reproduced byte for byte. The install measurements below were
not re-taken, and name the release they were taken on. The only editing is
hard-wrapping long lines to the page width. Nothing is elided, abbreviated, or
retyped from memory — where an earlier revision of this file showed `...` inside a
hash, that was illustrative text under a claim of verbatim, and it is gone.

```bash
pip install opik-rigor
```

**What that pulls in.** opik-rigor's own code is 0.33 MB in 0.1.1, and 0.2.0 adds
about 32 KB of source to that with the worked-example module described below. It
requires NumPy and SciPy, which come to **180.6 MB on disk** between them (numpy
2.5.2 and scipy 1.18.0 on CPython 3.14/Windows, counting the `numpy.libs` and
`scipy.libs` directories that carry the bundled BLAS/LAPACK). NumPy is used by
every gate. SciPy is used by exactly one function — `assert_no_regression`, for
`scipy.stats.mannwhitneyu` — and is imported on first call rather than at package
import, so a suite that never calls that gate never loads SciPy and does not pay
for it at import time. Both stay required, so `pip install opik-rigor` continues
to give you a working `assert_no_regression`.

**What that costs you at the prompt**, which is the number you feel: measured on
0.1.1 with `pip install --no-cache-dir opik-rigor` into an empty virtualenv
(Python 3.14.4, Windows), **54 seconds**, and the virtualenv grows from **11.5 MiB
to 192.8 MiB** across four packages. That is the disk figure above plus the
interpreter's own baseline; nothing else is hiding in it. The `[opik]` extra costs
a great deal more — see [Optional extras](#optional-extras) before you add it.

A worked example rubric ships **inside the package**, so the install gives you
something to point the judge at:

```bash
python -c "import opik_rigor, shutil; shutil.copy(opik_rigor.example_rubric_path(), 'rubric.md')"
```

Read it, then edit it into your own — a rubric is the measuring instrument, and
one copied from a library measures the library's idea of quality rather than
yours. It deliberately says nothing about JSON: `PinnedJudge` appends the
response-format instruction to the prompt itself, so a rubric that restates it
sends the same block twice. Then:

```python
from opik_rigor import EvidenceLog, FakeAdapter, PinnedJudge, assert_pass_rate, sample

log = EvidenceLog("evidence.jsonl")
adapter = FakeAdapter(  # a real judge would be AnthropicAdapter("claude-...-20250929")
    responses=['{"pass": true, "score": 5}'] * 9 + ['{"pass": false, "score": 2}'],
    seed=1,
)
judge = PinnedJudge(adapter, "rubric.md", log, name="summariser")

result = sample(lambda: judge.evaluate("Summarise this.", "A summary."), 20, evidence=log)
assert_pass_rate(result, min_rate=0.9, evidence=log)
```

This **fails**, and the failure is the point:

```
opik_rigor.distribution.PassRateError: pass rate gate failed: 18/20 passed (observed
0.9000); one-sided 95% Wilson lower bound 0.7383 < min_rate 0.9000. Two-sided 95%
interval [0.6990, 0.9721]. The observed rate 0.9000 clears min_rate 0.9000 but the
lower bound does not: this is an underpowered sample, not a demonstrated failure.
20 runs cannot distinguish a system at 90.0% from one at 73.8%. The observed rate
sits exactly on min_rate, and the lower bound approaches the observed rate from
below without ever reaching it: no sample size clears this bar at exactly 0.9000.
The system needs real headroom above min_rate, or min_rate has to come down.
```

`assert pass_rate >= 0.9` would have gone green on that sample.

Same judge, same seed, sampled properly and gated at a bar it can actually
defend. Change `20` to `200` and `min_rate` to `0.8` — and keep the report the
assertion hands back, because on success it prints nothing at all:

```python
report = assert_pass_rate(result, min_rate=0.8, evidence=log)
print(
    f"passed={report['passed']}  observed={report['pass_rate']:.4f}  "
    f"lower_bound={report['lower_bound']:.4f}  min_rate={report['min_rate']}"
)
```

```
passed=True  observed=0.9150  lower_bound=0.8768  min_rate=0.8
```

Note `report['pass_rate']` under the word `observed`. That mismatch is not a typo
here — it is the roadmap's "the key names are not guessable" complaint in its
sharpest form, and it is why this block now shows the `print` rather than a line
of output with no way to produce it. The full key set on success is `gate`,
`label`, `passed`, `n`, `successes`, `failures`, `pass_rate`, `lower_bound`,
`interval_lower`, `interval_upper`, `min_rate`, `confidence`, `method`.

And the other primitive. Append the line `Be stricter about caveats.` to
`rubric.md` — the exact text matters, because the second hash below is a hash of
the result — then build the same judge against the same evidence log:

```python
judge = PinnedJudge(adapter, "rubric.md", log, name="summariser")
```

```
opik_rigor.errors.RubricDriftError: rubric drift for judge 'summariser': evidence log
last recorded 556c1383350d73d71235e40c719cdf816bec8a5693cc4750ad01ae421128dc5d, rubric
file now hashes to 833331e342e2a6c05c4759621fde7b606fbb4a22dfadf283c35ca9d5176138c8.
Scores before and after this change are not comparable. Pass
accept_rubric_change=True to acknowledge and record the change.
```

The first hash is the rubric exactly as it ships in the wheel; the second is that
file with the line above appended. Both are printed in full, because half a hash
is not something you can compare against a log — the message never abbreviates
one, and an earlier revision of this README showed `...` in the middle of both.

A full worked example — corpus, judge, both gates, a baseline, a simulated
regression, and the audit trail — **ships inside the wheel** and runs offline with
no credentials:

```bash
python -m opik_rigor.examples.summarise_eval --seed 7 --n 40
```

A module, not a path, deliberately. This line used to read
`python examples/summarise_eval.py`, and `examples/` is a directory in the git
tree that is not in the artifact — an install contains `opik_rigor/` and
`opik_rigor-0.2.0.dist-info/` and nothing else — so the command this quickstart
ended on could not be run by anyone who had followed the install instruction four
paragraphs above it. The walkthrough of what it prints, screen by screen, is
[in the repository](https://github.com/ericwehmeyer/opik-rigor/blob/main/examples/README.md).

> **This one command is the exception to the paragraph at the top of this
> section.** Everything else here was first run against the published 0.1.1 wheel,
> and `opik_rigor.examples` is not in 0.1.1 — **0.2.0 is the release that ships
> it**, so this line was run against the 0.2.0 source tree rather than against a
> published wheel. If it reports `No module named`, the install predates 0.2.0 and
> `pip install --upgrade opik-rigor` is the fix rather than a checkout. A
> project's long description is also frozen at upload, so the page PyPI renders
> for an earlier version is that version's README — for 0.1.1, ellipses and broken
> paths and all — and no correction made here ever reaches it. This file
> self-heals in the repository; the index does not.

---

## Optional extras

```bash
pip install "opik-rigor[opik]"     # log samples and verdicts to Opik
pip install "opik-rigor[pytest]"   # @pytest.mark.rigor_repeat, rigor_judge fixture
```

**Opik** — two functions, not a framework. They live in
`opik_rigor.integrations.opik`, **not** at the package root:

```python
from opik_rigor.integrations.opik import log_assertion_to_opik, log_sample_to_opik
```

Neither name is on the package root, and importing either from there raises
`ImportError`. That is a deliberate arrangement rather than an oversight to be
tidied away: `opik_rigor`'s `__init__` imports no integration at module scope — a
test asserts it in a subprocess — so that `import opik_rigor` can never drag in a
vendor SDK. Putting these two names on the package root is the one change that
would break the property the last paragraph of this section is about.

`log_sample_to_opik` maps a sample to a trace with one span per run (a run that
*raised* is visibly distinct from one that *failed*), and `log_assertion_to_opik`
maps a gate's verdict to feedback scores. The verified API surface, the version
bounds, the reasoning behind them, and a correction to a claim this project got
wrong about Opik's own documentation are all in
[COMPATIBILITY.md](https://github.com/ericwehmeyer/opik-rigor/blob/main/COMPATIBILITY.md).

**"Two functions, not a framework" describes rigor's side of the seam, not the
install.** Measured on 0.1.1 the same way as the core install above, into its own
empty virtualenv: `pip install --no-cache-dir "opik-rigor[opik]"` takes **4
minutes 48 seconds** and produces a **414.9 MiB** virtualenv holding **74
packages**, against 54 seconds, 192.8 MiB and 4 packages for the bare install.
What arrives with it includes litellm (101.9 MiB), openai, tokenizers,
huggingface-hub, hf-xet, tiktoken, sentry-sdk, three `tree-sitter` grammars,
pydantic, boto3 type stubs —
and pytest, which Opik pulls in for its own plugin. Add the extra when you want
the dashboard; do not add it because the sentence above made it sound small.

**pytest** — `@pytest.mark.rigor_repeat(n=50, min_rate=0.9)` runs a test *n* times
and applies the gate to the outcomes. A body that returns passes; one that raises
`AssertionError` is a failure; anything else is an exception, counted separately.
Registered as `rigor`, and verified to co-exist with Opik's own pytest plugin.

**The core never imports either.** If a vendor SDK breaks, you lose a dashboard,
not a test suite.

---

## Designed to support SR 11-7 model validation

[SR 11-7](https://www.federalreserve.gov/boarddocs/srletters/2011/sr1107.pdf)
(Federal Reserve / OCC 2011-12, April 2011) is the US supervisory guidance on model
risk management. It is not a checklist and this library does not make you compliant
with it — compliance is an institutional programme with independent review,
governance, and validators, and no Python package delivers that.

What rigor does is make three things it asks for cheap to produce as a by-product
of testing, rather than reconstructed from memory at review time:

**Conceptual soundness** is documented where the choices are made. Why Wilson over
Clopper-Pearson (coverage vs power at the small *n* an eval can afford), why
Mann-Whitney over a t-test (ordinal, non-normal, multi-modal scores), and why a
parse failure is never a fail-verdict — all in the docstrings of the functions
that implement them, not in a slide deck that drifts away from the code.

**Ongoing monitoring** is what the gates are. A recorded baseline carries a sha256
of its own contents and is verified on load, so a regression cannot be made to
disappear by editing the file it is compared against.

**Outcomes analysis** is what the evidence log holds: every verdict, every sample,
every gate decision, appended and never rewritten, each carrying the judge's pinned
model id and the sha256 of the exact rubric revision that produced it. The question
"what exactly was this number measured with, and has that changed since?" has a
file-backed answer.

**Effective challenge** is a property of your organisation, not your tooling. But
challenge needs something to bite on, and "the rubric hashed to `e62bdbb2…` and the
judge was pinned to `claude-sonnet-4-5-20250929`" is a materially better starting
point than "we ran the eval and it looked fine."

If you work in a regulated setting, treat this as plumbing that makes evidence
falsifiable and cheap — and treat the guidance as your compliance team's to
interpret.

---

## Roadmap

0.2.0 is still two primitives done rigorously; it fixes and packages, and adds no
third. These are the good ideas that were deliberately parked, most of them
discovered by writing the example and finding the library annoying to use:

- **A non-raising `check_*` beside each `assert_*`.** Today success returns a
  report dict and failure carries the same numbers on `exc.stats` — and
  `underpowered`/`runs_needed` exist only on the failure path. Printing "what did
  the gate conclude" means `try/except` around every gate.
- **Typed report objects.** The reports are `dict[str, Any]`: no autocomplete, no
  typo protection, and the key names are not guessable (`lower_bound` vs
  `interval_lower` vs `min_rate`).
- **`sample_over(items, fn)`.** `sample(fn, n)` hands `fn` nothing, so every caller
  writing a real eval reimplements the same dataset-cycling closure. Not to be
  confused with the shipped `sample_of(values)`, which wraps values you already
  have; the collision of names is itself a reason this one may end up called
  something else.
- **A seedable callable `FakeAdapter`.** `seed=` is rejected in exactly the
  `responses=<callable>` mode a realistic fake needs — the one shape that can react
  to its input is the one that cannot take a seed.
- Cost and latency gates. `SampleResult` already records per-run durations.
- Clopper-Pearson as an option, for settings that need guaranteed coverage.
- A configurable score range (currently fixed at 1–5).

Explicitly **not** planned: becoming an eval platform. Datasets, dashboards,
prompt management, and orchestration are what Opik is for, and rigor integrates
with it rather than competing.

---

## Development

Linux and macOS:

```bash
python -m venv .venv
.venv/bin/python -m pip install -e ".[dev]"
.venv/bin/python -m pytest tests examples
.venv/bin/python -m ruff check src tests scripts
```

Windows (PowerShell), which is where this project is primarily developed — the
interpreter is under `Scripts\`, not `bin/`, and it is `python.exe`:

```powershell
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -e ".[dev]"
.\.venv\Scripts\python.exe -m pytest tests examples
.\.venv\Scripts\python.exe -m ruff check src tests scripts
```

`pytest` on its own honours `testpaths = ["tests"]` from `pyproject.toml` and does
not collect `examples/`, so it reports fewer tests than `pytest tests examples`.
Neither number is wrong; they are answers to different questions.

The suite is green with **no credentials and no network**. Anything needing a live
provider is marked `requires_network` or `requires_opik` and deselected in CI. The
Opik integration is tested against a real Opik client via
`opik.record_traces_locally()` pointed at a loopback server — not against mocks.

Two environments are maintained: one without Opik (the suite must pass without it)
and one with, for the integration and plugin co-installation tests.

## License

MIT — see
[LICENSE](https://github.com/ericwehmeyer/opik-rigor/blob/main/LICENSE). The full
text also ships inside the wheel, under
`opik_rigor-<version>.dist-info/licenses/LICENSE`; the link is absolute because
PyPI renders this file with no repository behind it, so a relative one 404s from
the project page.
