Metadata-Version: 2.4
Name: paired-evalkit
Version: 0.2.0
Summary: Per-query eval results and paired-bootstrap significance testing for retrieval and LLM systems
Project-URL: Homepage, https://github.com/skbugudi/evalkit
Project-URL: Issues, https://github.com/skbugudi/evalkit/issues
Author: Shanth Kumar Bugudi
License: MIT
License-File: LICENSE
Keywords: bootstrap,evaluation,llm,rag,retrieval,significance
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: anthropic>=0.120.2
Requires-Dist: httpx>=0.28.1
Requires-Dist: numpy>=2.0
Requires-Dist: pydantic>=2.13.4
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: typer>=0.27.1
Description-Content-Type: text/markdown

# evalkit

Per-query eval results and paired-bootstrap significance testing for retrieval
and LLM systems.

Answers the question most eval harnesses can't: **did that change actually
improve anything, or did the number just move?**

```python
from evalkit import run, paired_bootstrap

baseline = run("baseline", search_v1, queries, qrels, ["recall@10", "ndcg@10"])
candidate = run("candidate", search_v2, queries, qrels, ["recall@10", "ndcg@10"])

v = paired_bootstrap(candidate, baseline, "ndcg@10")
print(
    f"{v.observed:+.4f}  95% CI [{v.ci_low:+.4f}, {v.ci_high:+.4f}]  p={v.p_value:.4f}"
)
# +0.0207  95% CI [+0.0106, +0.0320]  p=0.0003
if not v.significant:
    raise SystemExit("difference is within noise — don't ship on this evidence")
```

## Install

```bash
uv add paired-evalkit
# or
pip install paired-evalkit
```

```python
import evalkit  # the import name is `evalkit`, not `paired_evalkit`
```

**The distribution is `paired-evalkit`; the import is `evalkit`.** `evalkit` on
PyPI is an unrelated project (evolutionary algorithms), so the published name
carries the differentiator this package actually has. If you happen to have both
installed they share the `evalkit` module name and will conflict — uninstall one.

To pin a specific commit instead of a release:

```bash
uv add "evalkit @ git+https://github.com/skbugudi/evalkit@<sha>"
```

Pin a rev rather than a bare URL. Unpinned, a resolve tracks whatever `main`
holds, so an upstream merge changes your CI with no commit in your repo — and a
metric-definition change makes previously saved baselines incomparable with
nothing saying so.

Python 3.12+. Runtime dependencies: `numpy`, `httpx`, `pydantic`, `pyyaml`,
`typer`, `anthropic` (the last used only by the LLM judge).

## Why

**1. Aggregates hide the thing you need to know.** A mean tells you a system
scores 0.7377. It can't tell you *which* queries it won or lost, so it can't
tell you whether A beats B or merely averages higher. Two systems with identical
means can disagree on every single query.

This isn't hypothetical. On real SciFact data a hybrid retriever beat a dense one
on the aggregate (+0.0091 nDCG) while **losing on more queries than it won** —
14 wins, 16 losses, 70 ties. The aggregate said ship it. It was noise (p = 0.65).

**2. "The number went up" is not evidence.** A paired bootstrap resamples your
query set with replacement 10,000 times and reports the spread. If the 95%
interval straddles zero, a different sample of queries could plausibly have
reversed your verdict.

**3. Underpowered evals produce meaningless nulls.** Simulated at n=100:

| true effect | detected |
|---|---|
| +0.02 | 12% of the time |
| +0.05 | 60% |
| +0.10 | 100% |

A 3-point improvement is *missed more often than found* at n=100. Knowing that
before you run tells you whether the eval is worth running at all.

## API

### `run(name, retriever, queries, qrels, metrics) -> RunResult`

```python
@dataclass
class QueryResult:
    query_id: str
    retrieved: list[str]
    metrics: dict[str, float]  # {"recall@1": 1.0, "ndcg@10": 0.63}


@dataclass
class RunResult:
    name: str
    per_query: list[QueryResult]  # always retained

    def mean(self, metric: str) -> float: ...  # aggregates are DERIVED
```

- **`retriever`** — any callable `(query) -> list[str]`, ids best first.
- **`queries`** — any objects exposing `.id`. Bring your own type.
- **`qrels`** — `dict[str, set[str]]`, query id to relevant ids.
- **`metrics`** — spec strings parsed against `runner.METRICS`: `"recall@10"`,
  `"ndcg@10"`. Add your own with `METRICS["precision"] = precision_at_k`.
  **Note:** this is the *runner* metric table, shape `(retrieved, relevant, k)
  -> float`. It is separate from the `@register_metric` registry used by the
  target/gate path — see "Two metric tables" below.

A query missing from `qrels` **raises** rather than scoring 0.0. Silently
scoring an unjudged query as zero makes a broken eval look like a worse system,
which sends you to debug the model when the harness is what broke.

### `paired_bootstrap(a, b, metric, n_resamples=10_000, seed=42) -> BootstrapResult`

Returns `observed`, `ci_low`, `ci_high`, `p_value`, and `significant`, plus
provenance (`metric`, `name_a`, `name_b`, `n_queries`, `n_resamples`).

**Paired is load-bearing.** It resamples query *ids* and looks up both systems'
scores for those same ids. Query difficulty dominates the variance, and pairing
subtracts it out before the statistics see it — comparing independently drawn
samples throws away most of your power. Alignment is by query id, never by row
position; there's a test asserting that shuffling one system's rows changes
nothing.

Implementation details that matter:

- CI from percentiles of the resampled distribution.
- p-value from a **centred** null (`resampled - observed`), which simulates a
  world where the two systems are equivalent.
- `p = (extreme + 1) / (n_resamples + 1)` — floors p at 1/10001 rather than
  reporting exactly 0. Never seeing something in 10,000 draws is evidence it's
  rare, not impossible.

**Cost:** the bootstrap resamples results you already collected. It never
re-runs a query, so 10,000 resamples cost nothing beyond CPU. Run the eval once,
persist the per-query rows, bootstrap offline as often as you like.

### `corpus` — optional BEIR loaders

`load_corpus()`, `load_queries()`, `load_qrels()` read BEIR-format JSONL and TSV.
Convenience only; `run()` never requires them.

> **Known limitation:** `corpus.DATA` resolves relative to the installed module,
> so the loaders only work from a source checkout with a sibling `data/`
> directory. If you're bringing your own data — which you are, if you installed
> this as a package — build `queries` and `qrels` yourself. See
> [examples/custom_client](examples/custom_client/). Making the data root
> configurable is tracked as the first API change.

## Extension points

A client project adds behaviour by decorating its own functions. Nothing in this
package needs editing.

```python
from evalkit import register_metric, register_aggregate, register_loader


# per-case: (TargetResponse, GoldenCase) -> float
@register_metric("precision", higher_is_better=True)
def precision(response, case) -> float:
    hits = sum(1 for i in response.ranked_ids[:3] if i in case.relevant)
    return hits / 3


# the direction that makes this one a regression is the opposite one
@register_metric("unsupported_claims", higher_is_better=False)
def unsupported_claims(response, case) -> float:
    return float(count_unsupported(response.text))


# whole-run: (Sequence[CaseResult]) -> float
@register_aggregate("cost_per_query", higher_is_better=False)
def cost_per_query(rows) -> float:
    return sum(r.response.meta.get("cost_usd", 0.0) for r in rows) / len(rows)


@register_loader("parquet")  # (Path) -> GoldenSet
def load_parquet(path) -> GoldenSet: ...
```

Duplicate names raise. A name may be a per-case metric **or** an aggregate,
never both, so check resolution is unambiguous.

### `higher_is_better` is required

It has no default, on purpose. Which direction counts as a regression is a
property of the metric, not of the check that reads it — recall falling is a
regression, an unsupported-claim count falling is an improvement, and
`max_item_share` and `entropy` point opposite ways on the very same run. Nothing
outside the metric can know.

Defaulting it to `True` would mean the first lower-is-better metric anyone
registers silently gates backwards: a regression gate that passes the exact
change it exists to block, while still reading as coverage. `polarity(name)`
returns it; unknown names raise rather than assuming a direction.

### Two metric tables — read this before registering anything

There are currently two, because the retrieval path predates the target path:

| | shape | used by |
|---|---|---|
| `runner.METRICS` | `(retrieved, relevant, k) -> float` | `run()` with spec strings like `"recall@10"` |
| `@register_metric` | `(TargetResponse, GoldenCase) -> float` | the Target/gate path |

Register into the one matching the path you're using. A metric registered in the
wrong table is simply never found. These converge in a later release.

## Targets and goldens

```python
from evalkit import CallableTarget, HttpTarget, load_goldens

goldens = load_goldens("goldens/cases.jsonl")  # jsonl | yaml | csv

local = CallableTarget(my_ranker, name="local")

with HttpTarget(
    url="http://localhost:3001/api/search",
    name="prod",
    build_payload=lambda case: {"q": case.query},
    extract=lambda body: [r["id"] for r in body["results"]],
    meta_keys=("cost_usd", "my_custom_field"),
) as remote:
    response = remote.invoke(goldens.get("q1"))
```

`HttpTarget` retries once on a 5xx or a transport failure, never on a 4xx, and
measures latency across the whole attempt sequence. It reuses one connection
pool, so use it as a context manager (or call `.close()`).

Everything fails loudly: an unreachable target, a malformed golden, a JSON body
that isn't an object, or a golden file that loads zero cases all raise rather
than producing a zero that looks like a quality regression.

## Built-in metrics

Importing `evalkit` registers these. `registered()` returns the live list.

| Name | Kind | Polarity | Notes |
|---|---|---|---|
| `recall@1/3/5/10/20`, `ndcg@10` | per-case | higher better | adapts the verified core functions to `(TargetResponse, GoldenCase)` |
| `latency_p50/p95/p99`, `latency_mean` | aggregate | lower better | **nearest-rank**, not `floor(n·p)` |
| `max_item_share`, `max_item_share@1/3/10` | aggregate | lower better | fraction of **queries** an item appears in |
| `entropy` | aggregate | higher better | normalised to [0, 1] so catalogue sizes compare |

Add a `k` without editing the package:

```python
from evalkit.metrics.retrieval import register_retrieval_metric
from evalkit.metrics.core import recall_at_k

# registers "recall@50"
register_retrieval_metric("recall", recall_at_k, 50, higher_is_better=True)
```

**Percentiles are nearest-rank on purpose.** The common hand-written form
`sorted[floor(n * p / 100)]` returns the *maximum* whenever `n · p / 100` is an
integer — at p95 that is every run size divisible by 20. A 20-case eval then
reports its worst case and labels it p95.

**`max_item_share` counts queries, not slots.** If 100 queries return 3 items
each and one item appears in 40 of them, its share is 0.40, not 0.13. The
question "am I showing everyone the same thing" is about queries; a slot-count
denominator dilutes exactly that signal. Ids are deduped within one response, so
a target that repeats an id cannot inflate its own share.

### LLM-as-judge

```python
from evalkit import register_metric
from evalkit.metrics import LLMJudge

judge = LLMJudge(
    rubric="Score 5 if the answer is correct and complete, 1 if it is wrong.",
    model="claude-haiku-4-5",  # required — there is no default
    scale=5,
    cache_path=".evalkit/judge-cache.json",
)
register_metric("helpfulness", higher_is_better=True)(judge)
```

The model is a **required** argument. It is part of the eval's definition, not a
runtime detail: swapping judges makes scores incomparable to a stored baseline,
which is a silent baseline invalidation. It is folded into the cache key, so a
swap is a cache miss rather than a reuse of the previous model's grades.
Grading is a forced, `strict` tool call — there is no free-text parsing step
that can fail partway through a paid run. Latency is deliberately *not* hashed,
so rerunning an unchanged eval costs nothing.

## Checks

A check declares **either** a fixed threshold **or** a comparison against the
stored baseline — never both, never neither.

```yaml
checks:
  - id: recall_floor
    metric: recall@10
    op: ">="            # "<=" | ">=" | "=="
    threshold: 0.60

  - id: p95_ceiling
    metric: latency_p95
    op: "<="
    threshold: 800
    level: warn          # "block" (default) | "warn"

  - id: no_ndcg_regression
    metric: ndcg@10
    vs_baseline: no_significant_regression
    alpha: 0.05
```

```python
from evalkit import load_checks

specs = load_checks("evals/checks.yaml")
```

Declaring both leaves two verdicts for one check with no rule for combining
them. Declaring neither is worse — a line in a config file that gates nothing
while reading as coverage. Both raise at load time, as do an unknown key (a
typo'd `treshold` would otherwise gate the wrong thing), a duplicate id, a file
that loads zero checks, and an `alpha` on a threshold check, which has no
significance test to apply it to.

Metric names are **not** resolved at load time — a config may legitimately name
a metric the client registers later.

### `scope` — gate on named cases

A check with no `scope` covers every case in the run. `scope` restricts it to
the ones you name:

```yaml
checks:
  # the cases that already work hold an absolute floor
  - id: controls_hold
    metric: ndcg@3
    scope: [cp-surry-any, cp-newtown-pub, cp-bondi-any]
    op: ">="
    threshold: 1.0

  # everything else, including the known-broken cases, only may not get worse
  - id: no_regression
    metric: ndcg@3
    vs_baseline: no_significant_regression
```

That pairing is the point. On a set where some cases are known-broken and
baselined at zero, a single absolute threshold across everything is red on day
one — and a gate that is red on day one is switched off inside a week. Scoped
floors let the working cases hold a hard line while the broken ones are gated
only on further degradation, so the gate is green today and still fails the
moment something real regresses. As each defect is fixed you promote a new
baseline and the old failures become the new floor.

It is also what keeps per-case gating out of every client. Without it, each
project reimplements the same loop over `per_query` in its own runner, which is
the boundary this package exists to hold: clients write their cases, their
metrics and their thresholds — not their gate logic.

`scope` is orthogonal to the either/or above: it works on threshold and
`vs_baseline` checks alike.

**Two ways a scoped check can lie, and both raise.**

*Naming a case that does not exist* is refused at **load**, when you pass the
ids you have:

```python
specs = load_checks("evals/checks.yaml", known_case_ids=goldens.ids())
# CheckError: scope names 1 case(s) that do not exist: cp-newton-pub
```

A typo'd case id is a config error, and a config error should surface when the
config is read — not after a full scored run has been paid for. `known_case_ids`
is optional, so a client that has not loaded its goldens yet can still read
config; supply it wherever you can.

*Passing on cases it never evaluated* is refused at **run**, by `select`:

```python
spec.select(result.per_query)
# CheckError: check 'controls_hold': scoped to 3 case(s) but 1 produced no
# result: cp-bondi-any. A scoped check must evaluate every case it names —
# a floor checked on a subset is a floor that was not checked.
```

Reaching that error means the case was known and produced no row — it errored,
timed out, or the runner dropped it. Evaluating the survivors would report the
same green as a full pass, and the difference is exactly the case that fell
over. An empty `scope: []` is refused outright, as is a repeated case id, which
would be weighted twice.

Report `spec.coverage(rows)` alongside the verdict — `controls_hold: pass
(4/4 cases)`. A scoped check that prints only "pass" leaves the reader unable to
tell a gate that held from a gate that matched nothing.

### Comparison

`<=` and `>=` are exact. A threshold is a line, and a tolerance around it just
moves the line somewhere undocumented.

`==` uses a **fixed** tolerance —
`math.isclose(observed, threshold, rel_tol=1e-9, abs_tol=1e-12)`. Exact equality
would mostly work: essentially every legitimate `==` in an eval config is a
zero-tolerance count (`error_count == 0`) or a boundary rate
(`pass_rate == 1.0`), and nobody sanely writes `ndcg@10 == 0.7377`. The problem
is not wrongness but a confusing failure — a report printing `0.0000` beside a
check that failed on `1e-17`. The `abs_tol` is the load-bearing half: `isclose`
with only a relative tolerance never matches against zero, which is the main
case.

The tolerance takes no argument and is not a config key. A per-check tolerance
is a strictness dial, and a gate whose strictness is tunable per check is a gate
someone quietly loosens under deadline.

**For a zero-tolerance count, prefer `op: "<="` with `threshold: 0`.** It is
unambiguous and sidesteps what `==` means on a float entirely.

```yaml
  - id: no_unsupported_claims
    metric: unsupported_claims
    op: "<="
    threshold: 0
```

## Example

[examples/custom_client](examples/custom_client/) evaluates an HTTP-style client
against a hand-written golden set, with no network. Run it:

```bash
uv run python examples/custom_client/evaluate.py
```

```
metric        baseline  candidate    delta
------------------------------------------
recall@1        0.2000     0.7000  +0.5000
ndcg@10         0.7750     1.0000  +0.2250

Paired bootstrap (10,000 resamples):
  recall@1   +0.5000  95% CI [+0.1000, +0.9000]  p=0.0203  REAL
  recall@5   +0.0000  95% CI [+0.0000, +0.0000]  p=1.0000  not significant
  ndcg@10    +0.2250  95% CI [+0.0899, +0.3566]  p=0.0006  REAL
```

## Development

```bash
uv sync
uv run pytest          # 101 tests, all offline
uv run mypy src tests  # strict
uv run ruff check .
```

The four core modules are copied verbatim from a verified source and are
deliberately byte-identical to it — `ruff` line-length is pinned to 88 so
formatting never rewrites them.

The significance tests are worth reading before trusting the statistics: they
construct systems whose answer is known in advance (one better on every query;
a system compared against itself) and assert the bootstrap agrees. Calibration
was also checked by simulation — 300 trials under a true null produced a 7.7%
false-positive rate against a nominal 5%, which is the mild anti-conservatism
the percentile bootstrap is known for at n≈100. Treat a borderline p = 0.04 with
proportionate caution.

## License

MIT
