# Evalogy leaderboard — meridian

Results as of 2026-06-12. Deterministic engine (`det-4`), k=3: the headline is **strict pass^3** — a probe counts only if it passed every one of its 3 repetitions; `mean` alongside is capability when the dice land well. Protocol: [ADR-0006](adr/0006-meridian-and-the-leaderboard-protocol.md).

| # | Model | pass^3 | mean | none | resolvable | unresolvable | void | ANSWER fmt | scorer | run date | notes |
|--:|---|--:|--:|--:|--:|--:|--:|--:|---|---|---|
| 1 | anthropic/claude-sonnet-4-6 | **86.4%** | 90.9% | 100% | 100% | 40% | 100% | 98.5% | det-4 | 2026-06-11 |  |
| 2 | openai/gpt-4o | **45.5%** | 54.5% | 0% | 83.3% | 0% | 100% | 22.7% | det-4 | 2026-06-12 | skips the CRM leg of cross-silo joins |
| 3 | openai/gpt-4o-mini | **27.3%** | 40.9% | 0% | 16.7% | 0% | 100% | 92.4% | det-4 | 2026-06-12 | fails the joins and all five ties |

## Methodology

- Every row is one run of the `meridian` org: all probes, 3 epochs each, scored on accuracy, provenance (mechanical, per-field for the CRM — credited only for data that actually came back), and committed refusal.
- The per-category columns are strict pass^k by conflict type. `unresolvable` is the column to watch: it measures whether a model fabricates a tie-break rather than refusing when two systems of record disagree with equal authority.
- The `meridian` blueprint is public — it is the answer key. Honesty over purity: results are date-stamped, training-data contamination becomes more likely over time, and seeded variants are the planned mitigation (see ADR-0006).
- Evalogy scores policy execution, not discovery: the agent is told the reconciliation policy; the question is whether it executes it reliably.
- `ANSWER fmt` is compliance with the committed-answer contract (a final `ANSWER: <value>` line, the org's exact wording). Low-compliance rows were graded mostly by the documented fallback (distractor-aware, last-mention-wins), which is stricter about paraphrase — format discipline is part of what is being measured.

This file is generated from [`leaderboard.rows.json`](leaderboard.rows.json) — the committed source of truth (ADR-0010). Never edit the table by hand; CI regenerates it from the manifest and fails on drift, and `evalogy leaderboard verify` re-derives every log-backed row from its committed log (ADR-0012). To add or update a row, produce a run, commit its log under `logs/leaderboard/`, extract the row, merge it into the manifest, then regenerate and verify:

```bash
evalogy run --suite meridian --model <provider/model> --k 3 --seed 0 \
  --scaffold baseline   # deterministic grading is the default
cp ~/.evalogy/runs/<id>/log.eval logs/leaderboard/<run>.eval
evalogy leaderboard extract logs/leaderboard/<run>.eval  # -> a manifest row
evalogy leaderboard render --manifest docs/leaderboard.rows.json -o docs/leaderboard.md
evalogy leaderboard verify --manifest docs/leaderboard.rows.json
```
