# FEATURE — Judges: the evaluation primitive and its accuracy record

**Status: LIVE 2026-08-16.** Contract + two modes + Selector + accuracy ledger +
Cohen's-kappa calibration. First consumer migrated: the Hindsight replay judge.

> **A judge nobody measures is an opinion with a schema.** Engram VISION §3.3
> makes a Judge a first-class specialist — "its own contract and its own
> Hindsight accuracy tracking" — and §3.6 makes the Selector its sibling. Before
> this module the platform ran three unrelated judges with three verdict
> vocabularies and **zero** accuracy records. That is what this replaces.

## The five files

| File | What it owns |
|---|---|
| `judge.py` | `JudgeContract` (the declaration) + `Judge` (the two modes) + the D-15 guardrail |
| `selector.py` | `Selector` — pick among N by comparison, never by absolute score |
| `ledger.py` | Writes `platform.judge_verdict`; stamps the agreement bit |
| `calibration.py` | Cohen's kappa + the usable/production bands. Pure, no DB |
| `ai_judge.py` | The original strict-JSON pass/fail evaluator. **Kept** — it is the funnel `Judge` is built on, and the test tier uses it |

Host consumer (agreement + calibration + admin surface):
[`aidream/services/judge_accuracy/FEATURE.md`](../../../../aidream/services/judge_accuracy/FEATURE.md).

## The invariants

1. **Declare before you judge.** A judge without a `JudgeContract` has no
   identity, so it has no accuracy record. `key` + `version` IS that identity —
   **bump `version` on any rubric or contract change**, or two different judges
   silently share one track record.
2. **Rank when a reference exists; score only against a NAMED rubric** (D-14,
   research #7). `mode='rubric'` with no `rubric_name`/`rubric` is refused at
   *construction*. There is no unnamed absolute score, and no 0-100 scale
   anywhere — verdicts are an enumeration, nuance rides `confidence`.
3. **🚨 The judged unit never authors and never sees its judge's rubric (D-15).**
   Rubrics live on the contract, nowhere else. Three layers, each sufficient
   alone, each loud: `assert_not_self_authored` (subject == rubric author →
   refuse, before any paid call) · `_assert_rubric_isolation` (the rubric text
   appears inside the subject → refuse; this is the realistic way the rule dies,
   a caller "helpfully" giving the agent its grading criteria) · every ledger row
   stamps `rubric_fingerprint` + `rubric_provenance` + `rubric_author`, so a
   violation that lands anyway is visible afterwards.
4. **The contract owns the verdict vocabulary.** The funnel path enforces it in
   the provider schema; the slot path RAISES on an out-of-vocabulary verdict and
   never coerces one. A drifted judge agent is a defect, not a rounding error.
5. **Judges are platform agents under the hood.** Declare `mandate` + pass the
   `NamedAgent` class and the judge runs as that DB-managed, user-swappable,
   versioned agent. Declare neither and it runs on the funnel with `model`.
   There is no third runner.
6. **Every invocation writes a ledger row**, and the row a live surface reads is
   always authored on live. See the first trap below — this one already bit.
7. **A ledger failure never sinks a verdict.** It screams (`vcprint` red + a
   logged warning) and returns `None`. The verdict is the product; the accuracy
   row is the record of it.

## The Selector

A tournament of comparative judgements — `champion = candidates[0]`, each
challenger compared against the incumbent, `better` unseats it. Deliberately
thin: it is `Judge` in comparison mode, so a Selector's accuracy is measured by
the same machinery, and a counterfactual replay of a discarded branch can later
stamp the agreement bit on the exact comparison that discarded it (§4.2).

* The constructor **refuses a rubric-mode contract.** There is no absolute-score
  path into the Selector, by construction. That is the specialization.
* A `same` verdict **keeps the incumbent.** Churning on a tie would make the
  result depend on candidate ORDER far more than on quality.

## Earned traps

* **🚨 A `ledger_id` is meaningless outside the database that minted it.** A
  contained replay (D-25) executes in a child process whose ONE database is the
  MIRROR, while the terminal `hindsight.replay` row is written on LIVE by a
  different process. Copying the child's `ledger_id` across would hand the live
  row a **mirror id** — the same class of mistake `envelope_terminal_fields`
  already refuses for the fork conversation id. The rule: **whoever writes the
  terminal row authors the ledger row on their own database.**
  `verdict_payload()` builds a row as plain data without writing it, so it can
  cross the boundary; `replay_queue_worker._land_judge_ledger` writes it on live
  and swaps in the real id. **This must never become a "am I on the mirror?"
  branch** inside the replay path — containment is a deployment input, not a
  code fork, and `test_replay_py_has_no_mirror_branch` fails the build if one
  appears. The producer writes unconditionally and always ships the payload; the
  live-row writer decides.
* **Two `run_mandated` twins exist** — `matrx_ai.mandates` (package-internal,
  injected resolver) and aidream's host-side one that defaults the principal
  from the ambient `AppContext`. A host judge MUST pass its own via
  `mandate_runner=`, or it silently loses that defaulting.
* **`organization_id` is required.** The ledger is a canonical entity table, so
  the column is NOT NULL, and a NULL org is never "global" — it is invisible.
  The writer refuses loudly rather than inventing one.
* **`JudgeVerdict` (ai_judge) ≠ `JudgeAssessment` (judge).** The former is
  AIJudge's pass/fail shape; the latter is this primitive's contract-typed one.
  Both are exported; do not merge them.

## Known gap (named, not hidden)

The pinned `hindsight_replay_judge` v1 agent's output schema has no
`confidence` or `evidence` field, so slot-run verdicts record
`confidence = 0.5` (the primitive's honest default) and an empty evidence list.
Closing it means authoring a v2 of that agent AND rebinding in **both** places
(`agents.py` CAPS *and* the DB mandate pin — `sync_declared_mandates` never rebinds an
existing row). Tracked in the Hindsight `FEATURE.md`.

## Verification

`uv run python scripts/_verify_judge_primitive.py` — 33 live checks against the
real DB with real LLM calls: comparative judgement + ledger row, the agreement
bit, rollup + calibration, both D-15 layers, the Selector tournament, and the
contained-replay write-back. `--skip-paid` runs the guardrails only.
