# ADR-047: Verifier Calibration & Judge Reliability

**Status:** Implemented 2026-07-03 (epic #417, v0.31.0)
**Date:** 2026-07-03
**Decision Makers:** Chris Joseph, LLM Council
**Council Review:** 2026-07-03 (4 models, balanced) — feedback incorporated: screening thresholds defined, blocking-path invariant operationalized, DoD covers P3/P4, reference consistency
**Related:** ADR-036 (Phase 2 calibration report — this implements that slice), ADR-034 (verify), ADR-015/017/018 (bias & randomization), ADR-022 (tiers), #397/#403 (error surfacing)

---

## Context

Operational evidence from driving five release epics through the verify gate
(2026-07-01→03):

1. **Confidence is miscalibrated.** On real code with zero blocking issues,
   confidence pins at 0.49–0.6 — below the 0.7 PASS threshold — producing
   multi-round "asymptotes" where each re-verify surfaces finer or inverting
   nits instead of converging (documented across PRs #364/#387/#395…). PASS is
   effectively unreachable for scrutinized prose/code even when reviewers find
   nothing blocking.
2. **UNCLEAR conflates three unlike things:** infra failure (chairman call
   errored — billing outage, #397), genuine low confidence, and global
   timeout. Callers (CI gates, epic-loop) need to treat these differently;
   today they all read as exit code 2.
3. **Cost asymmetry:** every gate spends a full council (~4 models × 3 stages)
   even for changes a cheap screen could pass/flag in seconds.

2026 judge research matches: multi-agent judges can *amplify* rather than
cancel bias; lightweight sub-second screening judges are production practice;
confidence signals require calibration against observed outcomes. We own the
calibration corpus: every verify run persists a full transcript + verdict to
`.council/logs`.

## Decision

### Phase 1 — UNCLEAR disambiguation (semantics first)
Split UNCLEAR into machine-readable causes on `VerifyResponse`:
`unclear_reason ∈ {infra_failure, low_confidence, timeout}` (infra detection
uses the #403 `error_status`; timeout uses `timeout_fired`). Exit code stays 2
(compat); the reason field lets automation apply distinct policies (retry
infra, accept-and-audit low-confidence, re-tier timeout). Loop/skill docs
updated to consume it.

### Phase 2 — Confidence calibration from the transcript corpus
Build a calibration analysis over `.council/logs` (verdict, confidence, rubric
scores, blocking count vs. eventual disposition where recorded): fit a simple
monotonic recalibration mapping (e.g. isotonic/binned) and surface BOTH raw
and calibrated confidence on responses. The PASS rule may then use calibrated
confidence — behind a flag, default off, until the mapping is validated
(ADR-036 P2's cross-model calibration report is a by-product).

### Phase 3 — Lightweight screening judge
An opt-in pre-gate: a single quick-tier model scores the change against the
rubric in seconds; unambiguous outcomes short-circuit to a cheap
PASS-with-audit-note, everything else proceeds to the full council unchanged.
**Concrete gates (council feedback; initial values, env-tunable):** a change
may be screen-passed only if ALL hold — screen rubric score ≥ 9/10 on every
dimension; diff < 5,000 chars; no changed file matches a risk glob
(`**/auth*`, `**/security*`, `**/crypto*`, `**/*payment*`); and the request
is not blocking-capable. **Blocking-capable (operationalized):** any request
carrying `evidence` items with `strength=blocking`, or `rubric_focus` in
{security}, is NEVER screened — full council always runs. Default OFF;
shadow-first: every screen decision is logged with its score so the screen's
own precision is measured before trusting it (like ADR-044 P2).

### Phase 4 — Bias-amplification check (report only)
Extend the ADR-015/018 bias pipeline with a reviewer-agreement decomposition
per ADR-036 P2 (do reviewers converge because the work is good, or because of
shared bias/position effects?). Report-only; no gating.

## Consequences

**Positive:** the flagship CI-gate feature becomes trustworthy (distinct
UNCLEAR causes; confidence that means something); screening cuts gate cost for
easy changes; aligns with current judge-reliability research; the asymptote
pattern gets a principled fix instead of a per-session cap heuristic.

**Negative / risks:** calibration data is observational and modest-N
(mitigation: publish CIs, flag-gated adoption, keep raw confidence);
a screening judge is itself a judge (mitigation: shadow-first with logged
decisions; never screens BLOCKING-capable paths silently); changing PASS
semantics affects automation (mitigation: compat exit codes + additive fields
only).

## Definition of Done (per phase — council feedback: all phases covered)
- **P1:** `unclear_reason` on every UNCLEAR response + tests for all three
  causes; verify tool description documents it.
- **P2:** calibration analysis reproducible from `.council/logs`; raw AND
  calibrated confidence on responses; flag-off behaviour unchanged (test).
- **P3:** screen decisions logged with scores; blocking-capable invariant
  enforced by test; flag-off byte-identical.
- **P4:** report renders from existing bias store; no gating side-effects
  (test).
- All phases: README verify section, CLAUDE.md, CHANGELOG; epic-loop guidance
  updated to consume new fields.

## References
- `docs/roadmap-2026-h2.md` item 4 (sources: agent-as-judge survey, bias
  amplification, lightweight judges); memory: verify asymptote + #397 infra
  misdiagnosis; ADR-036 §Phase 2.
