A plain-English guide to HyperMind's evaluation metrics, why the old "Collective IQ" number was broken, what replaces it, and exactly when each measurement should run.
EIG/Q (bits) is in the code, goes negative when the swarm is confidently wrong, and can be compared fairly to a single LLM.hmctl eval bench prints every number with a confidence interval and a Single-LLM baseline column.collective_iq.score was a blend of four things:
| Ingredient | Weight | Does it care if the answer is correct? |
|---|---|---|
| Convergence (did the panel agree?) | 40% | No — agreement ≠ correctness |
| Calibration (L1 distance to truth) | 30% | Yes (weakly — it's a non-proper rule) |
| Coordination density (citations/disputes) | 20% | No — "chatter" |
| Reputation motion (how much trust moved) | 10% | No — sign-blind (falling = rising) |
So 70% of the score can be maxed out without a single correct forecast. Worse, it can't be compared to a single LLM at all — three of the four ingredients are meaningless without a panel. A reviewer finds the formula in 30 seconds and dismisses the whole result. We're retiring it.
Every number below is a proper scoring rule — meaning you can only maximise it by telling the truth. That's the minimum bar for a metric you publish.
| Metric | Unit | What it tells you (plain English) |
|---|---|---|
| Brier score | [0, 0.25] | How far off the predicted probability was from the real outcome. Lower = better. 0.25 = a coin flip. |
| ECE (calibration error) | [0, 1] | When the swarm says "90% sure", is it right 90% of the time? Lower = better-calibrated. |
| EIG/Q (info gain / question) | bits | How much real information the swarm added over a naive baseline. Goes negative when confidently wrong. This is our new headline number. |
| WoC-Δ (wisdom-of-crowds delta) | Brier | The only number that tests the core claim: "the swarm beat its single best member." Negative = the crowd won. |
| Logarithmic score | nats | A second proper rule that punishes overconfidence harder than Brier. We report both as a robustness check. |
The key insight: proper metrics need the real answer to be known first. You can't compute Brier on a forecast that hasn't resolved. So not everything runs on every output. Here are the three schedules:
One command runs the periodic batch and prints every metric with a confidence interval + the Single-LLM baseline:
hmctl eval bench --model gpt-5-mini --dataset eb100 --rounds 3
# Output:
# Brier : 0.082 (95% CI [0.071, 0.094]) | Single-LLM: 0.069
# ECE : 0.041
# EIG/Q : +0.61 bits | Single-LLM: +0.55 bits
# WoC-Δ : -0.013 (crowd beat best member)
#
# Sources: Brier 1950 · Gneiting & Raftery 2007 · Guo et al. 2017Every number is reproducible from the run's trace.json. The source footer under each metric makes the citation traceable.
When RELATA_URL is set, five MCP tools upgrade the retrieval layer from in-process lexical search to verified, attributed, contestable knowledge retrieval over the signed ledger.
| Tool | Requires Relata? | What it does |
|---|---|---|
hm_retrieve | Optional (degrades) | BM25 hybrid search over the signed ledger — every result carries author, reputation, trust score |
hm_justify | Optional (degrades) | Full evidence chain for any claim: author, confidence, dispute history, citations |
hm_recall | Optional (degrades) | Memory search — "what does the swarm believe about X?" with optional time-travel |
hm_related | Yes (PATHS_BETWEEN) | Graph traversal: find connected claims/entities via CITES/CORROBORATES links |
hm_learn | Yes (write) | Trigger focused OrgSim research on a topic; publishes signed findings to the ledger |
Without RELATA_URL, all tools degrade gracefully to the in-process lexical fallback. Wire Relata with hmctl mcp install --env RELATA_URL=http://localhost:8000.
Tracked under the parent programme #226:
legacy.score, surface EIG/Q + Brier + ECE everywhere.record_outcome never fires during bench.dispute_lab from the deleted branch; load-test the 7-state FSM end-to-end.--rounds N + shuffle.