The experiment arc
Every measured experiment, newest concepts last. Colour = bankability: green banked, amber directional, grey closed-negative, blue in-progress.
| ID | Experiment | Bankability | Verdict |
|---|---|---|---|
| 0001 | Do gemma confidence signals predict groundedness? | NEGATIVE | AUROC 0.96 was a header-vs-claim confound (18/19 ungrounded = header fragments); 1 real ungrounded in 189 -> detector untestable on this substrate; gemma fabrication rate ~0.5%; build fabrication-inducing corpora |
| 0002 | Does P(grounded) detect injected fabrications? | DIRECTIONAL | detects BLATANT injected fabrications near-perfectly (AUROC 0.998, recall 0.96 @ perfect precision, all 7 domains, balanced n=340, Claude-verified 92% valid positives); scoped to obvious fabrications — subtle fabrications (0003) untested |
| 0003 | Does P(grounded) detect subtle fabrications? | DIRECTIONAL | detects SUBTLE fabrications AUROC 0.989 (~blatant 0.998); context-grounding defeats plausibility (checks vs the evidence's ground truth, not world-plausibility); ruler confirms 51/52 subtle ungrounded; directional — synthetic + mostly-contradicting + Claude-labeled, 0004 (real human-labeled RAGTruth) is the bankable test + the silent-evidence frontier |
| 0004 | Does P(grounded) detect real human-labeled RAG hallucinations? | BANKABLE | REAL RAGTruth AUROC 0.896 BANKABLE (test-retest stable: run1 0.896 / run2 0.893, Δ0.003). Production reqs quantified: natural-AUPRC 0.36 @6.5% base rate, recalibration ECE 0.17->0.03 (isotonic), per-claim noise 12% move>0.1 (avg multiple calls), gemma BINARY verdict useless (use continuous); Summary weakest 0.827. The honest real-world baseline for the #231 hallucination engine; synthetic 0002/0003 (0.99) were optimistic |
Operators by stage
Each pipeline stage and the operators measured against it. Declared cost is the operator’s own cost formula; a live per-run cost ledger (RunStats) is not yet stored, so an unknown cost shows —.
GATHER
| Operator | Status | Mechanism / result | Declared cost |
|---|---|---|---|
| default@v1 | LIVE (incumbent) | iterative gather is THE coverage engine; removing it ~halves coverage | fanout x count x rounds LLM+search calls |
| reflect_evidence@v1 | REJECTED | ~0 lift, costs a gather round (supersedes ADR-0023) | — |
| broad@v2 | REJECTED | wider net dilutes into mis-transcribed sources; real only as a routed lever | — |
PLAN
| Operator | Status | Mechanism / result | Declared cost |
|---|---|---|---|
| facet@v1 | REJECTED | drifts off-topic, loses pairwise both corpora | — |
| deliberate@v1 | REJECTED | fixed-budget reallocation; ejects simple facts; 0-3 pairwise vs off | — |
| default@v1 | LIVE (incumbent) | 1 LLM call |
EXTRACT
| Operator | Status | Mechanism / result | Declared cost |
|---|---|---|---|
| default@v1 | LIVE (incumbent) | beats the value-preserving variant | 1 LLM call/query |
| value_preserving@v2 | REJECTED | works mechanically (0 fabrication) but leak is upstream GATHER search-inclusion | 1 LLM call/query |
TRIAGE
| Operator | Status | Mechanism / result | Declared cost |
|---|---|---|---|
| default@v1 | LIVE (incumbent) | the source-cap is the conditional fabrication firewall | 0-1 LLM call (only when candidates > cap) |
| semantic_rerank@v1 | REJECTED | lateral retrieval shift, net-negative on strict coverage | — |
SYNTHESIZE
| Operator | Status | Mechanism / result | Declared cost |
|---|---|---|---|
| default@v1 | LIVE (incumbent) | commit/harden are conciseness levers; near-optimal fixed-budget allocator | 1 LLM call |
| synthesize_completely@v1 | REJECTED | no-op — facts die upstream, nothing to recover (ties off) | — |
VERIFY
| Operator | Status | Mechanism / result | Declared cost |
|---|---|---|---|
| default@v1 | LIVE (incumbent) | independent re-check; default off | 0-1 LLM call (default off) |
| hallucination@v1 | SHELVED (hypothesis) | per-claim ground + 1 correct |
PLANxSYNTH
| Operator | Status | Mechanism / result | Declared cost |
|---|---|---|---|
| deliberate+synthesize@v1 | REJECTED | recovers delib's drops but never exceeds baseline (not super-additive) | — |
DECIDE
| Operator | Status | Mechanism / result | Declared cost |
|---|---|---|---|
| default@v1 | LIVE (incumbent) | 1 LLM call/iteration |
SEARCH
| Operator | Status | Mechanism / result | Declared cost |
|---|---|---|---|
| default@v1 | LIVE (incumbent) | search+fetch+rank per query |
REFLECT
| Operator | Status | Mechanism / result | Declared cost |
|---|---|---|---|
| default@v1 | LIVE (incumbent) | 1 LLM call/round |
Needs action 3
Open lifecycle steps, classified by the same engine as wsk next. Each card names where the item sits and the concrete next options.
journal 0002 — Does P(grounded) detect injected fabrications?
P3: directional — needs a powered re-measure (the bankability gate)
- re-measure at the powered N / through the bankability gate to bank or close it
- run the adversarial pass (held-out / outliers / fabrication) before treating it as a win
journal 0003 — Does P(grounded) detect subtle fabrications?
P3: directional — needs a powered re-measure (the bankability gate)
- re-measure at the powered N / through the bankability gate to bank or close it
- run the adversarial pass (held-out / outliers / fabrication) before treating it as a win
journal 0004 — Does P(grounded) detect real human-labeled RAG hallucinations?
P4: BANKED, not cataloged (registry_ref is null)
- P4 catalog: land it in the 4 places (or P4-ALT if it's a detector/threshold, not a stage)
- P5 orchestration-fit: decide where/whether it pins in the live pipeline
- P6 composed-measure: measure the whole pipeline pinned vs without (still a partial gap)