Reconciliation report: bandit-algorithms

Generated 2026-09-16T23:40:32.338802Z by prereg 0.1.0. Ledger: 69 entries, head 643a8fe31504a563. Integrity check: PASSED.

Intervals are percentile bootstrap intervals at level 1 - 2 alpha, not multiplicity-adjusted; verdicts use adjusted p-values.

Summary

PredictionMetricClaimVerdictEstimateIntervalp (claim, adj.)
P1cumulative_regretthompson - ucb < -20CONFIRMED-76.390% [-83.18, -67.71]0.0003
P2cumulative_regretucb - eps_greedy < 0REFUTED80.1290% [61.83, 98.03]1.0000
P3cumulative_regretthompson - eps_greedy < 0INCONCLUSIVE6.62890% [-12.36, 25.33]1.0000
P4optimal_arm_ratethompson - eps_greedy > 0.1INADMISSIBLEn/an/an/a

Timeline

EntryTime (UTC)EventDetailCommit
02026-09-16T23:40:12.551785Zinitprereg 0.1.0e65d6b8ad9
12026-09-16T23:40:12.588980ZregisterP1 hash e05499515c3ce65d6b8ad9
22026-09-16T23:40:12.620167ZregisterP2 hash 3735615c961ee65d6b8ad9
32026-09-16T23:40:12.656981ZregisterP3 hash 7ae985801ddbe65d6b8ad9
42026-09-16T23:40:12.694466ZregisterP4 hash d8da1126d126e65d6b8ad9
52026-09-16T23:40:12.881742ZgateP1: passede65d6b8ad9
62026-09-16T23:40:12.918858ZgateP2: passede65d6b8ad9
72026-09-16T23:40:12.950546ZgateP3: passede65d6b8ad9
82026-09-16T23:40:13.075068ZgateP4: FAILEDe65d6b8ad9
9-682026-09-16T23:40:31.852202Zlink 60 run(s)eps_greedy: 20, ucb: 20, thompson: 20e65d6b8ad9
692026-09-16T23:40:32.635016ZanalysisP1=CONFIRMED, P2=REFUTED, P3=INCONCLUSIVE, P4=INADMISSIBLEe65d6b8ad9

P1: Thompson sampling accumulates at least 20 less regret than UCB1.

Claim thompson - ucb < -20 on cumulative_regret; bootstrap test, iqm over 20 seeds, alpha 0.05, correction holm. Registered 2026-09-16T23:40:12.588980Z, v1 hash e05499515c3c50d6.

Verdict: CONFIRMED

effect in the predicted direction; claim significant at the registered alpha.

Estimate of iqm(thompson) - iqm(ucb): -76.3, interval 90% [-83.18, -67.71]. p(claim) <0.0001 (adj. 0.0003); p(opposite) 1.0000 (adj. 1.0000).

Gates

GateResultMessage
MetricDefinedpassmetric 'cumulative_regret' fingerprint matches registration
ConfigFrozenpasscondition configs match registered hashes
NoPeekingpassno linked runs predate registration
MinSeedspass20 seeds per condition planned, minimum 10
PowerAnalysispasspower 0.91 at 20 seeds for margin 25 (target 0.80, ~15 required; alpha 0.01667, normal)

Per-seed values

thompsonthompson-s0 seed 0: 85.15thompson-s1 seed 1: 98.6thompson-s2 seed 2: 132.2thompson-s3 seed 3: 55.75thompson-s4 seed 4: 115.95thompson-s5 seed 5: 80.95thompson-s6 seed 6: 52.85thompson-s7 seed 7: 95.45thompson-s8 seed 8: 75.85thompson-s9 seed 9: 77.7thompson-s10 seed 10: 61.2thompson-s11 seed 11: 110.8thompson-s12 seed 12: 85.1thompson-s13 seed 13: 84.5thompson-s14 seed 14: 81.85thompson-s15 seed 15: 99.8thompson-s16 seed 16: 79.95thompson-s17 seed 17: 87.55thompson-s18 seed 18: 103.6thompson-s19 seed 19: 87.5iqm = 86.66ucbucb-s0 seed 0: 150.85ucb-s1 seed 1: 177.4ucb-s2 seed 2: 163.6ucb-s3 seed 3: 169.05ucb-s4 seed 4: 167.7ucb-s5 seed 5: 159.05ucb-s6 seed 6: 156.1ucb-s7 seed 7: 161.25ucb-s8 seed 8: 151.9ucb-s9 seed 9: 163.5ucb-s10 seed 10: 144.2ucb-s11 seed 11: 174.4ucb-s12 seed 12: 172.6ucb-s13 seed 13: 168.65ucb-s14 seed 14: 161.35ucb-s15 seed 15: 166.9ucb-s16 seed 16: 137.25ucb-s17 seed 17: 161.45ucb-s18 seed 18: 175ucb-s19 seed 19: 140iqm = 162.95546.62115.1183.6
Table
seedthompsonucb
085.15150.85
198.6177.4
2132.2163.6
355.75169.05
4115.95167.7
580.95159.05
652.85156.1
795.45161.25
875.85151.9
977.7163.5
1061.2144.2
11110.8174.4
1285.1172.6
1384.5168.65
1481.85161.35
1599.8166.9
1679.95137.25
1787.55161.45
18103.6175
1987.5140

Deviations

None.

Excluded runs

None.

Rejected runs

None.

P2: UCB1 accumulates less regret than epsilon-greedy at this horizon.

Claim ucb - eps_greedy < 0 on cumulative_regret; welch test, mean over 20 seeds, alpha 0.05, correction holm. Registered 2026-09-16T23:40:12.620167Z, v1 hash 3735615c961e520b.

Verdict: REFUTED

the opposite of the claim is significant at the registered alpha.

Estimate of mean(ucb) - mean(eps_greedy): 80.12, interval 90% [61.83, 98.03]. p(claim) 1.0000 (adj. 1.0000); p(opposite) <0.0001 (adj. <0.0001).

Gates

GateResultMessage
MetricDefinedpassmetric 'cumulative_regret' fingerprint matches registration
ConfigFrozenpasscondition configs match registered hashes
NoPeekingpassno linked runs predate registration
MinSeedspass20 seeds per condition planned, minimum 10
ControlPresentpasscontrol 'eps_greedy' declared and configured
PowerAnalysispasspower 0.87 at 20 seeds for margin 40 (target 0.80, ~17 required; alpha 0.01667, normal)

Per-seed values

ucbucb-s0 seed 0: 150.85ucb-s1 seed 1: 177.4ucb-s2 seed 2: 163.6ucb-s3 seed 3: 169.05ucb-s4 seed 4: 167.7ucb-s5 seed 5: 159.05ucb-s6 seed 6: 156.1ucb-s7 seed 7: 161.25ucb-s8 seed 8: 151.9ucb-s9 seed 9: 163.5ucb-s10 seed 10: 144.2ucb-s11 seed 11: 174.4ucb-s12 seed 12: 172.6ucb-s13 seed 13: 168.65ucb-s14 seed 14: 161.35ucb-s15 seed 15: 166.9ucb-s16 seed 16: 137.25ucb-s17 seed 17: 161.45ucb-s18 seed 18: 175ucb-s19 seed 19: 140mean = 161.11eps_greedyeps_greedy-s0 seed 0: 118.45eps_greedy-s1 seed 1: 116.05eps_greedy-s2 seed 2: 25.4eps_greedy-s3 seed 3: 34.3eps_greedy-s4 seed 4: 100.25eps_greedy-s5 seed 5: 32.15eps_greedy-s6 seed 6: 127.4eps_greedy-s7 seed 7: 61.35eps_greedy-s8 seed 8: 32.4eps_greedy-s9 seed 9: 109.1eps_greedy-s10 seed 10: 120.75eps_greedy-s11 seed 11: 26.2eps_greedy-s12 seed 12: 35.05eps_greedy-s13 seed 13: 190.2eps_greedy-s14 seed 14: 50.45eps_greedy-s15 seed 15: 114.3eps_greedy-s16 seed 16: 118.8eps_greedy-s17 seed 17: 131.75eps_greedy-s18 seed 18: 48.05eps_greedy-s19 seed 19: 27.35mean = 80.987517.16107.8198.4
Table
seeducbeps_greedy
0150.85118.45
1177.4116.05
2163.625.4
3169.0534.3
4167.7100.25
5159.0532.15
6156.1127.4
7161.2561.35
8151.932.4
9163.5109.1
10144.2120.75
11174.426.2
12172.635.05
13168.65190.2
14161.3550.45
15166.9114.3
16137.25118.8
17161.45131.75
1817548.05
1914027.35

Deviations

None.

Excluded runs

None.

Rejected runs

None.

P3: Thompson sampling accumulates less regret than epsilon-greedy.

Claim thompson - eps_greedy < 0 on cumulative_regret; bootstrap test, mean over 20 seeds, alpha 0.05, correction holm. Registered 2026-09-16T23:40:12.656981Z, v1 hash 7ae985801ddbada0.

Verdict: INCONCLUSIVE

neither the claim nor its opposite is significant.

Estimate of mean(thompson) - mean(eps_greedy): 6.628, interval 90% [-12.36, 25.33]. p(claim) 0.7207 (adj. 1.0000); p(opposite) 0.2794 (adj. 0.5587).

Gates

GateResultMessage
MetricDefinedpassmetric 'cumulative_regret' fingerprint matches registration
ConfigFrozenpasscondition configs match registered hashes
NoPeekingpassno linked runs predate registration
MinSeedspass20 seeds per condition planned, minimum 10
ControlPresentpasscontrol 'eps_greedy' declared and configured
PowerAnalysispasspower 0.87 at 20 seeds for margin 45 (target 0.80, ~17 required; alpha 0.01667, normal)

Per-seed values

thompsonthompson-s0 seed 0: 85.15thompson-s1 seed 1: 98.6thompson-s2 seed 2: 132.2thompson-s3 seed 3: 55.75thompson-s4 seed 4: 115.95thompson-s5 seed 5: 80.95thompson-s6 seed 6: 52.85thompson-s7 seed 7: 95.45thompson-s8 seed 8: 75.85thompson-s9 seed 9: 77.7thompson-s10 seed 10: 61.2thompson-s11 seed 11: 110.8thompson-s12 seed 12: 85.1thompson-s13 seed 13: 84.5thompson-s14 seed 14: 81.85thompson-s15 seed 15: 99.8thompson-s16 seed 16: 79.95thompson-s17 seed 17: 87.55thompson-s18 seed 18: 103.6thompson-s19 seed 19: 87.5mean = 87.615eps_greedyeps_greedy-s0 seed 0: 118.45eps_greedy-s1 seed 1: 116.05eps_greedy-s2 seed 2: 25.4eps_greedy-s3 seed 3: 34.3eps_greedy-s4 seed 4: 100.25eps_greedy-s5 seed 5: 32.15eps_greedy-s6 seed 6: 127.4eps_greedy-s7 seed 7: 61.35eps_greedy-s8 seed 8: 32.4eps_greedy-s9 seed 9: 109.1eps_greedy-s10 seed 10: 120.75eps_greedy-s11 seed 11: 26.2eps_greedy-s12 seed 12: 35.05eps_greedy-s13 seed 13: 190.2eps_greedy-s14 seed 14: 50.45eps_greedy-s15 seed 15: 114.3eps_greedy-s16 seed 16: 118.8eps_greedy-s17 seed 17: 131.75eps_greedy-s18 seed 18: 48.05eps_greedy-s19 seed 19: 27.35mean = 80.987517.16107.8198.4
Table
seedthompsoneps_greedy
085.15118.45
198.6116.05
2132.225.4
355.7534.3
4115.95100.25
580.9532.15
652.85127.4
795.4561.35
875.8532.4
977.7109.1
1061.2120.75
11110.826.2
1285.135.05
1384.5190.2
1481.8550.45
1599.8114.3
1679.95118.8
1787.55131.75
18103.648.05
1987.527.35

Deviations

None.

Excluded runs

None.

Rejected runs

None.

P4: Quick check on 5 seeds: Thompson sampling spends at least 10 percentage points more of the late phase on the best arm than epsilon-greedy.

Claim thompson - eps_greedy > 0.1 on optimal_arm_rate; mann_whitney test, median over 5 seeds, alpha 0.05, correction none. Registered 2026-09-16T23:40:12.694466Z, v1 hash d8da1126d126452f.

Verdict: INADMISSIBLE

admissibility gates failed or were not evaluated; analysis withheld.

Note: 30 linked run(s) use seeds planned by other predictions

Note: admissibility gates failed; analysis withheld

Gates

GateResultMessage
MetricDefinedpassmetric 'optimal_arm_rate' fingerprint matches registration
ConfigFrozenpasscondition configs match registered hashes
NoPeekingpassno linked runs predate registration
MinSeedspass5 seeds per condition planned, minimum 3
PowerAnalysisFAILpower 0.09 at 5 seeds for margin 0.1 (target 0.80, ~278 required; alpha 0.05, normal)

Deviations

None.

Excluded runs

None.

Rejected runs

None.

Exploratory analyses (not pre-registered)

Not registered before the runs; no verdicts or p-values.

IdMetricComparisonAggregationEstimateIntervalnNote
E1optimal_arm_ratethompson - eps_greedy > 0median-0.09495% [-0.281, 0.668]40
E2cumulative_regretucb / thompson > 1iqm1.8895% [1.702, 2.056]40

Ledger integrity

All 69 entries verified.