timeout: 1 safer, 1 worse · violated: 6 safer, 0 worse
(aalen_johansen + mcnemar_bh, BH q<0.05 over 18 tests, paired)
Trade: violated fell in 1 cell where timeout rose.
"no cell changed" is rejected (p = 0.000249938 < q 0.05) over 9 cells x 2 events
min-p of location and dispersion; 4000 within-cell permutations, pair_swap, seed 0
p floor 2.50e-04: the smallest p 4000 permutations can express; a p at the floor is a bound, not a measurement
| event | mean delta (pp) | p | rms delta (pp) | p |
|---|---|---|---|---|
| timeout | -2.4 | 0.028 | 19.3 | 2.50e-04 |
| violated | -27.2 | 2.50e-04 | 36.1 | 2.50e-04 |
informative cells: 6 of 9; 3 unchanged by every exchange
grouping task_id (axis), 3 groups; cells per group min 3 / median 3 / max 3; 0 degenerate; 3 of 3 groups resolve 20pp at alpha 0.0167
"no cell in the group changed" tested by BH across 3 groups at q 0.05: 3 rejected
group p floor 2.50e-04: the same 4000 permutations; a group p at the floor is a bound, not a measurement
| task_id | cells | informative | unchanged by every exchange | units (a/b) | group MDE (pp) | discordant pairs | p | q | verdict |
|---|---|---|---|---|---|---|---|---|---|
| checkout | 3 | 2 | 1 | 300/300 | 13.1pp | 89 | 2.50e-04 | 2.50e-04 | rejected |
| refund | 3 | 2 | 1 | 300/300 | 13.1pp | 141 | 2.50e-04 | 2.50e-04 | rejected |
| search | 3 | 2 | 1 | 300/300 | 13.1pp | 78 | 2.50e-04 | 2.50e-04 | rejected |
"this cell did not change" tested by BH within each of 3 rejecting groups at q 0.05: 8 of 18 rejected
MDE is the smallest detectable increase from arm A's rate, capped at the headroom, so on a cell at ceiling it reads the headroom and is not a power number.
task_id=checkout: 6 tests, 2 rejected; cell MDE median 9pp / max 19pp
| cell | event | delta (pp) | q | MDE |
|---|---|---|---|---|
| task_id=checkout, tool_fail_rate=0.3 | violated | -25.0 | 1.79e-07 | 19pp |
| task_id=checkout, tool_fail_rate=0.55 | violated | -59.0 | 2.08e-17 | 9pp |
task_id=refund: 6 tests, 4 rejected; cell MDE median 14pp / max 19pp
| cell | event | delta (pp) | q | MDE |
|---|---|---|---|---|
| task_id=refund, tool_fail_rate=0.3 | timeout | -49.0 | 9.54e-13 | 19pp |
| task_id=refund, tool_fail_rate=0.3 | violated | -29.0 | 7.45e-09 | 19pp |
| task_id=refund, tool_fail_rate=0.55 | timeout | 31.0 | 2.45e-06 | 17pp |
| task_id=refund, tool_fail_rate=0.55 | violated | -55.0 | 3.33e-16 | 12pp |
task_id=search: 6 tests, 2 rejected; cell MDE median 9pp / max 18pp
| cell | event | delta (pp) | q | MDE |
|---|---|---|---|---|
| task_id=search, tool_fail_rate=0.3 | violated | -19.0 | 1.14e-05 | 18pp |
| task_id=search, tool_fail_rate=0.55 | violated | -58.0 | 4.16e-17 | 13pp |
depth 3 (3 configured); outer-nodes false discovery rate at most 0.432 = 2 x 3 x 0.05 x 1.44
Yekutieli (2008) Corollary A.2 assumes each p-value is independent of its ancestors; this tree's group and root statistics aggregate their own cells, so that assumption is not established here.
| task_id | tool_fail_rate | n (a/b) | rate_a (pp) | rate_b (pp) | delta (pp) | p | q | verdict | MDE | interval (pp) |
|---|---|---|---|---|---|---|---|---|---|---|
| refund | 0.55 | 100/100 | 16.0 | 47.0 | 31.0 | 1.64e-06 | 4.21e-06 | worse | 17pp | [17.3, 44.0] |
| search | 0.3 | 100/100 | 0.0 | 0.0 | 0.0 | 1 | 1 | unchanged | 8pp | [-5.0, 5.0] |
| refund | 0 | 100/100 | 0.0 | 0.0 | 0.0 | 1 | 1 | unchanged | 8pp | [-5.0, 5.0] |
| search | 0 | 100/100 | 0.0 | 0.0 | 0.0 | 1 | 1 | unchanged | 8pp | [-5.0, 5.0] |
| checkout | 0 | 100/100 | 0.0 | 0.0 | 0.0 | 1 | 1 | unchanged | 8pp | [-5.0, 5.0] |
| checkout | 0.3 | 100/100 | 1.0 | 0.0 | -1.0 | 1 | 1 | unchanged | 9pp | [-6.7, 4.0] |
| search | 0.55 | 100/100 | 1.0 | 0.0 | -1.0 | 1 | 1 | unchanged | 9pp | [-6.7, 4.0] |
| checkout | 0.55 | 100/100 | 4.0 | 2.0 | -2.0 | 0.688 | 1 | unchanged | 12pp | [-9.6, 4.9] |
| refund | 0.3 | 100/100 | 51.0 | 2.0 | -49.0 | 3.18e-13 | 1.43e-12 | safer | 19pp | [-60.7, -36.1] |
Worst-powered unchanged cell (task_id=checkout, tool_fail_rate=0.55, n=100/100) can only detect a 12pp rise. "1 worse" is an honest claim only above that.
Worst-powered cell overall: 19pp (task_id=refund, tool_fail_rate=0.3).
| task_id | tool_fail_rate | n (a/b) | rate_a (pp) | rate_b (pp) | delta (pp) | p | q | verdict | MDE | interval (pp) |
|---|---|---|---|---|---|---|---|---|---|---|
| refund | 0 | 100/100 | 0.0 | 0.0 | 0.0 | 1 | 1 | unchanged | 8pp | [-5.0, 5.0] |
| search | 0 | 100/100 | 0.0 | 0.0 | 0.0 | 1 | 1 | unchanged | 8pp | [-5.0, 5.0] |
| checkout | 0 | 100/100 | 0.0 | 0.0 | 0.0 | 1 | 1 | unchanged | 8pp | [-5.0, 5.0] |
| search | 0.3 | 100/100 | 19.0 | 0.0 | -19.0 | 3.81e-06 | 8.58e-06 | safer | 18pp | [-29.4, -11.7] |
| checkout | 0.3 | 100/100 | 26.0 | 1.0 | -25.0 | 5.96e-08 | 1.79e-07 | safer | 19pp | [-36.0, -16.5] |
| refund | 0.3 | 100/100 | 30.0 | 1.0 | -29.0 | 3.73e-09 | 1.34e-08 | safer | 19pp | [-40.2, -19.9] |
| refund | 0.55 | 100/100 | 84.0 | 29.0 | -55.0 | 5.55e-17 | 3.33e-16 | safer | 12pp | [-65.8, -43.7] |
| search | 0.55 | 100/100 | 80.0 | 22.0 | -58.0 | 6.94e-18 | 6.25e-17 | safer | 13pp | [-68.6, -46.6] |
| checkout | 0.55 | 100/100 | 90.0 | 31.0 | -59.0 | 3.47e-18 | 6.25e-17 | safer | 9pp | [-69.5, -47.6] |
Worst-powered unchanged cell (task_id=refund, tool_fail_rate=0, n=100/100) can only detect a 8pp rise. "0 worse" is an honest claim only above that.
Worst-powered cell overall: 19pp (task_id=refund, tool_fail_rate=0.3).
intervals: 97.7778% (alpha* = 0.0222222 = q x 8 / 18 over the flat BH family), basis score_rd_paired
interval excludes zero, verdict unchanged: 0 cells
verdict significant, interval covers zero: 0 cells
This is empirical evidence under a stated test envelope, not formal verification. Cells outside the envelope were not tested; cells inside it were tested at the stated replicate count and can only detect effects at or above the reported MDE.