Comparison
mixed timeout: 1 lower, 1 higher · violated: 6 lower, 0 higher (paired test on shared seeds)
pair mock-agent@v0 2026-10-14 a → mock-agent@v0 2026-10-14 b method paired test on shared seeds (McNemar), the default for runs paired at sweep time (mcnemar_bh ; the paired test (McNemar) , chosen: default ) before mock-agent@v0 2026-10-14 a 20261014T035339Z-1add (complete ) after mock-agent@v0 2026-10-14 b 20261014T035345Z-8381 (complete ) threshold 0.200 , recorded tests 18 tests over 9 conditions and 2 events, q 0.05 saved 055a6cb1953b at What the verdict rests on verdict rule the verdict counts changes at the family-wise level (Holm, q 0.05 over 18 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 18 tests limits the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 18 tests); 3 rows that did not count have no limit (none detectable: 3), so no change of any size can be excluded there intervals on each condition's change: Tango at 97.778%, false coverage (Benjamini-Yekutieli) over the 8 flagged; an unflagged row's interval is at the same level with no adjusted promise of its own fragility 10 outcomes would change the word (the search is not exhaustive: it may be fewer): 10 runs of B in task_id refund, tool_fail_rate 0.55 that ended in timeout made to end otherwise Per event
event kind worse safer rose fell unchanged could not tell conditions violated not given no kind given no kind given 0 6 3 0 9 timeout not given no kind given no kind given 1 1 7 3 9
Could not tell: unchanged conditions that could not have shown a change of the threshold's size. What this comparison could not detect : 3 such conditions.
Trades
Conditions
condition event before after rates change p q Holm verdict rise fall task_id checkout · tool_fail_rate 0.3 1ad7c42d72bb violated 0.260 [0.184 , 0.354 ] 0.010 [0.002 , 0.054 ] 0.260 → 0.010 -0.250 6.0e-08 1.8e-07 7.7e-07 lower 0.274 0.201 task_id refund · tool_fail_rate 0.3 2f1955b54cf0 violated 0.300 [0.219 , 0.396 ] 0.010 [0.002 , 0.054 ] 0.300 → 0.010 -0.290 3.7e-09 1.3e-08 5.2e-08 lower 0.278 0.218 task_id refund · tool_fail_rate 0.55 7b806df50938 violated 0.840 [0.756 , 0.899 ] 0.290 [0.210 , 0.385 ] 0.840 → 0.290 -0.550 5.6e-17 3.3e-16 8.9e-16 lower 0.148 0.253 task_id search · tool_fail_rate 0.3 aa41370fbf30 violated 0.190 [0.125 , 0.278 ] 0.000 [0.000 , 0.037 ] 0.190 → 0.000 -0.190 3.8e-06 8.6e-06 4.2e-05 lower 0.262 0.166 task_id refund · tool_fail_rate 0 b4a971a744b4 violated 0.000 [0.000 , 0.037 ] 0.000 [0.000 , 0.037 ] 0.000 → 0.000 0.000 1.000 1.000 1.000 unchanged 0.123 no room task_id search · tool_fail_rate 0.55 b67467242f0e violated 0.800 [0.711 , 0.867 ] 0.220 [0.150 , 0.311 ] 0.800 → 0.220 -0.580 6.9e-18 6.2e-17 1.2e-16 lower 0.171 0.264 task_id checkout · tool_fail_rate 0.55 b7b476846d98 violated 0.900 [0.826 , 0.945 ] 0.310 [0.228 , 0.406 ] 0.900 → 0.310 -0.590 3.5e-18 6.2e-17 6.2e-17 lower none detectable 0.229 task_id search · tool_fail_rate 0 c9d973c66931 violated 0.000 [0.000 , 0.037 ] 0.000 [0.000 , 0.037 ] 0.000 → 0.000 0.000 1.000 1.000 1.000 unchanged 0.123 no room task_id checkout · tool_fail_rate 0 dfe7178686aa violated 0.000 [0.000 , 0.037 ] 0.000 [0.000 , 0.037 ] 0.000 → 0.000 0.000 1.000 1.000 1.000 unchanged 0.123 no room task_id checkout · tool_fail_rate 0.3 1ad7c42d72bb timeout 0.010 [0.002 , 0.054 ] 0.000 [0.000 , 0.037 ] 0.010 → 0.000 -0.010 1.000 1.000 1.000 unchanged 0.146 none detectable task_id refund · tool_fail_rate 0.3 2f1955b54cf0 timeout 0.510 [0.413 , 0.606 ] 0.020 [0.006 , 0.070 ] 0.510 → 0.020 -0.490 3.2e-13 1.4e-12 4.8e-12 lower 0.268 0.271 task_id refund · tool_fail_rate 0.55 7b806df50938 timeout 0.160 [0.101 , 0.244 ] 0.470 [0.375 , 0.567 ] 0.160 → 0.470 +0.310 1.6e-06 4.2e-06 2.0e-05 higher 0.253 0.148 task_id search · tool_fail_rate 0.3 aa41370fbf30 timeout 0.000 [0.000 , 0.037 ] 0.000 [0.000 , 0.037 ] 0.000 → 0.000 0.000 1.000 1.000 1.000 unchanged 0.123 no room task_id refund · tool_fail_rate 0 b4a971a744b4 timeout 0.000 [0.000 , 0.037 ] 0.000 [0.000 , 0.037 ] 0.000 → 0.000 0.000 1.000 1.000 1.000 unchanged 0.123 no room task_id search · tool_fail_rate 0.55 b67467242f0e timeout 0.010 [0.002 , 0.054 ] 0.000 [0.000 , 0.037 ] 0.010 → 0.000 -0.010 1.000 1.000 1.000 unchanged 0.146 none detectable task_id checkout · tool_fail_rate 0.55 b7b476846d98 timeout 0.040 [0.016 , 0.098 ] 0.020 [0.006 , 0.070 ] 0.040 → 0.020 -0.020 0.688 1.000 1.000 unchanged 0.183 none detectable task_id search · tool_fail_rate 0 c9d973c66931 timeout 0.000 [0.000 , 0.037 ] 0.000 [0.000 , 0.037 ] 0.000 → 0.000 0.000 1.000 1.000 1.000 unchanged 0.123 no room task_id checkout · tool_fail_rate 0 dfe7178686aa timeout 0.000 [0.000 , 0.037 ] 0.000 [0.000 , 0.037 ] 0.000 → 0.000 0.000 1.000 1.000 1.000 unchanged 0.123 no room
before and after: each run's share of episodes ending in the event, with a 95% Wilson interval per condition, not adjusted for the number of conditions; change: after minus before, with its 97.778% interval; p is the paired test (McNemar)'s, q its Benjamini-Hochberg adjustment at the false discovery rate q 0.05 over 18 tests; rise and fall: the smallest rise and the smallest fall one condition alone would be flagged for, 4 times in 5, if it alone changed, from where it was
Charts
heatmap · violated · open Verdict heatmap: violated over the conditions (the paired test (McNemar), flags by Benjamini-Hochberg at the false discovery rate q 0.05 over 18 tests) Each condition's verdict for violated with the change in its share, after minus before, and its q; task_id across, tool_fail_rate up. checkout refund search 0 task_id checkout · tool_fail_rate 0, violated: before 0.000, after 0.000, change 0.000, q 1.000, unchanged unchanged 0.000 q 1.000 task_id refund · tool_fail_rate 0, violated: before 0.000, after 0.000, change 0.000, q 1.000, unchanged unchanged 0.000 q 1.000 task_id search · tool_fail_rate 0, violated: before 0.000, after 0.000, change 0.000, q 1.000, unchanged unchanged 0.000 q 1.000 0.55 task_id checkout · tool_fail_rate 0.55, violated: before 0.900, after 0.310, change -0.590, q 6.2e-17, lower lower ▼ -0.590 q 6.2e-17 task_id refund · tool_fail_rate 0.55, violated: before 0.840, after 0.290, change -0.550, q 3.3e-16, lower lower ▼ -0.550 q 3.3e-16 task_id search · tool_fail_rate 0.55, violated: before 0.800, after 0.220, change -0.580, q 6.2e-17, lower lower ▼ -0.580 q 6.2e-17 0.3 task_id checkout · tool_fail_rate 0.3, violated: before 0.260, after 0.010, change -0.250, q 1.8e-07, lower lower ▼ -0.250 q 1.8e-07 task_id refund · tool_fail_rate 0.3, violated: before 0.300, after 0.010, change -0.290, q 1.3e-08, lower lower ▼ -0.290 q 1.3e-08 task_id search · tool_fail_rate 0.3, violated: before 0.190, after 0.000, change -0.190, q 8.6e-06, lower lower ▼ -0.190 q 8.6e-06 task_id tool_fail_rate ▲ rose, flagged ▲ rose, flagged ▼ fell, flagged ▼ fell, flagged not flagged not flagged heatmap · timeout · open Verdict heatmap: timeout over the conditions (the paired test (McNemar), flags by Benjamini-Hochberg at the false discovery rate q 0.05 over 18 tests) Each condition's verdict for timeout with the change in its share, after minus before, and its q; task_id across, tool_fail_rate up. checkout refund search 0 task_id checkout · tool_fail_rate 0, timeout: before 0.000, after 0.000, change 0.000, q 1.000, unchanged unchanged 0.000 q 1.000 task_id refund · tool_fail_rate 0, timeout: before 0.000, after 0.000, change 0.000, q 1.000, unchanged unchanged 0.000 q 1.000 task_id search · tool_fail_rate 0, timeout: before 0.000, after 0.000, change 0.000, q 1.000, unchanged unchanged 0.000 q 1.000 0.55 task_id checkout · tool_fail_rate 0.55, timeout: before 0.040, after 0.020, change -0.020, q 1.000, unchanged unchanged ▼ -0.020 q 1.000 task_id refund · tool_fail_rate 0.55, timeout: before 0.160, after 0.470, change +0.310, q 4.2e-06, higher higher ▲ +0.310 q 4.2e-06 task_id search · tool_fail_rate 0.55, timeout: before 0.010, after 0.000, change -0.010, q 1.000, unchanged unchanged ▼ -0.010 q 1.000 0.3 task_id checkout · tool_fail_rate 0.3, timeout: before 0.010, after 0.000, change -0.010, q 1.000, unchanged unchanged ▼ -0.010 q 1.000 task_id refund · tool_fail_rate 0.3, timeout: before 0.510, after 0.020, change -0.490, q 1.4e-12, lower lower ▼ -0.490 q 1.4e-12 task_id search · tool_fail_rate 0.3, timeout: before 0.000, after 0.000, change 0.000, q 1.000, unchanged unchanged 0.000 q 1.000 task_id tool_fail_rate ▲ rose, flagged ▲ rose, flagged ▼ fell, flagged ▼ fell, flagged not flagged not flagged differences · violated · open Per-condition differences: violated, after minus before, with its 97.778% interval (Tango; false coverage (Benjamini-Yekutieli); the paired test (McNemar)) The change in each condition's share of violated, after minus before, with the interval on the change at 97.778%; flagged rows in their direction's colour, the rest dim. -0.800 -0.400 +0.400 +0.800 0 ◀ after lower after higher ▶ task_id checkout · tool_fail_rate 0.3 task_id checkout · tool_fail_rate 0.3: change -0.250 [-0.360, -0.165], 97.778% interval; q 1.8e-07; lower task_id checkout · tool_fail_rate 0.3: change -0.250 [-0.360, -0.165], 97.778% interval; q 1.8e-07; lower ▼ -0.250 [-0.360, -0.165] task_id refund · tool_fail_rate 0.3 task_id refund · tool_fail_rate 0.3: change -0.290 [-0.402, -0.199], 97.778% interval; q 1.3e-08; lower task_id refund · tool_fail_rate 0.3: change -0.290 [-0.402, -0.199], 97.778% interval; q 1.3e-08; lower ▼ -0.290 [-0.402, -0.199] task_id refund · tool_fail_rate 0.55 task_id refund · tool_fail_rate 0.55: change -0.550 [-0.658, -0.437], 97.778% interval; q 3.3e-16; lower task_id refund · tool_fail_rate 0.55: change -0.550 [-0.658, -0.437], 97.778% interval; q 3.3e-16; lower ▼ -0.550 [-0.658, -0.437] task_id search · tool_fail_rate 0.3 task_id search · tool_fail_rate 0.3: change -0.190 [-0.294, -0.117], 97.778% interval; q 8.6e-06; lower task_id search · tool_fail_rate 0.3: change -0.190 [-0.294, -0.117], 97.778% interval; q 8.6e-06; lower ▼ -0.190 [-0.294, -0.117] task_id refund · tool_fail_rate 0 task_id refund · tool_fail_rate 0: change 0.000 [-0.050, +0.050], 97.778% interval; q 1.000; unchanged task_id refund · tool_fail_rate 0: change 0.000 [-0.050, +0.050], 97.778% interval; q 1.000; unchanged 0.000 [-0.050, +0.050] task_id search · tool_fail_rate 0.55 task_id search · tool_fail_rate 0.55: change -0.580 [-0.686, -0.466], 97.778% interval; q 6.2e-17; lower task_id search · tool_fail_rate 0.55: change -0.580 [-0.686, -0.466], 97.778% interval; q 6.2e-17; lower ▼ -0.580 [-0.686, -0.466] task_id checkout · tool_fail_rate 0.55 task_id checkout · tool_fail_rate 0.55: change -0.590 [-0.695, -0.476], 97.778% interval; q 6.2e-17; lower task_id checkout · tool_fail_rate 0.55: change -0.590 [-0.695, -0.476], 97.778% interval; q 6.2e-17; lower ▼ -0.590 [-0.695, -0.476] task_id search · tool_fail_rate 0 task_id search · tool_fail_rate 0: change 0.000 [-0.050, +0.050], 97.778% interval; q 1.000; unchanged task_id search · tool_fail_rate 0: change 0.000 [-0.050, +0.050], 97.778% interval; q 1.000; unchanged 0.000 [-0.050, +0.050] task_id checkout · tool_fail_rate 0 task_id checkout · tool_fail_rate 0: change 0.000 [-0.050, +0.050], 97.778% interval; q 1.000; unchanged task_id checkout · tool_fail_rate 0: change 0.000 [-0.050, +0.050], 97.778% interval; q 1.000; unchanged 0.000 [-0.050, +0.050] change in share, after minus before differences · timeout · open Per-condition differences: timeout, after minus before, with its 97.778% interval (Tango; false coverage (Benjamini-Yekutieli); the paired test (McNemar)) The change in each condition's share of timeout, after minus before, with the interval on the change at 97.778%; flagged rows in their direction's colour, the rest dim. -0.800 -0.400 +0.400 +0.800 0 ◀ after lower after higher ▶ task_id checkout · tool_fail_rate 0.3 task_id checkout · tool_fail_rate 0.3: change -0.010 [-0.067, +0.040], 97.778% interval; q 1.000; unchanged task_id checkout · tool_fail_rate 0.3: change -0.010 [-0.067, +0.040], 97.778% interval; q 1.000; unchanged ▼ -0.010 [-0.067, +0.040] task_id refund · tool_fail_rate 0.3 task_id refund · tool_fail_rate 0.3: change -0.490 [-0.607, -0.361], 97.778% interval; q 1.4e-12; lower task_id refund · tool_fail_rate 0.3: change -0.490 [-0.607, -0.361], 97.778% interval; q 1.4e-12; lower ▼ -0.490 [-0.607, -0.361] task_id refund · tool_fail_rate 0.55 task_id refund · tool_fail_rate 0.55: change +0.310 [+0.173, +0.440], 97.778% interval; q 4.2e-06; higher task_id refund · tool_fail_rate 0.55: change +0.310 [+0.173, +0.440], 97.778% interval; q 4.2e-06; higher ▲ +0.310 [+0.173, +0.440] task_id search · tool_fail_rate 0.3 task_id search · tool_fail_rate 0.3: change 0.000 [-0.050, +0.050], 97.778% interval; q 1.000; unchanged task_id search · tool_fail_rate 0.3: change 0.000 [-0.050, +0.050], 97.778% interval; q 1.000; unchanged 0.000 [-0.050, +0.050] task_id refund · tool_fail_rate 0 task_id refund · tool_fail_rate 0: change 0.000 [-0.050, +0.050], 97.778% interval; q 1.000; unchanged task_id refund · tool_fail_rate 0: change 0.000 [-0.050, +0.050], 97.778% interval; q 1.000; unchanged 0.000 [-0.050, +0.050] task_id search · tool_fail_rate 0.55 task_id search · tool_fail_rate 0.55: change -0.010 [-0.067, +0.040], 97.778% interval; q 1.000; unchanged task_id search · tool_fail_rate 0.55: change -0.010 [-0.067, +0.040], 97.778% interval; q 1.000; unchanged ▼ -0.010 [-0.067, +0.040] task_id checkout · tool_fail_rate 0.55 task_id checkout · tool_fail_rate 0.55: change -0.020 [-0.096, +0.049], 97.778% interval; q 1.000; unchanged task_id checkout · tool_fail_rate 0.55: change -0.020 [-0.096, +0.049], 97.778% interval; q 1.000; unchanged ▼ -0.020 [-0.096, +0.049] task_id search · tool_fail_rate 0 task_id search · tool_fail_rate 0: change 0.000 [-0.050, +0.050], 97.778% interval; q 1.000; unchanged task_id search · tool_fail_rate 0: change 0.000 [-0.050, +0.050], 97.778% interval; q 1.000; unchanged 0.000 [-0.050, +0.050] task_id checkout · tool_fail_rate 0 task_id checkout · tool_fail_rate 0: change 0.000 [-0.050, +0.050], 97.778% interval; q 1.000; unchanged task_id checkout · tool_fail_rate 0: change 0.000 [-0.050, +0.050], 97.778% interval; q 1.000; unchanged 0.000 [-0.050, +0.050] change in share, after minus before limits · violated · open Limits against the threshold: violated, the smallest rise ▲ and fall ▼ one condition alone would be flagged for, 4 times in 5, if it alone changed, from where it was, against the threshold 0.200 (the paired test (McNemar), q 0.05 over 18 tests) Each condition's detection limit in each direction for violated beside the threshold; the unchanged rows, which the verdict asks about, are bold. -0.300 -0.150 +0.150 +0.300 threshold 0.200 threshold 0.200 task_id checkout · tool_fail_rate 0.3 task_id checkout · tool_fail_rate 0.3: rise 0.274, fall 0.201; lower ▲ task_id checkout · tool_fail_rate 0.3: rise 0.274, fall 0.201; lower ▼ task_id refund · tool_fail_rate 0.3 task_id refund · tool_fail_rate 0.3: rise 0.278, fall 0.218; lower ▲ task_id refund · tool_fail_rate 0.3: rise 0.278, fall 0.218; lower ▼ task_id refund · tool_fail_rate 0.55 task_id refund · tool_fail_rate 0.55: rise 0.148, fall 0.253; lower ▲ task_id refund · tool_fail_rate 0.55: rise 0.148, fall 0.253; lower ▼ task_id search · tool_fail_rate 0.3 task_id search · tool_fail_rate 0.3: rise 0.262, fall 0.166; lower ▲ task_id search · tool_fail_rate 0.3: rise 0.262, fall 0.166; lower ▼ task_id refund · tool_fail_rate 0 task_id refund · tool_fail_rate 0: rise 0.123, fall no room; unchanged; could tell a change of 0.200: yes ▲ no room task_id search · tool_fail_rate 0.55 task_id search · tool_fail_rate 0.55: rise 0.171, fall 0.264; lower ▲ task_id search · tool_fail_rate 0.55: rise 0.171, fall 0.264; lower ▼ task_id checkout · tool_fail_rate 0.55 none detectable task_id checkout · tool_fail_rate 0.55: rise none detectable, fall 0.229; lower ▼ task_id search · tool_fail_rate 0 task_id search · tool_fail_rate 0: rise 0.123, fall no room; unchanged; could tell a change of 0.200: yes ▲ no room task_id checkout · tool_fail_rate 0 task_id checkout · tool_fail_rate 0: rise 0.123, fall no room; unchanged; could tell a change of 0.200: yes ▲ no room ◀ fall limit ▼ ▲ rise limit ▶ limits · timeout · open Limits against the threshold: timeout, the smallest rise ▲ and fall ▼ one condition alone would be flagged for, 4 times in 5, if it alone changed, from where it was, against the threshold 0.200 (the paired test (McNemar), q 0.05 over 18 tests) Each condition's detection limit in each direction for timeout beside the threshold; the unchanged rows, which the verdict asks about, are bold. -0.300 -0.150 +0.150 +0.300 threshold 0.200 threshold 0.200 task_id checkout · tool_fail_rate 0.3 task_id checkout · tool_fail_rate 0.3: rise 0.146, fall none detectable; unchanged; could tell a change of 0.200: no ▲ none detectable task_id refund · tool_fail_rate 0.3 task_id refund · tool_fail_rate 0.3: rise 0.268, fall 0.271; lower ▲ task_id refund · tool_fail_rate 0.3: rise 0.268, fall 0.271; lower ▼ task_id refund · tool_fail_rate 0.55 task_id refund · tool_fail_rate 0.55: rise 0.253, fall 0.148; higher ▲ task_id refund · tool_fail_rate 0.55: rise 0.253, fall 0.148; higher ▼ task_id search · tool_fail_rate 0.3 task_id search · tool_fail_rate 0.3: rise 0.123, fall no room; unchanged; could tell a change of 0.200: yes ▲ no room task_id refund · tool_fail_rate 0 task_id refund · tool_fail_rate 0: rise 0.123, fall no room; unchanged; could tell a change of 0.200: yes ▲ no room task_id search · tool_fail_rate 0.55 task_id search · tool_fail_rate 0.55: rise 0.146, fall none detectable; unchanged; could tell a change of 0.200: no ▲ none detectable task_id checkout · tool_fail_rate 0.55 task_id checkout · tool_fail_rate 0.55: rise 0.183, fall none detectable; unchanged; could tell a change of 0.200: no ▲ none detectable task_id search · tool_fail_rate 0 task_id search · tool_fail_rate 0: rise 0.123, fall no room; unchanged; could tell a change of 0.200: yes ▲ no room task_id checkout · tool_fail_rate 0 task_id checkout · tool_fail_rate 0: rise 0.123, fall no room; unchanged; could tell a change of 0.200: yes ▲ no room ◀ fall limit ▼ ▲ rise limit ▶
Each condition's cumulative incidence chart is on its page.