Condition task_id refund · tool_fail_rate 0.55

comparison: mock-agent@v0 2026-10-14 a → mock-agent@v0 2026-10-14 b 6669c14388db
before: mock-agent@v0 2026-10-14 a
after: mock-agent@v0 2026-10-14 b
hash: 7b806df50938571e

violated (kind not given)

lowerviolated fell from 0.840 to 0.290 (-0.550) (the paired test (McNemar), q 3.3e-16)

before: 0.840 [0.756, 0.899] 95% wilson interval; 84 of 100 episodes · list them (mock-agent@v0 2026-10-14 a, 20261014T035339Z-1add)
after: 0.290 [0.210, 0.385] 95% wilson interval; 29 of 100 episodes · list them (mock-agent@v0 2026-10-14 b, 20261014T035345Z-8381)
change: -0.550 [-0.658, -0.437], 97.778% Tango interval
test: the paired test (McNemar): 0 pairs had the event only after, 55 only before; statistic 55; p 5.6e-17, q 3.3e-16, Holm 8.9e-16
rise: 0.148
fall: 0.253
limits: computed by the paired test at level 0.003 over 18 tests, power 0.8 (4 times in 5), from the observed baseline; the smallest possible p here is 1.6e-30, so this condition can be flagged (yes)

incidence of violated, each run · open

highertimeout rose from 0.160 to 0.470 (+0.310) (the paired test (McNemar), q 4.2e-06)

before: 0.160 [0.101, 0.244] 95% wilson interval; 16 of 100 episodes · list them (mock-agent@v0 2026-10-14 a, 20261014T035339Z-1add)
after: 0.470 [0.375, 0.567] 95% wilson interval; 47 of 100 episodes · list them (mock-agent@v0 2026-10-14 b, 20261014T035345Z-8381)
change: +0.310 [+0.173, +0.440], 97.778% Tango interval
test: the paired test (McNemar): 37 pairs had the event only after, 6 only before; statistic -31; p 1.6e-06, q 4.2e-06, Holm 2.0e-05
rise: 0.253
fall: 0.148
limits: computed by the paired test at level 0.003 over 18 tests, power 0.8 (4 times in 5), from the observed baseline; the smallest possible p here is 1.6e-30, so this condition can be flagged (yes)

incidence of timeout, each run · open

the verdict counts changes at the family-wise level (Holm, q 0.05 over 18 tests); a condition's flag is Benjamini-Hochberg's, at the false discovery rate q 0.05 over 18 tests
the smallest change one condition alone would be flagged for, 4 times in 5, if it alone changed, from where the condition was (the paired test (McNemar), q 0.05 over 18 tests); 3 rows that did not count have no limit on their fall (none detectable: 3), so a fall of any size cannot be excluded there
on each condition's change: Tango at 97.778%, false coverage (Benjamini-Yekutieli) over the 8 flagged; an unflagged row's interval is at the same level with no adjusted promise of its own
before and after: each run's share with its 95% Wilson interval, per condition, not adjusted; the rise and the fall are each direction's limit