What the comparison could not detect
mixedtimeout: 1 lower, 1 higher · violated: 6 lower, 0 higher (paired test on shared seeds)
- comparison
- mock-agent@v0 2026-10-14 a → mock-agent@v0 2026-10-14 b 6669c14388db
- asked
- could each unchanged condition have caught a change of 0.200, if it alone changed, from where it was?
- answer
- 7 of 10 unchanged rows could; 3 could not tell (3 of 10 unchanged rows)
- flaggable
- 18 of 18 rows can be flagged: with 100 pairs the smallest possible p is 1.6e-30, and a flag needs less than 0.003 (q 0.05 over 18 tests) (smallest p 1.6e-30; a flag needs less than 0.003 over 18 tests)
- threshold
- 0.200, recorded
- design
- 0.262 at its worst base rate, 0.400 (Fisher, q 0.05 over 9 tests, 4 times in 5)
recorded as limit 0.262 · at the worst base rate 0.400 · direction rise · test Fisher · level 0.006 · over tests 9 · conditions 9 · events 1 · q 0.05 · power 0.8 (4 times in 5) · flaggable yes · computed now from the recorded replicates yes
Unchanged rows and their limits
| condition | event | before | rise | fall | could tell |
|---|---|---|---|---|---|
| task_id checkout · tool_fail_rate 0.3 1ad7c42d72bb | timeout | 0.010 | 0.146 | none detectable | no |
| task_id search · tool_fail_rate 0.55 b67467242f0e | timeout | 0.010 | 0.146 | none detectable | no |
| task_id checkout · tool_fail_rate 0.55 b7b476846d98 | timeout | 0.040 | 0.183 | none detectable | no |
| task_id refund · tool_fail_rate 0 b4a971a744b4 | violated | 0.000 | 0.123 | no room | yes |
| task_id search · tool_fail_rate 0 c9d973c66931 | violated | 0.000 | 0.123 | no room | yes |
| task_id checkout · tool_fail_rate 0 dfe7178686aa | violated | 0.000 | 0.123 | no room | yes |
| task_id search · tool_fail_rate 0.3 aa41370fbf30 | timeout | 0.000 | 0.123 | no room | yes |
| task_id refund · tool_fail_rate 0 b4a971a744b4 | timeout | 0.000 | 0.123 | no room | yes |
| task_id search · tool_fail_rate 0 c9d973c66931 | timeout | 0.000 | 0.123 | no room | yes |
| task_id checkout · tool_fail_rate 0 dfe7178686aa | timeout | 0.000 | 0.123 | no room | yes |
rise and fall: the smallest change each way this condition alone would be flagged for, 4 times in 5, if it alone changed, from where it was (the paired test (McNemar), Holm q 0.05 over 18 tests) Charts: limits, violated · limits, timeout.