Verdict heatmap: timeout over the conditions (the paired test (McNemar), flags by Benjamini-Hochberg at the false discovery rate q 0.05 over 18 tests)
- chart
- heatmap of timeout
- comparison
- mock-agent@v0 2026-10-14 a → mock-agent@v0 2026-10-14 b 6669c14388db
- condition
- every condition
The data
| condition | task_id | tool_fail_rate | before | after | delta | q | verdict |
|---|---|---|---|---|---|---|---|
| task_id checkout · tool_fail_rate 0.3 | checkout | 0.3 | 0.010 | 0.000 | -0.010 | 1.000 | unchanged |
| task_id refund · tool_fail_rate 0.3 | refund | 0.3 | 0.510 | 0.020 | -0.490 | 1.4e-12 | lower |
| task_id refund · tool_fail_rate 0.55 | refund | 0.55 | 0.160 | 0.470 | +0.310 | 4.2e-06 | higher |
| task_id search · tool_fail_rate 0.3 | search | 0.3 | 0.000 | 0.000 | 0.000 | 1.000 | unchanged |
| task_id refund · tool_fail_rate 0 | refund | 0 | 0.000 | 0.000 | 0.000 | 1.000 | unchanged |
| task_id search · tool_fail_rate 0.55 | search | 0.55 | 0.010 | 0.000 | -0.010 | 1.000 | unchanged |
| task_id checkout · tool_fail_rate 0.55 | checkout | 0.55 | 0.040 | 0.020 | -0.020 | 1.000 | unchanged |
| task_id search · tool_fail_rate 0 | search | 0 | 0.000 | 0.000 | 0.000 | 1.000 | unchanged |
| task_id checkout · tool_fail_rate 0 | checkout | 0 | 0.000 | 0.000 | 0.000 | 1.000 | unchanged |
From .poisson/artifacts/comparison/