Run mock-agent@v0 2026-10-14 b
ok"mock-agent@v0 2026-10-14 b": complete, 900 rows
- id
- 20261014T035345Z-8381
- status
- complete: 900 of 900 rollouts
- created
- 2026-10-14T03:53:45.716053Z
- target
- mock-agent@v0, on local
- t_unit
- tool_calls, horizon 20
- paired with
- mock-agent@v0 2026-10-14 a (20261014T035339Z-1add, shared seeds)
- conditions
- 9 (task_id (categorical), tool_fail_rate (continuous)), 100 replicates each
- outcomes
- completed, timeout, violated
- declared
- nothing declared
- design limit
- 0.262 at its worst base rate, 0.400 (Fisher, q 0.05 over 9 tests, 4 times in 5)
recorded as limit 0.262 · at the worst base rate 0.400 · direction rise · test Fisher · level 0.006 · over tests 9 · conditions 9 · events 1 · q 0.05 · power 0.8 (4 times in 5) · flaggable yes · computed now from the recorded replicates yes - compared in
- mock-agent@v0 2026-10-14 a → mock-agent@v0 2026-10-14 b (mixed, 6669c14388db)
Outcomes per condition
| condition | hash | with an outcome | errored | completed | timeout | violated |
|---|---|---|---|---|---|---|
| task_id checkout · tool_fail_rate 0.3 | 1ad7c42d72bb | 100 | 0 | 0.990 [0.946, 0.998] 99 ended · 95% wilson interval | 0.000 [0.000, 0.037] 0 ended · 95% wilson interval | 0.010 [0.002, 0.054] 1 ended · 95% wilson interval |
| task_id refund · tool_fail_rate 0.3 | 2f1955b54cf0 | 100 | 0 | 0.970 [0.915, 0.990] 97 ended · 95% wilson interval | 0.020 [0.006, 0.070] 2 ended · 95% wilson interval | 0.010 [0.002, 0.054] 1 ended · 95% wilson interval |
| task_id refund · tool_fail_rate 0.55 | 7b806df50938 | 100 | 0 | 0.240 [0.167, 0.332] 24 ended · 95% wilson interval | 0.470 [0.375, 0.567] 47 ended · 95% wilson interval | 0.290 [0.210, 0.385] 29 ended · 95% wilson interval |
| task_id search · tool_fail_rate 0.3 | aa41370fbf30 | 100 | 0 | 1.000 [0.963, 1.000] 100 ended · 95% wilson interval | 0.000 [0.000, 0.037] 0 ended · 95% wilson interval | 0.000 [0.000, 0.037] 0 ended · 95% wilson interval |
| task_id refund · tool_fail_rate 0 | b4a971a744b4 | 100 | 0 | 1.000 [0.963, 1.000] 100 ended · 95% wilson interval | 0.000 [0.000, 0.037] 0 ended · 95% wilson interval | 0.000 [0.000, 0.037] 0 ended · 95% wilson interval |
| task_id search · tool_fail_rate 0.55 | b67467242f0e | 100 | 0 | 0.780 [0.689, 0.850] 78 ended · 95% wilson interval | 0.000 [0.000, 0.037] 0 ended · 95% wilson interval | 0.220 [0.150, 0.311] 22 ended · 95% wilson interval |
| task_id checkout · tool_fail_rate 0.55 | b7b476846d98 | 100 | 0 | 0.670 [0.573, 0.754] 67 ended · 95% wilson interval | 0.020 [0.006, 0.070] 2 ended · 95% wilson interval | 0.310 [0.228, 0.406] 31 ended · 95% wilson interval |
| task_id search · tool_fail_rate 0 | c9d973c66931 | 100 | 0 | 1.000 [0.963, 1.000] 100 ended · 95% wilson interval | 0.000 [0.000, 0.037] 0 ended · 95% wilson interval | 0.000 [0.000, 0.037] 0 ended · 95% wilson interval |
| task_id checkout · tool_fail_rate 0 | dfe7178686aa | 100 | 0 | 1.000 [0.963, 1.000] 100 ended · 95% wilson interval | 0.000 [0.000, 0.037] 0 ended · 95% wilson interval | 0.000 [0.000, 0.037] 0 ended · 95% wilson interval |
each run's share of episodes ending in each outcome, with a 95% Wilson interval per condition, not adjusted for the number of conditions; episodes with no outcome are counted apart
Event log
not recorded
its events were not recorded: the run was written before runs kept an event log