Eval Report: ci-nightly

Profile: gdm-swebench-lite-v1 | Tasks: 8 | Pass rate: 37.5% | Cost: $0.0080

Task IDBandScorePassedCost
python-security-fix-easy-001easy0.740$0.0010
canary-shell-ops-004hard0.310$0.0010
canary-python-cache-005hard0.310$0.0010
canary-python-regression-002hard0.310$0.0010
typescript-security-fix-easy-001easy0.740$0.0010
canary-typescript-auth-006hard0.310$0.0010
python-bugfix-easy-001easy0.740$0.0010
canary-typescript-session-003hard0.310$0.0010

Leaderboard Snapshot

Latest run: a0291428-794e-4ccc-b1e8-508808b682a9 | Latest model: coder | Latest score: 0.310 | Recorded at: 2026-05-23T18:57:10.324681+00:00

Recent Trend

Run IDModelGit SHAScoreCreated
a0291428-794e-4ccc-b1e8-508808b682a9coder4669773b4fbe9d507f1396f38777a1b36998faf30.3102026-05-23T18:57:10.324681+00:00
936f4629-86b4-4529-b924-d90fede38b4dcoder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-05-23T18:57:10.243258+00:00
058ff750-91ee-4d47-9fb2-b3b81df6cfe4coder4669773b4fbe9d507f1396f38777a1b36998faf30.3102026-05-23T18:57:10.180202+00:00
98f401b3-7341-4182-83aa-3a7cca0e9cc2coder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-05-23T18:57:10.109097+00:00
d03198cb-156a-4b1a-a029-0a1095abbdb4coder4669773b4fbe9d507f1396f38777a1b36998faf30.3102026-05-23T18:57:10.021813+00:00

Failure Breakdown

TaxonomyFailuresBar
wrong-file5
#####
wrong-logic3
###

Recent Failures

Run IDTask IDTaxonomyScoreCostCreated
a0291428-794e-4ccc-b1e8-508808b682a9canary-typescript-session-003wrong-file0.310$0.00102026-05-23T18:57:10.324681+00:00
936f4629-86b4-4529-b924-d90fede38b4dpython-bugfix-easy-001wrong-logic0.740$0.00102026-05-23T18:57:10.243258+00:00
058ff750-91ee-4d47-9fb2-b3b81df6cfe4canary-typescript-auth-006wrong-file0.310$0.00102026-05-23T18:57:10.180202+00:00
98f401b3-7341-4182-83aa-3a7cca0e9cc2typescript-security-fix-easy-001wrong-logic0.740$0.00102026-05-23T18:57:10.109097+00:00
d03198cb-156a-4b1a-a029-0a1095abbdb4canary-python-regression-002wrong-file0.310$0.00102026-05-23T18:57:10.021813+00:00
d9fb4c37-cbdc-4dc7-b420-98df5f6c9d30canary-python-cache-005wrong-file0.310$0.00102026-05-23T18:57:09.942168+00:00
5a4c056f-733d-487b-9451-5a53518d2627canary-shell-ops-004wrong-file0.310$0.00102026-05-23T18:57:09.878455+00:00
fdb484ff-b5c0-40f0-9869-e6fa3bf5d412python-security-fix-easy-001wrong-logic0.740$0.00102026-05-23T18:57:09.814986+00:00

Cost Frontier

pass_rate vs cost_usd (Pareto frontier marked with *)
* [#######-------------] 37.5% @ $0.0010  (coder)