Eval Report: ci-pr-smoke

Profile: gdm-swebench-lite-v1 | Tasks: 15 | Pass rate: 100.0% | Cost: $0.0150

Task IDBandScorePassedCost
python-security-fix-easy-001easy0.740$0.0010
typescript-security-fix-easy-001easy0.740$0.0010
python-bugfix-easy-001easy0.740$0.0010
typescript-bugfix-easy-001easy0.740$0.0010
python-performance-easy-001easy0.740$0.0010
typescript-performance-easy-001easy0.740$0.0010
python-test-writing-easy-001easy0.740$0.0010
typescript-test-writing-easy-001easy0.740$0.0010
python-multi-file-easy-001easy0.740$0.0010
typescript-multi-file-easy-001easy0.740$0.0010
python-refactor-easy-001easy0.740$0.0010
typescript-refactor-easy-001easy0.740$0.0010
python-config-easy-001easy0.740$0.0010
typescript-config-easy-001easy0.740$0.0010
python-recovery-easy-001easy0.740$0.0010

Leaderboard Snapshot

Latest run: 187f48ce-00a7-42c4-9f1f-e7ef1140cc61 | Latest model: coder | Latest score: 0.740 | Recorded at: 2026-04-27T16:10:28.194438+00:00

Recent Trend

Run IDModelGit SHAScoreCreated
187f48ce-00a7-42c4-9f1f-e7ef1140cc61coder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:10:28.194438+00:00
ff0d0c92-5115-4034-a1c0-734a1e1664c4coder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:10:28.156522+00:00
e59e5264-10d9-440d-bee1-1538f24fd45ecoder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:10:28.120072+00:00
0244cb7f-8fce-4d16-a3c1-61286907e7afcoder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:10:28.090682+00:00
94f1e917-7bcd-45f5-ae87-91c8f13584cbcoder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:10:28.055819+00:00

Failure Breakdown

TaxonomyFailuresBar
wrong-logic62
##############################################################

Recent Failures

Run IDTask IDTaxonomyScoreCostCreated
187f48ce-00a7-42c4-9f1f-e7ef1140cc61python-recovery-easy-001wrong-logic0.740$0.00102026-04-27T16:10:28.194438+00:00
ff0d0c92-5115-4034-a1c0-734a1e1664c4typescript-config-easy-001wrong-logic0.740$0.00102026-04-27T16:10:28.156522+00:00
e59e5264-10d9-440d-bee1-1538f24fd45epython-config-easy-001wrong-logic0.740$0.00102026-04-27T16:10:28.120072+00:00
0244cb7f-8fce-4d16-a3c1-61286907e7aftypescript-refactor-easy-001wrong-logic0.740$0.00102026-04-27T16:10:28.090682+00:00
94f1e917-7bcd-45f5-ae87-91c8f13584cbpython-refactor-easy-001wrong-logic0.740$0.00102026-04-27T16:10:28.055819+00:00
c3edf1bb-0328-47e6-a18f-c0f08afd006ftypescript-multi-file-easy-001wrong-logic0.740$0.00102026-04-27T16:10:28.012569+00:00
933e8068-4e64-4b45-991e-5020675341e6python-multi-file-easy-001wrong-logic0.740$0.00102026-04-27T16:10:27.992973+00:00
bcd4ebea-f3f0-46cf-8638-8cfb638f8abatypescript-test-writing-easy-001wrong-logic0.740$0.00102026-04-27T16:10:27.965341+00:00
6e495d28-220f-4066-a714-db0653f58236python-test-writing-easy-001wrong-logic0.740$0.00102026-04-27T16:10:27.911374+00:00
723170c9-fe39-4377-add2-8865ef328c1btypescript-performance-easy-001wrong-logic0.740$0.00102026-04-27T16:10:27.848197+00:00

Cost Frontier

pass_rate vs cost_usd (Pareto frontier marked with *)
* [####################] 100.0% @ $0.0056  (coder)