Eval Report: ci-pr-smoke

Profile: gdm-swebench-lite-v1 | Tasks: 15 | Pass rate: 100.0% | Cost: $0.0150

Task IDBandScorePassedCost
python-bugfix-easy-001easy0.740$0.0010
python-config-easy-001easy0.740$0.0010
python-dependency-easy-001easy0.740$0.0010
python-explain-easy-001easy0.740$0.0010
python-multi-file-easy-001easy0.740$0.0010
python-performance-easy-001easy0.740$0.0010
python-recovery-easy-001easy0.740$0.0010
python-refactor-easy-001easy0.740$0.0010
python-security-fix-easy-001easy0.740$0.0010
python-test-writing-easy-001easy0.740$0.0010
typescript-bugfix-easy-001easy0.740$0.0010
typescript-config-easy-001easy0.740$0.0010
typescript-dependency-easy-001easy0.740$0.0010
typescript-explain-easy-001easy0.740$0.0010
typescript-multi-file-easy-001easy0.740$0.0010

Leaderboard Snapshot

Latest run: 6209fc06-6dcc-434b-bf94-775356a2fc64 | Latest model: coder | Latest score: 0.740 | Recorded at: 2026-04-27T16:05:44.789461+00:00

Recent Trend

Run IDModelGit SHAScoreCreated
6209fc06-6dcc-434b-bf94-775356a2fc64coder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:05:44.789461+00:00
abf48a9b-4db2-47c5-b65c-5c2c5e681fa3coder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:05:44.749019+00:00
5a7afe15-8aa1-4e97-abad-fa609c91017dcoder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:05:44.712390+00:00
a684150b-c575-4cff-8529-eeb3f473fdcbcoder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:05:44.674436+00:00
2c1b5792-aaef-4fb9-b83e-cebfb43d5730coder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:05:44.632844+00:00

Failure Breakdown

TaxonomyFailuresBar
wrong-logic47
###############################################

Recent Failures

Run IDTask IDTaxonomyScoreCostCreated
6209fc06-6dcc-434b-bf94-775356a2fc64typescript-multi-file-easy-001wrong-logic0.740$0.00102026-04-27T16:05:44.789461+00:00
abf48a9b-4db2-47c5-b65c-5c2c5e681fa3typescript-explain-easy-001wrong-logic0.740$0.00102026-04-27T16:05:44.749019+00:00
5a7afe15-8aa1-4e97-abad-fa609c91017dtypescript-dependency-easy-001wrong-logic0.740$0.00102026-04-27T16:05:44.712390+00:00
a684150b-c575-4cff-8529-eeb3f473fdcbtypescript-config-easy-001wrong-logic0.740$0.00102026-04-27T16:05:44.674436+00:00
2c1b5792-aaef-4fb9-b83e-cebfb43d5730typescript-bugfix-easy-001wrong-logic0.740$0.00102026-04-27T16:05:44.632844+00:00
8f4b1fae-f3e5-4655-9b71-77898e9102c6python-test-writing-easy-001wrong-logic0.740$0.00102026-04-27T16:05:44.609658+00:00
ae3a73d2-5b65-4a5c-9fbe-8f38165897f9python-security-fix-easy-001wrong-logic0.740$0.00102026-04-27T16:05:44.580193+00:00
48686b39-226f-4131-9efc-de02c2f0451apython-refactor-easy-001wrong-logic0.740$0.00102026-04-27T16:05:44.546089+00:00
7798175b-63f1-4e2c-ac1c-9588cf2c4828python-recovery-easy-001wrong-logic0.740$0.00102026-04-27T16:05:44.516886+00:00
aa226740-6e79-4229-aa32-e059fb1bd1efpython-performance-easy-001wrong-logic0.740$0.00102026-04-27T16:05:44.476851+00:00

Cost Frontier

pass_rate vs cost_usd (Pareto frontier marked with *)
* [####################] 100.0% @ $0.0057  (coder)