Eval Report: ci-pr-smoke

Profile: gdm-swebench-lite-v1 | Tasks: 15 | Pass rate: 100.0% | Cost: $0.0150

Task IDBandScorePassedCost
python-security-fix-easy-001easy0.740$0.0010
typescript-security-fix-easy-001easy0.740$0.0010
python-bugfix-easy-001easy0.740$0.0010
typescript-bugfix-easy-001easy0.740$0.0010
python-performance-easy-001easy0.740$0.0010
typescript-performance-easy-001easy0.740$0.0010
python-test-writing-easy-001easy0.740$0.0010
typescript-test-writing-easy-001easy0.740$0.0010
python-multi-file-easy-001easy0.740$0.0010
typescript-multi-file-easy-001easy0.740$0.0010
python-refactor-easy-001easy0.740$0.0010
typescript-refactor-easy-001easy0.740$0.0010
python-config-easy-001easy0.740$0.0010
typescript-config-easy-001easy0.740$0.0010
python-recovery-easy-001easy0.740$0.0010

Leaderboard Snapshot

Latest run: dee1db7c-ce07-4710-b07a-833848794aa8 | Latest model: coder | Latest score: 0.740 | Recorded at: 2026-04-27T16:14:32.387794+00:00

Recent Trend

Run IDModelGit SHAScoreCreated
dee1db7c-ce07-4710-b07a-833848794aa8coder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:14:32.387794+00:00
491336af-ea5d-42d4-821e-ea1552be6574coder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:14:32.314045+00:00
a8caf138-e4e4-4f26-a352-26abb1321769coder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:14:32.249922+00:00
ccf3f268-459e-4d9c-9d19-b73407b37a0ccoder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:14:32.194352+00:00
c7faff11-c7a3-4fc6-9936-2ff18b547349coder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:14:32.151347+00:00

Failure Breakdown

TaxonomyFailuresBar
wrong-logic156
############################################################################################################################################################

Recent Failures

Run IDTask IDTaxonomyScoreCostCreated
dee1db7c-ce07-4710-b07a-833848794aa8python-recovery-easy-001wrong-logic0.740$0.00102026-04-27T16:14:32.387794+00:00
491336af-ea5d-42d4-821e-ea1552be6574typescript-config-easy-001wrong-logic0.740$0.00102026-04-27T16:14:32.314045+00:00
a8caf138-e4e4-4f26-a352-26abb1321769python-config-easy-001wrong-logic0.740$0.00102026-04-27T16:14:32.249922+00:00
ccf3f268-459e-4d9c-9d19-b73407b37a0ctypescript-refactor-easy-001wrong-logic0.740$0.00102026-04-27T16:14:32.194352+00:00
c7faff11-c7a3-4fc6-9936-2ff18b547349python-refactor-easy-001wrong-logic0.740$0.00102026-04-27T16:14:32.151347+00:00
13b72298-2ff9-44c0-948c-9976e3c3ed71typescript-multi-file-easy-001wrong-logic0.740$0.00102026-04-27T16:14:32.101694+00:00
5e9627a1-0cda-4e92-afb3-745e8fc0fec3python-multi-file-easy-001wrong-logic0.740$0.00102026-04-27T16:14:32.044298+00:00
debd5260-6599-48a8-9713-fb0cb60972d4typescript-test-writing-easy-001wrong-logic0.740$0.00102026-04-27T16:14:31.977927+00:00
cb92e9bc-394d-4778-8535-55e704614f9apython-test-writing-easy-001wrong-logic0.740$0.00102026-04-27T16:14:31.865273+00:00
1cb71b23-c83d-4e3a-b851-67ac5bef3f41typescript-performance-easy-001wrong-logic0.740$0.00102026-04-27T16:14:31.807855+00:00

Cost Frontier

pass_rate vs cost_usd (Pareto frontier marked with *)
* [####################] 100.0% @ $0.0055  (coder)