Eval Report: ci-pr-smoke

Profile: gdm-swebench-lite-v1 | Tasks: 2 | Pass rate: 100.0% | Cost: $0.0020

Task IDBandScorePassedCost
python-security-fix-easy-001easy0.740$0.0010
typescript-security-fix-easy-001easy0.740$0.0010

Leaderboard Snapshot

Latest run: 9891df5b-b3e7-4904-ac6c-b756915e9d37 | Latest model: coder | Latest score: 0.740 | Recorded at: 2026-04-27T16:14:32.749084+00:00

Recent Trend

Run IDModelGit SHAScoreCreated
9891df5b-b3e7-4904-ac6c-b756915e9d37coder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:14:32.749084+00:00
92541717-ec8b-4d17-a4b5-4e0db7bc9857coder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:14:32.680215+00:00
dee1db7c-ce07-4710-b07a-833848794aa8coder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:14:32.387794+00:00
491336af-ea5d-42d4-821e-ea1552be6574coder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:14:32.314045+00:00
a8caf138-e4e4-4f26-a352-26abb1321769coder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:14:32.249922+00:00

Failure Breakdown

TaxonomyFailuresBar
wrong-logic158
##############################################################################################################################################################

Recent Failures

Run IDTask IDTaxonomyScoreCostCreated
9891df5b-b3e7-4904-ac6c-b756915e9d37typescript-security-fix-easy-001wrong-logic0.740$0.00102026-04-27T16:14:32.749084+00:00
92541717-ec8b-4d17-a4b5-4e0db7bc9857python-security-fix-easy-001wrong-logic0.740$0.00102026-04-27T16:14:32.680215+00:00
dee1db7c-ce07-4710-b07a-833848794aa8python-recovery-easy-001wrong-logic0.740$0.00102026-04-27T16:14:32.387794+00:00
491336af-ea5d-42d4-821e-ea1552be6574typescript-config-easy-001wrong-logic0.740$0.00102026-04-27T16:14:32.314045+00:00
a8caf138-e4e4-4f26-a352-26abb1321769python-config-easy-001wrong-logic0.740$0.00102026-04-27T16:14:32.249922+00:00
ccf3f268-459e-4d9c-9d19-b73407b37a0ctypescript-refactor-easy-001wrong-logic0.740$0.00102026-04-27T16:14:32.194352+00:00
c7faff11-c7a3-4fc6-9936-2ff18b547349python-refactor-easy-001wrong-logic0.740$0.00102026-04-27T16:14:32.151347+00:00
13b72298-2ff9-44c0-948c-9976e3c3ed71typescript-multi-file-easy-001wrong-logic0.740$0.00102026-04-27T16:14:32.101694+00:00
5e9627a1-0cda-4e92-afb3-745e8fc0fec3python-multi-file-easy-001wrong-logic0.740$0.00102026-04-27T16:14:32.044298+00:00
debd5260-6599-48a8-9713-fb0cb60972d4typescript-test-writing-easy-001wrong-logic0.740$0.00102026-04-27T16:14:31.977927+00:00

Cost Frontier

pass_rate vs cost_usd (Pareto frontier marked with *)
* [####################] 100.0% @ $0.0055  (coder)