Eval Report: ci-nightly

Profile: gdm-swebench-lite-v1 | Tasks: 8 | Pass rate: 37.5% | Cost: $0.0080

Task IDBandScorePassedCost
python-security-fix-easy-001easy0.740$0.0010
canary-typescript-session-003hard0.310$0.0010
canary-python-security-001hard0.310$0.0010
canary-typescript-auth-006hard0.310$0.0010
canary-python-cache-005hard0.310$0.0010
typescript-security-fix-easy-001easy0.740$0.0010
python-bugfix-easy-001easy0.740$0.0010
canary-shell-ops-004hard0.310$0.0010

Leaderboard Snapshot

Latest run: 111f619d-b00f-40cd-bbec-0eaa47d133ab | Latest model: coder | Latest score: 0.310 | Recorded at: 2026-04-27T17:13:10.216948+00:00

Recent Trend

Run IDModelGit SHAScoreCreated
111f619d-b00f-40cd-bbec-0eaa47d133abcoder4669773b4fbe9d507f1396f38777a1b36998faf30.3102026-04-27T17:13:10.216948+00:00
6a0103fd-6727-4468-9219-cce2f0d4c487coder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T17:13:10.125955+00:00
b9766030-19a5-454d-bde5-e3dcd2df25afcoder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T17:13:10.033585+00:00
53e986df-1c77-4ba6-85ab-4c4c851e847bcoder4669773b4fbe9d507f1396f38777a1b36998faf30.3102026-04-27T17:13:09.953441+00:00
5687afc6-7806-41a0-bf95-4aa39a50184bcoder4669773b4fbe9d507f1396f38777a1b36998faf30.3102026-04-27T17:13:09.874652+00:00

Failure Breakdown

TaxonomyFailuresBar
wrong-file5
#####
wrong-logic3
###

Recent Failures

Run IDTask IDTaxonomyScoreCostCreated
111f619d-b00f-40cd-bbec-0eaa47d133abcanary-shell-ops-004wrong-file0.310$0.00102026-04-27T17:13:10.216948+00:00
6a0103fd-6727-4468-9219-cce2f0d4c487python-bugfix-easy-001wrong-logic0.740$0.00102026-04-27T17:13:10.125955+00:00
b9766030-19a5-454d-bde5-e3dcd2df25aftypescript-security-fix-easy-001wrong-logic0.740$0.00102026-04-27T17:13:10.033585+00:00
53e986df-1c77-4ba6-85ab-4c4c851e847bcanary-python-cache-005wrong-file0.310$0.00102026-04-27T17:13:09.953441+00:00
5687afc6-7806-41a0-bf95-4aa39a50184bcanary-typescript-auth-006wrong-file0.310$0.00102026-04-27T17:13:09.874652+00:00
a29a61c2-74b3-4b01-9039-8abb84db9bcdcanary-python-security-001wrong-file0.310$0.00102026-04-27T17:13:09.790262+00:00
c851e2f0-dc8c-4444-9f9b-fd521d9fc71dcanary-typescript-session-003wrong-file0.310$0.00102026-04-27T17:13:09.702949+00:00
292afb14-abe9-4acd-a826-d258597605c9python-security-fix-easy-001wrong-logic0.740$0.00102026-04-27T17:13:09.610616+00:00

Cost Frontier

pass_rate vs cost_usd (Pareto frontier marked with *)
* [#######-------------] 37.5% @ $0.0010  (coder)