Eval Report: ci-nightly

Profile: gdm-swebench-lite-v1 | Tasks: 300 | Pass rate: 100.0% | Cost: $0.3000

Task IDBandScorePassedCost
python-security-fix-easy-001easy0.740$0.0010
typescript-security-fix-easy-001easy0.740$0.0010
python-bugfix-easy-001easy0.740$0.0010
typescript-bugfix-easy-001easy0.740$0.0010
python-performance-easy-001easy0.740$0.0010
typescript-performance-easy-001easy0.740$0.0010
python-test-writing-easy-001easy0.740$0.0010
typescript-test-writing-easy-001easy0.740$0.0010
python-multi-file-easy-001easy0.740$0.0010
typescript-multi-file-easy-001easy0.740$0.0010
python-refactor-easy-001easy0.740$0.0010
typescript-refactor-easy-001easy0.740$0.0010
python-config-easy-001easy0.740$0.0010
typescript-config-easy-001easy0.740$0.0010
python-recovery-easy-001easy0.740$0.0010
typescript-recovery-easy-001easy0.740$0.0010
python-dependency-easy-001easy0.740$0.0010
typescript-dependency-easy-001easy0.740$0.0010
python-explain-easy-001easy0.740$0.0010
typescript-explain-easy-001easy0.740$0.0010
python-security-fix-medium-001medium0.740$0.0010
shell-security-fix-medium-001medium0.740$0.0010
python-bugfix-medium-001medium0.740$0.0010
shell-bugfix-medium-001medium0.740$0.0010
python-performance-medium-001medium0.740$0.0010
shell-performance-medium-001medium0.740$0.0010
python-test-writing-medium-001medium0.740$0.0010
shell-test-writing-medium-001medium0.740$0.0010
python-multi-file-medium-001medium0.740$0.0010
shell-multi-file-medium-001medium0.740$0.0010
python-refactor-medium-001medium0.740$0.0010
shell-refactor-medium-001medium0.740$0.0010
python-config-medium-001medium0.740$0.0010
shell-config-medium-001medium0.740$0.0010
python-recovery-medium-001medium0.740$0.0010
shell-recovery-medium-001medium0.740$0.0010
python-dependency-medium-001medium0.740$0.0010
shell-dependency-medium-001medium0.740$0.0010
python-explain-medium-001medium0.740$0.0010
shell-explain-medium-001medium0.740$0.0010
typescript-security-fix-hard-001hard0.740$0.0010
shell-security-fix-hard-001hard0.740$0.0010
typescript-bugfix-hard-001hard0.740$0.0010
shell-bugfix-hard-001hard0.740$0.0010
typescript-performance-hard-001hard0.740$0.0010
shell-performance-hard-001hard0.740$0.0010
typescript-test-writing-hard-001hard0.740$0.0010
shell-test-writing-hard-001hard0.740$0.0010
typescript-multi-file-hard-001hard0.740$0.0010
shell-multi-file-hard-001hard0.740$0.0010
typescript-refactor-hard-001hard0.740$0.0010
shell-refactor-hard-001hard0.740$0.0010
typescript-config-hard-001hard0.740$0.0010
shell-config-hard-001hard0.740$0.0010
typescript-recovery-hard-001hard0.740$0.0010
shell-recovery-hard-001hard0.740$0.0010
typescript-dependency-hard-001hard0.740$0.0010
shell-dependency-hard-001hard0.740$0.0010
typescript-explain-hard-001hard0.740$0.0010
shell-explain-hard-001hard0.740$0.0010
python-security-fix-easy-001easy0.740$0.0010
typescript-security-fix-easy-001easy0.740$0.0010
python-bugfix-easy-001easy0.740$0.0010
typescript-bugfix-easy-001easy0.740$0.0010
python-performance-easy-001easy0.740$0.0010
typescript-performance-easy-001easy0.740$0.0010
python-test-writing-easy-001easy0.740$0.0010
typescript-test-writing-easy-001easy0.740$0.0010
python-multi-file-easy-001easy0.740$0.0010
typescript-multi-file-easy-001easy0.740$0.0010
python-refactor-easy-001easy0.740$0.0010
typescript-refactor-easy-001easy0.740$0.0010
python-config-easy-001easy0.740$0.0010
typescript-config-easy-001easy0.740$0.0010
python-recovery-easy-001easy0.740$0.0010
typescript-recovery-easy-001easy0.740$0.0010
python-dependency-easy-001easy0.740$0.0010
typescript-dependency-easy-001easy0.740$0.0010
python-explain-easy-001easy0.740$0.0010
typescript-explain-easy-001easy0.740$0.0010
python-security-fix-medium-001medium0.740$0.0010
shell-security-fix-medium-001medium0.740$0.0010
python-bugfix-medium-001medium0.740$0.0010
shell-bugfix-medium-001medium0.740$0.0010
python-performance-medium-001medium0.740$0.0010
shell-performance-medium-001medium0.740$0.0010
python-test-writing-medium-001medium0.740$0.0010
shell-test-writing-medium-001medium0.740$0.0010
python-multi-file-medium-001medium0.740$0.0010
shell-multi-file-medium-001medium0.740$0.0010
python-refactor-medium-001medium0.740$0.0010
shell-refactor-medium-001medium0.740$0.0010
python-config-medium-001medium0.740$0.0010
shell-config-medium-001medium0.740$0.0010
python-recovery-medium-001medium0.740$0.0010
shell-recovery-medium-001medium0.740$0.0010
python-dependency-medium-001medium0.740$0.0010
shell-dependency-medium-001medium0.740$0.0010
python-explain-medium-001medium0.740$0.0010
shell-explain-medium-001medium0.740$0.0010
typescript-security-fix-hard-001hard0.740$0.0010
shell-security-fix-hard-001hard0.740$0.0010
typescript-bugfix-hard-001hard0.740$0.0010
shell-bugfix-hard-001hard0.740$0.0010
typescript-performance-hard-001hard0.740$0.0010
shell-performance-hard-001hard0.740$0.0010
typescript-test-writing-hard-001hard0.740$0.0010
shell-test-writing-hard-001hard0.740$0.0010
typescript-multi-file-hard-001hard0.740$0.0010
shell-multi-file-hard-001hard0.740$0.0010
typescript-refactor-hard-001hard0.740$0.0010
shell-refactor-hard-001hard0.740$0.0010
typescript-config-hard-001hard0.740$0.0010
shell-config-hard-001hard0.740$0.0010
typescript-recovery-hard-001hard0.740$0.0010
shell-recovery-hard-001hard0.740$0.0010
typescript-dependency-hard-001hard0.740$0.0010
shell-dependency-hard-001hard0.740$0.0010
typescript-explain-hard-001hard0.740$0.0010
shell-explain-hard-001hard0.740$0.0010
python-security-fix-easy-001easy0.740$0.0010
typescript-security-fix-easy-001easy0.740$0.0010
python-bugfix-easy-001easy0.740$0.0010
typescript-bugfix-easy-001easy0.740$0.0010
python-performance-easy-001easy0.740$0.0010
typescript-performance-easy-001easy0.740$0.0010
python-test-writing-easy-001easy0.740$0.0010
typescript-test-writing-easy-001easy0.740$0.0010
python-multi-file-easy-001easy0.740$0.0010
typescript-multi-file-easy-001easy0.740$0.0010
python-refactor-easy-001easy0.740$0.0010
typescript-refactor-easy-001easy0.740$0.0010
python-config-easy-001easy0.740$0.0010
typescript-config-easy-001easy0.740$0.0010
python-recovery-easy-001easy0.740$0.0010
typescript-recovery-easy-001easy0.740$0.0010
python-dependency-easy-001easy0.740$0.0010
typescript-dependency-easy-001easy0.740$0.0010
python-explain-easy-001easy0.740$0.0010
typescript-explain-easy-001easy0.740$0.0010
python-security-fix-medium-001medium0.740$0.0010
shell-security-fix-medium-001medium0.740$0.0010
python-bugfix-medium-001medium0.740$0.0010
shell-bugfix-medium-001medium0.740$0.0010
python-performance-medium-001medium0.740$0.0010
shell-performance-medium-001medium0.740$0.0010
python-test-writing-medium-001medium0.740$0.0010
shell-test-writing-medium-001medium0.740$0.0010
python-multi-file-medium-001medium0.740$0.0010
shell-multi-file-medium-001medium0.740$0.0010
python-refactor-medium-001medium0.740$0.0010
shell-refactor-medium-001medium0.740$0.0010
python-config-medium-001medium0.740$0.0010
shell-config-medium-001medium0.740$0.0010
python-recovery-medium-001medium0.740$0.0010
shell-recovery-medium-001medium0.740$0.0010
python-dependency-medium-001medium0.740$0.0010
shell-dependency-medium-001medium0.740$0.0010
python-explain-medium-001medium0.740$0.0010
shell-explain-medium-001medium0.740$0.0010
typescript-security-fix-hard-001hard0.740$0.0010
shell-security-fix-hard-001hard0.740$0.0010
typescript-bugfix-hard-001hard0.740$0.0010
shell-bugfix-hard-001hard0.740$0.0010
typescript-performance-hard-001hard0.740$0.0010
shell-performance-hard-001hard0.740$0.0010
typescript-test-writing-hard-001hard0.740$0.0010
shell-test-writing-hard-001hard0.740$0.0010
typescript-multi-file-hard-001hard0.740$0.0010
shell-multi-file-hard-001hard0.740$0.0010
typescript-refactor-hard-001hard0.740$0.0010
shell-refactor-hard-001hard0.740$0.0010
typescript-config-hard-001hard0.740$0.0010
shell-config-hard-001hard0.740$0.0010
typescript-recovery-hard-001hard0.740$0.0010
shell-recovery-hard-001hard0.740$0.0010
typescript-dependency-hard-001hard0.740$0.0010
shell-dependency-hard-001hard0.740$0.0010
typescript-explain-hard-001hard0.740$0.0010
shell-explain-hard-001hard0.740$0.0010
python-security-fix-easy-001easy0.740$0.0010
typescript-security-fix-easy-001easy0.740$0.0010
python-bugfix-easy-001easy0.740$0.0010
typescript-bugfix-easy-001easy0.740$0.0010
python-performance-easy-001easy0.740$0.0010
typescript-performance-easy-001easy0.740$0.0010
python-test-writing-easy-001easy0.740$0.0010
typescript-test-writing-easy-001easy0.740$0.0010
python-multi-file-easy-001easy0.740$0.0010
typescript-multi-file-easy-001easy0.740$0.0010
python-refactor-easy-001easy0.740$0.0010
typescript-refactor-easy-001easy0.740$0.0010
python-config-easy-001easy0.740$0.0010
typescript-config-easy-001easy0.740$0.0010
python-recovery-easy-001easy0.740$0.0010
typescript-recovery-easy-001easy0.740$0.0010
python-dependency-easy-001easy0.740$0.0010
typescript-dependency-easy-001easy0.740$0.0010
python-explain-easy-001easy0.740$0.0010
typescript-explain-easy-001easy0.740$0.0010
python-security-fix-medium-001medium0.740$0.0010
shell-security-fix-medium-001medium0.740$0.0010
python-bugfix-medium-001medium0.740$0.0010
shell-bugfix-medium-001medium0.740$0.0010
python-performance-medium-001medium0.740$0.0010
shell-performance-medium-001medium0.740$0.0010
python-test-writing-medium-001medium0.740$0.0010
shell-test-writing-medium-001medium0.740$0.0010
python-multi-file-medium-001medium0.740$0.0010
shell-multi-file-medium-001medium0.740$0.0010
python-refactor-medium-001medium0.740$0.0010
shell-refactor-medium-001medium0.740$0.0010
python-config-medium-001medium0.740$0.0010
shell-config-medium-001medium0.740$0.0010
python-recovery-medium-001medium0.740$0.0010
shell-recovery-medium-001medium0.740$0.0010
python-dependency-medium-001medium0.740$0.0010
shell-dependency-medium-001medium0.740$0.0010
python-explain-medium-001medium0.740$0.0010
shell-explain-medium-001medium0.740$0.0010
typescript-security-fix-hard-001hard0.740$0.0010
shell-security-fix-hard-001hard0.740$0.0010
typescript-bugfix-hard-001hard0.740$0.0010
shell-bugfix-hard-001hard0.740$0.0010
typescript-performance-hard-001hard0.740$0.0010
shell-performance-hard-001hard0.740$0.0010
typescript-test-writing-hard-001hard0.740$0.0010
shell-test-writing-hard-001hard0.740$0.0010
typescript-multi-file-hard-001hard0.740$0.0010
shell-multi-file-hard-001hard0.740$0.0010
typescript-refactor-hard-001hard0.740$0.0010
shell-refactor-hard-001hard0.740$0.0010
typescript-config-hard-001hard0.740$0.0010
shell-config-hard-001hard0.740$0.0010
typescript-recovery-hard-001hard0.740$0.0010
shell-recovery-hard-001hard0.740$0.0010
typescript-dependency-hard-001hard0.740$0.0010
shell-dependency-hard-001hard0.740$0.0010
typescript-explain-hard-001hard0.740$0.0010
shell-explain-hard-001hard0.740$0.0010
python-security-fix-easy-001easy0.740$0.0010
typescript-security-fix-easy-001easy0.740$0.0010
python-bugfix-easy-001easy0.740$0.0010
typescript-bugfix-easy-001easy0.740$0.0010
python-performance-easy-001easy0.740$0.0010
typescript-performance-easy-001easy0.740$0.0010
python-test-writing-easy-001easy0.740$0.0010
typescript-test-writing-easy-001easy0.740$0.0010
python-multi-file-easy-001easy0.740$0.0010
typescript-multi-file-easy-001easy0.740$0.0010
python-refactor-easy-001easy0.740$0.0010
typescript-refactor-easy-001easy0.740$0.0010
python-config-easy-001easy0.740$0.0010
typescript-config-easy-001easy0.740$0.0010
python-recovery-easy-001easy0.740$0.0010
typescript-recovery-easy-001easy0.740$0.0010
python-dependency-easy-001easy0.740$0.0010
typescript-dependency-easy-001easy0.740$0.0010
python-explain-easy-001easy0.740$0.0010
typescript-explain-easy-001easy0.740$0.0010
python-security-fix-medium-001medium0.740$0.0010
shell-security-fix-medium-001medium0.740$0.0010
python-bugfix-medium-001medium0.740$0.0010
shell-bugfix-medium-001medium0.740$0.0010
python-performance-medium-001medium0.740$0.0010
shell-performance-medium-001medium0.740$0.0010
python-test-writing-medium-001medium0.740$0.0010
shell-test-writing-medium-001medium0.740$0.0010
python-multi-file-medium-001medium0.740$0.0010
shell-multi-file-medium-001medium0.740$0.0010
python-refactor-medium-001medium0.740$0.0010
shell-refactor-medium-001medium0.740$0.0010
python-config-medium-001medium0.740$0.0010
shell-config-medium-001medium0.740$0.0010
python-recovery-medium-001medium0.740$0.0010
shell-recovery-medium-001medium0.740$0.0010
python-dependency-medium-001medium0.740$0.0010
shell-dependency-medium-001medium0.740$0.0010
python-explain-medium-001medium0.740$0.0010
shell-explain-medium-001medium0.740$0.0010
typescript-security-fix-hard-001hard0.740$0.0010
shell-security-fix-hard-001hard0.740$0.0010
typescript-bugfix-hard-001hard0.740$0.0010
shell-bugfix-hard-001hard0.740$0.0010
typescript-performance-hard-001hard0.740$0.0010
shell-performance-hard-001hard0.740$0.0010
typescript-test-writing-hard-001hard0.740$0.0010
shell-test-writing-hard-001hard0.740$0.0010
typescript-multi-file-hard-001hard0.740$0.0010
shell-multi-file-hard-001hard0.740$0.0010
typescript-refactor-hard-001hard0.740$0.0010
shell-refactor-hard-001hard0.740$0.0010
typescript-config-hard-001hard0.740$0.0010
shell-config-hard-001hard0.740$0.0010
typescript-recovery-hard-001hard0.740$0.0010
shell-recovery-hard-001hard0.740$0.0010
typescript-dependency-hard-001hard0.740$0.0010
shell-dependency-hard-001hard0.740$0.0010
typescript-explain-hard-001hard0.740$0.0010
shell-explain-hard-001hard0.740$0.0010

Leaderboard Snapshot

Latest run: fb94a095-5b8a-47c1-a966-99bcac1f9c99 | Latest model: coder | Latest score: 0.740 | Recorded at: 2026-04-27T16:59:31.377280+00:00

Recent Trend

Run IDModelGit SHAScoreCreated
fb94a095-5b8a-47c1-a966-99bcac1f9c99coder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:59:31.377280+00:00
aa2ece66-36a9-466f-b321-917c7f877819coder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:59:31.305532+00:00
7c6bd19b-f3ca-4530-9dae-c1b9d6e332dbcoder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:59:31.240704+00:00
4fd31c2c-2436-4f31-81fa-99e7e1a04962coder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:59:31.153548+00:00
f1b11e8a-22b1-4d62-8102-94bd1eae288fcoder4669773b4fbe9d507f1396f38777a1b36998faf30.7402026-04-27T16:59:31.087606+00:00

Failure Breakdown

TaxonomyFailuresBar
wrong-logic3600
################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################

Recent Failures

Run IDTask IDTaxonomyScoreCostCreated
fb94a095-5b8a-47c1-a966-99bcac1f9c99shell-explain-hard-001wrong-logic0.740$0.00102026-04-27T16:59:31.377280+00:00
aa2ece66-36a9-466f-b321-917c7f877819typescript-explain-hard-001wrong-logic0.740$0.00102026-04-27T16:59:31.305532+00:00
7c6bd19b-f3ca-4530-9dae-c1b9d6e332dbshell-dependency-hard-001wrong-logic0.740$0.00102026-04-27T16:59:31.240704+00:00
4fd31c2c-2436-4f31-81fa-99e7e1a04962typescript-dependency-hard-001wrong-logic0.740$0.00102026-04-27T16:59:31.153548+00:00
f1b11e8a-22b1-4d62-8102-94bd1eae288fshell-recovery-hard-001wrong-logic0.740$0.00102026-04-27T16:59:31.087606+00:00
aabf7d4e-0104-43db-b10d-2044c57a7ddftypescript-recovery-hard-001wrong-logic0.740$0.00102026-04-27T16:59:31.014974+00:00
e32fe9a1-db1b-42f8-b58a-274b522453d2shell-config-hard-001wrong-logic0.740$0.00102026-04-27T16:59:30.952893+00:00
d602c1b0-f380-4765-bf48-aeab472a04b1typescript-config-hard-001wrong-logic0.740$0.00102026-04-27T16:59:30.895021+00:00
96d15ecb-6507-4263-ae32-eb4a3469b725shell-refactor-hard-001wrong-logic0.740$0.00102026-04-27T16:59:30.828553+00:00
4b467d3f-c5d6-4ed0-bb14-fdfc2d061911typescript-refactor-hard-001wrong-logic0.740$0.00102026-04-27T16:59:30.761869+00:00

Cost Frontier

pass_rate vs cost_usd (Pareto frontier marked with *)
* [###################-] 100.0% @ $0.0057  (coder)