Eval Report: ci-post-merge

Profile: gdm-swebench-lite-v1 | Tasks: 50 | Pass rate: 100.0% | Cost: $0.0500

Task ID	Band	Score	Passed	Cost
python-security-fix-easy-001	easy	0.740	✓	$0.0010
typescript-security-fix-easy-001	easy	0.740	✓	$0.0010
python-bugfix-easy-001	easy	0.740	✓	$0.0010
typescript-bugfix-easy-001	easy	0.740	✓	$0.0010
python-performance-easy-001	easy	0.740	✓	$0.0010
typescript-performance-easy-001	easy	0.740	✓	$0.0010
python-test-writing-easy-001	easy	0.740	✓	$0.0010
typescript-test-writing-easy-001	easy	0.740	✓	$0.0010
python-multi-file-easy-001	easy	0.740	✓	$0.0010
typescript-multi-file-easy-001	easy	0.740	✓	$0.0010
python-refactor-easy-001	easy	0.740	✓	$0.0010
typescript-refactor-easy-001	easy	0.740	✓	$0.0010
python-config-easy-001	easy	0.740	✓	$0.0010
typescript-config-easy-001	easy	0.740	✓	$0.0010
python-recovery-easy-001	easy	0.740	✓	$0.0010
typescript-recovery-easy-001	easy	0.740	✓	$0.0010
python-dependency-easy-001	easy	0.740	✓	$0.0010
typescript-dependency-easy-001	easy	0.740	✓	$0.0010
python-explain-easy-001	easy	0.740	✓	$0.0010
typescript-explain-easy-001	easy	0.740	✓	$0.0010
python-security-fix-medium-001	medium	0.740	✓	$0.0010
shell-security-fix-medium-001	medium	0.740	✓	$0.0010
python-bugfix-medium-001	medium	0.740	✓	$0.0010
shell-bugfix-medium-001	medium	0.740	✓	$0.0010
python-performance-medium-001	medium	0.740	✓	$0.0010
shell-performance-medium-001	medium	0.740	✓	$0.0010
python-test-writing-medium-001	medium	0.740	✓	$0.0010
shell-test-writing-medium-001	medium	0.740	✓	$0.0010
python-multi-file-medium-001	medium	0.740	✓	$0.0010
shell-multi-file-medium-001	medium	0.740	✓	$0.0010
python-refactor-medium-001	medium	0.740	✓	$0.0010
shell-refactor-medium-001	medium	0.740	✓	$0.0010
python-config-medium-001	medium	0.740	✓	$0.0010
shell-config-medium-001	medium	0.740	✓	$0.0010
python-recovery-medium-001	medium	0.740	✓	$0.0010
shell-recovery-medium-001	medium	0.740	✓	$0.0010
python-dependency-medium-001	medium	0.740	✓	$0.0010
shell-dependency-medium-001	medium	0.740	✓	$0.0010
python-explain-medium-001	medium	0.740	✓	$0.0010
shell-explain-medium-001	medium	0.740	✓	$0.0010
python-security-fix-easy-001	easy	0.740	✓	$0.0010
typescript-security-fix-easy-001	easy	0.740	✓	$0.0010
python-bugfix-easy-001	easy	0.740	✓	$0.0010
typescript-bugfix-easy-001	easy	0.740	✓	$0.0010
python-performance-easy-001	easy	0.740	✓	$0.0010
typescript-performance-easy-001	easy	0.740	✓	$0.0010
python-test-writing-easy-001	easy	0.740	✓	$0.0010
typescript-test-writing-easy-001	easy	0.740	✓	$0.0010
python-multi-file-easy-001	easy	0.740	✓	$0.0010
typescript-multi-file-easy-001	easy	0.740	✓	$0.0010

Leaderboard Snapshot

Latest run: 27583d32-2993-4b6d-96c9-0cd73e32d4a4 | Latest model: coder | Latest score: 0.740 | Recorded at: 2026-04-27T16:18:49.670944+00:00

Recent Trend

Run ID	Model	Git SHA	Score	Created
27583d32-2993-4b6d-96c9-0cd73e32d4a4	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:18:49.670944+00:00
a354d7d8-686c-437e-a48f-36d62c1e3ef3	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:18:49.600466+00:00
655e640b-4a80-45d1-a92f-bd3cab3e9715	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:18:49.526936+00:00
4502966d-2ab3-458f-9708-d14e34d2605e	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:18:49.443751+00:00
3898ad65-e7d5-4a52-b75d-c1b1b41aa717	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:18:49.375883+00:00

Failure Breakdown

Taxonomy	Failures	Bar
wrong-logic	450	##################################################################################################################################################################################################################################################################################################################################################################################################################################################################

Taxonomy

Failures

Bar

wrong-logic

450

##################################################################################################################################################################################################################################################################################################################################################################################################################################################################

Recent Failures

Run ID	Task ID	Taxonomy	Score	Cost	Created
27583d32-2993-4b6d-96c9-0cd73e32d4a4	typescript-multi-file-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:18:49.670944+00:00
a354d7d8-686c-437e-a48f-36d62c1e3ef3	python-multi-file-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:18:49.600466+00:00
655e640b-4a80-45d1-a92f-bd3cab3e9715	typescript-test-writing-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:18:49.526936+00:00
4502966d-2ab3-458f-9708-d14e34d2605e	python-test-writing-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:18:49.443751+00:00
3898ad65-e7d5-4a52-b75d-c1b1b41aa717	typescript-performance-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:18:49.375883+00:00
84893604-2e2f-48b8-8e03-18ca6594d167	python-performance-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:18:49.288034+00:00
4aff9805-3643-47b9-8761-46ce9bced098	typescript-bugfix-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:18:49.203405+00:00
cb487c38-ea1b-4e9e-8c24-941be92f7f15	python-bugfix-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:18:49.132056+00:00
f7d15373-13db-4833-acb2-256f7144678b	typescript-security-fix-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:18:49.061689+00:00
df335ee0-61c9-4bcb-8766-fdfac5dc4417	python-security-fix-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:18:48.996110+00:00

Cost Frontier

pass_rate vs cost_usd (Pareto frontier marked with *)
* [####################] 100.0% @ $0.0056  (coder)