Eval Report: ci-pr-smoke

Profile: gdm-swebench-lite-v1 | Tasks: 15 | Pass rate: 100.0% | Cost: $0.0150

Task ID	Band	Score	Passed	Cost
python-security-fix-easy-001	easy	0.740	✓	$0.0010
typescript-security-fix-easy-001	easy	0.740	✓	$0.0010
python-bugfix-easy-001	easy	0.740	✓	$0.0010
typescript-bugfix-easy-001	easy	0.740	✓	$0.0010
python-performance-easy-001	easy	0.740	✓	$0.0010
typescript-performance-easy-001	easy	0.740	✓	$0.0010
python-test-writing-easy-001	easy	0.740	✓	$0.0010
typescript-test-writing-easy-001	easy	0.740	✓	$0.0010
python-multi-file-easy-001	easy	0.740	✓	$0.0010
typescript-multi-file-easy-001	easy	0.740	✓	$0.0010
python-refactor-easy-001	easy	0.740	✓	$0.0010
typescript-refactor-easy-001	easy	0.740	✓	$0.0010
python-config-easy-001	easy	0.740	✓	$0.0010
typescript-config-easy-001	easy	0.740	✓	$0.0010
python-recovery-easy-001	easy	0.740	✓	$0.0010

Leaderboard Snapshot

Latest run: ea9299da-1a96-41fe-b3dd-b39e8b5e9077 | Latest model: coder | Latest score: 0.740 | Recorded at: 2026-04-27T16:18:51.331703+00:00

Recent Trend

Run ID	Model	Git SHA	Score	Created
ea9299da-1a96-41fe-b3dd-b39e8b5e9077	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:18:51.331703+00:00
ddea4704-ceab-40e5-a875-78427e02cbc7	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:18:51.257179+00:00
3b626c50-e082-446b-90d3-ec1181899ff9	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:18:51.163982+00:00
7ffaa278-85a9-433d-8948-d7971480f048	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:18:51.082056+00:00
baa10dc9-282a-4f6d-9f92-f32f65a35470	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:18:51.021607+00:00

Failure Breakdown

Taxonomy	Failures	Bar
wrong-logic	235	###########################################################################################################################################################################################################################################

Taxonomy

Failures

Bar

wrong-logic

235

###########################################################################################################################################################################################################################################

Recent Failures

Run ID	Task ID	Taxonomy	Score	Cost	Created
ea9299da-1a96-41fe-b3dd-b39e8b5e9077	python-recovery-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:18:51.331703+00:00
ddea4704-ceab-40e5-a875-78427e02cbc7	typescript-config-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:18:51.257179+00:00
3b626c50-e082-446b-90d3-ec1181899ff9	python-config-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:18:51.163982+00:00
7ffaa278-85a9-433d-8948-d7971480f048	typescript-refactor-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:18:51.082056+00:00
baa10dc9-282a-4f6d-9f92-f32f65a35470	python-refactor-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:18:51.021607+00:00
9d6ef5f5-ed36-4de0-a6e3-806f782fc7f5	typescript-multi-file-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:18:50.966138+00:00
809d18cd-67ca-493a-ac45-d8995367d82f	python-multi-file-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:18:50.886412+00:00
6708d6b2-0386-4efd-862d-1b9426a692a8	typescript-test-writing-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:18:50.822814+00:00
90530486-ccdd-476b-ad2a-21bc1cba4948	python-test-writing-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:18:50.747380+00:00
8c60f4e8-b383-403c-9922-5f9b77fd2815	typescript-performance-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:18:50.685352+00:00

Cost Frontier

pass_rate vs cost_usd (Pareto frontier marked with *)
* [####################] 100.0% @ $0.0055  (coder)