Eval Report: ci-pr-smoke

Profile: gdm-swebench-lite-v1 | Tasks: 15 | Pass rate: 100.0% | Cost: $0.0150

Task ID	Band	Score	Passed	Cost
python-security-fix-easy-001	easy	0.740	✓	$0.0010
typescript-security-fix-easy-001	easy	0.740	✓	$0.0010
python-bugfix-easy-001	easy	0.740	✓	$0.0010
typescript-bugfix-easy-001	easy	0.740	✓	$0.0010
python-performance-easy-001	easy	0.740	✓	$0.0010
typescript-performance-easy-001	easy	0.740	✓	$0.0010
python-test-writing-easy-001	easy	0.740	✓	$0.0010
typescript-test-writing-easy-001	easy	0.740	✓	$0.0010
python-multi-file-easy-001	easy	0.740	✓	$0.0010
typescript-multi-file-easy-001	easy	0.740	✓	$0.0010
python-refactor-easy-001	easy	0.740	✓	$0.0010
typescript-refactor-easy-001	easy	0.740	✓	$0.0010
python-config-easy-001	easy	0.740	✓	$0.0010
typescript-config-easy-001	easy	0.740	✓	$0.0010
python-recovery-easy-001	easy	0.740	✓	$0.0010

Leaderboard Snapshot

Latest run: ee89089b-cc45-4efc-ae18-666be182b5c6 | Latest model: coder | Latest score: 0.740 | Recorded at: 2026-04-27T16:30:53.346072+00:00

Recent Trend

Run ID	Model	Git SHA	Score	Created
ee89089b-cc45-4efc-ae18-666be182b5c6	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:30:53.346072+00:00
18bf30f9-e25e-484e-8027-99b712852847	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:30:53.234283+00:00
3c0e6b23-e3a7-42ea-b4f3-efdcb992bfc3	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:30:53.132404+00:00
86fa10d3-5702-4f60-9da5-8d665a772dab	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:30:53.051331+00:00
a47603df-bae7-4ab9-ab7b-37148cb0c5d2	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:30:52.965320+00:00

Failure Breakdown

Taxonomy	Failures	Bar
wrong-logic	408	########################################################################################################################################################################################################################################################################################################################################################################################################################

Taxonomy

Failures

Bar

wrong-logic

408

########################################################################################################################################################################################################################################################################################################################################################################################################################

Recent Failures

Run ID	Task ID	Taxonomy	Score	Cost	Created
ee89089b-cc45-4efc-ae18-666be182b5c6	python-recovery-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:30:53.346072+00:00
18bf30f9-e25e-484e-8027-99b712852847	typescript-config-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:30:53.234283+00:00
3c0e6b23-e3a7-42ea-b4f3-efdcb992bfc3	python-config-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:30:53.132404+00:00
86fa10d3-5702-4f60-9da5-8d665a772dab	typescript-refactor-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:30:53.051331+00:00
a47603df-bae7-4ab9-ab7b-37148cb0c5d2	python-refactor-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:30:52.965320+00:00
14be093c-4619-49f6-9e23-e624e78f0ab5	typescript-multi-file-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:30:52.885871+00:00
c6875329-71cf-492f-bd9f-85f199ae1ec7	python-multi-file-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:30:52.760270+00:00
f1522627-9115-43d3-8910-710fabe406cd	typescript-test-writing-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:30:52.678183+00:00
78c2cfe2-8213-4293-ba27-fc33d239fd4e	python-test-writing-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:30:52.609999+00:00
2f84ce54-f8ff-43ed-aa78-7667ef3ac87f	typescript-performance-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:30:52.531856+00:00

Cost Frontier

pass_rate vs cost_usd (Pareto frontier marked with *)
* [####################] 100.0% @ $0.0053  (coder)