Eval Report: ci-pr-smoke

Profile: gdm-swebench-lite-v1 | Tasks: 15 | Pass rate: 100.0% | Cost: $0.0150

Task ID	Band	Score	Passed	Cost
python-security-fix-easy-001	easy	0.740	✓	$0.0010
typescript-security-fix-easy-001	easy	0.740	✓	$0.0010
python-bugfix-easy-001	easy	0.740	✓	$0.0010
typescript-bugfix-easy-001	easy	0.740	✓	$0.0010
python-performance-easy-001	easy	0.740	✓	$0.0010
typescript-performance-easy-001	easy	0.740	✓	$0.0010
python-test-writing-easy-001	easy	0.740	✓	$0.0010
typescript-test-writing-easy-001	easy	0.740	✓	$0.0010
python-multi-file-easy-001	easy	0.740	✓	$0.0010
typescript-multi-file-easy-001	easy	0.740	✓	$0.0010
python-refactor-easy-001	easy	0.740	✓	$0.0010
typescript-refactor-easy-001	easy	0.740	✓	$0.0010
python-config-easy-001	easy	0.740	✓	$0.0010
typescript-config-easy-001	easy	0.740	✓	$0.0010
python-recovery-easy-001	easy	0.740	✓	$0.0010

Leaderboard Snapshot

Latest run: 0bfaa44c-76b0-4f60-80af-4d39533c78ee | Latest model: coder | Latest score: 0.740 | Recorded at: 2026-04-27T16:27:01.495394+00:00

Recent Trend

Run ID	Model	Git SHA	Score	Created
0bfaa44c-76b0-4f60-80af-4d39533c78ee	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:27:01.495394+00:00
61d71e9f-603f-4a3c-bd0f-c89103e948e1	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:27:01.370729+00:00
16a74901-1765-4ff5-b779-d70d863af41f	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:27:01.294506+00:00
ca9c9bcd-0213-40ab-8b90-6d7c8fe260c6	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:27:01.194015+00:00
ec9fc8b9-78db-4a93-bdf9-22c5b9fd6dbd	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:27:01.133723+00:00

Failure Breakdown

Taxonomy	Failures	Bar
wrong-logic	314	##########################################################################################################################################################################################################################################################################################################################

Taxonomy

Failures

Bar

wrong-logic

314

##########################################################################################################################################################################################################################################################################################################################

Recent Failures

Run ID	Task ID	Taxonomy	Score	Cost	Created
0bfaa44c-76b0-4f60-80af-4d39533c78ee	python-recovery-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:27:01.495394+00:00
61d71e9f-603f-4a3c-bd0f-c89103e948e1	typescript-config-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:27:01.370729+00:00
16a74901-1765-4ff5-b779-d70d863af41f	python-config-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:27:01.294506+00:00
ca9c9bcd-0213-40ab-8b90-6d7c8fe260c6	typescript-refactor-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:27:01.194015+00:00
ec9fc8b9-78db-4a93-bdf9-22c5b9fd6dbd	python-refactor-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:27:01.133723+00:00
d2bee63d-2a2a-4e80-b149-ce3273b4fb27	typescript-multi-file-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:27:01.046022+00:00
9fbf0040-dfc2-466a-956d-08f4388decb5	python-multi-file-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:27:00.955678+00:00
c5f22aa7-2ebf-4831-bb21-c115a7ca4195	typescript-test-writing-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:27:00.881808+00:00
b5471786-1cd7-4f7d-acff-fff9ee396bec	python-test-writing-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:27:00.767129+00:00
7db1a379-7720-4357-8b7f-068f984d502b	typescript-performance-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:27:00.688678+00:00

Cost Frontier

pass_rate vs cost_usd (Pareto frontier marked with *)
* [####################] 100.0% @ $0.0054  (coder)