Eval Report: ci-pr-smoke

Profile: gdm-swebench-lite-v1 | Tasks: 15 | Pass rate: 100.0% | Cost: $0.0150

Task ID	Band	Score	Passed	Cost
python-security-fix-easy-001	easy	0.740	✓	$0.0010
typescript-security-fix-easy-001	easy	0.740	✓	$0.0010
python-bugfix-easy-001	easy	0.740	✓	$0.0010
typescript-bugfix-easy-001	easy	0.740	✓	$0.0010
python-performance-easy-001	easy	0.740	✓	$0.0010
typescript-performance-easy-001	easy	0.740	✓	$0.0010
python-test-writing-easy-001	easy	0.740	✓	$0.0010
typescript-test-writing-easy-001	easy	0.740	✓	$0.0010
python-multi-file-easy-001	easy	0.740	✓	$0.0010
typescript-multi-file-easy-001	easy	0.740	✓	$0.0010
python-refactor-easy-001	easy	0.740	✓	$0.0010
typescript-refactor-easy-001	easy	0.740	✓	$0.0010
python-config-easy-001	easy	0.740	✓	$0.0010
typescript-config-easy-001	easy	0.740	✓	$0.0010
python-recovery-easy-001	easy	0.740	✓	$0.0010

Leaderboard Snapshot

Latest run: 094feb3b-0505-45cb-8ea6-a66caeede468 | Latest model: coder | Latest score: 0.740 | Recorded at: 2026-04-27T16:51:50.584211+00:00

Recent Trend

Run ID	Model	Git SHA	Score	Created
094feb3b-0505-45cb-8ea6-a66caeede468	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:51:50.584211+00:00
4fa67e5d-c8f6-4b3d-a5ba-807b8d3367c6	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:51:50.517148+00:00
fad950ed-e471-4cec-bf23-efaf2edc0aa0	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:51:50.441024+00:00
3d6f706a-5b12-4487-821d-956d736649c0	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:51:50.368522+00:00
87e1a0eb-1e51-4e8b-a489-3ee9c6911e3c	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:51:50.279380+00:00

Failure Breakdown

Taxonomy	Failures	Bar
wrong-logic	438	######################################################################################################################################################################################################################################################################################################################################################################################################################################################

Taxonomy

Failures

Bar

wrong-logic

438

######################################################################################################################################################################################################################################################################################################################################################################################################################################################

Recent Failures

Run ID	Task ID	Taxonomy	Score	Cost	Created
094feb3b-0505-45cb-8ea6-a66caeede468	python-recovery-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:51:50.584211+00:00
4fa67e5d-c8f6-4b3d-a5ba-807b8d3367c6	typescript-config-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:51:50.517148+00:00
fad950ed-e471-4cec-bf23-efaf2edc0aa0	python-config-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:51:50.441024+00:00
3d6f706a-5b12-4487-821d-956d736649c0	typescript-refactor-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:51:50.368522+00:00
87e1a0eb-1e51-4e8b-a489-3ee9c6911e3c	python-refactor-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:51:50.279380+00:00
9420870c-0a2d-4697-bf1c-e5e3db20034c	typescript-multi-file-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:51:50.215093+00:00
30a678eb-06c9-473f-9b06-f822e0f7f5a4	python-multi-file-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:51:50.149620+00:00
b00f2a73-f160-4535-b415-0aeee8bd2d7f	typescript-test-writing-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:51:50.069290+00:00
3a940c09-5772-484d-ad0a-e20e8fff7c3d	python-test-writing-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:51:49.984269+00:00
6c48b2f5-7008-4e54-b387-1593f15b9f2a	typescript-performance-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:51:49.908790+00:00

Cost Frontier

pass_rate vs cost_usd (Pareto frontier marked with *)
* [####################] 100.0% @ $0.0052  (coder)