Eval Report: ci-pr-smoke

Profile: gdm-swebench-lite-v1 | Tasks: 15 | Pass rate: 100.0% | Cost: $0.0150

Task ID	Band	Score	Passed	Cost
python-security-fix-easy-001	easy	0.740	✓	$0.0010
typescript-security-fix-easy-001	easy	0.740	✓	$0.0010
python-bugfix-easy-001	easy	0.740	✓	$0.0010
typescript-bugfix-easy-001	easy	0.740	✓	$0.0010
python-performance-easy-001	easy	0.740	✓	$0.0010
typescript-performance-easy-001	easy	0.740	✓	$0.0010
python-test-writing-easy-001	easy	0.740	✓	$0.0010
typescript-test-writing-easy-001	easy	0.740	✓	$0.0010
python-multi-file-easy-001	easy	0.740	✓	$0.0010
typescript-multi-file-easy-001	easy	0.740	✓	$0.0010
python-refactor-easy-001	easy	0.740	✓	$0.0010
typescript-refactor-easy-001	easy	0.740	✓	$0.0010
python-config-easy-001	easy	0.740	✓	$0.0010
typescript-config-easy-001	easy	0.740	✓	$0.0010
python-recovery-easy-001	easy	0.740	✓	$0.0010

Leaderboard Snapshot

Latest run: ab88deeb-da0b-433c-a317-1aefd4c4f701 | Latest model: coder | Latest score: 0.740 | Recorded at: 2026-04-27T16:26:07.936876+00:00

Recent Trend

Run ID	Model	Git SHA	Score	Created
ab88deeb-da0b-433c-a317-1aefd4c4f701	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:26:07.936876+00:00
97f9b1d1-902b-4732-96f8-bf3974780aff	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:26:07.846515+00:00
11e49a8c-d127-4573-8326-aceb197f0f2d	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:26:07.787647+00:00
41023bc0-52c8-4f36-824f-734ff692b8a2	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:26:07.702442+00:00
510e34d9-e187-42cf-b3d5-bfc791ab2d85	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:26:07.616440+00:00

Failure Breakdown

Taxonomy	Failures	Bar
wrong-logic	250	##########################################################################################################################################################################################################################################################

Taxonomy

Failures

Bar

wrong-logic

250

##########################################################################################################################################################################################################################################################

Recent Failures

Run ID	Task ID	Taxonomy	Score	Cost	Created
ab88deeb-da0b-433c-a317-1aefd4c4f701	python-recovery-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:26:07.936876+00:00
97f9b1d1-902b-4732-96f8-bf3974780aff	typescript-config-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:26:07.846515+00:00
11e49a8c-d127-4573-8326-aceb197f0f2d	python-config-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:26:07.787647+00:00
41023bc0-52c8-4f36-824f-734ff692b8a2	typescript-refactor-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:26:07.702442+00:00
510e34d9-e187-42cf-b3d5-bfc791ab2d85	python-refactor-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:26:07.616440+00:00
3fffafb4-8344-4f1a-a8d7-6ec071e415ac	typescript-multi-file-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:26:07.492856+00:00
4b79bedb-97b2-4507-9df2-ea98ced97327	python-multi-file-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:26:07.391639+00:00
d00c3723-b217-4246-92db-079d056daf51	typescript-test-writing-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:26:07.304229+00:00
fc79baa8-e1b6-4bba-bfb6-c35952573e01	python-test-writing-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:26:07.230756+00:00
acd78248-7a98-4a10-b6e0-a9596d46b768	typescript-performance-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:26:07.090173+00:00

Cost Frontier

pass_rate vs cost_usd (Pareto frontier marked with *)
* [####################] 100.0% @ $0.0054  (coder)