Eval Report: ci-pr-smoke

Profile: gdm-swebench-lite-v1 | Tasks: 2 | Pass rate: 100.0% | Cost: $0.0020

Task ID	Band	Score	Passed	Cost
python-security-fix-easy-001	easy	0.740	✓	$0.0010
typescript-security-fix-easy-001	easy	0.740	✓	$0.0010

Leaderboard Snapshot

Latest run: 2bd9018c-9c51-4e57-852b-d51c5b1587f4 | Latest model: coder | Latest score: 0.740 | Recorded at: 2026-04-27T16:26:08.507750+00:00

Recent Trend

Run ID	Model	Git SHA	Score	Created
2bd9018c-9c51-4e57-852b-d51c5b1587f4	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:26:08.507750+00:00
419a0c12-e37d-4811-ad3e-6a6efc77ba3e	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:26:08.419359+00:00
ab88deeb-da0b-433c-a317-1aefd4c4f701	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:26:07.936876+00:00
97f9b1d1-902b-4732-96f8-bf3974780aff	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:26:07.846515+00:00
11e49a8c-d127-4573-8326-aceb197f0f2d	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:26:07.787647+00:00

Failure Breakdown

Taxonomy	Failures	Bar
wrong-logic	252	############################################################################################################################################################################################################################################################

Taxonomy

Failures

Bar

wrong-logic

252

############################################################################################################################################################################################################################################################

Recent Failures

Run ID	Task ID	Taxonomy	Score	Cost	Created
2bd9018c-9c51-4e57-852b-d51c5b1587f4	typescript-security-fix-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:26:08.507750+00:00
419a0c12-e37d-4811-ad3e-6a6efc77ba3e	python-security-fix-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:26:08.419359+00:00
ab88deeb-da0b-433c-a317-1aefd4c4f701	python-recovery-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:26:07.936876+00:00
97f9b1d1-902b-4732-96f8-bf3974780aff	typescript-config-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:26:07.846515+00:00
11e49a8c-d127-4573-8326-aceb197f0f2d	python-config-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:26:07.787647+00:00
41023bc0-52c8-4f36-824f-734ff692b8a2	typescript-refactor-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:26:07.702442+00:00
510e34d9-e187-42cf-b3d5-bfc791ab2d85	python-refactor-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:26:07.616440+00:00
3fffafb4-8344-4f1a-a8d7-6ec071e415ac	typescript-multi-file-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:26:07.492856+00:00
4b79bedb-97b2-4507-9df2-ea98ced97327	python-multi-file-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:26:07.391639+00:00
d00c3723-b217-4246-92db-079d056daf51	typescript-test-writing-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:26:07.304229+00:00

Cost Frontier

pass_rate vs cost_usd (Pareto frontier marked with *)
* [####################] 100.0% @ $0.0054  (coder)