Eval Report: ci-post-merge

Profile: gdm-swebench-lite-v1 | Tasks: 50 | Pass rate: 100.0% | Cost: $0.0500

Task ID	Band	Score	Passed	Cost
python-security-fix-easy-001	easy	0.740	✓	$0.0010
typescript-security-fix-easy-001	easy	0.740	✓	$0.0010
python-bugfix-easy-001	easy	0.740	✓	$0.0010
typescript-bugfix-easy-001	easy	0.740	✓	$0.0010
python-performance-easy-001	easy	0.740	✓	$0.0010
typescript-performance-easy-001	easy	0.740	✓	$0.0010
python-test-writing-easy-001	easy	0.740	✓	$0.0010
typescript-test-writing-easy-001	easy	0.740	✓	$0.0010
python-multi-file-easy-001	easy	0.740	✓	$0.0010
typescript-multi-file-easy-001	easy	0.740	✓	$0.0010
python-refactor-easy-001	easy	0.740	✓	$0.0010
typescript-refactor-easy-001	easy	0.740	✓	$0.0010
python-config-easy-001	easy	0.740	✓	$0.0010
typescript-config-easy-001	easy	0.740	✓	$0.0010
python-recovery-easy-001	easy	0.740	✓	$0.0010
typescript-recovery-easy-001	easy	0.740	✓	$0.0010
python-dependency-easy-001	easy	0.740	✓	$0.0010
typescript-dependency-easy-001	easy	0.740	✓	$0.0010
python-explain-easy-001	easy	0.740	✓	$0.0010
typescript-explain-easy-001	easy	0.740	✓	$0.0010
python-security-fix-medium-001	medium	0.740	✓	$0.0010
shell-security-fix-medium-001	medium	0.740	✓	$0.0010
python-bugfix-medium-001	medium	0.740	✓	$0.0010
shell-bugfix-medium-001	medium	0.740	✓	$0.0010
python-performance-medium-001	medium	0.740	✓	$0.0010
shell-performance-medium-001	medium	0.740	✓	$0.0010
python-test-writing-medium-001	medium	0.740	✓	$0.0010
shell-test-writing-medium-001	medium	0.740	✓	$0.0010
python-multi-file-medium-001	medium	0.740	✓	$0.0010
shell-multi-file-medium-001	medium	0.740	✓	$0.0010
python-refactor-medium-001	medium	0.740	✓	$0.0010
shell-refactor-medium-001	medium	0.740	✓	$0.0010
python-config-medium-001	medium	0.740	✓	$0.0010
shell-config-medium-001	medium	0.740	✓	$0.0010
python-recovery-medium-001	medium	0.740	✓	$0.0010
shell-recovery-medium-001	medium	0.740	✓	$0.0010
python-dependency-medium-001	medium	0.740	✓	$0.0010
shell-dependency-medium-001	medium	0.740	✓	$0.0010
python-explain-medium-001	medium	0.740	✓	$0.0010
shell-explain-medium-001	medium	0.740	✓	$0.0010
python-security-fix-easy-001	easy	0.740	✓	$0.0010
typescript-security-fix-easy-001	easy	0.740	✓	$0.0010
python-bugfix-easy-001	easy	0.740	✓	$0.0010
typescript-bugfix-easy-001	easy	0.740	✓	$0.0010
python-performance-easy-001	easy	0.740	✓	$0.0010
typescript-performance-easy-001	easy	0.740	✓	$0.0010
python-test-writing-easy-001	easy	0.740	✓	$0.0010
typescript-test-writing-easy-001	easy	0.740	✓	$0.0010
python-multi-file-easy-001	easy	0.740	✓	$0.0010
typescript-multi-file-easy-001	easy	0.740	✓	$0.0010

Leaderboard Snapshot

Latest run: 11c73777-92af-453a-86ea-d40ad630ef6c | Latest model: coder | Latest score: 0.740 | Recorded at: 2026-04-27T16:14:41.212258+00:00

Recent Trend

Run ID	Model	Git SHA	Score	Created
11c73777-92af-453a-86ea-d40ad630ef6c	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:14:41.212258+00:00
7127abcd-a923-4245-8b10-e8d421530447	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:14:41.170446+00:00
49e1da2a-065a-4369-a96b-0a01b3969413	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:14:41.128920+00:00
f66e7356-23f6-45b9-8b0e-4352ee278001	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:14:41.070165+00:00
ab915b23-e292-445c-9ff1-1aa931f4d48c	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:14:40.996137+00:00

Failure Breakdown

Taxonomy	Failures	Bar
wrong-logic	400	################################################################################################################################################################################################################################################################################################################################################################################################################

Taxonomy

Failures

Bar

wrong-logic

400

################################################################################################################################################################################################################################################################################################################################################################################################################

Recent Failures

Run ID	Task ID	Taxonomy	Score	Cost	Created
11c73777-92af-453a-86ea-d40ad630ef6c	typescript-multi-file-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:14:41.212258+00:00
7127abcd-a923-4245-8b10-e8d421530447	python-multi-file-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:14:41.170446+00:00
49e1da2a-065a-4369-a96b-0a01b3969413	typescript-test-writing-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:14:41.128920+00:00
f66e7356-23f6-45b9-8b0e-4352ee278001	python-test-writing-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:14:41.070165+00:00
ab915b23-e292-445c-9ff1-1aa931f4d48c	typescript-performance-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:14:40.996137+00:00
2284b1f6-d6de-4742-ac40-7596de5af649	python-performance-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:14:40.958092+00:00
dceb3e83-6cb7-4b9c-9d2b-83accbc67266	typescript-bugfix-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:14:40.849505+00:00
119bae5d-e6b0-4800-a8d9-f65a9206fafe	python-bugfix-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:14:40.810349+00:00
e70c7004-13a7-4608-badb-bdc96666b6ce	typescript-security-fix-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:14:40.757691+00:00
d2ea3b99-f879-4a01-b124-fe691a48168e	python-security-fix-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:14:40.719685+00:00

Cost Frontier

pass_rate vs cost_usd (Pareto frontier marked with *)
* [####################] 100.0% @ $0.0056  (coder)