Eval Report: ci-nightly

Profile: gdm-swebench-lite-v1 | Tasks: 8 | Pass rate: 37.5% | Cost: $0.0080

Task ID	Band	Score	Passed	Cost
python-bugfix-easy-001	easy	0.740	✓	$0.0010
canary-typescript-session-003	hard	0.310	✗	$0.0010
canary-python-security-001	hard	0.310	✗	$0.0010
canary-typescript-auth-006	hard	0.310	✗	$0.0010
canary-python-cache-005	hard	0.310	✗	$0.0010
python-config-easy-001	easy	0.740	✓	$0.0010
python-dependency-easy-001	easy	0.740	✓	$0.0010
canary-shell-ops-004	hard	0.310	✗	$0.0010

Leaderboard Snapshot

Latest run: a5c60717-5a90-4d04-8ef8-a9812218b7af | Latest model: coder | Latest score: 0.310 | Recorded at: 2026-04-27T16:00:54.295822+00:00

Run ID	Model	Git SHA	Score	Created
a5c60717-5a90-4d04-8ef8-a9812218b7af	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.310	2026-04-27T16:00:54.295822+00:00
7b676907-e55b-45a1-ae6c-c373cb05d09b	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:00:54.263680+00:00
295a362c-a327-4f8a-ba8d-dd9302103116	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T16:00:54.215820+00:00
3595da29-4f83-420e-a8f6-4cd7ca84309d	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.310	2026-04-27T16:00:54.168436+00:00
1f3a7798-7144-4dca-a44f-3a8322549f7d	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.310	2026-04-27T16:00:54.100564+00:00

Taxonomy	Failures	Bar
wrong-file	5	#####
wrong-logic	3	###

Run ID	Task ID	Taxonomy	Score	Cost	Created
a5c60717-5a90-4d04-8ef8-a9812218b7af	canary-shell-ops-004	wrong-file	0.310	$0.0010	2026-04-27T16:00:54.295822+00:00
7b676907-e55b-45a1-ae6c-c373cb05d09b	python-dependency-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:00:54.263680+00:00
295a362c-a327-4f8a-ba8d-dd9302103116	python-config-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:00:54.215820+00:00
3595da29-4f83-420e-a8f6-4cd7ca84309d	canary-python-cache-005	wrong-file	0.310	$0.0010	2026-04-27T16:00:54.168436+00:00
1f3a7798-7144-4dca-a44f-3a8322549f7d	canary-typescript-auth-006	wrong-file	0.310	$0.0010	2026-04-27T16:00:54.100564+00:00
8685f9fc-cbff-48f5-9197-b2da4afe0d9f	canary-python-security-001	wrong-file	0.310	$0.0010	2026-04-27T16:00:54.055854+00:00
671e00e9-f0b9-4c9f-abdd-ec77f17cbb0a	canary-typescript-session-003	wrong-file	0.310	$0.0010	2026-04-27T16:00:54.015229+00:00
2da271d2-6c7b-4420-9676-45fb147f47f9	python-bugfix-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T16:00:53.981147+00:00

pass_rate vs cost_usd (Pareto frontier marked with *)
* [#######-------------] 37.5% @ $0.0010  (coder)