Eval Report: ci-nightly

Profile: gdm-swebench-lite-v1 | Tasks: 8 | Pass rate: 37.5% | Cost: $0.0080

Task ID	Band	Score	Passed	Cost
python-security-fix-easy-001	easy	0.740	✓	$0.0010
canary-typescript-session-003	hard	0.310	✗	$0.0010
canary-python-security-001	hard	0.310	✗	$0.0010
canary-typescript-auth-006	hard	0.310	✗	$0.0010
canary-python-cache-005	hard	0.310	✗	$0.0010
typescript-security-fix-easy-001	easy	0.740	✓	$0.0010
python-bugfix-easy-001	easy	0.740	✓	$0.0010
canary-shell-ops-004	hard	0.310	✗	$0.0010

Leaderboard Snapshot

Latest run: 00faf957-981e-4148-b56e-eef2d13dd24a | Latest model: coder | Latest score: 0.310 | Recorded at: 2026-05-09T19:41:38.186147+00:00

Run ID	Model	Git SHA	Score	Created
00faf957-981e-4148-b56e-eef2d13dd24a	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.310	2026-05-09T19:41:38.186147+00:00
d2a59902-9d37-4aa7-9a0f-7e4cceff114a	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-05-09T19:41:38.119257+00:00
d1fa1f23-33bc-416c-8e0c-9330c4f33919	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-05-09T19:41:38.039058+00:00
f6ef3b8e-d725-479c-a9e8-24a9aa034728	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.310	2026-05-09T19:41:37.968562+00:00
4ca9d648-923a-444a-abc7-fe0d10fe76e2	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.310	2026-05-09T19:41:37.892663+00:00

Taxonomy	Failures	Bar
wrong-file	5	#####
wrong-logic	3	###

Run ID	Task ID	Taxonomy	Score	Cost	Created
00faf957-981e-4148-b56e-eef2d13dd24a	canary-shell-ops-004	wrong-file	0.310	$0.0010	2026-05-09T19:41:38.186147+00:00
d2a59902-9d37-4aa7-9a0f-7e4cceff114a	python-bugfix-easy-001	wrong-logic	0.740	$0.0010	2026-05-09T19:41:38.119257+00:00
d1fa1f23-33bc-416c-8e0c-9330c4f33919	typescript-security-fix-easy-001	wrong-logic	0.740	$0.0010	2026-05-09T19:41:38.039058+00:00
f6ef3b8e-d725-479c-a9e8-24a9aa034728	canary-python-cache-005	wrong-file	0.310	$0.0010	2026-05-09T19:41:37.968562+00:00
4ca9d648-923a-444a-abc7-fe0d10fe76e2	canary-typescript-auth-006	wrong-file	0.310	$0.0010	2026-05-09T19:41:37.892663+00:00
b18329ac-10fa-4141-80cc-d9839b57076a	canary-python-security-001	wrong-file	0.310	$0.0010	2026-05-09T19:41:37.816902+00:00
29d2debc-0bf7-447f-90b2-a8b89a653fa9	canary-typescript-session-003	wrong-file	0.310	$0.0010	2026-05-09T19:41:37.747527+00:00
0f73da4a-132f-42d8-8579-f1891aa5f3c0	python-security-fix-easy-001	wrong-logic	0.740	$0.0010	2026-05-09T19:41:37.656738+00:00

pass_rate vs cost_usd (Pareto frontier marked with *)
* [#######-------------] 37.5% @ $0.0010  (coder)