Eval Report: ci-nightly

Profile: gdm-swebench-lite-v1 | Tasks: 8 | Pass rate: 37.5% | Cost: $0.0080

Task ID	Band	Score	Passed	Cost
python-security-fix-easy-001	easy	0.740	✓	$0.0010
canary-shell-ops-004	hard	0.310	✗	$0.0010
canary-python-cache-005	hard	0.310	✗	$0.0010
canary-python-regression-002	hard	0.310	✗	$0.0010
typescript-security-fix-easy-001	easy	0.740	✓	$0.0010
canary-typescript-auth-006	hard	0.310	✗	$0.0010
python-bugfix-easy-001	easy	0.740	✓	$0.0010
canary-typescript-session-003	hard	0.310	✗	$0.0010

Leaderboard Snapshot

Latest run: 8ffa1a1e-7347-40a1-b388-4dc03f3cde6a | Latest model: coder | Latest score: 0.310 | Recorded at: 2026-05-22T13:57:01.025619+00:00

Run ID	Model	Git SHA	Score	Created
8ffa1a1e-7347-40a1-b388-4dc03f3cde6a	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.310	2026-05-22T13:57:01.025619+00:00
7793d6f2-6678-41f3-ab6f-3c170b0db867	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-05-22T13:57:00.940455+00:00
f0054904-b9f2-4a1f-86c7-49b4038f29a3	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.310	2026-05-22T13:57:00.847922+00:00
92185a49-98ce-4446-82d5-153ace505470	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-05-22T13:57:00.770789+00:00
e24e42ac-b988-4aad-978a-7c384dc216de	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.310	2026-05-22T13:57:00.684887+00:00

Taxonomy	Failures	Bar
wrong-file	5	#####
wrong-logic	3	###

Run ID	Task ID	Taxonomy	Score	Cost	Created
8ffa1a1e-7347-40a1-b388-4dc03f3cde6a	canary-typescript-session-003	wrong-file	0.310	$0.0010	2026-05-22T13:57:01.025619+00:00
7793d6f2-6678-41f3-ab6f-3c170b0db867	python-bugfix-easy-001	wrong-logic	0.740	$0.0010	2026-05-22T13:57:00.940455+00:00
f0054904-b9f2-4a1f-86c7-49b4038f29a3	canary-typescript-auth-006	wrong-file	0.310	$0.0010	2026-05-22T13:57:00.847922+00:00
92185a49-98ce-4446-82d5-153ace505470	typescript-security-fix-easy-001	wrong-logic	0.740	$0.0010	2026-05-22T13:57:00.770789+00:00
e24e42ac-b988-4aad-978a-7c384dc216de	canary-python-regression-002	wrong-file	0.310	$0.0010	2026-05-22T13:57:00.684887+00:00
de46a676-c089-4d6b-8ac2-6200a6fef221	canary-python-cache-005	wrong-file	0.310	$0.0010	2026-05-22T13:57:00.596733+00:00
cab23baf-7f28-4bea-820b-677d9e53f3d4	canary-shell-ops-004	wrong-file	0.310	$0.0010	2026-05-22T13:57:00.501579+00:00
fe6ff0bd-b171-4492-ab2f-888acd21d240	python-security-fix-easy-001	wrong-logic	0.740	$0.0010	2026-05-22T13:57:00.429532+00:00

pass_rate vs cost_usd (Pareto frontier marked with *)
* [#######-------------] 37.5% @ $0.0010  (coder)