Eval Report: ci-nightly

Profile: gdm-swebench-lite-v1 | Tasks: 8 | Pass rate: 37.5% | Cost: $0.0080

Task ID	Band	Score	Passed	Cost
python-security-fix-easy-001	easy	0.740	✓	$0.0010
canary-typescript-session-003	hard	0.310	✗	$0.0010
canary-python-security-001	hard	0.310	✗	$0.0010
canary-typescript-auth-006	hard	0.310	✗	$0.0010
canary-python-cache-005	hard	0.310	✗	$0.0010
typescript-security-fix-easy-001	easy	0.740	✓	$0.0010
python-bugfix-easy-001	easy	0.740	✓	$0.0010
canary-shell-ops-004	hard	0.310	✗	$0.0010

Leaderboard Snapshot

Latest run: 19aba696-48c1-480a-b700-b321f1067490 | Latest model: coder | Latest score: 0.310 | Recorded at: 2026-04-27T23:48:42.232882+00:00

Run ID	Model	Git SHA	Score	Created
19aba696-48c1-480a-b700-b321f1067490	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.310	2026-04-27T23:48:42.232882+00:00
cc75890e-56ec-4034-bd4e-a9e4ca0a6bb3	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T23:48:42.172628+00:00
2d1c0a54-63d6-45f4-9b58-8ee365eecb07	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.740	2026-04-27T23:48:42.109056+00:00
b7eef47a-9bae-4254-908d-23198bfa0631	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.310	2026-04-27T23:48:42.037644+00:00
64493495-5b87-4be0-8b30-45f03d156df3	coder	4669773b4fbe9d507f1396f38777a1b36998faf3	0.310	2026-04-27T23:48:41.960800+00:00

Taxonomy	Failures	Bar
wrong-file	5	#####
wrong-logic	3	###

Run ID	Task ID	Taxonomy	Score	Cost	Created
19aba696-48c1-480a-b700-b321f1067490	canary-shell-ops-004	wrong-file	0.310	$0.0010	2026-04-27T23:48:42.232882+00:00
cc75890e-56ec-4034-bd4e-a9e4ca0a6bb3	python-bugfix-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T23:48:42.172628+00:00
2d1c0a54-63d6-45f4-9b58-8ee365eecb07	typescript-security-fix-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T23:48:42.109056+00:00
b7eef47a-9bae-4254-908d-23198bfa0631	canary-python-cache-005	wrong-file	0.310	$0.0010	2026-04-27T23:48:42.037644+00:00
64493495-5b87-4be0-8b30-45f03d156df3	canary-typescript-auth-006	wrong-file	0.310	$0.0010	2026-04-27T23:48:41.960800+00:00
ca5e9904-b264-4833-9d36-c291f8f6f709	canary-python-security-001	wrong-file	0.310	$0.0010	2026-04-27T23:48:41.893821+00:00
49b8a97b-3cc8-4a17-b10a-c4197fcff23e	canary-typescript-session-003	wrong-file	0.310	$0.0010	2026-04-27T23:48:41.829625+00:00
ddfaf9da-1fa3-4c21-9a3b-67064f2740c2	python-security-fix-easy-001	wrong-logic	0.740	$0.0010	2026-04-27T23:48:41.759044+00:00

pass_rate vs cost_usd (Pareto frontier marked with *)
* [#######-------------] 37.5% @ $0.0010  (coder)