Eval Report: ci-nightly

Profile: gdm-swebench-lite-v1 | Tasks: 8 | Pass rate: 37.5% | Cost: $0.0080

Task ID	Band	Score	Passed	Cost
python-security-fix-easy-001	easy	0.740	✓	$0.0010
canary-shell-ops-004	hard	0.310	✗	$0.0010
canary-python-cache-005	hard	0.310	✗	$0.0010
canary-python-regression-002	hard	0.310	✗	$0.0010
typescript-security-fix-easy-001	easy	0.740	✓	$0.0010
canary-typescript-auth-006	hard	0.310	✗	$0.0010
python-bugfix-easy-001	easy	0.740	✓	$0.0010
canary-typescript-session-003	hard	0.310	✗	$0.0010

Leaderboard Snapshot

Latest run: 4ea91dac-df81-4ba4-91a3-002671bf4f0c | Latest model: coder | Latest score: 0.310 | Recorded at: 2026-05-23T20:38:58.267306+00:00

Run ID	Model	Git SHA	Score	Created
4ea91dac-df81-4ba4-91a3-002671bf4f0c	coder	07ca25bc2d511f5aee15446c60081c184e9c9122	0.310	2026-05-23T20:38:58.267306+00:00
35d0e18d-9c00-4ead-b299-2a32cac5cf9b	coder	07ca25bc2d511f5aee15446c60081c184e9c9122	0.740	2026-05-23T20:38:58.179743+00:00
ac036fcb-902b-46e2-bbd9-7b654d27dd06	coder	07ca25bc2d511f5aee15446c60081c184e9c9122	0.310	2026-05-23T20:38:58.074630+00:00
7c73c1b3-b859-4758-bb79-7534a7836e88	coder	07ca25bc2d511f5aee15446c60081c184e9c9122	0.740	2026-05-23T20:38:57.965181+00:00
66d78fc3-466d-4064-afc8-38c91b1110c7	coder	07ca25bc2d511f5aee15446c60081c184e9c9122	0.310	2026-05-23T20:38:57.854104+00:00

Taxonomy	Failures	Bar
wrong-file	5	#####
wrong-logic	3	###

Run ID	Task ID	Taxonomy	Score	Cost	Created
4ea91dac-df81-4ba4-91a3-002671bf4f0c	canary-typescript-session-003	wrong-file	0.310	$0.0010	2026-05-23T20:38:58.267306+00:00
35d0e18d-9c00-4ead-b299-2a32cac5cf9b	python-bugfix-easy-001	wrong-logic	0.740	$0.0010	2026-05-23T20:38:58.179743+00:00
ac036fcb-902b-46e2-bbd9-7b654d27dd06	canary-typescript-auth-006	wrong-file	0.310	$0.0010	2026-05-23T20:38:58.074630+00:00
7c73c1b3-b859-4758-bb79-7534a7836e88	typescript-security-fix-easy-001	wrong-logic	0.740	$0.0010	2026-05-23T20:38:57.965181+00:00
66d78fc3-466d-4064-afc8-38c91b1110c7	canary-python-regression-002	wrong-file	0.310	$0.0010	2026-05-23T20:38:57.854104+00:00
1f2bcd45-6049-4d29-b839-13b72e078b31	canary-python-cache-005	wrong-file	0.310	$0.0010	2026-05-23T20:38:57.758935+00:00
4bce2766-a875-45f7-9fa6-7d61bbbf9f41	canary-shell-ops-004	wrong-file	0.310	$0.0010	2026-05-23T20:38:57.647593+00:00
41f9e4dd-12c5-4e5d-bff3-edcd32d9a988	python-security-fix-easy-001	wrong-logic	0.740	$0.0010	2026-05-23T20:38:57.551933+00:00

pass_rate vs cost_usd (Pareto frontier marked with *)
* [#######-------------] 37.5% @ $0.0010  (coder)