Eval Report: ci-nightly

Profile: gdm-swebench-lite-v1 | Tasks: 8 | Pass rate: 37.5% | Cost: $0.0080

Task ID	Band	Score	Passed	Cost
python-security-fix-easy-001	easy	0.740	✓	$0.0010
canary-shell-ops-004	hard	0.310	✗	$0.0010
canary-python-cache-005	hard	0.310	✗	$0.0010
canary-python-regression-002	hard	0.310	✗	$0.0010
typescript-security-fix-easy-001	easy	0.740	✓	$0.0010
canary-typescript-auth-006	hard	0.310	✗	$0.0010
python-bugfix-easy-001	easy	0.740	✓	$0.0010
canary-typescript-session-003	hard	0.310	✗	$0.0010

Leaderboard Snapshot

Latest run: 919a15fa-3368-496f-876f-eb8a27a2ec1b | Latest model: coder | Latest score: 0.310 | Recorded at: 2026-05-23T20:07:49.195844+00:00

Run ID	Model	Git SHA	Score	Created
919a15fa-3368-496f-876f-eb8a27a2ec1b	coder	89f0f5456c5b8670ca70d1a941ab0d7272df1310	0.310	2026-05-23T20:07:49.195844+00:00
1ca83d35-77e5-474c-9f29-a2d0589e8cd1	coder	89f0f5456c5b8670ca70d1a941ab0d7272df1310	0.740	2026-05-23T20:07:49.101685+00:00
e2e6b00b-fd7b-4037-bf1d-47f37f0e0e93	coder	89f0f5456c5b8670ca70d1a941ab0d7272df1310	0.310	2026-05-23T20:07:49.039952+00:00
1e13ad55-b70e-496c-a2e6-ab5dc2ba0b54	coder	89f0f5456c5b8670ca70d1a941ab0d7272df1310	0.740	2026-05-23T20:07:48.960329+00:00
b5c1aa67-86cf-4953-aec8-4da5df8cd4fa	coder	89f0f5456c5b8670ca70d1a941ab0d7272df1310	0.310	2026-05-23T20:07:48.911326+00:00

Taxonomy	Failures	Bar
wrong-file	5	#####
wrong-logic	3	###

Run ID	Task ID	Taxonomy	Score	Cost	Created
919a15fa-3368-496f-876f-eb8a27a2ec1b	canary-typescript-session-003	wrong-file	0.310	$0.0010	2026-05-23T20:07:49.195844+00:00
1ca83d35-77e5-474c-9f29-a2d0589e8cd1	python-bugfix-easy-001	wrong-logic	0.740	$0.0010	2026-05-23T20:07:49.101685+00:00
e2e6b00b-fd7b-4037-bf1d-47f37f0e0e93	canary-typescript-auth-006	wrong-file	0.310	$0.0010	2026-05-23T20:07:49.039952+00:00
1e13ad55-b70e-496c-a2e6-ab5dc2ba0b54	typescript-security-fix-easy-001	wrong-logic	0.740	$0.0010	2026-05-23T20:07:48.960329+00:00
b5c1aa67-86cf-4953-aec8-4da5df8cd4fa	canary-python-regression-002	wrong-file	0.310	$0.0010	2026-05-23T20:07:48.911326+00:00
02239f36-145d-4e10-af2d-1223618f3201	canary-python-cache-005	wrong-file	0.310	$0.0010	2026-05-23T20:07:48.848404+00:00
e651f240-d7ee-4b5f-8d5d-0834ad603e5a	canary-shell-ops-004	wrong-file	0.310	$0.0010	2026-05-23T20:07:48.784903+00:00
dea0d64f-720a-4da3-b3ff-deb173330bcd	python-security-fix-easy-001	wrong-logic	0.740	$0.0010	2026-05-23T20:07:48.721645+00:00

pass_rate vs cost_usd (Pareto frontier marked with *)
* [#######-------------] 37.5% @ $0.0010  (coder)