OSPREY model-capability benchmark

Open-weight & self-hosted models across the full tests/e2e/ suite · 11 models × 3 seeds

passed failed skipped error timeout

Outcome breakdown by model × seed

Each cell shows the full mix for one run — pass% (passed / passed+failed+timeout+error; a timeout counts as a failure, only skips are excluded) plus a stacked bar and counts (✓ pass · ✗ fail · ⧗ timeout · ∅ skip · ! error).

model	seed 1	seed 2	seed 3	mean pass
gpt-oss-20b [cborg]	67% 24✓ 12✗ 36t · 30m	67% 24✓ 12✗ 36t · 26m	56% 20✓ 16✗ 36t · 40m	63%
gemma-4 [cborg]	89% 32✓ 4✗ 36t · 182m	94% 34✓ 2✗ 36t · 134m	92% 33✓ 2✗ 1⧗ 36t · 196m	92%
cborg-coder [cborg]	94% 34✓ 2✗ 36t · 266m	97% 35✓ 1✗ 36t · 248m	92% 33✓ 3✗ 36t · 266m	94%
gpt-oss-120b [cborg]	92% 33✓ 3✗ 36t · 39m	81% 29✓ 7✗ 36t · 41m	81% 29✓ 7✗ 36t · 48m	84%
qwen-3-coder [cborg]	94% 34✓ 2✗ 36t · 45m	83% 30✓ 6✗ 36t · 76m	94% 34✓ 2✗ 36t · 55m	91%
qwen-3 [cborg]	89% 32✓ 4✗ 36t · 44m	78% 28✓ 8✗ 36t · 73m	83% 30✓ 6✗ 36t · 51m	83%
deepseek-v4-flash [ds4 · macstudio (self-hosted)]	94% 34✓ 2✗ 36t · 730m	N/A	N/A	94%
deepseek-v4-pro [ds4 · macstudio (self-hosted)]	97% 35✓ 1✗ 36t · 544m	N/A	N/A	97%

claude-haiku-4-5-20251001 [als-apg]	100% 36✓ 36t · 30m	N/A	N/A	100%
claude-sonnet-4-6 [als-apg]	100% 36✓ 36t · 31m	N/A	N/A	100%
claude-opus-4-6 [als-apg]	100% 36✓ 36t · 32m	N/A	N/A	100%

Per-test outcomes — one square per seed

For every test, the three squares are seed 1·2·3 colored by outcome (green pass · red fail · orange timeout · yellow skip · purple error · grey pending). Lets you spot a test a model passes once but times out on another seed.

test	gpt-oss-20b	gemma-4	cborg-coder	gpt-oss-120b	qwen-3-coder	qwen-3	deepseek-v4-flash	deepseek-v4-pro	claude-haiku-4-5-20251001	claude-sonnet-4-6	claude-opus-4-6
claude_code/test_agent_delegation.py
TestAgentDelegation::test_channel_finder_delegation
TestAgentDelegation::test_data_visualizer_delegation
TestAgentDelegation::test_logbook_search_delegation
claude_code/test_audit_observability.py
TestAuditObservability::test_channel_read_archiver_plot_with_audit
claude_code/test_data_visualizer.py
TestDataVisualizerAgent::test_archiver_read_to_interactive_plot
TestDataVisualizerAgent::test_interactive_plot_with_data_source
claude_code/test_feedback_capture_e2e.py
test_feedback_hook_captures_search_results
test_feedback_hook_is_silent
claude_code/test_safety_approval_e2e.py
test_approval_denial_propagates_to_claude
test_channel_write_triggers_approval_callback
test_default_policy_always_asks_for_reads
test_limits_deny_blocks_before_approval
claude_code/test_safety_error_guidance_e2e.py
test_error_response_no_retry_protocol
test_execution_error_triggers_guidance
claude_code/test_safety_kill_switch.py
test_channel_write_denied_when_writes_disabled
test_python_write_denied_when_writes_disabled
claude_code/test_safety_python.py
test_safe_python_execution_succeeds
claude_code/test_safety_reads.py
test_channel_read_succeeds
claude_code/test_safety_writes.py
test_write_over_limits_denied
test_write_to_readonly_denied
test_write_to_unlisted_channel_succeeds
test_write_within_limits_succeeds
claude_code/test_sdk_workflows.py
TestClaudeCodeSDKIntegration::test_3d_scatter_plot_via_data_visualizer
TestClaudeCodeSDKIntegration::test_archiver_and_plot_via_sdk
TestClaudeCodeSDKIntegration::test_full_bpm_correlation_pipeline_via_sdk
TestClaudeCodeSDKIntegration::test_tool_call_observability_smoke
test_corrector_limit_honest_refusal_scenario.py
test_corrector_limit_honest_refusal
test_facility_knowledge_e2e.py
test_facility_knowledge_delegation
test_facility_knowledge_pss_reset
test_facility_knowledge_vacuum_recovery
test_preset_agentic.py
test_control_assistant_channel_finder_flow
test_control_assistant_write_triggers_approval_hook
test_hello_world_canonical_flow
test_hello_world_write_triggers_approval_hook
test_rf_cavity_correlation_scenario.py
test_rf_cavity_c1_correlation_flow
test_vacuum_burst_scenario.py
test_sector7_vacuum_burst_flow

Methodology. The 36 model-driving e2e tests — the full tests/e2e/ suite minus the 13 files that don't call an LLM and/or don't route through the model under test (hover for the reason: test_llm_providers.py, test_ariel_search.py, test_dockerfile_e2e.py, test_claude_code_build_integration.py, test_llm_channel_namer.py, test_channel_finder_mcp_benchmarks.py, test_mcp_readiness.py, test_sdk_helpers.py, test_dispatch_deploy.py, test_proxy_full_chain_e2e.py, test_proxy_live_roundtrip_e2e.py, test_proxy_open_model_harness_e2e.py, test_dispatch_tutorial.py) — are forced onto each CBORG model suite-wide via OSPREY_E2E_FORCE_PROVIDER=cborg + OSPREY_E2E_FORCE_MODEL (all tiers collapse to one model). Open models route through OSPREY's Anthropic↔OpenAI translation proxy. The per-test timeout is 1800s; a timeout counts as a failure (the model did not complete the task within the cap) and is shown as a distinct category so it can be told apart from a logic failure. Models were chosen from a 14-model lightweight probe to span weak/fast→big/slow across distinct open families (gpt-oss, gemma, cborg). qwen-3 / qwen-3-coder are open-weight but CBORG-proxied (google/ route, not lbl/ self-hosted), included as an additional open family. Dropped at probe: cborg-instant, cborg-instant-short (could not relay a tool result; short context truncates the harness prompt). Closed-weight commercial models (GPT/Gemini/…) are out of scope as study subjects, but single-seed Anthropic Claude reference models — claude-haiku-4-5 (the suite's literal default tier), claude-sonnet-4-6, and claude-opus-4-6 — bracket the open models weak→strong as a control/ceiling, showing how they compare against the models the tests were written for. Open subjects run on the Mac Studio via CBORG; the Anthropic reference rows route natively via als-apg (see the [provider] line under each model).