Every model that touches NomosGuard's evidence edge reads raw logs and writes structured claims. This benchmark measures how well each model performs that one job: 16 models, 4 hand-derived scenarios, 76 measured runs — same prompt, script-scored, nothing by hand.
| Model | Recall | Precision | Schema | Halluc. | |
|---|---|---|---|---|---|
| 1 | gpt-oss-120bAntigravity | 100% | 100% | 100% | 0 |
| 2 | gemini-3.8-flashAntigravity | 100% | 96% | 100% | 1 |
| 3 | claude-opus-4.6Antigravity | 100% | 88% | 100% | 2 |
| 4 | opencode/step-5-preview-free | 100% | 88% | 100% | 2 |
| 5 | gemini-3.1-proAntigravity | 95% | 95% | 100% | 1 |
| 6 | opencode/mimo-v2.6-flash-free | 95% | 95% | 100% | 1 |
| 7 | opencode/nemotron-3-ultra-freeOpenCode Zen | 95% | 95% | 100% | 1 |
| 8 | opencode/space-bunny-freeOpenCode Zen | 95% | 87% | 75% | 1 |
| 9 | claude-sonnet-4.6Antigravity | 95% | 83% | 100% | 2 |
| 10 | opencode/longcat-2.5-preview-freeOpenCode Zen | 95% | 83% | 100% | 2 |
| 11 | opencode/ling-3.1-flash-free | 93% | 82% | 100% | 2 |
| 12 | opencode/big-pickleOpenCode Zen | 88% | 88% | 100% | 1 |
| 13 | mistral-medium-3.5Vibe CLI | 87% | 78% | 100% | 2 |
| 14 | opencode/muse-spark-1.3-contributor-freeOpenCode Zen | 55% | 62% | 100% | 1 |
| 15 | gemini-3.7-flashAntigravity | 25% | 25% | 25% | 0 |
| 16 | gemini-3.6-flashAntigravity | 0% | 0% | 0% | 0 |
Brute force & pivot · benign admin window · lateral movement · privilege escalation
| Model | Brute Force Then Pivot | Benign Admin Window | Lateral Movement Pair | Privilege Escalation |
|---|---|---|---|---|
| gpt-oss-120b | 100% | 100% | 100% | 100% |
| gemini-3.8-flash | 100% | 100% | 100% | 100% |
| claude-opus-4.6 | 100% | 100% | 100% | 100% |
| opencode/step-5-preview-free | 100% | 100% | 100% | 100% |
| gemini-3.1-pro | 80% | 100% | 100% | 100% |
| opencode/mimo-v2.6-flash-free | 80% | 100% | 100% | 100% |
| opencode/nemotron-3-ultra-free | 80% | 100% | 100% | 100% |
| opencode/space-bunny-free | 80% | 100% | 100% | 100% |
| claude-sonnet-4.6 | 80% | 100% | 100% | 100% |
| opencode/longcat-2.5-preview-free | 80% | 100% | 100% | 100% |
| opencode/ling-3.1-flash-free | 80% | 100% | 100% | — |
| opencode/big-pickle | 100% | 50% | 100% | 100% |
| mistral-medium-3.5 | 80% | 100% | 100% | 67% |
| opencode/muse-spark-1.3-contributor-free | 20% | 0% | 100% | 100% |
| gemini-3.7-flash | 0% | 0% | 100% | 0% |
| gemini-3.6-flash | 0% | 0% | 0% | 0% |