LLM benchmark — security telemetry parsing
NomosGuard LLM-edge evaluation · 9 models · 4 scenarios · 40 runs · all values measured by script
| Model | Source | JSON | Schema |
Recall | Precision | Evidence | Halluc. |
| claude-opus-4.6 |
Antigravity (your quota) |
100% |
100% |
100% |
88% |
100% |
2 |
| gemini-3.8-flash |
Antigravity (your quota) |
100% |
100% |
100% |
96% |
100% |
1 |
| gpt-oss-120b |
Antigravity (your quota) |
100% |
100% |
100% |
100% |
100% |
0 |
| opencode/step-5-preview-free |
|
100% |
100% |
100% |
88% |
100% |
2 |
| claude-sonnet-4.6 |
Antigravity (your quota) |
100% |
100% |
95% |
83% |
100% |
2 |
| gemini-3.1-pro |
Antigravity (your quota) |
100% |
100% |
95% |
95% |
100% |
1 |
| opencode/mimo-v2.6-flash-free |
|
100% |
100% |
95% |
95% |
100% |
1 |
| opencode/longcat-2.5-preview-free |
OpenCode Zen (free) |
100% |
100% |
95% |
83% |
100% |
2 |
| opencode/nemotron-3-ultra-free |
OpenCode Zen (free) |
100% |
100% |
95% |
95% |
100% |
1 |
| opencode/space-bunny-free |
OpenCode Zen (free) |
100% |
75% |
95% |
87% |
100% |
1 |
| opencode/ling-3.1-flash-free |
|
100% |
100% |
93% |
82% |
100% |
2 |
| opencode/big-pickle |
OpenCode Zen (free) |
100% |
100% |
88% |
88% |
100% |
1 |
| mistral-medium-3.5 |
vibe |
100% |
100% |
87% |
78% |
100% |
2 |
| opencode/muse-spark-1.3-contributor-free |
OpenCode Zen (free) |
100% |
100% |
55% |
62% |
75% |
1 |
| gemini-3.7-flash |
Antigravity (your quota) |
25% |
25% |
25% |
25% |
25% |
0 |
| gemini-3.6-flash |
Antigravity (your quota) |
0% |
0% |
0% |
0% |
0% |
0 |
Recall by scenario
| Model | brute force then pivot | benign admin window | lateral movement pair | privilege escalation |
|---|
| claude-opus-4.6 | 100% | 100% | 100% | 100% |
| gemini-3.8-flash | 100% | 100% | 100% | 100% |
| gpt-oss-120b | 100% | 100% | 100% | 100% |
| opencode/step-5-preview-free | 100% | 100% | 100% | 100% |
| claude-sonnet-4.6 | 80% | 100% | 100% | 100% |
| gemini-3.1-pro | 80% | 100% | 100% | 100% |
| opencode/mimo-v2.6-flash-free | 80% | 100% | 100% | 100% |
| opencode/longcat-2.5-preview-free | 80% | 100% | 100% | 100% |
| opencode/nemotron-3-ultra-free | 80% | 100% | 100% | 100% |
| opencode/space-bunny-free | 80% | 100% | 100% | 100% |
| opencode/ling-3.1-flash-free | 80% | 100% | 100% | — |
| opencode/big-pickle | 100% | 50% | 100% | 100% |
| mistral-medium-3.5 | 80% | 100% | 100% | 67% |
| opencode/muse-spark-1.3-contributor-free | 20% | 0% | 100% | 100% |
| gemini-3.7-flash | 0% | 0% | 100% | 0% |
| gemini-3.6-flash | 0% | 0% | 0% | 0% |
Methodology: each model receives the same prompt on the same 4 hand-derived
scenarios; outputs are scored by script — JSON validity, schema conformance,
claim recall/precision against ground truth, verbatim evidence citation, and
hallucination count. Ground truth was derived from the raw logs before any model
output was seen. All numbers come from actual runs; none are hand-computed.
Runs executed 2026-10-09 via OpenCode Zen (free tier) and Antigravity CLI
(user account quota).