Exact predictions
2/15
Improved from 0/15 before the source-first evidence pass.
Prediction Engine v2 precision benchmark
Source-first evidence improved exact hits, but target recall is now the bottleneck.
Vaner prepared locally on the RTX 5090 with qwen3.5:35b.
Claude Sonnet generated the user-facing answers and Claude Opus judged them.
The run confirms the evidence hygiene fix worked: no supporting-only packages.
It also shows the next problem clearly: when Vaner chooses the wrong primary source file,
the warm answer can get worse.
Related coverage
5/15
Dropped from 6/15; the stricter gate discarded broad docs-only context.
Primary evidence
9/15
Source/config/test evidence was present in 60% of turns.
Quality A/B
1 / 1 / 1
One Vaner win, one tie, one cold-start win.
Before vs after
The old run used Claude Sonnet as judge; this run used Claude Opus, so compare directionally, not as a perfectly controlled A/B.
Current failure modes
| Miss reason | Count | Meaning |
|---|---|---|
| No primary evidence | 6/15 | No source/config/test target was ready. |
| Wrong primary target | 4/15 | Vaner found source files, but not the mechanism the user asked for. |
| Tangential primary evidence | 3/15 | Source evidence was nearby but incomplete. |
| Supporting-only misses | 0/15 | The hygiene/readiness fix removed the old docs/generated/data-only failure. |
Local 35B exploration, serial concurrency, 90s cycle cap.
Warm lookup stayed interactive; preparation cost is background work.
Vaner won 1/3 sampled answer comparisons under Opus judging.
What improved
1
Primary evidence is now visibleThe report tracks primary vs supporting evidence, so failure is no longer hidden behind one relevance score.
2
Docs/generated artifacts stopped dominatingGenerated bindings, diagrams, runtime data, and broad docs no longer make mechanism predictions look evidence-ready by themselves.
3
Exact hits appearedTwo turns landed direct mechanism evidence under a stricter Opus judge.
What to fix next
A
Component-name target recallQueries naming components such as stores, scorers, assemblers, daemon runners, and config loaders need better symbol-to-file resolution.
B
Wrong-primary guardrailIf primary evidence points to a neighboring mechanism, Vaner should abstain or keep the package evidence-ready only, not let it drive answer quality down.
C
Answer generation robustnessOne quality sample tied because both Claude answers failed to produce useful content. The harness should capture answer errors separately from Vaner quality.