Why better?
Evidence
Vaner put direct docs/files first and gave the primary model a plan for using them.
Public session-replay benchmark - April 28, 2026
Vaner is better when the question needs the right evidence, not just a fluent answer.
This run compares the same primary model answering the same public-corpus questions with no context, naive RAG context, raw Vaner context, and Vaner's answerable evidence package. The short version: Vaner wins because it turns scattered files into usable answer evidence.
Prediction accuracy?
Partial
58.3% had a prepared package; exact next-prompt prediction was weak. The win came from packaging and evidence floors.
Quality lift?
+1.65
Best Vaner path beat best(naked, RAG), not only the no-context baseline.
Cost story?
Not yet
Quality improved; final-answer latency and total cloud-cost reduction are not proven by this run.
Average answer quality
Score is a blind judge grade out of 10. The main product gain is the jump from raw Vaner to answerable Vaner: same general evidence pool, better packaging.
Plain-English answer
1
Vaner found task-specific evidence.For code questions, it surfaced the exact test or implementation file instead of nearby docs. For docs questions, it surfaced the canonical English tutorial page instead of broad translated or incidental mentions.
2
Vaner shaped evidence for the answer model.The answerable layer separated direct evidence, supporting context, lower-confidence context, and answer plan. That reduced vague “the context does not say” answers.
3
Vaner usually avoided false confidence.Exact intent prediction was not strong, but false-ready rate was only 1.6% and correct abstain rate was 97.9%.
Answerability labels: 276 full, 96 weak, 12 none. Most briefings were usable before final answer generation.
The primary model used evidence correctly or partially in 349 judged live paths.
Prompt prediction had 20 partial matches and no exact public intent hits. This is the biggest product caveat.
Concrete examples
How quality improved in practice.
The numbers matter, but the user-visible difference is simpler: Vaner gave the primary model the missing source material and a better interpretation of which source mattered.
Question: “Where are the existing CLI tests? I need to add one for this new command.”
Naked model
No project context was provided, so it could only ask the user to share a file tree. Score: 2/10.
Naive RAG
Still said there was no direct information about existing CLI tests. Score: 4/10.
Vaner
Surfaced tests/test_cli.py plus related test fixtures. The answer identified where to add the test and what neighboring test structure to follow. Score: 9/10.
Question: “Give me the simple OAuth2 example with password flow — no JWT yet.”
Naked model
Could not ground the answer in the docs and fell back to saying no context was available. Score: 1/10.
Naive RAG
Gave a partial generic summary of password flow, but missed the stronger source ordering. Score: 6/10.
Vaner
Put docs/en/docs/tutorial/security/simple-oauth2.md, first-steps.md, and related security references in the package. Score: 9/10.
Question: “What’s the path-sum problem? Show me both variants covered in the repo.”
Naked model
No repository context, so it could not identify the repo variants. Score: 2/10.
Naive RAG
Retrieved irrelevant math/problem snippets and said the context did not contain path-sum details. Score: 2/10.
Vaner
Found binary_tree_path_sum.py and binary_tree_node_sum.py, letting the answer compare the actual variants. Score: 9/10.
Question: “Quicksort — compare its partition scheme with merge sort’s split.”
Vaner often had useful sorting context, but extra context sometimes made the final answer worse than the stronger baseline. This is why the report separates retrieval, packaging, and final answer use. The next layer should not blindly add more context; it should decide when context is helpful, redundant, or distracting.
Prediction and readiness
Vaner did not win because it perfectly guessed the next prompt.
The benchmark is useful precisely because it separates “prediction readiness” from “answer quality.” In this run, prompt prediction was weak; evidence packaging carried the product result.
Prediction metrics
| Metric | Value | What it means |
|---|---|---|
| Prediction available rate | 58.3% | Vaner often had a pre-turn package, but availability is not the same as exact prediction. |
| Intent prediction buckets | 20 partial, 364 unrelated | No exact public intent hits. The predictor saw broad session direction more often than exact next prompt. |
| Adoptable prediction rate | 2.1% | Very few predictions were ready enough to adopt directly as an answer. |
| False-ready rate | 1.6% | Low, but any false-ready case is high-risk for trust. |
| Correct abstain rate | 97.9% | Vaner usually avoided pretending it had a ready answer when it did not. |
So what worked?
A
Answerable evidence packaging.Raw Vaner scored 5.85; answerable Vaner scored 6.78. That delta is packaging, ranking, and answer-plan value.
B
Evidence-use reliability.The primary model used the evidence correctly in 226 cases and partially in 123 cases.
C
Domain fit.Docs and code tasks benefited most. Literature was still positive, but more mixed.
Best Vaner uplift over best baseline. Strongest domain in the run.
Large gains when exact implementation/test files matter.
Positive but mixed; synthesis tasks do not always need more retrieved context.
Developer appendix
The detailed tables are still here.
The user-facing story should be readable without these tables, but the benchmark remains auditable: protocol, corpus, scores, cost, readiness, and reproducibility are preserved below.
Protocol
| Vaner prep model | qwen3.5:35b via local Ollama on RTX 5090 |
|---|---|
| Primary user model | claude-sonnet-4-6 |
| Judge model | claude-opus-4-7 |
| Answer paths | naked primary model, naive RAG, raw Vaner, answerable Vaner, best Vaner product path |
| Live configs | idle multipliers 0, 0.25, 1.0 crossed with evidence assembly shadow and safe |
| Pricing snapshot | anthropic-2026-04-28-manual, as-run estimates preserved |
Public corpora
| Corpus | Use |
|---|---|
| Flask | Codebase and documentation tasks |
| FastAPI | Documentation and API usage tasks |
| TheAlgorithms/Python | Learning-code and implementation-study tasks |
| Alice in Wonderland | Literary synthesis tasks |
Results by domain
| Domain | N | Naked | RAG | Answerable | Best Vaner | Uplift | Win/Tie/Loss | Context tokens |
|---|---|---|---|---|---|---|---|---|
| Code | 96 | 3.30 | 5.20 | 6.30 | 7.42 | +2.03 | 83 / 11 / 2 | 979 |
| Docs | 96 | 2.99 | 4.55 | 7.97 | 8.10 | +2.70 | 83 / 8 / 5 | 1,000 |
| Learning code | 96 | 4.46 | 4.82 | 7.25 | 7.70 | +1.45 | 52 / 19 / 25 | 983 |
| Literature | 96 | 3.71 | 5.23 | 5.61 | 6.32 | +0.43 | 45 / 26 / 25 | 794 |
Live conditions
| Idle | Assembly | N | Naked | RAG | Raw | Answerable | Best | Uplift | Win/Tie/Loss | Context |
|---|---|---|---|---|---|---|---|---|---|---|
| 0.0 | safe | 64 | 3.78 | 4.97 | 5.62 | 7.02 | 7.56 | +1.81 | 41 / 14 / 9 | 978 |
| 0.0 | shadow | 64 | 3.62 | 4.98 | 5.78 | 7.19 | 7.72 | +1.89 | 45 / 14 / 5 | 971 |
| 0.25 | safe | 64 | 3.31 | 4.89 | 5.66 | 6.56 | 7.20 | +1.62 | 43 / 10 / 11 | 948 |
| 0.25 | shadow | 64 | 3.44 | 4.92 | 6.12 | 6.72 | 7.30 | +1.67 | 48 / 8 / 8 | 863 |
| 1.0 | safe | 64 | 3.94 | 5.03 | 5.94 | 6.50 | 7.19 | +1.25 | 42 / 10 / 12 | 932 |
| 1.0 | shadow | 64 | 3.59 | 4.91 | 6.00 | 6.72 | 7.34 | +1.66 | 44 / 8 / 12 | 942 |
Cost ledger view
| Total estimated cloud cost | $5.7672 |
|---|---|
| Primary answer model cost | $2.2329 |
| Judge model cost | $3.5343 |
| Mean incremental primary input cost | $0.0028 per turn |
| Cost per best-quality point | $0.0031 |
Reproducibility
| Raw artifact | .vaner/bench-runs/20260428T094217Z-publishable-public-full/public-session-replay-full.json |
|---|---|
| Summary artifact | docs/benchmarks/2026-04-28-public-session-replay-benchmark.summary.json |
| Vaner repo SHA | 07f519dfd4fdd6b45ea18b15b6b61fb91764240e |
| Evaluation harness SHA | 0df58b519e0ebda7b217bd957e5a024acf878329 |
| Random seed | 42 |
Caveats
- This is a public-corpus session-replay benchmark with a repeatable protocol, not a third-party leaderboard.
- Prepared-only predictive readiness was weak. The headline result is evidence packaging quality, not exact next-prompt prediction.
- File recall did not improve over the cold baseline in this run: cold mean recall was 75.7%, live best mean recall was 68.5%.
- Final answer generation was slower than naked and RAG in this harness. A speed win needs prepared draft adoption or tighter client-side timing signals.
- Cost is an as-run estimate from the configured pricing snapshot. Runtime cost ledgers should be used for product-facing cost claims.