Better than naked?
Yes
Best Vaner path: 7.83/10 vs naked Sonnet: 2.90/10.
Fair user-architecture benchmark
Vaner improves answer quality without bloating context versus RAG.
Latest run uses the intended product shape: Vaner prepares with a local model, the user's primary AI answers with Claude Sonnet, and Claude Opus judges. The headline is no longer raw briefing alone; it reports raw Vaner, answerable Vaner, and the best valid Vaner product path.
Better than RAG?
Often
29 wins, 13 ties, 6 losses against best(naked, RAG).
Does it bloat context?
No
Vaner sent 840 tokens on average, about 4 tokens less than RAG.
Golden trifecta?
Quality + cost
Quality and context efficiency are strong. Answer latency still trails naked and is slightly slower than RAG.
Average answer quality
Scale is Opus judge score out of 10. Best product path takes the best valid Vaner arm per turn: raw, answerable, prepared/adoption when present.
Context and cost efficiency
| Metric | Value | Interpretation |
|---|---|---|
| Vaner context sent | 840 tokens | Mean answerable context injected into Sonnet. |
| RAG context sent | 844 tokens | Naive top-K RAG used essentially the same budget. |
| Overhead vs RAG | -4 tokens | Vaner did not win by filling the context window. |
| Incremental Sonnet input cost | $0.0025 / turn | Estimated from Sonnet 4.6 input pricing. |
| Cost per Vaner quality point | $0.0031 | Estimated primary cost per point of best-path uplift. |
| Total cloud cost of run | $0.4166 | Primary + judge estimate for the benchmark run. |
Cost is estimated from configured pricing: Sonnet 4.6 at $0.003/$0.015 per 1k input/output and Opus 4.7 at $0.005/$0.025 per 1k input/output.
Best Vaner product-path uplift over best(naked, RAG).
Quality points per 1k injected tokens on positive-uplift turns.
Answerable path mean answer latency, versus 4.12s for RAG and 2.76s naked.
Benchmark architecture
V
Vaner prep modelLocal Ollama
qwen3.5:35b on the RTX 5090. This is the cheap/background worker, not the user's primary model.P
Primary user model
claude-sonnet-4-6. Naked, RAG, raw Vaner, and answerable Vaner all ask the same primary model.J
Judge
claude-opus-4-7. Separate from the primary model for cleaner evaluation.D
DatasetPublic 2-turn sessions across Flask, FastAPI docs, TheAlgorithms/Python, and Alice in Wonderland. No local Vaner-codebase diagnostics are included in headline claims.
What changed from the old canvas
| Old framing | New framing |
|---|---|
| Same 122B model for Vaner, answer, and judge | Local Vaner, cloud primary, separate judge |
| Raw Vaner briefing as headline | Raw, answerable, and best product path separated |
| Quality and latency only | Quality, latency, context tokens, and estimated cost |
| Context size noted descriptively | Context overhead vs RAG and cost per quality point |
Page 2: where Vaner helps and what to optimize
The next product layer is an evidence budget optimizer.
The run shows Vaner can beat RAG without using more context. The next step is making this adaptive in the product: include more when it buys quality, send less when the marginal evidence is weak, and avoid treating a fixed token cap as the strategy.
Best and worst slices
| Slice | Signal | Read |
|---|---|---|
| FastAPI auth | +3 to +4 best uplift | Good evidence shaping; Vaner beats RAG without extra context. |
| FastAPI dependency injection | +2 to +2.5 | Stable product win; both raw and answerable paths strong. |
| Flask debug session cookie | +2 to +3.5 | Code/doc context helps primary model answer beyond RAG. |
| Algorithms sorting | +3 to +4 | Answerable packaging was the main win. |
| Algorithms trees | -1 to -2 | RAG found better evidence; Vaner selected or shaped the wrong context. |
| Alice wordplay | -1 to 0 | Naked/RAG often sufficient; Vaner should abstain or compress harder. |
Optimization target
Maximize useful answer lift per primary-model token. Do not optimize for small context alone, and do not optimize for quality by dumping large context. Vaner should estimate marginal evidence value and stop when extra context no longer improves answerability.
1
Direct evidence firstRank exact-language, topic-specific, official/current evidence before predictive or broad support.
2
Marginal utility per tokenAdd an excerpt only if it contributes answer-critical facts not already covered.
3
Adaptive budgetUse compact, balanced, or expanded packets based on answerability and query breadth, not a blunt cap.
4
Abstain or deferIf RAG/naked is likely enough or Vaner’s evidence is weak, avoid sending distracting context.
Current gap
Live file recall was 0.719, below cold recall 0.813. That means quality gains are coming from packaging and synthesis in many turns, but recall still needs work.
Not proven here
The run reports no prepared-only wins: prepared_briefing_available_rate=0. This is a product-quality run, not a proof that Vaner had the final answer before the prompt.
Next benchmark
Composer/pre-submit text should be a separate condition. If Vaner sees the user typing before send, it can prepare sharper evidence and may reduce context while improving quality.
Acceptance gates for the next layer
| Gate | Target |
|---|---|
| Best Vaner uplift | > +1.0 vs best(naked, RAG) |
| Context overhead vs RAG | ≤ 0 average unless quality gain is material |
| Cost per quality point | Tracked per primary model and visible in report |
| Severe loss rate | Decrease on direct lookup / literature slices |
| Answerability | Improve full/weak classification without inflating false-ready claims |
Public claim wording
✓
Use“In a fair local-Vaner/cloud-primary benchmark, Vaner improved answer quality by +1.40 points over best(naked, RAG) while using about the same context as RAG.”
✓
Use“Answerable packaging, not raw context volume, was the main product improvement.”
!
Avoid“Vaner always beats RAG” or “Vaner predicted the exact next prompt.” This run does not prove those claims.