Benchmark results
Accuracy by tier, latency, and cost for the strategies recorded in each run.
kb-arena benchmark to evaluate your corpus.| Strategy | Tier 1 LookupiSingle fact retrieval from one document. Example: 'What is the default timeout?' | Tier 2 How-ToiStep-by-step procedure within one topic. Example: 'How do I enable server-side encryption?' | Tier 3 ComparisoniChoosing between two options or configurations. Example: 'Compare hot storage vs cold archive for compliance.' | Tier 4 IntegrationiCross-topic dependencies requiring 3-4 connected components. Example: 'What permissions does service A need for B and C?' | Tier 5 ArchitectureiFull system design spanning 3-5+ topics. Example: 'How does a request flow from ingress through processing to storage?' | Avg % v | Latency | Cost/Q |
|---|---|---|---|---|---|---|---|---|
| QnA PairsiGenerates question-answer pairs at index time and retrieves against those pairs instead of only source chunks. | 79% | 85% | 83% | 84% | 66% | 79.4% | 9043 ms | $0.4800 |
| Knowledge GraphiExtracts entities and relationships into Neo4j, then queries the graph through intent-matched Cypher templates. | 72% | 69% | 61% | 77% | 79% | 71.6% | 20322 ms | $1.3700 |
| HybridiRoutes by intent between vector and graph paths, then uses reciprocal rank fusion when both paths contribute. | 39% | 81% | 61% | 80% | 62% | 64.6% | 41549 ms | $3.0200 |
| RAPTORiBuilds a recursive tree of chunk clusters and summaries, then queries leaf and summary levels together. | 30% | 16% | 15% | 36% | 30% | 25.4% | 7240 ms | $0.6900 |
| Naive VectoriChunks documents, embeds each chunk, and retrieves top-k by cosine similarity as a dense baseline. | 27% | 15% | 14% | 26% | 22% | 20.8% | 6421 ms | $0.3300 |
| Contextual VectoriPrepends parent topic context to each chunk before embedding so you can measure whether document context changes retrieval. | 25% | 11% | 9% | 26% | 11% | 16.4% | 5114 ms | $0.2900 |
| PageIndexiBuilds a hierarchical tree from document structure and uses model-guided traversal without an embedding index. | 19% | 12% | 7% | 21% | 12% | 14.2% | 10933 ms | $0.2900 |
| BM25iUses BM25 keyword matching as a keyless lexical baseline with no embeddings or graph service. | 24% | 11% | 9% | 16% | 10% | 14.0% | 4514 ms | $0.2600 |
Results from your benchmark runs. Run kb-arena benchmark to regenerate with different corpora or strategies.
Accuracy by tier
Methodology
Each question is sent to the strategies recorded in the run. Answers are evaluated through a 4-pass pipeline: structural checks (must_mention / must_not_claim), entity coverage against source documentation, source attribution, and LLM-as-judge scoring for accuracy, completeness, and faithfulness.
Composite ranking: 0.5 * accuracy + 0.3 * reliability + 0.2 * latency_score. Latency score inverts p95 so lower is better.
Tiers: 1 = lookup (single fact retrieval), 2 = how-to (procedure within one topic), 3 = comparison (option A vs B), 4 = integration (cross-topic dependencies), 5 = architecture (3+ topics, system design).