This report uses real synthetic Markdown fixtures. Read .owledge/benchmark-kit/BENCHMARK_EXPLAINED.md before interpreting the scores.
Profiles tested: metadata_scan, owledge_context_pack, oracle. The final product verdict is at the end of this report and is based on the Owledge context-pack profile.
Baseline metadata_scan is the naive before-state. owledge_context_pack is the product behavior under test. oracle is the ideal reference ceiling.
| Metric | Baseline metadata_scan | Owledge context pack | Oracle reference |
|---|---|---|---|
| Context pollution | 0.7356 | 0.0856 | 0.0 |
| Privacy failures | 1 | 0 | 0 |
| Stale failures | 1 | 0 | 0 |
| Tokens per correct answer | 7485.0 | 1062.67 | 898.83 |
| Verdict | Fail | Pass | Pass |
The benchmark injects private Markdown notes that look relevant to the question. A reliable context pack must exclude those notes, even when their wording appears useful.
Baseline selected private trap files: 1
Owledge selected private trap files: 0
Baseline selected files: notes/0011-privacy-trap-target.md, notes/0039-privacy-trap-distractor-a.md, notes/0088-privacy-trap-private-trap.md, notes/0000-background-filler.md, notes/0001-background-filler.md
Owledge selected files: notes/0011-privacy-trap-target.md
Interpretation: if the baseline includes a private trap file and Owledge does not, Owledge prevented the privacy leak for this scenario.
Chart direction: lower pollution, duration, and token cost are better; higher tokens/sec is better. The standalone SVG is also written to .owledge/reports/generated/benchmark-kit/charts.svg.
| Total tokens | How much prompt and completion budget the run consumed. |
|---|---|
| Tokens per correct answer | The practical token cost of useful retrieval. |
| Context pollution | How much irrelevant, stale, private, or background context entered selected packs. |
| Privacy failures | Whether private trap notes entered selected context. This must stay at zero. |
| Stale failures | Whether superseded records entered selected context. |
| tokens/sec | Approximate local generation throughput when a local model is used. |
| Duration | End-to-end benchmark runtime for this run. |
| Status | Scenario | Profile | Model | Precision | Recall | Pollution | Total tokens | tokens/sec | Duration ms |
|---|---|---|---|---|---|---|---|---|---|
| Warn | needle | metadata_scan | glm-5.1:cloud | 0.2 | 1.0 | 0.805 | 2177 | 153.8959 | 2821 |
| Pass | needle | owledge_context_pack | glm-5.1:cloud | 1.0 | 1.0 | 0.0 | 768 | 134.7275 | 2670 |
| Pass | needle | oracle | glm-5.1:cloud | 1.0 | 1.0 | 0.0 | 758 | 141.7993 | 2555 |
| Warn | multi-hop | metadata_scan | glm-5.1:cloud | 0.5 | 1.0 | 0.5056 | 2369 | 52.751 | 6878 |
| Pass | multi-hop | owledge_context_pack | glm-5.1:cloud | 1.0 | 1.0 | 0.0 | 1683 | 75.5505 | 8688 |
| Pass | multi-hop | oracle | glm-5.1:cloud | 1.0 | 1.0 | 0.0 | 1329 | 88.0626 | 3404 |
| Warn | stale-conflict | metadata_scan | glm-5.1:cloud | 0.2 | 1.0 | 0.8036 | 2049 | 56.3107 | 5468 |
| Pass | stale-conflict | owledge_context_pack | glm-5.1:cloud | 1.0 | 1.0 | 0.0 | 816 | 69.1734 | 6134 |
| Pass | stale-conflict | oracle | glm-5.1:cloud | 1.0 | 1.0 | 0.0 | 790 | 76.3643 | 5111 |
| Fail | privacy-trap | metadata_scan | glm-5.1:cloud | 0.2 | 1.0 | 0.8065 | 2272 | 80.0893 | 6896 |
| Pass | privacy-trap | owledge_context_pack | glm-5.1:cloud | 1.0 | 1.0 | 0.0 | 755 | 99.7656 | 3657 |
| Pass | privacy-trap | oracle | glm-5.1:cloud | 1.0 | 1.0 | 0.0 | 701 | 34.3603 | 9383 |
| Warn | distractor-heavy | metadata_scan | glm-5.1:cloud | 0.1111 | 1.0 | 0.8937 | 3817 | 109.1573 | 5856 |
| Warn | distractor-heavy | owledge_context_pack | glm-5.1:cloud | 0.5 | 1.0 | 0.5137 | 1086 | 23.5395 | 14929 |
| Pass | distractor-heavy | oracle | glm-5.1:cloud | 1.0 | 1.0 | 0.0 | 541 | 86.5034 | 1736 |
| Warn | handoff-resume | metadata_scan | glm-5.1:cloud | 0.4 | 1.0 | 0.5989 | 2286 | 56.0179 | 9683 |
| Pass | handoff-resume | owledge_context_pack | glm-5.1:cloud | 1.0 | 1.0 | 0.0 | 1268 | 134.1525 | 3821 |
| Pass | handoff-resume | oracle | glm-5.1:cloud | 1.0 | 1.0 | 0.0 | 1274 | 125.1156 | 4173 |
Fixture content is deterministic and synthetic. Local model timings vary by hardware, model quantization, Ollama version, and background load. The benchmark is designed to expose context pollution and handoff reliability, not to certify every real-world repository.
Fixture directory: .owledge/tmp/benchmark-kit/fixtures/20260629100314-seed42-small/small
Baseline: Baseline failed as expected: naive metadata scanning over-selected unsafe, stale, or noisy context.
Owledge: Owledge passed: the context-pack profile kept private and stale records out while staying inside the target pollution band.
Oracle: Oracle is the reference ceiling and should be treated as an ideal comparison point, not a product claim.
Conclusion: Owledge passed: the baseline profile over-selected unsafe or stale context, while the Owledge context-pack profile kept private and stale records out.