# Roadmap

Kairix is a contextual retrieval platform for human-agent teams that keeps your knowledge private and on your own infrastructure. This document describes the current state and near-term direction.

Discussion of priorities and feature direction happens in [GitHub Discussions → Roadmap & Priorities](https://github.com/three-cubes/kairix/discussions).

---

## Why this exists

The dominant pattern for AI agent memory is to send your knowledge to a third-party LLM service. This creates three problems:

1. **Privacy** — your organisation's knowledge, decisions, and relationships leave your infrastructure
2. **Context quality** — generic retrieval without entity awareness, temporal reasoning, or team-specific patterns produces mediocre results
3. **Team coherence** — agents and humans draw from different sources, breaking the shared context that makes teams effective

Kairix is the alternative: a private, on-infrastructure retrieval layer that both human team members and AI agents query against the same indexed knowledge base. Your data never leaves your servers.

---

## Current state — v2026.4.27

**R10 0.8171 · NDCG@10 0.8385** on a 160-query reference library gold suite with graded relevance gold labels (TREC-DL 0/1/2 rubric). **Hit@5 0.9629** — a relevant document in the top 5 for 96% of queries. Phase 4 gate passed (weighted total 0.8171 ≥ 0.80).

The benchmark uses strict NDCG@10 scoring with graded relevance (0/1/2), generated by `kairix eval enrich` (GPL-inspired: hybrid search → LLM judge → gold_titles). See [EVALUATION.md](EVALUATION.md) for methodology and scoring interpretation.

| Capability | Status | Notes |
|---|---|---|
| Hybrid BM25 + vector search | ✅ Shipped | RRF fusion, concurrent dispatch, all intents |
| Entity graph + alias resolution | ✅ Shipped | Neo4j canonical store, typed relationships |
| Multi-hop QueryPlanner | ✅ Shipped | LLM-decomposed sub-queries, parallel retrieval |
| Temporal query routing | ✅ Shipped | Date-aware chunking, timeline index |
| Session briefing synthesis | ✅ Shipped | GPT-4o-mini, 8-source concurrent pipeline |
| Auto-classification of writes | ✅ Shipped | Rule-based + LLM fallback |
| LLM-typed relationship enrichment | ✅ Shipped | Nightly cron, GPT-4o-mini batch classifier |
| Procedural query boost | ✅ Shipped | Path-weighted re-rank for how-to and runbook queries |
| Neo4j graph layer | ✅ Shipped | Community Edition, Bolt, `kairix.graph` module |
| Vault crawler | ✅ Shipped | Entity discovery from vault structure |
| LLM backend abstraction | ✅ Shipped | `LLMBackend` protocol, `AzureOpenAIBackend` adapter |
| Deployment-specific collections | ✅ Shipped | `KAIRIX_EXTRA_COLLECTIONS` env var |
| Contradiction detection | ✅ Shipped | `kairix contradict check` — hybrid search + LLM conflict detection |
| MCP server | ✅ Shipped | `kairix mcp serve` — search, entity, prep, timeline tools |
| Neo4j-native entity health | ✅ Shipped | Curator health reports entirely from Neo4j Cypher queries |
| entities.db retirement | ✅ Shipped | SQLite entity store removed; Neo4j is sole canonical store |
| Docker sidecar secrets | ✅ Shipped | vault-agent sidecar fetches Azure KV secrets to tmpfs at startup |
| Automated eval suite generation | ✅ Shipped | `kairix eval generate/enrich` — GPL pipeline, graded relevance, LLM judge |
| YAML retrieval config | ✅ Shipped | `kairix.config.yaml` with validation; `RerankConfig`, `TemporalBoostConfig` |
| Automated benchmark regression gate | ✅ Shipped | CI gate on PRs via mock backend; blocks NDCG regression > 0.02 |
| Temporal chunk-date boost fixes | ✅ Shipped | Explicit-only guard, midpoint date, LIKE suffix path match (TMP-7B) |
| Cross-encoder re-ranking | ✅ Shipped | `ms-marco-MiniLM-L-6-v2`; opt-in via `rerank.enabled: true`; `pip install kairix-agentic-knowledge-mgt[rerank]` |
| OpenAI SDK embed client | ✅ Shipped | `OpenAIEmbedProvider` via `openai` SDK (#43) |
| Multi-collection support | ✅ Shipped | `hybrid_search()` accepts multiple collection names |
| Port auto-detection | ✅ Shipped | `kairix mcp serve` and `kairix setup` auto-select available port |
| Docker Compose primary deployment | ✅ Shipped | `docker compose up -d` as recommended install path |
| Incremental file watcher | 🔲 Planned | `watchfiles`-based daemon; sub-60s document store sync latency |
| Observability dashboard | 🔲 Planned | Per-query latency, cache hit rate, entity graph density metrics |
| Search result feedback loop | 🔲 Planned | Thumbs-up/down signals to retrain gold suite and adjust weights |
| Multi-user isolation | 🔲 Planned | Per-agent Neo4j namespace, collection-level access control |
| Streaming search response | 🔲 Planned | Server-sent events for MCP and REST consumers |
| Webhook / push indexing | 🔲 Planned | HTTP endpoint to trigger incremental embed on external write events |
| Local/offline embedding | 🔲 Planned | `EmbedProvider` abstraction; Ollama + sentence-transformers adapters |
| REST API server mode | 🔲 Planned | FastAPI server mode exposing search, entity, and briefing as HTTP endpoints |

**Benchmark category breakdown (160-query reference library suite, NDCG@10):**

| Category | NDCG@10 | Notes |
|---|---|---|
| entity | 0.8626 | Neo4j entity boost, alias resolution |
| procedural | 0.8716 | Path-pattern re-rank active |
| recall | 0.8420 | Hybrid BM25 + vector |
| conceptual | 0.775 | Semantic intent routing |
| temporal | 0.7930 | Date-aware chunking + chunk_date boost |
| multi_hop | 0.721 | QueryPlanner functional |
| **Overall** | **R10 0.8171** | NDCG@10 0.8385, Hit@5 0.9629, MRR@10 0.7614 |

---

## Tune retrieval to your knowledge — and why it matters

Different knowledge stores reward different retrieval strategies. A reference library full of curated technical docs reads differently from a stream of daily working notes, which reads differently from project plans or meeting transcripts. There is no single "best" search configuration that works equally well across all of them. A configuration that performs strongly on one corpus is often mediocre on another.

Kairix lets you tune retrieval per collection so each gets the strategy that suits its shape.

**The shipped reference library (≈6,000 open-source engineering docs) runs through a standard 200-query benchmark every release.** This is the single most useful number we publish: it shows what kairix actually delivers on a corpus you have access to and can re-run yourself.

Latest sweep (2026-05-08, post eval-pipeline standardisation, against `suites/reflib-gold-v3.yaml`):

| Strategy | What it does | NDCG@10 | Hit@5 | MRR@10 |
|---|---|---|---|---|
| **Hybrid (BM25 + vector, RRF k=20)** ★ | Keyword + meaning, fused by reciprocal rank | **0.949** | **0.965** | **0.896** |
| Hybrid (BM25 + vector, RRF k=40) | Same, broader candidate pool | 0.947 | 0.965 | 0.894 |
| Hybrid (BM25 + vector, RRF k=60) | Same, broadest pool | 0.947 | 0.965 | 0.894 |
| Hybrid (BM25 + vector, k=60, default boosts) | Hybrid + entity + procedural boosts | 0.945 | 0.965 | 0.892 |
| BM25-primary (vector backup, top-20) | Keywords first, vectors break ties | 0.725 | 0.845 | 0.675 |
| BM25-primary (top-10) | Same, smaller backup pool | 0.696 | 0.840 | 0.671 |
| BM25-primary (top-5) | Same, narrowest backup | 0.672 | 0.840 | 0.671 |
| BM25-only (keywords only) | No semantic understanding | 0.524 | 0.685 | 0.567 |

**Hit@5 = "the right document is in the top 5 results."** For a curated technical library, hybrid search puts the answer in the top 5 for **96.5% of queries** — vs **68.5%** for keyword-only.

The lift between BM25-only (0.524 NDCG) and hybrid-RRF (0.949 NDCG) is **+81%**. That's the whole game: keyword search alone misses anything that's worded differently from how it was indexed; semantic search alone misses exact-term matches; the two together cover both. RRF with a small k tightens the fusion further.

**Why your knowledge probably needs different settings**

The reference library is dense, technical, structured, with consistent terminology. Real working knowledge usually isn't:

- **Daily notes / journals** — date-shaped paths, conversational tone, reward temporal boosts (kairix `chunk_date_boost: enabled`).
- **People-heavy content** — meeting notes, CRM exports, performance reviews — reward entity boosts (`entity.factor`) so a query for "Acme Corp" surfaces the docs that genuinely mention them.
- **How-to and runbook content** — procedural-path boosts (`procedural.factor`) push tutorials and runbooks above general conceptual hits.
- **Long-form prose / research papers** — vector-primary fusion (`fusion_strategy: rrf`) outperforms keyword-primary because the wording is intentionally varied.
- **Wikis with strict naming conventions** — keyword-primary (`fusion_strategy: bm25_primary`) outperforms vector-primary because filename and heading matches carry signal.

**What you get from tuning**

- **Faster correct answers.** Your agent finds the right doc on the first attempt, not after three follow-up queries.
- **Less wasted context.** A higher Hit@5 means more of the top results are genuinely relevant — fewer adjacent-but-wrong docs eating context window.
- **Confident synthesis.** The agent's answers are grounded in the right sources, not in adjacent-topic ones it had to settle for.
- **Visible regressions.** Once you have a gold suite for your knowledge, every kairix upgrade runs the same benchmark and tells you concretely whether it got better or worse.

**How to tune for your own knowledge**

1. Build a small gold suite from your corpus — typically 30–50 questions with the documents you'd want returned. `kairix eval auto-gold` generates a starter suite from your indexed content.
2. Run a sweep against it: `kairix eval hybrid-sweep --suite suites/your-gold.yaml --collection your-collection`. Takes ~10–15 minutes for 200 queries × 8 configs.
3. Look at the top config's `weighted_total` and `NDCG@10`. Set the matching `retrieval:` block on that collection in `kairix.config.yaml`.
4. Re-run periodically. Recent kairix work standardised the eval pipeline (issue [#143](https://github.com/three-cubes/kairix/issues/143)) so the same benchmark machinery, gold-builder, and judge are now used consistently across the reference library, the bundled examples, and your own knowledge — meaning the numbers compare cleanly across releases and across corpora.

The reference-library numbers above are the upper bound for clean, well-curated content. Yours will likely be lower in absolute terms but the *shape* of the sweep (which strategy wins for which content) is the actionable signal.

---

## Near-term

- **CLI / MCP feature parity** ([#168](https://github.com/three-cubes/kairix/issues/168)) — every kairix feature exposed via both the CLI and the MCP server with uniform UX. One use case per operation in `kairix/use_cases/`; CLI and MCP become thin adapters with shared parameter names, defaults, output shapes, and help text. Closes the timeline code-path divergence ([#163](https://github.com/three-cubes/kairix/issues/163)) by collapsing it onto the same use case both surfaces consume; subsumes the MCP error envelope shape gap ([#165](https://github.com/three-cubes/kairix/issues/165)) by pushing error-envelope construction into the use case. 8 surface-parity gaps identified (entity suggest/validate missing on MCP; entity get missing on CLI; prep, research, brief, usage_guide each missing on one side). Phased: Phase 1 timeline (template + #163 fix), Phase 2 UX parity audit on already-converged tools, Phase 3 fill missing surfaces in operator-priority order, Phase 4 uniformity polish + help-text single source of truth. See [cli-mcp-feature-parity.md](../architecture/cli-mcp-feature-parity.md). **Targeted for next sprint.**
- **Eval-module rectification** ([#143](https://github.com/three-cubes/kairix/issues/143)) — fix accumulated structural debt across `kairix/quality/eval/`: 6 silent bugs (4 silently-skipped tests, procedural-pattern filter applied to titles, inverted credential-merge logic, DB-leak in gold_builder), 2 BLOCKER S2083 vulnerabilities + prompt-injection vectors, 11+ `*_fn=None` test-substitution kwargs (eliminated via `LLMJudge` / `QueryGenerator` / `Retriever` / `ChatBackend` protocols + fakes + DI), `_call_llm` private-symbol cross-module import, raw-SQL FTS access in gold_builder (moved onto `DocumentRepository`), missing BDD + integration coverage for the gold-builder / judge / generator pipelines. `generate.py` (876 lines) split into 3 modules. Phased: Phase 0 bug fixes, Phase 0b security pass, Phase 1 protocols + fakes (foundation), Phase 2a owner-led judge refactor, Phase 2b parallel agent fan-out (file-isolated: sweep, gold_builder, generate), Phase 3 cli.py integration + deprecated-kwarg removal, Phase 4 VM deploy, Phase 5 reference-library benchmark validation against v2026.4.27 baseline (R10 0.8171, NDCG@10 0.8385). See [eval-module-refactor.md](../architecture/eval-module-refactor.md).
- **Configurable default search scope** — operators control which collections participate in default search scopes from `kairix.config.yaml` via a per-collection `in_default: bool` flag, replacing the hardcoded `_RESERVED_COLLECTIONS = {"reference-library"}` carve-out. Includes a `DefaultCollectionResolver` refactor that consolidates today's 5-branch elif chain into a `match` dispatch with predicates owned by `CollectionsConfig` / `AgentDef` rather than the resolver. **Phase 2** (composable named scopes — `scopes.research`, `scopes.with-history`) is design-tracked but uncommitted; held until a second concrete use case lands. **Phase 3** lifts the parallel `reference-library` retrieval-config hardcode in `config_loader.py:384` into per-collection `retrieval:` overrides. See [configurable-default-scope.md](../architecture/configurable-default-scope.md).
- **Pre-release container registry** — CI builds and pushes alpha Docker images to GHCR on green `develop` builds (`ghcr.io/three-cubes/kairix:2026.5.1a1`). VM deploys via `docker compose pull` instead of building from source. Tests the exact same image end users will get. Stable tags pushed on merge to `main`.
- **File watcher** — `watchfiles`-based daemon replacing the 60-second embed cron. Document store changes embedded within seconds of write, reducing lag for session-prep queries against recently-added content.
- **Observability dashboard** — structured JSON log output (`LOG_LEVEL=json`) parsed by a lightweight dashboard. Per-query latency, intent distribution, entity hit rate, RRF score distributions surfaced without third-party tooling.
- **Local/offline embedding** — `EmbedProvider` abstraction; Ollama and sentence-transformers adapters (removes Azure OpenAI as hard dependency for non-Azure deployments).
- **REST API** — FastAPI server mode exposing search, entity, and briefing as local HTTP endpoints.

---

## How priorities are set

The project is maintained by [@quanyeomans](https://github.com/quanyeomans). Priorities are influenced by:

1. **Benchmark gaps** — categories with low NDCG@10 score get attention first
2. **Deployment blockers** — things that prevent new operators from adopting the system
3. **Community demand** — upvoted Discussions and well-articulated issues

If you want to influence the roadmap, the most effective approaches are:
- Open a Discussion in [Roadmap & Priorities](https://github.com/three-cubes/kairix/discussions) with a concrete use case
- Submit a PR — working code with tests moves faster than feature requests
- Share benchmark results from your own deployment — real-world data changes priorities

---

## Contributing

See [CONTRIBUTING.md](../CONTRIBUTING.md) for setup, testing standards, and PR process.
The issues labelled [`good first issue`](https://github.com/three-cubes/kairix/issues?q=label%3A%22good+first+issue%22) are explicitly scoped for new contributors.
