Metadata-Version: 2.5
Name: locus-rag
Version: 1.1.0
Summary: Vectorless RAG with MCP — BM25 + temporal KG + link walking. Zero dependencies.
Project-URL: Homepage, https://github.com/jmiaie/locus
Project-URL: Repository, https://github.com/jmiaie/locus
Project-URL: Changelog, https://github.com/jmiaie/locus/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/jmiaie/locus/issues
Project-URL: Documentation, https://github.com/jmiaie/locus#readme
Project-URL: Benchmarks, https://github.com/jmiaie/locus/blob/main/benchmarks/README.md
Author-email: Jeff Milam <jmiaie@users.noreply.github.com>
License: MIT
License-File: LICENSE
Keywords: ai,bm25,knowledge-graph,llm,mcp,rag,retrieval,vectorless
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Indexing
Classifier: Typing :: Typed
Requires-Python: >=3.11
Provides-Extra: bench
Requires-Dist: rank-bm25>=0.2.2; extra == 'bench'
Provides-Extra: dev
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Provides-Extra: ompa
Requires-Dist: ompa>=0.3.0; extra == 'ompa'
Provides-Extra: rerank
Requires-Dist: sentence-transformers>=3; extra == 'rerank'
Description-Content-Type: text/markdown

<!-- mcp-name: io.github.jmiaie/locus -->
<div align="center">

# Locus

### Vectorless RAG &nbsp;·&nbsp; MCP-Native &nbsp;·&nbsp; Zero Dependencies

[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue?logo=python&logoColor=white)](https://python.org)
[![License: MIT](https://img.shields.io/badge/license-MIT-green?logo=opensourceinitiative&logoColor=white)](LICENSE)
[![Version](https://img.shields.io/badge/version-1.1.0-blue)](CHANGELOG.md)
[![Tests](https://img.shields.io/badge/tests-468%20passing-brightgreen?logo=pytest&logoColor=white)](tests/)
[![Zero Dependencies](https://img.shields.io/badge/dependencies-zero-brightgreen)](pyproject.toml)
[![MCP Compatible](https://img.shields.io/badge/MCP-compatible-blueviolet)](https://modelcontextprotocol.io)
[![47 MCP Tools](https://img.shields.io/badge/MCP%20tools-47-blueviolet)](#mcp-tools)
[![CI](https://github.com/jmiaie/locus/actions/workflows/ci.yml/badge.svg)](https://github.com/jmiaie/locus/actions)

**Retrieval-Augmented Generation without vectors, without GPUs, without black boxes.**

Every result tells you *exactly* why it was returned.

[Quick Start](#quick-start) &nbsp;·&nbsp;
[MCP Demo](#mcp-sales-demo) &nbsp;·&nbsp;
[Architecture](#architecture) &nbsp;·&nbsp;
[MCP Tools](#mcp-tools) &nbsp;·&nbsp;
[Benchmark Results](#benchmark-results) &nbsp;·&nbsp;
[Companion Stack](#companion-stack) &nbsp;·&nbsp;
[CLI Reference](#cli-reference)

</div>

---

## Why Vectorless?

Standard RAG pipelines require embedding models, vector databases, and GPU compute. The retrieval signal — cosine similarity — is a single opaque float that tells you nothing about *why* a document was returned. Locus takes a different path:

| | Vector RAG | **Locus** |
|:---|:---|:---|
| **Dependencies** | sentence-transformers, faiss / pinecone / chroma | **Zero** |
| **GPU required** | Yes (or paid embedding API) | **No** |
| **Why was this returned?** | `"similarity = 0.82"` | `"BM25: [jwt, auth]; KG: Alice links to doc"` |
| **Temporal queries** | Not supported | **Yes — validity windows on every KG fact** |
| **Knowledge graph** | Not supported | **Auto-extracted from prose** |
| **Contradiction detection** | Not supported | **Built in** |
| **Multi-node** | Requires external infrastructure | **Built-in cluster** |
| **Deterministic results** | No (model-dependent) | **Yes** — ties broken by index order, no hash-order dependence, retrieval never writes |

---

## Retrieval Pipeline

Locus retrieves in two stages. Stage 1 is always BM25 over a **SQLite FTS5** inverted index
(stdlib `sqlite3`, no extra packages). Stage 2 is an **optional** re-rank of the BM25 candidates
with five graph/metadata signals, fused with weighted Reciprocal Rank Fusion.

```
  Query ──► Stage 1: BM25 (SQLite FTS5)  ──► top-k chunks          mode="bm25"  (default)
                     │
                     └─► candidate pool (rerank_depth=100)
                           ├─ KG            entity expansion + one hop (honours as_of)
                           ├─ Link Walk     wikilinks from the top BM25 hits
                           ├─ Structural    frontmatter date / tag / type
                           ├─ Recency       frontmatter date, newest first
                           └─ Link Popularity  inbound wikilinks
                         weighted RRF (intent-aware, rerank_alpha=0.1) ─► top-k   mode="hybrid"
```

**Why BM25 is the default:** on the public BEIR SciFact and NFCorpus test sets the hybrid re-rank
did **not** beat plain BM25 (paired bootstrap, see [Benchmark Results](#benchmark-results)), and every
larger re-rank weight we tried on the dev splits made nDCG@10 worse. The re-rank stays available for
link- and frontmatter-rich vaults: `engine.retrieve(q, mode="hybrid")`,
`LocusEngine(retrieval_mode="hybrid")`, `LOCUS_RETRIEVAL_MODE=hybrid`, `locus retrieve --mode hybrid`,
or `mode` on the `locus_retrieve` MCP tool. Passing `as_of` selects hybrid, since the temporal filter
acts on KG facts.

**Analyzer (how text becomes index terms).** The default is `porter+english`. It lower-cases text, drops
the Locus stop-words plus a larger English function-word list, and applies Porter stemming with a
pure-stdlib stemmer (`locus/memory/stemmer.py`). On every Porter-stemmable word in the BEIR corpora it
gives the same output as SQLite FTS5's `porter` tokenizer, apart from one degenerate word.
The other analyzers are `porter` (stemming only) and `plain` (the pre-2026-10-21 behaviour: no
stemming, small stop-word list).

How the analyzer is chosen:
- Set it with `LocusEngine(analyzer=...)`, `LOCUS_ANALYZER=...` or `locus reindex --analyzer ...`.
- A store remembers its analyzer.
- Asking for a different one re-indexes from the stored chunks; source files are not re-read.
- Stores created before analyzers existed are re-indexed with the default on first open.

Index-term APIs follow the analyzer:
- `top_terms()` and `inspect_doc()` report stems.
- `get_posting_list("running")` is the same as `get_posting_list("run")`.

Use `analyzer="plain"` to keep exact-token behaviour.

| Stage-2 signal | Weight (× `rerank_alpha`) | What it captures |
|:---|:---|:---|
| KG Entity | intent-weighted | Entities from the query linked to documents |
| Link Walk | intent-weighted | Candidates reachable via wikilinks from the top hits |
| Structural | 1.0 | Frontmatter date ranges, tags, doc types |
| Recency | 0.3 | Newer frontmatter `date` first |
| Link Popularity | 0.2 | Hub documents cited by many others |

Retrieval is **read-only**: it never writes to the store unless you pass `track=True`, which records
hits in the hot-context bulletin and token budget (the `locus_retrieve` MCP tool does this by default
so `locus_hot_context` keeps working; pass `track: false` for a pure read).

---

## Quick Start

```bash
pip install locus-rag     # macOS / Linux: python3 -m pip install locus-rag · Windows: py -3 -m pip install locus-rag
```

**Index and retrieve in three commands:**

```bash
# Index a directory of markdown files
locus index ./my-docs

# Retrieve — every result includes a provenance tag
locus retrieve "how does authentication work?"
# [1] auth/jwt.md  score=7.4121  via=bm25
#     JWT tokens are validated on every request. The token is...
# [2] auth/oauth.md  score=5.0318  via=bm25
#     OAuth 2.0 flow for third-party integrations...

# Optional graph/metadata re-rank of the BM25 candidates
locus retrieve "who owns the auth service?" --mode hybrid

# Explain why a specific result was returned
locus explain <chunk_id> --query "authentication"
# {
#   "narrative": "Chunk from 'auth/jwt.md' (section: 'JWT Validation').
#                 Retrieved because: BM25: terms [jwt, authentication]
#                 appear in this chunk; KG: entity Alice links to this
#                 document via 3 KG triples.",
#   "bm25_matched_terms": ["jwt", "authentication"],
#   "kg_matched_entities": ["Alice"],
#   ...
# }
```

**As a Python library:**

```python
from locus import LocusEngine

engine = LocusEngine(store_path=".locus")
engine.index("./my-docs")

chunks = engine.retrieve("JWT authentication flow", limit=5)
for chunk in chunks:
    print(f"{chunk.doc_path}  via={chunk.provenance}")
    print(chunk.content[:200])

# Explain a result
print(engine.explain(chunks[0].chunk_id, query="JWT")["narrative"])
```

**As an MCP server** (Claude Desktop / Claude Code / Cursor / Windsurf):

```bash
# macOS / Linux
claude mcp add locus -- python3 -m locus.mcp.server --store /path/to/.locus

# Windows (Python launcher)
claude mcp add locus -- py -3 -m locus.mcp.server --store C:\path\to\.locus

# Any OS, via the installed console script
claude mcp add locus -- locus mcp --store /path/to/.locus
```

Tool calls use the `--store` you configured; a call can still target another store by passing `store_path`.

Then ask Claude: *"What does the codebase say about the auth system?"* — Locus retrieves context, Claude answers.

### MCP sales demo

Run a deck-ready end-to-end walkthrough (sample notes → MCP `locus_index` / `locus_retrieve` / `locus_explain` / KG / doctor) with **zero extra deps**:

```bash
pip install -e .          # or: pip install locus-rag
python examples/mcp_sales_demo.py
python examples/mcp_sales_demo.py --keep   # leave the temp store for inspection
```

The script calls the same `locus.mcp.server` tool dispatcher an MCP client would hit — useful for consulting demos without wiring Claude Desktop first. See [`examples/mcp_sales_demo.py`](examples/mcp_sales_demo.py).

---

## Architecture

Locus is three layers — memory, retrieval, and interface — with no layer depending upward:

```
┌─────────────────────────────────────────────────────────────────┐
│  Interface Layer                                                  │
│  ┌────────────────┐  ┌──────────────────┐  ┌─────────────────┐  │
│  │  CLI (locus)   │  │  MCP stdio server│  │  HTTP JSON-RPC  │  │
│  │  42 commands   │  │  47 tools        │  │  POST /rpc      │  │
│  │                │  │  4 prompts       │  │  Bearer token   │  │
│  │                │  │  resources       │  │  CORS opt-in    │  │
│  └────────────────┘  └──────────────────┘  └─────────────────┘  │
├─────────────────────────────────────────────────────────────────┤
│  Retrieval Layer                                                  │
│  BM25Retriever  ·  KGRetriever  ·  LinkWalker                    │
│  StructuralRetriever  ·  RecencyRetriever  ·  LinkPopularity     │
│  ──────────────────────────────────────────────────────────────  │
│  RRF Fusion  ·  QueryClassifier  ·  ContextBulletin              │
│  TokenBudget  ·  LocusCluster (multi-node)                       │
├─────────────────────────────────────────────────────────────────┤
│  Memory Layer                                                     │
│  Corpus         SQLite FTS5 BM25 index, checksum dedup, SQLite   │
│  TemporalKG     Triples + validity windows, traversal, match     │
│  EntityResolver Alias table → canonical entity names            │
│  Chunker        Section-aware (heading boundaries) + word-count  │
└─────────────────────────────────────────────────────────────────┘
```

All storage is SQLite — no external databases, no servers, no configuration.

---

## Features

### Retrieval
- **BM25 on SQLite FTS5** — inverted index, so query cost tracks the matching postings, not the corpus size (100k docs: ~93 ms p50 vs ~19 s before); pure-Python fallback when FTS5 is not compiled in
- **Optional hybrid re-rank** — KG, link walk, structural, recency and link popularity over the BM25 candidates, fused via weighted RRF (`mode="hybrid"`)
- **Incremental indexing** — changed files replace their chunks, postings and auto-extracted KG triples in place; `forget` removes them; `add_fact` facts are never touched
- **Deterministic, read-only retrieval** — ties broken by index order; nothing is written unless `track=True`
- **Section-aware chunking**
  - Splits at `#` heading boundaries; long sections fall back to word windows (400 words, 50 overlap).
  - Opt-in options (stored with the index):
    - `chunk_words` / `overlap_words` set the window size and overlap.
    - `heading_context` prefixes every window with its heading path (`Spec > 3.1 > Slabs`).
    - `boundaries="lines"` ends windows at line or paragraph breaks.
  - See [Chunking, parent-child and RM3](#chunking-parent-child-retrieval-and-rm3-opt-in).
- **Parent-child retrieval** (`retrieve(..., parent="section"|"doc")`) — matches small chunks, then returns the enclosing section or document. Each result's `spans` cite the matched passages by character offset. Opt-in.
- **RM3 query expansion** (`retrieve(..., expansion="rm3")`) — LLM-free pseudo-relevance feedback. Opt-in.
  - NFCorpus: +0.014 nDCG@10 (significant).
  - Neutral elsewhere.
  - About 5–10× BM25 latency.
- **Cross-encoder reranking** (optional `locus[rerank]` extra) — reranks the BM25 top-k with a local MiniLM cross-encoder. NFCorpus +0.018 (significant); about 400 ms/query on CPU. Opt-in.
- **Symlink-safe indexing** — directory indexing skips symlinks by default, so it never reads outside the indexed tree. Opt in with `follow_symlinks="inside"` or `True` (equivalently `--follow-symlinks inside|all`, or `LOCUS_FOLLOW_SYMLINKS`).
- **Temporal retrieval** — `as_of="2024-01-01"` returns only facts valid on that date
- **Query intent routing** (hybrid) — auto-classifies `kg_first` / `bm25_first` / `balanced` and shifts re-rank weights
- **Checksum dedup** — unchanged files never re-indexed; safe to run `index` on every startup
- **Metadata filters** — restrict results by path glob, frontmatter `type`, tags (all/any), `date` range,
  point-in-time `as_of` (honours `supersedes`) and any frontmatter field (`eq ne in nin gt gte lt lte contains exists`).
  Available in the API (`retrieve(..., filters={...})`), the CLI (`--path --type --tag --date-from --in-force --meta`)
  and the `locus_retrieve` MCP tool (`filters`). Filters never change scores, and the candidate pool deepens until `limit`
  matches are found. See [Metadata filters](#metadata-filters).
- **Optional LLM query rewriting** — `rewrite`, `multi` (RRF merge) and HyDE through any OpenAI-compatible
  endpoint (`locus.llm`, `locus retrieve --llm multi`). Off by default and never imported by the core.
  With `gpt-4o-mini`, HyDE reaches NFCorpus nDCG@10 0.359 (+0.030, significant); none of the three is significant on SciFact.
  Costs about 0.8–2 s and $0.02–0.08 per 1,000 queries. See [LLM query transformations](#llm-query-transformations-optional).

### Knowledge Graph
- **Prose triple extraction** — 14 relation patterns (`leads`, `works_at`, `is_a`, `part_of`, `replaced`, `depends_on`...) auto-run during indexing — no annotation required
- **Entity resolution** — alias table maps name variants to canonical entities transparently
- **Contradiction detection** — surfaces same-predicate triples with different objects and overlapping validity windows
- **BFS traversal** — `kg.traverse("Alice", max_depth=2)` returns the full entity subgraph
- **Pattern matching** — `kg.match("*", "works_at", "Acme")` finds everyone at Acme; `*` is a wildcard
- **KG export** — GraphML (Gephi/Cytoscape), JSONL (scripting), DOT (Graphviz)

### Explainability
```python
engine.explain(chunk_id, query="authentication")
# {
#   "bm25_matched_terms": ["jwt", "authentication"],
#   "kg_matched_entities": ["Alice"],
#   "structural_matches": ["date 2025-02-10 in range Q1 2025"],
#   "kg_context": [{"entity": "Alice", "fact_count": 3, "sample_fact": "Alice --leads--> Auth team"}],
#   "narrative": "Chunk from 'auth.md'. Retrieved because: BM25: terms [jwt,
#                 authentication]; KG: Alice links to document..."
# }
```
This is structurally impossible with vector search. No cosine score has a "why."

### MCP Integration
- **47 tools** spanning retrieval, KG operations, reasoning, bulletin management, cluster, evaluation, export, snapshots, annotations and relevance feedback
- **MCP Resources** — all indexed documents browsable as `locus://doc/{path}`
- **4 MCP Prompts** — live-context templates pre-filled with retrieved knowledge before sending to the model
- **HTTP transport** — `locus serve` binds `127.0.0.1` by default; any other host requires `--token <secret>`. No CORS headers unless you pass `--cors-origin <origin>`

### Memory Management
- **Tiered bulletin board** (Tier 0 Pinned / Tier 1 Hot / Tier 2 Archive) with hit-boost + age-decay scoring
- **Persistent hot tier** — `bulletin.sqlite3` survives process restarts
- **Soft token budget** — WARNING / TREND / CRITICAL alerts; never blocks retrieval
- **File watcher** — `locus watch ./docs` auto-reindexes changes; deletions detected

### Operations
- **Health diagnostics** — `locus doctor` checks corpus, KG, bulletin, store size, version
- **Multi-node cluster** — cross-corpus RRF fusion with node-prefixed provenance (`jarv:bm25`)
- **OMPA bridge** — imports OMPA vaults (markdown + KG triples; OMPA's `source_file` maps to Locus `source`)
- **Evaluation** — `locus benchmark qa.json` measures Recall@K and MRR

### Metadata filters

```python
engine.retrieve("slab concrete strength", filters={
    "path": ["specs/*", "rfis/*"],          # glob(s) on the document path (any)
    "exclude_path": "*/archive/*",
    "type": "spec",                          # frontmatter `type` (any of; case-insensitive)
    "tags": ["concrete"], "tags_any": ["mri", "vibration"],
    "date_from": "2026-01-01", "date_to": "2026-06-30",   # frontmatter `date`, inclusive
    "as_of": "2026-02-15",                   # in force on that date: effective/date <= it, not superseded
    "meta": {"status": "Closed", "revision": {"gte": 1}, "spec_section": {"in": ["03 30 00"]}},
})
```

```bash
locus retrieve "slab concrete" --type spec --in-force 2026-02-15 --meta "spec_section=03 30 00"
locus retrieve "roof" --filters '{"path": "change-orders/*", "meta": {"status": {"ne": "Void"}}}'
```

How filters behave:
- They act on the source document's path and frontmatter.
- Tags may be `a, b`, `[a, b]` or `#a #b`. They are split on commas and whitespace.
- Dates compare as ISO strings.
- A document that is missing a field the filter needs does not match.

Invalid keys, operators or dates raise `ValueError`. MCP returns that as an `error`.

### Chunking, parent-child retrieval and RM3 (opt-in)

```python
engine = LocusEngine(".locus", chunking={"chunk_words": 200, "overlap_words": 0,
                                         "heading_context": True, "boundaries": "lines"})
engine.index("docs/")            # chunking is stored with the index; changing it re-chunks on the next index()
hits = engine.retrieve("slab strength", parent="section")   # enclosing sections, best chunk's score
hits[0].spans   # [{"chunk_id": ..., "start": 120, "end": 410, "score": 8.1}, ...] offsets into hits[0].content
engine.retrieve("statin muscle pain", expansion="rm3", rm3={"fb_docs": 10, "fb_terms": 20, "orig_weight": 0.9})
```

```bash
locus index docs/ --chunk-words 200 --overlap-words 0 --heading-context --boundaries lines
locus retrieve "slab strength" --parent section --expand rm3 --raw
```

All three are measured against the current default in [docs/COMPARISON.md](docs/COMPARISON.md#rag-feature-pack-2026-10-23).
Settings were tuned on dev splits only. None was significantly better without a large latency cost, so all three stay opt-in:

- **Chunking options:** no significant change on the BEIR test splits. On LongMemEval R@5 they are slightly worse (not significant).
- **Parent-child retrieval:** worth using for *what it returns*: a full section with cited spans instead of a fragment. With 100-word children it is significantly worse on LongMemEval R@5 (−0.017).
- **RM3:** significantly better on NFCorpus (+0.0136 nDCG@10) and neutral on SciFact, LongMemEval and the construction demo. It costs 5–10× BM25 latency, roughly 10–20 ms instead of 2–3 ms.

### Cross-encoder reranking (optional extra)

```bash
pip install "locus-rag[rerank]" # sentence-transformers + PyTorch; the Locus core stays dependency-free
locus retrieve "statin muscle pain" --rerank --rerank-depth 20
```
```python
from locus.rerank import CrossEncoderReranker
engine = LocusEngine(".locus", reranker=CrossEncoderReranker(depth=20))   # cross-encoder/ms-marco-MiniLM-L-6-v2, CPU
engine.retrieve("statin muscle pain")                 # BM25 top-20 reranked
engine.retrieve("statin muscle pain", rerank=False)   # plain BM25
```

**Measured result** (depth 20 tuned on dev; CPU with 2 threads):
- **Quality:** NFCorpus +0.0179 nDCG@10 (significant). SciFact, LongMemEval R@5 and the construction demo are not significantly changed.
- **Cost:** about **400 ms per query instead of 2–3 ms**, and about 750 MB RSS.

It is opt-in. Use it where a few hundred milliseconds is acceptable, or on a GPU.

The model is downloaded from the Hugging Face Hub on first use; after that, set `HF_HUB_OFFLINE=1`. Hub telemetry is disabled by default.

### LLM query transformations (optional)

```python
from locus.llm import OpenAICompatibleClient, llm_retrieve
client = OpenAICompatibleClient.from_env()   # OPENROUTER_API_KEY, or OPENAI_API_KEY (+ OPENAI_BASE_URL)
hits = llm_retrieve(engine, "why is the L2 slab thicker?", strategy="multi", client=client, limit=5)
client.usage   # {'calls': 1, 'prompt_tokens': ..., 'completion_tokens': ...}
```

| Strategy | What it sends |
|---|---|
| `rewrite` | One keyword-style rewrite of the query |
| `multi` | 3 alternative queries |
| `hyde` | A short hypothetical answer passage, used as the query |

The original query is always retrieved too, and the lists are merged with RRF (k=60), so a bad rewrite cannot drop the original hits.

**Configuration and safeguards:**
- Calls use temperature 0 and are memoised per client.
- `LOCUS_LLM_MODEL` picks the model.
- `LOCUS_LLM_MAX_TOKENS_TOTAL` caps the total tokens per client.
- Works with any OpenAI-compatible server, including local ones (vLLM, llama.cpp, Ollama).
- The core never imports `locus.llm` and makes no network calls.

**Measured 2026-10-23** with `openai/gpt-4o-mini` via OpenRouter, temperature 0, all BEIR test queries
(SciFact n=300, NFCorpus n=323), paired bootstrap B=10,000. Full table and the exact LLM outputs (so the run
reproduces without an API key) are in [`benchmarks/published/2026-10-23-llm/`](benchmarks/published/2026-10-23-llm/README.md).

| Strategy | SciFact nDCG@10 (Δ) | NFCorpus nDCG@10 (Δ) | Added p50 latency | $ / 1k queries |
|---|---|---|---|---|
| default (no LLM) | 0.681 | 0.329 | – | 0 |
| `rewrite` | 0.689 (+0.008, ns) | 0.347 (**+0.018, sig.**) | ~0.85 s | $0.015–0.019 |
| `multi` (3) | 0.684 (+0.003, ns) | 0.350 (**+0.021, sig.**) | ~1.1–1.4 s | $0.032–0.048 |
| `hyde` | 0.689 (+0.008, ns) | **0.359 (+0.030, sig.)** | ~1.9–2.1 s | $0.072–0.076 |

The transforms help on short, vocabulary-mismatched queries (NFCorpus) and are neutral on long, precise claims
(SciFact). LongMemEval results for the transforms are not measured yet. Numbers depend on the model; re-measure
with yours.

---

## MCP Tools

<details>
<summary><strong>All 47 tools</strong></summary>

| Tool | Description |
|:---|:---|
| `locus_index` | Index a file or directory; skips unchanged files |
| `locus_retrieve` | BM25 retrieval (FTS5) with optional hybrid re-rank — returns chunks with provenance tags |
| `locus_explain` | Explain *why* a chunk was retrieved for a query |
| `locus_add_fact` | Add a temporal KG triple (subject–predicate–object) |
| `locus_query_entity` | Query all KG facts about an entity; supports `as_of` |
| `locus_kg_stats` | Triple count, entity count, source count |
| `locus_kg_traverse` | BFS graph walk from a starting entity |
| `locus_kg_match` | Wildcard pattern match (`*` matches any) |
| `locus_contradictions` | Find conflicting triples (same predicate, different objects) |
| `locus_add_alias` | Register entity alias → canonical mapping |
| `locus_suggest_aliases` | Suggest entity name pairs that may be duplicates |
| `locus_hot_context` | Get the hot-tier bulletin board for context injection |
| `locus_promote` | Pin a chunk to Tier 0 (never auto-archived) |
| `locus_session_start` | Warm open: corpus + KG + bulletin stats + hot context |
| `locus_wrap_up` | Session close: tick bulletin decay, return summary |
| `locus_status` | Full store status: corpus, KG, bulletin, budget, resolver |
| `locus_forget` | Remove a document from the corpus |
| `locus_sync` | Full reindex: wipe and rebuild from directory |
| `locus_doctor` | Health diagnostics — PASS / WARN / FAIL per check |
| `locus_export_kg` | Export KG to GraphML / JSONL / DOT |
| `locus_ingest_ompa` | Import an OMPA vault (markdown + KG triples) |
| `locus_benchmark` | Recall@K + MRR evaluation against a QA JSON file |
| `locus_cluster_retrieve` | Cross-node retrieval fused via RRF |
| `locus_add_node` | Register a named node in the cluster |
| `locus_remove_node` | Unregister a node |
| `locus_list_nodes` | List nodes with corpus and KG stats |
| `locus_prepare_context` | Retrieve, re-rank and pack results into a token budget |
| `locus_cache_stats` | Query cache size, hit rate, hits, misses |
| `locus_clear_cache` | Invalidate the query result cache |
| `locus_reason` | Multi-hop KG reasoning over the entities in a question |
| `locus_find_paths` | Shortest KG paths between two entities |
| `locus_inspect` | A document's chunks, top terms, entities and links |
| `locus_top_terms` | Corpus-wide top terms by document frequency |
| `locus_ingest_github` | Fetch and index markdown from a GitHub repository |
| `locus_expand_query` | Expand a query via KG aliases and first-hop neighbours |
| `locus_multi_retrieve` | Run several query variants and fuse with RRF |
| `locus_timeline` | Chronological KG facts for an entity |
| `locus_snapshot` | Archive the store to a `.tar.gz` |
| `locus_restore` | Restore a store from a snapshot (MCP: relative target inside the working dir) |
| `locus_snapshot_info` | List a snapshot's files and sizes |
| `locus_related_docs` | Similar documents by TF-IDF cosine (no embeddings) |
| `locus_diff` | Preview new / changed / deleted files before re-indexing |
| `locus_annotate` | Attach a label and note to a chunk |
| `locus_get_annotations` | Labels and notes on a chunk |
| `locus_chunks_with_label` | Chunk IDs carrying a label |
| `locus_mark_relevant` | Persist a relevance boost for a chunk and query |
| `locus_mark_irrelevant` | Persist a relevance penalty for a chunk and query |

</details>

### MCP Prompts

Four templates rendered with live context before delivery to the model:

| Prompt | What it pre-fills |
|:---|:---|
| `locus_research` | Retrieved chunks for a topic |
| `locus_entity_summary` | All KG facts + document context for an entity |
| `locus_timeline` | Temporal fact diff for an entity between two dates |
| `locus_contradiction_analysis` | Contradictions framed for human resolution |

---

## CLI Reference

```
locus [--store PATH] COMMAND [args]
```

| Command | Description |
|:---|:---|
| `index <path>` | Index files; `--pattern **/*.md`; chunking `--chunk-words --overlap-words --heading-context --boundaries words\|lines`; `--follow-symlinks no\|inside\|all` (default no) |
| `retrieve <query>` | Retrieve; `--limit 5 --mode bm25\|hybrid --as-of 2024-01-01 --track --raw`; filters `--path --exclude-path --type --tag --any-tag --date-from --date-to --in-force --meta K=V --filters JSON`; `--expand rm3`; `--parent section\|doc`; `--llm rewrite\|multi\|hyde` (optional, network) |
| `explain <chunk_id>` | Explain retrieval; `--query "..."` |
| `status` | Corpus + KG + bulletin + budget stats |
| `session-start` | Warm context open |
| `wrap-up` | Session close + bulletin tick |
| `add-fact S P O` | Add KG triple; `--valid-from --valid-to --source` |
| `query-entity <name>` | KG entity facts; `--as-of` |
| `add-alias A B` | Register alias A → canonical B |
| `contradictions [entity]` | Find conflicting KG triples |
| `kg-traverse <start>` | BFS traversal; `--depth 2 --direction out` |
| `kg-match` | Pattern match; `--subject --predicate --obj --as-of` |
| `forget <doc_path>` | Remove document from corpus |
| `sync <path>` | Full reindex |
| `reindex` | Rebuild the search index from stored chunks; `--backend fts5\|legacy` |
| `watch <path>` | Auto-reindex on changes; `--interval 5` |
| `mcp` | Start MCP stdio server |
| `serve` | Start HTTP JSON-RPC server; `--host 127.0.0.1 --port 7391 --token --cors-origin` |
| `ingest-ompa <vault>` | Import OMPA vault |
| `benchmark <qa.json>` | Recall@K + MRR; `--k 1,3,5` |
| `doctor` | Health check report |
| `export-kg <path>` | Export KG; `--format graphml\|jsonl\|dot` |

---

## Multi-node Cluster

Locus has first-class support for running multiple named nodes — each indexing a different knowledge domain — with cross-node retrieval fused via RRF.

```python
from locus import LocusCluster

cluster = LocusCluster("cluster.json")
cluster.add_node("docs",    "/vaults/docs/.locus")
cluster.add_node("code",    "/vaults/code/.locus")
cluster.add_node("logs",    "/vaults/logs/.locus")

# Query all three simultaneously
chunks = cluster.retrieve("auth system deployment", limit=5)
for c in chunks:
    print(f"{c.provenance}  {c.doc_path}")
    # docs:bm25     auth/jwt.md
    # code:kg       src/auth/middleware.py.md
    # logs:recency  incidents/2025-02-auth-outage.md
```

Each result is tagged `node_name:signal` so you always know which knowledge base contributed which chunk. The registry persists across restarts — add a node once, always available.

---

## Companion stack

Locus is the **explainable retrieval** layer. Pair it with OMPA (memory) and CognitionOS (compliance product):

| Layer | Project | Role |
|---|---|---|
| Explainable retrieval | **Locus** ([`locus-rag`](https://pypi.org/project/locus-rag/)) | Vectorless RAG — BM25 + temporal KG + link walking, zero GPU, MCP-native |
| Vault memory | **[OMPA](https://github.com/jmiaie/ompa)** ([PyPI](https://pypi.org/project/ompa/)) | Vault · palace · temporal KG · agent lifecycle hooks |
| Compliance product | **[CognitionOS](https://github.com/jmiaie/cognition-os)** | Audit · RBAC · retention · PHI redaction on top of Locus + OMPA |

CognitionOS consumes Locus via git submodule / optional pip; keep the public `locus` API stable when changing this package.

## OMPA Bridge

If you already use [OMPA](https://github.com/jmiaie/ompa) as your agent memory layer, Locus can import your vault directly. OMPA's `triples` table (`source_file` column) is mapped onto the Locus KG (`source`). If the table is missing required columns the import raises `OMPASchemaError` rather than silently importing nothing.

```bash
locus ingest-ompa ~/path/to/ompa-vault --store .locus
```

```python
from locus.bridge.ompa import OMPABridge
bridge = OMPABridge(engine, vault_path="~/ompa-vault")
result = bridge.ingest()
# {"vault": "...", "chunks_indexed": ..., "triples_imported": ..., "kg_source": ".../.palace/knowledge_graph.sqlite3"}
```

Markdown files are indexed via the normal pipeline. KG triples from `.palace/knowledge_graph.sqlite3` are copied row by row after a schema check. Safe to re-run — checksum dedup prevents duplicate indexing.

---

## Health Diagnostics

```bash
locus doctor
```

```
Locus Doctor
============================================
[PASS]  corpus               47 docs | 203 chunks | avg 4.3 chunks/doc
[PASS]  knowledge_graph      1847 triples | 312 entities | 2 contradictions
[WARN]  bulletin             Tier 0 (pinned) is full — oldest entries will be demoted
[PASS]  entity_resolver      14 aliases registered
[PASS]  store_size           3.21 MB at .locus
[PASS]  version              Locus v1.1.0

Result: 5 pass  1 warn  0 fail
```

---

## Benchmark Results

### At a glance

| What | Result | Data / n | Source |
|---|---|---|---|
| Default retrieval quality (BM25 + `porter+english`) | SciFact nDCG@10 **0.681**, NFCorpus **0.329**, LongMemEval-S R@5 **0.977** | BEIR test (300 / 323 q), LME-S (470 q) | [2026-10-21-analyzer](benchmarks/published/2026-10-21-analyzer/README.md) |
| Search latency at 100k docs | p50 **93 ms** (was 19.0 s BM25-only / 54.6 s six-signal in 1.0.0) | synthetic Zipf corpus, 100 q | [2026-10-20](benchmarks/published/2026-10-20/scaling_before_after.csv) |
| Index build / store at 100k docs | 34 s / 419 MB (was 21.7 min / 1,324 MB) | same | [2026-10-20](benchmarks/published/2026-10-20/scaling_before_after.csv) |
| RM3 expansion (opt-in) | NFCorpus **+0.0136** (sig.), neutral elsewhere, 5–10× latency | BEIR test, LME-S, demo | [2026-10-23-ragpack](benchmarks/published/2026-10-23-ragpack/summary.json) |
| Cross-encoder rerank (opt-in extra) | NFCorpus **+0.0179** (sig.), neutral elsewhere, ~400 ms/query | same | [2026-10-23-ragpack](benchmarks/published/2026-10-23-ragpack/summary.json) |
| LLM HyDE (opt-in, `gpt-4o-mini`) | NFCorpus **0.359 (+0.030, sig.)**; SciFact +0.008 (ns) | BEIR test | [2026-10-23-llm](benchmarks/published/2026-10-23-llm/README.md) |
| Construction demo (synthetic, workflow demo) | R@5 0.952 (1.000 on point-in-time questions with as-of) | 24 docs, 21 q | [examples/construction](examples/construction/) |
| Head-to-head vs mem0 / Zep / Letta / LangMem | **pending** — not run yet; no numbers are claimed | – | – |

"sig." means the 95% paired-bootstrap CI of the difference excludes 0; "ns" means it does not. Latencies are from a
shared 8-vCPU VM and are indicative only.

### Details

Every number below is reproducible with the scripts in [`benchmarks/repro/`](benchmarks/repro/);
methodology, data sources and caveats are in [`benchmarks/README.md`](benchmarks/README.md).
Measured 2026-10-20 on a shared 8-vCPU / 16 GB Linux VM (Python 3.13, SQLite 3.46.1) — treat latencies as
indicative. 95% CIs are seeded bootstrap (B = 10,000) over queries.

### Public datasets — BEIR test splits, nDCG@10

| System | SciFact (300 q) | NFCorpus (323 q) | p50 latency (SciFact / NFCorpus) |
|---|---|---|---|
| **Locus default (`mode="bm25"`, FTS5, `porter+english` analyzer)** ¹ | **0.681** [0.638, 0.724] | **0.329** [0.294, 0.364] | 2.3 ms / 0.8 ms ² |
| Locus hybrid (`mode="hybrid"`, `porter+english`) ¹ | 0.678 [0.635, 0.721] | 0.328 [0.294, 0.363] | 7.2 ms / 4.1 ms |
| Locus BM25, `plain` analyzer (the 2026-10-20 default) | 0.668 [0.623, 0.713] | 0.306 [0.272, 0.341] | 2.7 ms / 1.2 ms |
| Locus hybrid, `plain` analyzer | 0.669 [0.624, 0.714] | 0.306 [0.272, 0.341] | 7.7 ms / 4.6 ms |
| LlamaIndex `BM25Retriever` (bm25s, stemmer + stop-words, defaults) ¹ | 0.687 [0.644, 0.730] | 0.323 [0.288, 0.358] | 0.5 ms / 0.4 ms |
| `rank_bm25` BM25Okapi (Locus `plain` tokens) | 0.664 [0.618, 0.709] | 0.308 [0.274, 0.343] | 7.2 ms / 1.6 ms |
| *Locus 1.0.0 six-signal pipeline (before)* | *0.581* [0.538, 0.624] | *0.284* [0.251, 0.318] | *878 ms / 404 ms* |

¹ Measured 2026-10-21 (analyzer change; artefacts in [`benchmarks/published/2026-10-21-analyzer/`](benchmarks/published/2026-10-21-analyzer/)).
² Interleaved in-process measurement, same session as `plain`: 1.23× / 1.01× the `plain` p50 (SciFact / NFCorpus). Other rows' latencies are from separate sessions.

- **Stemming + stop-words (default since 2026-10-21):** vs the `plain` analyzer, +0.0129 [−0.0064, +0.0325] nDCG@10 on SciFact (not significant), **+0.0223 [+0.0119, +0.0335]** on NFCorpus and **+0.0128 [+0.0021, +0.0255]** LongMemEval-S R@5 (0.964 → 0.977, n = 470). The NFCorpus and LongMemEval-S gains are significant. The analyzer was chosen on the dev splits only; k1/b stay at FTS5's 1.2/0.75 (see below).
- **vs LlamaIndex BM25 (bm25s):** −0.0059 [−0.0159, +0.0043] on SciFact (not significant), **+0.0058 [+0.0010, +0.0115]** on NFCorpus (significant) and +0.0064 [−0.0064, +0.0213] on LongMemEval-S (not significant). bm25s is still faster per query.
- **vs `rank_bm25`:** +0.0174 [−0.0020, +0.0378] (SciFact), **+0.0206 [+0.0104, +0.0317]** (NFCorpus), +0.0043 [−0.0064, +0.0170] (LongMemEval-S).
- **Earlier comparison (`plain` analyzer) — parity with `rank_bm25`:** paired difference +0.0045 [−0.0015, +0.0111] on SciFact and −0.0017 [−0.0043, +0.0009] on NFCorpus — not significant either way.
- **Hybrid vs BM25:** +0.0007 [−0.0056, +0.0080] (SciFact) and −0.0001 [−0.0012, +0.0009] (NFCorpus) — no significant gain, so BM25 is the default.
  The re-rank weight and depth were chosen on the dev splits only (SciFact train, NFCorpus dev); no larger weight helped there.
- **vs Locus 1.0.0:** +0.0872 [+0.0636, +0.1111] (SciFact) and +0.0220 [+0.0138, +0.0305] (NFCorpus) nDCG@10 over the old six-signal default; +0.0278 / +0.0124 over the old BM25-only path (both significant).
- **Monte Carlo, 10 random 2,000-doc SciFact sub-corpora:** mean nDCG@10 BM25 0.742 · `rank_bm25` 0.742 · hybrid 0.736 (hybrid lower in 8 of 10 seeds).

### Scaling — synthetic corpus (Zipf vocabulary, known-item queries, 100 queries)

| Docs | Index build | Store size | Query p50 / p95 (default) | Before (1.0.0): build · size · p50 |
|---|---|---|---|---|
| 1,000 | 0.24 s | 4.4 MB | 1.5 ms / 2.2 ms | 6.0 s · 13 MB · 207 ms |
| 10,000 | 2.5 s | 43 MB | 10.5 ms / 16.3 ms | 85 s · 130 MB · 1.36 s |
| 100,000 | 34 s | 419 MB | 93 ms / 137 ms | 21.7 min · 1,324 MB · 19.0 s |

Query cost grows with the size of the matching posting lists: the synthetic queries include very common
words, so FTS5 still scores most of the corpus. p95 at 100k is above 100 ms on this machine.

### Determinism and the old wiki claims

The 1.0.0 README reported R@5 0.947 / MRR 0.633 for the six-signal pipeline on the bundled Engineering
Wiki benchmark. That did not reproduce (0.867 / 0.594), and the simple-100 benchmark gave three different
answers across five runs. With this release both benchmarks give identical results in 5 of 5 fresh
processes with different `PYTHONHASHSEED` values:

| Benchmark (`benchmarks/run.py`) | System | R@5 | MRR |
|---|---|---|---|
| Engineering Wiki (113 queries, synthetic) | Locus default (BM25) | 1.000 | 0.727 |
| | Locus hybrid | 1.000 | 0.743 |
| | `rank_bm25` | 1.000 | 0.760 |
| simple-100 (synthetic) | Locus default / hybrid | 0.600 | 0.274 |
| | `rank_bm25` | 0.590 | 0.271 |

These two corpora are generated by Locus's own code and are small, so treat them as smoke tests rather than evidence;
the BEIR numbers above are the ones to quote.

## Benchmarking

Create a QA JSON file and run evaluation against your indexed corpus:

```json
[
  {"query": "how does JWT authentication work?",    "expected_docs": ["auth/jwt.md"]},
  {"query": "incident response runbook",            "expected_docs": ["ops/ir.md"]},
  {"query": "Q1 2025 architecture decisions",       "expected_docs": ["adrs/q1-2025.md"]}
]
```

```bash
locus benchmark qa.json --k 1,3,5
```

```
Locus Benchmark
========================================
Queries:  20

Recall@1: 0.650  (13/20)
Recall@3: 0.800  (16/20)
Recall@5: 0.850  (17/20)
MRR:      0.723

Misses (top 5):
  [MISS] "incident response process" expected=['ir.md'] got=['deploy.md', ...]
```

---

## KG Export & Visualization

```bash
locus export-kg knowledge.graphml   # Gephi / Cytoscape / yEd
locus export-kg knowledge.dot       # Graphviz → PNG/PDF
locus export-kg knowledge.jsonl     # scripting / grep
```

```bash
# Render a quick PNG with Graphviz
dot -Tpng knowledge.dot -o knowledge.png
```

---

## Installation

```bash
pip install locus-rag          # macOS / Linux: python3 -m pip install locus-rag · Windows: py -3 -m pip install locus-rag
```

Locus is also described for the MCP registry in [`server.json`](server.json) (`uvx locus-rag mcp --store PATH`).
Since 1.1.0 the wheel installs both a `locus` and a `locus-rag` console script (same CLI). On 1.0.0, which has
only `locus`, use `uvx --from locus-rag locus mcp`.

**Zero runtime dependencies.** Locus uses only Python stdlib: `sqlite3`, `re`, `pathlib`, `threading`, `http.server`, `hashlib`, `math`, `json`.

Optional extras:

```bash
pip install "locus-rag[ompa]"   # OMPA vault bridge
pip install "locus-rag[dev]"    # pytest, coverage
pip install "locus-rag[bench]"  # rank_bm25 baseline for benchmarks
pip install "locus-rag[rerank]" # optional cross-encoder reranker (sentence-transformers + PyTorch)
```

**Requirements:** Python 3.11+

### On-prem deployment (container or offline bundle)

Nothing in these targets uploads anything. There is no registry push and no PyPI upload.

```bash
make offline-bundle      # dist/locus-offline-<ver>.tar.gz: wheel + sdist + install.sh + SHA256SUMS
make image               # builds locus:local from dist/*.whl (docker, or podman with --format docker)
make image-save          # dist/locus-image-<ver>.tar for air-gapped hosts (docker/podman load -i …)
scripts/container-smoke-test.sh locus:local   # 24 checks, no network needed
```

**Offline bundle.** Copy the tarball to the target host and run `tar xzf locus-offline-*.tar.gz`. Then run
`./locus-offline-*/install.sh /path/to/venv/bin/python`. It checks the checksums and installs with
`pip --no-index`, which works because Locus has no runtime dependencies.

**Image.** Based on `python:3.12-slim`, pinned by digest, about 124 MB. It:
- runs as uid 10001;
- works with a read-only root filesystem;
- has a HEALTHCHECK on `/health`;
- stops cleanly on SIGTERM.

The index lives in `/data` (a volume). Mount source documents read-only at `/docs`.

**HTTP binding.** HTTP binds `127.0.0.1` inside the container by default, so a published port is unreachable.
For local use, use `docker exec` or MCP stdio (`docker run -i --rm -v locus-data:/data locus:local mcp`).
For remote use:
- set `LOCUS_HTTP_HOST=0.0.0.0`;
- provide the bearer token as a file or secret through `LOCUS_HTTP_TOKEN_FILE`;
- publish the port on the host loopback only, and put TLS in front of it.

Locus refuses to bind a non-loopback address without a token. [`deploy/docker-compose.yml`](deploy/docker-compose.yml)
sets all of this up with a Compose secret, `read_only`, `cap_drop: [ALL]`, `no-new-privileges` and memory/pid limits.

The smoke test checks the following with `--network=none`:
- the non-root user and the healthcheck;
- indexing and retrieval on a read-only rootfs;
- MCP stdio;
- that the loopback-bound port is unreachable from the host;
- the refusal to start without a token;
- 401/403 on a missing or wrong token or a foreign `Origin`;
- that the token does not appear in logs or config;
- clean stop;
- installing the offline bundle.

**Construction demo.** [`examples/construction/`](examples/construction/) has 24 **synthetic** project documents:
specs with revisions, RFIs, submittals, change orders, permits, a lease and OAC minutes. It also has 21
labelled questions (`python examples/construction/demo.py`). Results are R@5 0.952 overall, and 1.000 on
the point-in-time questions when the as-of view is used. They come from a small fictional set; treat them as a
workflow demo, not a benchmark.

---

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md). All contributions welcome — bug reports, feature requests, PRs.

```bash
git clone https://github.com/jmiaie/locus
cd locus
pip install -e ".[dev]"
pytest tests/ -v
```

---

## License

MIT — see [LICENSE](LICENSE).

---

## Acknowledgements

Locus is architecturally descended from [OMPA](https://github.com/jmiaie/ompa) (Obsidian-MemPalace-Agnostic), which pioneered the three-layer vault → palace → KG design for agent memory. The tiered bulletin board, temporal KG schema, and session lifecycle patterns are direct adaptations. The OMPA bridge enables seamless import of existing OMPA vaults. [CognitionOS](https://github.com/jmiaie/cognition-os) productizes Locus + OMPA into a compliance SaaS (audit, RBAC, retention) — see [Companion stack](#companion-stack).
