Metadata-Version: 2.4
Name: fusion-rag
Version: 0.8.34
Summary: Fusion-RAG — Apple Silicon native offline vector knowledge base backend
Author: dahai80
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/dahai80/fusion-rag
Project-URL: Repository, https://github.com/dahai80/fusion-rag
Project-URL: Issues, https://github.com/dahai80/fusion-rag/issues
Project-URL: Changelog, https://github.com/dahai80/fusion-rag/releases
Keywords: rag,vector-database,embeddings,bm25,hybrid-search,apple-silicon,mlx
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: MacOS
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.27.0
Requires-Dist: fusion-core>=0.12.1
Requires-Dist: fastapi>=0.115.0
Requires-Dist: uvicorn>=0.30.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: lancedb>=0.10.0
Requires-Dist: pyarrow>=15.0.0
Requires-Dist: PyMuPDF>=1.24.0
Requires-Dist: python-docx>=1.1.0
Requires-Dist: python-pptx>=1.0.0
Requires-Dist: openpyxl>=3.1.0
Requires-Dist: ebooklib>=0.18
Requires-Dist: markdownify>=0.12.0
Requires-Dist: aiofiles>=23.0.0
Requires-Dist: jieba>=0.42.1
Requires-Dist: rank_bm25>=0.2.2
Requires-Dist: pathspec>=0.12.0
Requires-Dist: setproctitle>=1.3.0
Provides-Extra: test
Requires-Dist: pytest>=8.0.0; extra == "test"
Requires-Dist: pytest-asyncio>=0.23.0; extra == "test"
Requires-Dist: pytest-cov>=5.0.0; extra == "test"
Requires-Dist: pytest-mock>=3.14.0; extra == "test"
Provides-Extra: fusion-store
Requires-Dist: numpy>=1.26.0; extra == "fusion-store"
Provides-Extra: qdrant
Requires-Dist: qdrant-client>=1.10.0; extra == "qdrant"
Dynamic: license-file

---
doc_id: fusion-rag/readme
product: fusion-rag
class: platform
entry: service
doc_type: readme
title: {zh: "Fusion-RAG 离线向量知识库后端", en: "Fusion-RAG Offline Vector Knowledge Base Backend"}
audience: engineer
status: draft
version: 0.8.21
updated: 2026-11-07
owner: "@dahai"
source_of_truth:
  - README.md
---

<div align="center">

# Fusion-RAG

**Apple Silicon Native Offline Vector Knowledge Base Backend**

Local vector knowledge base service for the Fusion-MLX ecosystem — offline by default, no data leaves your device unless you explicitly opt into the cloud embedding fallback (`FUSION_RAG_FALLBACK_URL`, empty by default).

[![Python](https://img.shields.io/badge/Python-3.12+-3776AB.svg)](https://www.python.org/)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
[![Tests](https://img.shields.io/badge/Tests-243-success.svg)](tests/)
[![Version](https://img.shields.io/badge/Version-0.8.3-blue.svg)]()

[Quick Start](#quick-start) · [API Reference](#api-reference) · [Architecture](#architecture) · [Documentation](docs/)

</div>

---

## Why Fusion-RAG?

| Feature | Fusion-RAG | Dify RAG | LangChain RAG | Claude RAG |
|---------|-----------|----------|---------------|------------|
| **MLX native** | ✅ fusion-mlx API | ❌ Ollama/Cloud | ❌ Cloud API | ❌ Cloud |
| **Apple Silicon optimized** | ✅ LanceDB + MLX | ❌ | ❌ | ❌ |
| **Multi-KB isolation** | ✅ | ✅ | ❌ | ❌ |
| **BM25 + Vector hybrid** | ✅ Okapi BM25 + RRF | ❌ | ❌ | ❌ |
| **Contextual Retrieval** | ✅ Anthropic-style | ❌ | ❌ | ✅ |
| **MCP Server** | ✅ Claude/Cursor native | ❌ | ❌ | ❌ |
| **GraphRAG** | ✅ Entity-relation graph (`/graph/*` routes, experimental) | ❌ | ❌ | ❌ |
| **Code AST chunking** | ✅ Python AST-aware | ❌ | ❌ | ❌ |
| **Version snapshots** | ✅ Hard-link snapshots | ❌ | ❌ | ❌ |
| **Incremental sync** | ✅ MD5 + mtime detection | ❌ | ❌ | ❌ |
| **Search templates** | ✅ Preset + custom | ❌ | ❌ | ❌ |
| **Audit logging** | ✅ Full search audit trail | ❌ | ❌ | ❌ |
| **Fine-grained permissions** | ✅ ACL with role inheritance | ❌ | ❌ | ❌ |
| **Cloud embed fallback** | ✅ Auto-fallback on local fail | ❌ | ✅ | ✅ |
| **Git repo indexing** | ✅ Clone + .gitignore aware | ❌ | ❌ | ❌ |
| **Bench API** | ✅ Latency benchmarks | ❌ | ❌ | ❌ |
| **Distributed storage** | ✅ StoreBackend abstraction | ❌ | ❌ | ❌ |
| **Local offline** | ✅ Offline by default | ⚠️ Partial | ❌ | ❌ |
| **Zero API cost** | ✅ | ❌ | ❌ | ❌ |

**One sentence:** Fusion-RAG is a unified local vector knowledge base backend for the Fusion-MLX ecosystem — all Embedding goes through fusion-mlx HTTP API, no direct MLX imports.

---

## Quick Start

### Prerequisites

- macOS with Apple Silicon (M1–M5)
- Python 3.12+
- [fusion-mlx](https://github.com/dahai80/fusion-mlx) running on `localhost:11432`

### Install

From PyPI (after the first tagged release is published):

```bash
pip install fusion-rag
# optional extras:
pip install "fusion-rag[qdrant]"   # Qdrant vector backend
```

From source (development):

```bash
git clone https://github.com/dahai80/fusion-rag.git
cd fusion-rag
pip install -e ".[test]"
```

### Start the Server

```bash
./start.sh start
# Fusion-RAG running on http://127.0.0.1:11436
```

### Minimal Example

```python
import asyncio
from fusion_rag import KnowledgeBase, DocumentParser, Chunker
from fusion_rag.embed.client import EmbeddingClient
from fusion_rag.store.vector_store import VectorStore

async def main():
    # 1. Parse a document
    parser = DocumentParser()
    result = await parser.parse("README.md")
    print(f"Parsed: {result.file_name} ({result.chars} chars)")

    # 2. Chunk the text
    chunker = Chunker(strategy="semantic")
    chunks = await chunker.chunk(result)
    print(f"Chunks: {len(chunks)}")

    # 3. Embed via fusion-mlx
    embed = EmbeddingClient(model="BGE-M3")
    vectors = await embed.embed_batch([c.text for c in chunks])
    print(f"Vectors: {len(vectors)}")

asyncio.run(main())
```

---

## API Reference

Fusion-RAG provides a REST API at `/kb/*` for knowledge base operations.

### Knowledge Base Management

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/kb/bases` | List all knowledge bases |
| POST | `/kb/bases` | Create a knowledge base |
| GET | `/kb/bases/{id}` | Get knowledge base details |
| DELETE | `/kb/bases/{id}` | Delete a knowledge base |
| GET | `/kb/bases/{id}/stats` | Get KB statistics |

### Document Operations

| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/kb/bases/{id}/documents` | Upload and index a document |
| POST | `/kb/bases/{id}/documents/batch` | Batch upload multiple documents |
| POST | `/kb/bases/{id}/documents/ingest` | Ingest inline content: text-ish types (`text`/`markdown`/`html`/`csv`) index directly; `pdf`/`docx`/`epub` treat `content` as base64-encoded binary and run the full parse pipeline (incl. the scanned-PDF OCR path) — invalid base64 returns 400 (issue #90) |
| GET | `/kb/bases/{id}/documents` | List all documents in a KB |
| PATCH | `/kb/bases/{id}/documents/{doc_id}` | Patch document lifecycle metadata (`status`: active/deprecated, `superseded_by`) — issue #86 |
| DELETE | `/kb/bases/{id}/documents/{doc_id}` | Delete a document and its chunks |
| PUT | `/kb/bases/{id}/documents/{doc_id}` | Replace a document (re-index) |
| GET | `/kb/bases/{id}/documents/{doc_id}/status` | Get document indexing status |
| POST | `/kb/bases/{id}/scan` | Scan directory and index all files |
| POST | `/kb/bases/{id}/watch` | Watch files for changes |
| POST | `/kb/bases/{id}/unwatch` | Stop watching files |
| GET | `/kb/bases/{id}/watch/status` | Get watch status |

### Search & RAG

| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/kb/bases/{id}/search` | Semantic search (hybrid, rerank, templates, status_filter) |
| POST | `/kb/bases/{id}/ask` | RAG Q&A (model, hybrid, rerank, folder_prefix, status_filter, citations) |
| POST | `/kb/bases/{id}/similar` | Near-duplicate / version-drift search by text or raw embedding (issue #83) |

#### Search Parameters

| Parameter | Default | Description |
|-----------|---------|-------------|
| `query` | (required) | Search query text |
| `top_k` | 10 | Number of results |
| `threshold` | 0.0 | Minimum similarity score |
| `hybrid` | false | Enable BM25+Vector hybrid search |
| `hybrid_alpha` | 0.7 | Vector weight (alpha fusion) |
| `hybrid_method` | "rrf" | Fusion method: "alpha" or "rrf" |
| `rerank` | false | Enable reranking (default ON when `FUSION_RAG_RERANK_MODEL` is set) |
| `rerank_backend` | from env | Rerank stage: `cross_encoder` (fusion-mlx `/v1/rerank`, real bge-reranker-v2-m3) or `llm` (legacy prompt-scoring) |
| `rerank_model` | from env | Cross-encoder model name (overrides `FUSION_RAG_RERANK_MODEL`) |
| `rerank_top_n` | 20 | Candidate pool size fed to the reranker before truncating to `top_k` (recall lift) |
| `folder_prefix` | (none) | Filter results by folder path prefix |
| `template` | (none) | Search template name (general/code/design) |
| `rewrite_mode` | (none) | Query rewrite: hyde, expand, condense |
| `status_filter` | "active" | Document status filter: `active` (default — drops deprecated docs), `deprecated`, `all` (issue #86) |

#### Ask Parameters

| Parameter | Default | Description |
|-----------|---------|-------------|
| `question` | (required) | Question text |
| `model` | "qwen3.5-9b" | LLM model for answer generation |
| `max_tokens` | 4096 | Max output tokens |
| `temperature` | 0.3 | Sampling temperature |
| `hybrid` | false | Enable hybrid search |
| `rerank` | false | Enable reranking (default ON when `FUSION_RAG_RERANK_MODEL` is set) |
| `rerank_backend` | from env | `cross_encoder` (fusion-mlx `/v1/rerank`) or `llm` (legacy) |
| `rerank_model` | from env | Cross-encoder model name |
| `rerank_top_n` | 20 | Candidate pool size before truncating to `top_k` |
| `folder_prefix` | (none) | Filter by folder path |
| `status_filter` | "active" | Document status filter: `active`, `deprecated`, `all` (issue #86). Deprecated chunks kept via `all` are marked `[已作废版本]` in the LLM context; the response carries per-chunk `citations[]` + `has_low_confidence` (issue #85) |

### Version Snapshots

| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/kb/bases/{id}/versions` | Create a snapshot |
| GET | `/kb/bases/{id}/versions` | List snapshots |
| GET | `/kb/bases/{id}/versions/{vid}` | Get snapshot details |
| POST | `/kb/bases/{id}/versions/{vid}/rollback` | Rollback to snapshot |
| DELETE | `/kb/bases/{id}/versions/{vid}` | Delete a snapshot |

### Search Templates

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/kb/bases/{id}/templates` | List search templates |
| POST | `/kb/bases/{id}/templates` | Create a custom template |
| DELETE | `/kb/bases/{id}/templates/{name}` | Delete a custom template |

### Permissions

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/kb/bases/{id}/permissions` | List permission rules |
| POST | `/kb/bases/{id}/permissions` | Add a permission rule |
| DELETE | `/kb/bases/{id}/permissions/{rule_id}` | Delete a permission rule |
| POST | `/kb/bases/{id}/permissions/check` | Check if action is allowed |

### Audit & Admin

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/kb/bases/{id}/audit` | Query audit logs |
| GET | `/kb/bases/{id}/audit/{log_id}` | Get a specific log entry |
| GET | `/kb/bases/{id}/audit/export` | Export audit logs (json/csv) |
| POST | `/kb/bases/{id}/sync` | Incremental directory sync |
| POST | `/kb/bases/{id}/bench` | Run search benchmark |
| GET | `/kb/bases/{id}/bench/results` | List benchmark results |
| GET | `/kb/config` | View runtime config (env-driven knobs, read-only) |

### Project-KB Mapping

| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/kb/projects/{project_id}/kb` | Map project to KB |
| GET | `/kb/projects/{project_id}/kb` | Get project KB mapping |
| DELETE | `/kb/projects/{project_id}/kb` | Remove project KB mapping |

### MCP Protocol

| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/mcp` | MCP JSON-RPC handler (tools/list, tools/call) |

### System

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/kb/status` | Service status |
| GET | `/health` | Health check |
| GET | `/metrics` | Prometheus metrics (RED: request count, latency histogram, error count by endpoint + kb_id) |

### BNUP Textbook Corpus (Issue #74)

BNUP (北师大版) K-12 math textbook corpus ingestion + retrieval, serving synthetic training-data generation for fusion-trainer. The corpus JSON (`beishi-math-g1-6.json`, grades → units → lessons → `knowledge_point_ids`) is POSTed by the operator; fusion-rag tags each lesson with `edition=bnup`, `grade`, `semester`, `unit`, `knowledge_point_ids`, `visual_type` and indexes it into a dedicated KB for hybrid retrieval.

| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/api/v1/bnup/ingest` | Ingest the BNUP corpus JSON body into the default KB (`bnup-default`, or `?kb_id=`). Idempotent — re-ingest overwrites. `verify_api_key` gated. |
| GET | `/api/v1/bnup/retrieve` | Retrieve lessons by `?knowledge_point=` and/or `?grade=` (`&limit=`, default 10). Hybrid vector + BM25 search filtered by KG metadata. |
| GET | `/api/v1/bnup/knowledge_graph` | Export the KG hierarchy (grades → units → lessons → knowledge_point_ids) + a flat `knowledge_points` index for the trainer. |
| GET | `/api/v1/bnup/misconceptions` | Retrieve common-error data by `?knowledge_point=` for DPO negative samples. Forward-compatible — returns empty until upstream [fusion-k12-teacher #21](https://github.com/dahai80/fusion-k12-teacher/issues/21) populates `common_misconceptions`; re-ingest then serves real data. |
| GET | `/api/v1/bnup/stats` | Corpus coverage per grade / semester / knowledge_point. |

Example:

```bash
# Ingest the corpus (operator POSTs the JSON from fusion-k12-teacher)
curl -X POST http://127.0.0.1:11436/api/v1/bnup/ingest \
  -H "Content-Type: application/json" \
  -d @beishi-math-g1-6.json

# Retrieve lessons for a knowledge point
curl "http://127.0.0.1:11436/api/v1/bnup/retrieve?knowledge_point=math-g1-na-01&limit=5"

# Export the knowledge graph
curl "http://127.0.0.1:11436/api/v1/bnup/knowledge_graph"
```

PDF/DOCX teacher manuals (菁优网/高思题集) can be ingested into the same BNUP KB via the standard `/kb/bases/bnup-default/docs/*` endpoints with `metadata={"edition":"bnup", ...}` tags.

### Evidence Metadata Contract (Issue #77)

Every retrieved chunk carries stable evidence anchors so consumers (e.g. [EnviPermit](https://github.com/dahai80/envipermit)) can bind extracted values back to their source location:

| Field | Where | Description |
|-------|-------|-------------|
| `chunk_id` | top level of each `/search` result + `/ask` source | Stable chunk identifier (`{doc_id}_{index}`), same as `id`. |
| `doc_id` | top level + chunk `metadata` | Document identifier returned at ingest time. Falls back to the chunk-id prefix for data ingested before v0.8.5. |
| `page` | top level + chunk `metadata` | 1-based source page for PDF chunks (per-page char spans tracked through parse → chunk). `0` = no page info. Non-PDF sources: pass `page` in the ingest `metadata` (see OCR below). |
| `chunk_index` | chunk `metadata` | 0-based chunk position within its document. |

PDF files ingested via `POST /kb/bases/{kb_id}/documents` get per-chunk `page` automatically — the parser records per-page char spans and the chunker maps each chunk's start offset back to its page.

**OCR pre-extracted text**: ingest one page per call via `POST /kb/bases/{kb_id}/documents/ingest` with the caller's own page number in the metadata:

```bash
curl -X POST http://127.0.0.1:11436/kb/bases/{kb_id}/documents/ingest \
  -H "Content-Type: application/json" \
  -d '{"content": "<OCR-extracted text for page 3>",
       "doc_name": "permit_2024_p3.txt",
       "metadata": {"source": "ocr", "page": 3},
       "contextualize": false}'
```

The full `metadata` dict is stamped on the document AND every chunk, so `metadata.source=ocr` and `metadata.page=3` round-trip through `/search` and are filterable via the `filter` parameter.

**Excel / tables**: there is no native table-chunk type. Convert spreadsheets to text pre-ingest (e.g. via fusion-office) and ingest the synthetic text chunks through `documents/ingest` with caller-defined metadata (`source=excel`, sheet name, etc.).

**Per-project KB pattern** (~100 concurrent projects, EnviPermit-style): one KB per project via `POST /kb/bases` (`kb_id=project_<id>`), ingest project documents, search with `POST /kb/bases/project_<id>/search`. For multi-tenant isolation enable `FUSION_RAG_REQUIRE_GATEWAY=1` (KB-level tenant scoping) or `FUSION_RAG_REQUIRE_IDENTITY=1` (JWT-verified tenant via fusion-identity); with `FUSION_RAG_STORE_BACKEND=qdrant`, tenants get separate vector collections. fusion-rag is single-process per stores dir (H3) — scale horizontally behind a stateless load balancer, never by running multiple processes against one `FUSION_RAG_STORES_DIR`.

**Horizontal scaling + auth (issue #102)**: the "stateless LB" contract covers the vector path, NOT the local API-key store. `auth.db` is a per-instance SQLite file (`~/.fusion-rag/auth.db`) — a key provisioned on node A does not exist on node B. Decision (option 1, per `architecture/fusion-db-conventions-1030.md` DB-5): horizontally-scaled deployments use **config-plane credentials only** — the shared `FUSION_RAG_API_KEY` env on every node, or gateway/identity-issued credentials (`FUSION_RAG_REQUIRE_GATEWAY=1` / `FUSION_RAG_REQUIRE_IDENTITY=1`). When either scaled mode is on, `POST /kb/auth/keys` refuses with 409 ("disabled in scaled mode") so no one can mint a single-node key; read-only key listing stays available per instance. Moving auth.db to Postgres (option 2) was rejected — it would add a PG dependency where the existing gateway/identity auth paths already cover the scaled case.

### GraphRAG (Issue #79)

Entity-relation graph retrieval, wired to HTTP (experimental). A build extracts entities + relations from a KB's chunks via the fusion-mlx LLM (one call per chunk, semaphore-bounded, capped) and stores them in a per-KB SQLite graph (`graph.db` under the KB's store dir); graph search expands entity matches over relation hops and resolves hit chunk ids back to text.

| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/kb/bases/{kb_id}/graph/build` | Extract entities/relations from the KB's chunks (LLM-backed). `max_chunks` (default 200, hard cap 1000) bounds the LLM spend. `verify_kb_action("write")` gated. |
| GET | `/kb/bases/{kb_id}/graph/stats` | Entity + relation counts for the KB. |
| POST | `/kb/bases/{kb_id}/graph/search` | Query → entity matches → relation-hop expansion → chunks. `max_hops` (default 2, cap 4). Returns evidence-anchored rows (`chunk_id`/`doc_id`/`page`). |

```bash
curl -X POST http://127.0.0.1:11436/kb/bases/{kb_id}/graph/build -H "Content-Type: application/json" -d '{"max_chunks": 200}'
curl -X POST http://127.0.0.1:11436/kb/bases/{kb_id}/graph/search -H "Content-Type: application/json" -d '{"query": "fusion"}'
```

### Recall Benchmark (Issue #79)

`scripts/recall_benchmark.py` measures retrieval quality on a deterministic ≥500-chunk corpus (markdown + Python mix through the real chunker) with 55 ground-truth queries — recall@5/@10 + MRR@10, and the ablation BM25-only vs vector-only vs RRF fusion:

```bash
python scripts/recall_benchmark.py --mode mock   # wiring / no-regression evidence (labeled mode=mock)
FUSION_MLX_URL=http://127.0.0.1:11434/v1 FUSION_MLX_API_KEY=<key> \
  python scripts/recall_benchmark.py --mode real --output docs/benchmarks/recall-$(date +%F).json
```

Mock mode uses deterministic hashed-bag embeddings and is wiring-regression evidence only — never quality evidence. Real mode hits the live fusion-mlx embeddings endpoint with the cache disabled. CI (`tests/test_recall_benchmark.py`) asserts the pipeline runs and RRF fusion never drops recall@10 below the best single mode. Recorded results live in `docs/benchmarks/` — see [docs/benchmarks/README.md](docs/benchmarks/README.md) for the ablation decision.

### Similar-Chunk Search (Issue #83)

`POST /kb/bases/{kb_id}/similar` finds near-duplicate / version-drift chunks. Input is either `text` (embedded with the same service-wide model as ingest, so scores are directly comparable) or a raw `embedding` vector:

```bash
curl -X POST http://127.0.0.1:11436/kb/bases/{kb_id}/similar \
  -H "Content-Type: application/json" \
  -d '{"text": "排污许可证 编号 X123", "top_k": 10, "threshold": 0.85, "exclude_doc_id": "<doc_id>"}'
```

Parameters: `top_k` (1-100, default 10), `threshold` (default: the KB's `similarity_threshold`), `exclude_doc_id` (drop a doc's own chunks — dedup-within-corpus), `snippet_chars` (1-2000, default 200), `status_filter` (`active`/`deprecated`/`all`). Returns `matches[]` with `doc_id`/`doc_name`/`chunk_id`/`score`/`snippet`/`page`/`status` + `count` + `threshold`.

### EPUB + Scanned-PDF OCR (Issues #84/#85)

`DocumentParser` now reads EPUB (each XHTML chapter becomes a `# <name>` headed section via ebooklib). Scanned PDFs (pages present, zero extractable text) are no longer silently ingested as empty docs: they are handed to the [fusion-drawing-parser](https://github.com/dahai80/fusion-drawing-parser) OCR service over HTTP and fail visibly when none is configured:

```bash
export FUSION_RAG_OCR_URL=http://127.0.0.1:11491     # drawing-parser base URL
export FUSION_RAG_OCR_API_KEY=<jwt>                  # Bearer token (may be empty)
export FUSION_RAG_OCR_TENANT_ID=<tenant>             # X-Tenant-Id fallback (single-tenant /
                                                      # background ingestion; issue #88)
```

OCR chunks bypass the chunker + contextualizer (they are already slice-sized) and carry per-chunk evidence metadata (issue #85): `ocr_confidence`, `low_confidence` (vs `FUSION_RAG_LOW_OCR_CONFIDENCE`, default 0.85), and `slice_ref` (`task_id`/`slice_id`/`url` linking back to the source slice image). Every `/search` result, `/ask` source, and `/ask` citation from an OCR doc carries these fields, so consumers can flag shaky OCR text and show the original image.

### Document Active/Deprecated Status (Issue #86)

`PATCH /kb/bases/{kb_id}/documents/{doc_id}` flips a document's lifecycle status without re-indexing:

```bash
curl -X PATCH http://127.0.0.1:11436/kb/bases/{kb_id}/documents/{doc_id} \
  -H "Content-Type: application/json" \
  -d '{"status": "deprecated", "superseded_by": "regulations_v2.pdf"}'
```

Status is resolved at search time from the documents table (the PATCH flips it there), so a deprecated doc vanishes from default `/search`, `/ask`, and `/similar` immediately. `status_filter=deprecated` retrieves only deprecated docs (audit/traceability); `status_filter=all` keeps everything — deprecated chunks then appear marked `[已作废版本]` in the LLM context so the model knows they are stale. `/ask` additionally returns per-chunk `citations[]` (every cited chunk, not deduped) and `has_low_confidence`.

---

## Architecture

```
┌───────────────────────────────────────────────────────────────────┐
│                   Fusion-RAG HTTP API (FastAPI)                   │
│  routes_kb.py  routes_docs.py  routes_search.py  routes_admin.py │
└───────────────────────────┬───────────────────────────────────────┘
                            │
┌───────────────────────────▼───────────────────────────────────────┐
│                      RAG Core Engine                              │
│  ┌────────────────┐  ┌────────────────┐  ┌──────────────────┐   │
│  │ DocumentParser │  │    Chunker     │  │  KnowledgeBase    │   │
│  │ PDF/DOCX/MD/   │  │ semantic/fixed │  │  Manager (CRUD)   │   │
│  │ TXT/HTML/Code  │  │ /code/AST      │  │  + isolation      │   │
│  └────────┬───────┘  └───────┬────────┘  └────────┬─────────┘   │
│  ┌────────┴───────┐  ┌───────┴────────┐  ┌────────┴─────────┐   │
│  │ VersionManager │  │ SearchTemplate │  │  AuditLogger     │   │
│  │ Snapshots/Roll │  │  Manager       │  │  PermissionMgr   │   │
│  └────────────────┘  └────────────────┘  └──────────────────┘   │
└───────────┼──────────────────┼────────────────────┼──────────────┘
            │                  │                    │
┌───────────▼──────────────────▼────────────────────▼──────────────┐
│                     Storage Layer                                 │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │   StoreBackend (ABC)                                       │  │
│  │   ├── LocalBackend (LanceDB + BM25)                        │  │
│  │   └── RemoteBackend (stub)                                 │  │
│  └────────────────────────────────────────────────────────────┘  │
│  ┌────────────────────────────┐  ┌────────────────────────────┐  │
│  │   MetadataStore (SQLite)   │  │  EmbeddingCache (SQLite)   │  │
│  └────────────────────────────┘  └────────────────────────────┘  │
└───────────────────────────┬──────────────────────────────────────┘
                            │
┌───────────────────────────▼──────────────────────────────────────┐
│                    Embedding Client                               │
│  Primary: fusion-mlx /v1/embeddings → Fallback: Cloud API       │
└───────────────────────────┬──────────────────────────────────────┘
                            │ HTTP
┌───────────────────────────▼──────────────────────────────────────┐
│  fusion-mlx (/v1/embeddings, /v1/chat/completions)              │
│  Apple Silicon MLX Runtime (Metal GPU)                           │
└──────────────────────────────────────────────────────────────────┘
```

### LLM Client Infrastructure

All LLM chat calls (`/v1/chat/completions`) go through **fusion-core**'s HTTP client (`fusion_core.http_client`):

- `get_async_client(base_url, timeout)` — shared connection pool (LRU, keyed by loop + base_url), reused across modules
- `with_retry(fn, retries=2)` — automatic retry on 429/5xx and transient connection errors (backoff + jitter)
- Auth headers passed **per request**, not baked into the pooled client (avoids header leak between instances sharing a base_url)
- **D-H3 guard**: success path checks for empty/whitespace content and degrades explicitly (logged error/fallback) rather than returning empty content as a valid result
- **Pool-key timeout caveat (A7)**: the pool key is `loop + base_url` only — the per-call `timeout` is not part of the key, so the first caller's timeout wins for that base_url on a given loop. fusion-rag mitigates by unifying all LLM `_call` timeouts to a shared 60s (`DocumentChain._CALL_TIMEOUT`), making first-call-wins benign. Root cause tracked upstream: [fusion-core#16](https://github.com/dahai80/fusion-core/issues/16)

Modules migrated to fusion-core: `contextualizer`, `query_rewriter`, `reranker`, `rag_chain` (MultiTurnRAG + DocumentChain), `graph_rag`, `streaming` (MetadataExtractor; SSEStreamer keeps raw `httpx.stream`), `evaluator`, `api/routes._generate_answer`.

### Key Modules

| Module | File | Description |
|--------|------|-------------|
| **Knowledge Base** | `engine/knowledge_base.py` | KB CRUD, config, persistence |
| **Document Parser** | `engine/document.py` | PDF, DOCX, EPUB, MD, TXT, HTML, code, scanned PDF via OCR service |
| **Chunker** | `engine/chunker.py` | Semantic, fixed, code, AST chunking |
| **AST Chunker** | `engine/ast_chunker.py` | Python AST-aware code chunking |
| **BM25 Index** | `engine/bm25_index.py` | Okapi BM25 with jieba Chinese tokenization |
| **Contextualizer** | `engine/contextualizer.py` | Anthropic Contextual Retrieval |
| **Reranker** | `engine/reranker.py` | Batch LLM reranking + HybridSearch (alpha/RRF) |
| **Query Rewriter** | `engine/query_rewriter.py` | HyDE, query expansion, condensation |
| **GraphRAG** | `engine/graph_rag.py` | Entity extraction and graph-based retrieval |
| **Evaluator** | `engine/evaluator.py` | RAG quality evaluation (faithfulness/relevance) |
| **Embedding Cache** | `engine/embedding_cache.py` | SQLite-backed embedding vector cache |
| **Version Manager** | `engine/version_manager.py` | KB snapshot and rollback |
| **Incremental Sync** | `engine/incremental_sync.py` | MD5+mtime change detection |
| **Search Templates** | `engine/search_template.py` | Preset + custom search strategies |
| **Audit Logger** | `engine/audit_logger.py` | Search audit trail with export |
| **Trajectory Writer** | `engine/trajectory_writer.py` | D1 retrieval trajectory sink (JSONL to `~/.fusion/trajectories/rag/`) |
| **Bench Runner** | `engine/bench.py` | Search latency benchmarking |
| **Permissions** | `permissions/acl.py` | Role-based ACL with path inheritance |
| **Git Loader** | `connectors/git_loader.py` | Git repo clone + index |
| **Store Backend** | `store/store_backend.py` | Abstract storage backend |
| **Local Backend** | `store/local_backend.py` | LanceDB + BM25 implementation. Vector search is a brute-force scan (no ANN index) — exact results, fine up to ~100k chunks per KB; for larger corpora use the `fusion-store` (HNSW) or `qdrant` backends. |
| **Remote Backend** | `store/remote_backend.py` | HTTP client to a remote fusion-rag node's `/store/*` vector surface |
| **Qdrant Backend** | `store/qdrant_backend.py` | Qdrant vector DB with per-tenant collection isolation (#66) |
| **Store Server** | `api/routes_store.py` | `/kb/bases/{kb_id}/store/*` M2M vector store endpoints (RemoteBackend server half) |
| **Auth** | `api/auth.py` | API key authentication |
| **MCP Server** | `api/mcp_server.py` | Model Context Protocol for Claude/Cursor |
| **Embedding** | `embed/client.py` | MLX Embedding + cloud fallback |
| **Vector Store** | `store/vector_store.py` | StoreBackend wrapper |
| **Metadata Store** | `store/metadata_store.py` | SQLite document/chunk tracking |
| **API Routes** | `api/routes.py` | Shared helpers + sub-router mount |
| **Server** | `api/server.py` | FastAPI server |

---

## Configuration

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `FUSION_RAG_PORT` | 11436 | Server port |
| `FUSION_RAG_HOST` | 127.0.0.1 | Listen address |
| `FUSION_MLX_URL` | http://127.0.0.1:11432/v1 | fusion-mlx URL |
| `FUSION_MLX_API_KEY` | (empty) | MLX gateway API key (auto-detected from `~/.fusion-mlx/settings.json` if unset) |
| `FUSION_RAG_EMBED` | BGE-M3 | Embedding model |
| `FUSION_RAG_API_KEY` | (empty) | API key auth (disabled if empty) |
| `FUSION_RAG_AUTH_BACKEND` | apikey | Auth backend: `apikey` or `none` |
| `FUSION_RAG_INGEST_ROOTS` | (empty) | Colon-separated allowed roots for file ingest/scan/watch. Empty = no confinement (local-first, unrestricted). Set to confine reads and block path traversal / LFI |
| `FUSION_RAG_SYSTEM_PROMPT` | (built-in) | Custom system prompt for RAG answer generation |
| `FUSION_RAG_FALLBACK_URL` | (empty) | Cloud embedding fallback URL |
| `FUSION_RAG_FALLBACK_API_KEY` | (empty) | Cloud fallback API key |
| `FUSION_RAG_STORE_BACKEND` | local | Vector store backend: `local` (LanceDB), `fusion-store` (HNSW via the in-tree fusion-store PyO3 binding; `pip install -e ../fusion-store`, not on PyPI), `qdrant` (Qdrant vector DB with per-tenant collection isolation; `pip install 'fusion-rag[qdrant]'`, see `QDRANT_*`), or `remote` (HTTP client to another fusion-rag node — see `FUSION_RAG_REMOTE_*`) |
| `FUSION_RAG_REMOTE_ENDPOINT` | (empty) | Remote node base URL for the `remote` backend (e.g. `http://node-b:11436`). Required when backend is `remote`. |
| `FUSION_RAG_REMOTE_API_KEY` | (empty) | `X-API-Key` sent to the remote node (empty = no auth). |
| `FUSION_RAG_REMOTE_KB_ID` | (derived) | KB id on the remote node; defaults to this node's kb_id. |
| `FUSION_RAG_REMOTE_TIMEOUT` | 30 | Per-request timeout (seconds) for the `remote` backend. |
| `QDRANT_URL` | :memory: | Qdrant endpoint for the `qdrant` backend. `:memory:` = in-process local mode (no server, for dev/test); otherwise the remote Qdrant URL (e.g. `http://qdrant:6333`). |
| `QDRANT_API_KEY` | (empty) | API key for the remote Qdrant server (ignored in `:memory:` mode). |
| `QDRANT_COLLECTION_PREFIX` | tenant_id_ | Prefix for per-tenant collection names. A request under tenant `T` lands in `{prefix}{T}`; no tenant (single-tenant dev) uses the shared `fusion_rag_kb_{kb_id}` collection. |
| `FUSION_TRAJECTORY_DIR` | ~/.fusion/trajectories/rag | RAG retrieval trajectory output dir (D1 轨迹飞轮) |
| `FUSION_RAG_SCAN_MAX_FILES` | 1000 | Max files a single `scan_directory` ingests (D4) |
| `FUSION_RAG_EMBED_CACHE_TTL` | 604800 | Embedding cache entry TTL in seconds (D4, default 7d) |
| `FUSION_RAG_EMBED_CACHE_MAX_ENTRIES` | 100000 | Max rows in the embedding cache before LRU eviction (D4) |
| `FUSION_RAG_TOKEN_BUDGET` | 8192 | Multi-turn RAG token budget per turn (D4) |
| `FUSION_RAG_MAX_HISTORY_TURNS` | 10 | Max conversation turns kept in RAG history (D4) |
| `FUSION_RAG_FETCH_K_MULTIPLIER` | 4 | Over-fetch factor for filtered search (D4) |
| `FUSION_RAG_RERANK_BACKEND` | llm | Rerank stage: `llm` (legacy LLM prompt-scoring `Reranker`) or `cross_encoder` (real cross-encoder via fusion-mlx `POST /v1/rerank`, e.g. `bge-reranker-v2-m3`). Issue #70. |
| `FUSION_RAG_RERANK_MODEL` | (empty) | Cross-encoder model name for the `cross_encoder` backend (e.g. `bge-reranker-v2-m3`). When set, `/search` and `/ask` default `rerank=true` (rerank ON when a model is available). Empty = rerank off unless the caller passes `rerank=true`. Issue #70. |
| `FUSION_RAG_RERANK_TOP_N` | 20 | Candidate pool size fed to the reranker before truncating to `top_k` — recall lift without changing the returned count (issue #70). |
| `FUSION_RAG_WATCH_CAP` | 16 | Max concurrent directory watches per process (R3) |
| `FUSION_RAG_TRAJECTORY_MAX_MB` | 100 | Max trajectory file size in MB before rotation (R6) |
| `FUSION_RAG_TRAJECTORY_KEEP` | 5 | Rotated trajectory files to keep (R6) |
| `FUSION_RAG_AUDIT_RETENTION_DAYS` | 30 | Audit log retention days (0 = keep forever) (R6) |
| `FUSION_RAG_STORES_DIR` | ~/.fusion-rag/stores | Per-KB store root dir (R3 watch registry) |
| `FUSION_RAG_LOG_DIR` | `./logs` | Log file dir (RotatingFileHandler, 10MB x 5 files) |
| `FUSION_RAG_LOG_FORMAT` | text | Log line format: `text` (readable) or `json` (one JSON object/line, for ELK/Loki) |
| `FUSION_RAG_SKIP_STARTUP_PROBE` | (empty) | Set `1` to skip the boot-time fusion-mlx reachability probe (MLX comes up later) |
| `FUSION_REGISTRY` | ~/fusion/architecture/port-registry.yaml | Port registry checked by `start.sh` before serving (issue #98). A port missing from the registry refuses to start; `FUSION_SKIP_REGISTRY_CHECK=1` bypasses; a missing registry file warns and continues (standalone deploy). |
| `FUSION_RAG_REQUIRE_GATEWAY` | (empty) | Set `1` to enforce multi-tenant isolation (#61): `/kb/*` requests require the `X-Fusion-Route: gateway-decision` header (403 without it), and KB access is scoped to the `X-Fusion-Tenant` header. Default off — single-tenant deployments are unaffected. |
| `FUSION_RAG_REQUIRE_IDENTITY` | (empty) | Set `1` to enable authoritative tenant resolution via fusion-identity (#68): `/kb/*` requests require a valid `Authorization: Bearer <JWT>` verified by the identity service, and the tenant is taken from the JWT `tid` claim. `X-Fusion-Tenant` is demoted to defense-in-depth (must match the JWT `tid` if present). Default off. |
| `FUSION_IDENTITY_URL` | http://127.0.0.1:11470 | fusion-identity base URL (HTTP service, port 11470). Used only when `FUSION_RAG_REQUIRE_IDENTITY=1`. |
| `FUSION_IDENTITY_SERVICE_TOKEN` | (empty) | Service token sent as `Authorization: Bearer` to fusion-identity `/api/v1/auth/verify`. Required when `FUSION_RAG_REQUIRE_IDENTITY=1` (fail-closed if unset). |
| `FUSION_RAG_OCR_URL` | (empty) | Base URL of the OCR service (fusion-drawing-parser, e.g. `http://127.0.0.1:11491`) for scanned-PDF ingestion. Empty = scanned PDFs fail visibly with "no OCR service configured" (issue #84). |
| `FUSION_RAG_OCR_API_KEY` | (empty) | Bearer token sent to the OCR service (issue #84). |
| `FUSION_RAG_OCR_TIMEOUT` | 300 | Total poll budget in seconds for one OCR task (issue #84). |
| `FUSION_RAG_OCR_TENANT_ID` | (empty) | `X-Tenant-Id` fallback for OCR calls when no request tenant is bound (background watch-loop ingestion, single-tenant deployments). The OCR service requires the header unconditionally (issue #88); during a request the calling tenant is passed through automatically. |
| `FUSION_RAG_LOW_OCR_CONFIDENCE` | 0.85 | OCR chunk confidence below this flags `low_confidence: true` on search results and `/ask` citations (issue #85). |
| `FUSION_RAG_CITATION_AUDIT` | 1 | `1` = write a queryable citation audit trail (issue #117, default ON since R2/#124): query chain + retrieved chunks per retrieval call to `citation_audit.db`, reconstructable via `GET /kb/bases/{kb_id}/citations/traces`. `0` = disable. |

**Multi-tenant isolation (#61).** When `FUSION_RAG_REQUIRE_GATEWAY=1`, fusion-rag enforces the backend half of fusion-gateway's tenant scoping: every `/kb/*` request must transit the gateway (proven by the `X-Fusion-Route: gateway-decision` header) and carries an authoritative `X-Fusion-Tenant`. KBs are stamped with their tenant at creation and scoped on list/get — a tenant-A caller never sees tenant-B's KBs. The client-supplied `X-Space-Id` is a non-authoritative passthrough, ignored for scoping. `/health`, `/ready`, `/metrics`, `/mcp`, and auth routes stay open so the service remains observable when the gateway is down. Default off; no migration needed (existing KBs have `tenant=null`).

**Authoritative tenant resolution (#68).** The #61 path trusts the `X-Fusion-Tenant` header, which is forgeable by any direct-port caller. When `FUSION_RAG_REQUIRE_IDENTITY=1`, fusion-rag resolves the tenant **authoritatively** from a fusion-identity JWT instead: every `/kb/*` request must carry `Authorization: Bearer <JWT>`, which fusion-rag verifies via fusion-identity's `POST /api/v1/auth/verify` (service-token-gated; revocation and tenant-status enforced by the identity service). The JWT `tid` claim becomes the authoritative tenant and feeds the same `_request_tenant` contextvar, so KB scoping (#61) and per-tenant collection isolation (#66) keep working unchanged. `X-Fusion-Tenant` is demoted to defense-in-depth: if present it must equal the JWT `tid` (mismatch → 401, a blocked forgery attempt). Requests with no/invalid/revoked JWT are 401; an unreachable identity service is fail-closed (deny, not fallback to the header). fusion-identity is an HTTP service dependency (port 11470), not a Python import — fusion-rag talks to it over plain HTTP. Default off; single-tenant local-first dev is unaffected.

### Using start.sh

```bash
./start.sh start      # Start the server
./start.sh stop       # Stop the server
./start.sh restart    # Restart the server
./start.sh status     # Check server status
```

---

## Deployment

> **Enterprise / commercial deployment**: see [`DEPLOYMENT.md`](DEPLOYMENT.md) for hard constraints (single-process, single-embedding-model, external MLX dependency), the cold-start runbook, the security checklist, the performance baseline, and the commercial-release gap tracker.

fusion-rag is a single-process service (see H3 in CLAUDE.md: do NOT run `--workers N` or multiple instances against one shared `stores` dir). Three supported deploy shapes:

### Docker / docker-compose

```bash
# Build from the repo root (issue #55): the canonical Dockerfile lives at the
# repo root so the monorepo compose overlay (`build: ../fusion-rag`) resolves.
docker build -t fusion-rag:0.8.0rc4 .
docker run -p 11436:11436 \
    -e FUSION_MLX_URL=http://host.docker.internal:11432/v1 \
    -v fusion-rag-stores:/root/.fusion-rag/stores fusion-rag:0.8.0rc4
curl http://127.0.0.1:11436/health   # → 200

# Or via the compose file (builds from the root Dockerfile):
docker compose -f deploy/docker-compose.yml up -d
docker compose -f deploy/docker-compose.yml logs -f
```

The image uses `python:3.12-slim` (arm64), the Aliyun pip mirror, and binds `0.0.0.0:11436`. fusion-mlx stays on the host (Apple Silicon metal), reached via `FUSION_MLX_URL=http://host.docker.internal:11432/v1`. fusion-rag has **no fusion-memory dependency** — no UDS socket mount is needed. The compose file mounts a named volume for `~/.fusion-rag/stores` (KB data survives container recreation) and sets `FUSION_RAG_LOG_FORMAT=json` so `docker logs` emits parseable JSON. See the root `Dockerfile` + `deploy/docker-compose.yml`.

### systemd (bare-metal / VM)

```bash
sudo cp deploy/fusion-rag.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now fusion-rag
journalctl -u fusion-rag -f
```

The unit drains in-flight requests for 40s on stop (`TimeoutStopSec=40`, matching the 30s graceful shutdown) before SIGKILL. Per-instance env overrides live in `/etc/fusion-rag/env` (`EnvironmentFile=-`). See `deploy/fusion-rag.service`.

### start.sh (dev / single-user)

```bash
./start.sh start    # nohup, logs to logs/stdout.log (bootstrap) + logs/fusion-rag.log (rotated)
./start.sh stop     # SIGTERM, drains up to 35s, then SIGKILL
```

### Health probes

| Endpoint | Purpose | Use for |
|----------|---------|---------|
| `GET /health` | Liveness — process up, event loop turns | restart policy (`restart: on-failure`, livenessProbe) |
| `GET /ready` | Readiness — embedding backend reachable + store writable | routing / load-balancer (readinessProbe); NOT restart |

`/health` never checks downstream deps (a transient MLX hiccup must not evict the pod). `/ready` returns 503 with a `checks` body when a dep is down.

### Snapshot-safe stores backup (O-P2-1)

The per-KB stores dir (`~/.fusion-rag/stores/{kb}/`) mixes SQLite WAL stores (metadata.db, bm25_index.db, versions.db, …) with LanceDB `vectors/`. A raw `tar`/`rsync` of the dir mid-write captures an inconsistent `.db` + a stale `-wal` that a restore would replay or drop. **Checkpoint before backing up**:

```bash
# Fold every WAL sidecar into its .db + compact LanceDB, THEN snapshot.
curl -X POST -H "Authorization: Bearer $FUSION_RAG_API_KEY" \
  http://127.0.0.1:11436/kb/bases/{kb_id}/checkpoint
tar -czf kb-snapshot.tgz -C ~/.fusion-rag/stores {kb_id}
```

`POST /kb/bases/{kb_id}/checkpoint` runs `PRAGMA wal_checkpoint(TRUNCATE)` on every SQLite store for the KB and `table.optimize()` on the LanceDB table. Run it, then snapshot.

### Structured logging + request-id (O-P1-2 / O-P2-2)

- Logs rotate: `logs/fusion-rag.log` at 10MB x 5 files (RotatingFileHandler), so a long run no longer fills the disk. `FUSION_RAG_LOG_DIR` overrides the path.
- `FUSION_RAG_LOG_FORMAT=json` emits one JSON object per line (`{ts, level, logger, msg, request_id, extra}`) for ELK/Loki; `text` (default) stays readable for dev.
- Every HTTP request gets an `X-Request-ID` (incoming header honored, else generated), echoed on the response. All log lines for a request carry the same `request_id`, so a call is traceable end-to-end across app logs + the aggregator.

---

## Data Layer Dependencies

Governing conventions: `architecture/fusion-db-conventions-1030.md` (single shared Qdrant instance per node; no alternative vector stores — DB-8).

### When Qdrant is required

- **Default (`FUSION_RAG_STORE_BACKEND=local`)** — no external database. Vectors live in embedded LanceDB + SQLite under `~/.fusion-rag/stores/{kb_id}/`.
- **Qdrant backend (`FUSION_RAG_STORE_BACKEND=qdrant`)** — required for per-tenant collection isolation (issue #66) and the multi-tenant deployments that rely on it. Install the client with `pip install 'fusion-rag[qdrant]'`.

### Running Qdrant locally

The canonical Qdrant instance is **`http://localhost:6333` (HTTP)**; the gRPC port `6334` is reserved by the conventions doc but **not used by fusion-rag** — the backend connects over HTTP only. Qdrant has no native Homebrew formula, so a local run needs a container runtime (OrbStack or Docker):

```bash
docker run -d --name qdrant -p 6333:6333 -p 6334:6334 \
  -v $(pwd)/qdrant_storage:/qdrant/storage qdrant/qdrant:v1.12.4
```

Production provides the same instance (same ports, same version `v1.12.4`) via `fusion-supervisor`'s `docker-compose.sidecar.yml` — do not spin up a second Qdrant.

### Connection + failure behavior

Connection is configured through `QDRANT_URL` (per DB-3 of the conventions doc):

```bash
export FUSION_RAG_STORE_BACKEND=qdrant
export QDRANT_URL=http://localhost:6333
```

The default `QDRANT_URL=:memory:` runs an in-process ephemeral Qdrant — dev/test only, data does not survive restart.

Expected behavior when Qdrant is unreachable: **ingest fails loudly** — upsert/connection errors propagate to a visible 4xx/5xx, never a silently successful ingest. Known caveat (tracked as a follow-up in issue #100): the search path currently treats any `get_collection` failure as "collection does not exist" and returns empty results, so an unreachable Qdrant can surface as an empty search rather than an error. Check `logs/fusion-rag.log` for `QdrantBackend` errors when results look unexpectedly empty.

---

## Development

```bash
# Install dev dependencies
pip install -e ".[test]"

# Run all tests
pytest tests/

# Run with coverage
pytest tests/ --cov=fusion_rag --cov-report=term-missing

# Lint
ruff check fusion_rag/
```

---

## What's New in v0.8.34

- **Rerank pool A/B — 20 saturates (R3, issue #125)** — `recall_bench.run_rerank_pool_ablation` + `scripts/rerank_pool_ablation.py` (RRF candidates per pool → cross-encoder → top-10, R12 metrics + rerank latency p50/p95). Live bge-reranker-v2-m3 run recorded (`docs/benchmarks/rerank-pool-ablation.json`): metrics identical at pool 20/50/100/150, latency 114 ms → 1298 ms p50 — `rerank_top_n=20` default stays, re-run on a real labeled set before concluding for production traffic.

## What's New in v0.8.33

- **PyPI packaging** — `pyproject.toml` PyPI-ready (readme/classifiers/urls, `fusion-core>=0.12.1` resolves from PyPI); `.github/workflows/pypi.yml` publishes via Trusted Publishing (OIDC) on `v*` tags; sdist + wheel verified (console assets included). `pip install fusion-rag` once the first tag publishes.
- **MCP `kb_id` optional (#136)** — `kb_search` / `kb_ask` fall back to the single existing KB (tenant-scoped) or the most recently created; `tools/list` documents the default; the ACL gate resolves the same id so the default path cannot skip it. Typed `-32602` outside app state.
- **`start.sh` venv contract (#137)** — already-activated `VIRTUAL_ENV` first, project-adjacent `.venv` second, monorepo fallback last; fails loudly when `fusion_rag` is not importable instead of starting uvicorn into a crash.

## What's New in v0.8.32

- **Eval completeness (R12, issue #134)** — NDCG@K + MAP@K added alongside recall/MRR in `recall_bench` (all three modes) and the offline eval harness; `scripts/eval_baseline.json` re-baselined with `ndcg@10` 0.9469 / `map@10` 0.9212, CI gate fails on regression. Retune checklist (`docs/retrieval-retune-checklist.md`) + printed on gate failure: any embedding-model / chunking / fusion-parameter change requires a re-run against the fixed labeled sample. `POST /search` accepts `source_weights` (per-`doc_type` trust multipliers, [0,10], invalid entries ignored with a warning) — deterministic post-fusion re-rank applied before the rerank stage; scored rows carry `source_weight` for audit.

## What's New in v0.8.31

- **Citation auditing ON by default (R2, issue #124)** — `FUSION_RAG_CITATION_AUDIT` now defaults to `1` (set `0` to disable). Measured overhead: ~0.11 ms per retrieval write (10 chunks + 1 rewrite, sqlite WAL batch) — noise against search latency. The evaluator (`RAGEvaluator`) gained a citation-accuracy judge (`CITATION_ACCURACY_PROMPT`: what fraction of the answer's claims are supported by the CITED sources — Anthropic/Glean production rubric) emitted per-case as `citation_accuracy` + aggregate `avg_citation_accuracy` (averaged only over cases that carry citations), and `citation_accuracy_from_trace()` scores an /ask response's citations against a citation-audit trace (chunk-id overlap).

## What's New in v0.8.30

- **RRF fusion consolidation (R1, issue #123)** — `HybridSearch(method="rrf")` is the single supported fusion path; the weighted-score `FusionRetriever` is deprecated (DeprecationWarning, experimental only). Recall ablation re-run on live BGE-M3 recorded as `docs/benchmarks/recall-2026-11-14.json` (RRF 0.7939/0.9212/1.0000, rrf >= max(single) holds).

## What's New in v0.8.29

- **Office format ingest: PPTX + XLSX** — server-side parsers (`python-pptx`, `openpyxl`): PPTX renders one `## Slide N` section per slide (text frames in reading order + speaker notes; image-only slides skipped), XLSX renders one `## Sheet: name` section per sheet with ` | `-joined cell rows (cached formula values, empty trailing cells trimmed, empty sheets skipped). Both types added to the #90 inline binary ingest set and the console file picker (`accept` updated). Legacy binary formats (.doc/.xls/.ppt) remain unsupported — they need a converter, not a parser; re-save as docx/xlsx/pptx. New deps: `python-pptx>=1.0.0`, `openpyxl>=3.1.0`.

## What's New in v0.8.28

- **UI: file-picker ingest in Docs / Search tab** — "选择文件入库 / Pick a file" button: pick a `txt/md/markdown/html/htm/csv` (text) or `pdf/docx/epub` (binary, #90 base64 inline path → full parse pipeline incl. scanned-PDF OCR) file; doc name = filename. The paste-text path stays as a secondary row.

## What's New in v0.8.27

- **UI: chat input doubled** — the 问答 / Chat input is now a 2-row textarea (Enter sends, Shift+Enter newline, IME-safe), send button bottom-aligned. Console rebuilt.

## What's New in v0.8.26

- **UI: chat tab height doubled** — the 问答 / Chat message area now scrolls within `max-h-[48rem]` (was `max-h-96`), min height raised to match. Console rebuilt.

## What's New in v0.8.25

- **Console chat tab** — new "问答 / Chat" tab in the KB detail view: a RAG Q&A box calling `POST /kb/bases/{kb_id}/ask`. Answers render with per-chunk citation chips (doc name, page, score; low-OCR-confidence citations flagged ⚠ and a per-answer warning). Plain text (no markdown lib) — it's an evidence tester on top of the /ask surface. Console rebuilt (v0.8.25, bundle +1.2 kB gzip).

## What's New in v0.8.24

- **Citation audit trail (issue #117, feature 1)** — every retrieval call (`/search`, `/ask`, `/similar`) can now write a queryable audit trail to `citation_audit.db`: one `rag_queries` row per query version (the original plus each rewrite step chained via `parent_id`/`depth`) and `rag_citations` rows for the retrieved chunks linked to the FINAL query version with per-chunk similarity + rank. An answer's citations are reconstructable from the DB alone: `GET /kb/bases/{kb_id}/citations/traces` (list) + `GET /kb/bases/{kb_id}/citations/traces/{query_id}` (full chain + evidence). Opt-in via `FUSION_RAG_CITATION_AUDIT=1` (default OFF — zero behavior change); write failures log and never affect the response.
- **Schema'd RAG eval set + offline eval gate (issue #117, feature 2)** — `engine/eval_set.py`: `eval_cases` (query / gold chunks, seeded deterministically from the recall_bench 55-query corpus, idempotent) + `eval_runs` (metric rows per run). `run_offline_eval()` is BM25-only — no embedding model, no LLM, no network — computing recall@5 / recall@10 / mrr@10 (baseline: 0.7939 / 0.9212 / 1.0000). `scripts/eval_gate.py` compares against `scripts/eval_baseline.json` and FAILS the build when any metric drops beyond tolerance (default 0.02) while always reporting deltas; new `.github/workflows/eval.yml` CI job runs the gate on every PR touching retrieval code.

## What's New in v0.8.23

- **Docs: frontmatter fix (PR #116)** — `owner: @dahai` quoted as `owner: "@dahai"` in all 15 frontmatter-bearing docs (bare `@` handle is an invalid YAML mapping value in strict parsers). No runtime change.

## What's New in v0.8.22

- **Docs: fusion docs-system v1.1 contract (PR #115)** — new `docs/` tree (product page, architecture, API reference REST+MCP, config reference, get-started, backup-and-restore + multi-tenant-isolation guides, troubleshooting, benchmarks) following the platform × service contract; `ci/validate_docs.py` contract validator wired as a CI gate (`.github/workflows/docs.yml`) — sibling-link integrity + required sections enforced on every PR. Version bump only; no runtime change.

## What's New in v0.8.21

- **Web surface gap closed (issue #113)** — the console covered 2 views over ~70 REST routes; the previously API-only capabilities now have UI. KB detail gained tabs: **Graph** (stats tiles, LLM entity-extraction build with explicit busy state, graph search with evidence hits), **Versions** (snapshot list/create, rollback + delete via destructive-action ConfirmDialog), **Templates** (preset + custom search templates, create/delete; builtin templates hide the doomed delete action). New `#/admin` page (linked from the console header) with: per-KB **ACL rules** (list/add/delete + live permission check), **audit log** (recent records + JSON/CSV export links), and **API-key management** (list/create/delete; a minted key is shown once). Server-side: the auth router (`/kb/auth/keys`) is now also mounted under the console subpath `/shell/fusion-rag/` — it is registered separately from the `/kb` router at root level, so its endpoints 404'd behind the gateway pass-through proxy. Console calls ride page-relative paths (#109 contract preserved); bundle checks extended. 472 tests passing; ruff clean.

## What's New in v0.8.20

- **Console rebuilt to the union web UI spec (user report: bilingual/styling non-compliant + dead delete button)** — the single-file vanilla console is replaced by a proper React 18 + TypeScript + Tailwind app (`web/`, Vite, Bun build) following the fusion-web-shell unified rules: design tokens via `@fusion/web-ui` (token-only colors, `rounded-card`/`rounded-md`, dark + light via the Shell theme bridge), bilingual copy via the SDK language bridge (`t()`, follows the Shell toggle), four mandatory states (loading skeleton / empty / error+retry / populated) on every data view. Delete-KB fix: the old page used native `confirm()`, which the shell sandboxed iframe blocks silently — the button appeared dead. Deletion now goes through `ConfirmDialog` (danger + type-the-KB-id guard) and feedback via the SDK toast bridge. The API-key localStorage input is removed (spec S8 — the same-origin shell session carries auth; no local token storage). Build output is committed under `fusion_rag/console/` (shipped as package data, served at `/shell/fusion-rag/` exactly as before); dev: `cd web && bun install && bun run dev`. 4 console tests (subpath parity, page-relative API base, no localStorage key, no origin-root fetch); full suite 472 passing.

## What's New in v0.8.19

- **Console works behind the gateway (issue #109)** — the console's API calls used origin-root-relative paths (`fetch("/kb/bases")`); behind the gateway pass-through proxy (origin `:11432`, which only routes `/shell/fusion-rag/*`) those 404'd. The page now derives its API base from `location.pathname` (page-relative), and fusion-rag serves the FULL `/kb` router plus `/health` under the console subpath `/shell/fusion-rag/` — registered BEFORE the StaticFiles mount so routes win over static-file matching. Direct access (`:11436`) unchanged; behind the gateway the browser now asks `/shell/fusion-rag/kb/...` and the subpath-carrying proxy forwards it to the same handlers. 4 console tests; full suite 472 passing.

## What's New in v0.8.18

- **Built-in RAG console (admin menu follow-up)** — fusion-rag now ships a self-contained static console served at `/shell/fusion-rag/` (alias `/console`): KB list/create/delete, inline text doc ingest + delete, hybrid search tester (top-5 with scores), live `/health` pill. Single HTML page owned by this repo (ships in the pip package, no build step); all API calls use relative URLs so the page works identically direct (`:11436`) and behind fusion-gateway (user auth already proxied via the existing `rag:` upstream). Gateway menu integration (manifest entry + `/shell/fusion-rag/` proxy) requested upstream: dahai80/fusion-gateway#481. 3 new tests; full suite 471 passing.

## What's New in v0.8.17

- **Local embedding fallback: one shared load + fail-fast mode (issue #106)** — when fusion-mlx was down, every embed request fell back to a local sentence-transformers BGE-M3 cold load (~2-4 min on MPS), and concurrent requests each triggered their OWN load (no lock on the module singleton) — search hung for minutes and looked hung. `get_local_model` now uses double-checked locking: first caller loads, concurrent callers wait and reuse (verified live: 5 fallback triggers, exactly 1 model load). New operator knob `FUSION_RAG_EMBED_LOCAL_FALLBACK=0` disables the local fallback entirely: embed calls fail FAST with a retryable `EmbeddingError` (search returns a clean 503 with a readable detail, ~2.5s instead of minutes) instead of cold-loading a model in-request. In fail-fast mode `health()` no longer reports healthy via the local model, and `/ready` now respects `health()`'s return value (it previously ignored the boolean — `False` still reported "ok"). 8 new tests; full suite 468 passing.

## What's New in v0.8.16

- **Startup probe actually runs (issue #104)** — the O-P2-3 startup gate called the async `EmbeddingClient.health()` bare: the coroutine was never awaited, so the probe passed unconditionally (a fusion-rag booted with fusion-mlx down reported healthy) and every boot emitted a `RuntimeWarning: coroutine ... never awaited`. The gate is now an extracted `_run_startup_probe()` that probes fusion-mlx `GET /models` directly — deliberately NOT via `health()`, whose local sentence-transformers fallback would mask a down MLX on boxes where the local model is installed. Boot-time console logging is configured before the probe (previously the passing probe's log line was dropped — no handler existed until `run_server`). MLX unreachable → loud ERROR + exit 1; `FUSION_RAG_SKIP_STARTUP_PROBE=1` still bypasses. 4 new tests; full suite 460 passing.

## What's New in v0.8.15

- **Scaled-mode local key creation refused (issue #102)** — the per-instance SQLite `auth.db` contradicted the documented "scale horizontally behind a stateless load balancer" contract: a key provisioned on node A 401s on node B. Decision: option 1 (config-plane keys only). When `FUSION_RAG_REQUIRE_GATEWAY=1` or `FUSION_RAG_REQUIRE_IDENTITY=1` is on, `POST /kb/auth/keys` refuses with 409 instead of minting a single-node key; scaled deployments use the shared `FUSION_RAG_API_KEY` env or gateway/identity-issued credentials. Single-tenant default unchanged. README documents the constraint and the rejected option 2 (auth.db to Postgres). 4 new tests; two-node drill verified.

## What's New in v0.8.14

- **Data layer dependencies documentation (issue #100)** — new README section documenting when Qdrant is required (only for the `qdrant` backend / per-tenant isolation — the default local LanceDB backend needs no external DB), how to run it locally (container runtime required, `qdrant/qdrant:v1.12.4`), the canonical connection `QDRANT_URL=http://localhost:6333` (HTTP-only — the gRPC port 6334 is reserved but unused by fusion-rag), fail-loud expectations on ingest, and a pointer to `architecture/fusion-db-conventions-1030.md` as the governing doc. Follow-up noted in the issue: search-path silent degradation when Qdrant is unreachable.

## What's New in v0.8.13

- **Config Phase 3: legacy key-name cleanup + register-then-start self-check (issue #98)** — the last literal references to legacy env key names are gone from the code: key resolution reads the canonical `FUSION_CLIENT_KEY` only, with the legacy alias chain owned entirely by `fusion_core.config.client_key()` (never spelled out in this repo, so `scripts/check-env.py` reports 0 legacy warns for fusion-rag — and stays green when the check auto-escalates to errors on 2027-01-01). `start.sh` now verifies its own `port-registry.yaml` entry before serving (config Phase 3.3): an unregistered port fails at start with a clear message instead of at CI, `FUSION_SKIP_REGISTRY_CHECK=1` bypasses, and a missing registry file warns + continues for standalone deploys. 4 new tests cover the refuse/pass/bypass/missing-file paths; full suite 473 passing.

## What's New in v0.8.12

- **Single-point config resolution via fusion_core (issue #96)** — all hand-rolled key/env resolution is replaced by `fusion_core.config` (config Phase 2.2): the MLX API key resolves through `client_key()` (alias chain `FUSION_CLIENT_KEY` → `FUSION_MLX_API_KEY` → `MLX_API_KEY`; dev profile: empty + one-shot warning; `FUSION_PROFILE=prod` without a key: RuntimeError fail-fast), the server port through `service_port("rag")` (`FUSION_RAG_PORT`), and the data-dir root through `service_data_dir` (`FUSION_RAG_DATA_DIR` override, default stays `~/.fusion-rag`) — used by the KB stores root, watch registry, auth/graph/cache/eval SQLite paths. `server.py` chains `client_key() or resolve_api_key()` so the local-first `~/.fusion-mlx/settings.json` fallback keeps working; local `_mlx_api_key()`-style resolvers are deleted (call sites stay one-line). No env renames; `FUSION_RAG_API_KEY` (the service's own auth) is unchanged. 11 new tests; full suite 469 passing; `scripts/check-env.py` reports 0 errors for fusion-rag.

## What's New in v0.8.11

- **Exact chunk-id deletion (issue #94)** — deleting one document no longer removes the physically-shared chunks of an identical-content doc. Two docs ingested with the same `doc_path` (e.g. the same inline `doc_name` twice) used to share one deletion key, so `DELETE` on either removed both docs' vectors and BM25 rows. All delete/replace/rollback paths now go through a new `delete_by_chunk_ids` store surface (implemented by the local, qdrant, fusion-store, and remote backends; the remote `/store/*` surface gained `POST /delete_by_chunk_ids`), with a logged `delete_by_doc` fallback only for old remote nodes. 5 new tests; full suite 458 passing.
- **LLM auth surfacing (issue #93)** — a stale `FUSION_MLX_API_KEY` used to break every `/ask` with a generic "upstream LLM unavailable". A 401/403 from fusion-mlx now surfaces distinctly: `/ask` returns 502 "LLM credential rejected", the startup probe logs a loud ERROR (never exits — search keeps working), and `/health` reports an informational `llm_auth: ok/rejected/unknown` field. A successful generation clears a stale rejected flag after key rotation; logs identify the key only by blake2b hash prefix. 11 new tests.
- **Process title (issue #92)** — the server process now shows as `fusion-rag serve :11436` in `ps`/Activity Monitor instead of a bare Python (setproctitle, soft dependency). `start.sh` stops by PID file, so lifecycle control is unaffected. 3 new tests.

## What's New in v0.8.10

- **Inline binary ingest fix (issue #90)** — `POST /kb/bases/{kb_id}/documents/ingest` with `content_type` `pdf`/`docx`/`epub` now treats `content` as base64-encoded binary: it is decoded to a temp file and run through the full `DocumentParser` pipeline (PDF text extraction, EPUB chapters, and the scanned-PDF OCR path from #84), instead of chunking and indexing the base64 text itself — which silently polluted retrieval with garbage chunks. Invalid base64 returns 400. The text-ish types (`text`/`markdown`/`html`/`csv`) keep the direct-text path unchanged. 5 new tests (PDF/EPUB parse round-trips, scanned-PDF OCR path, invalid base64, plain-text regression); full suite 439 passing.

## What's New in v0.8.9

- **OCR tenant header passthrough (issue #88)** — `OcrHook` now sends `X-Tenant-Id` on every call to the drawing-parser OCR service (its tenant middleware 401s header-less calls, breaking scanned-PDF ingest in multi-tenant deployments). The tenant is resolved from the calling request's context (works with both `#61` gateway and `#68` identity modes); `FUSION_RAG_OCR_TENANT_ID` is the fallback for background watch-loop ingestion and single-tenant deployments. Invalid charset values are dropped and logged, never forwarded.

## What's New in v0.8.8

- **Similar-chunk search (issue #83)** — new `POST /kb/bases/{kb_id}/similar` endpoint for near-duplicate / version-drift detection: query by `text` (embedded with the same service-wide model as ingest, so scores are comparable) or a raw `embedding` vector; `top_k`, `threshold`, `exclude_doc_id` (dedup-within-corpus), `snippet_chars`, and `status_filter`. Returns `matches[]` with full evidence anchors (doc_id/chunk_id/page/status).
- **EPUB + scanned-PDF OCR (issue #84)** — `DocumentParser` now reads EPUB (ebooklib, each XHTML chapter becomes a headed section). Scanned PDFs (pages present, zero extractable text) are handed to the fusion-drawing-parser OCR service (`FUSION_RAG_OCR_URL`, `FUSION_RAG_OCR_API_KEY`, `FUSION_RAG_OCR_TIMEOUT`) instead of being silently ingested as empty docs — and fail visibly when no OCR service is configured. OCR chunks bypass the chunker + contextualizer (already slice-sized) and carry per-chunk evidence metadata.
- **OCR evidence metadata + citations (issue #85)** — every `/search` result and `/ask` citation from an OCR-sourced chunk carries `ocr_confidence`, `low_confidence` (vs `FUSION_RAG_LOW_OCR_CONFIDENCE`, default 0.85), and `slice_ref` (`task_id`/`slice_id`/`url` linking back to the source slice image). `/ask` additionally returns per-chunk `citations[]` (not deduped) and a top-level `has_low_confidence` flag.
- **Document active/deprecated status (issue #86)** — `PATCH /kb/bases/{kb_id}/documents/{doc_id}` flips a doc's lifecycle `status` (`active`/`deprecated`, optional `superseded_by`) with no re-indexing. Status is resolved at search time from the documents table, so a deprecated doc vanishes from default `/search`, `/ask`, and `/similar` immediately; `status_filter` (`active`/`deprecated`/`all`) controls each query, and deprecated chunks kept via `all` are marked `[已作废版本]` in the LLM context.
- 19 new tests (EPUB parse + OCR roundtrip + failure paths, /similar validation + exclusion + status, PATCH + status filtering + /ask citations), full suite 428 green, ruff clean.

## What's New in v0.8.3

- **LLM backend URL + auth propagation (issue #72)** — every LLM-calling engine (`Contextualizer`, `QueryRewriter`, `Reranker`) now honors the same `FUSION_MLX_URL` / `FUSION_MLX_API_KEY` as the `EmbeddingClient` instead of hardcoding the gateway URL `http://127.0.0.1:11432/v1` and sending no auth header. A non-default backend (e.g. fusion-mlx directly at `:11434` with a different key) no longer 401s on every chat-completions / context-generation call. The route layer passes `embed.base_url` + `embed.api_key` into each engine it constructs; `start.sh` now activates the monorepo `.venv` before launching (was bare `python3`, failing with `ModuleNotFoundError: No module named 'fusion_core'` outside an activated venv). 4 new regression tests lock the URL + `Authorization` header wiring.

## What's New in v0.8.2

- **Cross-encoder rerank (issue #70)** — real cross-encoder reranking via fusion-mlx's Cohere/Jina-compatible `POST /v1/rerank` endpoint, replacing the legacy LLM prompt-scoring `Reranker` as the precision stage. New `CrossEncoderReranker` (`engine/cross_encoder_reranker.py`) is a plain async HTTP client (no MLX import — respects the project's "no direct MLX imports" + "only modify own project" rules) to fusion-mlx's rerank route, e.g. `bge-reranker-v2-m3`. Select it with `FUSION_RAG_RERANK_BACKEND=cross_encoder` + `FUSION_RAG_RERANK_MODEL=bge-reranker-v2-m3`; when `FUSION_RAG_RERANK_MODEL` is set, `/search` and `/ask` default `rerank=true` (rerank ON when a model is available — acceptance 1). `rerank_top_n` (default 20) fetches a wider candidate pool that the cross-encoder re-scores before truncating to `top_k`, lifting recall without changing the returned count. Fallback chain: cross-encoder → legacy LLM-prompt `Reranker` → original retrieval order (logged, never crashes — backward compat, acceptance 4). Hybrid retrieval (BM25 + vector + RRF fusion) was already present (`hybrid=true`, acceptance 2). A recall benchmark (`scripts/recall_benchmark.py`) runs offline (mocked embed + mocked cross-encoder) and guards the no-regression contract (rerank ≥ hybrid ≥ vector); for the absolute recall-lift magnitude, run it against live fusion-mlx with BGE-M3 + bge-reranker-v2-m3 loaded (acceptance 3). 10 new tests (unit: reorder/score-stamp/empty/truncation/404/network; integration: backend selection, fallback chain, default-off regression guard, default-on when model set, explicit `rerank=true`).

## What's New in v0.7.2

- **Audit P4 + architecture hardwounds (Task #10-18)** — maintainability fixes and 硬伤 A-F 收口 from the adversarial audit (`audit/fusion-rag-audit-0827.md`):
  - **硬伤 B 收口** — the three-store ingest write (vectors / bm25 / metadata) now writes the hard-undo side **first**: `vectors → bm25 → metadata` via a single `_write_doc_to_stores` helper shared by all four ingest paths (`_index_document`, `ingest_content`, `replace_document`, `scan_directory`). A crash now leaves recoverable orphan vectors (re-ingest over `doc_path`) instead of orphan metadata that makes `list_documents` lie. Unified rollback across all paths (P4-1).
  - **硬伤 C 收口** — auth→identity→ACL wired: `require_kb_action` chains `verify_api_key` (resolves subject) → `PermissionManager.check_permission`, enforced on every `/bases/{kb_id}/*` route including reads (P0-5/6/7 already landed; this PR confirms the full chain).
  - **P4 maintainability** — `BM25Index._init_db` no longer double-opens a SQLite connection (P4-2); `EmbeddingCache` eviction uses a `created_at`-index subquery instead of an O(n) `OFFSET` scan (P4-3), and `_hash` switches MD5→blake2b and drops the 1000-char text truncation that made the stored column disagree with the hash key (P4-4); `VectorStore` routes every backend through `StoreBackendFactory` uniformly (no more hardcoded 3-branch), backends accept `**_` to ignore irrelevant kwargs (P4-5); `SearchTemplateManager` seeds builtins once per DB (not per construction) and uses `ON CONFLICT DO UPDATE` so a new release's builtin defaults actually apply (P4-6); `KnowledgeBase`/`KnowledgeBaseConfig.from_dict` warn on unknown keys and raise on a load-time empty `id` instead of silently regenerating one (which orphaned storage) (P4-7); `validate_identifier` dead code removed and template validation now checks name charset/length, description cap, and `doc_type_filter` items (P4-8).
  - **硬伤 D 部分收口** — fusion-store backend uses `score = 1/(1+dist)` + threshold tuning (P0-10); the root fix (exposing `Store.open(metric=...)` in the upstream fusion-store binding) is tracked as an upstream issue. Default `local` backend (LanceDB cosine) is unaffected and recommended for production.
- 243 tests green (random + ordered), ruff clean.

## What's New in v0.7.1

- **Audit P2/P3 fixes (Task #8)** — architecture + performance hardening from the adversarial audit (`audit/fusion-rag-audit-0827.md`): `LocalBackend.close()` now actually releases the BM25 SQLite handle (was only nulling refs); `FusionStoreBackend.close()` raises on checkpoint failure instead of silently losing data; `EmbeddingCache` inherits `SqliteBase` (no more connection-per-call storm) with a batched `WHERE IN` lookup; `MetadataStore` splits read/write lock paths for concurrent reads; `BenchRunner`/`SearchTemplateManager`/`AuditLogger` adopt the `SqliteBase` write lock (no more cross-thread commit/rollback pollution); 5 admin managers pooled per-KB on `app.state` and closed at shutdown; `PermissionManager` uses `open_sqlite` (WAL + busy_timeout); `StoreBackendFactory` raises on unknown backend type instead of silently falling back to local; `RemoteBackend.count/clear` raise instead of masking a misconfigured empty KB.
- **P3 perf** — `delete_by_doc` uses a `doc_path→int_ids` KV index (was O(n) full vector scan per delete); BM25 `_df` maintained incrementally (was full Counter rebuild per op); `LocalBackend.search` skips rows missing `_distance` with a warning (was treating missing as best match); `get_document_by_metadata` uses `json_extract` (was O(n) full-table scan); `AuditLogger` prunes on construction + streams exports via keyset pagination; `version_manager` caches record counts (60s TTL) instead of a full COUNT per snapshot.
- **Deadlock fix** — `SqliteBase._db_lock` switched `Lock → RLock` to fix a re-entrant deadlock in `EmbeddingCache` (locks-then-calls-`_get_conn`, which also locks during lazy connection create). 243 tests green, ruff clean.

## What's New — Remote Backend (full-stack)

- **Remote vector store backend (#remote-store)** — `RemoteBackend` (`store/remote_backend.py`) is now a real httpx client that delegates vector add/search/keyword/delete/count/clear to a remote fusion-rag node over HTTP, instead of the old `NotImplementedError` stub. The matching server half is a new `/kb/bases/{kb_id}/store/*` surface (`api/routes_store.py`, auth via `X-API-Key`), so one fusion-rag instance can act as the vector store for another. Opt in with `FUSION_RAG_STORE_BACKEND=remote` + `FUSION_RAG_REMOTE_ENDPOINT=http://node-b:11436`; default stays `local` (zero behavior change). Metadata stays local (split-store model: the remote owns vectors/BM25, the caller owns document metadata/audit/trajectory). The `/store/*` path is exempt from the gateway-origin gate (tenant.py) since it is authenticated M2M, not a gateway-routed user request. 10 new tests (round-trip via in-process ASGI transport + config + server auth). Also fixes a pre-existing `LocalBackend.delete_by_doc` count bug (LanceDB `DeleteResult.num_deleted_rows` was never read, so deletes always reported 0).
- **Authoritative tenant resolution via fusion-identity (#68)** — retires the forgeable `X-Fusion-Tenant` blind-trust from #61. When `FUSION_RAG_REQUIRE_IDENTITY=1`, `/kb/*` requests require `Authorization: Bearer <JWT>`; fusion-rag verifies the JWT via fusion-identity's `POST /api/v1/auth/verify` (service-token-gated, revocation + tenant-status enforced by the identity service) and takes the authoritative tenant from the JWT `tid` claim. The resolved tenant feeds the same `_request_tenant` contextvar, so KB scoping (#61) and per-tenant collection isolation (#66) keep working unchanged. `X-Fusion-Tenant` is demoted to defense-in-depth (must match the JWT `tid` if present, else 401). No/invalid/revoked JWT → 401; unreachable identity → fail-closed deny. fusion-identity is an HTTP service dependency (port 11470), not a Python import — fusion-rag talks to it over plain HTTP. Default off; single-tenant local-first dev unaffected. 12 tests (forged header 401, invalid JWT 401, valid JWT scoped, cross-tenant 404, header/JWT mismatch 401, revoked 401, exempt paths, integration-off regression guard, client unit).
- **Qdrant vector backend + per-tenant collection isolation (#66)** — optional `QdrantBackend` (`store/qdrant_backend.py`) sinks vector storage to Qdrant and picks the collection PER REQUEST from the authoritative tenant (`X-Fusion-Tenant`): a tenant-A upsert lands in `tenant_id_A`, a tenant-B search queries `tenant_id_B` — cross-tenant vectors are physically invisible (collection-per-tenant is the data-layer isolation tier; the KB-level `tenant` field from #61 is the logical tier). When no tenant is in effect (single-tenant dev, isolation off), a shared `fusion_rag_kb_{kb_id}` collection is used so the default path still works. `chunk_id` (str) maps to a Qdrant point id (int) via a deterministic blake2b hash, so re-ingest overwrites cleanly. Keyword search stays in-process (BM25), exactly like the other backends — Qdrant is a vector primitive. Opt in with `FUSION_RAG_STORE_BACKEND=qdrant` + `QDRANT_URL` (`:memory:` = in-process local mode, no server needed; otherwise the remote Qdrant URL); `pip install 'fusion-rag[qdrant]'`. Default stays `local` (LanceDB), so no migration. 12 tests (round-trip recall, BM25, delete, cross-tenant isolation acceptance, default-collection, dim-mismatch) run with `:memory:` — no Qdrant server needed in CI.

## What's New in v0.7.0

- **fusion-store HNSW vector backend (#47)** — optional `FusionStoreBackend` sinks vector storage/retrieval to the in-tree fusion-store HNSW index via its PyO3 binding. Opt in with `FUSION_RAG_STORE_BACKEND=fusion-store`; default stays `local` (LanceDB), so no migration. fusion_store uses int ids while fusion-rag uses str chunk ids — the backend maintains a str↔int mapping in fusion-store KV and persists an id counter in a sidecar. Chunk metadata rides in fusion-store KV (`put_kv`); BM25 keyword search stays in-process (fusion-store is a vector primitive only, per its PRD boundary). Hybrid search (vector + BM25 + RRF) keeps working. 9 backend tests (skip on CI where the binding isn't installed).

## What's New in v0.6.7

- **fusion-core HTTP client migration (PR #41)** — LLM modules (reranker, contextualizer, query_rewriter, rag_chain, graph_rag, evaluator, streaming, routes._generate_answer) now use `fusion_core.http_client.get_async_client` (shared connection pool, LRU-keyed by loop+base_url) + `with_retry` (auto-retry on 429/5xx + transient errors). Auth headers passed per-request, not baked into pooled client. SSE streaming and embed/client keep raw httpx.
- **D-H3 empty-content guards** — success-path empty/whitespace LLM responses now degrade explicitly (logged error/fallback) instead of returning empty as a valid result.
- **CLAUDE.md re-checked in** — un-ignored and refreshed: accurate architecture tree (parse/, permissions/, store split, 7 route files, 7 new engine modules), 170 tests green, 5 missing env vars documented, fusion_core dependency + ruff lint documented.

## What's New in v0.6.6

- **RRF Hybrid Search Fix** — `_rrf_fusion` no longer applies a cosine-scaled `similarity_threshold` (0.3) to rank-fusion scores (max ~0.066); hybrid RRF search now returns matches instead of silently empty results. Regression test added.

## What's New in v0.6.0

- **Version Snapshots** — Hard-link based KB snapshots with rollback
- **Incremental Sync** — MD5 hash + mtime change detection for directories
- **AST Chunking** — Python AST-aware code chunking (auto-detect)
- **Search Templates** — 3 presets (general/code/design) + custom templates
- **Audit Logging** — Full search audit trail with JSON/CSV export
- **Fine-grained Permissions** — Role-based ACL with path-prefix inheritance
- **StoreBackend Abstraction** — LocalBackend (LanceDB) + RemoteBackend (extensible)
- **Bench API** — Search latency benchmarking with SQLite results
- **Cloud Embed Fallback** — Auto-fallback to cloud API when local fails
- **Git Repo Indexing** — Clone + .gitignore-aware file indexing
- **Routes Split** — Monolithic routes.py → 4 focused sub-modules

---

## License

Apache License 2.0 — see [LICENSE](LICENSE) for details.

## Acknowledgments

- [fusion-mlx](https://github.com/dahai80/fusion-mlx) — Apple Silicon model serving
- [LanceDB](https://github.com/lancedb/lancedb) — Vector database
- [LightRAG](https://github.com/HKUDS/LightRAG) — Reference architecture
