Phase 10 — RAG: Persistent Document Knowledge Base

1. Architecture Overview

A dedicated ChromaDB collection (collective.documents) holds chunked, embedded document passages. A search_documents tool lets Gemma query it semantically — same bus pattern as search_memory.

User ingest (CLI or UI)
  → DocumentService.ingest_file(path)
      → extract text (PDF / plain text / markdown)
      → chunk into overlapping passages
      → embed each chunk with nomic-embed-text
      → store in ChromaDB collection: collective.documents

Gemma tool call: search_documents(query="…")
  → DocumentService.search(query)
      → embed query with nomic-embed-text (search_query: prefix)
      → ChromaDB similarity search
      → return top-k passages with source filename + page

Relationship to existing services

Both share the same ChromaDB path (.chroma/) and the same embedding model (nomic-embed-text), but use separate collections so searches don't cross-contaminate.

2. New Files

Modified files

3. Chunking Strategy

Fixed-size character chunks with overlap. Character-based (not token-based) to avoid a tokenizer dependency.

Each chunk is stored with metadata:

{
  "source_file": "attention_is_all_you_need.pdf",
  "chunk_index": 3,
  "page": 2,            # PDF only; omitted for text files
  "ingested_at": 1717430000.0,
  "type": "document"
}

4. file_extract.py — Shared Extraction

Currently attachment_bar.py has private _extract_pdf_text() and _process_file() functions. These will be moved to src/local/utils/file_extract.py as public functions, and attachment_bar.py will import from there. This is the only UI file touched.

# src/local/utils/file_extract.py

TEXT_EXTS = {".txt", ".md", ".py", ".js", ".ts", ".yaml", ".json", ".csv"}
IMAGE_EXTS = {".jpg", ".jpeg", ".png", ".gif", ".webp"}

def extract_text(path: str) -> str:
    """Return plain text from a PDF or text file. Raises on unsupported types."""
    ...

def extract_for_attachment(path: str) -> dict:
    """Return {type, name, data} dict for AttachmentBar — images as base64, text as str."""
    ...

5. DocumentService API

# src/local/services/document_service.py

class DocumentService:
    def __init__(self, chroma_path=None, collection_name=None, embed_model=None): ...

    def ingest_file(self, path: str) -> int:
        """Chunk, embed, and store file. Returns number of chunks written."""
        ...

    def ingest_text(self, text: str, source_name: str) -> int:
        """Ingest already-extracted text (for programmatic use). Returns chunk count."""
        ...

    def search(self, query: str, n: int | None = None) -> list[dict]:
        """Return top-n chunks by similarity: [{content, source_file, chunk_index, page, score}]"""
        ...

    def list_sources(self) -> list[str]:
        """Return unique source filenames in the collection."""
        ...

    def delete_source(self, source_file: str) -> int:
        """Delete all chunks for a given source file. Returns count deleted."""
        ...

6. search_documents Tool Schema

{
  "type": "function",
  "function": {
    "name": "search_documents",
    "description": "Searches the user's personal document knowledge base for relevant passages. ...",
    "parameters": {
      "type": "object",
      "properties": {
        "query": {
          "type": "string",
          "description": "What to look for in the knowledge base."
        }
      },
      "required": ["query"]
    }
  }
}

Trigger conditions live in the tool description (same principle as Phase 8/9). The description will say:

"Searches the user's personal document knowledge base for relevant passages. Call this tool when the user asks about content from documents they have added to their knowledge base, or when a question is likely answered by their stored documents rather than general knowledge or the web."

7. Tool Result Format

[Knowledge base results for "transformer positional encoding"]

1. attention_is_all_you_need.pdf (p.4, chunk 8)
   Since our model contains no recurrence and no convolution, in order for the model
   to make use of the order of the sequence, we must inject some information about
   the relative or absolute position of the tokens in the sequence…

2. transformer_survey.pdf (p.12, chunk 31)
   Positional encodings can be fixed (sinusoidal, as in the original Transformer)
   or learned. Both approaches yield similar results in practice…

8. CLI Ingest Script

# scripts/ingest.py
# Usage: python scripts/ingest.py file.pdf [file2.txt ...]
# Runs without the full stack (no ZMQ, no Ollama chat endpoint).

import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent / "src"))

from local.services.document_service import DocumentService

def main():
    paths = sys.argv[1:]
    if not paths:
        print("Usage: python scripts/ingest.py file.pdf [file2.txt ...]")
        sys.exit(1)
    svc = DocumentService()
    for path in paths:
        n = svc.ingest_file(path)
        print(f"  {path}: {n} chunks ingested")

if __name__ == "__main__":
    main()

9. run_local.py Changes

# Create DocumentService alongside MemoryService (both are ChromaDB; init in main thread)
from local.services.document_service import DocumentService
shared_documents = DocumentService()

# Start tool (after existing tools, before generator)
threading.Thread(
    target=_start_search_documents,
    args=(shared_documents,),
    daemon=True, name="search_documents"
).start()

10. config/documents.yaml

collection: "collective.documents"
chroma_path: ".chroma"
embed_model: "nomic-embed-text"
chunk_size: 1500
chunk_overlap: 200
n_results: 5

description: |
  Searches the user's personal document knowledge base for relevant passages.
  Call this tool when the user asks about content from documents they have added,
  or when a question is likely answered by stored documents rather than general
  knowledge or the web. Do not call this for general research questions — use
  search_papers or web_search for those.

param_query: |
  What to look for in the knowledge base. Be specific — use terms likely to
  appear in the source documents.

11. Testing Plan

Unit tests — tests/test_document_service.py

Unit tests — tests/test_search_documents_tool.py

Story — tests/stories/s12_rag.yaml

• Turn 1: ingest a short text via CLI first; then ask "what does the document say about X?"
• Turn 2: follow-up question using conversation history
• Turn 3: ask about something not in the document — Gemma should not hallucinate from the KB

12. Build Order

1. Create src/local/utils/file_extract.py — move extraction logic from attachment_bar
2. Update attachment_bar.py to import from file_extract
3. Write src/local/services/document_service.py
4. Add subjects to protocol/subjects.py
5. Write src/local/tools/search_documents_tool.py
6. Wire run_local.py
7. Add TOOL_ACTIVITY_SEARCH_DOCUMENTS to main_window.py
8. Write config/documents.yaml
9. Write scripts/ingest.py
10. Write unit tests and story
11. Run full test suite