# Knowledge Skill - Mnemosyne, Titan of Memory

Retrieval-Augmented Generation (RAG) for Familiar.
Build a searchable knowledge base from your documents.

## Features

- **Document ingestion** - PDF, Word, text, markdown, HTML, CSV, JSON
- **URL ingestion** - Fetch and index web pages
- **Intelligent chunking** - Overlapping chunks preserve context
- **Local embeddings** - Works fully offline with sentence-transformers
- **Semantic search** - Find relevant content by meaning, not just keywords
- **Hybrid search** - Combines semantic + keyword matching
- **Source attribution** - Know where information came from
- **Collections** - Organize documents by topic/project

## Usage Examples

```
"Add this PDF to my knowledge base"
"Search my documents for budget information"
"What do my notes say about the board meeting?"
"Add all files from my project folder"
"Create a collection for grant applications"
```

## How It Works

```
Documents → Chunks → Embeddings → Vector Store
                                      ↓
Query → Embedding → Similarity Search → Relevant Context → Answer
```

1. **Ingest**: Documents are split into overlapping chunks
2. **Embed**: Each chunk gets a vector embedding (local model)
3. **Store**: Chunks and embeddings saved to disk
4. **Search**: Query is embedded, similar chunks retrieved
5. **Answer**: Retrieved context helps generate accurate answers

## Installation

### Basic (Required)
```bash
# Embedding model (pick one)
pip install sentence-transformers  # Full-featured, ~500MB
# OR
pip install fastembed              # Lighter, faster, ~100MB
```

### For PDF Support
```bash
pip install pymupdf    # Recommended
# OR
pip install pypdf      # Lighter alternative
```

### For Word Documents
```bash
pip install python-docx
```

### For Better HTML Parsing
```bash
pip install beautifulsoup4
```

### Complete Setup
```bash
pip install sentence-transformers pymupdf python-docx beautifulsoup4
```

## Embedding Models

| Model | Size | Speed | Quality | Best For |
|-------|------|-------|---------|----------|
| all-MiniLM-L6-v2 | 80MB | Fast | Good | Pi, general use |
| all-mpnet-base-v2 | 420MB | Medium | Better | Desktop |
| bge-small-en-v1.5 | 130MB | Fast | Good | fastembed |

Default: `all-MiniLM-L6-v2` (works great on Pi 4)

## Collections

Organize documents into collections:

```
"Create collection 'grants' for grant documents"
"Add this proposal to the grants collection"
"Search grants for deadline information"
```

Collections are stored in `~/.familiar/data/knowledge/collections/`

## Chunking Strategy

Documents are split intelligently:
- **Chunk size**: ~500 characters (configurable)
- **Overlap**: ~50 characters between chunks
- **Hierarchy**: Splits on paragraphs → sentences → words

This preserves context across chunk boundaries.

## Search Modes

### Semantic Search (Default)
Finds content by meaning:
- "budget concerns" finds "financial worries"
- "meeting notes" finds "discussion summary"

### Hybrid Search
Combines semantic + keyword:
- Better for specific terms
- Good for names, codes, numbers

## File Locations

- Collections: `~/.familiar/data/knowledge/collections/`
- Each collection: `{name}.json`

## Tools

| Tool | Description |
|------|-------------|
| search_knowledge | Search for information |
| get_context | Get raw context for LLM |
| add_document | Add file, URL, or text |
| add_folder | Batch add from folder |
| create_collection | Create new collection |
| list_collections | Show all collections |
| collection_info | Collection details |
| remove_document | Remove from collection |
| delete_collection | Delete collection |

## Raspberry Pi Tips

1. Use `fastembed` instead of `sentence-transformers` (lighter)
2. Start with small chunk sizes (300-500)
3. Index documents incrementally
4. Use `all-MiniLM-L6-v2` model (fast, small)

## Example Workflow

```bash
# 1. Create a collection
"Create a collection called 'board' for board documents"

# 2. Add documents
"Add board_minutes_jan.pdf to board collection"
"Add all files from ~/Documents/board/ to board"

# 3. Search
"Search board for executive director compensation"
"What did the board decide about the budget?"
```

## API for Developers

```python
from skills.knowledge.skill import (
    _load_collection,
    _search_collection,
    _embed_texts
)

# Load collection
collection = _load_collection("my_docs")

# Search
results = _search_collection(collection, "budget planning", top_k=5)

for result in results:
    print(f"{result.score:.2f}: {result.chunk.content[:100]}...")
```
