Metadata-Version: 2.4
Name: searchat
Version: 0.7.2
Summary: Semantic search and RAG for local AI coding agent conversations
Project-URL: Homepage, https://github.com/Mathews-Tom/searchat
Project-URL: Repository, https://github.com/Mathews-Tom/searchat.git
Project-URL: Issues, https://github.com/Mathews-Tom/searchat/issues
Project-URL: Changelog, https://github.com/Mathews-Tom/searchat/blob/master/CHANGELOG.md
Author: Searchat Contributors
License: MIT
License-File: LICENSE
Keywords: ai-coding,conversation-search,duckdb,faiss,local-first,rag,semantic-search
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Environment :: Web Environment
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: duckdb==1.4.3
Requires-Dist: fastapi>=0.100.0
Requires-Dist: jinja2>=3.1.0
Requires-Dist: litellm>=1.81.3
Requires-Dist: numpy<2
Requires-Dist: pandas>=2.0.0
Requires-Dist: pygments>=2.17.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: python-multipart>=0.0.31
Requires-Dist: reportlab>=4.2.0
Requires-Dist: rich>=13.0.0
Requires-Dist: sentence-transformers>=2.3.0
Requires-Dist: tomli>=2.0.1
Requires-Dist: uvicorn>=0.20.0
Requires-Dist: watchdog>=3.0.0
Requires-Dist: zstandard>=0.25.0
Provides-Extra: all
Requires-Dist: cryptography>=42.0.0; extra == 'all'
Requires-Dist: faiss-cpu>=1.7.4; extra == 'all'
Requires-Dist: keyring>=25.0.0; extra == 'all'
Requires-Dist: llama-cpp-python>=0.2.0; extra == 'all'
Requires-Dist: mcp>=0.1.0; extra == 'all'
Requires-Dist: nbformat>=5.9.0; extra == 'all'
Requires-Dist: pyarrow>=14.0.0; extra == 'all'
Requires-Dist: tree-sitter-languages>=1.10.2; extra == 'all'
Requires-Dist: tree-sitter>=0.21.3; extra == 'all'
Requires-Dist: weasyprint>=60.0; extra == 'all'
Provides-Extra: embedded
Requires-Dist: llama-cpp-python>=0.2.0; extra == 'embedded'
Provides-Extra: gpu
Requires-Dist: faiss-gpu>=1.7.2; extra == 'gpu'
Provides-Extra: jupyter
Requires-Dist: nbformat>=5.9.0; extra == 'jupyter'
Provides-Extra: legacy
Requires-Dist: pyarrow>=14.0.0; extra == 'legacy'
Provides-Extra: mcp
Requires-Dist: mcp>=0.1.0; extra == 'mcp'
Provides-Extra: palace
Requires-Dist: faiss-cpu>=1.7.4; extra == 'palace'
Requires-Dist: rank-bm25>=0.2.2; extra == 'palace'
Provides-Extra: pdf
Requires-Dist: weasyprint>=60.0; extra == 'pdf'
Provides-Extra: secure
Requires-Dist: cryptography>=42.0.0; extra == 'secure'
Requires-Dist: keyring>=25.0.0; extra == 'secure'
Provides-Extra: treesitter
Requires-Dist: tree-sitter-languages>=1.10.2; extra == 'treesitter'
Requires-Dist: tree-sitter>=0.21.3; extra == 'treesitter'
Description-Content-Type: text/markdown

# Searchat

Semantic search and RAG-powered Q&A for AI coding agent conversations. Find past solutions by meaning, not just keywords, and ask questions about your conversation history.

## Project Lineage

This repository is inspired by and built on the original [Process-Point-Technologies-Corporation/searchat](https://github.com/Process-Point-Technologies-Corporation/searchat).

The original repository is also accompanied by and stems from the paper [Structured Distillation for Personalized Agent Memory: 11× Token Reduction with Retrieval Preservation](https://arxiv.org/html/2603.13017) (arXiv:2603.13017, March 13, 2026).

Credit to the original repository and its authors for the foundation:

- local-first semantic search for AI coding conversations
- append-only indexing and live file watching
- the initial FastAPI + web UI shape
- the original Claude Code and Mistral Vibe support
- the structured distillation and personalized agent memory framing described in the paper

This fork has since expanded substantially beyond that baseline. It now includes:

- 8 agent connectors across local coding tools and chat clients
- RAG chat, session chat, MCP integration, and embedded local LLM support
- an expertise layer for extraction, validation, priming, and onboarding workflows
- an L3 knowledge graph for lineage, contradiction detection, and resolution
- bookmarks, saved queries, dashboards, exports, backups, snapshots, and analytics
- a materially larger API, CLI, frontend, documentation, and test surface

## Supported Agents

| Agent        | Location                                                     | Format   |
| ------------ | ------------------------------------------------------------ | -------- |
| Claude Code  | `~/.claude/projects/**/*.jsonl`                              | JSONL    |
| Mistral Vibe | `~/.vibe/logs/session/*.json`                                | JSON     |
| OpenCode     | `~/.local/share/opencode/storage/session/*/*.json`           | JSON     |
| OpenAI Codex | `~/.codex/sessions/**/rollout-*.jsonl`                       | JSONL    |
| Gemini CLI   | `~/.gemini/tmp/<project_hash>/chats/*.json`                  | JSON     |
| Continue     | `~/.continue/sessions/*.json`                                | JSON     |
| Cursor       | `.../Cursor/User/.../*.vscdb`                                | SQLite   |
| Aider        | `.aider.chat.history.md` (set `SEARCHAT_AIDER_PROJECT_DIRS`) | Markdown |

## Features

### Core Search

- **Hybrid Search** — DuckDB FTS keyword + FAISS semantic vectors with RRF fusion
- **Query Synonyms** — Automatic expansion of common search terms
- **Cross-Encoder Re-ranking** — Optional re-ranking with cross-encoder models
- **Multi-Agent** — Search across Claude Code, Mistral Vibe, and OpenCode sessions
- **Tool Filters** — Filter results by specific agent (Claude, Vibe, or OpenCode)
- **Autocomplete** — Smart search suggestions as you type
- **Search History** — Persistent search history with LocalStorage

### AI-Powered Features

- **RAG Chat** — Ask questions about your conversation history with AI-powered answers
- **Session Chat** — Multi-turn RAG conversations with session persistence (30-min TTL)
- **Pattern Mining** — Extract recurring coding patterns from conversation archives
- **Embedded LLM** — Run RAG chat locally with a GGUF model (llama-cpp-python)
- **Semantic Highlights** — Optional LLM-generated highlight terms for search results
- **Conversation Similarity** — Discover related conversations using semantic similarity
- **Code Extraction** — Extract and view code snippets with syntax highlighting

### Organization & Management

- **Bookmarks** — Save and annotate favorite conversations
- **Saved Queries** — Save reusable searches (query + filters + mode)
- **Dashboards** — Build dashboards from saved queries (widgets + auto-refresh)
- **Search Analytics** — Track search patterns and usage statistics
- **Export** — Export conversations in JSON/Markdown/Text (optional PDF + Jupyter notebook)
- **Bulk Export** — Export multiple conversations at once
- **Agent Config Export** — Generate CLAUDE.md, copilot-instructions, or cursorrules from patterns
- **Pagination** — Navigate large result sets efficiently

### Data Safety & Performance

- **Live Indexing** — Auto-indexes new/modified files (5min debounce)
- **Append-Only** — Never deletes existing data, safe for long-term use
- **Backups** — Create and restore backups from UI or API
- **Snapshots** — Browse backups as read-only datasets ("snapshot" mode)
- **Safe Shutdown** — Detects ongoing indexing, prevents data corruption
- **DuckDB Storage** — Efficient Parquet-based storage with fast queries
- **FAISS Vectors** — High-performance semantic search

### User Experience

- **Keyboard Shortcuts** — Power user navigation and commands
- **Cross-Platform** — Windows, WSL, Linux, macOS
- **Local-First** — All data stays on your machine
- **Self-Search** — Agents can search their own history via API
- **MCP Server** — Let MCP clients (Claude Desktop, Cursor, etc.) query your local index
- **Terminal Resume** — Resume conversations directly in terminal

## 📊 Interactive Documentation

Visual architecture diagrams and flow charts:

- [System Overview](docs/architecture.md) - Current end-to-end architecture across connectors, indexing, storage, retrieval, knowledge layers, and interfaces
- [RAG Chat Pipeline](docs/infographics/rag-chat-pipeline.html) - AI-powered Q&A over conversations
- [Backup & Restore](docs/infographics/backup-restore-flow.html) - Data safety and recovery procedures
- [File Watching](docs/infographics/file-watching-indexing.html) - Live indexing and event handling
- [Multi-Agent Connectors](docs/infographics/multi-agent-connectors.html) - Agent support architecture

Use the HTML diagrams in a browser for interactive walkthroughs, and use `docs/architecture.md` for the canonical Mermaid system map. [See all infographics →](docs/infographics/README.md)

## Quick Start

### Install And Run (Standalone)

Install Searchat and build the initial index:

```bash
pip install searchat

# Optional: create ~/.searchat/config/settings.toml interactively
python -m searchat.setup

# Build the initial search index
searchat setup-index

# Start the web server
searchat web
```

Open http://localhost:8000 (searches ports up to 8010 if 8000 is busy; override with `SEARCHAT_PORT`).

### Install From Source

```bash
git clone https://github.com/Mathews-Tom/searchat.git
cd searchat
pip install -e .

# First-time setup: build search index
python scripts/setup-index

# Start web server
searchat web
```

Open http://localhost:8000 (searches ports up to 8010 if 8000 is busy; override with `SEARCHAT_PORT`).

The setup script indexes all conversations from supported agents. On subsequent runs, the web server automatically indexes new conversations via live file watching.

### Rebuilding The Index

Two different operations can be confused — know which one you need:

- **`searchat rebuild-derived [--force]`** — always safe. Rebuilds the
  keyword and vector search indexes from conversations already stored in
  the local database. Never opens a source conversation file, so it's safe
  to run at any time, including after a compaction or backup restore, or
  with no source files present at all.
- **`searchat reingest-sources [--force]`** — guarded and dangerous if
  source files are incomplete. Re-scans source JSONL/session files and
  replaces the existing index with what it finds there. If source files are
  missing or incomplete, indexed conversations absent from those files are
  permanently lost. Refuses with an error unless an existing index is
  absent or `--force` is explicitly passed.

For routine maintenance (fixing a corrupted index, restoring after backup,
recovering from disk compaction) use `rebuild-derived`. Only use
`reingest-sources --force` if you have personally verified every source
file is present and complete.

## MCP Server (Claude Desktop, Cursor, ...)

```bash
pip install "searchat[mcp]"
searchat mcp
```

See `docs/mcp-setup.md` for client configuration.

Available MCP tools:

- `search_conversations`
- `get_conversation`
- `find_similar_conversations`
- `ask_about_history`
- `list_projects`
- `get_statistics`
- `extract_patterns`
- `generate_agent_config`

## Embedded LLM (Local GGUF)

Install the optional embedded dependency and download the default GGUF model:

```bash
pip install "searchat[embedded]"
searchat download-model --activate
```

When `llm.default_provider = "embedded"`, the server will auto-download the default model if `embedded_model_path` is not set.

## Enable Claude Self-Search

Add to `~/.claude/CLAUDE.md`:

````markdown
## Conversation History Search

Search past Claude Code conversations via local API (requires server running).

**Search:**

```bash
curl -s "http://localhost:8000/api/search?q=QUERY&limit=5" | jq '.results[] | {id: .conversation_id, title, snippet}'
```
````

**Get full conversation:**

```bash
curl -s "http://localhost:8000/api/conversation/CONVERSATION_ID" | jq '.messages[] | {role, content: .content[:500]}'
```

**Ask questions (RAG):**

```bash
curl -X POST "http://localhost:8000/api/chat" \
  -H "Content-Type: application/json" \
  -d '{"query": "How did we implement authentication?", "model_provider": "openai", "model_name": "gpt-4.1-mini"}'
```

**Ask questions (RAG, embedded/local):**

```bash
curl -X POST "http://localhost:8000/api/chat" \
  -H "Content-Type: application/json" \
  -d '{"query": "How did we implement authentication?", "model_provider": "embedded"}'
```

**When to use:**

- User asks "did we discuss X before" or "find that conversation about Y"
- Looking for previous solutions to similar problems
- Checking how something was implemented in past sessions
- Asking questions about past work (use RAG chat)

**Start server:** `searchat web` from the searchat directory

````

See `CLAUDE.example.md` for the full template.

## Usage

### Web UI

```bash
searchat web
````

Features:

- **Search Modes:** hybrid/semantic/keyword with autocomplete
- **Filters:** project, date range, tool (agent), similarity search
- **View Conversations:** Full message history with code extraction
- **Bookmarks:** Save and annotate favorite conversations
- **RAG Chat:** Ask questions about your conversation history
- **Analytics:** View search patterns and statistics
- **Saved Queries:** Save and re-run complex searches
- **Dashboards:** Create dashboards and widgets from saved queries
- **Export:** Download conversations in multiple formats
- **Backups:** Create and restore backups (left sidebar)
- **Snapshots:** Switch between active index and read-only backup snapshots
- **Keyboard Shortcuts:** Press `?` to see all shortcuts
- **Terminal Resume:** Resume conversations in terminal
- **Helpful Tips:** Search tips + API integration sidebars

### CLI

```bash
searchat  # interactive mode
searchat web
searchat mcp
searchat setup-index [--force]
searchat rebuild-derived [--force]   # rebuild search indexes from already-indexed data — always safe
searchat reingest-sources [--force]  # re-scan source files and replace the index — guarded, see above

# Download a default embedded GGUF model and update ~/.searchat/config/settings.toml
searchat download-model --activate

# Build the initial index (first-time setup)
searchat-setup-index
```

### API

#### Search & Discovery

```bash
# Search
curl "http://localhost:8000/api/search?q=authentication&mode=hybrid&limit=10"

# Search with tool filter (claude, vibe, opencode)
curl "http://localhost:8000/api/search?q=authentication&tool=claude&limit=10"

# Autocomplete suggestions
curl "http://localhost:8000/api/search/suggestions?q=auth&limit=5"

# Find similar conversations
curl "http://localhost:8000/api/conversation/{conversation_id}/similar?limit=5"

# Optional: request highlight terms for the UI (LLM)
curl "http://localhost:8000/api/search?q=auth&mode=hybrid&highlight=true&highlight_provider=ollama"

# Search with pagination
curl "http://localhost:8000/api/search?q=api&limit=20&offset=0"
```

#### Conversations

```bash
# Get conversation
curl "http://localhost:8000/api/conversation/{conversation_id}"

# List all conversations
curl "http://localhost:8000/api/conversations/all?limit=50"

# Extract code snippets
curl "http://localhost:8000/api/conversation/{conversation_id}/code"

# Code highlighting (Pygments)
curl -X POST "http://localhost:8000/api/code/highlight" \
  -H "Content-Type: application/json" \
  -d '{"blocks":[{"code":"print(123)","language":"python","language_source":"fence"}]}'

# Conversation diff
curl "http://localhost:8000/api/conversation/{conversation_id}/diff?target_id={other_conversation_id}"

# Export conversation
curl "http://localhost:8000/api/conversation/{conversation_id}/export?format=markdown"

# Bulk export
curl -X POST "http://localhost:8000/api/conversations/bulk-export" \
  -H "Content-Type: application/json" \
  -d '{"conversation_ids": ["id1", "id2"], "format": "json"}'

# Resume in terminal
curl -X POST "http://localhost:8000/api/resume?conversation_id={id}"
```

#### Bookmarks

```bash
# List bookmarks
curl "http://localhost:8000/api/bookmarks"

# Add bookmark
curl -X POST "http://localhost:8000/api/bookmarks" \
  -H "Content-Type: application/json" \
  -d '{"conversation_id": "abc123", "notes": "Important auth solution"}'

# Remove bookmark
curl -X DELETE "http://localhost:8000/api/bookmarks/{conversation_id}"
```

#### RAG Chat

```bash
# Ask question about conversation history
curl -X POST "http://localhost:8000/api/chat" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "How did we implement authentication?",
    "model_provider": "openai",
    "model_name": "gpt-4.1-mini",
    "session_id": "optional-session-id"
  }'
# Response includes X-Session-Id header for session tracking

# Non-streaming RAG response with citations
curl -X POST "http://localhost:8000/api/chat-rag" \
  -H "Content-Type: application/json" \
  -d '{"query":"Summarize how backups work","model_provider":"ollama","model_name":"ollama/gemma3"}'

# Streaming response (default)
curl -X POST "http://localhost:8000/api/chat" \
  -H "Content-Type: application/json" \
  -d '{"query": "Explain the API design", "model_provider": "ollama", "model_name": "llama3"}' \
  --no-buffer
```

#### Analytics & Statistics

```bash
# Index statistics
curl "http://localhost:8000/api/statistics"

# Search analytics
curl "http://localhost:8000/api/stats/analytics/summary?days=7"

# Top queries / trends
curl "http://localhost:8000/api/stats/analytics/top-queries?limit=10&days=30"
curl "http://localhost:8000/api/stats/analytics/trends?days=30"

# List projects
curl "http://localhost:8000/api/projects"

# Project summary
curl "http://localhost:8000/api/projects/summary"
```

#### Saved Queries & Dashboards

```bash
# Saved queries
curl "http://localhost:8000/api/queries"

# Dashboards
curl "http://localhost:8000/api/dashboards"
```

#### Tech Docs (Optional)

Requires `export.enable_tech_docs=true`.

```bash
curl -X POST "http://localhost:8000/api/docs/summary" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Searchat Notes",
    "format": "markdown",
    "sections": [
      {"name": "Backups", "query": "backup restore", "mode": "hybrid", "filters": {"project": "myapp"}}
    ]
  }'
```

#### Snapshots (Read-only)

```bash
# Search within a backup snapshot
curl "http://localhost:8000/api/search?q=auth&snapshot=backup_YYYYMMDD_HHMMSS"
```

#### Indexing & Management

```bash
# Watcher status
curl "http://localhost:8000/api/watcher/status"

# Index missing conversations (append-only)
curl -X POST "http://localhost:8000/api/index_missing"

# Safe shutdown (checks for ongoing indexing)
curl -X POST "http://localhost:8000/api/shutdown"

# Force shutdown (override safety check)
curl -X POST "http://localhost:8000/api/shutdown?force=true"
```

#### Backups

```bash
# Create full backup (plaintext)
curl -X POST "http://localhost:8000/api/backup/create"

# Create full backup (encrypted; backups only)
# Requires: SEARCHAT_BACKUP_KEY_B64 (base64-encoded 32 bytes)
curl -X POST "http://localhost:8000/api/backup/create?encrypted=true"

# Create incremental backup (delta) based on a parent backup
curl -X POST "http://localhost:8000/api/backup/incremental/create?parent=backup_YYYYMMDD_HHMMSS"

# Create encrypted incremental backup (parent must also be encrypted)
curl -X POST "http://localhost:8000/api/backup/incremental/create?parent=backup_YYYYMMDD_HHMMSS&encrypted=true"

# List backups
curl "http://localhost:8000/api/backup/list"

# Validate a backup chain and (optionally) verify hashes
curl "http://localhost:8000/api/backup/validate/backup_YYYYMMDD_HHMMSS?verify_hashes=true"

# Inspect ancestry chain (base -> target)
curl "http://localhost:8000/api/backup/chain/backup_YYYYMMDD_HHMMSS"

# Restore backup
curl -X POST "http://localhost:8000/api/backup/restore?backup_name=backup_YYYYMMDD_HHMMSS"

# Delete backup
curl -X DELETE "http://localhost:8000/api/backup/delete/backup_YYYYMMDD_HHMMSS"
```

Notes:

- Snapshot browsing (`?snapshot=...`) only supports snapshot-browsable backups (plaintext full backups). Incremental and encrypted backups are restore-only.
- Incremental backups support chained deltas, with a max total chain length of 10.
- Generate a new encryption key (example): `python -c "import os,base64; print(base64.b64encode(os.urandom(32)).decode())"`

### Utilities

```bash
# Add missing conversations to index
python scripts/index-missing

# Initial setup (interactive, safe options)
python scripts/setup-index

# Convert Vibe plaintext history to searchable sessions
python utils/vibe_converter.py
```

### As Library

```python
from searchat.core.search_engine import SearchEngine
from searchat.config.settings import Config

config = Config.load()
engine = SearchEngine(config.paths.search_directory, config)

results = engine.search("python async", mode="hybrid")
for r in results.results[:5]:
    print(f"{r.title}: {r.score:.3f}")
```

## Architecture

**Code Organization:**

- `src/searchat/api/` - FastAPI app with 14 modular routers (50+ endpoints)
- `src/searchat/core/` - Core indexing and search logic
- `src/searchat/services/` - Business services (chat, bookmarks, analytics, backup)
- `src/searchat/web/` - Modular frontend (HTML + CSS modules + ES6 JS)
- `tests/` - Comprehensive test suite (840+ tests)

**Data Flow:**

```plaintext
~/.claude/projects/**/*.jsonl     (source conversations)
~/.vibe/logs/session/*.json
~/.local/share/opencode/.../*.json
        │
        ▼ index_append_only()
        │
~/.searchat/data/
├── conversations/*.parquet       (DuckDB queryable)
└── indices/
    ├── embeddings.faiss          (semantic vectors)
    ├── embeddings.metadata.parquet
    └── index_metadata.json
```

**Search Flow:**

1. Query → DuckDB FTS keyword search + FAISS semantic search
2. Results merged via Reciprocal Rank Fusion
3. Hybrid ranking returns best of both approaches
4. Optional: Cross-encoder re-ranking of top results
5. Query synonym expansion for keyword matching
6. Optional: Find similar conversations via vector similarity

**RAG Flow:**

1. User question → Search for relevant conversations
2. Top results used as context
3. LLM generates answer with conversation references
4. Streaming response to client

**Live Watching:**

- `watchdog` monitors conversation directories
- New files → indexed immediately
- Modified files → re-indexed after 5min debounce (configurable)
- `index_append_only()` adds to existing index
- Never deletes existing data

**Documentation:**

- `docs/features.md` - Complete feature list and descriptions
- `docs/architecture.md` - System design and components
- `docs/api-reference.md` - Complete API endpoint documentation
- `docs/terminal-launching.md` - Platform-specific terminal launching

## Configuration

Create `~/.searchat/config/settings.toml`:

```toml
[paths]
search_directory = "~/.searchat"
claude_directory_windows = "~/.claude/projects"
claude_directory_wsl = "//wsl$/Ubuntu/home/{username}/.claude/projects"

[indexing]
batch_size = 1000
auto_index = true
reindex_on_modification = true  # Re-index modified conversations
modification_debounce_minutes = 5  # Wait time before re-indexing
enable_connectors = true
enable_adaptive_indexing = true

[search]
default_mode = "hybrid"
max_results = 100
snippet_length = 200

[embedding]
model = "all-MiniLM-L6-v2"
batch_size = 32
device = "auto"  # auto|cuda|mps|cpu

[llm]
default_provider = "ollama"
openai_model = "gpt-4.1-mini"
ollama_model = "ollama/gemma3"

# Embedded (local GGUF via llama-cpp-python)
embedded_model_path = ""
embedded_n_ctx = 4096
embedded_n_threads = 0
embedded_auto_download = true
embedded_default_preset = "qwen2.5-coder-1.5b-instruct-q4_k_m"

[chat]
enable_rag = true
enable_citations = true

[analytics]
enabled = false
retention_days = 30

[export]
enable_ipynb = false
enable_pdf = false
enable_tech_docs = false

[dashboards]
enabled = true

[snapshots]
enabled = true

[performance]
memory_limit_mb = 3000
query_cache_size = 100

[reranking]
enabled = false
model = "cross-encoder/ms-marco-MiniLM-L-6-v2"
top_k = 50

[server]
cors_origins = ["http://localhost:8000", "http://127.0.0.1:8000"]
```

Or use environment variables:

```bash
export SEARCHAT_DATA_DIR=~/.searchat
export SEARCHAT_PORT=8000
export SEARCHAT_EMBEDDING_MODEL=all-MiniLM-L6-v2
export SEARCHAT_REINDEX_ON_MODIFICATION=true
export SEARCHAT_MODIFICATION_DEBOUNCE_MINUTES=5
export SEARCHAT_OPENCODE_DATA_DIR=~/.local/share/opencode
export OPENAI_API_KEY=sk-...  # For RAG chat
export OLLAMA_BASE_URL=http://localhost:11434
```

## Requirements

- Python 3.10+
- ~2-3GB RAM (embeddings model + FAISS index)
- ~10MB disk per 1K conversations
- Optional: OpenAI API key or Ollama for RAG chat

### Dependencies

| Package               | Purpose                       |
| --------------------- | ----------------------------- |
| sentence-transformers | Embeddings (all-MiniLM-L6-v2) |
| faiss-cpu             | Vector similarity search      |
| pyarrow               | Parquet storage               |
| duckdb                | SQL queries on parquet        |
| fastapi + uvicorn     | Web API                       |
| watchdog              | File system monitoring        |
| litellm               | Multi-provider LLM interface  |
| rich                  | CLI formatting                |

## Safety

**Append-only indexing:** Never deletes existing data.

```python
indexer.index_append_only(file_paths)  # Safe: only adds new data
indexer.index_all()                     # Blocked if index exists
indexer.index_all(force=True)           # Explicit override required
```

**Safe shutdown:** Detects ongoing indexing operations.

```bash
# Check status, wait if indexing in progress
curl -X POST "http://localhost:8000/api/shutdown"

# Override safety check (may corrupt data)
curl -X POST "http://localhost:8000/api/shutdown?force=true"
```

**Backups:** Create backups before risky operations.

```bash
# Automatic pre-restore backup before any restore operation
curl -X POST "http://localhost:8000/api/backup/restore" -d '{"name": "backup_20260129_120000"}'
```

Protects against:

- Data loss from deleted/moved source files
- Corrupted Parquet/FAISS files during indexing
- Inconsistent metadata from interrupted operations

## Performance

| Metric            | Value                                              |
| ----------------- | -------------------------------------------------- |
| Search latency    | <100ms (hybrid), <50ms (semantic), <30ms (keyword) |
| Filtered queries  | <20ms (DuckDB predicate pushdown)                  |
| Index build       | ~60s per 100K conversations                        |
| Embedding         | Batched (CPU: 0.1s/conv, GPU: 0.008s/conv)         |
| Memory            | ~2-3GB                                             |
| Startup           | <3s                                                |
| RAG chat response | <2s (with OpenAI), <5s (with Ollama)               |
| Code extraction   | <50ms per conversation                             |
| Similarity search | <100ms (FAISS nearest neighbors)                   |

## Testing

```bash
pytest                          # Run the full test suite
pytest tests/api/              # Run API tests only
pytest -v                      # Verbose output
pytest -k test_search          # Run specific tests
pytest --cov=searchat          # Coverage report
pytest --cov-report=html       # HTML coverage report
searchat validate release      # Run the local pre-release validation matrix
```

**Test Coverage:**

- 840+ tests (API, UI contract tests, unit tests, perf gates)
- ~5,900 lines of test code
- Comprehensive coverage of all features
- API endpoint tests, unit tests, integration tests

See [docs/release.md](docs/release.md) for the Wave 6 release validation flow and packaging gate.

## Troubleshooting

**Port in use:**

```bash
SEARCHAT_PORT=8001 searchat-web
```

**No conversations found:**

```bash
ls ~/.claude/projects/  # Verify conversations exist
```

**WSL not tracked:**
Configure `claude_directory_wsl` in `~/.searchat/config/settings.toml`:

```toml
claude_directory_wsl = "//wsl.localhost/Ubuntu/home/username/.claude/projects"
```

**Missing conversations after setup:**

```bash
python scripts/index-missing  # Index files not yet in search index
```

**Slow on WSL:**
Run from Windows Python or move repo to WSL filesystem (`~/projects/`).

**Import errors:**

```bash
pip install -e . --force-reinstall
```

**Empty environment variables override config:**
Remove empty values from `~/.searchat/config/.env` or set proper values.

**RAG chat not working:**

```bash
# For OpenAI
export OPENAI_API_KEY=sk-...

# For Ollama (local)
ollama serve  # Start Ollama server
export OLLAMA_BASE_URL=http://localhost:11434
```

## Fork Enhancements

Compared with the original upstream repository, this fork adds significant new capabilities:

- **RAG Chat** - AI-powered Q&A over conversation history
- **Session Chat** - Multi-turn RAG with session persistence
- **Bookmarks System** - Save and organize favorite conversations
- **Search Analytics** - Track and analyze search patterns
- **Conversation Similarity** - Discover related conversations
- **Code Extraction** - Extract code snippets with syntax highlighting
- **Saved Queries** - Reusable searches with stored filters
- **Dashboards** - Builder UI + widgets rendered from saved queries
- **Snapshots** - Browse backups as read-only datasets
- **Export Features** - JSON/Markdown/Text exports (optional PDF + Jupyter)
- **Bulk Export** - Export multiple conversations at once
- **Pagination** - Efficient navigation of large result sets
- **Autocomplete** - Smart search suggestions
- **Search History** - Persistent search history
- **Keyboard Shortcuts** - Power user shortcuts
- **Expanded Connector Coverage** - Support for OpenCode, Codex, Gemini CLI, Continue, Cursor, and Aider
- **Tool Filtering** - Filter by specific agent across the broader connector set
- **MCP Server** - Query local history from MCP clients
- **Embedded LLM Support** - Run local GGUF-backed RAG without a hosted provider
- **Expertise Store** - Extract, validate, search, and prime reusable engineering knowledge
- **Knowledge Graph** - Track lineage, contradictions, and record relationships
- **Pattern Mining** - Extract recurring patterns via LLM workflows
- **Agent Config Export** - Generate agent config from patterns and expertise
- **DuckDB FTS** - Replaced BM25 with DuckDB full-text search
- **Cross-Encoder Re-ranking** - Optional result re-ranking
- **Query Synonyms** - Automatic synonym expansion
- **Operational Health Tooling** - Health endpoints, validation commands, CI checks, and release validation flows
- **Modern Typing** - Python 3.12 type hints throughout
- **CORS Hardening** - Configurable CORS origins (default localhost)

See `docs/features.md` for complete feature documentation.

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

## License

MIT
