Metadata-Version: 2.4
Name: loom-pkm
Version: 0.2.0
Summary: Loom — local-first personal knowledge system. Vector + graph retrieval over your vault.
Author-email: Karl Kloss <klosskyle92@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/KlossKarl/loom
Project-URL: Repository, https://github.com/KlossKarl/loom
Project-URL: Issues, https://github.com/KlossKarl/loom/issues
Keywords: pkm,knowledge-base,rag,graphrag,obsidian,local-first
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: End Users/Desktop
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Office/Business
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml>=6.0
Requires-Dist: requests>=2.31.0
Requires-Dist: certifi>=2023.7.22
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: chromadb>=1.0.0
Requires-Dist: neo4j>=5.0.0
Requires-Dist: anthropic>=0.86.0
Requires-Dist: mcp>=1.0.0
Requires-Dist: PyMuPDF>=1.27.0
Requires-Dist: beautifulsoup4>=4.12.0
Requires-Dist: lxml>=5.0.0
Requires-Dist: watchdog>=4.0.0
Requires-Dist: pandas>=2.0.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: tqdm>=4.0.0
Requires-Dist: tenacity>=8.0.0
Requires-Dist: sentence-transformers>=3.0.0
Requires-Dist: pyfiglet>=1.0.0
Provides-Extra: transcribe
Requires-Dist: openai-whisper>=20250625; extra == "transcribe"
Provides-Extra: tray
Requires-Dist: pystray>=0.19.0; extra == "tray"
Requires-Dist: pillow>=10.0.0; extra == "tray"
Requires-Dist: tkinterdnd2>=0.3.0; extra == "tray"
Provides-Extra: lightrag
Requires-Dist: lightrag-hku>=1.4.0; extra == "lightrag"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Dynamic: license-file

# loom

<p align="center">
  <img src="docs/loom_hero.jpg" width="800" alt="Loom - structural memory">
</p>

<p align="center">
  <img src="https://img.shields.io/badge/python-3.10+-58a6ff?style=flat-square&logo=python&logoColor=white" alt="Python">
  <img src="https://img.shields.io/badge/license-MIT-3fb950?style=flat-square" alt="License">
  <img src="https://img.shields.io/badge/platform-Windows%20%7C%20Linux%20%7C%20Mac-161b22?style=flat-square" alt="Platform">
  <img src="https://img.shields.io/badge/local--first-data%20stays%20local-161b22?style=flat-square" alt="Local First">
  <img src="https://img.shields.io/badge/MCP-Claude%20Desktop-58a6ff?style=flat-square" alt="MCP">
  <img src="https://img.shields.io/github/stars/KlossKarl/loom?style=social" alt="Stars">
  <img src="https://img.shields.io/github/last-commit/KlossKarl/loom?style=flat-square" alt="Last Commit">
  <img src="https://img.shields.io/github/commit-activity/m/KlossKarl/loom?style=flat-square" alt="Commits">
  <a href="https://github.com/KlossKarl/loom/actions/workflows/tests.yml"><img src="https://github.com/KlossKarl/loom/actions/workflows/tests.yml/badge.svg" alt="Tests"></a>
</p>

> Loom builds a knowledge graph of your notes and lets you ask questions across all of them. It runs through Claude Desktop, fully on your machine.

**Connect Loom to Claude Desktop and ask your vault anything.** Loom runs as a local [MCP server](#connect-to-claude-desktop-the-primary-way-to-use-loom) that gives Claude cited answers from your notes without anything leaving your machine. Claude Desktop, Cursor, or any MCP client works. → **[Setup below.](#connect-to-claude-desktop-the-primary-way-to-use-loom)**

Point Loom at your notes folder and it indexes everything and maps the relationships between your ideas. Your vault, vector index, and graph stay on your machine. Only the specific passages behind an answer go to Claude for synthesis, and you can skip even that by running a local model instead. Prefer a web UI or the terminal? [Both work.](#install)

**Who this is for:** researchers, lawyers, quants, journalists, and anyone with thousands of documents who needs to reason across all of them privately, on their own machine. If you have 50 notes, a simple LLM-over-files setup works fine. If you have 5,000 and need to find what you forgot you knew two years ago, you need infrastructure underneath. That's Loom.

> I built this because I kept losing things I had already read.

Drop in audio, PDFs, web threads, browser history, or markdown notes. Loom indexes everything and lets you chat with all of it. Synthesis runs through Claude by default, or a local LLM if you want to stay completely offline. Your data never leaves your machine.

Most RAG tools are stateless: they find chunks and forget them. Loom keeps a persistent knowledge graph. Entities and typed relationships accumulate over time, so you can ask structural questions that vector search can't answer no matter how big the context window gets: what connects X and Y, what contradicts this, which notes are related but never linked.

<p align="center">
  <img src="docs/images/demo_clinical.png" width="800" alt="Loom answering a clinical polypharmacy question by retrieving from vault notes and synthesizing with Claude">
</p>

> *Query: "Search my vault first, then answer" — a clinical polypharmacy question about a 74-year-old CHF patient. Loom retrieved passages from her clinical reference notes on Beers Criteria and drug interactions. Claude synthesized across them — flagging the ibuprofen contraindication, oxybutynin's ACB score of 3 as the likely cause of confusion and falls, and the triple AKI risk.*

<p align="center">
  <img src="docs/images/demo_gradient_descent.png" width="800" alt="Loom retrieving across a research vault and identifying a missing note on adaptive optimization methods">
</p>

> *Query across a research vault — Loom pulled from Cornell CS4780 lectures, a standalone SGD note, and papers, then identified a gap: no dedicated note on adaptive methods (Adam, AdaGrad, RMSProp).*

---

## Table of Contents
- [Download](#download)
- [Connect to Claude Desktop (primary)](#connect-to-claude-desktop-the-primary-way-to-use-loom)
- [Quick Start: UI Mode](#quick-start-ui-mode)
- [Why Loom](#why-loom)
- [How Loom Compares](#how-loom-compares)
- [Features](#features)
- [Retrieval Benchmarks](#retrieval-benchmarks)
- [Install](#install)
- [Control Panel](#control-panel)
- [What It Actually Does](#what-it-actually-does)
- [Architecture](#architecture)
- [Loom Capture](#loom-capture)
- [Roadmap](#roadmap)
- [Advanced: GPU Acceleration](#advanced-gpu-acceleration)
- [Contributing](#contributing)

---

## Download

**Windows:** Download [loom-installer.exe](https://github.com/KlossKarl/loom/releases/latest) and run it. No Python or Neo4j required. The installer handles setup and connects Claude Desktop automatically.

**Mac / Linux:** Install from source with pipx:

```bash
git clone https://github.com/KlossKarl/loom
cd loom
LOOM_EMBEDDED=true pipx install .
loom init     # interactive wizard: pick your vault folder
loom chat
```

`loom init` walks you through picking a vault folder, then `loom chat` is live. Full options (Docker, dev install) are in [Install](#install) below. On a Mac? See the [Mac setup guide](docs/mac-setup.md) for the `pip3`/`python3` conventions and SSL/cert notes.

---

## Connect to Claude Desktop (the primary way to use Loom)

Loom runs as a local **MCP server** (`loom_mcp.py`) that connects your vault and knowledge graph to Claude Desktop, Cursor, Continue.dev, or any MCP-compatible client. You ask questions in Claude Desktop; Loom does the retrieval locally and returns cited passages and graph connections.

**Setup (Claude Desktop):**

1. Install Loom from source (see [Install](#install)) and run `loom init` once. The
   wizard auto-detects your OS paths, writes `config.yaml`, and registers Loom in
   Claude Desktop's config automatically on Windows and Mac. The next two steps are
   only needed if you want to wire it up by hand.
2. Add Loom to your Claude Desktop config: `%APPDATA%\Claude\claude_desktop_config.json`
   on Windows, `~/Library/Application Support/Claude/claude_desktop_config.json` on Mac:

```json
{
  "mcpServers": {
    "loom": {
      "command": "python",
      "args": ["C:\\path\\to\\loom\\loom_mcp.py"]
    }
  }
}
```

3. Restart Claude Desktop. Ask: *"Search my vault for anything about knowledge graphs."*

> On Windows, if `python` doesn't resolve in Claude Desktop's environment, use the
> full interpreter path (e.g. `C:\\Python314\\python.exe`). Full instructions (Mac
> paths, the `python` resolution fix, and troubleshooting) are in
> **[docs/claude-desktop-config.md](docs/claude-desktop-config.md)**.

**What stays local:** your vault, the ChromaDB vector index, and the Neo4j graph
never leave your machine. Claude Desktop only ever sees the specific passages Loom
retrieves for a given question. It's the same data flow as any retrieval-augmented chat.

**The 14 tools Claude gets:**

| Tool | What it does |
|------|-------------|
| `search_vault` | Hybrid search (semantic + keyword + graph) with citations |
| `get_entity_neighbors` | Explore the knowledge graph around any concept |
| `get_archaeology_report` | Find notes that are semantically related but never linked |
| `ingest_url` | Add a web URL to your vault |
| `get_vault_stats` | Vault statistics and top concepts |
| `add_fact` | Write an entity or typed relationship into the graph |
| `delete_edge` | Remove a single relationship by edge id |
| `forget_entity` | Remove an entity and all its edges |
| `get_vault_health` | Orphan notes, broken links, stale content, tag hygiene |
| `create_note` | Create a new markdown file in the vault |
| `edit_section` | Edit a specific section of a note by heading |
| `read_section` | Read a specific section of a note by heading |
| `append_to_note` | Append content to an existing note |
| `run_loop` | Trigger a background loop on demand (archaeology, audit, etc.) |

---

## Quick Start: UI Mode

Prefer a browser to Claude Desktop? Loom also exposes an OpenAI-compatible API (`loom_serve.py`), and [Open WebUI](https://github.com/open-webui/open-webui) connects to it for a polished chat experience: citations, model switching, conversation history, no terminal.

<p align="center">
  <img src="docs/loom_ui_demo.png" width="820" alt="Loom answering a graduate-level question about Perceptron convergence and PAC learning, with citations pulled from the user's vault">
</p>

**Prerequisites:** Loom installed (see above), [Docker Desktop](https://www.docker.com/products/docker-desktop) for Open WebUI.

**One-time setup:**

```bash
# 1. Start Open WebUI (one Docker command)
docker run -d -p 3030:8080 --add-host=host.docker.internal:host-gateway ^
  -v open-webui:/app/backend/data --name open-webui --restart always ^
  ghcr.io/open-webui/open-webui:main

# 2. Start the Loom API server
pip install fastapi uvicorn
python loom_serve.py
```

**Connect Open WebUI to Loom (one-time, in the browser):**

1. Open <http://localhost:3030>
2. Admin Settings → Connections → **+** (OpenAI-compatible)
3. URL: `http://host.docker.internal:11435` &nbsp;·&nbsp; Key: `loom`
4. Add model ID `loom` → Save

**Daily use, just run:**

```bash
python loom_serve.py
# then open http://localhost:3030 and pick the "loom" model
```

On Windows, double-click `Loom.bat` in the install directory for one-click launch.

Every reply runs through the full retrieval pipeline (vector + graph + BM25 + HyDE + rerank) and cites the specific vault files behind each answer.

---

## Synthesis Modes

Retrieval (embedding, vector search, BM25, graph traversal, reranking) always runs
**locally and in-process**: `sentence-transformers` + ChromaDB, no external services
required for indexing or search. Only the final answer-generation step needs a model,
and you choose where that runs:

**API mode (recommended).** Set `ANTHROPIC_API_KEY` in a `.env` file at the repo root.
Loom uses Claude for synthesis: fast, high quality, no local GPU required. Because
embedding and retrieval are in-process, this is the lightest possible setup.

**Local mode.** Start Ollama and pull a chat model (`ollama pull deepseek-r1:14b`).
Synthesis runs fully on-device: slower and needs a capable GPU, but zero API cost and
fully air-gapped.

---

## Why Loom

Most LLM-over-notes tools are stateless: they read your files fresh every conversation, re-derive connections on every query, and degrade as your vault grows past a few hundred notes.

Loom is stateful. A persistent vector index and knowledge graph accumulate over time. The relationships between your ideas are stored, traversable, and queryable, not re-derived on every request. When your vault hits 5,000 notes and you need to know what you forgot you knew two years ago, stateless tools can't help. Loom can.

**What Loom does:**
- **Ask your notes anything**: semantic search + knowledge graph traversal with cited answers. Retrieval runs locally; synthesis uses Claude by default (or a local model)
- See what you forgot: weekly reports find notes you read months ago that connect to what you're working on now (Knowledge Archaeology, live)
- Track how your thinking evolved: monthly reports show how your understanding of a topic shifted over time (Temporal Reasoning, live)
- Know where you're thin: weekly reports rank concepts where your coverage is shallow (Epistemic Audit, live)
- **Ingest everything automatically**: drop PDFs, audio, and web URLs into one folder. Loom does the rest.

**What Loom will never do:**
- Store your vault, index, or graph anywhere but your machine
- Require a subscription or a vendor account
- Stop working because a vendor got acquired

The only thing that ever leaves your machine is the handful of retrieved passages sent to Claude for synthesis. Run Ollama locally to keep even that on-device.

**How Loom differs from agent-memory tools.** Mem0, Graphiti, and Cognee store facts extracted from conversation histories. They're optimized for "remember what this user prefers across sessions." Loom is built for document corpora: thousands of notes, papers, transcripts, and web threads you've accumulated over years. The retrieval problem is different, the graph schema is different, and the use case is different. If you need your agent to remember a conversation, use Mem0. If you need to reason across your document library, that's Loom.

---

## How Loom Compares

| | Loom | NotebookLM | Khoj | Smart Connections | InfraNodus | Mem0 |
|---|---|---|---|---|---|---|
| Surfaces forgotten connections | Yes (local) | No | No | Semantic only | Yes (cloud, €12+/mo) | No |
| Knowledge graph | Full entity-relation | No | No | Semantic neighborhoods | Yes | Cloud only ($249/mo) |
| Temporal reasoning | Yes (local) | No | No | No | No | No |
| Knowledge gap detection | Yes (local) | No | No | No | Yes (cloud) | No |
| Local-first (your data stays on device) | Yes | No (Google) | Self-host | Yes (plugin) | No (EU cloud) | No (cloud) |
| Automated intake | Yes | Manual upload | Connectors | No | Import-based | No |
| Works without Obsidian | Yes | Yes | No | No | Partial | Yes |

### How Loom is different from InfraNodus

InfraNodus is the closest competitor: it also surfaces structural gaps and
forgotten connections in a knowledge graph, and it ships as an Obsidian plugin.
The difference is shape, not feature checkboxes. **InfraNodus is cloud SaaS,
single-purpose, and import-based**: you upload a body of text, it analyzes that
snapshot, you read the result. **Loom is local, continuous, and integrated**:
it runs on your machine with no subscription, the graph grows every time you
drop a file into intake, and the analysis lives inside a queryable second brain
you can chat with. You don't export to Loom. Loom *is* where your notes live.

---

## Features

| | Feature | Status |
|---|---|---|
| ⚡ | **Semantic search**: ask your notes anything, get cited answers | ✅ Live |
| 🕸️ | **Knowledge graph**: typed relationships, Neo4j-powered | ✅ Live |
| 🎙️ | **Whisper transcription**: audio and video → indexed notes | ✅ Live |
| 📄 | **PDF extraction**: drop any PDF, it lands in your vault | ✅ Live |
| 🌐 | **Web digest**: Wikipedia, arXiv, HN, SEC, IRS, LessWrong | ✅ Live |
| 📚 | **Topic packs**: 46 curated corpora via `loom pack install` | ✅ Live |
| 📦 | **Loom Capture**: standalone intake daemon, pipx installable | ✅ Live |
| 🖥️ | **Web UI via Open WebUI**: chat interface, no terminal needed | ✅ Live |
| 🏛️ | **Knowledge Archaeology**: weekly report of forgotten connections | ✅ Live (loop) |
| ⏱️ | **Temporal Reasoning**: monthly arc report of concept evolution | ✅ Live (loop) |
| 🧠 | **Epistemic Audit**: weekly knowledge gap analysis | ✅ Live (loop) |
| 🔍 | **Vault health diagnostics**: orphans, broken links, stale notes | ✅ Live |
| ⏰ | **Temporal search lane**: "last week" queries activate recency scoring | ✅ Live |
| 📝 | **Note write tools**: create, edit, append via MCP | ✅ Live |
| 🔄 | **Real-time vault sync**: file watcher re-indexes on save | ✅ Live |
| 🔁 | **Loop runner**: 6 scheduled background analyses | ✅ Live |

---

## Install

> The one-click [Windows installer](#download) is live (v0.2.0). On Mac/Linux, or prefer source? The pipx path below is the quickest, and Docker brings up the full stack. Works on Windows, Mac, and Linux.

> Prefer to inspect the installer before running it? See [Installer Trust](docs/INSTALLER_TRUST.md) — exactly what it touches, how to read it yourself, and how to pin to a release.

**1. Claude Desktop (MCP): the recommended way to use Loom**

Install Loom from source (the pipx step below is fine), then wire it into Claude
Desktop and ask your vault questions right from the chat. Full walkthrough:
[Connect to Claude Desktop](#connect-to-claude-desktop-the-primary-way-to-use-loom).

```bash
git clone https://github.com/KlossKarl/loom
cd loom
LOOM_EMBEDDED=true pipx install .   # or: pip install -e .
loom init                           # creates config.yaml (required before connecting)
# then add loom to claude_desktop_config.json (see the section linked above)
```

**2. Docker: full stack, no manual dependency setup**

Brings up the vector store, graph database, and local LLM with no manual
Python/Neo4j/Ollama install:

```bash
git clone https://github.com/KlossKarl/loom
cd loom

# 1. Configure: copy the env template and set your vault path
cp .env.example .env
# then edit .env: LOOM_VAULT_PATH is required (point it at your notes folder)

# 2. CPU-only or Mac? Comment out the `deploy: resources: ... nvidia` block
#    in docker-compose.yml first, or the Ollama container will fail to start.

# 3. Bring up the services (Neo4j + Ollama + Loom)
docker compose up -d
```

`docker compose up` starts the services but does **not** index or chat on its
own. The Loom container idles until you run commands against it:

```bash
docker compose run --rm loom python src/second_brain/second_brain.py --index
docker compose run --rm loom python src/second_brain/second_brain.py --chat
```

**3. pipx: quickest start (no Neo4j, no Ollama required)**

```bash
LOOM_EMBEDDED=true pipx install .   # skips Neo4j; vector search only, graph features unlock later
loom init
loom chat
```

Embedded mode skips Neo4j entirely. You get full vector search immediately,
and graph features unlock when you're ready.

**4. Development install (from source)**

```bash
git clone https://github.com/KlossKarl/loom
cd loom
pip install -e .
LOOM_EMBEDDED=true loom init
loom chat
```

Drop files into `intake/`, then ask your notes anything.

### Gotchas

- **First graph index is slow**: the `--graph-index` pass uses your local LLM (or Claude Haiku) to extract entities from every file. On a mid-range GPU, expect it to run overnight. Vector index is fast (~10 minutes). For a free, fast first pass, `--quick-graph-index` uses [GLiNER v2.1](https://github.com/urchade/GLiNER) locally. See below.
- **Tested on Windows**: Loom uses `Path()` throughout and should work on Mac/Linux, but it's only been tested on Windows. PRs welcome.
- **Neo4j is optional**: `LOOM_EMBEDDED=true` mode skips Neo4j entirely. You get full vector search without it. Graph features unlock when you're ready.

**Entity extraction: local and free, or Claude Haiku.** The knowledge graph is
built by `--graph-index` (Claude Haiku via the Anthropic API when a key is set,
falling back to your local `chat_model`) or by `--quick-graph-index`, which uses
[GLiNER v2.1](https://github.com/urchade/GLiNER) (`urchade/gliner_multi-v2.1`),
a 209M-param local NER model with zero API cost that runs entirely on your
machine. See [docs/usage.md](docs/usage.md#entity-extraction-llm-vs-gliner) for
the trade-offs.

---

## Control Panel

Loom ships with a desktop control panel (`src/whisper/intake_tray.py`):

- **Intake tab**: drag-and-drop files + YouTube URL queue
- **Search tab**: vector and graph search against your vault
- **Chat tab**: persistent local LLM conversation
- **Status tab**: vault stats, watcher status, index runner

Runs in the system tray. Start it with:

```bash
python src/whisper/intake_tray.py
```

The Windows installer launches this automatically on first run.

---

## What it actually does

Drop a file into the intake folder. It routes itself.

```
lecture.mp3          ->  Whisper transcription -> vault -> indexed
paper.pdf            ->  PDF to markdown -> vault -> indexed
thread.txt           ->  Web digest -> structured note -> vault -> indexed  
https://...url       ->  Same as above
note.md              ->  Copied directly to vault -> indexed
```

Then ask questions:

```
> what did the stanford cs229 lecture say about attention mechanisms?
> compare the risk frameworks across my last 5 papers
> what connects OODA loop to predictive coding?
> find everything I've read about CLO structures
```

Retrieves from ChromaDB (vector search), traverses Neo4j (knowledge graph), and generates the answer with Claude by default (or a local LLM via Ollama). Indexing, retrieval, and your data stay on your hardware.

---

## Example: 30 Papers on RAG Over 6 Months

You've been reading about retrieval-augmented generation for months. Papers, blog posts, podcast transcripts, HN threads. All dropped into `intake/` as they came in. You never organized them.

Six months later, you need to write a synthesis. You type:

```bash
loom chat
> What are the main approaches to RAG and how do they compare?
```

Loom searches the vector index, walks the knowledge graph, and returns an answer citing 14 of your notes across 8 sources, including a podcast transcript from January you completely forgot about, and a connection between two papers you never would have made manually.

The answer includes citations back to the exact vault files. You click through, verify, and start writing. The system got smarter while you weren't looking, because the graph accumulated relationships every time you fed it something new.

That's what stateful retrieval means. A stateless tool would have started from scratch.

---

## Architecture

```
intake/                     <- drop anything here
    |
src/whisper/intake_watcher.py   <- watches folder, routes by file type
    |
+--------------------------------------------------+
|  transcribe.py   pdf_to_md.py   web_digest.py   |
|        Whisper      pymupdf        Claude Code   |
+--------------------------------------------------+
    |
Obsidian Vault              <- all content lands here as markdown
    |
src/second_brain/second_brain.py --index         <- chunks + embeds into ChromaDB
src/second_brain/second_brain.py --graph-index   <- extracts entities/relationships into Neo4j
    |
    +--> src/second_brain/second_brain.py --chat <- CLI: hybrid retrieval (vector + graph + HyDE + rerank)
    |
    +--> loom_serve.py                            <- OpenAI-compatible HTTP API
              |
              +--> Open WebUI (browser)           <- full chat UI with citations
              +--> Continue.dev / any OpenAI client
```

```mermaid
graph LR
    A[📁 intake/] --> B[Folder Watcher]
    C[🎙️ Audio] --> D[Whisper]
    E[📄 PDF] --> F[PyMuPDF]
    G[🌐 Web URL] --> H[Web Digest]
    B --> I[(ChromaDB\nVector Index)]
    D --> I
    F --> I
    H --> I
    I --> J[Neo4j\nKnowledge Graph]
    I --> K[💬 loom chat]
    J --> K
    I --> M[loom_serve.py\nOpenAI-compatible API]
    J --> M
    M --> N[🖥️ Open WebUI]
    K --> L[📝 Obsidian Vault]
```

For details on the knowledge graph schema, see [docs/graph_schema.md](docs/graph_schema.md).

---

## What's actually different

Loom is opinionated about graph schema and entity resolution. It trades flexibility for long-term coherence: every relationship is typed, every entity is resolved, and the graph gets smarter the more you feed it.

A few specific things, since "local-first RAG" is a crowded space.

**Constrained typed relationships, not unconstrained predicate generation.** The Neo4j schema uses a fixed set of relationship types: CITES, INFLUENCES, EXTENDS, CONTRASTS_WITH, UNCERTAIN_SAME_AS, UNTYPED_RELATION, CO_OCCURS_WITH, and REFERS_TO. Anything the LLM tries to emit outside that set is rewritten to UNTYPED_RELATION with the original predicate preserved on `r.raw_type`. This is more restrictive than letting the model invent edge types, but the graph stays coherent at scale instead of fragmenting into thousands of one-off predicate names. The validation happens at the graph write layer, not just in the prompt.

**Evidence-backed graph extraction.** Every semantic edge (and every `MENTIONS` link from a chunk to an entity) carries an `evidence_span` (a ≤200-char verbatim quote from the source chunk) and an `edge_confidence` score. Every `:Entity` node carries an `extraction_confidence` (max of all scores ever seen for it). You can ask "where did this come from?" and get the actual line of text the LLM was looking at when it made the claim. Most graph-RAG tools throw this provenance away the moment extraction finishes.

**Entity resolution with canonical keys + aliases.** "PAC-learning", "PAC learning", and "pac learning" all collapse to the same canonical key in the graph (lowercase, hyphens → spaces, stripped possessives/articles/accents, conservative plural strip), with the original surface forms preserved as Alias nodes linked via HAS_ALIAS. This handles the entity dedup problem most LLM-extracted graphs ignore. Without it, the graph fills with near-duplicate nodes and cross-document traversal breaks down.

**Wikilink-aware graph.** Obsidian `[[wikilinks]]` between vault notes are extracted and written as Document→Document `REFERS_TO` edges: user-curated structure, no LLM call. The graph respects the connections you drew by hand, not just the ones a model inferred.

**Adaptive query routing, not blind hybrid retrieval.** Most personal RAG tools run the same retrieval pipeline regardless of query type. This one classifies the query first (semantic, relational, or hybrid), routes to the appropriate store (ChromaDB, Neo4j, or both), then runs a sufficiency check and loops up to 3 times if context is insufficient. The router falls back to vector if graph comes up empty, or expands into graph if vector results don't answer the question. The route taken is logged so you can see how the system is thinking.

**Cross-document queries vector search cannot answer.** Because entities are shared nodes across documents, you can traverse Document -> Chunk -> Entity <- Chunk <- Document to find pairs of documents that both reference the same concept. That's a single Cypher traversal. Pure vector RAG cannot answer this structurally no matter how big the context window gets.

---

## Retrieval Benchmarks

Loom ships with a reproducible retrieval-evaluation harness (`eval/`). It samples
50 factual chunks from the vault, uses Claude to generate 150 questions whose
answers are stated in those chunks, then runs every question through five
retrieval modes built from Loom's own `retrieve()` primitives and scores
Recall@5/@10 and MRR@10.

Latest run: 150 queries, 18,180-chunk corpus, `mxbai-embed-large`:

| Mode | Recall@5 | Recall@10 | MRR@10 |
|------|---------:|----------:|-------:|
| BM25 only | 0.773 | 0.827 | 0.648 |
| Vector only | 0.653 | 0.720 | 0.468 |
| Hybrid (BM25 + vector + RRF) | 0.760 | 0.853 | 0.589 |
| Hybrid + rerank | **0.940** | **0.947** | **0.851** |
| Hybrid + rerank + HyDE | _pending_ | _pending_ | _pending_ |

> **Config note:** Reranking is enabled by default (`rerank: true`) and produces the Recall@10
> 0.947 result. A GPU is strongly recommended (~257ms/query on an RTX 4070 Ti, fp16); CPU latency
> is ~14s/query. GPU machines need a CUDA build of torch (e.g. the `cu126` wheel). Set
> `rerank: false` if running CPU-only (Recall@10 falls to 0.853 but queries stay ~0.2s).

Honest takeaways: **the reranker upgrade to `bge-reranker-v2-m3` now wins every
column.** The previous small cross-encoder (`ms-marco-MiniLM-L-6-v2`) regressed on
this corpus: it truncated at 512 tokens, and Loom's chunks are longer, so it
reordered the fused pool *worse* than RRF (Recall@10 dropped to 0.673). Swapping
in `bge-reranker-v2-m3` (1024-token window, multilingual XLM-RoBERTa) flips that:
Recall@10 climbs from the 0.853 no-rerank baseline to **0.947**, and MRR@10 from
0.589 to **0.851** (+26.2 pts). BM25 remains a remarkably strong baseline on this
query distribution (the eval questions reuse source-passage vocabulary, which
favours lexical matching), but reranking the fused pool now clearly sharpens the
top. The cost is latency: the larger model runs ~14 s/query on CPU (these numbers
were measured on a CPU-only PyTorch build, so the reranker ran on CPU even though
the box has a GPU), versus ~0.2 s for plain Hybrid. The **Hybrid + rerank + HyDE** row is
pending a valid `ANTHROPIC_API_KEY`. That mode makes one Claude call per query, so
it only runs when a working key is configured.

Full methodology, analysis, and reproduction steps: **[eval/results.md](eval/results.md)**.

```bash
python eval/sample_chunks.py   # sample factual chunks (seeded, reproducible)
python eval/gen_queries.py     # Claude generates 3 Qs/chunk -> eval/queries.jsonl
python eval/run_eval.py        # score all modes -> eval/results.md
```

---

## Requirements

> This list is for source installs. The [Windows installer](#download) (v0.2.0, live) bundles Python and handles setup automatically.

- **Python 3.10+**
- **[Ollama](https://ollama.ai)** - local LLM inference for chat. Pull a model based on your hardware:

  | Profile | VRAM | Chat model | Quality |
  |---------|------|------------|---------|
  | Budget | 4-8GB | `ollama pull llama3:8b` | Good for chat, basic graph |
  | Mid (default) | 8-16GB | `ollama pull deepseek-r1:14b` | Solid all-around |
  | High | 16-24GB | `ollama pull deepseek-r1:32b` | Better local graph extraction |
  | Workstation | 48GB+ | `ollama pull llama3:70b` | Near-frontier quality |

  Embeddings run in-process via `sentence-transformers` (current default `mxbai-embed-large`). No separate Ollama pull required. For new installs we recommend `qwen3-embedding`, which leads current retrieval benchmarks; `mxbai-embed-large` remains a solid fallback.

  **Graph extraction defaults to Claude Haiku via the Anthropic API** when `anthropic.api_key` is set in `config.yaml`. This is the recommended path: it's faster, more accurate, has prompt caching (~5× cost reduction), and costs about $5 for a full 18K-chunk index. If no API key is configured, Loom falls back to your local `chat_model` for extraction.
- **[Neo4j Desktop](https://neo4j.com/download/)** - knowledge graph database
  - Free, but requires manual setup (see below)
- **[Obsidian](https://obsidian.md)** - vault is just a folder of markdown, Obsidian is optional but recommended
- **[Claude Code](https://claude.ai/code)** - used for free-tier web digest processing (optional but recommended)
- Decent hardware. 16GB RAM minimum. A GPU with 8GB+ VRAM makes graph indexing significantly faster.

---

## Loom Capture

If you only want the intake half (automatic Whisper transcription, PDF → markdown, web/HN/Reddit/Wikipedia digests, all dropping into any folder you point at), there's a standalone product:

```bash
pipx install loom-capture
loom-capture init
loom-capture watch
```

→ [loom-capture/](./loom-capture/). Free, MIT, no API key, no vector DB, no graph. Works with Obsidian, Logseq, or any folder of markdown files.

### What a day with Loom Capture looks like

Morning: you listen to a podcast and drop the mp3 into your vault. Capture transcribes it via Whisper and files it as markdown.

Afternoon: you paste three URLs into a text file in `intake/`: an arXiv paper, a Wikipedia article, and an HN thread. Capture digests all three into clean markdown notes.

Evening: you open Obsidian and everything is there, searchable, formatted, filed. You did zero manual work.

---

## Manual config

The `loom init` wizard handles this for you, but if you prefer to set things up by hand:

```bash
cp config.template.yaml config.yaml
# edit config.yaml with your paths
```

**Neo4j setup** (only needed for the full stack, not embedded mode):

1. Install Neo4j Desktop
2. Create a new Project, add a Local DBMS
3. Set a password, start the instance
4. Put the password in `config.yaml` under `second_brain.neo4j_password`

---

## config.yaml reference

```yaml
paths:
  obsidian_vault: C:\Users\you\Documents\Obsidian Vault
  chroma_dir: C:\Users\you\Documents\second_brain_db

second_brain:
  vaults:
    - C:\Users\you\Documents\Obsidian Vault
  
  embed_model: mxbai-embed-large
  chat_model: deepseek-r1:14b
  
  neo4j_uri: neo4j://127.0.0.1:7687
  neo4j_password: yourpassword
  
  # customize entity types for your domain
  entity_types:
    - Person
    - Concept
    - Method
    - Paper
    - Organization
    - Dataset

intake:
  folder: C:\Users\you\Documents\loom\intake
  auto_index: true
  web_digest_free: true    # true = Claude Code (free), false = Anthropic API
```

---

## Usage

The fastest path is the intake watcher. Run it once and drop files into
`intake/`; they get transcribed/digested and indexed automatically:

```bash
python src/whisper/intake_watcher.py
```

The full command reference covers indexing, graph building (LLM and GLiNER),
chat, web digests, transcription, research-source batches, and the in-chat
retrieval toggles. See **[docs/usage.md](docs/usage.md)**.

---

## Pre-built topic packs

loom ships with 46 pre-built semantic corpora, curated knowledge bases (AI,
quant finance, mathematics, law, philosophy, and more) you can ingest in a few
hours to start with a connected, queryable foundation instead of an empty vault.

List and install them straight from the CLI:

```bash
loom pack list                 # show all 46 packs with URL and install counts
loom pack install ai_agents    # ingest a pack via the web-digest pipeline, then index
```

Installs are resumable, so a failed URL or an interrupted run picks up where it
left off.

→ See [docs/topic-packs.md](docs/topic-packs.md) for the full list and usage.

---

## Known issues

- Tested on Windows. Paths use `Path()` throughout so it should work on Mac/Linux, but that hasn't been tested. PRs welcome.
- Graph indexing is slow on large vaults: roughly 0.5 seconds per chunk on the current default hardware profile. For a vault with thousands of files this means running overnight. Batched UNWIND writes (Phase 2) will cut this significantly on multi-core machines.
- ~~No web UI.~~ Web UI is now live via `loom_serve.py` + Open WebUI. See [Quick Start: UI Mode](#quick-start-ui-mode).

---

## Roadmap

**P0: Onboarding (shipped)**
- [x] docker-compose.yml: one command full stack
- [x] LOOM_EMBEDDED=true: zero-dependency first run
- [x] pipx installable: `pipx install .`
- [x] `loom init` wizard: OS-aware path detection, auto-configures Claude Desktop
- [x] One-click Windows installer (v0.2.0): bundles Python, no Neo4j required
- [x] `loom pack list` / `loom pack install`: 46 curated topic packs from the CLI

**P1: Loom Capture (shipped)**
- [x] Standalone intake pipeline: `pipx install loom-capture`
- [x] Three commands: `loom-capture init`, `watch`, `digest`

**Extraction Quality: Phase 0 + Phase 1 (shipped 2026-05-20)**
- [x] Claude Haiku as default graph extraction model (with prompt caching, ~5× cost reduction)
- [x] Evidence spans + confidence scores on every entity and every semantic/MENTIONS edge
- [x] Typed exception handling + quality metrics summary at end of every `--graph-index` run
- [x] `UNCERTAIN_SAME_AS` / `UNTYPED_RELATION` split (retires `POSSIBLY_SAME_AS`)
- [x] Hallucination guard: entity names validated against source text
- [x] Obsidian `[[wikilinks]]` → Document→Document `REFERS_TO` edges
- [x] In-process sentence-transformers embedding (eliminates the old Ollama HTTP 500s under load)
- [x] BM25 disk cache (avoids cold rebuild every session)
- [x] `ingested_at` on every graph node: temporal foundation

**Extraction Quality: Phase 2 (pending, new hardware)**
- [ ] Batched UNWIND Neo4j writes: utilises full CPU core count
- [x] Reranker upgrade to `bge-reranker-v2-m3` (Recall@10 0.853 → 0.947, now on by default)
- [ ] Degree cap on graph traversal (hub-node protection)
- [ ] Retrieval deduplication
- [ ] First full clean graph index run on the new PC

**Performance**
- [ ] Python 3.12 venv for CUDA-accelerated reranking (current PyTorch has no Python 3.14 CUDA builds; CPU reranking works at ~10s/query which is acceptable for local use)

**P2: Next Generation**
- [x] Knowledge Archaeology: weekly loop surfaces forgotten notes ✅ shipped
- [x] Epistemic Audit: weekly knowledge gap report ✅ shipped
- [x] Temporal Reasoning: monthly arc report of how your thinking evolved ✅ shipped
- [x] `loom_serve.py`: OpenAI-compatible HTTP API (Open WebUI, Continue.dev) ✅
- [ ] Argument layer: extract claims/evidence/debate structure into the graph
- [ ] Graph visualization UI
- [ ] Mac/Linux testing and fixes
- [ ] Browser extension for web digest

**Infrastructure**
- [ ] SQLite-graph backend: replace Neo4j with embedded SQLite graph (stdlib, zero server, zero JVM). FalkorDB spike completed and failed gate (server-only). SQLite spike spec written. This eliminates the last install friction point for the full graph stack.

---

## Project structure

```
loom/
├── src/                  # all Python modules
│   ├── whisper/          # audio/video to markdown + intake watcher/tray
│   ├── second_brain/     # core: index, chat, graph
│   └── web_digest/       # all ingestion scripts + topic files
├── experiments/          # exploratory scripts (lightrag_test, etc.)
├── docs/                 # guides, ADRs, usage, topic packs
├── config.template.yaml  # starting point, copy to config.yaml
└── config.yaml           # your config (gitignored)
```

---

## Support

If Loom is useful to you, a ⭐ on GitHub helps more people find it.

Questions or ideas? [Open an issue](https://github.com/KlossKarl/loom/issues).

---

## Advanced: GPU Acceleration

The installer uses CPU-only PyTorch by default (~200 MB). For GPU-accelerated embeddings, install CUDA torch manually:

```bash
pip install torch --index-url https://download.pytorch.org/whl/cu121
```

Note: Loom's LLM inference (chat, analysis, graph extraction) runs through Ollama, which manages GPU access independently. CUDA torch only affects the embedding model (sentence-transformers).

### Benchmarking Embedding Models

To compare embedding models on your actual vault content:

```bash
python experiments/embedding_benchmark.py --sample 500
```

Samples chunks from your existing ChromaDB collection, re-embeds them with each candidate model, runs a set of queries against each, and reports latency + retrieval results. Defaults to comparing `mxbai-embed-large`, `nomic-embed-text-v1.5`, and `bge-m3`. Results saved to `experiments/embedding_benchmark_results.json`.

---

## Contributing

Loom is a solo project but issues and PRs are welcome.

- **Bug reports:** open a GitHub issue with steps to reproduce
- **Feature requests:** check the roadmap first, then open an issue
- **Pull requests:** keep them focused. One thing per PR

---

## License

MIT
