Metadata-Version: 2.5
Name: archiver-rag
Version: 0.2.0
Summary: Agent-agnostic semantic memory for your vault. Turns notes into a queryable knowledge graph any MCP agent can search, log to, and reorganize.
Project-URL: Homepage, https://fernandojrr.github.io/archiver-rag
Project-URL: Repository, https://github.com/FernandoJRR/archiver-rag
Project-URL: Issues, https://github.com/FernandoJRR/archiver-rag/issues
Project-URL: Changelog, https://github.com/FernandoJRR/archiver-rag/blob/main/CHANGELOG.md
Author-email: FernandoJRR <fercitotron@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: ai-agents,chromadb,embeddings,knowledge-graph,mcp,obsidian,rag,semantic-search
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.11
Requires-Dist: chromadb<2,>=1.5
Requires-Dist: click<9,>=8
Requires-Dist: mcp<2,>=1.9
Requires-Dist: numpy
Requires-Dist: platformdirs
Requires-Dist: pyyaml
Requires-Dist: rich
Requires-Dist: sentence-transformers<6,>=5
Requires-Dist: starlette>=1.2
Requires-Dist: typer<0.26,>=0.25
Requires-Dist: uvicorn>=0.49
Requires-Dist: watchdog<7,>=6
Description-Content-Type: text/markdown

<p align="center">
  <img src="https://raw.githubusercontent.com/FernandoJRR/archiver-rag/main/assets/archiver-rag-lockup.svg" alt="Archiver RAG" width="320" />
</p>

<p align="center">
  <em>A finding aid for your knowledge graph</em>
</p>

<p align="center">
  The agent-agnostic memory management system for your Obsidian vault
</p>

<p align="center">
  <a href="https://github.com/FernandoJRR/archiver-rag/actions/workflows/test.yml"><img src="https://github.com/FernandoJRR/archiver-rag/actions/workflows/test.yml/badge.svg" alt="tests" /></a>
  <a href="https://pypi.org/project/archiver-rag/"><img src="https://img.shields.io/pypi/v/archiver-rag" alt="PyPI version" /></a>
</p>

> **Beta.** archiver-rag is under active development and I'm looking for feedback.
> Tools and config may change before 1.0; see the [changelog](https://github.com/FernandoJRR/archiver-rag/blob/main/CHANGELOG.md).
> Bugs, ideas, rough edges: [open an issue](https://github.com/FernandoJRR/archiver-rag/issues/new/choose).

Archiver RAG turns your Obsidian vault into a live, queryable knowledge graph that any MCP-compatible AI agent can search, update, and reorganize — without ever leaving its native interface.

Connect it once. Every agent you use (Claude Code, Cursor, Gemini CLI, or your own) gets semantic search, automatic knowledge logging, wikilink-aware graph traversal, and vault health monitoring out of the box.

---

## How it works

```
Your Obsidian vault (.md files)
         ↓  file watcher + ingest pipeline
     ChromaDB  (persistent vector store)
         ↓  MCP server
     Any MCP-compatible agent
```

Three layers make search smarter than plain embeddings:

1. **Contextual prefix** — each chunk is embedded with its note's metadata (folder, tags, wikilinks), so vectors carry structural context
2. **Rich metadata filtering** — ChromaDB stores folder, type, tags, incoming link count, and wikilinks for filtered retrieval
3. **Graph reranking** — after vector search, results are re-scored by wikilink proximity to a context note and hub importance

The file watcher runs as a background service. Edit a note in Obsidian, save it, and it's indexed and auto-linked within seconds — no manual sync needed.

---

## Features

- **Semantic search** with graph reranking — finds notes by meaning, then boosts results connected via wikilinks
- **Adaptive auto-linking** — after every ingest, the `## Related` section is rebuilt with `[[wikilinks]]` to every note that clears a relevance margin of the top match — the list grows and shrinks with the graph instead of accumulating
- **Knowledge logging** — create categorized notes (`decision`, `lesson`, `gotcha`, `pattern`, …) from any agent; date lives in frontmatter, filename is the slug identity wikilinks are written against
- **Semantic folder placement** — folders declare a description (`_folder.md`, generated by `archiver-rag describe` or auto-regenerated by the watcher); new notes are scored against those descriptions and moved to the best match above a threshold, falling back to the note's frontmatter `type:` field
- **Soft delete** — `archiver-rag delete` moves notes to `.trash/` (Obsidian's delete convention, recoverable) and sweeps inbound `[[wikilinks]]` from the rest of the vault
- **Vault health** — single call returns orphaned notes, broken links, missing frontmatter, tag stats, and recent activity
- **Wikilink-aware reorganization** — move files and every `[[link]]` across the vault is rewritten automatically
- **Smart clustering** — label-propagation algorithm groups notes by wikilink structure and suggests folder organization (manual command — the watcher runs semantic placement, not full-graph clustering)
- **Inbox routing (opt-in)** — when no folder claims a note, it can be parked in `inbox/` and a new real folder is spun out automatically once a group of embedding-similar notes accumulates there (off by default)
- **Agent-agnostic** — exposes a standard MCP interface over stdio or HTTP; works with any MCP-compatible client

---

## Requirements

- Python >= 3.11
- [pipx](https://pipx.pypa.io/) or [uv](https://docs.astral.sh/uv/) for installation
- An Obsidian vault (local `.md` files)
- An MCP-compatible agent (Claude Code, Cursor, etc.)

---

## Installation

With pipx:

```bash
pipx install archiver-rag
```

Or with uv:

```bash
uv tool install archiver-rag
```

If your system Python is older than 3.11, uv can fetch a suitable one: `uv tool install --python 3.12 archiver-rag` (pipx: `pipx install --python python3.12 archiver-rag`, with that Python already installed).

> Use `pipx` or `uv tool`, not `pip install` — both create an isolated environment and expose the CLI globally on `PATH`, which is required for MCP registration to find the correct executable.

For local development from a clone of this repo, use `pipx install --editable .` instead.

---

## Development

```bash
git clone https://github.com/FernandoJRR/archiver-rag && cd archiver-rag
pipx install --editable .   # global CLI — required for MCP registration
pip install --group dev -e .  # adds pytest, ruff
pytest                      # 447 tests, ~5 s
```

Tests marked `slow` load the sentence-transformers model; skip them with `-m "not slow"`.

`tests/` layout (highlights — see `AGENTS.md` for the full file-by-file list):
- `conftest.py` — `_no_real_vault` (autouse): patches `get_vault_path` in every module that imports it, so no test ever touches the real vault. `_no_real_home_paths` (autouse): redirects `archiver_rag.paths`' config/data/cache dirs so no test ever touches the real `~/.config/archiver-rag/`, `~/.local/share/archiver-rag/`, or `~/.cache/archiver-rag/`. Opt-in `tmp_vault` and `tmp_install` fixtures for tests that need real files.
- `test_wikilinks.py` — 28 unit tests for the offset-based wikilink extractor
- `test_linker_section.py` — 11 characterization tests for `_append_links_section`

---

## Setup

Run the one-time setup wizard:

```bash
archiver-rag init
```

This will:
1. Ask for your vault path
2. Index your vault into ChromaDB
3. Register the MCP server in `~/.claude.json` (or prompt you to do it manually for other clients)
4. Install the background watcher as a launchd agent (Mac) or systemd service (Linux)

---

## MCP registration (manual)

If you prefer to register manually, add this to your MCP client config:

```json
{
  "mcpServers": {
    "archiver-rag": {
      "command": "/path/to/archiver-rag",
      "args": ["serve"]
    }
  }
}
```

Find the executable path with `which archiver-rag`.

For Claude Code specifically, use:

```bash
claude mcp add --scope user archiver-rag $(which archiver-rag) serve
```

---

## HTTP transport

By default the server speaks MCP over **stdio**: your client spawns it as a child
process. That means one client per server, and no way to reach the vault from another
machine. `--transport http` serves the same six tools over streamable HTTP instead, so
several clients can share one warm process:

```bash
archiver-rag serve --transport http            # foreground, http://127.0.0.1:8077/mcp
```

Or run it as a supervised background daemon (launchd on Mac, systemd on Linux) instead of holding a terminal:

```bash
archiver-rag start http          # installs + starts the daemon (asks about auto-start at login)
archiver-rag stop http           # stop it (settings are kept)
archiver-rag restart http        # restart
archiver-rag status              # includes the HTTP daemon's state and URL
```

`start http` refuses to start if the port is already bound, so it can never silently crash-loop behind a busy foreground `serve`.

Register it with Claude Code:

```bash
claude mcp add --scope user --transport http archiver-rag http://127.0.0.1:8077/mcp
```

### Other clients

Both transports work with any MCP-compatible client. Replace `/path/to/archiver-rag`
with the output of `which archiver-rag`.

#### opencode

Add an `mcp` block to `~/.config/opencode/opencode.jsonc` (or `opencode.json`, or a
project-local file of either name — opencode's schema allows comments and trailing
commas in both):

```jsonc
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    // stdio — opencode launches the server itself
    "archiver-rag": {
      "type": "local",
      "command": ["/path/to/archiver-rag", "serve"],
      "enabled": true,
      "timeout": 30000
    },
    // HTTP — connects to a server you started separately
    "archiver-rag-http": {
      "type": "remote",
      "url": "http://127.0.0.1:8077/mcp",
      "enabled": true,
      "timeout": 30000
    }
  }
}
```

Use one or the other — both are shown here only to give the shape of each.

> **Raise `timeout`.** opencode defaults to 5000 ms per request, and the *first*
> `search_vault` call loads the embedding model — measured at ~5 s on a warm disk, right
> at the limit. Without a higher timeout the first search may fail and then succeed on a
> retry, which is a confusing way to meet the tool. Later calls take ~70 ms.

#### Codex CLI

```bash
# stdio — Codex launches the server itself
codex mcp add archiver-rag -- $(which archiver-rag) serve

# HTTP — connects to a server you started separately
codex mcp add archiver-rag-http --url http://127.0.0.1:8077/mcp
```

Both write to `~/.codex/config.toml`, and you can equally hand-edit it:

```toml
[mcp_servers.archiver-rag]
command = "/path/to/archiver-rag"
args = ["serve"]

[mcp_servers.archiver-rag-http]
url = "http://127.0.0.1:8077/mcp"
```

Verify with `codex mcp list`. If your server sits behind a proxy that wants a token,
`--bearer-token-env-var NAME` reads it from the environment — archiver-rag itself never
checks it (see below).

### Reaching it from another machine

> **archiver-rag performs no authentication and terminates no TLS.** Anyone who can
> reach the port has full access: the entire vault is readable, and `log_note` and
> `move_notes` can modify it.

It is deliberately not this tool's job to decide how you secure that. Keep the server on
loopback and put a layer you already trust in front of it — a reverse proxy terminating
TLS, an SSH tunnel, a VPN, or a private overlay network. The server does not need to know
which; it stays on plain HTTP at `127.0.0.1:8077` in every case.

If you bind beyond loopback (`--host 0.0.0.0`), the CLI prints a warning — heed it. You
can additionally enable DNS-rebinding protection by naming the hostnames you expect to
serve:

```bash
archiver-rag serve --transport http --allowed-host vault.internal.example
```

Requests arriving with any other `Host` header are rejected with `421`. The bind address
itself is always allowed, so local access keeps working.

### Options

| Flag | Default | Meaning |
|---|---|---|
| `--transport` | `stdio` | `stdio` or `http` |
| `--host` | `127.0.0.1` | Bind address |
| `--port` | `8077` | Bind port |
| `--path` | `/mcp` | HTTP route |
| `--allowed-host` | *(none)* | Enable DNS-rebinding protection for this Host (repeatable) |
| `--stateful` | off | Use HTTP sessions + SSE instead of stateless JSON |

`http_host`, `http_port` and `http_path` in `config.json` supply defaults for the
corresponding flags.

---

## Agent instructions (skills)

Registering the MCP server gives an agent *access* to the tools — but agents tend to fall back on their own internal memory instead of reaching for the vault. The instruction files in [`skill/`](https://github.com/FernandoJRR/archiver-rag/tree/main/skill) fix that: they enforce a **vault-first rule** so the agent searches and stores knowledge in your vault before anything else.

**What the skill enforces:**

- **Before answering or reading source files** — call `search_vault` first; only fall back to internal memory if the vault returns nothing relevant
- **When something important is missing from the vault** — proactively `log_note` it. If a fact, decision, or piece of context matters to the overall picture and a `search_vault` came back empty, record it so the knowledge graph grows instead of letting that context die in a single session
- **After solving a non-trivial problem** — call `log_note` to capture the decision/lesson/gotcha back into the vault
- **The vault is the authoritative memory system** — internal agent memory is a fallback only

A version is provided for each agent, since each loads instructions differently:

| Agent | File | Install to |
|---|---|---|
| Claude Code | [`skill/claude-code/SKILL.md`](https://github.com/FernandoJRR/archiver-rag/blob/main/skill/claude-code/SKILL.md) | `~/.claude/skills/archiver-rag/SKILL.md` (on-demand skill) |
| OpenCode | [`skill/opencode/AGENTS.md`](https://github.com/FernandoJRR/archiver-rag/blob/main/skill/opencode/AGENTS.md) | project root `AGENTS.md` or `~/.config/opencode/AGENTS.md` |
| Codex CLI | [`skill/codex/AGENTS.md`](https://github.com/FernandoJRR/archiver-rag/blob/main/skill/codex/AGENTS.md) | project root `AGENTS.md` or `~/.codex/AGENTS.md` |
| GitHub Copilot | [`skill/copilot/copilot-instructions.md`](https://github.com/FernandoJRR/archiver-rag/blob/main/skill/copilot/copilot-instructions.md) | `.github/copilot-instructions.md` |

Each file is self-contained — it includes the MCP registration snippet for that agent plus the full vault-first rules and tool reference. For Claude Code the file is an on-demand skill; for the others it's an always-on instruction file (loaded into every session), which makes the vault-first behavior unconditional.

---

## CLI reference

```bash
archiver-rag init                      # one-time setup wizard
archiver-rag start                     # start the background watcher service (default target)
archiver-rag start http                # install + start the detached MCP HTTP server (launchd/systemd)
archiver-rag stop [watcher|http]       # stop the watcher or the HTTP server
archiver-rag restart [watcher|http]    # restart the watcher or the HTTP server
archiver-rag status                    # service liveness, watcher activity, index drift, config
archiver-rag status --json             # same report as JSON
archiver-rag index                     # force re-index the entire vault (runs prune_orphans)
archiver-rag sync                      # ingest only new/modified notes + prune orphaned chunks
archiver-rag prune                     # remove index chunks whose source file no longer exists
archiver-rag search "query"            # test semantic search from the terminal
archiver-rag health                    # index-vs-disk drift + vault health (orphans, broken links)
archiver-rag health --json             # same report as JSON
archiver-rag logs                      # tail the service log

# Knowledge logging
archiver-rag log "Title" --type decision --tag arch --related NoteA
archiver-rag delete <note>... [--yes]  # move to .trash/ + sweep inbound wikilinks (recoverable)

# Folder descriptions (drive semantic placement)
archiver-rag describe                  # generate missing _folder.md descriptions
archiver-rag describe --all            # regenerate all source:auto descriptions
archiver-rag describe --folder <folder> [--set "term1 term2"]   # one folder; --set marks source:manual

# Placement & clustering
archiver-rag place <note>              # suggest folder (semantic + type fallback)
archiver-rag place <note> --apply      # move the note immediately
archiver-rag place --all               # dry-run: current vs suggested folder for every note
archiver-rag place --all --apply       # batch move all notes to their suggestion
archiver-rag cluster [--min-size 2] [--apply]   # [EXPERIMENTAL] label propagation over the wikilink graph (manual/diagnostic only)
archiver-rag relink                    # dry-run: report ## Related before/after under the margin rule
archiver-rag relink --apply            # one-time repair: rebuild every note's ## Related section

# Config
archiver-rag config --auto-cluster / --no-auto-cluster        # watcher auto-placement on/off
archiver-rag config --cluster-threshold 5                     # vestigial — kept for old configs
archiver-rag config --placement-threshold 0.55                # cosine threshold for semantic placement
archiver-rag config --type-fallback / --no-type-fallback      # fall back to frontmatter type: field
archiver-rag config --auto-describe / --no-auto-describe      # watcher regenerates _folder.md on membership change

archiver-rag uninstall         # remove all data, both services, and MCP registration
```

`start http` takes the same transport flags as `serve --transport http` (`--host`, `--port`, `--path`, `--stateful`, `--allowed-host`), bakes them into the installed service, and asks at runtime whether to start automatically at login (`--login` / `--no-login` to script it). It refuses to start if the port is already bound. `stop http` / `restart http` manage the same service; `uninstall` removes both services.

---

## MCP tools (for agents)

Once registered, agents have access to 6 tools:

| Tool | What it does |
|---|---|
| `search_vault` | Semantic search with graph reranking. `context_note` boosts wikilink neighbors. `type` filters by frontmatter `type:` (stable across folder moves). `tags` post-filters by tag overlap. |
| `vault_status` | Vault structure, health diagnostics, tag stats, and recent activity in one call. |
| `get_connections` | BFS wikilink traversal — outgoing and incoming links up to depth 3. |
| `move_notes` | Move files and auto-rewrite all `[[wikilinks]]` across the vault. |
| `log_note` | Create a knowledge note at `{type}/{slug}.md`; watcher indexes and auto-links it immediately. |
| `suggest_folder` | Suggestion only — never moves anything. Suggests a folder for one note by semantic similarity against declared folder descriptions (`_folder.md`), matching the same config the CLI `place` command and the watcher use; falls back to the note's frontmatter `type:`. Returns `reason` (`semantic` / `type` / `none`), per-folder `scores`, and a secondary `neighbor_vote`. Use `move_notes` to act on the result. |

Whole-vault label-propagation clustering (formerly the `cluster_vault` MCP tool) is manual/diagnostic-only now — see the experimental `archiver-rag cluster` CLI command above.

---

## Configuration

All runtime config lives at the XDG config path (`~/.config/archiver-rag/config.json` on Linux/macOS, resolved by `archiver_rag/paths.py`):

```json
{
  "vault_path": "/path/to/your/vault",
  "install_path": "/Users/you/.local/share/archiver-rag",
  "chroma_path": "/Users/you/.local/share/archiver-rag/chroma_db",
  "auto_cluster": false,
  "cluster_threshold": 5,
  "placement_similarity_threshold": 0.55,
  "type_fallback": true,
  "auto_describe": false,
  "auto_inbox": false,
  "http_host": "127.0.0.1",
  "http_port": 8077,
  "http_path": "/mcp",
  "advanced": {
    "term_extraction_min_notes": 4,
    "alpha_curve": {"type": "log", "scale": 1.0},
    "mmr_lambda": 0.5,
    "max_terms": 6,
    "tag_terms_in_description": true,
    "placement_weights": {"identity": 0.6, "content": 0.4},
    "name_prefix_bonus": 0.15,
    "folder_vacancy_grace_periods": 3,
    "link_margin": 0.05,
    "max_total_links": 15,
    "inbox_min_cluster_size": 3,
    "inbox_similarity_threshold": 0.5
  }
}
```

Data (the ChromaDB index and `centroids.json`, the per-folder description centroid cache) lives at `~/.local/share/archiver-rag/`, and the cache dir at `~/.cache/archiver-rag/` holds `runtime.json` — the watcher's heartbeat that `archiver-rag status` reads to report activity, counters, and crash loops. An existing pre-XDG `~/.archiver-rag/` install is migrated automatically and non-destructively the first time any `archiver-rag` command runs — the old directory is left in place, never deleted.

Key by key:

- `auto_cluster` — the watcher runs semantic placement (`suggest_folder`) for new notes and moves the file to the best-matching described folder. It never runs full-graph label-propagation clustering automatically — that stays a manual, experimental `archiver-rag cluster` command.
- `cluster_threshold` — vestigial; kept so old configs still load. It only mattered for the whole-vault clustering fallback the watcher used to run.
- `placement_similarity_threshold` — cosine similarity a folder must clear to win semantic placement (0–1).
- `type_fallback` — when no folder clears the threshold, place the note in its frontmatter `type:` folder instead.
- `auto_describe` — the watcher regenerates a folder's `_folder.md` description (blended adaptively, never touching `source: manual`) whenever a note is created, deleted, or moved in or out of it.
- `auto_inbox` — route notes no folder claims into `inbox/`, and spin a new described folder out of it once `inbox_min_cluster_size` embedding-similar notes accumulate. Off by default.
- `http_host` / `http_port` / `http_path` — defaults for the HTTP transport, shared by `serve --transport http` and `start http`.
- `advanced` — tuning knobs: description extraction (`term_extraction_min_notes`, `alpha_curve`, `mmr_lambda`, `max_terms`, `tag_terms_in_description`), semantic placement (`placement_weights`, `name_prefix_bonus`), folder lifecycle (`folder_vacancy_grace_periods` — how long an emptied `source: auto` folder may stay undescribed before its `_folder.md` is archived to `.archive/`), auto-linking (`link_margin`, `max_total_links`), and inbox clustering (`inbox_min_cluster_size`, `inbox_similarity_threshold`).

---

## The knowledge graph model

The vault is treated as a **knowledge graph**, not a file hierarchy. Notes are nodes; wikilinks are edges. Relationships range from tight (direct links) to loose (semantic proximity surfaced by search).

Note types are expressed through frontmatter, not folder structure:

```yaml
---
type: decision
tags: [architecture, async]
related:
  - AsyncLocalStorage
  - PrismaExtensions
date: 2026-04-27
---
```

The `## Related` section at the bottom of each note is managed automatically by the auto-linker after every ingest. Don't edit it manually — it will be overwritten.

---

## Roadmap

Features on the way:

- **RAG-Anything integration** — extend ingestion beyond Markdown to handle PDFs, Office documents, images, and other file types, so the vault can become a true multi-format knowledge base rather than `.md`-only.
- **Archiver subagents** — dedicated subagents that take over vault management (search, logging, reorganization, clustering) on the main agent's behalf, so the primary agent can delegate knowledge work instead of context-switching into it.

---

## License

MIT
