Metadata-Version: 2.4
Name: mediarag
Version: 0.1.20
Summary: Multimodal local search over documents and media — pdf, office files, notes, images, audio and subtitles — from any MCP-capable agent.
Author: D Yiapanis
License: MIT
Project-URL: Homepage, https://github.com/dyiapanis/mediarag
Project-URL: Issues, https://github.com/dyiapanis/mediarag/issues
Keywords: rag,mcp,search,embeddings,onnx,multimodal
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: onnxruntime==1.29.0
Requires-Dist: numpy<3,>=2.0
Requires-Dist: pillow<14,>=11.0
Requires-Dist: tokenizers<1,>=0.20
Requires-Dist: transformers==5.19.0
Requires-Dist: torch<3,>=2.2
Requires-Dist: torchvision<1,>=0.17
Requires-Dist: huggingface_hub<3,>=0.26
Dynamic: license-file

# MediaRAG

![MediaRAG — one semantic index over documents, images and audio](assets/banner.jpg)

Multimodal local semantic search over documents and media, packaged as an
agent-plugin per the [agent-plugins.org](https://agent-plugins.org/) 1.0
standard.

It indexes documents (pdf, office files, notes, html) and media (images, audio,
subtitles) into one local SQLite corpus, and you search both by meaning from any
MCP-capable agent. Video is not an ingest format: pull the soundtrack out first
(`ffmpeg -i clip.mp4 -vn -ac 1 -ar 16000 soundtrack.wav`) and index that — the
audio tower then makes the clip findable by what was said in it. "Which file mentions the
landing" and "where is the clip where I talked about it" are the same kind of
question.

How it runs:

- In-gateway: five native tools (`mediarag_search`, `mediarag_add_source`,
  `mediarag_ingest_dir`, `mediarag_add_note`, `mediarag_status`) register once,
  so one gateway serves all agents and the embedding model loads once.
- Cross-client: `mcp.json` declares a stdio MCP server, so Claude Desktop,
  Cursor, and other Agent Plugins compliant clients load the same package.
  Per the spec the entry uses a plugin-relative `command`
  (`./scripts/mcp_launch.py`), never a bare `python3`: portable plugins must not
  depend on `PATH` or ambient variables for resolution. That launcher is
  stdlib-only and finds an interpreter that can import the engine
  (numpy/onnxruntime/transformers/torch, ~500 MB), which Agent Plugins 1.0 has
  no field to declare — so a client without a provisioned interpreter needs one
  command, once:

  ```bash
  python3 -m venv .venv && .venv/bin/python -m pip install -r requirements.txt
  ```

  Order of preference: `$MEDIARAG_PYTHON`, the package's own `.venv/`, the
  client's `$PLUGIN_DATA/venv`, then the interpreter that started it. With none
  present it exits 3 naming what's missing instead of vanishing mid-handshake.

  If `uv` is installed, it can own the environment instead — same
  `requirements.txt`, resolved and cached by the installer:

  ```bash
  uv run --with-requirements requirements.txt -- python -c pass   # once
  ```

  Set `MEDIARAG_AUTO_BOOTSTRAP=1` to have the launcher run that on start. It is
  off by default: a first run pulls ~1.7 GB, longer than most clients wait.
- Local-only: no cloud services. The corpus lives at `<root>/.corpus/store.db`.

Documents and media are embedded by the same model into the same vector space
(see [Embedding model](#embedding-model)), which is why one index answers both
kinds of question.

## System Requirements

| | |
|---|---|
| **Python** | 3.11 or newer (`requires-python >=3.11`) |
| **OS** | Linux x86_64 · Windows 10+ x86_64 · macOS 14+ on Apple Silicon |
| **Memory** | ~1.5 GB resident once everything is loaded — measured 834 MB after the text trunk, 1.06 GB with vision and audio added, 1.5 GB with the processor |
| **Disk** | ~480 MB of q4 towers on first run (measured 505 MB: 174 MB text + 109 MB vision + 189 MB audio + a 32 MB tokenizer), and up to ~450 MB more if the cache stores them as symlinks and they are materialised into `~/.cache/mediarag`; the corpus then grows with whatever you index |

**macOS is Apple Silicon only.** `onnxruntime` and `torch` publish no x86_64
macOS wheels, and their arm64 wheels require macOS 14.0 — an Intel Mac or a
macOS 13 machine cannot install this engine. CI runs the real engine on macOS 14
(arm64), Windows and Linux, ingest to search, on every commit.

System tools are needed only by the lanes that use them:

| Tool | Used for | Install |
|---|---|---|
| `ffmpeg` | audio | `apt install ffmpeg` · `winget install Gyan.FFmpeg` · `brew install ffmpeg` |
| `pandoc` | docx, odt, epub, rst, html, csv, rtf | `apt install pandoc` · `winget install JohnMacFarlane.Pandoc` · `brew install pandoc` |
| `pdftotext` | pdf | `apt install poppler-utils` · `winget install oschwartz10612.Poppler` · `brew install poppler` |
| `curl` | `add_source` on an http(s) URL | preinstalled on all three |

`curl` ships with Linux and macOS and is present on Windows 10+ (a built-in
`curl.exe`); `ffmpeg`, `pandoc` and `pdftotext` are not installed by default
anywhere. Everything else — numpy, onnxruntime, transformers, torch — is
pip-installed from the declared dependencies.

## Install

Two ways in; both end at the same `mediarag-mcp` stdio server.

**As a Python package** — any MCP client, no clone:

```bash
uvx --from git+https://github.com/dyiapanis/mediarag mediarag-mcp   # runs it
pipx install git+https://github.com/dyiapanis/mediarag              # or: mediarag-mcp on PATH
```

**As a plugin** — for hosts that provision dependencies from the package manifest:

```bash
hermes plugins install dyiapanis/mediarag    # Hermes reads python_dependencies
```

The engine needs numpy, onnxruntime, transformers and torch (~1.7 GB with the
model weights); `pipx`/`uvx`/`hermes` install that for you. Running from a bare
checkout instead, see the MCP section below.

Python dependencies: `pip install -r requirements.txt`.

Runs on Linux, Windows and macOS 14+ on Apple Silicon — see [System
Requirements](#system-requirements) for memory, disk and the system-tool table.
On Windows and macOS those tools are not preinstalled; the table has the
`winget`/`brew` commands. The corpus cache lives at `~/.cache/mediarag`
(`%USERPROFILE%\.cache\mediarag` on Windows); adapters are python scripts run
through the current interpreter, so they need neither the executable bit nor a
shebang.

The embedding weights are not downloaded on startup. Once, before first search:

```bash
python scripts/fetch_model.py
```

That is about 480MB into the Hugging Face cache; what is in it, and why, is
covered under [Embedding model](#embedding-model). After the download everything
runs offline (`HF_HUB_OFFLINE=1`).

## Features

- Hybrid retrieval: semantic vectors plus an FTS5/BM25 keyword lane, fused with
  reciprocal rank fusion. Exact-string lookups and meaning searches both work.
- Documents and media in one index: pdf via `pdftotext`, docx/odt/epub/rst/html/csv/rtf
  via `pandoc`, plus images, audio, and subtitles.
- Incremental re-indexing: unchanged files are skipped on an mtime+size
  fingerprint.
- Source adapters: external collections through `sources.d` executables (see
  below).
## Embedding model

One model embeds everything: [EmbeddingGemma
2](https://huggingface.co/onnx-community/embeddinggemma-2-ONNX) in its ONNX q4
build, on onnxruntime, on the CPU. It ships three towers — the text trunk, a
vision encoder and an audio encoder — and all three write into the same
768-dimensional space, so a pdf, a photograph and a clip of you talking about
both are directly comparable.

- **Snapshot** `daa72c51243991dfcaf9f9137d2c573d8f7790c0` of
  `onnx-community/embeddinggemma-2-ONNX`. `scripts/fetch_model.py` pulls only the
  q4 towers plus the tokenizer and processor configs — 174 MB text, 109 MB
  vision, 189 MB audio, 32 MB tokenizer, about 505 MB — and never at startup. A
  cache that has also seen other quantisations of this model will be larger
  still, because those blobs are not evicted.
- **768 dimensions**, L2-normalised, stored as f32 blobs (3,072 bytes per row).
  Retrieval is exact cosine over the whole corpus; there is no approximate index.
- **1024-token cap** on the text trunk. Attention is quadratic in sequence
  length, so tool input is sliced at that boundary rather than trusted.
- **Chunking**: documents and notes at 1,500 characters, subtitles at 600,
  audio in 11.2-second windows.
- If the Hugging Face cache stores the ONNX shards as symlinks, they are
  materialised once into `~/.cache/mediarag/onnx-real`: onnxruntime will not
  load external-data blobs out of a sharded cache.

The three embedders are parity-locked against the reference model — 12/12 golden
vectors at cosine 1.000000 (`tests/fixtures/golden.json`, enforced by
`tests/test_parity_slices.py`).

## What it touches

No telemetry. It shells out to `curl` (http sources), `ffmpeg`, `pandoc`,
`pdftotext`, and `rclone` when those lanes are used, and to whatever adapter
you drop in `sources.d/`. It reads the paths you pass plus the corpus at `root`
(`mediarag-config.yaml` `root:`, else `MEDIARAG_ROOT`, else `~/.mediarag`). The Raindrop adapter calls
`api.raindrop.io` with `RAINDROP_API_TOKEN` — optional; inside Hermes the token
is resolved through the profile secret scope, and a missing scope does not
inherit another profile's value.

To run it as a Hermes plugin, point the plugin loader at this directory (or at
the agent plugin catalog entry). The manifests (`mcp.json`, `plugin.yaml`,
`plugin.json`) describe both the stdio MCP server (`scripts/mcp_server.py`,
spawned by the client) and the in-gateway native tools. Neither mode opens a
network listener.

## Configuration

Everything is optional and set through the environment:

- `mediarag-config.yaml` — named corpus config in the plugin dir (`root:` key); wins over env and the default (see `mediarag-config.example.yaml`)
- `MEDIARAG_ROOT` — corpus root (default: `~/.mediarag`)
- `MEDIARAG_ALLOW_PRIVATE_URLS=1` — let `add_source` fetch loopback/private-network URLs (standalone; inside Hermes the host's `security.allow_private_urls` applies). Link-local and cloud-metadata addresses are always refused.
- `RAINDROP_API_TOKEN`, `PCLOUD_MOUNT`, `OBSIDIAN_VAULT` — adapter inputs

## Adapters

`add_source` accepts public `http(s)://` URLs (fetched with curl, capped at 256MB;
every redirect hop is checked and loopback/private/link-local/metadata hosts are refused)
and source-scheme URIs (`<scheme>://...`) handled by adapters. Five ship in the
box:

- `dropbox://` — Dropbox through the desktop client's local folder, or any mirror
  (`DROPBOX_DIR` env). Skips Dropbox's own `.dropbox/` bookkeeping.
- `rclone://remote:path` — any [rclone](https://rclone.org/) remote. Files copy
  into a persistent cache at `~/.cache/mediarag/rclone` and index from there.
- `pcloud://` — walks a locally mounted pCloud drive (`PCLOUD_MOUNT` env).
- `obsidian://` — an Obsidian vault; skips `.obsidian/`, `.trash/`, `.git/`.
- `raindrop://` — Raindrop.io bookmarks through the REST API
  (`RAINDROP_API_TOKEN` env).

To write your own adapter, drop an executable at `sources.d/<scheme>` under the
configured corpus root (`mediarag-config.yaml` / `MEDIARAG_ROOT` / `~/.mediarag`; a
per-call `root` argument never adds adapters) that
accepts `fetch <rest>` and prints one JSON object per line, either
`{"uri","title","text"}` or `{"uri","path"}`. `status` reports the adapters it
found; an unknown scheme fails with the list of what is available.

## Development

Tests are direct-run scripts, not pytest-collectable:

- `python tests/test_contract.py` — tool/store/adapter contracts (no model)
- `python tests/test_ingest_units.py` — chunking, subtitle timings, store integrity
- `python tests/test_engine_deref.py` — sharded-cache materialisation (no model)
- `python tests/test_config_root.py` — corpus-root precedence (no model)
- `python tests/test_parity_slices.py` — FTS lane, RRF fusion, doc bridge (needs the model)

The fixtures under `tests/fixtures/` are synthetic and deterministic, and nothing
contacts the network. `tests/bench_engine.py` and `scripts/bench.sh` are the
local benchmark harness. CI runs the model-free suites on Linux and Windows,
plus the full parity lane on Linux.

## License

MIT. For security reports, see `SECURITY.md`.

## Acknowledgements

This project was developed with AI assistance using [Hermes Agent](https://github.com/NousResearch/hermes-agent).
