Metadata-Version: 2.4
Name: retrieval-mcp
Version: 0.4.10
Summary: Installable MCP server for the Retrieval API: academic-paper search + journal memory + ACE playbook + semantic code search
Project-URL: Homepage, https://retrieval.rnarket.com
License: MIT
Keywords: ace,code-search,llm,mcp,model-context-protocol,papers,retrieval,search
Requires-Python: >=3.10
Requires-Dist: chonkie[code]<1.7,>=1.6
Requires-Dist: httpx[socks]>=0.27
Requires-Dist: mcp<1.30,>=1.29
Requires-Dist: tomli>=2.0; python_version < '3.11'
Requires-Dist: tree-sitter-language-pack<1.13,>=1.8
Requires-Dist: tree-sitter<0.26,>=0.23
Requires-Dist: watchfiles>=1.1
Requires-Dist: zstandard>=0.22
Description-Content-Type: text/markdown

# retrieval-mcp

Current PyPI release: **0.4.9** (2026-09-04), 33 tools. Platform architecture:
[../overview.md](../overview.md).

**0.4.10 is a candidate in this repo — NOT published.** It fixes a cross-project
scope bug: the singleton daemon resolved a relative scope (`scope_dir="."`,
`repo_root="."`) against its OWN working directory — whichever project's shim
started it — so journal reads returned a foreign repo and `journal_record` wrote
into that repo's namespace. The shim now sends its directory in the raw handshake
and the daemon binds it per session. `DAEMON_PROTOCOL_SCHEMA_VERSION` goes 1 → 2 so
a fixed shim cannot reuse an unfixed daemon. Tool surface is unchanged (33 tools).

Two changes over 0.4.8:

- `index_code` and `journal_index_dir` document the nested-scope guard's
  gitignore exception — a child scope indexing with `respect_gitignore=False`
  under a filtered parent now warns instead of being refused.
- Request bodies over 64 KiB are compressed with zstd level 3, which is what
  `index_code`/`index_dir` uploads are (a real 19.3 MiB body becomes 2.2 MiB in
  54 ms). This adds a `zstandard` dependency; it also makes httpx advertise zstd,
  so replies come back compressed with no further work.

  **It does not require a backend upgrade.** Compression is negotiated per
  request: the client learns a backend accepts compressed bodies from the
  `Accept-Encoding` *response* header (RFC 7694), and a backend that does not
  rejects the body with 400/415/422 — all decided before the route function runs,
  so the automatic uncompressed resend cannot duplicate an index submission.
  After one such rejection the client stops compressing for the session.

`find_duplicate_code` was added in 0.4.8 and its backend endpoint is live on `.88`.

The package declares `mcp>=1.29,<1.30`. In addition to the public SDK v1
`FastMCP` API, the raw JSONL daemon bridge uses the verified 1.29 low-level
`SessionMessage` and server-stream APIs. The narrow upper bound is therefore a
private-API compatibility contract, not merely a v2 exclusion; raising it needs
the daemon concurrency and clean-wheel MCP regressions first.

The daemon's 33-tool compatibility digest covers each tool name, full
`inputSchema`, and semantic description text. Description canonicalization uses
`inspect.cleandoc` only to remove the shared docstring margin emitted differently
by the supported Python 3.10/Pydantic and Windows renderers; it does not reduce
the fence to tool names or discard meaningful relative indentation.

Code indexing deliberately declares `chonkie[code]>=1.6,<1.7`,
`tree-sitter-language-pack>=1.8,<1.13`, and `tree-sitter>=0.23,<0.26`. The local
`CodeChunker` path is verified on Chonkie 1.6.x, the 1.12.x language-pack line,
and the native tree-sitter 0.25.x ABI. Chonkie's `code` extra only gives its
parser dependencies lower bounds; the explicit companion constraints prevent a
fresh install from silently crossing into a newer parser-pack or native ABI line.
The native constraint is intentional because local indexing executes native
parser code and its crash boundary must not drift. Raising any upper bound
requires a CodeChunker regression and a clean wheel-install MCP smoke; this is a
compatibility contract, not a package-cache workaround.

An **MCP server** for the Retrieval academic-paper API - semantic paper search,
document matching, ACE journal memory, and index inventory. Self-contained: it talks
to the backend over HTTP only (just `mcp` + `httpx`), so it installs anywhere with
`uvx` / `pip` - no repo checkout, no GPU, no models.

By default it targets the compute box on the lab LAN (`http://10.100.100.111:8000`),
which trusts LAN callers so **no key is needed**. Off-LAN, point `RETRIEVAL_API_URL` at
the public gateway (`https://retrieval.rnarket.com`) and set `RETRIEVAL_API_KEY` (`sk-...`).

## Tools

Every tool's full docstring (purpose + each argument with its default + an example)
is what your LLM sees - call them by name. Summary:

**Paper retrieval**

| Tool | What it does |
|------|--------------|
| `search_papers` | Semantic hybrid search over the live top-venue corpus (filters: `venue`, `year`, `title_only`) |
| `search_within_paper` | Every matching passage inside one paper |
| `get_paper_markdown` | Return one paper's complete Markdown, or Range-stream it to a caller-local file/directory |
| `download_paper_pdf` | Range-stream the original PDF to a caller-local path with validated resume and atomic publication |
| `download_papers` | Download `pdf`, `markdown`, or `both` for 1-10 paper IDs with two bounded concurrent transfers |
| `match_document` / `match_paper` | Content-nearest papers to a passage / to a paper |
| `list_conferences` / `corpus_stats` | Venue registry / corpus size |

**Journal work-memory** (scoped to the current project by default)

| Tool | What it does |
|------|--------------|
| `journal_record` | Record a work note (`memory`) or a file's current content (`doc`, latest-wins) |
| `journal_search` | Search memory - `keyword` (FTS5, no embedding) or `hybrid`/`dense`/`sparse` |
| `journal_recent` | List recent entries |
| `journal_index_dir` | Batch-index a local dir's files into the journal (latest-wins per file) |
| `journal_index_status` | Poll an asynchronous Journal directory-index job |
| `delete_journal_docs` | Explicitly remove batch-indexed Journal documents for one directory scope |

**Code KB** (source stays local - only chunks are uploaded)

| Tool | What it does |
|------|--------------|
| `index_code` | AST-chunk a repo locally (40+ languages) and index it, scoped to you |
| `index_code_status` | Poll local preparation plus backend Code KB embedding/upsert progress |
| `search_code` | Semantic code search with `path:line` citations |
| `find_duplicate_code` | Report file pairs in an indexed scope that duplicate each other |
| `delete_code` | Explicitly remove one Code KB scope from vectors and inventory |
| `index_inventory` | Your indexed-file tree: user -> host -> project -> dir -> file |

**Public Code** (admin-managed shared source corpus)

| Tool | What it does |
|------|--------------|
| `list_public_code_sources` | List configured sources, active revisions, coverage, profiles, and update state |
| `add_public_code_source` | Register an HTTPS Git upstream using server-side proxy/credential profile references |
| `check_public_code_updates` | Queue remote revision checks for selected sources or domains |
| `index_public_code_sources` | Queue bare mirror refresh, AST chunking, shared GPU embedding, and atomic promotion |
| `set_public_code_update_policy` | Change manual/scheduled update policy without exposing credentials |
| `public_code_job_status` | Poll durable source/file/chunk progress for checks and indexing |
| `search_public_code` | Flat domain or exact-repository hybrid/dense/keyword search with full code citations |

**ACE playbook** (accumulated, curated lessons per project)

| Tool | What it does |
|------|--------------|
| `ace_context_aware` / `ace_playbook` | Retrieve relevant / list all curated bullets |
| `ace_enhance_prompt` / `ace_smart_generate` | Attach playbook lessons to a prompt (no LLM call) |
| `ace_smart_reflect` | Curate a transferable lesson into the playbook (grow-and-refine dedup) |

### Code KB language coverage

`index_code` chunks 40+ languages structurally via tree-sitter (chonkie CodeChunker): Python,
TypeScript/TSX/JS/JSX (React), Java, Kotlin (incl. Jetpack Compose `.kt`/`.kts`),
Swift, Go, Rust, C/C++, C#, Ruby, PHP, Lua, Scala, Dart, R, Julia, Elixir, Erlang,
Haskell, OCaml, SQL, GraphQL, Protobuf, HTML, CSS/SCSS (Tailwind = CSS classes),
Vue, Svelte, shell, PowerShell, Dockerfile, Terraform/HCL, CMake, YAML/JSON/TOML/XML,
and more. Grammarless config/text files fall back to line-window chunks; docs (`.md`)
and binaries are skipped (docs belong in the journal via `journal_index_dir`).

## Singleton behavior and command-mode MCP configuration

Use the same command-style stdio entry for Claude, Codex, Grok, and other MCP
clients. The normal no-argument invocation automatically starts or reuses one
per-user daemon and transparently bridges just this client's JSONL session. No
client URL, transport option, or manual `start` step is required.

Install the published package once and use the same command-style configuration
for every client. Existing version-pinned `uvx` configurations should pin
`retrieval-mcp==0.4.8`.

```json
{
  "mcpServers": {
    "retrieval": {
      "command": "retrieval-mcp",
      "args": []
    }
  }
}
```

Existing `uvx` command configurations remain compatible. A once-installed
`retrieval-mcp` avoids a package-runner wrapper per client and is the recommended
singleton setup. `status` and `stop` are explicit local control commands;
`retrieval-mcp stdio-server` is only a direct debug/compatibility server and is
not the normal client configuration.

## Automatic code-index refresh

`index_code(...)` returns a local job ID before repository walking, hashing, AST chunking,
upload, GPU embedding, or Qdrant upsert completes. Poll that same ID with
`index_code_status()` through the `preparing`, backend queue, and terminal phases. A second
index request for the same scope reuses the active job instead of starting another scan.

With `auto_refresh=True`, the client then keeps one filesystem watcher for that absolute
repository path. Ordinary file events hash and chunk only the touched paths; ignore-rule
changes trigger a full reconcile. `search_code()` never waits for the watcher or indexing:
it returns the last completed snapshot and reports freshness separately.

Git repositories continue to honor Git's ignore rules by default. Any directory, including
non-Git projects, can add scope-relative patterns to `.retrievalignore`; callers can add
temporary patterns with `exclude_globs=["generated/**", "private.py"]`. Explicit
`delete_code()` cancels/fences stale refresh work and removes vectors, inventory, and the
saved refresh policy.

Windows drive and UNC scopes use `pathlib`'s Windows path semantics and one
case-insensitive canonical identity, while POSIX scope casing remains significant.
Historical mixed-case Windows scopes are included through bounded catalog aliases.
`path_glob` is always scope-relative; both slash styles are accepted, and `pathspec`
supplies basename, directory, `**`, and exact-file matching semantics.
Windows extended-length/device spellings (`\\?\`, `\\.\`) are rejected with guidance
to use the equivalent ordinary drive or UNC path, preventing duplicate scope identities.

## Config (env)

| Var | Default | Notes |
|-----|---------|-------|
| `RETRIEVAL_API_URL` | `http://10.100.100.111:8000` | LAN compute box (no key). Off-LAN, set to `https://retrieval.rnarket.com`. |
| `RETRIEVAL_API_KEY` | - | `sk-...` key for the gateway (create under `/auth/keys`). Required off-LAN. |
| Journal/code scope | current absolute directory path | The client derives scope from the directory you run or pass to each tool, so projects do not leak into each other. |

## Paper artifact transport

- `get_paper_markdown(..., save_path=None)` returns the complete JSON-backed Markdown
  to the MCP caller. With `save_path`, it instead uses the raw
  `GET /api/papers/{paper_id}/markdown` file route, so large Markdown is not buffered
  as JSON by the gateway or MCP process.
- Markdown files and PDFs use HTTP Range with validated `Content-Range`. If the
  connection closes after writing a valid prefix, the next request resumes from the
  exact local byte offset. Completed files are published atomically and are not
  overwritten unless `overwrite=True`.
- `download_papers` accepts at most 10 paper IDs, preflights all destination names,
  disambiguates sanitized filename collisions with a stable digest, and reuses the
  single-artifact Range paths with at most two concurrent transfers. Errors are
  reported per paper/artifact.
- The separate authenticated HTTP `POST /api/papers/download-batch` endpoint serves a
  one-shot ZIP for browser/API clients. It is capped at 10 papers and 1 GiB of source
  artifacts, allows at most two concurrent archive builds, and intentionally rejects
  Range because each generated archive is a new file.

## License

MIT
