Metadata-Version: 2.4
Name: punt-quarry
Version: 1.19.0
Summary: Extract searchable knowledge from any document. Expose it to LLMs via MCP.
Author: Punt Labs
License-Expression: MIT
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Indexing
Requires-Dist: pymupdf>=1.25.0
Requires-Dist: lancedb>=0.17.0
Requires-Dist: tokenizers>=0.20.0
Requires-Dist: huggingface-hub>=0.26.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pydantic-settings>=2.0.0
Requires-Dist: typer>=0.15.0
Requires-Dist: rich>=13.0.0
Requires-Dist: mcp>=1.0.0,<2.0.0
Requires-Dist: starlette>=0.40.0,<1.0.0
Requires-Dist: uvicorn>=0.32.0
Requires-Dist: websockets>=13.0
Requires-Dist: pyarrow>=18.0.0
Requires-Dist: numpy>=1.26.0
Requires-Dist: pillow>=10.0.0
Requires-Dist: python-docx>=1.1.0
Requires-Dist: tree-sitter>=0.23.0
Requires-Dist: tree-sitter-language-pack>=0.5.0,<1.0.0
Requires-Dist: rapidocr>=3.6.0
Requires-Dist: onnxruntime>=1.18.0
Requires-Dist: opencv-python-headless>=4.8.0
Requires-Dist: openpyxl>=3.1.0
Requires-Dist: python-pptx>=1.0.0
Requires-Dist: beautifulsoup4>=4.12.0
Requires-Dist: markdownify>=0.14.0
Requires-Dist: pathspec>=0.12.0
Requires-Dist: ultimate-sitemap-parser>=1.8.0
Requires-Dist: pyyaml>=6.0.0
Requires-Dist: cryptography>=46.0.0
Requires-Dist: httpx>=0.27.0 ; extra == 'dev'
Requires-Dist: ruff>=0.13.0 ; extra == 'dev'
Requires-Dist: mypy>=1.18.1 ; extra == 'dev'
Requires-Dist: pyright>=1.1.0 ; extra == 'dev'
Requires-Dist: pytest>=8.4.2 ; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.0 ; extra == 'dev'
Requires-Dist: pytest-cov>=6.0.0 ; extra == 'dev'
Requires-Dist: pytest-timeout>=2.3.0 ; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0.0 ; extra == 'dev'
Requires-Dist: ranx>=0.3.20 ; extra == 'eval'
Requires-Python: >=3.13
Project-URL: Homepage, https://github.com/punt-labs/quarry
Project-URL: Bug Tracker, https://github.com/punt-labs/quarry/issues
Provides-Extra: dev
Provides-Extra: eval
Description-Content-Type: text/markdown

# punt-quarry

> Local semantic search for AI agents and humans.

[![License](https://img.shields.io/github/license/punt-labs/quarry)](LICENSE)
[![CI](https://img.shields.io/github/actions/workflow/status/punt-labs/quarry/test.yml?label=CI)](https://github.com/punt-labs/quarry/actions/workflows/test.yml)
[![PyPI](https://img.shields.io/pypi/v/punt-quarry)](https://pypi.org/project/punt-quarry/)
[![Python](https://img.shields.io/pypi/pyversions/punt-quarry)](https://pypi.org/project/punt-quarry/)
[![Working Backwards](https://img.shields.io/badge/Working_Backwards-hypothesis-lightgrey)](./prfaq.pdf)

Quarry indexes documents in 20+ formats, embeds them with a local ONNX model (snowflake-arctic-embed-m-v1.5, 768-dim), stores vectors in LanceDB, and serves semantic search to Claude Code, Claude Desktop, and the CLI. Everything runs locally — no API keys, no cloud accounts. The embedding model (~120 MB int8) downloads once on first use. CUDA GPUs are auto-detected for faster inference.

**Platforms:** macOS, Linux

## Quick Start

```bash
curl -fsSL https://raw.githubusercontent.com/punt-labs/quarry/ec2fb7f/install.sh | sh
```

Restart Claude Code, then:

```text
> /ingest report.pdf                    # index a document (runs in background)
> /quarry status                        # after a moment, confirm it's there
> /find "what does the report say about margins"   # search by meaning
```

Once installed, a plugin hook auto-indexes your current project directory on every session start — you don't need to `/ingest` your codebase manually.

<details>
<summary>Manual install (if you already have uv)</summary>

```bash
uv tool install punt-quarry
quarry install
quarry doctor
```

</details>

<details>
<summary>Verify before running</summary>

```bash
curl -fsSL https://raw.githubusercontent.com/punt-labs/quarry/ec2fb7f/install.sh -o install.sh
shasum -a 256 install.sh
cat install.sh
sh install.sh
```

</details>

### Remote Server

Run quarry on a GPU server and connect from any Mac or Linux client over TLS.

**Server** (GPU host, serves remote clients):

```bash
export QUARRY_API_KEY=$(openssl rand -hex 32)
curl -fsSL https://raw.githubusercontent.com/punt-labs/quarry/ec2fb7f/install.sh | sh -s -- --network
```

Generates TLS certificates, binds daemon to 0.0.0.0, registers a systemd service, and prints a CA fingerprint. NVIDIA GPUs are auto-detected for CUDA inference.

**Client** (connects to remote server):

```bash
curl -fsSL https://raw.githubusercontent.com/punt-labs/quarry/ec2fb7f/install.sh | sh
quarry login <server-hostname> --api-key <token>
```

No special flag needed --- the default install runs a local daemon on localhost. `quarry login` redirects queries to the remote server over `wss://` with TOFU certificate pinning.

### Claude Desktop

[**Download punt-quarry.mcpb**](https://github.com/punt-labs/quarry/releases/latest/download/punt-quarry.mcpb) and double-click to install. Alternatively, `quarry install` configures Claude Desktop automatically.

**Note:** Uploaded files in Claude Desktop live in a sandbox that quarry cannot access. Use `remember` for uploaded content, or provide local file paths to `ingest`.

## Features

- **20+ formats** --- PDFs (with OCR for scanned pages), source code (AST-aware splitting), spreadsheets, presentations, HTML, Markdown, LaTeX, DOCX, images
- **Semantic search** --- retrieval is by meaning, not keyword. A query about "margins" finds passages about profitability even if they never use that word
- **Daemon architecture** --- one `quarry serve` process loads the embedding model once and serves all Claude Code sessions via [mcp-proxy](https://github.com/punt-labs/mcp-proxy) over WebSocket
- **Passive knowledge capture** --- `quarry enable` sets up three scoped collections per project: file sync, passive captures (web fetches + session transcripts), and per-agent memory. Captures are separated from the code index so research doesn't pollute code search
- **Named databases** --- isolated LanceDB directories with independent sync registries. Switch with `use` for work/personal separation
- **Research agent** --- `researcher` subagent combines quarry local search with web research, auto-ingests valuable findings

## What It Looks Like

### Ingest a document

```text
> /ingest report.pdf

▶ Ingesting report.pdf (background)
```

### Check what's indexed

```text
> /quarry

▶ Database: default
  Documents: 47
  Chunks: 1,203
  Size: 12.4 MB
  Model: snowflake-arctic-embed-m-v1.5 (768-dim)
```

### Search by meaning

```text
> /find "what were the Q3 revenue figures"

▶ [report.pdf p.12 | text/.pdf] (similarity: 0.4521)
  Third quarter revenue reached $142M, up 18% year-over-year,
  driven primarily by expansion in the enterprise segment.
  Gross margins improved to 71% from 68% in Q2.
```

## Commands

### Slash Commands (Claude Code)

| Command | What it does |
|---------|-------------|
| `/ingest <source>` | Ingest a URL, directory, or file |
| `/remember <name>` | Ingest inline text under a document name |
| `/find <query>` | Semantic search. Questions get synthesized answers; keywords get raw results |
| `/explain <topic>` | Search and synthesize an explanation |
| `/source <claim>` | Find which document a claim comes from |
| `/quarry [sub]` | Manage: `status`, `sync`, `collections`, `databases`, `registrations` |

### MCP Tools

| Tool | Purpose | Execution |
|------|---------|-----------|
| `ingest` | Index a file or URL | Background |
| `remember` | Index inline text | Background |
| `register_directory` | Register directory for sync | Background |
| `sync_all_registrations` | Re-index all registered directories | Background |
| `find` | Semantic search with filters | Sync |
| `show` | Document metadata or page text | Sync |
| `list` | Documents, collections, databases, registrations | Sync |
| `status` | Database statistics | Sync |
| `delete` | Remove document or collection | Background |
| `deregister_directory` | Remove registration (validates; errors if unknown) | Sync |
| `use` | Switch active database | Sync |

### CLI

```bash
quarry ingest report.pdf                       # index a file
quarry ingest https://example.com              # index a webpage
echo "notes" | quarry remember --name notes.md # index inline text
quarry find "revenue trends"                   # hybrid search (vector + FTS)
quarry list documents                          # list indexed documents
quarry register ~/Documents/notes              # watch a directory
quarry sync                                    # re-index registered dirs
quarry use work                                # switch database
quarry enable                                  # set up project collections + captures
quarry disable                                 # remove project registration + data
quarry captures init                           # bootstrap the private capture shadow repo
quarry captures push                           # re-scrub + push captures to the shadow
quarry status                                  # database dashboard
quarry doctor                                  # health check
quarry serve                                   # start daemon on :8420
quarry install                                 # set up daemon, TLS certs, mcp-proxy

# Remote connections
quarry login okinos.local --api-key <token>    # TOFU login to remote server
quarry logout                                  # disconnect, revert to local daemon
quarry remote list --ping                      # show remote config and health

# Agent memory tagging
quarry ingest notes.md --agent-handle claude --memory-type fact
quarry find "deployment steps" --agent-handle claude
echo "key insight" | quarry remember --name insight.md --agent-handle claude \
  --memory-type observation --summary "Key insight from review"
```

## Setup

Quarry works with zero configuration. These environment variables are available for customization:

| Variable | Default | Description |
|----------|---------|-------------|
| `QUARRY_PROVIDER` | *(auto)* | ONNX execution provider: `cpu`, `cuda`, or unset (auto-detect) |
| `QUARRY_API_KEY` | *(none)* | Bearer token for `quarry serve` |
| `QUARRY_ROOT` | `~/.punt-labs/quarry/data` | Base directory for all databases |
| `CHUNK_MAX_CHARS` | `1800` | Max characters per chunk (~450 tokens) |
| `CHUNK_OVERLAP_CHARS` | `200` | Overlap between consecutive chunks |

For the full configuration reference, see [Architecture](docs/architecture.tex) section 7.

## Passive Knowledge Capture

Beyond explicit `/ingest` and `/find` commands, quarry runs as a Claude Code plugin with hooks that capture knowledge automatically during your sessions:

| Hook | When it fires | What it does |
|------|--------------|-------------|
| **Session start** | On every session start | Auto-registers your project directory and syncs it in the background. Your codebase is searchable without manual ingestion. |
| **Web fetch** | After any `WebFetch` tool call | URLs Claude fetches during research are auto-ingested into the project's `<name>-captures` collection (or global `web-captures` if no project is enabled). |
| **Pre-compact** | Before context compaction | Captures the conversation transcript into the project's `<name>-captures` collection (or global `session-notes` if no project is enabled). |

All hooks are fail-open — failures are ignored and never block Claude Code. Each hook is individually toggleable via `.punt-labs/quarry/config.md` YAML frontmatter. See [AGENTS.md](AGENTS.md) for the full integration model.

### Privacy: captures are scrubbed before write

Captures (session transcripts and auto-ingested web fetches) are redacted **at write time**, before anything touches disk or the `<name>-captures`/`web-captures` collection. A single write choke point scrubs, in order, secrets → filesystem paths (`/Users`/`/home/<user>/` → `~/`) → emails (→ `[REDACTED:email]`) → the local hostname (→ `[REDACTED:hostname]`) → profanity. Web-fetch captures also redact the source URL's userinfo, query, and fragment so a URL like `…/reset?email=…&token=…` cannot leak. Redaction is idempotent and fail-closed (on any scrub error, no capture is written). Deliberately-ingested content (`quarry ingest`, `remember`) is **not** scrubbed — only the automatic capture paths are, since those feed the shared/pushable collections. See [DES-036](DESIGN.md).

### Private capture shadow repo (`<repo>` → `<repo>-quarry`)

Redacted capture `.md` files live in the gitignored `.punt-labs/quarry/captures/` dir with no durable home. Enable the optional **shadow sync** to push them to a per-project **private** repo `<repo>-quarry` instead. Opt in by uncommenting the `shadow:` block in `.punt-labs/quarry/config.md`:

```yaml
shadow:
  enabled: true                  # off by default — a network + security action
  remote: ""                     # empty → derive <origin>-quarry from the public remote
  acknowledge_unverified: false  # required to push when gh cannot confirm the remote is private
```

**Pre-create the private repo first.** quarry does not create it by default (a wrong owner or an accidental public repo would leak). Create `<repo>-quarry` as **private** and let the remote derive, or run `quarry captures init --create` to create it via `gh` (which verifies visibility is private before any push).

Once enabled, the captures dir becomes a standalone nested git repo. `quarry captures push` — and the tail of every `quarry sync` — re-scrubs the staged captures with the same DES-036 scrubber, aborts the commit if any file is not clean, then commits and pushes. Auth reuses your existing git credentials; quarry stores no new secret. The push is fail-open: a network or auth failure never blocks a session.

**Visibility is load-bearing.** The re-scrub cannot catch every residual (IDN emails, non-`/Users`/`/home` paths, a hostname from another machine), so those are backstopped **only** by the remote being private. quarry therefore **refuses to push to a verifiably public remote**, and requires `acknowledge_unverified: true` in `.punt-labs/quarry/config.md` when `gh` is absent and it cannot confirm the remote is private. `quarry doctor` reports the shadow state, and flags a **required** failure if the public repo already tracks capture files (which `git rm --cached` stops going forward, but an already-pushed capture needs a history purge — `git filter-repo`/BFG + force-push — coordinated with the repo owner). See [DES-039](DESIGN.md).

## How It Works

Quarry runs as a daemon — one `quarry serve` process per machine holds the engine (embedding model, LanceDB, pipeline, sync registry). Claude Code sessions reach it through mcp-proxy over WebSocket:

```text
                    stdio                       wss:// (TLS)
Claude Code <-----------------> mcp-proxy <---------------------> quarry serve
             MCP JSON-RPC       (~5 MB Go)      pinned CA cert    (one engine)
```

One resident engine means the model loads once and MCP state is shared across sessions — no per-session ~200 MB reload. The mcp-proxy and remote connections use TLS with a self-signed, pinned CA — even on localhost.

The **target** is that *every* interface — CLI, library, and MCP — is a thin client of that one daemon over a versioned REST/WebSocket contract; this daemon-first model is specified in [DES-031 v2](DESIGN.md) (ACCEPTED, in implementation). Today the CLI and library still load the engine in-process on local calls, and MCP can fall back to an in-process `quarry mcp` when mcp-proxy is absent; those local-engine paths are being removed.

`quarry install` downloads mcp-proxy (SHA256-verified, correct platform) and configures MCP clients.

## Documentation

[Architecture](docs/architecture.tex) |
[Z Specification](docs/claude-code-quarry.tex) |
[Design](DESIGN.md) |
[Agents](AGENTS.md) |
[Changelog](CHANGELOG.md)

## Development

```bash
uv sync                        # install dependencies
make check                     # run all quality gates (lint, type, test)
make test                      # test suite only
make format                    # auto-format code
make docs                      # build LaTeX documents
make eval                      # retrieval-quality eval harness (MRR/success@k, dev/CI only)
```

## Roadmap

Direction tracks the [PR/FAQ](prfaq.tex). Current focus:

- **Retrieval quality** — a `make eval` harness (per-bucket MRR/success@k + a metadata-pollution diagnostic) now baselines search on quarry's own data; embedding levers (contextual embeddings, late chunking) are measured against it before adoption. See [docs/eval-harness-design.md](docs/eval-harness-design.md).
- **Private capture sync** — captures redact PII at write time, and the opt-in per-project private shadow repo (`<repo>` → `<repo>-quarry`, above) now pushes the redacted captures off the public repo entirely.

## License

MIT
