Metadata-Version: 2.4
Name: refindery
Version: 0.1.0
Summary: A local, single-machine retrieval engine over the web pages you read.
Keywords: retrieval,search,rag,embeddings,mcp,browser-history
Author: Harold Martin
Author-email: Harold Martin <harold.martin@gmail.com>
License-Expression: MIT
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3.13
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Developers
Classifier: Topic :: Text Processing :: Indexing
Requires-Dist: aiosqlite>=0.22.1
Requires-Dist: catsu>=0.1.8
Requires-Dist: chonkie>=1.7.0
Requires-Dist: duckdb>=1.5.4
Requires-Dist: fastapi>=0.139.0
Requires-Dist: fastapi-mcp>=0.4.0
Requires-Dist: fastembed>=0.8.0
Requires-Dist: httpx>=0.28.1
Requires-Dist: huey>=3.1.1
Requires-Dist: inflect>=7.5.0
Requires-Dist: lancedb>=0.34.0
Requires-Dist: model2vec>=0.8.2
Requires-Dist: numpy>=2.2
Requires-Dist: opentelemetry-api>=1.43.0
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.43.0
Requires-Dist: opentelemetry-instrumentation-fastapi>=0.64b0
Requires-Dist: opentelemetry-sdk>=1.43.0
Requires-Dist: prometheus-client>=0.25.0
Requires-Dist: pyarrow>=24.0.0
Requires-Dist: pydantic>=2.13.4
Requires-Dist: pydantic-settings>=2.14.2
Requires-Dist: pypdf>=6.14.2
Requires-Dist: qdrant-client>=1.18.0
Requires-Dist: rerankers>=0.10.0
Requires-Dist: scikit-learn>=1.9.0
Requires-Dist: scipy>=1.18.0
Requires-Dist: spacy>=3.8.3,<4
Requires-Dist: transformers>=5.13.0
Requires-Dist: umap-learn>=0.5.12
Requires-Dist: uuid6>=2025.0.1
Requires-Dist: uvicorn>=0.51.0
Requires-Dist: gliner>=0.2.24 ; extra == 'gliner'
Requires-Dist: gliner-spacy>=0.0.11 ; extra == 'gliner'
Requires-Dist: onnxruntime>=1.27.0 ; extra == 'gliner'
Requires-Dist: torch>=2.13.0 ; extra == 'html'
Requires-Dist: igraph>=1.0.0 ; extra == 'leiden'
Requires-Dist: leidenalg>=0.12.0 ; extra == 'leiden'
Requires-Dist: en-core-web-sm ; extra == 'ner'
Requires-Python: >=3.13
Project-URL: Homepage, https://github.com/hbmartin/refindery
Project-URL: Repository, https://github.com/hbmartin/refindery
Project-URL: Changelog, https://github.com/hbmartin/refindery/releases
Provides-Extra: gliner
Provides-Extra: html
Provides-Extra: leiden
Provides-Extra: ner
Description-Content-Type: text/markdown

# Refindery

A local, single-machine retrieval engine over the web pages you read.

Upstream capture systems (browser extensions, history readers) extract
main-body text and POST it here. Refindery chunks, embeds, indexes, clusters,
and extracts entities from that text, then serves hybrid retrieval over it via
a local HTTP API and an MCP server.

**This is a retrieval engine, not a Q&A system.** It returns ranked, grounded
passages with provenance. Synthesis is the caller's job — typically an LLM
agent (e.g. Claude via MCP) that treats Refindery as a tool. No generation
appears on the query path.

## Jobs to be done

1. **Refind** — "I read something about X, take me back to it."
2. **Synthesize** — "What have I learned about Y?" (agent-mediated; Refindery supplies the passages)
3. **Resurface** — "What have I been reading a lot about?" (clusters, similarity)

## Architecture

Hexagonal / ports-and-adapters. A single non-blocking asyncio process hosts
the FastAPI app, the MCP server, and the job queue consumer; CPU-bound work
runs in a process pool.

| Port              | Default adapter                | Alternatives                       |
| ----------------- | ------------------------------ | ---------------------------------- |
| `VectorStore`     | Qdrant (Docker)                | LanceDB (in-process, zero daemon)  |
| `MetadataStore`   | SQLite (WAL)                   | Postgres (v2)                      |
| `Embedder`        | Voyage (via catsu)             | Cohere, OpenAI, local              |
| `Reranker`        | Cohere / Voyage (via rerankers)| local cross-encoders               |
| `EntityExtractor` | spaCy + gazetteer              | GLiNER (extra), LLM                |
| `ClusterEngine`   | UMAP + HDBSCAN                 | KMeans, Leiden (extra)             |

Observability: OpenTelemetry traces (off by default), structured JSON logs,
Prometheus `/metrics`, and a DuckDB append-only query log — the substrate for
offline retrieval evals: `refindery eval score` computes nDCG/MRR/recall from
logged queries + `/v1/feedback` labels, and `refindery eval replay` diffs two
configurations (models or rerank on/off) over the same golden set.

## Quickstart

### Minimal profile (no Docker)

```bash
uv sync --extra ner
export REFINDERY_AUTH_TOKEN="$(openssl rand -hex 24)"
export REFINDERY_VECTOR_STORE=lancedb
export VOYAGE_API_KEY=...           # or configure another embedding provider
python -m refindery
```

### Docker profile (Qdrant, the default store)

```bash
uv sync --extra ner
docker compose up -d qdrant
export REFINDERY_AUTH_TOKEN="$(openssl rand -hex 24)"
python -m refindery
```

### Fully containerized

The multi-stage `Dockerfile` builds a slim image with the `ner` extra
(no torch/gliner; add extras to the sync lines if you need them). Data
lives on the `refindery_data` volume, model caches on `refindery_models`.

```bash
export REFINDERY_AUTH_TOKEN="$(openssl rand -hex 24)"
export VOYAGE_API_KEY=...
docker compose up -d --build
curl -s http://127.0.0.1:8000/healthz
```

### Ingest and search

```bash
curl -s -X POST http://127.0.0.1:8000/v1/pages \
  -H "Authorization: Bearer $REFINDERY_AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/article", "title": "An Article",
       "body_extracted": "Plain text main body content...",
       "fetched_at": "2026-07-08T10:00:00Z", "source": "extension"}'

curl -s -X POST http://127.0.0.1:8000/v1/search \
  -H "Authorization: Bearer $REFINDERY_AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "main body content"}'
```

### MCP

The MCP server is served over streamable HTTP at `/mcp` (same bearer token):

```bash
claude mcp add --transport http refindery http://127.0.0.1:8000/mcp \
  --header "Authorization: Bearer $REFINDERY_AUTH_TOKEN"
```

Read-only tools are exposed by default (`search`, `get_page`, `similar_to`,
`list_clusters`, …). Mutating tools (`add_page`, `forget`) are opt-in via
`REFINDERY_MCP__ENABLE_MUTATING_TOOLS=true`. That flag controls visibility
only — authorization comes from the token's scopes on every transport.

### Auth tokens and scopes

`REFINDERY_AUTH_TOKEN` is a single full-access token. To hand each capture
source or agent its own revocable token, configure named tokens with `read`
or `write` scopes (write implies read; both forms can coexist):

```bash
export REFINDERY_AUTH_TOKENS='[
  {"name": "chrome-capture", "token": "...", "scopes": ["write"]},
  {"name": "agent",          "token": "...", "scopes": ["read"]}
]'
```

Read-scoped tokens can search, browse, compare, and record feedback;
mutating endpoints (`add_page`, `forget`, model management, …) return 403
without the `write` scope.

## HTTP API

```
POST   /v1/pages                      ingest (body_extracted XOR body_html; neither → fetch)
GET    /v1/pages/{id}                 full body_text + metadata
GET    /v1/pages/{id}/status          queued|indexing|indexed|failed|dead
GET    /v1/pages/{id}/similar         ?mediation=vector|cluster|entity&k=
GET    /v1/pages/{id}/entities
POST   /v1/search                     hybrid dense+sparse, RRF, rerank, filters
POST   /v1/compare                    A/B embedding models (Jaccard@k, RBO, Kendall's τ)
GET    /v1/clusters                   ?include_tombstoned=false
GET    /v1/clusters/{id}
POST   /v1/clusters/recompute
GET    /v1/entities/{id_or_form}
POST   /v1/forget                     purge + blacklist atomically
GET    /v1/blacklist
DELETE /v1/blacklist/{id}
POST   /v1/models                     register embedding model
POST   /v1/models/{id}/backfill       dry-run estimate, then confirm
POST   /v1/models/{id}/activate
DELETE /v1/models/{id}
POST   /v1/feedback                   { query_id, page_id, relevant }
GET    /healthz  /readyz  /metrics
```

Bearer token is always required, even on loopback. The server binds to
`127.0.0.1` by default.

## Optional extras

| Extra       | Enables                                        | Pulls in            |
| ----------- | ---------------------------------------------- | ------------------- |
| `html`      | `body_html` / fetched-HTML extraction (pulpie) | torch (~2 GB)       |
| `gliner`    | GLiNER zero-shot NER                           | gliner, onnxruntime |
| `ner`       | spaCy NER model                                | en_core_web_sm      |
| `leiden`    | Leiden clustering                              | igraph, leidenalg   |

Entity extraction is required at startup. The default extractor chain needs
the `ner` extra unless you configure a healthy gazetteer, GLiNER, or LLM
extractor.

See [Architecture](docs/architecture.md) for the ports/adapters map, the
ingest→index→cluster data flow, and the search pipeline. See
[Operations](docs/operations.md) for alpha reset commands, query-log
purging, job lease behavior, vector-store caveats, and accepted risks.

## Development

See [CONTRIBUTING.md](CONTRIBUTING.md). TL;DR: `uv sync --all-groups --extra ner`,
then `uv run ruff format . && uv run ruff check . && uv run pytest && uv run ty check && uv run pyrefly check`.
