Metadata-Version: 2.4
Name: refindery
Version: 0.2.0
Summary: A local, single-machine retrieval engine over the web pages you read.
Keywords: retrieval,search,rag,embeddings,mcp,browser-history
Author: Harold Martin
Author-email: Harold Martin <harold.martin@gmail.com>
License-Expression: MIT
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3.13
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Developers
Classifier: Topic :: Text Processing :: Indexing
Requires-Dist: aiosqlite>=0.22.1
Requires-Dist: catsu>=0.1.8
Requires-Dist: chonkie>=1.7.0
Requires-Dist: duckdb>=1.5.4
Requires-Dist: fastapi>=0.139.0
Requires-Dist: fastapi-mcp>=0.4.0
Requires-Dist: fastembed>=0.8.0
Requires-Dist: httpx>=0.28.1
Requires-Dist: huey>=3.1.1
Requires-Dist: inflect>=7.5.0
Requires-Dist: lancedb>=0.34.0
Requires-Dist: model2vec>=0.8.2
Requires-Dist: numpy>=2.2
Requires-Dist: opentelemetry-api>=1.43.0
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.43.0
Requires-Dist: opentelemetry-instrumentation-fastapi>=0.64b0
Requires-Dist: opentelemetry-sdk>=1.43.0
Requires-Dist: prometheus-client>=0.25.0
Requires-Dist: pyarrow>=24.0.0
Requires-Dist: pydantic>=2.13.4
Requires-Dist: pydantic-settings>=2.14.2
Requires-Dist: pypdf>=6.14.2
Requires-Dist: qdrant-client>=1.18.0
Requires-Dist: rerankers>=0.10.0
Requires-Dist: scikit-learn>=1.9.0
Requires-Dist: scipy>=1.18.0
Requires-Dist: spacy>=3.8.3,<4
Requires-Dist: transformers>=5.13.0
Requires-Dist: umap-learn>=0.5.12
Requires-Dist: uuid6>=2025.0.1
Requires-Dist: uvicorn>=0.51.0
Requires-Dist: gliner>=0.2.24 ; extra == 'gliner'
Requires-Dist: gliner-spacy>=0.0.11 ; extra == 'gliner'
Requires-Dist: onnxruntime>=1.27.0 ; extra == 'gliner'
Requires-Dist: torch>=2.13.0 ; extra == 'html'
Requires-Dist: igraph>=1.0.0 ; extra == 'leiden'
Requires-Dist: leidenalg>=0.12.0 ; extra == 'leiden'
Requires-Dist: en-core-web-sm ; extra == 'ner'
Requires-Python: >=3.13
Project-URL: Homepage, https://github.com/hbmartin/refindery
Project-URL: Repository, https://github.com/hbmartin/refindery
Project-URL: Changelog, https://github.com/hbmartin/refindery/releases
Provides-Extra: gliner
Provides-Extra: html
Provides-Extra: leiden
Provides-Extra: ner
Description-Content-Type: text/markdown

# Refindery

A local, single-machine retrieval engine over the web pages you read.

Upstream capture systems (browser extensions, history readers) extract
main-body text and POST it here. Refindery chunks, embeds, indexes, clusters,
and extracts entities from that text, then serves hybrid retrieval over it via
a local HTTP API and an MCP server.

📖 **Full documentation: <https://hbmartin.github.io/refindery/>**

## Companion projects

- **[Refindery Chrome Extension](https://github.com/hbmartin/refindery-chrome-extension)** — capture pages as you browse.
- **[Browser History Refindery](https://github.com/hbmartin/browser-history-refindery)** — import and search your browser history.

**This is a retrieval engine, not a Q&A system.** It returns ranked, grounded
passages with provenance. Synthesis is the caller's job — typically an LLM
agent (e.g. Claude via MCP) that treats Refindery as a tool. No generation
appears on the query path.

## Jobs to be done

1. **Refind** — "I read something about X, take me back to it."
2. **Synthesize** — "What have I learned about Y?" (agent-mediated; Refindery supplies the passages)
3. **Resurface** — "What have I been reading a lot about?" (clusters, similarity)

## Quickstart

The fastest path on macOS (Homebrew, no Docker) writes a private `.env` with a
generated auth token and the daemon-free LanceDB profile:

```bash
VOYAGE_API_KEY=... ./scripts/setup-macos.sh --start
```

On later runs, start the server with the generated environment:

```bash
uv run --env-file .env refindery serve
```

Then ingest a page and search it:

```bash
curl -s -X POST http://127.0.0.1:8000/v1/pages \
  -H "Authorization: Bearer $REFINDERY_AUTH_TOKEN" -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/article", "title": "An Article",
       "body_extracted": "Plain text main body content...",
       "fetched_at": "2026-07-08T10:00:00Z", "source": "extension"}'

curl -s -X POST http://127.0.0.1:8000/v1/search \
  -H "Authorization: Bearer $REFINDERY_AUTH_TOKEN" -H "Content-Type: application/json" \
  -d '{"query": "main body content"}'
```

Other install paths (Docker/Qdrant, manual, fully containerized) are in the
[Installation guide](https://hbmartin.github.io/refindery/getting-started/installation/).

## Documentation

The [documentation site](https://hbmartin.github.io/refindery/) is the canonical
reference:

- [**Getting started**](https://hbmartin.github.io/refindery/getting-started/) — install, quickstart, and validation.
- [**Guides**](https://hbmartin.github.io/refindery/guides/) — ingest, search, MCP, evaluation, clustering, entities, models, deletion.
- [**Configuration**](https://hbmartin.github.io/refindery/configuration/) — the settings model, deployment profiles, auth, tuning, observability.
- [**Architecture**](https://hbmartin.github.io/refindery/architecture/) — hexagonal ports/adapters, data flow, and the data model.
- [**Operations**](https://hbmartin.github.io/refindery/operations/) — reset, purge, job leases, and accepted risks.
- [**Reference**](https://hbmartin.github.io/refindery/reference/) — HTTP API, upstream ingest API, MCP tools, CLI, and the Python API.

## Development

See [CONTRIBUTING.md](CONTRIBUTING.md) and the
[Contributing docs](https://hbmartin.github.io/refindery/contributing/).
TL;DR: `uv sync --all-groups --extra ner`, then
`uv run ruff format . && uv run ruff check . && uv run pytest && uv run ty check && uv run pyrefly check`.

Preview the docs locally with `uv run zensical serve`.

## License

MIT — see [LICENSE.txt](LICENSE.txt).
