Metadata-Version: 2.4
Name: retrieval-observatory
Version: 0.5.1
Summary: Local-first retrieval reliability for integrating, evaluating, and investigating multi-stage retrieval pipelines
Author-email: Ameya Kiwalkar <akiwalkar@berkeley.edu>
License-Expression: MIT
Project-URL: PyPI, https://pypi.org/project/retrieval-observatory/
Project-URL: Homepage, https://github.com/AmeyaKI/retrieval-observatory
Project-URL: Documentation, https://github.com/AmeyaKI/retrieval-observatory#readme
Project-URL: Repository, https://github.com/AmeyaKI/retrieval-observatory
Project-URL: Issues, https://github.com/AmeyaKI/retrieval-observatory/issues
Project-URL: Changelog, https://github.com/AmeyaKI/retrieval-observatory/releases
Keywords: rag,retrieval,evaluation,diagnostics,observability
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.0
Requires-Dist: httpx>=0.27
Requires-Dist: aiosqlite>=0.20
Requires-Dist: rich>=13.0
Requires-Dist: typer>=0.12
Requires-Dist: pyyaml>=6.0
Requires-Dist: numpy>=1.26
Provides-Extra: beir
Requires-Dist: datasets>=2.0; extra == "beir"
Requires-Dist: beir>=2.0; extra == "beir"
Provides-Extra: cohere
Requires-Dist: cohere>=5.0; extra == "cohere"
Provides-Extra: hf
Requires-Dist: sentence-transformers>=3.0; extra == "hf"
Requires-Dist: torch; extra == "hf"
Provides-Extra: dense
Requires-Dist: sentence-transformers>=3.0; extra == "dense"
Requires-Dist: faiss-cpu>=1.7; extra == "dense"
Requires-Dist: torch; extra == "dense"
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.2; extra == "langchain"
Provides-Extra: llamaindex
Requires-Dist: llama-index-core>=0.10; extra == "llamaindex"
Provides-Extra: haystack
Requires-Dist: haystack-ai>=2.0; extra == "haystack"
Provides-Extra: dspy
Requires-Dist: dspy-ai>=2.4; extra == "dspy"
Provides-Extra: openai-agents
Requires-Dist: openai-agents>=0.0.3; extra == "openai-agents"
Provides-Extra: pgvector
Requires-Dist: asyncpg>=0.29; extra == "pgvector"
Requires-Dist: pgvector>=0.3; extra == "pgvector"
Provides-Extra: postgres
Requires-Dist: asyncpg>=0.29; extra == "postgres"
Provides-Extra: dashboard
Requires-Dist: fastapi>=0.111; extra == "dashboard"
Requires-Dist: uvicorn>=0.29; extra == "dashboard"
Requires-Dist: python-multipart>=0.0.9; extra == "dashboard"
Provides-Extra: serve
Requires-Dist: fastapi>=0.111; extra == "serve"
Requires-Dist: uvicorn>=0.29; extra == "serve"
Requires-Dist: python-multipart>=0.0.9; extra == "serve"
Provides-Extra: qdrant
Requires-Dist: qdrant-client>=1.9; extra == "qdrant"
Provides-Extra: llm-judge
Requires-Dist: google-generativeai>=0.8; extra == "llm-judge"
Requires-Dist: anthropic>=0.28; extra == "llm-judge"
Requires-Dist: openai>=1.0; extra == "llm-judge"
Provides-Extra: demo
Requires-Dist: beir>=2.0; extra == "demo"
Requires-Dist: datasets>=2.0; extra == "demo"
Requires-Dist: rank-bm25>=0.2.2; extra == "demo"
Provides-Extra: classifier
Requires-Dist: scikit-learn>=1.4; extra == "classifier"
Requires-Dist: joblib>=1.3; extra == "classifier"
Provides-Extra: mcp
Requires-Dist: mcp>=1.2; extra == "mcp"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Requires-Dist: coverage; extra == "dev"
Requires-Dist: respx>=0.21; extra == "dev"
Requires-Dist: scikit-learn>=1.4; extra == "dev"
Requires-Dist: joblib>=1.3; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: types-PyYAML; extra == "dev"
Requires-Dist: mcp>=1.2; extra == "dev"
Requires-Dist: fastapi>=0.111; extra == "dev"
Requires-Dist: playwright>=1.45; extra == "dev"
Dynamic: license-file

# retobs

[![PyPI](https://img.shields.io/pypi/v/retrieval-observatory)](https://pypi.org/project/retrieval-observatory/)

retobs is a local-first reliability layer for retrieval pipelines. It helps you integrate observable retrieval stages, evaluate a callable, compare explicit Runs, and inspect recorded query evidence. It is not an answer evaluator or a leaderboard: when identity, topology, candidates, telemetry, or ground truth are unavailable, retobs reports that limit instead of inferring a conclusion.

## Install

```bash
pip install "retrieval-observatory[dashboard,mcp]"
```

## Integrate an existing project

Create and review a plan before any mutation. Apply consumes that reviewed plan; verify reports readiness only after it observes the declared topology, candidate evidence, and telemetry health.

```bash
retobs integrate . --phase plan --output retobs/integration-plan.json
retobs integrate . --phase apply --plan retobs/integration-plan.json
retobs integrate . --phase verify --plan retobs/integration-plan.json
```

Unresolved required mappings or stale file hashes block apply. The apply result lists every changed file and retains reversal information in its apply record.

## Evaluate a callable

```bash
retobs evaluate mypackage.search:retrieve --queries data/queries.jsonl --qrels data/qrels.jsonl --corpus data/corpus.jsonl
```

Use the returned Run ID with `retobs report`, `retobs compare`, and `retobs inspect-query`. A comparison is valid only when its required identities align; a query diagnosis is limited to evidence actually recorded.

## Investigate locally

```bash
retobs serve --db .retobs/results.db
```

The dashboard binds to `127.0.0.1` by default. It is unauthenticated and local-first; put it behind trusted controls before exposing it beyond loopback.

## What retobs records

- Evaluation Runs, manifests, query evidence, and complete or partial operator traces.
- Production traces scoped to a service and pipeline, including candidate transitions when instrumentation provides them.
- Instrumentation health: sampling, drops, serialization failures, retries, and permanent export failures.

These are evidence contracts, not guarantees that every integration can supply every field.

## Integration support

First-class integration paths are plain Python, HTTP, FastAPI, LangChain, and LlamaIndex. DSPy, Haystack, and OpenAI Agents are supported examples with narrower guarantees. See [integration support](docs/INTEGRATIONS.md) and the [agent runbook](docs/integrations/AGENT_QUICKSTART.md).

## Privacy and production safety

Queries, candidates, metadata, labels, and traces may be sensitive. Redaction runs before enqueue and persistence according to the integration manifest; queue capacity, overflow policy, and sampling are explicit telemetry configuration. Read [privacy](docs/PRIVACY.md) and [security](SECURITY.md) before production use.

## Documentation

- [Start](docs/START.md)
- [Workflow](docs/WORKFLOW.md)
- [Concepts](docs/CONCEPTS.md)
- [CLI, SDK, and MCP reference](docs/REFERENCE.md)
- [Architecture](docs/ARCHITECTURE.md)
- [Releases](https://github.com/AmeyaKI/retrieval-observatory/releases)

License: [MIT](LICENSE).
