Metadata-Version: 2.4
Name: mkd-retriever
Version: 0.3.0
Summary: Modular, project-agnostic RAG retrieval layer: independent retrieval strategies behind a small shared core.
Author: Mehmet Kaan Durupunar
License: MIT
Project-URL: Homepage, https://pypi.org/project/mkd-retriever/
Keywords: rag,retrieval,information-retrieval,intent-classification,text-to-sql,llm,nlp,modular
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2
Requires-Dist: python-dotenv>=1
Provides-Extra: top-n
Provides-Extra: query-rewriting
Provides-Extra: intent-classification
Requires-Dist: langchain-core>=0.3; extra == "intent-classification"
Requires-Dist: langchain-openai>=0.2; extra == "intent-classification"
Provides-Extra: raptor
Provides-Extra: agentic
Provides-Extra: db-query
Requires-Dist: text2sql-engine[openai]>=0.2; extra == "db-query"
Provides-Extra: eval
Provides-Extra: all
Requires-Dist: langchain-core>=0.3; extra == "all"
Requires-Dist: langchain-openai>=0.2; extra == "all"
Requires-Dist: text2sql-engine[openai]>=0.2; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: build>=1; extra == "dev"
Requires-Dist: twine>=5; extra == "dev"
Dynamic: license-file

# mkd-retriever

A **modular, project-agnostic RAG retrieval layer**. Several retrieval
strategies — intent classification, text-to-SQL database query, and more to
come (top-N vector search, query rewriting, RAPTOR, agentic flows) — live side
by side in one package **without depending on one another**.

The PyPI name is `mkd-retriever`; the import name is `retrieval`
(`import retrieval`).

```bash
pip install mkd-retriever                        # core only
pip install "mkd-retriever[intent_classification]"
pip install "mkd-retriever[db_query]"
pip install "mkd-retriever[all]"                 # everything implemented so far
```

## Why it is built this way

Building retrieval methods on top of one another is a Jenga tower: changing or
removing one piece topples the rest. Instead, every method is an **independent
module**. The hard rules (enforced, see [`ARCHITECTURE.md`](./ARCHITECTURE.md)):

1. **A module never imports another module.** Cross-module communication is the
   consuming project's job, not this library's.
2. **Everything shared lives in `retrieval.core`, and core stays small.** Only
   minimal interfaces and common data schemas — never a module's implementation
   detail. The dependency arrow is always *module → core*.
3. **Dependencies are isolated per module.** Each module has its own
   optional-dependency group; a library one module needs never leaks into
   another module's environment.
4. **Each module is understandable and testable on its own.**
5. **Intent classification only returns a label — it does not route.** Mapping a
   label to a retrieval method is the consuming project's decision.

## What's inside

```
retrieval/
  core/        shared interfaces + ingestion-contract schemas (small, stable)
  eval/        golden-set format + metrics + async runner (cross-cutting)
  modules/
    intent_classification/   query -> intent label   (usable)
    db_query/                query -> SQL -> rows     (usable)
    top_n/                   vector search            (planned)
    query_rewriting/         rewriting / expansion    (planned)
    raptor/                  hierarchical retrieval   (planned)
    agentic/                 multi-step flows         (planned)
```

### `retrieval.core`

The whole surface every module shares:

- `RetrievalResult` — a single scored item, returned identically by every
  backend (`id`, `score`, `text`, `metadata`).
- `Retriever` — the one async method a backend implements:
  `async def retrieve(query, k) -> list[RetrievalResult]`. `run_sync` is a thin
  convenience for synchronous callers.
- `IntentLabel` (9 classes) / `IntentResult`.
- Ingestion-contract schemas — pydantic models for the JSON an upstream
  ingestion/parsing project emits (chunks, product graph, document metadata).
  The contract is defined by *shape*, not by importing the producer, so any
  project emitting these shapes can feed this layer. Models set `extra="ignore"`
  to stay resilient to upstream format drift.

### `intent_classification`

A prompt-based classifier over any `langchain-core` `BaseChatModel`. It returns
one `IntentResult` and nothing more — the label → strategy mapping is the
consuming project's job.

```python
from retrieval.modules.intent_classification import (
    LLMIntentClassifier, build_default_chat_model,
)

clf = LLMIntentClassifier(build_default_chat_model())
result = await clf.classify("de1000 fiyati ne kadar")   # -> IntentResult
```

In tests, inject a fake `BaseChatModel` instead of the factory — no network,
no model download. From synchronous code, use the `classify_sync(clf, query)`
helper — the sync mirror of `run_sync` for retrievers.

### `db_query`

Retrieval by generating SQL and running it. A natural-language query is turned
into a SQL string by the [`text2sql-engine`](https://pypi.org/project/text2sql-engine/)
package (a 2-stage LLM pipeline), that SQL runs against a **ready-made**
database, and the rows come back as `RetrievalResult` items.

```python
from retrieval.modules.db_query import build_default_retriever

retriever = build_default_retriever(db_path="specs.db")
results = await retriever.retrieve("operating temperature of DE9001", k=5)
```

Two injected seams keep it testable and dependency-isolated:

- `SqlGenerator` (NL → SQL) — the production impl wraps a `text2sql` engine
  hitting a remote endpoint; unit tests inject a fake returning canned SQL.
- `SqlExecutor` (SQL → rows) — the shipped `SQLiteExecutor` uses only stdlib
  `sqlite3` and opens the database **read-only** by default, so a stray write in
  generated SQL fails at the engine level.

The score defaults to `1.0` (a SQL result is an exact match set, not a
similarity ranking; ordering is the query's own `ORDER BY`) and is overridable.
The full row is in `RetrievalResult.metadata`. This module connects to an
existing database — it does not build one.

**The pipeline is optional.** Every piece is importable on its own — you are not
forced through `DbQueryRetriever`:

```python
from retrieval.modules.db_query import SQLiteExecutor, rows_to_results, clean_sql

# Run your own SQL, reuse only the row -> RetrievalResult mapping:
cols, rows = SQLiteExecutor("specs.db").execute("SELECT * FROM products LIMIT 5")
results = rows_to_results(cols, rows, score=0.9)   # -> list[RetrievalResult]

# ...or just clean a model's SQL output, map a single row, swap either seam.
```

## Inference & the GPU rule

Model inference (LLM/embedding) is expected to run on a **separate machine
behind an OpenAI-compatible endpoint**. This library only sends requests to that
endpoint; it never downloads models or runs local inference. The default test
run (`pytest`) is fully mocked and never reaches a network endpoint — tests that
hit a live endpoint are marked `integration` and skipped by default.

## Development

```bash
pip install -e ".[all,dev]"
pytest
```

## License

MIT. See [LICENSE](LICENSE).
