Metadata-Version: 2.4
Name: groundrag
Version: 0.3.0
Summary: Production-grade advanced RAG: multi-format ingest, hybrid recall, cross-encoder rerank, cited generation and evaluation.
Author: andrea9293
License-Expression: MIT
Project-URL: Homepage, https://github.com/andrea9293/groundrag
Project-URL: Repository, https://github.com/andrea9293/groundrag
Project-URL: Issues, https://github.com/andrea9293/groundrag/issues
Keywords: rag,retrieval,llm,mcp,lancedb,docling,markitdown,translation,multilingual
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.6
Requires-Dist: pyyaml>=6.0
Requires-Dist: numpy>=1.26
Requires-Dist: pyarrow>=23.0.1
Requires-Dist: lancedb>=0.25
Requires-Dist: pypdf>=6.14.2
Requires-Dist: python-docx>=1.1
Requires-Dist: python-pptx>=1.0
Requires-Dist: openpyxl>=3.1
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: fastembed>=0.5
Requires-Dist: openai>=1.40
Requires-Dist: defusedxml>=0.7.1
Provides-Extra: api
Requires-Dist: fastapi>=0.110; extra == "api"
Requires-Dist: uvicorn>=0.29; extra == "api"
Provides-Extra: mcp
Requires-Dist: mcp>=2.2; extra == "mcp"
Provides-Extra: docling
Requires-Dist: docling>=2.0; extra == "docling"
Provides-Extra: markitdown
Requires-Dist: markitdown[all]>=0.1.8; extra == "markitdown"
Requires-Dist: markitdown-ocr>=0.1; extra == "markitdown"
Requires-Dist: rapidocr>=3; extra == "markitdown"
Requires-Dist: onnxruntime; extra == "markitdown"
Provides-Extra: parsers
Requires-Dist: groundrag[docling,markitdown]; extra == "parsers"
Provides-Extra: tiktoken
Requires-Dist: tiktoken>=0.7; extra == "tiktoken"
Provides-Extra: translation
Requires-Dist: ctranslate2>=4.4; extra == "translation"
Requires-Dist: sentencepiece>=0.2; extra == "translation"
Requires-Dist: lingua-language-detector>=2.0; extra == "translation"
Requires-Dist: huggingface-hub>=0.24; extra == "translation"
Requires-Dist: transformers>=4.40; extra == "translation"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Requires-Dist: reportlab>=4.0; extra == "dev"
Requires-Dist: types-PyYAML; extra == "dev"
Requires-Dist: fastapi>=0.110; extra == "dev"
Requires-Dist: httpx>=0.27; extra == "dev"
Requires-Dist: mcp>=2.2; extra == "dev"
Dynamic: license-file

# groundrag

A production-grade **advanced RAG** system in Python, usable from a CLI, an HTTP API or as an
**MCP server** (protocol revision 2026-07-28): multi-format ingestion, structure-aware
chunking, hybrid dense + BM25 retrieval on LanceDB, query rewriting / multi-query / HyDE,
cross-encoder reranking, cited generation with an explicit "not found" fallback, and an
evaluation gate (recall@k, MRR, nDCG, faithfulness, answer relevance, abstention).

**Docs:** [usage guide](https://github.com/andrea9293/groundrag/blob/main/docs/USAGE.md) · [configuration reference](https://github.com/andrea9293/groundrag/blob/main/docs/CONFIG.md) ·
[design and default rationale](https://github.com/andrea9293/groundrag/blob/main/docs/DESIGN.md).

> [!NOTE]
> **English models, automatic translation.**
>
> - The default models (`bge-small-en-v1.5`, `ms-marco-MiniLM-L-6-v2`) only understand English, so
>   documents and queries in other languages are translated locally to English (Opus-MT) while
>   citations keep the original text.
> - `translation.enabled: auto` (the default) turns it on when `pip install 'groundrag[translation]'`
>   is installed, otherwise it warns once and runs untranslated; `true` fails at startup without the
>   extra, `false` turns it off.
> - Use it **across languages** (e.g. English questions over Italian documents: 3 to 11 of 11
>   answerable questions answered correctly). With questions and documents in the **same**
>   language it adds nothing and costs a little quality: set `translation.enabled: false`.
> - List your corpus languages (`translation.languages: [it]`): with `[]` the detector loads every
>   language (~1 GB RAM) and a warning is logged.
> - Details, tiers and licenses: [`docs/CONFIG.md`](https://github.com/andrea9293/groundrag/blob/main/docs/CONFIG.md#translation) and
>   [`docs/DESIGN.md` §6](https://github.com/andrea9293/groundrag/blob/main/docs/DESIGN.md#6-translate-to-english-ingestion).
> - The alternative multilingual stack in [`docs/USAGE.md` §10](https://github.com/andrea9293/groundrag/blob/main/docs/USAGE.md#10-tuning-cheatsheet) is
>   ~20x slower and its reranker (`jina-reranker-v2`) is licensed **CC-BY-NC-4.0 (non-commercial only)**.

## Installation

```bash
pip install groundrag
# with extras (combine as needed):
pip install "groundrag[parsers,api,mcp,tiktoken,translation]"
```

> The GitHub repository stays private until December 2026: until then the git form below and the
> repository links in this README work only for collaborators.

Or straight from GitHub:

```bash
pip install "git+https://github.com/andrea9293/groundrag.git"
pip install "groundrag[parsers,api,mcp,tiktoken,translation] @ git+https://github.com/andrea9293/groundrag.git"
```

| Extra | Enables |
|---|---|
| `[parsers]` | layout-aware parsing and image description (Docling + MarkItDown, `ingest.parser: hybrid`) |
| `[api]` | HTTP API (`groundrag serve`) |
| `[mcp]` | MCP server (`groundrag mcp`) |
| `[tiktoken]` | exact token counts (`tokens.counter: tiktoken`) |
| `[translation]` | translation to English (lingua, CTranslate2, transformers, sentencepiece, huggingface-hub); switches on `translation.enabled: auto`. The first model conversion also needs `torch` |

Development install:

```bash
git clone https://github.com/andrea9293/groundrag.git && cd groundrag
make venv install          # python3 -m venv .venv && pip install -e ".[dev]"
```

The repository's `config.yaml` is picked up from the current directory. To keep personal settings
(endpoint, model, paths) out of git, copy it to a gitignored `config.local.yaml` and run
`GROUNDRAG_CONFIG=config.local.yaml groundrag ...` (or `--config config.local.yaml`).

## Quickstart

The `make` targets below need a **clone of the repository** (they use `data/sample/`, `config.yaml`
and the Makefile). If you installed from PyPI, skip to the `groundrag ...` commands in the next
block.

```bash
make ingest                # index data/sample (16 file types) into collection "default"
make query Q="What is the warranty period of the RX-200?"
make demo                  # ingest + questions answered from pdf, docx, xlsx, rst (+ a "not found")
make test                  # pytest (offline, deterministic stand-ins)
make test-real             # opt-in: real embedding model, cross-encoder, LLM (network)
make eval                  # gold set → metrics → reports/eval_latest.json; non-zero exit on regression
make serve                 # HTTP API (pip install -e ".[api]")
make mcp                   # MCP server, stdio (pip install -e ".[mcp]") — see docs/USAGE.md §8
```

For layout-aware parsing and image description (hybrid Docling + MarkItDown): `pip install -e ".[parsers]"`
(see [`docs/USAGE.md` §5](https://github.com/andrea9293/groundrag/blob/main/docs/USAGE.md#5-ingesting-documents)).

With an OpenAI-compatible LLM: `export OPENAI_API_KEY=...` (or set `llm.base_url` for
Ollama / vLLM). Without one, the pipeline runs fully offline (see *Limitations*).

Quick start commands (your own documents; these work with `pip install groundrag`):

```bash
# 1. index your documents into a collection (incremental: unchanged files are skipped)
groundrag ingest docs/ --collection handbook

# 2. ask a question, offline: no LLM, the answer quotes the best sentences
groundrag --set llm.backend=none query "How many vacation days do I get?" --collection handbook

# 3. ask with an LLM (cited answer, refuses when the documents don't say)
#    the key comes from the environment (never commit it); --set llm.base_url=... / llm.model=... pick another endpoint
export OPENAI_API_KEY=...
groundrag query "How many vacation days do I get?" --collection handbook

# 4. create a gold set with an LLM, review it, then set the collection's threshold
groundrag gold-generate --collection handbook --out data/eval/gold_handbook.jsonl
# dry run (nothing written), then --save
groundrag calibrate --collection handbook --gold data/eval/gold_handbook.jsonl
groundrag calibrate --collection handbook --gold data/eval/gold_handbook.jsonl --save

# 5. measure quality, inspect, clean up
groundrag eval --gold data/eval/gold_handbook.jsonl --collection handbook --no-ingest
groundrag stats                                    # collections, rows, calibration state
groundrag drop --collection handbook               # delete a collection (asks to confirm)
```

More CLI examples:

```bash
# these queries use the LLM: run the export from step 3 first
groundrag query "vacation days" --filter format=docx --filter date_from=2026-01-01
groundrag query "salary band?" --groups hr-admins
groundrag query "and the RX-400?" --history chat.json --json
groundrag --set rerank.top_k=8 query "..."                   # override any config key
groundrag ingest docs/ --collection handbook --prune         # incremental; --rebuild to reset
```

## Architecture

```
ingest ─► clean/dedup ─► translate to English (when active; keeps original) ─► chunk (recursive|semantic|parent-child, metadata) ─► embed (normalised)
       ─► LanceDB: vectors (flat→HNSW) + BM25 FTS (+ BM25 on original_text) + scalar/ACL columns + manifest
query ─► cleanup ─► translate to English (when active) ─► standalone rewrite ─► multi-query / HyDE ─► routing
      ─► recall: dense + BM25 per variant, pre-filtered ─► RRF (k=60) ─► top-50
      ─► rerank: cross-encoder ─► top-5 ─► score < min_score ? "not found"
      ─► context "[n] source — page N" (token budget) ─► generate (T=0.1, cited, 1 retry)
eval  ─► gold.jsonl ─► retrieval + answer metrics ─► JSON report + RAGAS JSONL ─► gate
```

| Module | Path |
|---|---|
| Config / logging / models | `src/groundrag/{config,logging_utils,models,tokens,text}.py` |
| Ingest (pdf, docx, pptx, xlsx, csv, tsv, md, html, txt, rst, eml, epub, json, xml) | `src/groundrag/ingest/` |
| Chunking | `src/groundrag/chunking/` |
| Embeddings (FastEmbed `bge-small-en-v1.5`, hashing fallback) | `src/groundrag/embeddings/` |
| Index (LanceDB, safe filters, ACL) | `src/groundrag/index/` |
| Translation (language detection, NMT tiers, placeholders, cache) | `src/groundrag/translation/` |
| Query transforms | `src/groundrag/query/` |
| Hybrid retrieval + metrics | `src/groundrag/retrieval/` |
| Rerankers | `src/groundrag/rerank/` |
| LLM client, context, generator | `src/groundrag/generation/` |
| Façade | `src/groundrag/pipeline.py` |
| Evaluation | `src/groundrag/eval/`, `data/eval/gold.jsonl` |
| CLI / API / MCP server | `src/groundrag/cli.py`, `src/groundrag/api.py`, `src/groundrag/mcp_server.py` |

Every log line is JSON with `trace_id` and per-stage `timings_ms`.

## Agent skills

The repository ships [Agent Skills](https://agentskills.io/specification) in [`skills/`](https://github.com/andrea9293/groundrag/tree/main/skills/)
that teach a coding agent how to use groundrag: `groundrag-cli`, `groundrag-python-integration`,
`groundrag-ingest-parsers`, `groundrag-tuning` and `groundrag-evaluation`. Install them with:

```bash
npx skills add andrea9293/groundrag                       # pick the skills interactively
npx skills add andrea9293/groundrag --skill groundrag-cli    # or a single one
```

## Configuration

**Full per-key reference (allowed values, defaults, effects): [`docs/CONFIG.md`](https://github.com/andrea9293/groundrag/blob/main/docs/CONFIG.md).**
Most important keys (code defaults; `config.yaml` has the same values except where noted):

| Key | Default | Meaning |
|---|---|---|
| `ingest.parser` | `hybrid` | `hybrid` \| `docling` \| `markitdown` \| `native` (extras `[parsers]`) |
| `ingest.extensions` | `[]` in code; 16 formats in `config.yaml` | files that are ingested |
| `chunking.strategy` / `chunk_size` / `chunk_overlap` | `recursive` / 400 / 60 (`config.yaml`: `parent_child` / 150 / 30) | how text is split |
| `embeddings.backend` / `model` | `auto` / `BAAI/bge-small-en-v1.5` | FastEmbed, hashing fallback |
| `translation.enabled` / `tier` / `languages` | `auto` / `fast` / `[]` | translate to English when the `[translation]` extra is installed; `true` \| `false` to force |
| `retrieval.dense_top_k` / `sparse_top_k` / `candidates_top_k` | 50 / 50 / 50 | recall stage (dense + BM25 + RRF) |
| `rerank.backend` / `model` | `auto` / `Xenova/ms-marco-MiniLM-L-6-v2` | cross-encoder, lexical fallback |
| `rerank.top_k` | 5 | chunks kept after reranking |
| `rerank.min_score.cross_encoder` / `min_score_llm.cross_encoder` | 0.5 / 0.05 | "not found" thresholds without / with an LLM |
| `llm.backend` / `model` / `base_url` / `api_key_env` | `auto` / `gpt-4o-mini` / `null` / `OPENAI_API_KEY` | any OpenAI-compatible endpoint; the key is never read from the file |
| `query.multi_query.n` / `query.hyde.enabled` | 3 / `false` | LLM query expansion |
| `generation.temperature` / `max_tokens` / `max_context_tokens` | 0.1 / 2000 / 3000 | answer building |
| `llm.max_tokens` / `llm.reasoning_effort` | 6000 / `null` | budget of the judge, gold-generate and query steps (reasoning models); optional reasoning effort |
| `eval.judge` / `eval.thresholds` | `auto` / `{}` in code; 8 gates in `config.yaml` | eval judge and pass/fail gate |
| `security.enforce_acl` | `true` | ACL filtering |
| `api.*` / `mcp.*` | `127.0.0.1:8000`, `/ingest` off / `stdio`, ingest off | HTTP API and MCP server |

Justifications for the defaults: [`docs/DESIGN.md`](https://github.com/andrea9293/groundrag/blob/main/docs/DESIGN.md) §2.

## Limitations

* **Offline fallbacks.** Without access to HuggingFace / an LLM API the `auto` backends use
  a hashing embedder, a lexical reranker and an extractive answerer (all logged as WARNING).
  They're deterministic and pass the eval gate on the sample set, but they don't understand
  synonyms (e.g. "EU" vs "European Union"). For production quality, use FastEmbed models and an LLM.
* The token count is a regex estimate unless `tokens.counter: tiktoken`. The contextual header
  (`title > section`) sits on top of `chunk_size`.
* **Translation.** Translation quality is that of the NMT model: tables, names and short queries can
  come out wrong, which cost a few answers on same-language Italian questions in the evaluation
  (one run, 13 questions). Only English is a target language. Routing keywords must be in English
  while translation is active. The first ingest after translation becomes active re-chunks the
  documents it sees; unseen ones need `--rebuild` or `--prune`.
* The BM25 index is rebuilt after each ingest batch, which is fine up to roughly 10⁵ chunks. For larger
  corpora, batch your ingests.
* The HTTP API has no authentication. Run it behind a gateway. The MCP server over HTTP requires
  OAuth (or an explicit opt-out for use behind a gateway).
* The MCP *Tasks* extension is not yet supported by the Python SDK, so MCP `ingest_documents` is
  synchronous (and disabled by default).
* The gold set is small (23 questions: 20 answerable and 3 unanswerable traps) and synthetic. Extend it with real questions before relying on it
  for tuning.
* **"Not found" thresholds.** Without an LLM no threshold can reject questions that are on topic
  but have no answer in the documents (the cross-encoder scores them ~0.98); only an LLM does
  that. `groundrag calibrate` reports how many it would reject (`trap_rejection`) instead of
  hiding it. With few gold questions (< 30 answerable) the threshold is flagged `few_data`.
* **Generated gold sets** (`groundrag gold-generate`) are easier than real questions and are marked
  `needs_review`. Use them as a starting point and recalibrate with real questions.
