Metadata-Version: 2.5
Name: ingestlib
Version: 1.5.0
Summary: Self-hosted document intelligence for RAG — parse, classify, split, and extract from PDF/DOCX/PPTX and images into searchable, cited chunks; query documents and your SQL databases in natural language; serve it all to agents over MCP
Project-URL: Homepage, https://github.com/LangModule/ingestlib
Project-URL: Documentation, https://langmodule.github.io/ingestlib/
Project-URL: Repository, https://github.com/LangModule/ingestlib
Project-URL: Issues, https://github.com/LangModule/ingestlib/issues
Author-email: ramacharanreddy-k <ramacharanreddykasireddy@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: chunking,document-intelligence,document-parsing,embeddings,hybrid-search,mcp,nl2sql,ocr,rag,retrieval,self-hosted,structured-extraction,text-to-sql,vector-database
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.12
Requires-Dist: alembic>=1.13
Requires-Dist: boto3>=1.43.29
Requires-Dist: botocore[crt]>=1.43
Requires-Dist: httpx>=0.27
Requires-Dist: langchain-aws>=1.5.1
Requires-Dist: langchain-openai>=1.0
Requires-Dist: mlx-vlm>=0.6.4; sys_platform == 'darwin' and platform_machine == 'arm64'
Requires-Dist: paddleocr>=3.7.0
Requires-Dist: paddlepaddle>=3.3.1
Requires-Dist: paddlex[genai-client,ocr]==3.7.1
Requires-Dist: pillow>=10.0
Requires-Dist: psycopg[binary]>=3.2
Requires-Dist: pydantic>=2.0
Requires-Dist: pypdfium2>=4.0
Requires-Dist: python-dotenv>=1.0
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: sqlalchemy>=2.0
Requires-Dist: sqlite-vec>=0.1.6
Provides-Extra: all
Requires-Dist: duckdb-engine>=0.13; extra == 'all'
Requires-Dist: duckdb>=1.1; extra == 'all'
Requires-Dist: fastembed>=0.8.0; extra == 'all'
Requires-Dist: mcp>=1.28; extra == 'all'
Requires-Dist: opensearch-py>=2.8; extra == 'all'
Requires-Dist: pgvector>=0.4; extra == 'all'
Requires-Dist: pinecone>=9.1.0; extra == 'all'
Requires-Dist: psycopg[binary]>=3.2; extra == 'all'
Requires-Dist: pymilvus>=2.6; extra == 'all'
Requires-Dist: pymongo>=4.10; extra == 'all'
Requires-Dist: pymysql>=1.1; extra == 'all'
Requires-Dist: qdrant-client>=1.18.0; extra == 'all'
Requires-Dist: snowflake-sqlalchemy>=1.6; extra == 'all'
Requires-Dist: sqlalchemy>=2.0; extra == 'all'
Requires-Dist: weaviate-client>=4.9; extra == 'all'
Provides-Extra: duckdb
Requires-Dist: duckdb-engine>=0.13; extra == 'duckdb'
Requires-Dist: duckdb>=1.1; extra == 'duckdb'
Requires-Dist: sqlalchemy>=2.0; extra == 'duckdb'
Provides-Extra: mcp
Requires-Dist: mcp>=1.28; extra == 'mcp'
Provides-Extra: milvus
Requires-Dist: pymilvus>=2.6; extra == 'milvus'
Provides-Extra: mongodb
Requires-Dist: pymongo>=4.10; extra == 'mongodb'
Provides-Extra: mysql
Requires-Dist: pymysql>=1.1; extra == 'mysql'
Requires-Dist: sqlalchemy>=2.0; extra == 'mysql'
Provides-Extra: opensearch
Requires-Dist: opensearch-py>=2.8; extra == 'opensearch'
Provides-Extra: pgvector
Requires-Dist: pgvector>=0.4; extra == 'pgvector'
Requires-Dist: psycopg[binary]>=3.2; extra == 'pgvector'
Provides-Extra: pinecone
Requires-Dist: pinecone>=9.1.0; extra == 'pinecone'
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.2; extra == 'postgres'
Requires-Dist: sqlalchemy>=2.0; extra == 'postgres'
Provides-Extra: qdrant
Requires-Dist: fastembed>=0.8.0; extra == 'qdrant'
Requires-Dist: qdrant-client>=1.18.0; extra == 'qdrant'
Provides-Extra: snowflake
Requires-Dist: snowflake-sqlalchemy>=1.6; extra == 'snowflake'
Requires-Dist: sqlalchemy>=2.0; extra == 'snowflake'
Provides-Extra: sql
Requires-Dist: sqlalchemy>=2.0; extra == 'sql'
Provides-Extra: weaviate
Requires-Dist: weaviate-client>=4.9; extra == 'weaviate'
Description-Content-Type: text/markdown

# ingestlib

![ingestlib — self-hosted document intelligence for RAG](assets/cover.png)

Self-hosted document intelligence for RAG pipelines. One library that takes a
raw document — PDF, DOCX, PPTX, or a PNG/JPEG/WebP image — and produces
searchable, **cited**, retrieval-ready chunks, plus schema-driven
extraction with verified citations: the territory of LlamaParse /
Reducto / Unstructured.io, running on your own stack.

```python
from ingestlib.services import ingest, retrieve

ingest("finance-10k.pdf")            # parse → classify → split → embed → vector store
result = retrieve("what were the total revenues?")
print(result.context)                # ranked chunks, each citing doc · page · section
```

**Documentation: [langmodule.github.io/ingestlib](https://langmodule.github.io/ingestlib/)** —
guides for every stage, the full configuration reference, and the API docs.

## What it does

| Stage | What you get |
|---|---|
| **Parse** | Layout-aware markdown per page: tables as HTML (merged cells intact), formulas as LaTeX, **charts converted to data tables** (estimated values marked `~`, printed callouts and growth labels captured), figures extracted as PNG crops with captions and AI descriptions — every block traceable to a bounding box on the page |
| **Classify** | Document-type label (`invoice`, `research_paper`, …) — open-ended, or constrained to your rules (per call or preset in `rules.yaml`, with page targeting) — confidence and ranked alternatives included. Works standalone with **no OCR** |
| **Split** | Sections (pages grouped by role: `methods`, `results`, … — LLM-discovered, or **your own categories** via rules) containing **natural chunks** — boundaries follow the content, tables never split, each chunk carries a `[category › section › heading]` breadcrumb in its `embedding_text` |
| **Extract** | **Your Pydantic schema, filled from the document** — one instance (`mode="one"`) or every instance in a batch (`mode="many"`, e.g. all receipts in a scanned expense bundle). Every field carries **verified provenance**: page + region citations checked against the parse, values grounded in the cited source text, and honest confidence — uncited or ungrounded answers are capped, hallucinated citations dropped |
| **Ingest** | The whole pipeline in one call — every stage's queryable output recorded in the internal **registry** (Postgres), its bytes (source, page images, figure crops, whole-doc markdown) in the artifact store (S3 or a local folder), vectors upserted, deduplicated by content checksum |
| **Retrieve** | Question → **hybrid search** (dense embeddings + lexical sparse, merged) → **rerank** (Jina by default; Amazon Rerank or none via `reranker:` in config.yaml) → hits with scores and citations, plus a prompt-ready context block |
| **Query databases** | The same `retrieve()` call also answers from your **SQL databases** — natural language → read-only generated SQL behind a permission boundary (read-only role + statement allowlist + LIMIT + timeout), with **verified-query** overrides for answers that must be exact. Postgres, MySQL, SQLite, DuckDB, Snowflake — merged with document results |

Engines: **PaddleOCR-VL-1.6** (0.9B VLM, runs on your GPU) for layout + recognition,
**Amazon Nova 2 Lite** for judgment (chart reading, review, classification,
chunk boundaries), **Nova multimodal embeddings**, **eight vector stores**
(Pinecone, Qdrant, SQLite, Postgres/pgvector, MongoDB, Milvus, OpenSearch,
Weaviate — all hybrid dense + sparse), **S3 or a local folder** for
artifacts (`artifact_store: s3 | local`). ~$0.002/page in LLM spend. An
**OpenAI backend** (GPT-5 vision-capable chat + text-embedding-3) ships
alongside Bedrock — flip `llm_provider: openai` / `embedding_provider:
openai` to run the whole pipeline on it instead — or `ollama` to keep
every LLM call on your own machine. See below.

## Quickstart

### 1. Requirements

- Python 3.12+ and [uv](https://github.com/astral-sh/uv)
- **AWS account** with Bedrock access (`us-east-1`): Nova 2 Lite + Nova 2
  multimodal embeddings — the default provider; the OpenAI backend or a
  local Ollama can run the whole pipeline instead (see below)
- **Vector database** — none at all by default: the sqlite connector
  stores vectors in a local file. Or a Pinecone account (serverless,
  free tier works), a Qdrant server (local docker or Qdrant Cloud), a Postgres
  with pgvector (RDS/Supabase/Neon or self-hosted), a MongoDB with search
  (Atlas any tier or 8.2+ self-managed), a Milvus (local docker or Zilliz
  Cloud), an OpenSearch (Amazon domain or local docker), a Weaviate (local
  docker or Weaviate Cloud) — each just one connection URL
- **Internal registry (Postgres)** — ingestlib's own metadata store, required for
  the **corpus path** (`ingest`/`retrieve`/lifecycle). Bring it up with the bundled
  compose file: `docker compose -f infra/docker-compose.yml --profile registry up -d`,
  then `ingestlib registry init`. Standalone operations (`parse`/`classify`/`split`/
  `extract` on a single document) need no registry
- **Jina AI account** for reranking (free tier: 100 RPM) — the default; or set
  `reranker: aws` (Amazon Rerank, same AWS credentials) or `reranker: none`
  in config.yaml and skip Jina entirely

### 2. Install

```bash
uv add ingestlib               # or: pip install ingestlib
```

The core install covers the full default stack (including sqlite vectors).
Server-backed vector stores are extras — add the one you'll use:

```bash
uv add "ingestlib[qdrant]"     # pinecone · qdrant · pgvector · mongodb
                               # · milvus · opensearch · weaviate · all
```

Or work from source:

```bash
git clone https://github.com/LangModule/ingestlib.git
cd ingestlib
uv sync
```

System dependency — LibreOffice (DOCX/PPTX → PDF conversion):

```bash
brew install --cask libreoffice          # macOS (binary is `soffice`)
sudo apt install libreoffice-core libreoffice-writer libreoffice-impress   # Linux
```

Optional — PostgreSQL client tools (`pg_dump`/`pg_restore`), only for
`ingestlib registry backup`/`restore`:

```bash
brew install libpq                       # macOS
sudo apt install postgresql-client       # Linux
```

### 3. Start the OCR inference server

Parse runs PaddleOCR-VL-1.6 behind an inference server. First launch downloads
~1.8 GB of weights; later launches load from cache in seconds.

```bash
# Apple Silicon (Metal GPU)
uv run python -m mlx_vlm.server --port 8111 --model PaddlePaddle/PaddleOCR-VL-1.6

# NVIDIA (then set paddle_vl.backend: vllm-server in config.yaml)
vllm serve PaddlePaddle/PaddleOCR-VL-1.6 --port 8111
```

The layout model (PP-DocLayoutV3, ~126 MB) auto-downloads on the first parse.

### 4. Configure

`ingestlib init` scaffolds the setup files into the current directory:

```bash
uv run ingestlib init            # default stack: Bedrock + sqlite + Jina → config.yaml + .env
uv run ingestlib init --local    # zero-cloud: Ollama + sqlite + local artifacts — no keys at all
```

For the default stack, add Bedrock-enabled credentials:

```bash
aws configure --profile your-aws-profile
```

Edit `config.yaml`: pick your providers, vector store, reranker, and
artifact store — everything else has working defaults (`vector_store:
sqlite` needs no server and no keys) — and fill `.env` with the keys your
choices need (Jina for the default reranker; `--local` needs none). The `aws` section is required only
while a choice uses AWS (the default bedrock provider, s3 artifacts, the
aws reranker, an Amazon OpenSearch domain) — delete it otherwise and the
config loader will tell you if something still needs it. **The S3 bucket
(default `ingestlib-{account_id}`) and the vector indexes/collections are
created automatically on first use** — no manual setup. Prefer no cloud
storage at all? `artifact_store: local` keeps every source file, page render,
and figure crop in a plain folder beside your config.yaml — browsable in a file
manager, and moving the artifact store between backends is a plain copy.

Config is discovered at call time, never at import: `INGESTLIB_CONFIG=/path/to/config.yaml`
wins, otherwise the working directory and its parents are searched — so
installed usage works the same as running inside this repo.

### 5. Verify

```bash
uv run ingestlib doctor
```

Doctor checks every configured choice with real calls — config discovery,
LibreOffice, the OCR server, an LLM ping, an embedding (with its dimension),
the reranker, the artifact store, the vector store, and the registry (reachable
and migrated). Every failed line
prints the one-sentence fix (wrong AWS profile → your available profiles,
missing key → where to get one, model not pulled → the exact `ollama pull`).

### 6. Run

```python
from ingestlib.services import ingest, retrieve

r = ingest("report.pdf")
print(r.status, r.category, r.chunks, r.durations)

res = retrieve("what does the report conclude?", top_k=5)
for hit in res.hits:
    print(hit.rerank_score, hit.citation, hit.chunk.heading)
```

## Manage a corpus

Real corpora change. Re-ingesting an edited file **replaces** the old version
(its vectors and artifacts are deleted — no stale chunks), a rename is a cheap
**move**, and `sync()` reconciles a whole folder in one call:

```python
from ingestlib.services import ingest, sync, remove, reindex

ingest("report.pdf")                       # edited file → status="replaced"
sync("corpus/", prune=True)                # add new, replace changed, drop deleted
remove("old.pdf")                          # erase one doc from both stores
reindex()                                  # rebuild the vector store from the registry
```

The same verbs are on the CLI — a corpus is managed from the shell, no Python
needed:

```bash
ingestlib ingest report.pdf docs/          # files or folders
ingestlib sync corpus/ --prune --dry-run   # preview, then drop --dry-run
ingestlib list                             # every stored document
ingestlib remove report.pdf                # erase one
ingestlib reindex                          # rebuild the index (provider switch, new store)
ingestlib recollect                        # re-sort the corpus after editing classify rules
ingestlib search "what were the risks?"    # cited retrieval from the shell
```

Documents carry a logical identity (`namespace` + source path), so lifecycle
knows a re-ingest from a brand-new file. Full guide:
[Manage a corpus](https://langmodule.github.io/ingestlib/how-to/manage-corpus/).

## Query your databases

`retrieve()` can also answer from your **SQL databases**, alongside the
document corpus, in the same call. Declare the databases in a `sources.yaml`
sidecar (beside config.yaml, like rules.yaml — see
[`sources.example.yaml`](./sources.example.yaml)) and pass their names:

```python
from ingestlib.services import retrieve

# documents AND databases behind one question
result = retrieve("how many prescriptions are ready?", sources=["prescriptions"])
for r in result.results:
    print(r.source_type, r.content, r.provenance["sql"])   # the exact query that ran
```

The model generates **read-only** SQL from your schema + `tables` hints, bounded
by a permission boundary whose floor is a read-only database role — so a wrong
query is a wrong *read*, never a write. A statement allowlist, an injected
`LIMIT`, and a timeout are defense in depth; **verified queries** let you pin
reviewed SQL for answers that must be exact. On a **wide schema** ingestlib
retrieves only the tables a question needs — plus their foreign-key bridges —
instead of dumping every table into the prompt (`schema_rag: auto`). Generate
hints for a cryptic schema with `ingestlib describe-schema`, and measure
generation accuracy on your own schema with `ingestlib eval-sql`. Each SQL
backend needs its pip extra (`ingestlib[postgres]` · `[mysql]` · `[duckdb]` ·
`[snowflake]`; sqlite needs none). From the shell:
`ingestlib search "…" --sources prescriptions`. Full guide:
[Query databases (SQL)](https://langmodule.github.io/ingestlib/how-to/structured-retrieval/).

## Serve it to agents (MCP)

`ingestlib mcp` exposes the whole loop — search, extract, ingest, sync, remove,
reindex, recollect, verify, and registry management — as [MCP](https://modelcontextprotocol.io) tools, so Claude Desktop,
Cursor, or any agent can drive your **self-hosted** corpus with citations,
nothing leaving your machine.

```bash
pip install "ingestlib[mcp]"
ingestlib mcp                                # stdio (local clients)
ingestlib mcp --transport http --port 8000   # remote; needs MCP_TOKEN (bearer auth)
```

Read tools (`search`/`extract`/`classify`/`list`/`doctor`) are always on; the
write tools hide under `--read-only`. Guide:
[Serve to agents (MCP)](https://langmodule.github.io/ingestlib/how-to/mcp-server/).

## Using the operations directly

Every operation also works standalone:

```python
from ingestlib.operations import parse, classify, split

result = parse("report.pdf")            # ParseResult: pages, regions, figures
print(result.markdown)                  # whole-document markdown
result.save_images("out/")              # extracted figures/charts as PNGs

label = classify("report.pdf")          # no OCR needed — native text + embedded images
chunks = split(result, category=label.category)
for c in chunks.chunks:
    print(c.token_estimate, c.embedding_text.splitlines()[0])
```

And the fourth operation pulls **structured data** out — your schema,
filled and cited:

```python
from pydantic import BaseModel
from ingestlib.operations import extract

class Receipt(BaseModel):
    merchant: str
    total: float
    currency: str

report = extract(parse("expenses.pdf"), schema=Receipt, mode="many")
for item in report.items:
    print(item.value.merchant, item.value.total, item.citation)
# BART 20.0 p.10  ·  Hilton 214.6 p.2  ·  …
print(report.items[0].fields["total"].grounded)   # True — verified in the cited region
```

`mode="one"` fills a single instance from the whole document (a 10-K's
headline financials); `mode="many"` finds every instance across the pages.
A `ParseResult` input gives region-level citations on scans; a raw path
reads the native text layer with page-level citations and no OCR server.
Confidence is honest: a field whose citation doesn't check out is capped,
and a value not found in its cited text is flagged `grounded=False`.

A stored document reads back explicitly — queryable metadata from the registry,
bytes from the blob store:

```python
from ingestlib.services import get_document

doc = get_document(doc_id)              # the stored document, from the registry
print(doc.category, doc.chunk_count)    # queryable metadata
doc.markdown()                          # whole-document markdown (blob store)
[e["schema_name"] for e in doc.extractions]   # any persisted extractions
```

## Classification & split rules

Classify and split are open-ended by default — the LLM decides the document
type and discovers the section vocabulary. Both can instead follow **your**
rules: pass them per call, or preset them once in `rules.yaml` beside your
config.yaml, and every bare call **and the whole ingest pipeline** uses
them automatically:

```python
classify("doc.pdf",
         {"invoice": "Itemized charges, tax info, and payment terms",
          "sec_filing": "10-K/10-Q style regulatory filings"},
         target_pages="1,3,5-7", max_pages=5)   # read only these pages
```

```python
split("report.pdf",
      vocabulary={"financial_statements": "Balance sheets, income statements",
                  "notes": "Footnotes and disclosures"},
      unmatched="other")   # pages fitting nothing: other (default) | require | skip
```

```yaml
# rules.yaml — beside your config.yaml; infra stays in config.yaml,
# what your documents MEAN lives here
classify:
  max_pages: 5
  rules:                       # up to 20 — result is one of these or "uncategorized"
    invoice: "Itemized charges, tax info, and payment terms"
    sec_filing: "10-K/10-Q style regulatory filings"
split:
  unmatched: other             # require | other | skip
  categories:                  # up to 50 — YOUR sections; Pass 1 is skipped
    financial_statements: "Balance sheets, income statements, cash flows"
    notes: "Footnotes and disclosures"
```

Classify returns one of your labels or `"uncategorized"`, with confidence,
reasoning, and ranked alternatives. Split labels every page against your
sections — unmatched pages become an honest `other` section (default), get
forced into the nearest category (`require`), or are dropped entirely
(`skip`). Precedence everywhere: explicit arguments beat the preset, and
`{}` forces the open-ended default even when a preset exists.

## OpenAI backend

The same LLM surface Bedrock provides is also available on OpenAI — GPT-5
chat with vision, thinking mode, schema-enforced structured output, and
text-embedding-3 embeddings. Add `OPENAI_API_KEY` to `.env` and pick models
in config.yaml's `openai:` section (defaults: `gpt-5-mini`,
`text-embedding-3-small`).

To run the whole `ingest`/`retrieve` pipeline on it, switch the providers
in config.yaml — every LLM and embedding call routes accordingly:

```yaml
llm_provider: openai          # chart reading, review, classify, chunking
embedding_provider: openai    # chunk + query embeddings
```

Combined with `artifact_store: local`, `vector_store: sqlite`, and
`reranker: jina` (or `none`), the pipeline needs no AWS at all. Two rules:
switching `embedding_provider` changes the vector space, so re-ingest
afterward (`skip_existing=False`) — vectors from different embedding models
never mix in one index. And text embeddings only: OpenAI has no
image-embedding model.

The backend is also importable directly, ignoring the config switch:

```python
from ingestlib.foundations.llm import Image
from ingestlib.foundations.llm.openai import chat, chat_structured, embed_text

chat("Read this chart", images=[Image(png_bytes, "png")])   # vision works
embed_text("a chunk of text")                               # 1024-dim default
```

## Local backend (Ollama)

The third provider keeps every LLM call on your machine — the fully
air-gapped pipeline. Point config.yaml at a local
[Ollama](https://ollama.com) (or any OpenAI-compatible server: vLLM,
LM Studio):

```yaml
llm_provider: ollama
embedding_provider: ollama

ollama:                                    # these are the defaults
  base_url: http://localhost:11434/v1
  llm_model_id: qwen3.5:9b
  embedding_model_id: qwen3-embedding:0.6b
```

```bash
ollama pull qwen3.5:9b
ollama pull qwen3-embedding:0.6b
```

No API key. Vision, schema-enforced structured output, and 1024-dim
embeddings all verified on the reference stack — and **local embedding
quality is measured, not assumed**: on the retrieval eval, qwen3-embedding
matched or beat the cloud stack (perfect reranked hit@1/3/5 = 1.00). Two
honest notes: use the GGUF builds, not `-mlx` (Ollama's MLX engine
silently drops images and schema enforcement), and a local 9B won't match
the cloud models on dense charts — test with your own documents before
committing.
Combined with `artifact_store: local` and `vector_store: sqlite`,
nothing leaves your machine but the optional Jina rerank call
(`reranker: none` closes even that).

## Architecture

```
src/ingestlib/
├── services/       ingest · retrieve · lifecycle (remove · sync · reindex · recollect) · verify — the product
├── operations/     parse · classify · split · extract — the tools (each standalone)
├── storage/        registry (Postgres metadata hub) · artifacts (S3 | local, bytes only) · base (VectorStore contract) · 8 connectors
│                   (pinecone · qdrant · sqlite · pgvector · mongodb · milvus
│                    · opensearch · weaviate)
├── sources/        structured retrieval — SQL databases & the corpus as queryable Sources
├── foundations/    llm (Bedrock Nova · OpenAI GPT-5 · Ollama Qwen · Jina) · ocr (PaddleOCR-VL)
├── cli/            the `ingestlib` command — init · doctor · ingest · sync · list · show · collections · remove · reindex · recollect · verify · search · describe-schema · eval-sql · registry · mcp
├── mcp/            MCP server (ingestlib[mcp]) — expose the pipeline to agents
├── utils/          logger · files · sync · aws
└── config.py       config.yaml + .env + rules.yaml + sources.yaml → typed configs

src/ingestlib_registry/   the registry's Postgres schema + Alembic migrations (a standalone package ingestlib depends on)
```

Strict downward dependencies. The `VectorStore` contract means backends drop
in as connectors — all eight ship **hybrid search**: **Pinecone** (dense +
hosted sparse model, merged client-side), **Qdrant** (dense + BM25 with
server-side IDF and RRF fusion; local docker or cloud), **SQLite**
(sqlite-vec KNN + built-in FTS5 BM25 with porter stemming, RRF fusion — one
local file, no server, no keys), **Postgres/pgvector** (HNSW cosine +
built-in full-text over a generated weighted tsvector, RRF fusion — the
extension and table bootstrap automatically), **MongoDB** (Atlas Vector
Search + Atlas Search true BM25, RRF fusion — Atlas any tier or self-managed
8.2+; both search indexes bootstrap automatically), **Milvus** (dense
ANN + server-computed BM25 sparse, fused server-side with RRF — local docker
or Zilliz Cloud), **OpenSearch** (faiss HNSW k-NN + Lucene BM25, RRF fused
client-side — an Amazon OpenSearch domain SigV4-signed with your aws
profile, or local docker), and **Weaviate** (HNSW dense + native BM25 fused
server-side in one hybrid call — local docker or Weaviate Cloud). Pick one
with `vector_store: pinecone | qdrant | sqlite | pgvector | mongodb |
milvus | opensearch | weaviate` in config.yaml. Connection secrets sit in
`.env` together (sqlite needs none) — only the selected connector ever
builds a client.

## Logging

```bash
INGESTLIB_LOG_LEVEL=INFO           # DEBUG | INFO | WARNING | ERROR (default INFO)
INGESTLIB_LOG_THIRD_PARTY=1        # also show paddlex/httpx/botocore chatter
INGESTLIB_LOG_COLOR=0              # disable colored output
```

## Testing

Tests hit **real APIs, never mocks**. Pure logic runs always; server-hitting
suites are opt-in via env gates. The sqlite connector's full suite runs
ungated in `make test` — there is no server, so in-process IS the real thing.

```bash
make test                  # fast suite (~630 tests, ~2min; e2e groups skip)
make test-openai           # OpenAI backend       (skips without OPENAI_API_KEY)
make test-ollama           # Ollama backend       (needs a local Ollama + models)
make test-parse            # parse e2e            (needs VL server + LLM provider)
make test-classify         # classify e2e         (needs the LLM provider)
make test-split            # split e2e            (needs the LLM provider)
make test-extract          # extract e2e          (needs the LLM provider; scans need the VL server)
make test-s3               # artifact store e2e   (needs AWS)
make test-pinecone         # vector connector e2e (needs Pinecone + embeddings)
make test-qdrant           # vector connector e2e (needs a Qdrant server + embeddings)
make test-sqlite           # vector connector suite (no gate — nothing to need)
make test-pgvector         # vector connector e2e (needs Postgres at PGVECTOR_URL)
make test-mongodb          # vector connector e2e (needs MongoDB at MONGODB_URL)
make test-milvus           # vector connector e2e (needs Milvus at MILVUS_URL)
make test-opensearch       # vector connector e2e (needs OpenSearch at OPENSEARCH_URL)
make test-weaviate         # vector connector e2e (needs Weaviate at WEAVIATE_URL)
make test-services         # full product e2e     (needs the entire stack)
make test-sources          # structured retrieval — SQL sources (deterministic; e2e gated)
make test-cli              # CLI: init/doctor + corpus commands (no gate)
make test-lifecycle        # remove/sync/reindex/recollect + replace-aware ingest (gates on the registry)
make test-mcp              # MCP server: tools, read_only, http auth (no gate)
make test-all              # everything
make eval                  # retrieval quality eval (see below)
make docs                  # live-preview the documentation site
```

Fixtures live in `tests/data/` — 15 real PDFs (research papers, earnings
decks, insurance forms, 10-Ks, a 16-page receipt scan, a password-protected
one), document images incl. WebP, and real DOCX/PPTX files. The
server-backed connector suites need their servers:
`docker compose -f infra/docker-compose.yml --profile all up -d` starts
every local store (or `--profile qdrant` etc. for just one).

### Retrieval quality

Beyond pass/fail tests, `evals/` measures retrieval quality: 22 ground-truth
questions over the fixture corpus, run through the real `retrieve()` flow
under dense/hybrid × rerank on/off, scored by hit@k and MRR. Measured so far
(consistent across all eight connectors): **with reranking, every answer
lands in the top 3 hits (hit@3 = 1.00)**; hit@1 ranges 0.86–1.00 across runs.
The fully-local stack holds the same bar — Ollama embeddings scored a
perfect 1.00 across hit@1/3/5 and MRR on dense+rerank.
Each run saves a timestamped snapshot to `evals/results/`, so quality changes
are visible over time.

## Disk footprint

| Component | Size | Location |
|---|---|---|
| Python deps | ~1.4 GB core (+ your vector-store extra) | `.venv/` |
| PaddleOCR-VL-1.6 weights | ~1.8 GB | `~/.cache/huggingface/hub/` |
| PP-DocLayoutV3 | ~126 MB | `~/.paddlex/official_models/` |
| LibreOffice | ~600 MB | system |

## Scope

English documents; PDF / DOCX / PPTX / PNG / JPEG / WebP input. Images,
charts, and tables **inside** documents are fully extracted and interpreted,
and a single image file parses as a one-page document; handwriting is out
of scope by design.

## The studio

[ingestlib-studio](https://github.com/LangModule/ingestlib-studio) is the
visual companion: a local web UI with a setup wizard, try-before-you-commit
pipeline runs, page-by-page review with hover-to-highlight bounding boxes,
committed ingestion with live progress, a content-rules editor, and a
retrieval playground where every answer points to its source on the page.

## Roadmap

- XLSX input (tables-first, not a PDF conversion)

Recently shipped: the **internal registry** — a Postgres metadata hub that makes
the whole corpus queryable, with `reindex`/`recollect`/`verify` for rebuilding
and auditing it and event-driven backups (v1.5); **schema-RAG for wide
databases** — retrieve the relevant tables (with foreign-key closure) instead of
dumping the whole schema, plus `describe-schema` auto-documentation and the
`eval-sql` accuracy harness (v1.4); **structured retrieval** — query your SQL
databases alongside documents through one `retrieve()` call, behind a read-only
permission boundary (v1.3); an **MCP server** to serve the corpus to agents
(v1.2); document lifecycle — replace-aware ingestion, folder `sync()`,
`reindex()`, and the corpus CLI (v1.1).

## License

See [LICENSE](./LICENSE).
