Metadata-Version: 2.1
Name: pagedoc-rag
Version: 0.1.0
Summary: CPU-First Multimodal Document Chunking, Ingestion & Retrieval Engine using Native PDF Extraction, OCR, Multi-Vector Embeddings, and MaxSim Late Interaction
Home-page: https://github.com/anoopchandra/pagedoc-rag
Author: PageDoc Contributors
Author-email: maintainers@pagedoc.dev
License: Apache-2.0
Project-URL: Documentation, https://github.com/anoopchandra/pagedoc-rag#readme
Project-URL: Source, https://github.com/anoopchandra/pagedoc-rag
Project-URL: Tracker, https://github.com/anoopchandra/pagedoc-rag/issues
Keywords: rag,document-chunking,pdf-ingestion,multimodal-rag,table-extraction,chart-parsing,maxsim,cpu-first,information-retrieval
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Provides-Extra: charts
Provides-Extra: tables
Provides-Extra: vector
Provides-Extra: api
Provides-Extra: all
License-File: LICENSE

# PageDoc-RAG: CPU-First Multimodal Document Ingestion & Retrieval Engine

[![PyPI Version](https://img.shields.io/badge/pypi-v0.1.0-blue.svg)](https://pypi.org/project/pagedoc-rag/)
[![License: Apache-2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)

**PageDoc-RAG** is a production-grade, CPU-first multimodal document engine that bridges the gap between raw documents (PDFs, scans, images) and modern RAG pipelines. It performs **native layout extraction**, **semantic multimodal chunking** (preserving tables, charts, and text reading order), **multi-vector indexing**, and **fine-grained MaxSim late-interaction retrieval** with authoritative page evidence.

---

## 🌟 Why PageDoc-RAG?

1. **End-to-End Document Lifecycle**: Not just a retriever. It handles the entire pipeline: high-resolution PDF rendering, native layout & table extraction, OCR fallback, semantic multimodal chunking, multi-vector generation, and database indexing.
2. **CPU-First Architecture**: High-speed, cost-effective inference on standard CPU hardware without requiring heavy multi-gigabyte GPU models for standard enterprise document corpora.
3. **Atomic Multimodal Chunking**:
   - **Tables**: Extracted as structured atomic units with header context preserved across rows.
   - **Charts**: Decomposed into 3 vector streams: Chart Title, Data Series pairs, and Synthesized Semantic Relational Insights (e.g., *"Q4 has highest revenue"*).
   - **Text**: Extracted with bounding boxes $[x_0, y_0, x_1, y_1]$ and reading-order continuity.
   - **Seals & Stamps**: Detected and isolated for compliance and verification workflows.
4. **ColPali/ColBERT-Style Late Interaction (MaxSim)**: Generates fine-grained multi-vector representations per page and scores queries using token-level late interaction:
   $$\\text{Score}(Q, P) = \\sum_{i=1}^{M} \\max_{j=1}^{N} \\text{sim}(q_i, p_j)$$
5. **Source-of-Truth Principle**: Returns original PDF page references, visual crops, and bounding boxes alongside similarity scores.
6. **Two-Stage Scalable Retrieval**: ANN candidate filtering before MaxSim reranking to scale effortlessly to tens of thousands of documents.

---

## 📦 Installation

```bash
# Core installation (Zero heavy dependencies, pure CPU)
pip install pagedoc-rag

# Optional extras:
pip install "pagedoc-rag[vector]"   # Adds FAISS-CPU vector index backend
pip install "pagedoc-rag[all]"      # Full suite (PyMuPDF, FAISS-CPU, Pillow, FastAPI)
```

---

## 🚀 Quickstart (Python API)

```python
from pagedoc_rag import PageDocRAG

# 1. Initialize engine
rag = PageDocRAG(
    embedding_model="small-cpu",
    index_backend="memory",   # or "faiss"
    enable_charts=True,
    enable_tables=True,
    enable_seals=True
)

# 2. Ingest document (PDF, PNG, JPG)
doc = rag.ingest("annual_report.pdf")
print(f"Ingested {doc.num_pages} pages ({doc.document_id})")

# 3. Perform natural language search
results = rag.search("What was the Net Operating Profit in FY2025?", top_k=5)

for res in results:
    print(f"Rank Page #{res.page_number} | Score: {res.score:.4f}")
    print(f"Snippet: {res.text[:140]}...")
    print(f"Source PDF: {res.source_pdf}")
    print(f"Evidence items found: {len(res.evidence)}")
```

---

## 💻 CLI Usage

```bash
# Index a directory of documents
pagedoc index ./documents --backend memory

# Search across indexed pages
pagedoc search "What was revenue in 2025?" --top-k 5

# Inspect structured elements & bounding boxes on a page
pagedoc inspect ./documents/report.pdf --page 2
```

---

## 🏛️ Architecture & Processing Pipeline

```
                       PDF / Scanned Document
                                 │
                                 ▼
                     PyMuPDF High-Res Rendering
                                 │
          ┌──────────────────────┴──────────────────────┐
          ▼                                             ▼
  Native Digital Stream                         Scanned Document Stream
  • Text Blocks & Coordinates                   • PaddleOCR DBNet / SVTR
  • Native Table Matrices (find_tables)         • PP-Structure Layout Router
  • Vector Drawings & Images                    • Chart2Table & Seal Detector
          │                                             │
          └──────────────────────┬──────────────────────┘
                                 ▼
                    Semantic Multimodal Chunking
               ├── Text Paragraphs with Bounding Boxes
               ├── Structured Table Matrices (CSV / Headers)
               └── Decomposed Chart Relations & Trends
                                 │
                                 ▼
                    Multi-Vector Embedding Model
                     [p1, p2, p3, ... pN per page]
                                 │
                                 ▼
                Two-Stage Candidate Search & Filtering
                                 │
                                 ▼
                 Vectorized MaxSim Late Interaction
                                 │
                                 ▼
             Authoritative Page Retrieval & RAG Context
```

---

## 📊 Evaluation & Benchmarking

PageDoc-RAG includes a built-in evaluation suite for standard information retrieval metrics:
- **Recall@1, Recall@5, Recall@10**
- **Mean Reciprocal Rank (MRR)**
- **nDCG@5, nDCG@10**
- **Latency Percentiles (p50, p95) & QPS**

```python
from pagedoc_rag.eval.benchmark import Evaluator, BenchmarkQuery

queries = [
    BenchmarkQuery(query="Revenue breakdown", target_page_ids=["doc_p2"], query_type="table"),
    BenchmarkQuery(query="Executive summary", target_page_ids=["doc_p1"], query_type="text"),
]

report = Evaluator.evaluate(rag, queries)
print(f"Recall@1: {report.recall_at_1:.2%}, MRR: {report.mrr:.4f}")
```

---

## 📄 License

Licensed under the [Apache License, Version 2.0](LICENSE).


