Metadata-Version: 2.4
Name: omnidoc-rag
Version: 0.1.4
Summary: RAG pipeline for omnidoc-sdk — intent-aware chunking, evaluation, streaming, graph linking, and vector DB integrations
Author-email: Ganesh Kinkar Giri <k.ganeshgiri@example.com>
License: Apache-2.0
Project-URL: Homepage, https://github.com/your-org/omnidoc-rag
Project-URL: Documentation, https://github.com/your-org/omnidoc-rag#readme
Project-URL: Issues, https://github.com/your-org/omnidoc-rag/issues
Keywords: rag,semantic-chunking,document-intelligence,vector-database,agentic-ai,llm
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: omnidoc-sdk>=0.4.2
Provides-Extra: chroma
Requires-Dist: chromadb>=0.4.0; extra == "chroma"
Provides-Extra: pinecone
Requires-Dist: pinecone-client>=3.0.0; extra == "pinecone"
Provides-Extra: weaviate
Requires-Dist: weaviate-client>=4.0.0; extra == "weaviate"
Provides-Extra: pgvector
Requires-Dist: psycopg2-binary>=2.9.0; extra == "pgvector"
Requires-Dist: pgvector>=0.2.0; extra == "pgvector"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Provides-Extra: all
Requires-Dist: chromadb>=0.4.0; extra == "all"
Requires-Dist: pinecone-client>=3.0.0; extra == "all"
Requires-Dist: weaviate-client>=4.0.0; extra == "all"
Requires-Dist: psycopg2-binary>=2.9.0; extra == "all"
Requires-Dist: pgvector>=0.2.0; extra == "all"
Dynamic: license-file

<div align="center">

<h1>omnidoc-rag</h1>

<p><strong>Modular, file-type-aware RAG pipeline for the OmniDoc document intelligence ecosystem</strong></p>

<p>
  <img src="https://img.shields.io/badge/python-3.9%2B-blue?style=flat-square" alt="Python 3.9+"/>
  <img src="https://img.shields.io/badge/version-0.1.3-green?style=flat-square" alt="v0.1.3"/>
  <img src="https://img.shields.io/badge/license-Apache--2.0-orange?style=flat-square" alt="Apache 2.0"/>
  <img src="https://img.shields.io/badge/file%20types-41-purple?style=flat-square" alt="41 file types"/>
  <img src="https://img.shields.io/badge/excel%20chunk%20types-57-teal?style=flat-square" alt="57 Excel chunk types"/>
  <img src="https://img.shields.io/badge/vector%20DBs-4-yellow?style=flat-square" alt="4 vector DB adapters"/>
</p>

</div>

---

## What is omnidoc-rag?

`omnidoc-rag` is the companion RAG SDK for [`omnidoc-sdk`](https://github.com/your-org/omnidoc-python-sdk). It converts `Document` objects from the extraction layer into vector-DB-ready `SemanticChunk` objects with a **dedicated chunker per file type** — each format gets its own splitting strategy, boundary detection, and intent mapping rather than a one-size-fits-all approach.

**What makes it different:**

- **41 file extensions** — each routed to a purpose-built chunker (PDF, DOCX, PPTX, XLSX, MD, Python, JS, Java, CSV, JSON, XML, YAML, HTML, EPUB, EML, IPYNB, and more)
- **57 Excel chunk types** — structural, semantic, analytical, validation, visual, VBA, cross-reference, connectivity, and operational chunks extracted from every layer of a workbook
- **Intent classification** — 6 canonical types (metric, table, process, value\_proposition, heading, narrative) with per-intent token budgets
- **Adaptive chunking** — boundary strategy differs by type (slide per chunk for PPTX, function per chunk for Python, cell per chunk for notebooks)
- **Deterministic confidence scoring** — quality signal for retrieval ranking, no LLM required
- **Streaming** — true lazy generator for large documents
- **Retrieval evaluation** — query-term coverage, source diversity, verdict scoring
- **Graph linking** — NEXT / SAME\_INTENT / METRIC\_OF edge graph
- **Cross-document stitching** — merge equivalent sections across multiple documents
- **4 vector DB adapters** — ChromaDB, Pinecone, Weaviate, PostgreSQL/pgvector

---

## Table of Contents

1. [Installation](#installation)
2. [Quick Start](#quick-start)
3. [Modular Chunking by File Type](#modular-chunking-by-file-type)
   - [Supported File Types](#supported-file-types)
   - [Auto-detection](#auto-detection)
   - [Explicit chunker selection](#explicit-chunker-selection)
   - [Chunking strategies by format](#chunking-strategies-by-format)
4. [Excel — 57 Chunk Types](#excel--57-chunk-types)
5. [Intent Types](#intent-types)
6. [Streaming](#streaming)
7. [Confidence Scoring](#confidence-scoring)
8. [Retrieval Evaluation](#retrieval-evaluation)
9. [Graph Linking](#graph-linking)
10. [Cross-Document Stitching](#cross-document-stitching)
11. [Vector DB Adapters](#vector-db-adapters)
    - [ChromaDB](#chromadb)
    - [Pinecone](#pinecone)
    - [Weaviate](#weaviate)
    - [PostgreSQL / pgvector](#postgresql--pgvector)
12. [Schema Reference](#schema-reference)
13. [Package Structure](#package-structure)
14. [Optional Extras](#optional-extras)
15. [Contributing & Development](#contributing--development)
16. [Changelog](#changelog)

---

## Installation

### Core (no vector DB)

```bash
pip install omnidoc-rag
```

### With a specific vector DB

```bash
pip install "omnidoc-rag[chroma]"
pip install "omnidoc-rag[pinecone]"
pip install "omnidoc-rag[weaviate]"
pip install "omnidoc-rag[pgvector]"
```

### Everything

```bash
pip install "omnidoc-rag[all]"
```

---

## Quick Start

```python
from omnidoc.loader.load import load_document      # omnidoc-sdk
from omnidocrag import chunk_document, evaluate_rag_result

# 1. Extract — any supported file type
doc = load_document("investor_deck.pdf")

# 2. Chunk — file type auto-detected, right chunker selected
chunks = chunk_document(doc)

for c in chunks:
    print(f"[{c.intent:<18}] conf={c.confidence:.2f}  p{c.page}  {c.text[:80]}")

# 3. Evaluate a retrieval result
result = evaluate_rag_result(
    query="What was the revenue growth rate?",
    answer="Revenue grew 24% year-over-year to $4.2B.",
    chunks=chunks,
)
print(result["overall"], result["verdict"])   # 0.87  excellent
```

---

## Modular Chunking by File Type

### Supported File Types

Every extension is routed to a dedicated chunker class. 41 extensions are registered:

| Category | Extensions | Chunker | Strategy |
|---|---|---|---|
| **Office — Documents** | `.pdf` | `PDFChunker` | Heading-aware sections + per-row table chunks |
| | `.docx` `.doc` | `DOCXChunker` | Paragraph-boundary + batched table chunks |
| | `.pptx` `.ppt` | `PPTXChunker` | One chunk per slide + speaker notes |
| | `.xlsx` `.xls` `.xlsm` | `XLSXChunker` | **57 chunk types** (see below) |
| **Text** | `.txt` | `TXTChunker` | Paragraph-boundary, no heading detection |
| **Markdown** | `.md` | `MDChunker` | Heading hierarchy + fenced code blocks + MD tables |
| **Code** | `.py` | `PythonChunker` | `def` / `class` / `async def` boundaries |
| | `.js` `.ts` | `JSChunker` | `function` / arrow function / `class` / `interface` |
| | `.java` | `JavaChunker` | `class` / `interface` / `enum` + package block |
| **Structured** | `.csv` | `CSVChunker` | Header-aware row batches (30 rows/chunk) |
| | `.json` | `JSONChunker` | Top-level key / array-item batches |
| | `.xml` | `XMLChunker` | Top-level element chunks |
| | `.yaml` `.yml` | `YAMLChunker` | Top-level key blocks |
| **Web** | `.html` `.htm` | `HTMLChunker` | `h1–h6` headings + `<code>` blocks + blockquotes |
| **Images** | `.png` `.jpg` `.jpeg` `.tiff` `.webp` `.gif` `.bmp` | `ImageChunker` | OCR text paragraphs (or metadata stub) |
| **Archives** | `.zip` `.tar` `.gz` `.7z` `.rar` `.bz2` `.xz` | `ArchiveChunker` | Manifest chunk + contained-file delegation |
| **E-Books** | `.epub` | `EPUBChunker` | Chapter/spine-item-aware |
| | `.odt` | `ODTChunker` | Heading-aware paragraphs + table batches |
| | `.rtf` | `RTFChunker` | Paragraph-based, RTF artifact stripping |
| **Email** | `.eml` | `EMLChunker` | Header + body + quoted-reply + attachments |
| | `.msg` | `MSGChunker` | Outlook header + body + attachments + meeting detection |
| **Notebooks** | `.ipynb` | `IPYNBChunker` | Cell-aware — code/markdown/output per cell |

### Auto-detection

`chunk_document` reads `doc.metadata["file"]` and selects the right chunker automatically:

```python
from omnidocrag import chunk_document

chunks = chunk_document(doc)                     # auto-detects from file path
chunks = chunk_document(doc, ext=".pdf")         # force a specific chunker
```

### Explicit chunker selection

```python
from omnidocrag.chunkers import get_chunker, chunk_by_type, supported_extensions

# Get an instantiated chunker by file path / extension
chunker = get_chunker("report.pdf")              # → PDFChunker()
chunker = get_chunker("notebook.ipynb")          # → IPYNBChunker()
chunker = get_chunker("data.xlsx")              # → XLSXChunker()

# Chunk directly
chunks = chunker.chunk(doc)
chunks = chunker.chunk(doc, overlap_chars=150, min_chars=30)

# Convenience dispatcher
chunks = chunk_by_type(doc)                      # same as chunk_document
chunks = chunk_by_type(doc, ext=".json")         # override extension

# Inspect all registered extensions
print(supported_extensions())
# ['.7z', '.bmp', '.bz2', '.csv', '.doc', '.docx', '.eml', '.epub', ...]
```

### Chunking strategies by format

#### Documents — PDF / DOCX

```python
# Headings flush the buffer and become standalone heading chunks
# Each table row → standalone metric chunk with header prepended
# Character overlap (default 100) carries context across boundaries
chunks = chunk_document(doc)          # PDFChunker or DOCXChunker
```

#### Presentations — PPTX

```python
# Each slide = one chunk group (title as heading)
# Bullet content chunked within the slide's token budget
# Speaker notes → separate narrative chunk tagged content_type="speaker_notes"
for c in chunks:
    print(c.metadata.get("slide_number"), c.intent, c.text[:60])
```

#### Source Code — Python / JS / Java

```python
# Python: def/class/async def blocks including decorators and docstrings
# JS/TS: function declarations, arrow functions, class/interface blocks
# Java: class/interface/enum top-level types + package + import preamble
# All code chunks carry: language, block_type metadata
for c in chunks:
    print(c.metadata["language"], c.metadata["block_type"], c.heading)
```

#### Notebooks — IPYNB

```python
# Each cell → one chunk
# code cells → intent="process"
# markdown cells → intent classified from content
# outputs (stdout, display_data, errors) → separate chunks
for c in chunks:
    ct = c.metadata.get("content_type")
    if ct == "notebook_cell":
        print(f"Cell {c.metadata['cell_index']} [{c.metadata['cell_type']}]: {c.text[:60]}")
    elif ct == "cell_output":
        print(f"  Output: {c.text[:40]}")
```

#### Structured Data — CSV / JSON / XML / YAML

```python
# CSV: header row prepended to every batch, metric intent if financial columns
# JSON: top-level keys become headings; arrays batched in groups of 20
# XML: root element summary + one chunk per top-level child element
# YAML: one chunk per top-level key block
```

#### Email — EML / MSG

```python
# Chunk 1: header fields (From, To, Subject, Date, …)
# Chunk 2+: body paragraphs
# Quoted replies ("> ") → separate narrative sub-chunks
# Attachment list → process chunk
for c in chunks:
    print(c.metadata.get("email_part"), c.text[:60])
    # "header" | "body" | "quoted_reply" | "attachment_list" | "signature"
```

---

## Excel — 57 Chunk Types

`XLSXChunker` uses `openpyxl` to extract every layer of a workbook into typed `SemanticChunk` objects. Each chunk carries a `chunk_type` field in `metadata`.

### Structural (chunks 1–9)

| # | Chunk Type | Key metadata fields |
|---|---|---|
| 1 | `WorkbookChunk` | `name`, `sheet_names`, `author`, `created_at`, `has_macros`, `file_size_kb` |
| 2 | `SheetChunk` | `sheet_name`, `sheet_index`, `sheet_type`, `is_visible`, `used_range`, `tab_color`, `is_protected` |
| 3 | `TableChunk` | `table_name`, `address`, `style`, `has_total_row`, `col_count`, `row_count` |
| 4 | `SchemaChunk` | `columns[name, data_type, null_rate, unique_rate, sample_values]`, `inferred_pk` |
| 5 | `RowChunk` | `row_index`, `values`, `parent_table`, `batch_start`, `batch_end`, `is_total_row` |
| 6 | `GroupChunk` | `direction`, `level`, `start_index`, `end_index`, `is_collapsed` |
| 7 | `ParentContextChunk` | `workbook`, `sheet`, `table`, `row_range` (breadcrumb) |
| 8 | `MergedCellChunk` | `merge_range`, `merged_value`, `row_span`, `col_span` |
| 9 | `FrozenPaneChunk` | `freeze_row`, `freeze_col`, `sheet_name` |

### Semantic (chunks 10–15)

| # | Chunk Type | Key metadata fields |
|---|---|---|
| 10 | `SummaryChunk` | `row_count`, `col_count`, `numeric_cols`, `date_cols`, `null_rate`, `dupe_rate` |
| 11 | `SemanticNarrativeChunk` | `narrative`, `subject`, `time_scope`, `grain`, `generated_by` |
| 12 | `ColumnSemanticChunk` | `col_name`, `role` (id/measure/dimension/date/flag), `unit`, `is_pk`, `is_fk` |
| 13 | `HeaderAliasChunk` | `original_header`, `aliases`, `abbreviations`, `normalized_name` |
| 14 | `EntityChunk` | `entity_type` (date/org/currency/product), `entities`, `frequencies` |
| 15 | `CellAnnotationChunk` | `cell_address`, `annotation_type`, `text`, `author` |

### Analytical (chunks 16–23)

| # | Chunk Type | Key metadata fields |
|---|---|---|
| 16 | `FormulaDefinitionChunk` | `formula_string`, `formula_type` (scalar/array/dynamic/lambda), `precedents` |
| 17 | `FormulaResultChunk` | `computed_value`, `value_type`, `has_error`, `error_type` |
| 18 | `KPIChunk` | `kpi_name`, `value`, `target`, `variance`, `variance_pct`, `source_cell` |
| 19 | `AggregationChunk` | `col_name`, `aggregations` {SUM/AVG/COUNT/MIN/MAX/MEDIAN}, `source_range` |
| 20 | `TemporalChunk` | `date_col`, `frequency`, `start_date`, `end_date`, `gap_count`, `is_sorted` |
| 21 | `TrendChunk` | `col_name`, `direction`, `delta_abs`, `delta_pct`, `regression_slope` |
| 22 | `OutlierChunk` | `col_name`, `outlier_type` (zscore/iqr), `cell_addr`, `value`, `z_score`, `severity` |
| 23 | `LookupMapChunk` | `lookup_type` (VLOOKUP/INDEX-MATCH/XLOOKUP), `formula`, `lookup_range` |

### Validation / QA (chunks 24–28)

| # | Chunk Type | Key metadata fields |
|---|---|---|
| 24 | `ValidationChunk` | `range_addr`, `validation_type`, `allowed_values`, `formula`, `error_msg` |
| 25 | `ErrorChunk` | `cell_addr`, `error_type` (#REF/#DIV0/…), `formula`, `likely_cause` |
| 26 | `ConditionalFormatChunk` | `range_addr`, `rule_type`, `condition_formula`, `business_meaning` |
| 27 | `ProtectionChunk` | `scope`, `is_password_protected`, `locked_ranges` |
| 28 | `DataQualityChunk` | `col_name`, `blank_count`, `dupe_count`, `type_mismatch_count`, `flagged_cells` |

### Visual / Embedded (chunks 29–41)

| # | Chunk Type | Key metadata fields |
|---|---|---|
| 29 | `ChartChunk` | `chart_type`, `title`, `x_axis`, `y_axis`, `series`, `sheet_anchor` |
| 30 | `ChartSeriesChunk` | `series_name`, `source_range`, `color`, `trendline_type` |
| 31 | `ChartAnnotationChunk` | `annotation_type`, `text`, `position` |
| 32 | `PivotTableChunk` | `pivot_name`, `row_fields`, `col_fields`, `value_fields`, `filter_fields` |
| 33 | `PivotFieldChunk` | `field_name`, `field_type`, `agg_function`, `sort_order` |
| 34 | `PivotCacheChunk` | `cache_range`, `last_refreshed`, `record_count` |
| 35 | `SlicerChunk` | `slicer_name`, `field_name`, `active_filters` |
| 36 | `TimelineChunk` | `timeline_name`, `date_field`, `granularity` |
| 37 | `SparklineChunk` | `sparkline_type`, `host_cell`, `source_range` |
| 38 | `ShapeChunk` | `shape_type`, `text_content`, `cell_anchor` |
| 39 | `ImageChunk` | `image_type`, `anchor_cell`, `width_px`, `height_px` |
| 40 | `FormControlChunk` | `control_type`, `linked_cell`, `value`, `macro_assigned` |
| 41 | `ActiveXControlChunk` | `control_type`, `name`, `linked_cell`, `event_macro` |

### VBA / Macro (chunks 42–46)

| # | Chunk Type | Key metadata fields |
|---|---|---|
| 42 | `MacroChunk` | `macro_name`, `module_name`, `trigger_type`, `line_count`, `scope` |
| 43 | `VBAModuleChunk` | `module_name`, `module_type`, `procedure_names`, `line_count` |
| 44 | `VBAEventChunk` | `event_name`, `module`, `trigger_condition`, `affected_range` |
| 45 | `CustomFunctionChunk` | `function_name`, `parameters`, `return_type`, `is_udf` |
| 46 | `RibbonCustomizationChunk` | `tab_name`, `button_label`, `macro_assigned` |

### Cross-Reference / Linkage (chunks 47–50)

| # | Chunk Type | Key metadata fields |
|---|---|---|
| 47 | `RelationshipChunk` | `source_cell`, `target_cell`, `target_sheet`, `rel_type`, `formula` |
| 48 | `NamedRangeChunk` | `range_name`, `refers_to`, `scope`, `usage_count` |
| 49 | `ExternalLinkChunk` | `source_cell`, `target_file`, `is_broken`, `update_mode` |
| 50 | `DependencyGraphChunk` | `nodes`, `edges`, `max_depth`, `has_circular_ref` |

### Connectivity / Power Features (chunks 51–54)

| # | Chunk Type | Key metadata fields |
|---|---|---|
| 51 | `PowerQueryChunk` | `query_name`, `source_type`, `source_path`, `transformation_steps` |
| 52 | `DataModelChunk` | `tables`, `relationships`, `dax_measures` |
| 53 | `PowerPivotMeasureChunk` | `measure_name`, `dax_expression`, `source_table` |
| 54 | `ConnectionChunk` | `connection_name`, `connection_type`, `connection_string_sanitized` |

### Operational / Metadata (chunks 55–57)

| # | Chunk Type | Key metadata fields |
|---|---|---|
| 55 | `PrintSettingsChunk` | `print_area`, `orientation`, `fit_to_pages`, `header_text`, `footer_text` |
| 56 | `ChangeLogChunk` | `cell_addr`, `old_value`, `new_value`, `author`, `changed_at` |
| 57 | `ChunkIndexChunk` | `all_chunk_ids`, `type_counts`, `source_address_map`, `created_at` |

#### Usage

```python
from omnidocrag.chunkers import get_chunker

chunker = get_chunker("financials.xlsx")
chunks = chunker.chunk(doc)

# Filter by chunk type
kpi_chunks    = [c for c in chunks if c.metadata.get("chunk_type") == "KPIChunk"]
formula_chunks = [c for c in chunks if c.metadata.get("chunk_type") == "FormulaDefinitionChunk"]
pivot_chunks  = [c for c in chunks if c.metadata.get("chunk_type") == "PivotTableChunk"]

# Access structured metadata
for c in kpi_chunks:
    print(c.metadata["kpi_name"], c.metadata["value"], c.metadata.get("variance_pct"))
```

> **Note:** `XLSXChunker` requires `openpyxl` and direct filesystem access to the `.xlsx` file via `doc.metadata["file"]`. If the file path is unavailable, it falls back to basic row-batch extraction using `doc.tables`.

---

## Intent Types

Every chunk is labelled with one of six canonical intents. The intent drives chunk sizing and confidence scoring.

| Intent | Token budget | Typical content |
|---|---|---|
| `heading` | 60 | Section/slide titles, worksheet names |
| `metric` | 150 | KPIs, financial figures, percentages, table rows |
| `table` | 200 | Structured data rows, CSV batches |
| `value_proposition` | 250 | Benefits, ROI claims, competitive statements |
| `narrative` | 350 | Prose, analysis, background paragraphs, email body |
| `process` | 400 | Code, workflows, steps, formulas, configuration |

Classification is deterministic — no LLM call required:

```python
from omnidocrag.core.intent import classify_intent
# or via top-level API:
from omnidocrag import classify_intent

classify_intent("Revenue grew 24% YoY to $4.2B")        # "metric"
classify_intent("Step 1: Configure the API key")         # "process"
classify_intent("This solution reduces costs by 30%")    # "value_proposition"
classify_intent("EXECUTIVE SUMMARY")                     # "heading"
classify_intent("The company was founded in 2012.")      # "narrative"
classify_intent("Col A | Col B\nVal 1 | Val 2")         # "table"
```

---

## Streaming

`stream_chunks` is a true Python generator — emits one chunk at a time without building a full list in memory:

```python
from omnidocrag.pipeline.stream import stream_chunks
# or:
from omnidocrag import stream_chunks

for chunk in stream_chunks(doc, overlap_chars=100):
    # process or upsert each chunk immediately
    vector_db.upsert([chunk])
```

Ideal for large documents or live vector DB indexing pipelines.

---

## Confidence Scoring

```python
from omnidocrag.core.confidence import score_chunk
# or:
from omnidocrag import score_chunk

score_chunk("Revenue grew 24% YoY to $4.2B in fiscal 2024.")  # ≥ 0.8
score_chunk("ROI: 38%", intent="metric")                       # ≥ 0.7 (not penalised for length)
score_chunk("See below")                                        # < 0.7
score_chunk("")                                                  # 0.1 (floor)
```

**Scoring factors:** length, multi-line structure, dense-fact density (financials/percentages/currency). Short-text penalty is **not applied** to `metric`, `table`, or `value_proposition` intents.

---

## Retrieval Evaluation

Score a RAG result without an LLM — fully deterministic and local:

```python
from omnidocrag.pipeline.evaluation import evaluate_rag_result
# or:
from omnidocrag import evaluate_rag_result

result = evaluate_rag_result(
    query="What was the EBITDA margin in fiscal 2024?",
    answer="EBITDA margin reached 28% driven by cost efficiencies.",
    chunks=chunks,
)
```

### Return value

```python
{
    "overall":          0.84,        # composite score 0.0–1.0
    "coverage":         0.75,        # fraction of query terms found in chunks
    "confidence":       0.91,        # average chunk confidence
    "source_diversity": 3,           # unique pages used
    "chunks_used":      4,
    "verdict":          "good",      # "excellent" | "good" | "weak" | "unsafe"
    "missing_terms":    ["ebitda"],
}
```

| Verdict | Score |
|---|---|
| `excellent` | ≥ 0.85 |
| `good` | ≥ 0.70 |
| `weak` | ≥ 0.50 |
| `unsafe` | < 0.50 or no chunks |

---

## Graph Linking

```python
from omnidocrag.pipeline.graph import link_chunks
# or:
from omnidocrag import link_chunks

graph = link_chunks(chunks, source="investor_deck.pdf")
graph["nodes"]   # [{id, intent, confidence, page, heading, text_preview}, ...]
graph["edges"]   # [{from, to, relation, weight}, ...]
```

| Relation | Description |
|---|---|
| `NEXT` | Sequential order — every adjacent pair |
| `SAME_INTENT` | Non-adjacent chunks sharing the same intent |
| `METRIC_OF` | Metric chunk linked back to its nearest heading |

---

## Cross-Document Stitching

Merge semantically equivalent sections from multiple documents into a unified de-duplicated set:

```python
from omnidocrag.pipeline.stitcher import stitch_documents
# or:
from omnidocrag import stitch_documents

docs = [
    {"metadata": {"file": "q3_report.pdf"}, "chunks": [c.to_dict() for c in chunks_q3]},
    {"metadata": {"file": "q4_report.pdf"}, "chunks": [c.to_dict() for c in chunks_q4]},
]

merged = stitch_documents(docs, similarity_threshold=0.80)

for chunk in merged:
    if len(chunk["sources"]) > 1:
        print(f"Merged from: {chunk['sources']}  {chunk['text'][:80]}")
```

Two chunks merge when their heading + intent match at `SequenceMatcher` similarity ≥ `similarity_threshold`. The merged chunk's `sources` lists all contributing files.

---

## Vector DB Adapters

All four adapters share the same `upsert(chunks)` / `query(text)` interface and accept either `SemanticChunk` objects or plain dicts.

---

### ChromaDB

```python
from omnidocrag.vectordb.chroma import ChromaAdapter

adapter = ChromaAdapter(collection_name="omnidoc")   # in-memory

# Persistent
import chromadb
adapter = ChromaAdapter(
    collection_name="omnidoc",
    client=chromadb.PersistentClient(path="/data/chroma"),
)

count = adapter.upsert(chunks)
results = adapter.query("Q3 revenue growth", n_results=5, where={"intent": "metric"})
```

Requires `pip install "omnidoc-rag[chroma]"`.

---

### Pinecone

```python
from omnidocrag.vectordb.pinecone import PineconeAdapter

adapter = PineconeAdapter(
    index_name="omnidoc-prod",
    embedding_fn=lambda text: model.encode(text).tolist(),
    api_key="pc-xxxxxxxxxxxxx",
    namespace="reports",
)

count = adapter.upsert(chunks)
results = adapter.query("EBITDA margin", top_k=5)
```

Requires `pip install "omnidoc-rag[pinecone]"`.

---

### Weaviate

```python
from omnidocrag.vectordb.weaviate import WeaviateAdapter

# Local Weaviate (localhost:8080)
adapter = WeaviateAdapter(
    collection_name="OmnidocChunks",
    embedding_fn=lambda text: model.encode(text).tolist(),
)

# Weaviate Cloud
adapter = WeaviateAdapter(
    collection_name="OmnidocChunks",
    embedding_fn=lambda text: model.encode(text).tolist(),
    wcd_url="https://my-cluster.weaviate.network",
    wcd_api_key="wcd-api-key",
)

count = adapter.upsert(chunks)
from weaviate.classes.query import Filter
results = adapter.query(
    "revenue growth", limit=5,
    filters=Filter.by_property("intent").equal("metric"),
)
adapter.close()
```

Requires `pip install "omnidoc-rag[weaviate]"`.

---

### PostgreSQL / pgvector

```python
from omnidocrag.vectordb.pgvector import PgVectorAdapter

adapter = PgVectorAdapter(
    embedding_fn=lambda text: model.encode(text).tolist(),
    dsn="postgresql://user:password@localhost:5432/ragdb",
    table="omnidoc_chunks",
    dimensions=384,
    create_index=True,
)

count = adapter.upsert(chunks)
results = adapter.query("revenue growth", top_k=5)
results = adapter.query(
    "revenue growth", top_k=5,
    where="intent = %s", where_params=("metric",),
)
adapter.delete(["id1", "id2"])

with PgVectorAdapter(embedding_fn=..., dsn=..., dimensions=384) as adapter:
    adapter.upsert(chunks)
```

Requires `pip install "omnidoc-rag[pgvector]"` and `CREATE EXTENSION IF NOT EXISTS vector;` on your PostgreSQL instance.

---

### Adapter comparison

| | ChromaDB | Pinecone | Weaviate | pgvector |
|---|---|---|---|---|
| Embedding fn required | No | Yes | No (or custom) | Yes |
| Self-hosted | Yes | No | Yes / WCD | Yes |
| Persistent by default | No (in-memory) | Yes | Yes | Yes |
| Filter on query | Yes (`where` dict) | No | Yes (Filter API) | Yes (raw SQL) |
| Similarity metric | Cosine (distance) | Cosine | Cosine / certainty | Cosine (`<=>`) |
| Install extra | `chroma` | `pinecone` | `weaviate` | `pgvector` |

---

## Schema Reference

```python
from omnidocrag.schema import SemanticChunk

chunk.id           # str  — SHA1 deterministic ID
chunk.text         # str  — chunk content
chunk.intent       # str  — metric|table|process|value_proposition|heading|narrative
chunk.confidence   # float — 0.1 … 1.0
chunk.page         # int  — source page number
chunk.heading      # str | None — nearest ancestor heading
chunk.keywords     # List[str] — top-15 BM25 non-stopword terms
chunk.metadata     # Dict[str, Any] — source, chunk_index, char_length,
                   #                  embedding_hint, chunk_type, + type-specific fields

chunk.to_dict()    # → plain dict, JSON-safe, vector DB ready
```

All Excel chunks carry additional `chunk_type` and type-specific metadata fields (see [Excel — 57 Chunk Types](#excel--57-chunk-types)).

---

## Package Structure

```
omnidoc-rag/
├── omnidocrag/
│   ├── __init__.py              Public API — chunk_document, stream_chunks,
│   │                            evaluate_rag_result, get_chunker, …
│   ├── schema.py                SemanticChunk dataclass
│   │
│   ├── core/                    NLP primitives
│   │   ├── intent.py            classify_intent() — deterministic regex classifier
│   │   ├── confidence.py        score_chunk() — quality scoring
│   │   ├── adaptive.py          tokens_for_intent() — per-intent token budgets
│   │   └── hybrid_metadata.py   BM25 keywords + SHA1 embedding hint
│   │
│   ├── pipeline/                RAG pipeline components
│   │   ├── chunker.py           chunk_document() — registry dispatcher
│   │   ├── stream.py            stream_chunks() — lazy generator
│   │   ├── graph.py             link_chunks() — NEXT/SAME_INTENT/METRIC_OF
│   │   ├── stitcher.py          stitch_documents() — cross-doc merging
│   │   └── evaluation.py        evaluate_rag_result() — retrieval scoring
│   │
│   ├── chunkers/                Modular file-type chunkers
│   │   ├── __init__.py          Registry — get_chunker(), chunk_by_type(),
│   │   │                        supported_extensions()
│   │   ├── base.py              BaseChunker ABC + shared helpers
│   │   │
│   │   ├── office/
│   │   │   ├── pdf.py           PDFChunker — heading-aware + table rows
│   │   │   ├── docx.py          DOCXChunker — paragraph + table batches
│   │   │   ├── pptx.py          PPTXChunker — slide-per-chunk + speaker notes
│   │   │   └── xlsx/            XLSXChunker — 57 chunk types
│   │   │       ├── __init__.py  XLSXChunker (orchestrator + fallback)
│   │   │       ├── _helpers.py  Shared helpers (make_chunk, infer_dtype, …)
│   │   │       ├── structural.py   Chunks 1–9
│   │   │       ├── semantic.py     Chunks 10–15
│   │   │       ├── analytical.py   Chunks 16–23
│   │   │       ├── validation.py   Chunks 24–28
│   │   │       ├── visual.py       Chunks 29–41
│   │   │       ├── vba.py          Chunks 42–46
│   │   │       ├── crossref.py     Chunks 47–50
│   │   │       ├── connectivity.py Chunks 51–54
│   │   │       └── operational.py  Chunks 55–57
│   │   │
│   │   ├── text/txt.py          TXTChunker
│   │   ├── markdown/md.py       MDChunker
│   │   ├── code/
│   │   │   ├── python.py        PythonChunker
│   │   │   ├── javascript.py    JSChunker (.js + .ts)
│   │   │   └── java.py          JavaChunker
│   │   ├── structured/
│   │   │   ├── csv.py           CSVChunker
│   │   │   ├── json_chunker.py  JSONChunker
│   │   │   ├── xml_chunker.py   XMLChunker
│   │   │   └── yaml_chunker.py  YAMLChunker
│   │   ├── web/html.py          HTMLChunker
│   │   ├── image/image.py       ImageChunker
│   │   ├── archive/archive.py   ArchiveChunker
│   │   ├── ebook/
│   │   │   ├── epub.py          EPUBChunker
│   │   │   ├── odt.py           ODTChunker
│   │   │   └── rtf.py           RTFChunker
│   │   ├── email/
│   │   │   ├── eml.py           EMLChunker
│   │   │   └── msg.py           MSGChunker
│   │   └── notebook/ipynb.py    IPYNBChunker
│   │
│   └── vectordb/                Vector DB adapters
│       ├── chroma.py            ChromaAdapter
│       ├── pinecone.py          PineconeAdapter
│       ├── weaviate.py          WeaviateAdapter (v4 Collections API)
│       └── pgvector.py          PgVectorAdapter (psycopg2 + pgvector)
│
├── tests/
│   ├── test_rag.py              111 tests — intent, confidence, chunker,
│   │                            evaluation, graph, stitcher, stream
│   └── test_vectordb.py         All 4 vector DB adapters (mock-based, no live DB)
├── pyproject.toml
└── README.md
```

---

## Optional Extras

| Extra | Install | Unlocks |
|---|---|---|
| `chroma` | `pip install "omnidoc-rag[chroma]"` | ChromaDB adapter |
| `pinecone` | `pip install "omnidoc-rag[pinecone]"` | Pinecone adapter |
| `weaviate` | `pip install "omnidoc-rag[weaviate]"` | Weaviate v4 adapter |
| `pgvector` | `pip install "omnidoc-rag[pgvector]"` | PostgreSQL + pgvector adapter |
| `all` | `pip install "omnidoc-rag[all]"` | All four adapters |

---

## Contributing & Development

### Setup

```bash
git clone https://github.com/your-org/omnidoc-rag.git
cd omnidoc-rag
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate

pip install -e ".[all]"
pip install build twine pytest pytest-cov ruff black mypy
```

`omnidoc-sdk` is a required dependency:

```bash
pip install omnidoc-sdk
# or from local source:
pip install -e ../omnidoc-sdk
```

### Running Tests

```bash
# All 111 tests
pytest tests/ -v

# Specific class
pytest tests/test_rag.py::TestChunkDocument -v
pytest tests/test_vectordb.py::TestChromaAdapter -v

# With coverage
pytest tests/ --cov=omnidocrag --cov-report=term-missing
```

All tests use fake `Document` objects and mocked DB clients — no `omnidoc-sdk` install or live database required.

### Lint and type-check

```bash
ruff check omnidocrag/
black --check omnidocrag/
mypy omnidocrag/ --ignore-missing-imports

# Auto-fix
black omnidocrag/
ruff check omnidocrag/ --fix
```

### Building & Publishing

```bash
rm -rf dist/ build/ omnidocrag.egg-info/
python -m build
twine check dist/*
twine upload --repository testpypi dist/*   # test first
twine upload dist/*                          # production
```

### Versioning

Follows [Semantic Versioning](https://semver.org/). Version is defined in `pyproject.toml`.

### Release Checklist

```
[ ] ruff check omnidocrag/          — zero errors
[ ] black --check omnidocrag/        — no formatting changes
[ ] pytest tests/ -v                 — all 111 tests pass
[ ] Version bumped in pyproject.toml
[ ] Changelog updated
[ ] rm -rf dist/ && python -m build
[ ] twine check dist/*               — both artifacts PASSED
[ ] TestPyPI round-trip verified
[ ] twine upload dist/*
[ ] git tag vX.Y.Z && git push origin vX.Y.Z
```

### Troubleshooting

**`ImportError: chromadb/pinecone/weaviate-client/psycopg2 not installed`**
→ Install the corresponding extra: `pip install "omnidoc-rag[chroma|pinecone|weaviate|pgvector]"`

**`XLSXChunker` produces only basic row chunks (fallback mode)**
→ The file path in `doc.metadata["file"]` must point to an accessible `.xlsx` file on disk. `openpyxl` reads it directly for full 57-chunk extraction.

**`evaluate_rag_result` returns `overall=0.0` / `verdict="unsafe"`**
→ The `chunks` list is empty. Verify `chunk_document(doc)` ran on a document with non-empty sections.

**`WeaviateConnectionError`** — start a local Weaviate with Docker:
```bash
docker run -d -p 8080:8080 -p 50051:50051 cr.weaviate.io/semitechnologies/weaviate:latest
```

**pgvector `could not open extension "vector"`**
```sql
CREATE EXTENSION IF NOT EXISTS vector;   -- run as superuser
```

---

## Changelog

### [0.1.3] — 2026-05-15

#### Added — Modular file-type chunkers
- **41 file extensions** now route to dedicated chunker classes via a central registry
- `chunkers/` package with one file per format, organised into sub-packages:
  - `office/` — `PDFChunker`, `DOCXChunker`, `PPTXChunker`, `XLSXChunker`
  - `text/` — `TXTChunker`
  - `markdown/` — `MDChunker` (heading hierarchy, fenced code blocks, MD tables, front matter)
  - `code/` — `PythonChunker` (def/class), `JSChunker` (function/class/interface), `JavaChunker` (class/method)
  - `structured/` — `CSVChunker`, `JSONChunker`, `XMLChunker`, `YAMLChunker`
  - `web/` — `HTMLChunker` (semantic tag-aware)
  - `image/` — `ImageChunker` (OCR text or metadata stub)
  - `archive/` — `ArchiveChunker` (manifest + contained-file delegation)
  - `ebook/` — `EPUBChunker`, `ODTChunker`, `RTFChunker`
  - `email/` — `EMLChunker`, `MSGChunker` (Outlook meeting detection)
  - `notebook/` — `IPYNBChunker` (cell-aware, output chunks)
- `chunkers/__init__.py` — `get_chunker(path)`, `chunk_by_type(doc)`, `supported_extensions()`
- `BaseChunker` ABC with shared `_make_chunk`, `_flush`, `_source`, `_is_heading` helpers

#### Added — Excel 57 chunk types
- `XLSXChunker` rewritten as a 10-file package (`xlsx/`) using `openpyxl` direct file access
- **Structural** (9): WorkbookChunk, SheetChunk, TableChunk, SchemaChunk, RowChunk, GroupChunk, ParentContextChunk, MergedCellChunk, FrozenPaneChunk
- **Semantic** (6): SummaryChunk, SemanticNarrativeChunk, ColumnSemanticChunk, HeaderAliasChunk, EntityChunk, CellAnnotationChunk
- **Analytical** (8): FormulaDefinitionChunk, FormulaResultChunk, KPIChunk, AggregationChunk, TemporalChunk, TrendChunk, OutlierChunk, LookupMapChunk
- **Validation** (5): ValidationChunk, ErrorChunk, ConditionalFormatChunk, ProtectionChunk, DataQualityChunk
- **Visual** (13): ChartChunk, ChartSeriesChunk, ChartAnnotationChunk, PivotTableChunk, PivotFieldChunk, PivotCacheChunk, SlicerChunk, TimelineChunk, SparklineChunk, ShapeChunk, ImageChunk, FormControlChunk, ActiveXControlChunk
- **VBA/Macro** (5): MacroChunk, VBAModuleChunk, VBAEventChunk, CustomFunctionChunk, RibbonCustomizationChunk
- **Cross-ref** (4): RelationshipChunk, NamedRangeChunk, ExternalLinkChunk, DependencyGraphChunk
- **Connectivity** (4): PowerQueryChunk, DataModelChunk, PowerPivotMeasureChunk, ConnectionChunk
- **Operational** (3): PrintSettingsChunk, ChangeLogChunk, ChunkIndexChunk (master manifest)
- Graceful fallback to basic row-batch extraction when file path is inaccessible

#### Added — Package restructuring
- `core/` sub-package: `intent.py`, `confidence.py`, `adaptive.py`, `hybrid_metadata.py`
- `pipeline/` sub-package: `chunker.py`, `stream.py`, `graph.py`, `stitcher.py`, `evaluation.py`
- `schema.py` remains at package root (shared by all layers)
- All imports updated to canonical new paths; no shim files retained

#### Changed
- `chunk_document()` now auto-routes to the right chunker by file extension instead of using the single generic algorithm for all types
- Import paths updated: `omnidocrag.core.intent`, `omnidocrag.pipeline.chunker`, etc.
- Test suite updated to canonical import paths; all 111 tests continue to pass

---

### [0.1.0] — 2026-04-10

Initial release of `omnidoc-rag` as a standalone SDK.

#### Added
- `SemanticChunk` dataclass with `to_dict()` serialisation
- `classify_intent()` — deterministic 6-label regex classifier
- `tokens_for_intent()` — per-intent token budgets (heading=60 → process=400)
- `score_chunk()` — confidence scoring with intent-aware length-penalty exemption
- `hybrid_metadata()` — BM25 keyword extraction + SHA1 embedding hint
- `chunk_document()` — heading-aware chunker with overlap carry-over and table row expansion
- `stream_chunks()` — true lazy generator
- `evaluate_rag_result()` — coverage / confidence / diversity / verdict scoring
- `link_chunks()` — NEXT, SAME\_INTENT, METRIC\_OF edge graph
- `stitch_documents()` — cross-document heading+intent merging
- `ChromaAdapter`, `PineconeAdapter`, `WeaviateAdapter` (v4), `PgVectorAdapter`
- 111-test suite; mock-based vector DB tests requiring no live connections

---

<div align="center">

**omnidoc-rag** &nbsp;·&nbsp; v0.1.3 &nbsp;·&nbsp; Apache 2.0 &nbsp;·&nbsp; Extraction layer → [omnidoc-sdk](https://github.com/your-org/omnidoc-python-sdk)

</div>
