Metadata-Version: 2.4
Name: omnidoc-sdk
Version: 0.4.1
Summary: Enterprise-grade SDK for document ingestion, OCR, semantic chunking, and RAG-ready processing
Author-email: Ganesh Kinkar Giri <k.ganeshgiri@example.com>
License: Apache-2.0
Keywords: document-intelligence,pdf,ocr,rag,semantic-chunking,agentic-ai,enterprise-ai
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Text Processing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pypdf>=4.0.0
Requires-Dist: pdfplumber>=0.11.0
Requires-Dist: pdf2image>=1.17.0
Requires-Dist: Pillow>=10.0.0
Requires-Dist: pytesseract>=0.3.10
Requires-Dist: opencv-python>=4.9.0
Requires-Dist: numpy>=1.24
Requires-Dist: chardet>=5.2.0
Requires-Dist: regex>=2023.12.25
Provides-Extra: office
Requires-Dist: python-docx>=1.1.0; extra == "office"
Requires-Dist: openpyxl>=3.1.2; extra == "office"
Requires-Dist: python-pptx>=0.6.23; extra == "office"
Provides-Extra: data
Requires-Dist: pandas>=2.1.0; extra == "data"
Requires-Dist: pyarrow>=15.0.0; extra == "data"
Provides-Extra: docs
Requires-Dist: markdown>=3.6; extra == "docs"
Requires-Dist: beautifulsoup4>=4.12.0; extra == "docs"
Requires-Dist: lxml>=5.1.0; extra == "docs"
Provides-Extra: archive
Requires-Dist: py7zr>=0.21.0; extra == "archive"
Requires-Dist: rarfile>=4.2; extra == "archive"
Provides-Extra: code
Requires-Dist: nbformat>=5.10.0; extra == "code"
Requires-Dist: pygments>=2.17.0; extra == "code"
Provides-Extra: cloud-ocr
Requires-Dist: boto3>=1.34.0; extra == "cloud-ocr"
Requires-Dist: azure-ai-formrecognizer>=3.3.0; extra == "cloud-ocr"
Provides-Extra: layout
Requires-Dist: layoutparser>=0.3.4; extra == "layout"
Provides-Extra: vlm
Requires-Dist: torch>=2.0; extra == "vlm"
Requires-Dist: transformers>=4.41.0; extra == "vlm"
Provides-Extra: privacy
Requires-Dist: presidio-analyzer>=2.2.0; extra == "privacy"
Requires-Dist: presidio-anonymizer>=2.2.0; extra == "privacy"
Provides-Extra: media
Requires-Dist: ffmpeg-python>=0.2.0; extra == "media"
Requires-Dist: openai-whisper>=20231117; extra == "media"
Provides-Extra: ebook
Requires-Dist: ebooklib>=0.18; extra == "ebook"
Requires-Dist: beautifulsoup4>=4.12.0; extra == "ebook"
Provides-Extra: openoffice
Requires-Dist: odfpy>=1.4.1; extra == "openoffice"
Provides-Extra: legacy
Requires-Dist: striprtf>=0.0.26; extra == "legacy"
Provides-Extra: email
Requires-Dist: extract-msg>=0.48.0; extra == "email"
Provides-Extra: image
Requires-Dist: pillow-heif>=0.16.0; extra == "image"
Provides-Extra: yaml
Requires-Dist: pyyaml>=6.0; extra == "yaml"
Provides-Extra: all
Requires-Dist: python-docx>=1.1.0; extra == "all"
Requires-Dist: openpyxl>=3.1.2; extra == "all"
Requires-Dist: python-pptx>=0.6.23; extra == "all"
Requires-Dist: pandas>=2.1.0; extra == "all"
Requires-Dist: pyarrow>=15.0.0; extra == "all"
Requires-Dist: markdown>=3.6; extra == "all"
Requires-Dist: beautifulsoup4>=4.12.0; extra == "all"
Requires-Dist: lxml>=5.1.0; extra == "all"
Requires-Dist: pyyaml>=6.0; extra == "all"
Requires-Dist: py7zr>=0.21.0; extra == "all"
Requires-Dist: rarfile>=4.2; extra == "all"
Requires-Dist: nbformat>=5.10.0; extra == "all"
Requires-Dist: pygments>=2.17.0; extra == "all"
Requires-Dist: boto3>=1.34.0; extra == "all"
Requires-Dist: azure-ai-formrecognizer>=3.3.0; extra == "all"
Requires-Dist: layoutparser>=0.3.4; extra == "all"
Requires-Dist: torch>=2.0; extra == "all"
Requires-Dist: transformers>=4.41.0; extra == "all"
Requires-Dist: presidio-analyzer>=2.2.0; extra == "all"
Requires-Dist: presidio-anonymizer>=2.2.0; extra == "all"
Requires-Dist: ffmpeg-python>=0.2.0; extra == "all"
Requires-Dist: openai-whisper>=20231117; extra == "all"
Requires-Dist: ebooklib>=0.18; extra == "all"
Requires-Dist: odfpy>=1.4.1; extra == "all"
Requires-Dist: striprtf>=0.0.26; extra == "all"
Requires-Dist: extract-msg>=0.48.0; extra == "all"
Requires-Dist: pillow-heif>=0.16.0; extra == "all"
Dynamic: license-file

<div align="center">

<h1>OmniDoc Python SDK</h1>

<p><strong>Enterprise-grade document intelligence for Agentic AI, RAG pipelines, and automation systems</strong></p>

<p>
  <img src="https://img.shields.io/badge/python-3.9%2B-blue?style=flat-square" alt="Python 3.9+"/>
  <img src="https://img.shields.io/badge/version-0.4.0-green?style=flat-square" alt="v0.4.0"/>
  <img src="https://img.shields.io/badge/license-Apache--2.0-orange?style=flat-square" alt="Apache 2.0"/>
  <img src="https://img.shields.io/badge/formats-40%2B-purple?style=flat-square" alt="40+ formats"/>
  <img src="https://img.shields.io/badge/OCR-Tesseract%20%7C%20Textract%20%7C%20Azure-teal?style=flat-square" alt="OCR"/>
</p>

</div>

---

## Table of Contents

1. [Overview](#overview)
2. [Installation](#installation)
3. [Supported Document Types](#supported-document-types)
4. [Architecture](#architecture)
5. [Quick Start](#quick-start)
6. [Core API — `extract_pdf()`](#core-api--extract_pdf)
7. [Universal Loader — `load_document()`](#universal-loader--load_document)
   - [Text, Markdown & Code](#text-markdown--code)
   - [Office Documents](#office-documents)
   - [Spreadsheets](#spreadsheets)
   - [Presentations](#presentations)
   - [OpenDocument (LibreOffice)](#opendocument-libreoffice)
   - [Web & Markup](#web--markup)
   - [Structured Data — JSON, XML, YAML](#structured-data--json-xml-yaml)
   - [Images (OCR)](#images-ocr)
   - [E-Books (EPUB)](#e-books-epub)
   - [Email (.eml / .msg)](#email-eml--msg)
   - [Legacy — RTF](#legacy--rtf)
   - [Jupyter Notebooks](#jupyter-notebooks)
   - [Archives](#archives)
   - [Audio & Video (Transcription)](#audio--video-transcription)
8. [Output Formats](#output-formats)
9. [OCR Engines](#ocr-engines)
10. [RAG Pipeline](#rag-pipeline)
11. [PII Masking](#pii-masking)
12. [Enterprise Configuration](#enterprise-configuration)
13. [Structured Logging](#structured-logging)
14. [Retry & Resilience](#retry--resilience)
15. [CLI](#cli)
16. [Optional Extras](#optional-extras)
17. [Contributing & Development](#contributing--development)
    - [Setup](#setup)
    - [Project Structure](#project-structure-1)
    - [Running Tests](#running-tests)
    - [Building & Publishing](#building--publishing)
    - [Versioning](#versioning)
    - [Release Checklist](#release-checklist)
    - [Troubleshooting](#troubleshooting)
18. [Changelog](#changelog)

---

## Overview

**OmniDoc** converts raw documents — PDFs, spreadsheets, slides, emails, scanned images, notebooks, source code, archives, audio/video, and more — into clean, structured, agent-ready outputs.

<table>
<tr><th>Layer</th><th>What it does</th></tr>
<tr><td>Detection</td><td>40+ file types detected automatically from extension</td></tr>
<tr><td>Extraction</td><td>Text, tables, layout blocks, metadata per document</td></tr>
<tr><td>Normalization</td><td>Headings, bullets, metrics, slide-to-report restructuring</td></tr>
<tr><td>Serialization</td><td>Python <code>Document</code>, JSON, or agent-native TOON</td></tr>
</table>

> **RAG pipeline?** Install [`omnidoc-rag`](https://github.com/your-org/omnidoc-rag) for intent-aware chunking, confidence scoring, streaming, graph linking, evaluation, and vector DB adapters.

---

## Installation

### Minimal (PDF + local OCR)

```bash
pip install omnidoc-sdk
```

### With Office documents (Word, Excel, PowerPoint)

```bash
pip install "omnidoc-sdk[office,data]"
```

### With web, email, and structured formats

```bash
pip install "omnidoc-sdk[docs,email,yaml,ebook,openoffice,legacy]"
```

### With code & notebooks

```bash
pip install "omnidoc-sdk[code]"
```

### With cloud OCR (AWS Textract / Azure Form Recognizer)

```bash
pip install "omnidoc-sdk[cloud-ocr]"
```

### With audio/video transcription (Whisper)

```bash
pip install "omnidoc-sdk[media]"
```

### Full enterprise install (everything)

```bash
pip install "omnidoc-sdk[all]"
```

### System dependencies

<table>
<tr><th>OS</th><th>Command</th></tr>
<tr><td>macOS</td><td><code>brew install poppler tesseract</code></td></tr>
<tr><td>Ubuntu / Debian</td><td><code>apt-get install poppler-utils tesseract-ocr libgl1</code></td></tr>
<tr><td>RHEL / CentOS</td><td><code>yum install poppler-utils tesseract</code></td></tr>
</table>

---

## Supported Document Types

Every type below is fully supported by `load_document()`. Optional extras are loaded lazily — only installed when the specific file type is used.

<table>
<thead>
<tr>
  <th>Category</th>
  <th>Extensions</th>
  <th>Extractor</th>
  <th>Install Extra</th>
  <th>Capabilities</th>
</tr>
</thead>
<tbody>

<tr>
  <td><strong>PDF</strong></td>
  <td><code>.pdf</code></td>
  <td>Full pipeline</td>
  <td><em>core</em></td>
  <td>Digital, scanned, slides; OCR, layout ML, tables, VLM</td>
</tr>

<tr>
  <td><strong>Plain Text</strong></td>
  <td><code>.txt</code></td>
  <td><code>TextExtractor</code></td>
  <td><em>core</em></td>
  <td>Auto-detects encoding (UTF-8 → latin-1 → chardet fallback)</td>
</tr>

<tr>
  <td><strong>Markdown / RST</strong></td>
  <td><code>.md</code> &nbsp; <code>.rst</code></td>
  <td><code>MarkdownExtractor</code></td>
  <td><em>core</em></td>
  <td>Headings, code blocks, lists preserved as sections</td>
</tr>

<tr>
  <td><strong>Word</strong></td>
  <td><code>.docx</code></td>
  <td><code>DocxExtractor</code></td>
  <td><code>office</code></td>
  <td>Paragraphs, headings, inline tables</td>
</tr>

<tr>
  <td><strong>Excel / Spreadsheet</strong></td>
  <td><code>.xlsx</code> &nbsp; <code>.xls</code> &nbsp; <code>.csv</code> &nbsp; <code>.tsv</code></td>
  <td><code>SpreadsheetExtractor</code></td>
  <td><code>office, data</code></td>
  <td>Multi-sheet; multi-encoding CSV/TSV; headers + typed rows</td>
</tr>

<tr>
  <td><strong>PowerPoint</strong></td>
  <td><code>.pptx</code></td>
  <td><code>PPTXExtractor</code></td>
  <td><code>office</code></td>
  <td>Slide title + body text per slide</td>
</tr>

<tr>
  <td><strong>OpenDocument</strong></td>
  <td><code>.odt</code> &nbsp; <code>.ods</code> &nbsp; <code>.odp</code></td>
  <td><code>OpenDocumentExtractor</code></td>
  <td><code>openoffice</code></td>
  <td>LibreOffice Writer, Calc, Impress — text, tables, slides</td>
</tr>

<tr>
  <td><strong>RTF</strong></td>
  <td><code>.rtf</code></td>
  <td><code>RTFExtractor</code></td>
  <td><code>legacy</code></td>
  <td>Legacy Word / WordPad format, full control-word stripping</td>
</tr>

<tr>
  <td><strong>Images (OCR)</strong></td>
  <td><code>.png</code> &nbsp; <code>.jpg</code> &nbsp; <code>.jpeg</code> &nbsp; <code>.tiff</code> &nbsp; <code>.tif</code> &nbsp; <code>.webp</code> &nbsp; <code>.heic</code> &nbsp; <code>.heif</code> &nbsp; <code>.bmp</code> &nbsp; <code>.gif</code></td>
  <td><code>ImageExtractor</code></td>
  <td><em>core</em> (<code>image</code> for HEIC)</td>
  <td>Tesseract OCR, per-image timeout, HEIC/HEIF via pillow-heif</td>
</tr>

<tr>
  <td><strong>SVG</strong></td>
  <td><code>.svg</code></td>
  <td><code>HTMLExtractor</code></td>
  <td><code>docs</code></td>
  <td>Text nodes extracted, markup stripped</td>
</tr>

<tr>
  <td><strong>HTML / Web</strong></td>
  <td><code>.html</code> &nbsp; <code>.htm</code></td>
  <td><code>HTMLExtractor</code></td>
  <td><code>docs</code></td>
  <td>Script/style removed; per-block sections (p, h1–h6, li, td)</td>
</tr>

<tr>
  <td><strong>JSON</strong></td>
  <td><code>.json</code></td>
  <td><code>JSONExtractor</code></td>
  <td><em>core</em></td>
  <td>Flattened key-value pairs as document sections</td>
</tr>

<tr>
  <td><strong>XML</strong></td>
  <td><code>.xml</code></td>
  <td><code>XMLExtractor</code></td>
  <td><em>core</em></td>
  <td>Element text + tail content extracted as sections</td>
</tr>

<tr>
  <td><strong>YAML</strong></td>
  <td><code>.yaml</code> &nbsp; <code>.yml</code></td>
  <td><code>YAMLExtractor</code></td>
  <td><code>yaml</code></td>
  <td>Parsed with PyYAML, serialised to pretty JSON for chunking</td>
</tr>

<tr>
  <td><strong>E-Book</strong></td>
  <td><code>.epub</code></td>
  <td><code>EPUBExtractor</code></td>
  <td><code>ebook</code></td>
  <td>Per-chapter sections, author + title metadata</td>
</tr>

<tr>
  <td><strong>Email — EML</strong></td>
  <td><code>.eml</code></td>
  <td><code>EMLExtractor</code></td>
  <td><em>core</em> (stdlib)</td>
  <td>Headers (From/To/Subject/Date), plain-text + HTML parts, attachment list</td>
</tr>

<tr>
  <td><strong>Email — Outlook</strong></td>
  <td><code>.msg</code></td>
  <td><code>MSGExtractor</code></td>
  <td><code>email</code></td>
  <td>Sender, recipients, subject, body, attachment list via extract-msg</td>
</tr>

<tr>
  <td><strong>Jupyter Notebook</strong></td>
  <td><code>.ipynb</code></td>
  <td><code>NotebookExtractor</code></td>
  <td><code>code</code></td>
  <td>Markdown + code + output cells as sections; kernel/language metadata</td>
</tr>

<tr>
  <td><strong>Source Code</strong></td>
  <td><code>.py</code> &nbsp; <code>.java</code> &nbsp; <code>.js</code> &nbsp; <code>.ts</code> &nbsp; <code>.go</code> &nbsp; <code>.cpp</code> &nbsp; <code>.c</code> &nbsp; <code>.rs</code></td>
  <td><code>CodeExtractor</code></td>
  <td><em>core</em> (<code>code</code> for token count)</td>
  <td>Block-level splitting (function/class boundaries), language metadata, optional Pygments token count</td>
</tr>

<tr>
  <td><strong>Archives</strong></td>
  <td><code>.zip</code> &nbsp; <code>.tar</code> &nbsp; <code>.tar.gz</code> &nbsp; <code>.tar.bz2</code> &nbsp; <code>.tar.xz</code> &nbsp; <code>.7z</code></td>
  <td><code>ArchiveExtractor</code></td>
  <td><em>core</em> (<code>archive</code> for .7z)</td>
  <td>Path-traversal guard (zip-slip), 512 MB per-member bomb limit; returns extracted file paths</td>
</tr>

<tr>
  <td><strong>Audio</strong></td>
  <td><code>.mp3</code> &nbsp; <code>.wav</code> &nbsp; <code>.m4a</code></td>
  <td><code>MediaExtractor</code></td>
  <td><code>media</code></td>
  <td>Whisper transcription, segment-level sections, auto language detection</td>
</tr>

<tr>
  <td><strong>Video</strong></td>
  <td><code>.mp4</code> &nbsp; <code>.mov</code> &nbsp; <code>.avi</code> &nbsp; <code>.mkv</code></td>
  <td><code>MediaExtractor</code></td>
  <td><code>media</code></td>
  <td>Audio extracted via ffmpeg, then Whisper transcription</td>
</tr>

</tbody>
</table>

---

## Architecture

```
Input File
    │
    ├─ detect_document_type()          ← 40+ extensions, compound (.tar.gz)
    │
    ├─ PDF Pipeline ────────────────────────────────────────────────────
    │   ├─ is_scanned_pdf()            ← digital vs. scanned detection
    │   ├─ extract_text()              ← pdfplumber / pypdf
    │   ├─ convert_from_path()         ← pdf2image rasterisation
    │   ├─ detect_layout()             ← Detectron2 layout ML (optional)
    │   ├─ order_blocks()              ← column-aware reading order
    │   ├─ OCR Engine                  ← Tesseract / Textract / Azure
    │   ├─ extract_tables()            ← camelot lattice extraction
    │   ├─ extract_ocr_tables()        ← table detection from images
    │   └─ normalize_slides()          ← slide → report restructuring
    │
    ├─ Non-PDF Extractors ──────────────────────────────────────────────
    │   ├─ TextExtractor               ← .txt
    │   ├─ MarkdownExtractor           ← .md / .rst
    │   ├─ DocxExtractor               ← .docx
    │   ├─ SpreadsheetExtractor        ← .xlsx / .xls / .csv / .tsv
    │   ├─ PPTXExtractor               ← .pptx
    │   ├─ OpenDocumentExtractor       ← .odt / .ods / .odp
    │   ├─ RTFExtractor                ← .rtf
    │   ├─ HTMLExtractor               ← .html / .htm / .svg
    │   ├─ JSONExtractor               ← .json
    │   ├─ XMLExtractor                ← .xml
    │   ├─ YAMLExtractor               ← .yaml / .yml
    │   ├─ EPUBExtractor               ← .epub
    │   ├─ EMLExtractor                ← .eml (stdlib, zero deps)
    │   ├─ MSGExtractor                ← .msg (Outlook)
    │   ├─ ImageExtractor              ← .png / .jpg / .tiff / .webp / .heic ...
    │   ├─ NotebookExtractor           ← .ipynb
    │   ├─ CodeExtractor               ← .py / .java / .js / .ts / .go / ...
    │   ├─ ArchiveExtractor            ← .zip / .tar / .tar.gz / .7z
    │   └─ MediaExtractor              ← .mp3 / .mp4 / .wav / .mov ...
    │
    ├─ Post-processing ─────────────────────────────────────────────────
    │   ├─ clean_text()
    │   ├─ normalize_slide_sections()
    │   ├─ build_document()            ← assembles Document dataclass
    │   └─ mask_pii()                  ← Presidio (optional)
    │
    └─ Output ──────────────────────────────────────────────────────────
        ├─ Document                    ← Python dataclass
        ├─ JSON                        ← RAG-ready dict
        └─ TOON                        ← agent-native format
```

---

## Quick Start

```python
from omnidoc.loader.load import load_document

# Works for every supported file type
doc = load_document("report.pdf")

print(doc.raw_text[:500])         # cleaned full text
print(doc.metadata)               # file info, page count, pipeline

# Pass to omnidoc-rag for chunking, evaluation, and vector DB ingestion
```

---

## Core API — `extract_pdf()`

The PDF-specific pipeline with full control over every stage.

```python
from omnidoc.pdf.pipeline import extract_pdf

doc = extract_pdf(
    path="document.pdf",          # required: path to PDF
    enable_layout=True,           # layout ML for reading-order (default: True)
    enable_cloud_ocr=False,       # use AWS Textract for scanned PDFs
    enable_vlm=False,             # use Donut VLM for complex layouts
    enable_pii_masking=False,     # redact PII via Presidio
    output_format="document",     # "document" | "json" | "toon"
    config=None,                  # OmnidocConfig (uses default when None)
)
```

### Parameter reference

<table>
<thead>
<tr><th>Parameter</th><th>Type</th><th>Default</th><th>Description</th></tr>
</thead>
<tbody>
<tr><td><code>path</code></td><td>str</td><td>—</td><td>Absolute or relative path to a PDF file. Raises <code>FileNotFoundError</code> if missing.</td></tr>
<tr><td><code>enable_layout</code></td><td>bool</td><td><code>True</code></td><td>Use layout ML for correct column / multi-column reading order.</td></tr>
<tr><td><code>enable_cloud_ocr</code></td><td>bool</td><td><code>False</code></td><td>Route scanned PDFs through AWS Textract. Requires <code>[cloud-ocr]</code>.</td></tr>
<tr><td><code>enable_vlm</code></td><td>bool</td><td><code>False</code></td><td>Use Donut VLM for complex scanned layouts. Requires <code>[vlm]</code>.</td></tr>
<tr><td><code>enable_pii_masking</code></td><td>bool</td><td><code>False</code></td><td>Redact PII (names, emails, SSNs). Requires <code>[privacy]</code>.</td></tr>
<tr><td><code>output_format</code></td><td>str</td><td><code>"document"</code></td><td><code>"document"</code> → <code>Document</code>; <code>"json"</code> → dict; <code>"toon"</code> → agent format.</td></tr>
<tr><td><code>config</code></td><td>OmnidocConfig</td><td><code>None</code></td><td>Override DPI, file size limit, OCR timeout, retries, and more.</td></tr>
</tbody>
</table>

### Example — digital PDF

```python
doc = extract_pdf("annual_report.pdf", enable_layout=True)

# Per-page sections
for section in doc.sections:
    print(f"Page {section.page}: {section.text[:200]}")

# Extracted tables
for table in doc.tables:
    print(f"Page {table.page}: headers={table.headers}")
    for row in table.rows[:3]:
        print(" ", row)
```

### Example — scanned PDF with cloud OCR

```python
import os
os.environ["AWS_DEFAULT_REGION"] = "us-east-1"

doc = extract_pdf("scanned_invoice.pdf", enable_cloud_ocr=True)
print(doc.raw_text)
```

---

## Universal Loader — `load_document()`

Accepts any supported file type. Extension is detected automatically.

```python
from omnidoc.loader.load import load_document

doc = load_document(
    path="file.ext",
    config=None,   # optional OmnidocConfig
)
```

### Text, Markdown & Code

```python
# Plain text — multi-encoding fallback (UTF-8 → latin-1 → chardet)
doc = load_document("notes.txt")
print(doc.raw_text)

# Markdown
doc = load_document("spec.md")
for sec in doc.sections:
    print(sec.text[:100])

# Python source — splits at function/class boundaries
doc = load_document("main.py")
for sec in doc.sections:
    print(f"Block: {sec.text[:80]}")

# Other code files — .java, .js, .ts, .go, .cpp, .c, .rs
doc = load_document("Main.java")
print(doc.metadata["language"])   # "java"
print(doc.metadata["lines"])      # line count
```

### Office Documents

```python
# Word
doc = load_document("contract.docx")
print(doc.sections[0].text)

# PowerPoint
doc = load_document("deck.pptx")
for i, sec in enumerate(doc.sections, 1):
    print(f"Slide {i}: {sec.text[:80]}")
```

### Spreadsheets

```python
# Excel
doc = load_document("financials.xlsx")
for table in doc.tables:
    print("Headers:", table.headers)
    print("Row 1:", table.rows[0])

# CSV (multi-encoding detection)
doc = load_document("users.csv")

# TSV
doc = load_document("export.tsv")
```

### Presentations

```python
doc = load_document("strategy.pptx")
for i, sec in enumerate(doc.sections, 1):
    print(f"Slide {i}: {sec.text[:100]}")
```

### OpenDocument (LibreOffice)

```python
# Writer document
doc = load_document("report.odt")
print(doc.raw_text)

# Calc spreadsheet — returns tables
doc = load_document("data.ods")
for table in doc.tables:
    print("Sheet:", table.headers)

# Impress presentation
doc = load_document("slides.odp")
for sec in doc.sections:
    print(sec.text[:100])
```

### Web & Markup

```python
# HTML — script/style stripped, per-block sections
doc = load_document("page.html")
print(doc.metadata.get("title"))   # <title> tag value
print(doc.raw_text[:500])

# SVG — text nodes extracted
doc = load_document("diagram.svg")
```

### Structured Data — JSON, XML, YAML

```python
# JSON
doc = load_document("config.json")
print(doc.raw_text)          # flattened key-value representation

# XML
doc = load_document("data.xml")

# YAML
doc = load_document("settings.yaml")
print(doc.raw_text)          # pretty-printed JSON representation of the YAML

# .yml alias also works
doc = load_document(".github/workflows/ci.yml")
```

### Images (OCR)

```python
# OCR via Tesseract
doc = load_document("screenshot.png")
print(doc.raw_text)

# TIFF, WebP, BMP, GIF — all handled
doc = load_document("scan.tiff")

# HEIC / HEIF (iPhone photos) — requires [image] extra
doc = load_document("photo.heic")

# Language hint for Tesseract
from omnidoc.extractors.image.extractor import ImageExtractor
doc = ImageExtractor(lang="eng+fra", timeout=120).extract("french_doc.jpg")
```

### E-Books (EPUB)

```python
# Each chapter becomes a Section
doc = load_document("book.epub")
print(doc.metadata["title"])
print(doc.metadata["authors"])

for i, sec in enumerate(doc.sections, 1):
    print(f"Chapter {i}: {sec.text[:100]}")
```

### Email (.eml / .msg)

```python
# RFC-822 email — zero extra dependencies
doc = load_document("message.eml")
print(doc.metadata["subject"])
print(doc.metadata["from"])
print(doc.metadata["to"])
print(doc.metadata.get("attachments", []))
print(doc.raw_text)

# Outlook .msg — requires [email] extra
doc = load_document("outlook_message.msg")
print(doc.metadata["subject"])
print(doc.raw_text)
```

### Legacy — RTF

```python
# Rich Text Format — requires [legacy] extra
doc = load_document("document.rtf")
print(doc.raw_text)
```

### Jupyter Notebooks

```python
# Markdown + code + outputs all extracted
doc = load_document("analysis.ipynb")
print(doc.metadata["kernel"])     # e.g. "Python 3 (ipykernel)"
print(doc.metadata["language"])   # "python"

for sec in doc.sections:
    print(sec.text[:150])          # one section per cell
```

### Archives

```python
# ZIP — returns extracted file paths (not a Document)
result = load_document("package.zip")
print(result["files"])    # ["/tmp/omnidoc_archive_xxx/file1.txt", ...]

# TAR / .tar.gz / .tar.bz2 / .tar.xz
result = load_document("backup.tar.gz")

# 7-Zip — requires [archive] extra
result = load_document("archive.7z")

# All archives enforce path-traversal (zip-slip) and 512 MB per-member limits
```

### Audio & Video (Transcription)

```python
# Audio — requires [media] extra (openai-whisper)
doc = load_document("podcast.mp3")
print(doc.metadata["language"])        # auto-detected language
for sec in doc.sections:
    print(sec.text)                    # one section per Whisper segment

# Video — audio extracted via ffmpeg, then transcribed
doc = load_document("meeting.mp4")
print(doc.raw_text)

# Custom model size and language
from omnidoc.extractors.media.extractor import MediaExtractor
doc = MediaExtractor(model="medium", language="en").extract("lecture.wav")
```

---

## Output Formats

### Document Object

The default Python dataclass output.

```python
from omnidoc.core.document import Document, Section, Table

doc: Document = load_document("report.pdf")

# Metadata
print(doc.metadata)
# {
#   "file": "report.pdf",
#   "pages": 12,
#   "source_type": "pdf",
#   "pipeline": "pdf",
#   "scanned": False,
# }

# Full text
print(doc.raw_text)

# Per-page sections
for section in doc.sections:
    print(f"  Page {section.page}: {section.text[:80]}")

# Tables
for table in doc.tables:
    print(f"  Page {table.page}: {table.headers}")
    for row in table.rows:
        print("    ", row)
```

### JSON Output

```python
from omnidoc.pdf.pipeline import extract_pdf

result = extract_pdf("report.pdf", output_format="json")

# result is a dict
print(result["metadata"])
print(result["sections"][0])
print(result["tables"])
print(result["raw_text"][:200])
```

### TOON Output

```python
result = extract_pdf("report.pdf", output_format="toon")

# TOON is Anthropic's agent-native structured format
print(result["type"])
print(result["content"])
```

---

## OCR Engines

### Tesseract (local)

```python
from omnidoc.ocr.tesseract import TesseractOCR

ocr = TesseractOCR(
    dpi=300,       # rendering DPI (higher = better accuracy, slower)
    timeout=60,    # per-page timeout in seconds (SIGALRM on Unix)
    lang="eng",    # Tesseract language code(s), e.g. "eng+fra"
)
```

### AWS Textract

```python
from omnidoc.ocr.aws_textract import TextractOCR

ocr = TextractOCR(
    region="us-east-1",
    timeout=120,
    max_retries=3,
    retry_backoff=2.0,
)
```

### Azure Form Recognizer

```python
from omnidoc.ocr.azure_form_recognizer import AzureFormRecognizer

ocr = AzureFormRecognizer(
    endpoint="https://my-resource.cognitiveservices.azure.com/",
    api_key="your-key",
    timeout=120,
    max_retries=3,
    model_id="prebuilt-read",   # or "prebuilt-layout" for tables
)
```

---

## RAG Pipeline

Semantic chunking, streaming, evaluation, graph linking, and vector DB adapters live in the companion package **`omnidoc-rag`**:

```bash
pip install omnidoc-rag
```

```python
from omnidoc.loader.load import load_document
from omnidoc_rag.chunker import chunk_document
from omnidoc_rag.evaluation import evaluate_rag_result
from omnidoc_rag.vectordb.chroma import ChromaAdapter

doc = load_document("investor_deck.pdf")

# Intent-aware semantic chunks
chunks = chunk_document(doc)
for c in chunks:
    print(f"[{c.intent}] {c.confidence:.2f} — {c.text[:100]}")

# Evaluate a retrieval result
score = evaluate_rag_result(
    query="What was the revenue growth?",
    answer="Revenue grew 24% YoY.",
    chunks=chunks,
)
print(score["overall"], score["verdict"])

# Upsert to ChromaDB
adapter = ChromaAdapter(collection_name="reports")
adapter.upsert(chunks)
```

See [omnidoc-rag documentation](https://github.com/your-org/omnidoc-rag) for the full API including streaming, graph linking, cross-document stitching, Pinecone/Weaviate/pgvector adapters, and LangChain/LlamaIndex integration.

---

## PII Masking

```python
from omnidoc.pdf.pipeline import extract_pdf

doc = extract_pdf("contract.pdf", enable_pii_masking=True)

# Names, emails, phone numbers, SSNs, credit cards replaced with <ENTITY_TYPE>
print(doc.raw_text)
# "The contract was signed by <PERSON> on behalf of <ORGANIZATION>."
```

Requires `pip install "omnidoc-sdk[privacy]"`.

---

## Enterprise Configuration

Override defaults via `OmnidocConfig` or environment variables.

```python
from omnidoc.config import OmnidocConfig

cfg = OmnidocConfig(
    max_file_mb=200,        # OMNIDOC_MAX_FILE_MB    (default 100)
    pdf_dpi=300,            # OMNIDOC_PDF_DPI        (default 250)
    ocr_timeout=120,        # OMNIDOC_OCR_TIMEOUT    (default 60)
    max_retries=5,          # OMNIDOC_MAX_RETRIES    (default 3)
    retry_backoff=2.0,      # OMNIDOC_RETRY_BACKOFF  (default 2.0)
    aws_region="us-east-1", # OMNIDOC_AWS_REGION
    enable_pii_masking=True,
    log_level="INFO",       # OMNIDOC_LOG_LEVEL
)

doc = load_document("large_report.pdf", config=cfg)
```

Or set environment variables before importing:

```bash
export OMNIDOC_MAX_FILE_MB=500
export OMNIDOC_OCR_TIMEOUT=180
export OMNIDOC_LOG_LEVEL=DEBUG
```

---

## Structured Logging

```python
from omnidoc.utils.logging_config import configure_logging

# Human-readable (default)
configure_logging(level="INFO")

# JSON logging for log aggregators (Datadog, CloudWatch, Splunk)
configure_logging(level="INFO", json=True)
```

JSON output example:

```json
{"ts": "2024-01-15T10:23:41.123Z", "level": "INFO", "logger": "omnidoc.loader.load", "msg": "load_document: path='report.pdf' size=1240.5KB"}
{"ts": "2024-01-15T10:23:42.456Z", "level": "DEBUG", "logger": "omnidoc.pdf.pipeline", "msg": "extract_pdf: 12 pages, scanned=False"}
```

---

## Retry & Resilience

Apply automatic retry to any function that calls a flaky external service.

```python
from omnidoc.utils.retry import with_retry

@with_retry(
    max_attempts=3,
    base_delay=1.0,
    backoff_factor=2.0,
    jitter=True,
    exceptions=(TimeoutError, ConnectionError),
)
def call_my_api(doc_bytes: bytes):
    ...
```

---

## CLI

```bash
# Extract any supported file — print JSON to stdout
omnidoc extract report.pdf

# With format control
omnidoc extract report.pdf --format json
omnidoc extract report.pdf --format toon

# With cloud OCR
omnidoc extract scanned.pdf --cloud-ocr aws

# Mask PII in output
omnidoc extract contract.pdf --pii

# Debug logging
omnidoc extract report.pdf --log-level DEBUG

# Extract non-PDF files
omnidoc extract data.xlsx
omnidoc extract notebook.ipynb
omnidoc extract message.eml
omnidoc extract podcast.mp3
```

---

## Optional Extras

Install only what you need. All heavy dependencies are loaded lazily at runtime.

<table>
<thead>
<tr><th>Extra</th><th>Install</th><th>Unlocks</th></tr>
</thead>
<tbody>
<tr><td><code>office</code></td><td><code>pip install "omnidoc-sdk[office]"</code></td><td><code>.docx</code>, <code>.xlsx</code>, <code>.xls</code>, <code>.pptx</code></td></tr>
<tr><td><code>data</code></td><td><code>pip install "omnidoc-sdk[data]"</code></td><td>pandas-backed CSV/TSV/XLSX with full type inference</td></tr>
<tr><td><code>docs</code></td><td><code>pip install "omnidoc-sdk[docs]"</code></td><td><code>.html</code>, <code>.htm</code>, <code>.svg</code>, <code>.xml</code></td></tr>
<tr><td><code>yaml</code></td><td><code>pip install "omnidoc-sdk[yaml]"</code></td><td><code>.yaml</code>, <code>.yml</code></td></tr>
<tr><td><code>ebook</code></td><td><code>pip install "omnidoc-sdk[ebook]"</code></td><td><code>.epub</code></td></tr>
<tr><td><code>openoffice</code></td><td><code>pip install "omnidoc-sdk[openoffice]"</code></td><td><code>.odt</code>, <code>.ods</code>, <code>.odp</code></td></tr>
<tr><td><code>legacy</code></td><td><code>pip install "omnidoc-sdk[legacy]"</code></td><td><code>.rtf</code></td></tr>
<tr><td><code>email</code></td><td><code>pip install "omnidoc-sdk[email]"</code></td><td><code>.msg</code> (Outlook) — <code>.eml</code> needs no extra</td></tr>
<tr><td><code>image</code></td><td><code>pip install "omnidoc-sdk[image]"</code></td><td>HEIC / HEIF image formats (iPhone photos)</td></tr>
<tr><td><code>code</code></td><td><code>pip install "omnidoc-sdk[code]"</code></td><td>Jupyter notebooks (<code>.ipynb</code>) + Pygments token count</td></tr>
<tr><td><code>archive</code></td><td><code>pip install "omnidoc-sdk[archive]"</code></td><td><code>.7z</code> archives via py7zr</td></tr>
<tr><td><code>media</code></td><td><code>pip install "omnidoc-sdk[media]"</code></td><td><code>.mp3</code>, <code>.mp4</code>, <code>.wav</code>, <code>.m4a</code>, <code>.mov</code>, <code>.avi</code>, <code>.mkv</code></td></tr>
<tr><td><code>cloud-ocr</code></td><td><code>pip install "omnidoc-sdk[cloud-ocr]"</code></td><td>AWS Textract + Azure Form Recognizer</td></tr>
<tr><td><code>layout</code></td><td><code>pip install "omnidoc-sdk[layout]"</code></td><td>Detectron2 layout ML for complex PDFs</td></tr>
<tr><td><code>vlm</code></td><td><code>pip install "omnidoc-sdk[vlm]"</code></td><td>Donut vision-language model for scanned PDFs</td></tr>
<tr><td><code>privacy</code></td><td><code>pip install "omnidoc-sdk[privacy]"</code></td><td>PII masking via Microsoft Presidio</td></tr>
<tr><td><code>all</code></td><td><code>pip install "omnidoc-sdk[all]"</code></td><td>Everything above</td></tr>
</tbody>
</table>

---

## Contributing & Development

### Setup

```bash
git clone https://github.com/your-org/omnidoc-python-sdk.git
cd omnidoc-python-sdk
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate

# Editable install with all extras
pip install -e ".[all]"

# Dev tools
pip install build twine pytest pytest-cov ruff black mypy
```

Verify:

```bash
python -c "import omnidoc; print('OK')"
omnidoc extract --help
```

**System dependencies:**

| OS | Command |
|----|---------|
| macOS | `brew install poppler tesseract` |
| Ubuntu/Debian | `apt-get install poppler-utils tesseract-ocr libgl1` |
| RHEL/CentOS | `yum install poppler-utils tesseract` |

---

### Project Structure

```
omnidoc-python-sdk/
├── omnidoc/
│   ├── __init__.py              # Public API
│   ├── config.py                # OmnidocConfig (env-var driven)
│   ├── version.py
│   ├── adapters/                # LangChain, LlamaIndex thin wrappers
│   ├── cli/                     # omnidoc CLI entry point
│   ├── core/                    # Document, Section, Table dataclasses
│   ├── extractors/
│   │   ├── archive/             # .zip / .tar / .7z
│   │   ├── code/                # .py / .java / .js / .ts / .go / .cpp / .rs
│   │   ├── ebook/               # .epub (ebooklib)
│   │   ├── email/               # .eml (stdlib) + .msg (extract-msg)
│   │   ├── image/               # .png / .jpg / .tiff / .webp / .heic …
│   │   ├── legacy/              # .rtf (striprtf)
│   │   ├── markdown/            # .md / .rst
│   │   ├── media/               # .mp3 / .mp4 / .wav (Whisper)
│   │   ├── notebook/            # .ipynb (nbformat)
│   │   ├── office/              # .docx / .xlsx / .pptx
│   │   ├── openoffice/          # .odt / .ods / .odp (odfpy)
│   │   ├── structured/          # .json / .xml / .yaml
│   │   ├── text/                # .txt
│   │   └── web/                 # .html / .htm / .svg (beautifulsoup4)
│   ├── layout/                  # Layout ML, block ordering
│   ├── loader/
│   │   ├── load.py              # Universal loader (40+ types)
│   │   ├── router.py            # detect_document_type()
│   │   └── types.py             # DocumentType enum
│   ├── ocr/                     # Tesseract, Textract, Azure engines
│   ├── pdf/                     # PDF pipeline, detector, normaliser
│   ├── postprocess/             # Slide normaliser, heading detection
│   ├── privacy/                 # PII masking
│   ├── tests/                   # Unit + smoke tests
│   └── utils/                   # text, retry, logging, serialize
├── testing/                     # Manual integration scripts
├── pyproject.toml
└── README.md
```

---

### Running Tests

```bash
# All tests
pytest omnidoc/tests/ -v

# Single file
pytest omnidoc/tests/test_text.py -v
pytest omnidoc/tests/test_new_extractors.py -v

# With coverage
pytest omnidoc/tests/ --cov=omnidoc --cov-report=term-missing

# Skip slow/cloud tests
pytest omnidoc/tests/ -v -m "not slow and not cloud"
```

**Which tests need which extras:**

| Test file | Required extra |
|-----------|---------------|
| `test_text.py` | none |
| `test_spreadsheet.py` | `data` |
| `test_pdf.py` | none (pdfplumber) |
| `test_archive.py` | none / `archive` for .7z |
| `test_image.py` | none (Pillow) |
| `test_docx.py` | `office` |
| `test_config.py` | none |
| `test_new_extractors.py` | `docs`, `code`, `ebook`, `openoffice` (skipped if absent) |

**Lint and type-check:**

```bash
ruff check omnidoc/
black --check omnidoc/
mypy omnidoc/ --ignore-missing-imports

# Auto-fix
black omnidoc/
ruff check omnidoc/ --fix
```

---

### Building & Publishing

**Build:**

```bash
rm -rf dist/ build/ omnidoc_sdk.egg-info/
python -m build
twine check dist/*
```

**Test on TestPyPI first:**

```bash
twine upload --repository testpypi dist/*

# Verify install
pip install \
  --index-url https://test.pypi.org/simple/ \
  --extra-index-url https://pypi.org/simple/ \
  omnidoc-sdk
```

**Publish to PyPI:**

```bash
twine upload dist/*
pip install omnidoc-sdk==0.4.0
```

**Credentials — `~/.pypirc`:**

```ini
[distutils]
index-servers = pypi testpypi

[testpypi]
repository = https://test.pypi.org/legacy/
username = __token__
password = pypi-YOUR_TEST_TOKEN

[pypi]
repository = https://upload.pypi.org/legacy/
username = __token__
password = pypi-YOUR_PROD_TOKEN
```

```bash
chmod 600 ~/.pypirc
```

**GitHub Actions CI/CD** — store `PYPI_API_TOKEN` and `TEST_PYPI_API_TOKEN` as repository secrets. The `.github/workflows/ci.yml` publishes to TestPyPI on merge to main and to PyPI on a version tag (`git tag v0.4.0 && git push origin v0.4.0`).

---

### Versioning

Version is defined once in `pyproject.toml`. Follow [Semantic Versioning](https://semver.org/):

| Change | Example | Bump |
|--------|---------|------|
| Bug fix | Fix path traversal guard | `0.4.0 → 0.4.1` |
| New feature | Add new extractor | `0.4.0 → 0.5.0` |
| Breaking change | Rename public API | `0.4.0 → 1.0.0` |

> PyPI does not allow re-uploading the same version. Always bump before rebuilding.

---

### Release Checklist

```
[ ] ruff check omnidoc/           — zero errors
[ ] black --check omnidoc/        — no formatting changes
[ ] pytest omnidoc/tests/ -v      — all pass
[ ] Version bumped in pyproject.toml
[ ] Changelog updated below
[ ] rm -rf dist/ && python -m build
[ ] twine check dist/*            — both artifacts PASSED
[ ] TestPyPI round-trip verified
[ ] twine upload dist/*           — production upload
[ ] git tag vX.Y.Z && git push origin vX.Y.Z
```

---

### Troubleshooting

**`HTTPError: 400 Bad Request — File already exists`** — PyPI does not allow overwriting. Bump version, rebuild, re-upload.

**`twine check` fails with "description failed to render"** — README contains unsupported HTML. Test with:

```bash
pip install readme-renderer[md]
python -m readme_renderer README.md -o /tmp/preview.html
```

**`403 Forbidden` on upload** — Token scope wrong (use *Entire account* for first upload), or 2FA not enabled on your PyPI account.

**`ModuleNotFoundError` for an optional dependency** — Install the relevant extra:

```bash
pip install "omnidoc-sdk[ebook,openoffice,email,legacy,yaml,media]"
```

---

## Changelog

### [0.4.0] — 2026-04-10

#### Added
- `omnidoc/config.py` — `OmnidocConfig` dataclass; all settings driven by `OMNIDOC_*` env vars
- `omnidoc/utils/retry.py` — `@with_retry` decorator with exponential backoff and jitter
- `omnidoc/utils/logging_config.py` — structured JSON or plain-text logging under `omnidoc.*`
- `omnidoc/ocr/azure_form_recognizer.py` — `AzureFormRecognizer` OCR engine (was missing)
- `omnidoc/extractors/web/html_extractor.py` — `HTMLExtractor` (bs4, per-block sections)
- `omnidoc/extractors/structured/yaml_extractor.py` — `YAMLExtractor`
- `omnidoc/extractors/email/eml_extractor.py` — `EMLExtractor` (stdlib, zero deps)
- `omnidoc/extractors/email/msg_extractor.py` — `MSGExtractor` (Outlook via extract-msg)
- `omnidoc/extractors/ebook/epub_extractor.py` — `EPUBExtractor` (ebooklib)
- `omnidoc/extractors/openoffice/extractor.py` — `OpenDocumentExtractor` (.odt/.ods/.odp)
- `omnidoc/extractors/legacy/rtf_extractor.py` — `RTFExtractor` (striprtf)
- `omnidoc/extractors/notebook/extractor.py` — `NotebookExtractor` (.ipynb)
- `omnidoc/extractors/code/extractor.py` — `CodeExtractor` (15 languages, block splitting)
- `omnidoc/extractors/media/extractor.py` — `MediaExtractor` (Whisper, ffmpeg)
- Full CI pipeline in `.github/workflows/ci.yml`

#### Changed
- `omnidoc/pdf/pipeline.py` — input validation, per-step error isolation, structured logging
- `omnidoc/loader/load.py` — 40+ types wired; validation, file-size check, config passthrough
- `omnidoc/ocr/aws_textract.py` — `@with_retry`, 10 MB limit enforced, correct confidence averaging
- `omnidoc/ocr/tesseract.py` — `SIGALRM`-based per-page timeout, `dpi`/`lang` constructor args
- `omnidoc/extractors/image/extractor.py` — corrupt-image detection, colour-mode normalisation
- `omnidoc/extractors/office/docx.py` — lazy import, corrupt-file guard, safe style access
- `omnidoc/extractors/office/pptx.py` — `is` identity check for title shape (was `==`, crashed)
- `omnidoc/extractors/office/spreadsheet.py` — multi-encoding CSV, `fillna("")`, `sections=` fix
- `omnidoc/extractors/archive/extractor.py` — zip-slip guard, 512 MB bomb limit, temp-dir cleanup
- `omnidoc/extractors/structured/json_xml.py` — XML text/tail content now extracted correctly
- `omnidoc/utils/serialize.py` — reuses `doc.chunks`; handles both dataclass and dict chunks
- RAG/streaming pipeline extracted to standalone `omnidoc-rag` package

#### Fixed
- `SpreadsheetExtractor` crashed with `TypeError` (missing `sections=` argument)
- `ArchiveExtractor` leaked temp directories on every call
- `ArchiveExtractor` vulnerable to zip-slip path traversal
- `TextExtractor` hard-coded UTF-8 causing `UnicodeDecodeError` on latin-1 files
- `PPTXExtractor` `AttributeError` when title placeholder absent
- `XMLExtractor` discarded all text node content
- `.gitignore` contained wrong AL/Dynamics-365 template

---

### [0.3.9] — 2025-12-01

Initial release: PDF extraction, Tesseract/Textract OCR, layout ML, semantic chunking, TOON/JSON output, LangChain/LlamaIndex adapters, PII masking, streaming, CLI. Supported: PDF, TXT, MD, DOCX, XLSX, CSV, PPTX, PNG, JPG, JSON, XML, ZIP.

---

<div align="center">

**OmniDoc Python SDK** &nbsp;·&nbsp; v0.4.0 &nbsp;·&nbsp; Apache 2.0 &nbsp;·&nbsp; 40+ document formats &nbsp;·&nbsp; RAG pipeline → [omnidoc-rag](https://github.com/your-org/omnidoc-rag)

</div>
