Metadata-Version: 2.4
Name: docvion
Version: 0.1.1
Summary: A canonical, engine-agnostic schema and adapter suite for parsed documents
Author-email: "Prolixis (OPC) Pvt Ltd" <engineering@prolixis.ai>
License-Expression: MIT
Project-URL: Homepage, https://github.com/prolixis/docvion
Project-URL: Repository, https://github.com/prolixis/docvion
Project-URL: Issues, https://github.com/prolixis/docvion/issues
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.0
Requires-Dist: psutil>=5.9.0
Provides-Extra: docling
Requires-Dist: docling>=2.0; extra == "docling"
Provides-Extra: tesseract
Requires-Dist: pytesseract; extra == "tesseract"
Requires-Dist: Pillow; extra == "tesseract"
Provides-Extra: paddleocr
Requires-Dist: paddleocr; extra == "paddleocr"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Provides-Extra: all
Requires-Dist: docling>=2.0; extra == "all"
Requires-Dist: pytesseract; extra == "all"
Requires-Dist: Pillow; extra == "all"
Requires-Dist: paddleocr; extra == "all"
Dynamic: license-file

# Docvion 📄⚡

> **One schema for every document parser.**

[![PyPI version](https://img.shields.io/pypi/v/docvion.svg?color=blue)](https://pypi.org/project/docvion/)
[![Docvion CI](https://github.com/prolixis-ai/docvion/actions/workflows/ci.yml/badge.svg)](https://github.com/prolixis-ai/docvion/actions)
[![Python Version](https://img.shields.io/badge/python-3.9%2B-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
[![Code Style: Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

---

## 1. Problem Statement

Every document-parsing engine (**Docling**, **Tesseract**, **PaddleOCR**, **Azure Document Intelligence**, **Google Document AI**, **AWS Textract**) returns output in its **own incompatible shape**. A developer building a RAG pipeline or memory ingestion engine against one parser's native output format is locked in. Switching parsers, comparing engine accuracy, or combining outputs from multiple engines requires rewriting downstream code every single time.

There has been no neutral, engine-agnostic representation of a parsed document — until now.

## 2. Solution: Before & After

`docvion` is a **schema-first Python library** created by [Prolixis](https://prolixis.ai) that defines one canonical representation of a parsed document — `DocvionDocument` — plus a suite of zero-overhead **adapters** that convert any supported parser's output into that canonical form.

### Before Docvion (Parser Lock-In)
```python
# Locked to Docling's specific dictionary layout:
def build_llm_prompt(docling_json):
    texts = docling_json["texts"]
    prompt = ""
    for item in texts:
        if item["label"] == "title":
            prompt += f"# {item['text']}\n"
        elif item["label"] == "paragraph":
            prompt += f"{item['text']}\n"
    return prompt
```

### After Docvion (Engine-Agnostic Downstream Pipeline)
```python
from docvion import DocvionDocument, convert

# Downstream code relies strictly on DocvionDocument. Swap engines freely!
doc: DocvionDocument = convert(engine="docling", raw_output=raw_data)
# Or swap to Tesseract / PaddleOCR without changing 1 line of downstream code:
# doc: DocvionDocument = convert(engine="tesseract", raw_output=tesseract_data)

prompt_context = doc.export_to_markdown()
json_payload = doc.export_to_json()
```

---

## 3. Landscape Comparison: Docvion vs. Alternatives

Docvion is **not** an OCR engine or an ETL platform—it is a **neutral schema specification and adapter layer**. Here is how Docvion compares to existing tools:

| Dimension | **Docvion** | **Unstructured.io** | **Raw Engine (e.g. Docling)** | **LlamaParse** |
| :--- | :--- | :--- | :--- | :--- |
| **Primary Architecture** | **Schema-First Neutral Layer** | Document ETL & Partition Suite | Specific Parser/OCR Model | Cloud Parsing API |
| **Schema Portability** | **High**: Single canonical `DocvionDocument` across all engines | **Medium**: Custom `Element` list schema | **None**: Locked to engine's native JSON output | **None**: Locked to LlamaParse proprietary JSON |
| **Engine Lock-In** | **Zero**: Swap Docling, Tesseract, PaddleOCR freely | **Low/Medium**: Tied to Unstructured partitioners | **100% Locked**: Downstream code tightly coupled | **100% Locked**: Coupled to LlamaCloud service |
| **Execution Environment** | **100% Local / Air-Gapped** | Hybrid (Local & Cloud options) | 100% Local / Local Models | Cloud API Required |
| **Core Base Weight** | **Ultra-Light** (`pydantic` & `psutil` only) | Heavy (Pulls torch, transformers, poppler) | Heavy (ML model dependencies) | Light (API Client wrapper) |
| **Cost & License** | **100% Free & Open Source (MIT)** | Freemium (Usage-based Cloud API) | Free & Open Source | Paid API (per-page credit cost) |
| **Downstream Pipeline Reusability** | **Build Once**: Write downstream code once against `DocvionDocument` | Medium: Custom element processing | Low: Rewrite downstream code per parser | Low: Tied to LlamaIndex ecosystem |

---

## 4. Quickstart

```python
from docvion import DocvionDocument, convert

# Option A: Bring your own engine output
doc = convert(engine="docling", raw_output=my_docling_result)

# Export clean, structured Markdown for LLM prompt context
markdown_text = doc.export_to_markdown()

# Export full-fidelity JSON (bboxes, confidence, table cells)
json_payload = doc.export_to_json()
```

Convenience one-liner that executes the engine and converts in a single call:

```python
from docvion import parse

# Parse invoice and get canonical schema immediately
doc = parse("invoice.pdf", engine="docling")
print(doc.export_to_markdown())
```

---

## 5. Structure-Aware Chunking

Most chunkers split by raw token count, which routinely breaks tables mid-row and cuts sentences in half. Docvion's `SemanticChunker` uses the document's own structure — block types and reading order — to make chunking decisions:

- **Tables are never split** across chunks (atomic table blocks)
- **Sentences are never split** across chunks
- **Headings attach to subsequent content**, propagating `heading_context` to every chunk for context reconstruction during retrieval

```python
from docvion import parse, SemanticChunker

doc = parse("report.pdf", engine="docling")
doc.chunks = SemanticChunker(target_tokens=512).chunk(doc)

for chunk in doc.chunks:
    print(chunk.heading_context, "->", chunk.token_count, "tokens")
```

> **Note on token counting:** Docvion uses a fast character-based approximation (~4 chars/token) for chunk sizing, not an exact tokenizer for any specific LLM. This is intentional — it keeps the core package dependency-light. For exact token counts matching a specific model, count tokens post-export using that model's own tokenizer.

---

## 6. Supported Engines & Confidence Calibration

| Engine | Native Features | Docvion Adapter | Per-Block Confidence Calibration |
| :--- | :--- | :---: | :--- |
| **Docling** | Layout, Tables, Bounding Boxes, Metadata | ✅ | **Full**: Native block-level confidence scores passed directly (`0.0–1.0`). |
| **Tesseract** | Text & Bounding Boxes (`image_to_data`) | ✅ | **Averaged**: Calculates mean confidence across word boxes in grouped paragraph. |
| **PaddleOCR** | Layout, PP-Structure HTML Tables | ✅ | **Full**: Region detection score & text recognition confidence. |
| *Azure Doc AI* | *Planned v0.2* | 🔄 | Coming soon |
| *AWS Textract* | *Planned v0.2* | 🔄 | Coming soon |

> [!NOTE]
> **Transparent Limitation Note on Confidence Calibration**:
> Engines handle confidence differently. Docling and PaddleOCR supply block/region confidence natively. Tesseract provides word-level confidence (`conf` between 0–100); Docvion averages valid word confidences across grouped lines into `Block.confidence`. Docvion never fabricates confidence scores—if an engine provides none, `confidence` remains `None`.

---

## 7. Core Schema (`DocvionDocument`)

All models are built on **Pydantic v2**, giving instant JSON serialization, strict validation, and full IDE autocomplete.

Key design principles:
- **Resolution-Independent Bounding Boxes**: Bounding boxes (`BoundingBox`) are normalized to `0.0–1.0` relative coordinates (`x0, y0, x1, y1`) and indexed by 1-based page number.
- **Flat Table Representation**: Tables (`Table`) are stored as flat cell lists (`TableCell`) with zero-based `row`, `col`, `row_span`, and `col_span` indices. Handles merged cells and irregular grids cleanly.
- **Multi-Page Tables**: Cells preserve global row/column indexing across page boundaries, preventing table fragmentation in multi-page documents.

```python
class BlockType(str, Enum):
    TITLE = "title"
    HEADING = "heading"
    PARAGRAPH = "paragraph"
    TABLE = "table"
    IMAGE = "image"
    FIGURE = "figure"
    LIST_ITEM = "list_item"
    CAPTION = "caption"
    HEADER = "header"
    FOOTER = "footer"
    FOOTNOTE = "footnote"
    OTHER = "other"
```

---

## 8. Installation

The core `docvion` package is lightweight and depends **only on `pydantic>=2.0` and `psutil`**. Heavy ML engine dependencies are kept optional so base installation takes seconds.

```bash
# Core package (zero heavy dependencies)
pip install docvion

# Optional engine extras (install only what you need):
pip install "docvion[docling]"     # Pulls docling
pip install "docvion[tesseract]"   # Pulls pytesseract & Pillow
pip install "docvion[paddleocr]"   # Pulls paddleocr
pip install "docvion[all]"         # Pulls all supported engines
```

---

## 9. Command Line Interface (CLI)

Docvion comes with a CLI for converting documents straight from your shell:

```bash
# Convert PDF using Docling into Markdown
docvion convert invoice.pdf --engine docling --format markdown

# Benchmark parsing latency, memory, and schema completeness across engines
docvion benchmark invoice.pdf --engines docling,tesseract,paddleocr --format markdown

# Check installed engine dependencies
docvion doctor
```

---

## 10. Runnable Examples

Check out the included scripts in `examples/`:

- [`examples/basic_usage.py`](examples/basic_usage.py): Quickstart demonstration converting raw outputs and exporting to Markdown/JSON/Dict.
- [`examples/engine_swap_demo.py`](examples/engine_swap_demo.py): Swap between Docling, Tesseract, and PaddleOCR without modifying downstream LLM prompt code.
- [`examples/chunking_demo.py`](examples/chunking_demo.py): Structure-aware chunking with table preservation and heading context.
- [`examples/benchmark_usage.py`](examples/benchmark_usage.py): Benchmark execution and report generation.

Run an example:
```bash
python examples/basic_usage.py
python examples/engine_swap_demo.py
python examples/chunking_demo.py
python examples/benchmark_usage.py
```

---

## 11. Third-Party Notices & Licenses

See [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md) for full licensing and attribution information for Docling, Tesseract OCR, PaddleOCR, and Pydantic.

---

## 12. Positioning: Docvion & Contivon

Docvion is **infrastructure for structured memory**.
- **Docvion**: Turns any document into a clean, engine-agnostic structure.
- **Contivon**: Turns that structure into governed, retrievable memory.

---

## 13. Development & Testing

Run unit & stress tests:
```bash
python -m unittest discover tests
```

---

## 14. License & Company

Maintained by **Prolixis (OPC) Pvt Ltd**.
Released under the [MIT License](LICENSE).
