Metadata-Version: 2.5
Name: paradox2
Version: 0.2.6
Summary: Fast, simple document extraction — from-scratch rewrite of paradox_pdf
Project-URL: Repository, https://github.com/CreAI-mx/paradox-v2
Project-URL: Issues, https://github.com/CreAI-mx/paradox-v2/issues
Author-email: CreAI <feliperodriguez@creai.mx>
License: Proprietary
License-File: LICENSE
Keywords: document,extraction,nlp,ocr,parsing,pdf,structured,table
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Markup
Requires-Python: >=3.11
Requires-Dist: numpy<3.0,>=1.24
Requires-Dist: opencv-python-headless<5.0,>=4.8
Requires-Dist: pymupdf<2.0,>=1.24
Provides-Extra: azure
Requires-Dist: requests<3.0,>=2.31; extra == 'azure'
Provides-Extra: formats
Requires-Dist: extract-msg<1.0,>=0.45; extra == 'formats'
Requires-Dist: odfpy<2.0,>=1.4; extra == 'formats'
Requires-Dist: openpyxl<4.0,>=3.1; extra == 'formats'
Requires-Dist: py7zr<1.0,>=0.20; extra == 'formats'
Requires-Dist: python-docx<2.0,>=1.0; extra == 'formats'
Requires-Dist: python-pptx<1.0,>=0.6.21; extra == 'formats'
Requires-Dist: pyxlsb<2.0,>=1.0; extra == 'formats'
Requires-Dist: rarfile<5.0,>=4.0; extra == 'formats'
Requires-Dist: striprtf<1.1,>=0.0.26; extra == 'formats'
Requires-Dist: xlrd<3.0,>=2.0; extra == 'formats'
Provides-Extra: gpu
Requires-Dist: paddleocr[doc-parser]>=3.0; extra == 'gpu'
Requires-Dist: paddlepaddle==3.2.1; extra == 'gpu'
Requires-Dist: paddlex<4.0,>=3.4; extra == 'gpu'
Requires-Dist: pillow<12.0,>=10.0; extra == 'gpu'
Provides-Extra: rapidai
Requires-Dist: lineless-table-rec<1.0,>=0.1; extra == 'rapidai'
Requires-Dist: rapidocr-onnxruntime<2.0,>=1.4; extra == 'rapidai'
Requires-Dist: table-cls<2.0,>=1.2; extra == 'rapidai'
Requires-Dist: wired-table-rec<2.0,>=1.2; extra == 'rapidai'
Provides-Extra: specialists
Requires-Dist: langdetect<2.0,>=1.0; extra == 'specialists'
Requires-Dist: pillow<12.0,>=10.0; extra == 'specialists'
Requires-Dist: pix2tex>=0.0.27; extra == 'specialists'
Requires-Dist: sentencepiece<1.0,>=0.1.99; extra == 'specialists'
Requires-Dist: torch<3.0,>=2.0; extra == 'specialists'
Requires-Dist: torchvision<1.0,>=0.15; extra == 'specialists'
Requires-Dist: transformers<5.0,>=4.30; extra == 'specialists'
Provides-Extra: yolo
Requires-Dist: huggingface-hub<1.0,>=0.20; extra == 'yolo'
Requires-Dist: ultralytics<9.0,>=8.0; extra == 'yolo'
Description-Content-Type: text/markdown

# paradox2

Fast, simple document extraction — a from-scratch rewrite of [paradox-pdf](https://pypi.org/project/paradox-pdf/), built around one idea: **most PDFs are digital and don't need a GPU.**

```python
import paradox2 as pdx

pages = pdx.extract("invoice.pdf")
```

## Why this rewrite exists

paradox-pdf grew into a 2,783-line facade doing routing, backend resolution,
and overlay rendering all in one file. paradox2 starts over with a hard rule:
**digital pages never pay for OCR or vision models.** Every PDF page is
classified independently — a digital page (has a text layer) goes through a
zero-ML fast path (PyMuPDF text + vector-line table detection); a scanned
page routes to the GPU OCR pipeline. A single mixed document is handled
correctly per-page, automatically.

## Install

```bash
pip install paradox2
```

Everything above works with just that — digital PDFs, text, vector-line
tables, key-value fields. Heavier features are opt-in extras so the base
install stays small:

| Extra | Adds | When you need it |
|---|---|---|
| `paradox2[gpu]` | PaddleOCR + PP-DocLayoutV2 | Scanned/photographed pages |
| `paradox2[rapidai]` | TableStructureRec (RapidAI), ONNX-only | Mid-tier table structure for scanned pages the OCR-grid heuristic rejects — no torch/torchvision, CPU-only, ~5.9s/table |
| `paradox2[yolo]` | YOLO26-document-layout | Independent table-bbox detector — crops the page before RapidAI/VLM instead of feeding them the whole page. Needed for the router's full accuracy; see below |
| `paradox2[formats]` | docx/xlsx/pptx/msg/rtf/7z/rar/odf readers | Non-PDF documents |
| `paradox2[specialists]` | torch + transformers (handwriting, signatures, formulas) | Handwriting (TrOCR), signature detection (Conditional-DETR), formula-to-LaTeX (pix2tex) |

## First-time setup (scanned/OCR support)

For scanned or photographed pages, run `paradox2-setup` right after
installing the base package — it detects whether the machine has an NVIDIA
GPU and installs the matching extras and the correct pinned `paddlepaddle`
build for you:

```bash
pip install paradox2
paradox2-setup
```

This installs `paradox2[gpu,rapidai,yolo,formats]` plus `paddlepaddle`
pinned to `3.2.1` (GPU build from PaddlePaddle's own index if a GPU is
detected, plain CPU build otherwise). Pinning matters: `paddlepaddle
3.3.1` — pip's unpinned default — has a real PIR/oneDNN inference bug;
`3.2.1` is the verified-good version.

If you'd rather install extras manually instead of running the script:

```bash
pip install "paradox2[gpu,rapidai,formats]"
```

**Use a clean virtual environment for `[gpu]`.** Installing into a shared/base
environment (e.g. Anaconda's `base`) that already has an older `paddleocr`
or `paddlepaddle` from a prior project can leave an incompatible version in
place — pip does not always resolve this cleanly against pre-existing
packages in a polluted environment. `python -m venv .venv && source
.venv/bin/activate` (or `conda create -n paradox2 python=3.12`) before
installing avoids this.

## What it does

```python
import paradox2 as pdx
from dataclasses import asdict

# Every page, as a list of PageResult dataclasses (NOT dicts — a page
# doesn't support page["blocks"]; use page.blocks, or asdict(page) /
# the CLI's --format json for a plain-dict/JSON form) — digital pages
# via the fast path, scanned pages via GPU OCR, decided per page.
pages = pdx.extract("document.pdf")
page_dicts = [asdict(p) for p in pages]  # if you want plain dicts

# Just the text
text = pdx.extract_text("document.pdf")

# Just the tables (vector-line detection on digital pages,
# OCR-grid heuristic + optional RapidAI/VLM fallback on scanned pages)
tables = pdx.extract_tables("document.pdf")

# Key-value pairs from an invoice/form-like page (digital only)
fields = pdx.extract_kie("invoice.pdf")

# Any non-PDF format too — same call, same output shape
data = pdx.extract("spreadsheet.xlsx")
data = pdx.extract("scan.docx")

# Before processing a scanned document: check whether it's actually
# processable with what's currently installed, without running OCR.
report = pdx.can_process("document.pdf")
# {"n_pages": 12, "n_scanned_pages": 12, "needs_ocr": True,
#  "can_process": False, "problems": [...]}
```

`page_workers > 1` (multi-page process-pool parallelism) uses
`multiprocessing`'s `spawn` start method — a caller script MUST guard its
top-level code with `if __name__ == "__main__":`, or the worker processes
die on import and every page silently falls back to sequential processing
(you'll see a `RuntimeWarning` when this happens).

Each worker loads its own full OCR model set (~2.4GB RSS measured) —
`page_workers` is automatically capped against both available CPU cores
(respecting `taskset`/cgroup/Docker `--cpus` limits, not just total system
cores) and available RAM, so a request for more workers than the machine
can actually hold gets a smaller worker count instead of an OOM kill.

### Selecting pages, features, and output format

```python
pdx.extract("report.pdf", pages="1-3,7")           # specific pages, same list[PageResult] shape
pdx.extract("report.pdf", output_format="markdown") # str, not list[PageResult]
pdx.extract("report.pdf", fields=True)               # turn on key-value extraction inline, still list[PageResult]
```

**`feature=` changes the return type**, unlike the calls above - real
documentation gap found 2026-09-24 (this used to say "still full
extract() under the hood" without mentioning that): the full pipeline
does still run internally, but the RETURN VALUE is projected down to
`list[dict]` (or a single value for a scalar feature), not
`list[PageResult]`. Code written against `page.blocks` breaks with
`AttributeError: 'dict' object has no attribute 'blocks'` if `feature=`
is added later without updating the calling code.

```python
pdx.extract("report.pdf", feature="tables")          # list[dict], not list[PageResult]
pdx.extract("report.pdf", feature="text")             # list[str]
```

`extract()` normally returns `list[PageResult]` (dataclasses - see the
first code example above for how to get plain dicts via `dataclasses.
asdict`); only `feature=` and `output_format=` change that.

### Optional specialists (all off by default, all lazy-loaded)

```python
pdx.extract("form.pdf", handwriting=True)   # TrOCR on handwritten blocks
pdx.extract("contract.pdf", signatures=True) # Conditional-DETR signature boxes
pdx.extract("paper.pdf", formulas=True)      # inline math -> LaTeX
```

None of these import their heavy dependencies unless the flag is set —
`import paradox2` alone never touches torch, paddle, or transformers.

## The scanned-table router

Scanned pages route tables through up to five tiers, each opt-in past the
first (real gap found and fixed in the README 2026-09-24: this used to
describe only 3 of the 5, and described RapidAI as the standing mid-tier
when PP-StructureV3 has actually taken that slot by default since
2026-09-24):

```
OCR-grid heuristic (free, always on)
        |  rejects merged/borderless/dense tables by design,
        |  or under-reads a real table (low_confidence flag)
        v
YOLO26 bbox (PARADOX2_YOLO_DETECT=1) -- crop the page to the detected
        |  table region before handing it to the tiers below. A
        |  borderline-confidence detection (<0.85) skips the mid-tier
        |  entirely and goes straight to the VLM tier - VLM tolerates
        |  a loose/imprecise crop better than a structure model does.
        |  A table covering >55% of the page area also skips straight
        |  to VLM (large tables measured harder in this session).
        v
PP-StructureV3 (PARADOX2_PPSTRUCTURE_TABLES=1, the default occupant of
        |  this slot) OR RapidAI (PARADOX2_RAPIDAI_TABLES=1, used only
        |  when PP-StructureV3 is disabled or has no GPU/CPU headroom -
        |  see paradox2.doctor()/resource_detect for that check). Either
        |  one's result is cross-checked against OCR line density in the
        |  same crop - implausibly few OR implausibly many rows re-route
        |  to the VLM tier
        v
VLM fallback (PARADOX2_VLM_TABLES=1)
        |  PaddleOCR-VL single-pass, reserved for tables that look
        |  genuinely complex - not run on everything the mid-tier touches
        v
Azure Document Intelligence (PARADOX2_AZURE_DI_TABLES=1 AND
        |  AZURE_DI_ENDPOINT/AZURE_DI_KEY both set - double opt-in,
        |  never called by default). PAID, external API - only reaches
        |  this tier if every free tier above still flags the result as
        |  implausible. Measured 86.5% mean accuracy across 12 tables,
        |  97.4% on the one 32-row table where the three local engines
        |  scored 0.9-4.7% - but total failure on the 3 simplest tables,
        |  hence last resort rather than first choice.
        v
   best available result
```

Timeouts for each network/GPU-bound tier are configurable
(`PARADOX2_{RAPIDAI,VLM,YOLO,PPSTRUCTURE}_TIMEOUT_S`,
`PARADOX2_PPSTRUCTURE_CPU_TIMEOUT_S`), as is the CPU-core floor below
which PP-StructureV3 is skipped entirely
(`PARADOX2_PPSTRUCTURE_MIN_CPU_CORES`) and the per-machine state cache
directory (`PARADOX2_STATE_DIR`) — see `.env.example` for every variable
with its default and rationale.

Every tier is measured end-to-end (`paradox2.extract()`, not the isolated
engine wrapper) on the same 20 real full-page tables (OmniDocBench), not
simulated. **This table predates PP-StructureV3 becoming the default
mid-tier occupant (2026-09-24)** - it measures the RapidAI-as-mid-tier
chain only; the 35%/80% numbers are NOT what the current default
(PARADOX2_PPSTRUCTURE_TABLES=1) produces, and haven't been re-measured
against it yet:

| config | exact row match | within +/-3 rows | mean time/page |
|---|---|---|---|
| heuristic only (pre-router) | 5% | 15% | 0.8s |
| + RapidAI, full page (bug, fixed) | 15% | 50% | 6.0s |
| + RapidAI, cropped to heuristic's own bbox (bug, fixed - made 3/20 pages *worse*) | 15% | 40% | 4.3s |
| + RapidAI, cropped to YOLO26 bbox | 25% | 75% | 5.5s |
| + selective VLM escalation (low YOLO confidence or implausible RapidAI row count) | **35%** | **80%** | 6.4s |

The isolated RapidAI engine alone scores 65% exact on tight ground-truth
crops — the gap to the router's end-to-end number is the YOLO26 crop's
margin versus a perfect crop, not a RapidAI accuracy problem. Cropping to
the heuristic's *own* bbox instead of an independent detector actively
hurts: that bbox only spans the rows the heuristic already found, so it
bakes its own under-read into the pixels before the next tier ever sees
them. A same-crop self-consistency check (RapidAI's row count vs. OCR
line density) can't catch a loose crop itself, since both numbers inflate
together on the same imprecise crop — that's what the YOLO-confidence
trigger (<0.85) is for instead.

The heuristic never gets replaced by a worse result — a lower tier's
output is kept unless a higher tier actually produces something.

```bash
export PARADOX2_YOLO_DETECT=1
export PARADOX2_RAPIDAI_TABLES=1
export PARADOX2_VLM_TABLES=1
```

## Health check

```python
from paradox2 import doctor
doctor()  # {"cuda": True/False, "pymupdf": "1.24.x", "ok": True/False, "problems": [...], ...}
```

```bash
paradox2 doctor          # full report
paradox2 doctor --fix    # prints only the pip install command(s) needed, exit 1 if any
```

**First OCR call downloads model weights** (PP-OCRv5, no progress bar) —
a first `extract()` on a scanned page that appears to hang for 10-30s on a
slow connection is this, not a crash. **Backend stderr noise** (onnxruntime/
TensorFlow/cuDNN registration warnings) is upstream noise from paddleocr's
own dependencies, not paradox2 — there is currently no flag to suppress it.

## Design principles

- **Digital-first**: a PDF with a text layer never pays for OCR, vision
  layout models, or GPU inference — verified per-page, not per-document.
- **Facade stays thin**: `paradox2/api.py` is dispatch only; business logic
  lives in `pipeline/`, `tables/`, `engines/`, `formats/`. If the facade
  creeps toward 100+ lines, that's a signal something leaked out of place.
- **Specialists are lazy and swallow their own failures**: an optional
  engine (handwriting, signatures, formulas, RapidAI, VLM) that fails to
  load or errors mid-call returns `[]`/`None` — it never takes down the
  base extraction path.
- **No simulated benchmarks**: every accuracy/speed number in this README
  and in the codebase's docstrings comes from a real run against real
  documents, with the script path noted alongside it.

## Status

Early-stage rewrite (Alpha) - the API and table router are still
stabilizing. Development docs referenced in some docstrings
(`docs/PLAN.md`, `docs/AGENT_SYNC.md`, `docs/TASKS.md`) are internal
working notes, not shipped with the package - real gap found 2026-09-24:
those references are dead ends for anyone who only has the installed
package, not the source repo. See the
[repository](https://github.com/CreAI-mx/paradox-v2) for current source
and issues.
