Metadata-Version: 2.4
Name: mag-pdf
Version: 0.3.0
Summary: PDF extraction that OCRs only the pages that need it - a gated LiteParse wrapper
Author-email: Magure <aman.p@magureinc.com>
Maintainer-email: Magure <aman.p@magureinc.com>
License: Apache-2.0
Project-URL: Homepage, https://github.com/Magure-Tech/magoneai-file-handler
Project-URL: Source, https://github.com/Magure-Tech/magoneai-file-handler
Project-URL: Issues, https://github.com/Magure-Tech/magoneai-file-handler/issues
Project-URL: Changelog, https://github.com/Magure-Tech/magoneai-file-handler/blob/main/magpdf/CHANGELOG.md
Keywords: pdf,ocr,liteparse,extraction,tables,markdown
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: liteparse<3,>=2.11
Provides-Extra: tables
Requires-Dist: docling-ibm-models[opencv-python-headless]<4,>=3.13; extra == "tables"
Requires-Dist: pypdfium2<6,>=4; extra == "tables"
Requires-Dist: onnxruntime<2,>=1.16; extra == "tables"
Requires-Dist: huggingface-hub<2,>=0.20; extra == "tables"
Requires-Dist: numpy<3,>=1.24; extra == "tables"
Requires-Dist: pillow>=10; extra == "tables"
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Requires-Dist: pillow>=10; extra == "test"
Requires-Dist: numpy<3,>=1.24; extra == "test"
Provides-Extra: tables-test
Requires-Dist: mag-pdf[tables]; extra == "tables-test"
Requires-Dist: reportlab>=4; extra == "tables-test"
Requires-Dist: pytest>=7; extra == "tables-test"
Provides-Extra: serve
Requires-Dist: fastapi<1,>=0.110; extra == "serve"
Requires-Dist: uvicorn[standard]<1,>=0.27; extra == "serve"
Requires-Dist: python-multipart>=0.0.9; extra == "serve"
Provides-Extra: serve-test
Requires-Dist: mag-pdf[serve]; extra == "serve-test"
Requires-Dist: httpx>=0.27; extra == "serve-test"
Requires-Dist: pytest>=7; extra == "serve-test"
Dynamic: license-file

# mag-pdf

PDF extraction with **two profiles and one result shape**.

| | `fast` (default) | `tables` |
|---|---|---|
| What | [LiteParse](https://github.com/run-llama/liteparse) behind an OCR gate | DocLayout-YOLO + TableFormer over the pypdfium2 text layer |
| For | most documents; scanned pages via Tesseract | documents whose value is in their tables |
| Install | `pip install mag-pdf` (one dependency) | `pip install 'mag-pdf[tables]'` |
| Cost | ~100-200 ms with no OCR needed | ~0 for a plain page; a model only where the gate says so |

```python
from magpdf import extract, warmup

warmup()                       # optional, ~160 ms, at worker startup
doc = extract(pdf_bytes)

doc.markdown                   # whole document, one unified string
doc.pages                      # per-page, 1-indexed page_number
doc.ocr_pages                  # which pages OCR actually ran on
doc.timings                    # {"gate_ms", "text_ms", "ocr_ms", "total_ms"}
```

Both profiles return the same `Document`, so a caller can switch with one
keyword and read the result the same way.

## Why

OCR costs ~0.2-1.3 s **per page** and has no warm-up to amortise (a reused
parser measured 5,797 ms against a fresh one's 5,864 ms). So the only lever on
latency is not OCRing a page at all.

`magpdf` reads LiteParse's own per-page signals in 6-105 ms and OCRs a page only
when it can add something the text layer cannot:

| Trigger | Fires when | Why it earns its cost |
|---|---|---|
| `image` | page carries substantial raster content | recovers tables and text rendered **inside images**. On one corpus page a table went from **0 markdown pipes to 16** |
| `dead_text` | text layer garbled or near-empty | catches true scans, and fonts subset without ToUnicode CMaps, where a text layer exists but decodes to bullet glyphs. One corpus file: **160 chars to 7,315** |
| `table` (**off**) | table detected in the text layer | measured across 53 such pages: **zero** change to table structure for ~72 s of OCR |

Over a 232-page corpus that OCRs **55% of pages instead of 100%, with identical
output**.

## Latency

A 10-page PDF, warm. OCR page count is the only variable that matters:

| Pages needing OCR | Total |
|---|---|
| 0 | **~100-200 ms** |
| ~5 | ~5-6 s |
| 10 | ~13 s |

The gate itself is under 1% of any run that OCRs.

> **Running one document at a time?** Set `num_workers=4`. The default of `1` is
> tuned for concurrent Temporal activities and is **2.8x slower** on OCR-heavy
> documents (14,779 ms vs 5,318 ms on a 4-page all-OCR file). See
> [`docs/KNOWLEDGE_HANDOFF.md`](docs/KNOWLEDGE_HANDOFF.md) section 5.

## Configuration

```python
from magpdf import ExtractConfig, OcrTriggers, extract

doc = extract(pdf_bytes, config=ExtractConfig(
    triggers=OcrTriggers(image=True, dead_text=True, table=False),
    dead_text_coverage=0.02,
    max_ocr_pages=100,      # cap; dead_text pages win, rest reported as skipped
    num_workers=1,
    ocr_enabled=True,       # False = text layer only
))
```

There are **no document-class heuristics** in the gate. It reads general
per-page signals and applies them uniformly to every PDF. Retuning is config,
never a code change.

## Vision escalation (optional)

A page that carried an image and *still* came back near-empty is where Tesseract
most plausibly failed. Inject a client and those pages get a second look:

```python
doc = extract(pdf_bytes, vision_client=my_client)
doc.escalated_pages          # [3]
```

The client only needs `ocr_image(image_bytes, media_type, *, prompt=None,
max_tokens=...) -> (text, usage)`. That is a structural match for
`mag-file-handler`'s `VisionClient`, so an existing client works as-is - but
neither package imports the other.

**Without a client, magpdf is fully offline, deterministic, and costs no
tokens.**

---

## The `tables` profile

```bash
pip install 'mag-pdf[tables]'
```

```python
from magpdf import extract, warmup

warmup(profile="tables")               # loads both models; seconds, not ms
doc = extract(pdf_bytes, profile="tables")

doc.markdown                           # one unified, ordered document
doc.table_pages                        # pages TableFormer actually ran on
doc.text_pages                         # pages served free from the text layer
doc.warnings                           # non-fatal degradations, in the result
```

Per page, once: render, ask a layout model what is on the page, and pay for
TableFormer only where it is worth it. A plain digital page is served from the
pdfium text layer for **zero** model time.

```
page ─ render once @2.0 ─┬─ YOLO ─→ table? figure? sparse text?
                         │              │
                         │        ┌─────┴──────┬────────────┐
                         │      table       figure       neither
                         │        │            │            │
                         │   TableFormer   vision (if a   pdfium
                         │                  client)       text layer
                         └─ text cells ────→ tokens for cell matching
```

**It does not use docling.** `docling_ibm_models` never imports it, so the two
models are called directly. That avoids monkey-patching docling's private
layout-engine factory, which is what the pipeline this ports has to do, and it
means no `DocumentConverter`, no temp file, and no pinned orchestration layer.

### Output

Clean GitHub-flavoured markdown. Tables become pipe tables, titles become `#`
headings, and blocks are emitted in reading order so a caption stays with its
table. Pages are joined in order into one document — `Document.markdown` is
the artifact, not a starting point.

### GPU

```python
from magpdf import ExtractConfig, TablesConfig, extract

cfg = ExtractConfig(profile="tables", tables=TablesConfig(device="auto"))
doc = extract(pdf_bytes, config=cfg)
```

`table_workers` defaults to `0`, meaning **derive it from the device**: a
page's tables are predicted concurrently on an accelerator and sequentially on
CPU. That is not a hedge — TableFormer is already parallel on CPU via
`num_threads`, so N concurrent tables there oversubscribe the same cores and
every table gets slower. Set `table_workers` explicitly to override either way.

### Vision (optional)

Inject a `VisionClient` and two things become possible: a scanned page is
**replaced** by its transcription, and a figure description is **inserted at
the figure's position**. With no client injected, neither runs and the profile
stays fully offline and deterministic.

### Concurrency

> `extract(profile="tables")` serialises at process scope. Concurrent callers
> block; they never race and never crash. **Scale with processes, not threads.**

pypdfium2 is not thread-safe. This is enforced rather than documented, so a
threaded caller gets slow instead of corrupt.

### Known limits

- **Reading order is a `(top, left)` sort.** Correct for single-column
  documents; it interleaves columns line by line on multi-column layouts.
- **The extra is heavy.** `docling-ibm-models` pulls torch, torchvision,
  transformers and accelerate — on the order of a couple of GB.
- Weights are fetched once from HuggingFace and cached. With no access, the
  profile degrades to the pdfium text layer and says so in `Document.warnings`;
  it does not hang or fail.

---

## Scope

**PDF only.** LiteParse parses PDF natively and reaches every other format by
shelling out to LibreOffice (measured: ~9.5 s for one XLSX, returned as 36
paginated "pages"). Office, email, HTML and tabular formats belong to
[`mag-file-handler`](../file_handler), which this package neither imports nor is
imported by.

Non-PDF input raises `UnsupportedFormat`. Encrypted PDFs return
`ok=False, error="pdf is encrypted"` unless a `password=` is supplied.

## Known ceilings

LiteParse reconstructs tables by spatial column projection and does not detect
where a table ends, so it can absorb adjacent content into the last row -
identically with OCR on or off. Tesseract also drops cells on image-tables. Both
are documented with evidence in
[`docs/KNOWLEDGE_HANDOFF.md`](docs/KNOWLEDGE_HANDOFF.md) section 7. This package
wraps those flaws and reports them; it does not claim to fix them.

## Command line

Installing the package puts two commands on `PATH`, one per pipeline:

```bash
magpdf-fast   in.pdf -o out.md      # gated LiteParse
magpdf-tables in.pdf -o out.md      # DocLayout-YOLO + TableFormer
```

Two commands rather than one with a `--profile` flag: they do not share
dependencies or failure modes, and one typo should not silently run the other
engine. Markdown goes to stdout, every diagnostic to stderr, so `> out.md`
yields a clean file.

```bash
magpdf-fast  - < in.pdf > out.md       # stdin/stdout
magpdf-tables a.pdf b.pdf --out-dir o  # batch; models load once, not per file
magpdf-tables in.pdf --json            # metadata envelope (JSON Lines if batched)
magpdf-fast  --self-test               # does OCR actually work on this platform?
magpdf-tables --doctor                 # environment + where models resolve from
```

Exit codes: `0` ok, `1` extraction error, `2` usage, `3` not a PDF,
`4` engine or models unavailable.

`--workers` defaults to **4** here, not the library's 1. The library default is
tuned for a caller that already runs documents in parallel; a CLI is the
opposite case, and 4 measured ~2.8x faster.

`magpdf-tables` **warms the models up before extracting**. This is deliberate: a
missing model does not make `extract()` raise - the layout gate swallows it and
the document degrades to plain text-layer markdown with no warning and exit 0.
The pre-flight turns that into exit 4. `--no-warmup` opts out.

## Serve (sidecar)

Both pipelines over HTTP, for callers that are not Python or not on the same
box:

```bash
pip install 'mag-pdf[tables,serve]'
magpdf-serve --port 8109
```

```
GET  /healthz   {"status","fast_ready","tables_ready",...}
POST /extract   multipart: file, profile=fast|tables, password?, table_mode?
```

```bash
curl -sf -F file=@in.pdf -F profile=tables localhost:8109/extract | jq -r .markdown
```

The response body is **the same envelope** `magpdf-fast --json` prints. One
shape, so a caller can move between the CLI and the sidecar without a second
parser.

- **`/healthz` returns 503 until both profiles are warm**, and stays 503 if the
  tables models never load. That is deliberate: a missing model does not make
  `extract()` fail, it makes it quietly return text-layer markdown, so
  readiness is the only place that failure can be made visible.
- **A bad document is `200` with `ok:false`** (an encrypted PDF is a fact about
  the file, not an outage); a bad server is `4xx`/`5xx`. Callers need to tell
  those apart.
- **Run one worker per container.** The tables pipeline serialises on a
  process-scope lock, and a second worker in the same container loads its own
  copy of the models for no extra throughput. Scale with replicas.

Environment: `MAGPDF_SERVE_TABLES`, `MAGPDF_FAST_CONCURRENCY`,
`MAGPDF_FAST_WORKERS`, `MAGPDF_TABLE_MODE`, `MAGPDF_NUM_THREADS`,
`MAGPDF_MAX_UPLOAD_BYTES`, `MAGPDF_QUEUE_TIMEOUT_SECONDS`, `LOG_LEVEL`.

## Docker

One image, all three entry points, weights baked in. It never touches the
network.

```bash
docker pull veermedi/mag-pdf:0.3.0

docker run --rm -v "$PWD:/data" veermedi/mag-pdf:0.3.0 \
    magpdf-tables /data/in.pdf -o /data/out.md

docker run --rm -i veermedi/mag-pdf:0.3.0 magpdf-fast - < in.pdf > out.md
docker run --rm --network none veermedi/mag-pdf:0.3.0 magpdf-tables --self-test

docker run --rm -p 8109:8109 veermedi/mag-pdf:0.3.0 magpdf-serve
```

- **Never pass `-t`** when capturing output. A TTY merges stderr into stdout and
  translates newlines to CRLF, corrupting the markdown.
- Writing to a bind mount as a non-root container user needs
  `--user "$(id -u):$(id -g)"` on Linux, or the write is denied.
- The image is built `linux/amd64` only. This is now a build choice rather than
  a constraint: liteparse 2.13.0 does publish a Linux arm64 wheel, so an arm64
  image is buildable if someone wants one. Until it is built, Apple Silicon
  needs `--platform linux/amd64` and should expect emulation to be slow.
- The tables profile serialises to one document per process, so scale with
  replicas or `docker run` invocations, never threads.

Build it yourself:

```bash
docker build -t mag-pdf:local ./magpdf
```

The build runs six gates and fails rather than shipping something subtly wrong:
CPU-only torch, models loading from the baked cache, a real tables extraction
with no degradation warnings, the OCR canary, an assertion that only the
TableFormer weights are baked in and the `docling` orchestration package is
absent, and that the HTTP surface builds and still emits tables a downstream
chunker can find.

## Install

```bash
pip install mag-pdf
```

Python 3.10+, **except on macOS, which needs 3.11+**: liteparse publishes no
macOS cp310 wheel, and pip's fallback of building the Rust sdist omits the
bundled pdfium library — so the install succeeds and every parse then dies with
`PanicException: failed to load pdfium shared library`. Linux and Windows are
fine on 3.10. CI excludes that one cell rather than pretending it works.

Apache-2.0.
