Metadata-Version: 2.4
Name: pyferro-pdf
Version: 0.1.0
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: Implementation :: CPython
License-File: LICENSE
Summary: Fast, layout-aware PDF text extraction in Rust with Python bindings
Requires-Python: >=3.8
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/anshgoyalevil/ferro-pdf
Project-URL: Repository, https://github.com/anshgoyalevil/ferro-pdf

# ferro-pdf

Fast, layout-aware PDF text extraction written in Rust, with Python bindings.

Most PDF text extractors are either **slow** (pdfminer.six is pure Python) or
**flat** (they emit glyphs in content-stream order, which scrambles multi-column
pages). `ferro-pdf` is built for RAG / document-ingestion pipelines: it
reconstructs **reading order** (columns, lines, spacing) and is fast enough to
not be the bottleneck.

## Benchmark

### Synthetic corpus (clean, standard fonts)
9 generated PDFs, 198 pages, single- and two-column.
Machine: Apple M4 Pro (14 cores), macOS 26.5. Best-of-5, warm cache.

| tool              | best (s) | speedup vs pdfminer | content (words vs pdfminer) |
|-------------------|---------:|--------------------:|----------------------------:|
| pdfminer.six      |    2.68 |                1.0× |                        100% |
| pypdfium2 (C++)   |    0.098 |               27.3× |                        100% |
| **ferro-pdf**     |  **0.019** |          **138.6×** |                    **100%** |

### Real research papers (arXiv, embedded fonts, math, two-column)
6 papers (Attention, BERT, GPT-3, InstructGPT, LLaMA, ResNet), 213 pages total.

| tool              | best (s) | speedup vs pdfminer | content (words vs pdfminer) |
|-------------------|---------:|--------------------:|----------------------------:|
| pdfminer.six      |    4.33 |                1.0× |                        100% |
| pypdfium2 (C++)   |    0.224 |               19.4× |                        100% |
| **ferro-pdf**     |  **0.135** |           **32.1×** |                     **98%** |

Per-paper: 19–43× faster, 91–100% word parity. The parity gap on dense
two-column papers (ResNet 91%, BERT 95%) comes from math/figure/table content
where word tokenization diverges from pdfminer — the next thing to harden.

> **Update:** since this table was recorded, real-paper word parity improved to
> ~99% after implementing full font-encoding support (Identity-H CID,
> `/Differences`) and real-glyph-width spacing. Re-run `bench.py papers` to
> refresh these numbers on your machine.

**Takeaway:** on clean PDFs the win is ~140×; on messy real-world papers it's a
still-excellent ~30× (and ~1.7× faster than C++ pypdfium2), at 98% content
parity. Quote the real-paper number publicly — it survives scrutiny.

Reproduce:

```bash
python gen_corpus.py corpus      # synthetic PDFs
python bench.py corpus           # synthetic benchmark
python bench.py papers           # real-paper benchmark (after downloading PDFs)
```

The speedup is **content-preserving**: ferro-pdf extracts ~the same word count
as pdfminer (the reference), so the win isn't from extracting less text.

## How it works

```
PDF bytes
  │  lopdf            parse objects + decode content streams (don't reinvent this)
  ▼
glyph extraction      walk content operators, track CTM/text-matrix, decode
  │  (src/extract.rs)  Unicode via ToUnicode CMap, Identity-H CID, or simple-font
  │                    /Encoding + /Differences; measure real glyph widths
  ▼
layout engine ★       column detection (x-coverage histogram + gutter finding),
  │  (src/layout.rs)   line bucketing by y, gap-based spacing, reading order
  ▼
rayon                 lay out pages in parallel (pdfminer is single-threaded)
  ▼
clean reading-order text
```

The layout engine is the moat — it's what flat Rust crates (`pdf-extract`) and
even C++ `pypdfium2` don't do for you.

## Usage

### Python

```bash
pip install pyferro-pdf
```

```python
import ferro_pdf
text = ferro_pdf.extract_text("paper.pdf")
text = ferro_pdf.extract_text_from_bytes(open("paper.pdf", "rb").read())
```

### Rust
```rust
let text = ferro_pdf::extract_text_from_path("paper.pdf")?;
```

### CLI
```bash
cargo build --release
./target/release/ferro-cli paper.pdf            # print text
./target/release/ferro-cli paper.pdf --time     # print timing to stderr
```

## Build

```bash
# Rust library + CLI
cargo build --release

# Python wheel (into the active venv)
maturin develop --release
```

## Honest limitations (current PoC)

- **Text-only**: table reconstruction (ruling lines → Markdown grid) is **not**
  implemented yet — that's the next moat to build.
- Scanned / image-only PDFs need an OCR fallback (not included).
- Column detection is heuristic (gutter finding); pathological layouts may need
  tuning of the thresholds in `detect_columns`.
- Font decoding handles ToUnicode CMaps (incl. array-form `bfrange`), Identity-H
  CID fonts, and simple fonts via WinAnsi/MacRoman + `/Differences`. Truly exotic
  encodings without any of these signals may still mis-decode.

These are the honest gaps to close before claiming production parity — but the
core speed + reading-order story is real and reproducible above.

## Performance: where the time goes & further scope

Stage profile (`FERRO_PROFILE=1`, GPT-3, 75 pages), after optimization:

| stage  | time | notes |
|--------|-----:|-------|
| load   | ~42 ms (~85%) | `lopdf` parse + inflate — **the bottleneck**, single-threaded |
| extract| ~4 ms  | content-stream + font decode (was ~31 ms before the font cache) |
| layout | ~3 ms  | column/line reconstruction — negligible |

Optimizations already applied:
- **Font/ToUnicode cache** (parse each CMap once, not per page): ~**7× on the
  extract stage** (GPT-3 31 ms → 4 ms).
- **Parallel page extraction + parallel batch API** (`extract_text_batch`):
  **9.2× scaling** on 14 cores.

Throughput (60-PDF / 2,130-page batch, Apple M4 Pro):

| mode | time | vs pdfminer | notes |
|------|-----:|------------:|-------|
| pdfminer.six (sequential)   | 43.9 s | 1.0× | |
| pypdfium2 — 1 core          |  2.32 s | 18.9× | C++; fastest single-core |
| pypdfium2 — threads         | crashes | — | pdfium is **not thread-safe** |
| pypdfium2 — process pool    |  1.42 s | 31× | fork + IPC overhead |
| ferro — 1 core              |  5.63 s | 7.8× | lopdf parse/inflate-bound |
| **ferro — parallel batch**  | **0.59 s** | **74×** | no-GIL Rust threads |

Honest read vs pypdfium2: **per single core, pdfium (C++) is ~2.4× faster** than
ferro — it's heavily optimized and ferro is bound by `lopdf`. ferro wins at
**batch scale (2.4× faster than pdfium's process pool)** because pdfium can't use
threads, while ferro parallelizes freely with no GIL. ferro's edge is
parallelism + pure-Rust embeddability + reading-order layout output, *not* raw
single-core speed.

Remaining scope, ranked by expected payoff:
1. **Attack `load` (85% of single-core time).** This is also exactly the gap to
   pypdfium2. SIMD inflate (flate2 `zlib-ng`, needs cmake) ≈ 2–3× on
   decompression; skip image/XObject streams we never use; or a lazier parser.
   Closing this would make ferro competitive with pdfium single-core too.
2. **Parallel granularity.** 9.2× / 14 cores ≈ 66% efficiency; file-level-only
   parallelism for batch (avoid nested rayon overhead) could reach ~12×.
3. **Allocation trimming** in the operator loop (arena / reuse buffers): modest.

Beyond ~2× more you'd need to drop `lopdf` for an FFI to pdfium/mupdf — at which
point you lose the "pure Rust" story and pdfium would roughly match you.

