Metadata-Version: 2.4
Name: prettypipeline-ocr
Version: 0.4.0
Summary: Turn PDFs into structured JSON with local OCR, GPT vision, and training exports.
Author: greatshoor
License-Expression: MIT
Project-URL: Homepage, https://github.com/virajshoor/PrettyPipeline
Project-URL: Repository, https://github.com/virajshoor/PrettyPipeline
Project-URL: Issues, https://github.com/virajshoor/PrettyPipeline/issues
Project-URL: PyPI, https://pypi.org/project/prettypipeline-ocr/
Keywords: ocr,pdf,llm,extraction,json
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Requires-Python: <3.14,>=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pymupdf>=1.24
Requires-Dist: openai>=1.0
Requires-Dist: pillow
Provides-Extra: ocr
Requires-Dist: transformers<5,>=4.57.1; extra == "ocr"
Requires-Dist: torch; extra == "ocr"
Requires-Dist: torchvision; extra == "ocr"
Requires-Dist: einops; extra == "ocr"
Requires-Dist: addict; extra == "ocr"
Requires-Dist: easydict; extra == "ocr"
Requires-Dist: safetensors; extra == "ocr"
Requires-Dist: tqdm; extra == "ocr"
Requires-Dist: matplotlib; extra == "ocr"
Requires-Dist: requests; extra == "ocr"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Requires-Dist: transformers<5,>=4.57.1; extra == "dev"
Requires-Dist: torch; extra == "dev"
Requires-Dist: torchvision; extra == "dev"
Requires-Dist: einops; extra == "dev"
Requires-Dist: addict; extra == "dev"
Requires-Dist: easydict; extra == "dev"
Requires-Dist: safetensors; extra == "dev"
Requires-Dist: tqdm; extra == "dev"
Requires-Dist: matplotlib; extra == "dev"
Requires-Dist: requests; extra == "dev"
Dynamic: license-file

# PrettyPipeline

[![PyPI](https://img.shields.io/pypi/v/prettypipeline-ocr.svg)](https://pypi.org/project/prettypipeline-ocr/)
[![Python](https://img.shields.io/pypi/pyversions/prettypipeline-ocr.svg)](https://pypi.org/project/prettypipeline-ocr/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

Turn PDFs into structured JSON. Text comes from embedded PDF content or local OCR. **Figures are segregated and sent directly to GPT-5.4** — never the whole PDF as one image. Export to CSV and common LLM fine-tuning formats.

## Architecture

```
                         ┌─────────────────────────────────────┐
                         │              PDF input              │
                         └─────────────────┬───────────────────┘
                                           │
              ┌────────────────────────────┼────────────────────────────┐
              │                            │                            │
              ▼                            ▼                            ▼
     ┌────────────────┐         ┌──────────────────┐         ┌──────────────────┐
     │ Embedded text  │         │ Embedded figures │         │ Scan-only pages  │
     │ (digital PDF)  │         │ (PyMuPDF crop)   │         │ (low-res fallback│
     └───────┬────────┘         └────────┬─────────┘         │  120 DPI max)    │
             │                           │                   └────────┬─────────┘
             │                           │                            │
             │              ┌────────────┴────────────┐               │
             │              │  NOT full-page images   │               │
             │              │  detail=low by default  │               │
             │              └────────────┬────────────┘               │
             │                           │                            │
             └───────────────────────────┼────────────────────────────┘
                                         │
                                         ▼
                            ┌────────────────────────┐
                            │   GPT-5.4 nano (cloud) │
                            │   text + figure images │
                            │   strict JSON schema   │
                            └───────────┬────────────┘
                                        │
                    ┌───────────────────┼───────────────────┐
                    ▼                   ▼                   ▼
              ┌──────────┐      ┌─────────────┐    ┌──────────────┐
              │   JSON   │      │ needs_review│    │ Export formats│
              │  + _meta │      │   flags     │    │ csv alpaca   │
              └──────────┘      └─────────────┘    │ sharegpt qa  │
                                                   │ openai jsonl │
                                                   └──────────────┘

Local OCR (Unlimited-OCR) only when a page has no usable embedded text
and `--force-ocr` is set or hybrid routing needs it. Requires `[ocr]` extra.
```

## Why this is cheap

- **Digital PDFs** — embedded text is free and accurate; no GPU needed.
- **Figures only** — segregated embedded images go to GPT vision at `detail=low` (not full-page raster).
- **Local OCR** — [baidu/Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR) for scan pages only (`pip install prettypipeline-ocr[ocr]`).
- **GPT-5.4 nano** — ~$0.20 / 1M input tokens for structuring ([pricing](https://developers.openai.com/api/docs/models/gpt-5.4-nano)).

## Install

Python **3.10–3.13** (3.12 recommended).

```bash
# Digital PDFs + GPT vision (smallest install)
pip install prettypipeline-ocr

# + local OCR for scans
pip install prettypipeline-ocr[ocr]
```

From this repo:

```bash
python3.12 -m venv .venv && source .venv/bin/activate
pip install -e ".[ocr,dev]"
```

**NVIDIA Linux:**

```bash
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124
pip install prettypipeline-ocr[ocr]
```

**Docker (CUDA):** see `Dockerfile.cuda`.

## Quick start

```bash
export OPENAI_API_KEY=sk-...

# Extract JSON (text + segregated figures → GPT)
prettypipeline run invoice.pdf --schema examples/invoice.schema.json

# Also export fine-tuning formats
prettypipeline run invoice.pdf --schema examples/invoice.schema.json \
  -o out.json --export csv,alpaca,sharegpt,openai,qa

# Convert existing JSON to a training file
prettypipeline export out.json --schema examples/invoice.schema.json \
  --format sharegpt -o train.jsonl
```

Demo invoice:

```bash
prettypipeline run tests/fixtures/sample_invoice.pdf \
  --schema examples/invoice.schema.json -o out.json
```

## Figure segregation (vision)

PrettyPipeline **does not** rasterize entire PDF pages for GPT unless a page is scan-only with no embedded figures.

| Page type | Text source | Vision sent to GPT |
|---|---|---|
| Digital + figures | Embedded PDF text | Embedded figures only (`detail=low`) |
| Digital, no figures | Embedded PDF text | Nothing |
| Scan, no figures | Local OCR or low-res page | One low-res page image (120 DPI) |
| `--no-vision` | Same as above | Disabled |

Images are **visual context only** — no CSV or table dump is generated from figures.

## Export formats (LLM training)

| Format | File | Use case |
|---|---|---|
| `csv` | `.csv` | Spreadsheets, flat field/value |
| `alpaca` | `.alpaca.jsonl` | Instruction tuning (LLaMA-Factory, Axolotl) |
| `sharegpt` | `.sharegpt.jsonl` | Multi-turn chat fine-tuning |
| `openai` | `.openai.jsonl` | OpenAI fine-tuning `messages` format |
| `qa` | `.qa.jsonl` | Simple question/answer pairs |

ShareGPT and OpenAI JSONL are preferred for chat models in 2026; Alpaca for single-turn extraction tasks.

## Schema examples

| File | Use case |
|---|---|
| `examples/invoice.schema.json` | Invoices |
| `examples/receipt.schema.json` | Receipts |
| `examples/purchase_order.schema.json` | Purchase orders |

## CLI

```
prettypipeline run FILE.pdf --schema SCHEMA.json
  -o, --output PATH           JSON output (also printed)
  --export csv,alpaca,...     comma-separated training exports
  --no-vision                 skip figure images to GPT
  --image-detail low|auto|high  vision token budget (default: low)
  --ocr-only                  skip GPT step
  --force-ocr                 always OCR text locally
  --model, --base-url         LLM settings

prettypipeline batch FILES... --schema SCHEMA.json --output-dir DIR
prettypipeline export RESULT.json --schema SCHEMA.json --format FORMAT -o OUT
```

Environment: `OPENAI_API_KEY`, `PRETTYPIPELINE_MODEL`, `PRETTYPIPELINE_BASE_URL`.

## Output

```json
{
  "data": { "invoice_number": "INV-3337", "total": 93.5 },
  "needs_review": [],
  "source": "pdf_text",
  "_meta": {
    "version": "0.4.0",
    "elapsed_ms": 3200,
    "pages": 1,
    "vision_images": 1,
    "segments": { "text_chars": 842, "figures": 1 },
    "token_usage": { "prompt_tokens": 1100, "completion_tokens": 95, "total_tokens": 1195 }
  }
}
```

## Programmatic use

```python
from prettypipeline import run, segment_pdf, to_csv

result = run("invoice.pdf", "examples/invoice.schema.json", export_formats=["alpaca"], export_stem=Path("out.json"))
seg = segment_pdf("invoice.pdf")  # inspect text vs figures before calling GPT
```

## Hardware

| Device | Role |
|---|---|
| NVIDIA CUDA | Recommended for local OCR (`[ocr]` extra) |
| Apple Silicon MPS | OCR supported; `--force-ocr` may degrade |
| CPU | OCR works, slow |
| None | Digital PDFs + GPT vision work without GPU |

## Development

```bash
pip install -e ".[ocr,dev]"
pytest
python -m build && twine check dist/*
```

## License

MIT. [Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR) is also MIT.

Fixtures: [Sliced Invoices sample](https://slicedinvoices.com/pdf/wordpress-pdf-invoice-plugin-sample.pdf), [IRS Form W-9](https://www.irs.gov/pub/irs-pdf/fw9.pdf).
