Metadata-Version: 2.5
Name: paper-siphon
Version: 0.5.0
Summary: Extract clean Markdown from academic PDFs - like drinking through a straw
Requires-Python: >=3.11
Requires-Dist: click>=8.1
Requires-Dist: docling>=2.0
Requires-Dist: onnxruntime>=1.20
Description-Content-Type: text/markdown

# Paper Siphon

**Extract clean Markdown from academic PDFs** - like drinking through a straw.

Academic papers come with artifacts: awkward page breaks, mangled tables, or even line numbers.
Paper Siphon filters them out, leaving you with clean, readable Markdown.

```
paper-siphon paper.pdf
```

That's it. Your paper is now `paper.md`.

---

## Features

- **Smart whitespace** - Collapses excessive blank lines, normalizes spacing
- **Table preservation** - Keeps your data tables intact and formatted
- **Formula support** - Optional enrichment for mathematical expressions
- **Line number removal** - Automatically strips the margin numbers (when present)
- **VLM backend** - `--vlm` uses Nemotron float32 on Apple Silicon (marker elsewhere) for
  complex layouts, heavy math, and broken encodings. See [the benchmark](bench/README.md)
- **Self-healing** - Auto-escalates to the VLM backend when the fast default
  output looks garbled or drops equations

## Installation

```bash
# With uv (recommended)
uv pip install paper-siphon

# With pip
pip install paper-siphon
```

The VLM backends run in isolated environments provisioned on demand. They
require [`uv`](https://docs.astral.sh/uv/) on your PATH. The Apple Silicon
backends also need Git to fetch the pinned MLX-VLM revision. The first run
installs the runtime and downloads model weights, then caches them. Nemotron
float32 weights are about 2.5 GB, Nemotron BF16 about 1.3 GB, and Xiaomi about
1.7 GB. Runtime packages require additional disk space.

## Usage

### Quick start (no install)

```bash
uvx paper-siphon paper.pdf                # Run directly with uvx
```

### Basic

```bash
paper-siphon paper.pdf                    # Creates paper.md
paper-siphon paper.pdf -o notes.md        # Custom output path
```

### From URL (including arXiv)

```bash
paper-siphon https://arxiv.org/pdf/1706.03762.pdf
```

**Tip:** For arXiv papers, just change `/abs/` to `/pdf/` in the URL:
```
https://arxiv.org/abs/1706.03762  →  https://arxiv.org/pdf/1706.03762.pdf
```

(That's "Attention Is All You Need" - the Transformer paper)

### Advanced

```bash
paper-siphon --vlm paper.pdf              # Force Nemotron float32 / marker
paper-siphon --vlm --vlm-backend xiaomi paper.pdf  # Local formula-focused model
paper-siphon --vlm --vlm-backend nemotron-bf16 paper.pdf  # Faster local VLM
paper-siphon --no-escalate paper.pdf      # Don't auto-retry with the VLM backend
paper-siphon --enrich-formula paper.pdf   # Formula enrichment on the default pipeline
paper-siphon --no-mlx --vlm paper.pdf     # Force the marker backend even on a Mac
paper-siphon -v paper.pdf                 # Verbose logging
```

By default, the fast pipeline runs first and paper-siphon **automatically
re-runs with the VLM backend** if that output looks garbled (a font-decoding
failure) or dropped its equations. Use `--vlm` to force the VLM backend from the
start, or `--no-escalate` to keep the fast output as-is.

## How It Works

Paper Siphon uses [Docling](https://github.com/DS4SD/docling) for PDF parsing, then applies
post-processing to clean up common academic paper artifacts:

1. **PDF parsing** - Extracts structure, text, and tables (fast Docling pipeline)
2. **Quality check** - Detects font-decoding failures (garbled glyphs) and
   dropped equations and escalates to the VLM backend when needed
3. **Line number filtering** - Removes standalone 1-4 digit numbers (common in journal formats)
4. **Whitespace normalization** - Collapses multiple blank lines

The VLM backend uses Nemotron 3.5 float32 on Apple Silicon and marker elsewhere.
`--vlm-backend` selects the model for forced VLM extraction and automatic retries.
Xiaomi MLX BF16 is the formula-focused choice, and Nemotron BF16 is faster with
some additional formula errors. GLM remains available as `--vlm-backend glm-ocr`.
Explicit MLX choices require Apple Silicon. `--no-mlx` uses marker.

Automatic escalation reruns the complete document. Xiaomi can duplicate
bibliography entries, and all tested local models can assign values to the wrong
table columns. Check important formulas and table values against the source.
See the [measured comparison](bench/baseline/ALL_MODELS.md) for the results.
The isolated runtimes keep their dependencies separate from Docling.

## Options

| Flag | Description |
|------|-------------|
| `-o, --output` | Output file path (default: input with `.md` extension) |
| `--vlm` | Force the selected VLM (Nemotron float32 on Apple Silicon, marker elsewhere) |
| `--vlm-backend` | `auto`, `nemotron`, `nemotron-bf16`, `xiaomi`, `glm-ocr`, or `marker` (default: `auto`) |
| `--mlx/--no-mlx` | Enable MLX on Apple Silicon. `--no-mlx` forces marker |
| `--escalate/--no-escalate` | Auto-retry with the VLM backend on garbled/math-dropping output (default: on) |
| `--enrich-formula` | Enable formula enrichment on the default pipeline (slow, CPU-bound) |
| `-v, --verbose` | Enable debug logging |

## Backend benchmark

The [combined model comparison](bench/baseline/ALL_MODELS.md) covers all measured
configurations, including both Nemotron precisions and Xiaomi on MLX and MPS.

The [page-aligned comparison](bench/baseline/RESULTS.md) evaluates Docling,
the current GLM integration, and GPT-6.1 Sol on 16 pages from eight papers.
It compares complete transcriptions with matching rendered pages and includes
a separate three-page LightOnOCR-3 test. Sol scored highest in this sample,
and Docling was fastest. See the
[reproduction guide](bench/baseline/README.md) for commands and saved evidence.
Follow-up experiments cover [Nemotron 3.5](bench/baseline/NEMOTRON.md) and
[Xiaomi-OCR-0](bench/baseline/XIAOMI.md) on the same 16 pages.
Version 0.5.0 keeps Docling first and uses Nemotron float32 for Apple Silicon VLM retries.

The [older 14-PDF comparison](bench/README.md) remains available as historical
results. An [audit](docs/research/2026-10-22-pdf-extraction/README.md) found
truncated candidate inputs and weak verdict validation in that benchmark.
Its scores should not determine model selection without a new comparison.

## Development

```bash
# Clone and install
git clone https://github.com/mrshu/paper-siphon.git
cd paper-siphon
uv sync --dev

# Run tests
uv run pytest

# Run tests with coverage
uv run pytest --cov=paper_siphon
```

## License

MIT

---

*Stop wrestling with PDFs. Just siphon the good stuff.*
