Metadata-Version: 2.4
Name: rc-docparser
Version: 0.2.1
Summary: Convert research literature (.docx, .xlsx, .pdf, .html, .pptx, .epub, .txt, .md, .csv) into structured Markdown + JSON corpora, with optional VLM image semantic captioning.
Project-URL: Homepage, https://github.com/Research-Commons/docparser
Project-URL: Repository, https://github.com/Research-Commons/docparser
Project-URL: Issues, https://github.com/Research-Commons/docparser/issues
Project-URL: Changelog, https://github.com/Research-Commons/docparser/blob/main/CHANGELOG.md
Project-URL: Research Commons, https://lab.researchcommons.ai/
Author: rc-docparser contributors
License-Expression: MIT
License-File: LICENSE
Keywords: corpus,csv,docx,epub,html,literature,markdown,ocr,parser,pdf,pptx,rag,vlm,xlsx
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Text Processing :: Markup
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: lxml>=5.3.0
Requires-Dist: openpyxl>=3.1.5
Requires-Dist: pillow>=10.4.0
Requires-Dist: python-docx>=1.1.2
Requires-Dist: python-dotenv>=1.0.1
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: tqdm>=4.66.5
Provides-Extra: all
Requires-Dist: beautifulsoup4>=4.12.0; extra == 'all'
Requires-Dist: ebooklib>=0.18; extra == 'all'
Requires-Dist: numpy>=1.24.0; extra == 'all'
Requires-Dist: pdfplumber>=0.11.0; extra == 'all'
Requires-Dist: pymupdf>=1.24.0; extra == 'all'
Requires-Dist: python-pptx>=1.0.0; extra == 'all'
Requires-Dist: rapidocr-onnxruntime>=1.3.0; extra == 'all'
Requires-Dist: requests>=2.32.3; extra == 'all'
Requires-Dist: trafilatura>=1.12.0; extra == 'all'
Provides-Extra: dev
Requires-Dist: build>=1.2.0; extra == 'dev'
Requires-Dist: ebooklib>=0.18; extra == 'dev'
Requires-Dist: mypy>=1.10.0; extra == 'dev'
Requires-Dist: pandas>=2.2.0; extra == 'dev'
Requires-Dist: pdfplumber>=0.11.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: python-pptx>=1.0.0; extra == 'dev'
Requires-Dist: ruff>=0.6.0; extra == 'dev'
Requires-Dist: twine>=5.1.0; extra == 'dev'
Provides-Extra: docling
Requires-Dist: docling>=2.0.0; extra == 'docling'
Provides-Extra: epub
Requires-Dist: beautifulsoup4>=4.12.0; extra == 'epub'
Requires-Dist: ebooklib>=0.18; extra == 'epub'
Provides-Extra: html
Requires-Dist: beautifulsoup4>=4.12.0; extra == 'html'
Requires-Dist: trafilatura>=1.12.0; extra == 'html'
Provides-Extra: localvlm
Requires-Dist: pillow>=10.4.0; extra == 'localvlm'
Requires-Dist: torch>=2.2.0; extra == 'localvlm'
Requires-Dist: transformers>=4.40.0; extra == 'localvlm'
Provides-Extra: marker
Requires-Dist: marker-pdf>=1.0.0; extra == 'marker'
Provides-Extra: ocr
Requires-Dist: numpy>=1.24.0; extra == 'ocr'
Requires-Dist: rapidocr-onnxruntime>=1.3.0; extra == 'ocr'
Provides-Extra: pdf
Requires-Dist: pymupdf>=1.24.0; extra == 'pdf'
Provides-Extra: pptx
Requires-Dist: python-pptx>=1.0.0; extra == 'pptx'
Provides-Extra: pymupdf4llm
Requires-Dist: pymupdf4llm>=0.0.17; extra == 'pymupdf4llm'
Provides-Extra: tables
Requires-Dist: pdfplumber>=0.11.0; extra == 'tables'
Provides-Extra: vlm
Requires-Dist: requests>=2.32.3; extra == 'vlm'
Description-Content-Type: text/markdown

# docparser

Convert research literature (`.docx`, `.xlsx`, `.pdf`, `.html`, `.pptx`,
`.epub`, `.txt`, `.md`, `.csv`) into a clean, reproducible **Markdown + JSON**
corpus, with optional **vision-language captioning** of every embedded figure
via OpenRouter, OpenAI, Gemini, a local server, or a fully-local model.

```text
                              ┌────────────────┐
  data/raw/*.docx             │   docparser    │       data/parsed/<slug>/
  data/raw/*.xlsx             │  - parse_docx  │       document.md
  data/raw/*.pdf      ─────►  │  - parse_xlsx  │ ────► document.json
  data/raw/*.html             │  - parse_pdf   │
  data/raw/*.pptx             │  - parse_html  │       data/assets/<slug>/
  data/raw/*.epub             │  - parse_pptx  │       img-*.png
  data/raw/*.txt|md|csv       │  - parse_epub  │
                              │  - VLM caption │
                              └────────────────┘
```

## Install

The package is published on PyPI as **`rc-docparser`**; the Python import name
is `docparser` (i.e. `import docparser`).

```bash
pip install rc-docparser              # core: docx + xlsx + txt/md + csv/tsv
pip install 'rc-docparser[pdf]'       # + PyMuPDF for PDFs
pip install 'rc-docparser[html]'      # + trafilatura + bs4 for HTML
pip install 'rc-docparser[pptx]'      # + python-pptx for PowerPoint
pip install 'rc-docparser[epub]'      # + EbookLib + bs4 for EPUB
pip install 'rc-docparser[vlm]'       # + requests for API VLM captions
pip install 'rc-docparser[all]'       # everything above (recommended)
```

Higher-fidelity / heavier features are separate opt-in extras (so the core
install stays small and MIT):

```bash
pip install 'rc-docparser[tables]'       # + pdfplumber for PDF table extraction
pip install 'rc-docparser[ocr]'          # + rapidocr-onnxruntime for scanned PDFs
pip install 'rc-docparser[pymupdf4llm]'  # PyMuPDF4LLM PDF backend (AGPL/commercial)
pip install 'rc-docparser[docling]'      # IBM Docling PDF backend (MIT)
pip install 'rc-docparser[marker]'       # Datalab Marker PDF backend (GPL-3.0)
pip install 'rc-docparser[localvlm]'     # transformers/torch local captioning
```

`rc-docparser` requires Python 3.10+.

## Quick start (library)

```python
from docparser import WorkspaceLayout, run_all

layout = WorkspaceLayout.under("./project")   # data/raw, data/parsed, data/assets, .cache
layout.ensure()

run_all(layout, use_vlm=False)                # parse everything in data/raw
```

For a single file:

```python
from docparser import parse_path, WorkspaceLayout

layout = WorkspaceLayout.under(".")
payload = parse_path("paper.pdf", layout)
print(payload["stats"])
```

## Quick start (CLI)

```bash
# parse a single file
docparser parse paper.pdf --workspace ./out --no-vlm

# walk a whole directory
docparser parse-all --workspace ./project --no-vlm

# enable VLM captioning (requires OPENROUTER_API_KEY in env or .env)
export OPENROUTER_API_KEY=sk-or-v1-...
docparser parse-all --workspace ./project --max-images 50

# higher-fidelity PDF: pick a backend, OCR scanned pages, extract tables
docparser parse paper.pdf --pdf-backend docling --ocr auto --pdf-tables --no-vlm

# caption with a different provider
docparser parse-all --workspace ./project --vlm-provider openai --vlm-model gpt-4o-mini

docparser version
```

## What gets captured

### `.docx`
- Walks the document body in **document order** (paragraphs + tables + drawings).
- Preserves heading hierarchy (`section_path`) on every block.
- Extracts every embedded image to `data/assets/<slug>/` with a stable
  `img-<seq>-<sha10>.<ext>` name.
- Detects figure/table captions (style `Caption` or text matching
  `Figure 1: …` / `Fig. 1.` / `Table 1.`) and associates the caption with the
  preceding image.
- Captures `context_before` and `context_after` for every image so the VLM has
  document-grounded context.

### `.xlsx`
- Iterates **every sheet, every row, every column**.
- For each cell stores: address, row/col indices, value, openpyxl `data_type`,
  `number_format`, `hyperlink`, `comment`, and the **formula** (from a second
  pass with `data_only=False`).
- Stores `merged_ranges`, `frozen_panes`, and any embedded images.
- Markdown rendering uses the first non-empty row as a header heuristic and
  preserves multi-line cells with `<br>`.

### `.pdf`
- Page-by-page text extraction in reading order via PyMuPDF's blocks API.
- Best-effort heading detection from font size (≥120% of the body-text median
  promotes a line to a heading; bold flag tracked).
- Embedded raster images extracted via `doc.extract_image(xref)`.
- **Pluggable backends** (`backend="pymupdf4llm" | "docling" | "marker"`) route
  conversion to a higher-fidelity engine; their Markdown is normalized into the
  same block schema. Images are still extracted via PyMuPDF.
- **OCR** (`ocr="auto" | "force"`, `[ocr]` extra) recognizes text on scanned /
  low-text pages; OCR'd blocks carry `"ocr": true`.
- **Tables** (`extract_tables=True`, `[tables]` extra) emit real `table` blocks
  via `pdfplumber`.

### `.html`
- Article-grade body extraction via `trafilatura`.
- Plus a **structural** BeautifulSoup walk that emits typed blocks
  (`heading` / `paragraph` / `list` / `table` / `image`) so downstream RAG
  layers can rely on the JSON.

### `.pptx`
- Walks slides in presentation order; each slide becomes a section.
- Emits per-slide headings (slide title), bulleted text frames (with list
  level), tables, pictures, and **speaker notes**.
- Embedded pictures extracted and optionally captioned.

### `.epub`
- Walks the spine in reading order; per-chapter BeautifulSoup structural walk.
- Captures metadata (title/author/language), headings, paragraphs, lists,
  tables, and embedded images (resolved from the EPUB image manifest).

### `.txt` / `.md` and `.csv` / `.tsv` (core, no extras)
- Plain text is split into paragraph blocks; Markdown is passed through and
  also decomposed into heading / list / code / paragraph blocks.
- CSV/TSV: delimiter sniffing, header detection, a Markdown table, and one JSON
  record per row.

### Images (`[vlm]` extra)
Each image is sent to a vision-language model (default provider OpenRouter,
model `anthropic/claude-sonnet-4`) along with its surrounding caption +
context. Any OpenAI-compatible provider works via `--vlm-provider`
(`openrouter` / `openai` / `gemini` / `local`), or use a fully-local
`transformers` model with `--vlm-provider transformers` (`[localvlm]` extra).
The model returns a strict JSON object:

```json
{
  "caption":           "one-sentence figure caption",
  "description":       "2–5 sentence paragraph",
  "visible_text":      "OCR-style transcription",
  "tags":              ["world-model", "diagram", "..."],
  "image_kind":        "diagram | plot | screenshot | photo | equation | table | ...",
  "domain_relevance":  "how this relates to the document's topic"
}
```

Results are cached on disk at `<cache_dir>/vlm/<model>/<sha1>.json`, keyed by
**SHA-1 of the image bytes × model**, so re-runs are free until the source
image bytes change.

## Configuration (`.env`)

| Var | Default | Purpose |
| --- | --- | --- |
| `DOCPARSER_VLM_PROVIDER` | `openrouter` | `openrouter` / `openai` / `gemini` / `local` |
| `OPENROUTER_API_KEY` | _required for OpenRouter_ | OpenRouter key (`sk-or-...`) |
| `OPENROUTER_VLM_MODEL` | `anthropic/claude-sonnet-4` | any vision-capable OpenRouter model |
| `OPENROUTER_BASE_URL` | `https://openrouter.ai/api/v1` | override for a proxy |
| `OPENROUTER_REFERER` / `OPENROUTER_TITLE` | repo URL / `docparser` | OpenRouter attribution headers |
| `OPENAI_API_KEY` / `OPENAI_VLM_MODEL` | _required for OpenAI_ / `gpt-4o-mini` | OpenAI provider |
| `GEMINI_API_KEY` / `GEMINI_VLM_MODEL` | _required for Gemini_ / `gemini-1.5-flash` | Gemini provider |
| `DOCPARSER_VLM_BASE_URL` | `http://localhost:11434/v1` | base URL for the `local` provider |
| `DOCPARSER_VLM_API_KEY` / `DOCPARSER_VLM_MODEL` | — / `llava` | key + model for the `local` provider |
| `DOCPARSER_LOCAL_VLM_MODEL` | `Salesforce/blip-image-captioning-large` | model for the `transformers` backend |

## API reference (highlights)

- `WorkspaceLayout(raw_dir, parsed_dir, assets_dir, cache_dir)` —
  dataclass describing where parser output lives. Use `.under(root)` for the
  default `data/raw + data/parsed + data/assets + .cache` layout under a root.
- `parse_docx(source, layout=None, *, captioner=None, write_outputs=True)` →
  payload dict.
- `parse_xlsx(source, layout=None, *, captioner=None, write_outputs=True)` →
  payload dict.
- `parse_pdf(source, layout=None, *, captioner=None, write_outputs=True, extract_images=True, backend="builtin", ocr="off", extract_tables=False)` →
  payload dict. (requires `[pdf]`; backends/OCR/tables require their extras)
- `parse_html(source, layout=None, *, captioner=None, write_outputs=True, use_trafilatura=True)` →
  payload dict. `source` may be a path or `http(s)://` URL. (requires `[html]`)
- `parse_pptx(source, layout=None, *, captioner=None, write_outputs=True)` →
  payload dict. (requires `[pptx]`)
- `parse_epub(source, layout=None, *, captioner=None, write_outputs=True)` →
  payload dict. (requires `[epub]`)
- `parse_text(source, layout=None, ...)` / `parse_csv(source, layout=None, ...)` —
  core parsers for `.txt`/`.md` and `.csv`/`.tsv`.
- `parse_path(source, layout=None, **kwargs)` — dispatches by extension;
  PDF-only kwargs (`backend`, `ocr`, `extract_tables`) are forwarded to PDFs.
- `run_all(layout, *, use_vlm=True, only=None, max_images=None, continue_on_error=False, vlm_provider=None, vlm_model=None, pdf_backend="builtin", ocr="off", extract_tables=False)` —
  walks `layout.raw_dir`, parses everything supported, writes a top-level
  `CORPUS.md` and `data/parsed/corpus.json`.
- `caption_image(image_bytes, *, mime, doc_name, nearby_caption, context, provider=None, model=None, layout=None, ...)` →
  `VLMResult`. (requires `[vlm]`)

## Development

```bash
git clone https://github.com/Research-Commons/docparser
cd docparser
python -m venv .venv && source .venv/bin/activate
pip install -e ".[all,dev]"
pytest -ra
ruff check src tests
mypy
python -m build           # produces dist/*.whl + *.tar.gz
twine check dist/*
```

### Publishing

CI runs lint + mypy + tests on Python 3.10-3.12 and builds the distribution on
every push/PR (`.github/workflows/ci.yml`). Pushing a version tag (e.g.
`v0.2.0`) triggers `.github/workflows/publish.yml`, which builds and uploads to
PyPI via **Trusted Publishing** (OIDC, no stored token) — configure a trusted
publisher for the project on PyPI first. To publish manually instead:

```bash
python -m build
twine check dist/*
twine upload dist/*
```

## License

MIT — see [LICENSE](LICENSE).
