Metadata-Version: 2.4
Name: rc-docparser
Version: 0.2.0
Summary: Convert research literature (.docx, .xlsx, .pdf, .html, .pptx, .epub, .txt, .md, .csv) into structured Markdown + JSON corpora, with optional VLM image semantic captioning.
Project-URL: Homepage, https://github.com/Research-Commons/docparser
Project-URL: Repository, https://github.com/Research-Commons/docparser
Project-URL: Issues, https://github.com/Research-Commons/docparser/issues
Project-URL: Changelog, https://github.com/Research-Commons/docparser/blob/main/CHANGELOG.md
Author-email: Research Commons <shubhankitsingh@researchcommons.ai>
License: MIT License
        
        Copyright (c) 2026 Research Commons
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: corpus,csv,docx,epub,html,literature,markdown,ocr,parser,pdf,pptx,rag,vlm,xlsx
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Text Processing :: Markup
Requires-Python: >=3.10
Requires-Dist: lxml>=5.3.0
Requires-Dist: openpyxl>=3.1.5
Requires-Dist: pillow>=10.4.0
Requires-Dist: python-docx>=1.1.2
Requires-Dist: python-dotenv>=1.0.1
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: tqdm>=4.66.5
Provides-Extra: all
Requires-Dist: beautifulsoup4>=4.12.0; extra == 'all'
Requires-Dist: ebooklib>=0.18; extra == 'all'
Requires-Dist: numpy>=1.24.0; extra == 'all'
Requires-Dist: pdfplumber>=0.11.0; extra == 'all'
Requires-Dist: pymupdf>=1.24.0; extra == 'all'
Requires-Dist: python-pptx>=1.0.0; extra == 'all'
Requires-Dist: rapidocr-onnxruntime>=1.3.0; extra == 'all'
Requires-Dist: requests>=2.32.3; extra == 'all'
Requires-Dist: trafilatura>=1.12.0; extra == 'all'
Provides-Extra: dev
Requires-Dist: build>=1.2.0; extra == 'dev'
Requires-Dist: ebooklib>=0.18; extra == 'dev'
Requires-Dist: mypy>=1.10.0; extra == 'dev'
Requires-Dist: pandas>=2.2.0; extra == 'dev'
Requires-Dist: pdfplumber>=0.11.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: python-pptx>=1.0.0; extra == 'dev'
Requires-Dist: ruff>=0.6.0; extra == 'dev'
Requires-Dist: twine>=5.1.0; extra == 'dev'
Provides-Extra: docling
Requires-Dist: docling>=2.0.0; extra == 'docling'
Provides-Extra: epub
Requires-Dist: beautifulsoup4>=4.12.0; extra == 'epub'
Requires-Dist: ebooklib>=0.18; extra == 'epub'
Provides-Extra: html
Requires-Dist: beautifulsoup4>=4.12.0; extra == 'html'
Requires-Dist: trafilatura>=1.12.0; extra == 'html'
Provides-Extra: localvlm
Requires-Dist: pillow>=10.4.0; extra == 'localvlm'
Requires-Dist: torch>=2.2.0; extra == 'localvlm'
Requires-Dist: transformers>=4.40.0; extra == 'localvlm'
Provides-Extra: marker
Requires-Dist: marker-pdf>=1.0.0; extra == 'marker'
Provides-Extra: ocr
Requires-Dist: numpy>=1.24.0; extra == 'ocr'
Requires-Dist: rapidocr-onnxruntime>=1.3.0; extra == 'ocr'
Provides-Extra: pdf
Requires-Dist: pymupdf>=1.24.0; extra == 'pdf'
Provides-Extra: pptx
Requires-Dist: python-pptx>=1.0.0; extra == 'pptx'
Provides-Extra: pymupdf4llm
Requires-Dist: pymupdf4llm>=0.0.17; extra == 'pymupdf4llm'
Provides-Extra: tables
Requires-Dist: pdfplumber>=0.11.0; extra == 'tables'
Provides-Extra: vlm
Requires-Dist: requests>=2.32.3; extra == 'vlm'
Description-Content-Type: text/markdown

# docparser

Convert research literature (`.docx`, `.xlsx`, `.pdf`, `.html`, `.pptx`,
`.epub`, `.txt`, `.md`, `.csv`) into a clean, reproducible **Markdown + JSON**
corpus, with optional **vision-language captioning** of every embedded figure
via OpenRouter, OpenAI, Gemini, a local server, or a fully-local model.

```text
                              ┌────────────────┐
  data/raw/*.docx             │   docparser    │       data/parsed/<slug>/
  data/raw/*.xlsx             │  - parse_docx  │       document.md
  data/raw/*.pdf      ─────►  │  - parse_xlsx  │ ────► document.json
  data/raw/*.html             │  - parse_pdf   │
  data/raw/*.pptx             │  - parse_html  │       data/assets/<slug>/
  data/raw/*.epub             │  - parse_pptx  │       img-*.png
  data/raw/*.txt|md|csv       │  - parse_epub  │
                              │  - VLM caption │
                              └────────────────┘
```

## Install

```bash
pip install docparser              # core: docx + xlsx + txt/md + csv/tsv
pip install 'docparser[pdf]'       # + PyMuPDF for PDFs
pip install 'docparser[html]'      # + trafilatura + bs4 for HTML
pip install 'docparser[pptx]'      # + python-pptx for PowerPoint
pip install 'docparser[epub]'      # + EbookLib + bs4 for EPUB
pip install 'docparser[vlm]'       # + requests for API VLM captions
pip install 'docparser[all]'       # everything above (recommended)
```

Higher-fidelity / heavier features are separate opt-in extras (so the core
install stays small and MIT):

```bash
pip install 'docparser[tables]'       # + pdfplumber for PDF table extraction
pip install 'docparser[ocr]'          # + rapidocr-onnxruntime for scanned PDFs
pip install 'docparser[pymupdf4llm]'  # PyMuPDF4LLM PDF backend (AGPL/commercial)
pip install 'docparser[docling]'      # IBM Docling PDF backend (MIT)
pip install 'docparser[marker]'       # Datalab Marker PDF backend (GPL-3.0)
pip install 'docparser[localvlm]'     # transformers/torch local captioning
```

`docparser` requires Python 3.10+.

## Quick start (library)

```python
from docparser import WorkspaceLayout, run_all

layout = WorkspaceLayout.under("./project")   # data/raw, data/parsed, data/assets, .cache
layout.ensure()

run_all(layout, use_vlm=False)                # parse everything in data/raw
```

For a single file:

```python
from docparser import parse_path, WorkspaceLayout

layout = WorkspaceLayout.under(".")
payload = parse_path("paper.pdf", layout)
print(payload["stats"])
```

## Quick start (CLI)

```bash
# parse a single file
docparser parse paper.pdf --workspace ./out --no-vlm

# walk a whole directory
docparser parse-all --workspace ./project --no-vlm

# enable VLM captioning (requires OPENROUTER_API_KEY in env or .env)
export OPENROUTER_API_KEY=sk-or-v1-...
docparser parse-all --workspace ./project --max-images 50

# higher-fidelity PDF: pick a backend, OCR scanned pages, extract tables
docparser parse paper.pdf --pdf-backend docling --ocr auto --pdf-tables --no-vlm

# caption with a different provider
docparser parse-all --workspace ./project --vlm-provider openai --vlm-model gpt-4o-mini

docparser version
```

## What gets captured

### `.docx`
- Walks the document body in **document order** (paragraphs + tables + drawings).
- Preserves heading hierarchy (`section_path`) on every block.
- Extracts every embedded image to `data/assets/<slug>/` with a stable
  `img-<seq>-<sha10>.<ext>` name.
- Detects figure/table captions (style `Caption` or text matching
  `Figure 1: …` / `Fig. 1.` / `Table 1.`) and associates the caption with the
  preceding image.
- Captures `context_before` and `context_after` for every image so the VLM has
  document-grounded context.

### `.xlsx`
- Iterates **every sheet, every row, every column**.
- For each cell stores: address, row/col indices, value, openpyxl `data_type`,
  `number_format`, `hyperlink`, `comment`, and the **formula** (from a second
  pass with `data_only=False`).
- Stores `merged_ranges`, `frozen_panes`, and any embedded images.
- Markdown rendering uses the first non-empty row as a header heuristic and
  preserves multi-line cells with `<br>`.

### `.pdf`
- Page-by-page text extraction in reading order via PyMuPDF's blocks API.
- Best-effort heading detection from font size (≥120% of the body-text median
  promotes a line to a heading; bold flag tracked).
- Embedded raster images extracted via `doc.extract_image(xref)`.
- **Pluggable backends** (`backend="pymupdf4llm" | "docling" | "marker"`) route
  conversion to a higher-fidelity engine; their Markdown is normalized into the
  same block schema. Images are still extracted via PyMuPDF.
- **OCR** (`ocr="auto" | "force"`, `[ocr]` extra) recognizes text on scanned /
  low-text pages; OCR'd blocks carry `"ocr": true`.
- **Tables** (`extract_tables=True`, `[tables]` extra) emit real `table` blocks
  via `pdfplumber`.

### `.html`
- Article-grade body extraction via `trafilatura`.
- Plus a **structural** BeautifulSoup walk that emits typed blocks
  (`heading` / `paragraph` / `list` / `table` / `image`) so downstream RAG
  layers can rely on the JSON.

### `.pptx`
- Walks slides in presentation order; each slide becomes a section.
- Emits per-slide headings (slide title), bulleted text frames (with list
  level), tables, pictures, and **speaker notes**.
- Embedded pictures extracted and optionally captioned.

### `.epub`
- Walks the spine in reading order; per-chapter BeautifulSoup structural walk.
- Captures metadata (title/author/language), headings, paragraphs, lists,
  tables, and embedded images (resolved from the EPUB image manifest).

### `.txt` / `.md` and `.csv` / `.tsv` (core, no extras)
- Plain text is split into paragraph blocks; Markdown is passed through and
  also decomposed into heading / list / code / paragraph blocks.
- CSV/TSV: delimiter sniffing, header detection, a Markdown table, and one JSON
  record per row.

### Images (`[vlm]` extra)
Each image is sent to a vision-language model (default provider OpenRouter,
model `anthropic/claude-sonnet-4`) along with its surrounding caption +
context. Any OpenAI-compatible provider works via `--vlm-provider`
(`openrouter` / `openai` / `gemini` / `local`), or use a fully-local
`transformers` model with `--vlm-provider transformers` (`[localvlm]` extra).
The model returns a strict JSON object:

```json
{
  "caption":           "one-sentence figure caption",
  "description":       "2–5 sentence paragraph",
  "visible_text":      "OCR-style transcription",
  "tags":              ["world-model", "diagram", "..."],
  "image_kind":        "diagram | plot | screenshot | photo | equation | table | ...",
  "domain_relevance":  "how this relates to the document's topic"
}
```

Results are cached on disk at `<cache_dir>/vlm/<model>/<sha1>.json`, keyed by
**SHA-1 of the image bytes × model**, so re-runs are free until the source
image bytes change.

## Configuration (`.env`)

| Var | Default | Purpose |
| --- | --- | --- |
| `DOCPARSER_VLM_PROVIDER` | `openrouter` | `openrouter` / `openai` / `gemini` / `local` |
| `OPENROUTER_API_KEY` | _required for OpenRouter_ | OpenRouter key (`sk-or-...`) |
| `OPENROUTER_VLM_MODEL` | `anthropic/claude-sonnet-4` | any vision-capable OpenRouter model |
| `OPENROUTER_BASE_URL` | `https://openrouter.ai/api/v1` | override for a proxy |
| `OPENROUTER_REFERER` / `OPENROUTER_TITLE` | repo URL / `docparser` | OpenRouter attribution headers |
| `OPENAI_API_KEY` / `OPENAI_VLM_MODEL` | _required for OpenAI_ / `gpt-4o-mini` | OpenAI provider |
| `GEMINI_API_KEY` / `GEMINI_VLM_MODEL` | _required for Gemini_ / `gemini-1.5-flash` | Gemini provider |
| `DOCPARSER_VLM_BASE_URL` | `http://localhost:11434/v1` | base URL for the `local` provider |
| `DOCPARSER_VLM_API_KEY` / `DOCPARSER_VLM_MODEL` | — / `llava` | key + model for the `local` provider |
| `DOCPARSER_LOCAL_VLM_MODEL` | `Salesforce/blip-image-captioning-large` | model for the `transformers` backend |

## API reference (highlights)

- `WorkspaceLayout(raw_dir, parsed_dir, assets_dir, cache_dir)` —
  dataclass describing where parser output lives. Use `.under(root)` for the
  default `data/raw + data/parsed + data/assets + .cache` layout under a root.
- `parse_docx(source, layout=None, *, captioner=None, write_outputs=True)` →
  payload dict.
- `parse_xlsx(source, layout=None, *, captioner=None, write_outputs=True)` →
  payload dict.
- `parse_pdf(source, layout=None, *, captioner=None, write_outputs=True, extract_images=True, backend="builtin", ocr="off", extract_tables=False)` →
  payload dict. (requires `[pdf]`; backends/OCR/tables require their extras)
- `parse_html(source, layout=None, *, captioner=None, write_outputs=True, use_trafilatura=True)` →
  payload dict. `source` may be a path or `http(s)://` URL. (requires `[html]`)
- `parse_pptx(source, layout=None, *, captioner=None, write_outputs=True)` →
  payload dict. (requires `[pptx]`)
- `parse_epub(source, layout=None, *, captioner=None, write_outputs=True)` →
  payload dict. (requires `[epub]`)
- `parse_text(source, layout=None, ...)` / `parse_csv(source, layout=None, ...)` —
  core parsers for `.txt`/`.md` and `.csv`/`.tsv`.
- `parse_path(source, layout=None, **kwargs)` — dispatches by extension;
  PDF-only kwargs (`backend`, `ocr`, `extract_tables`) are forwarded to PDFs.
- `run_all(layout, *, use_vlm=True, only=None, max_images=None, continue_on_error=False, vlm_provider=None, vlm_model=None, pdf_backend="builtin", ocr="off", extract_tables=False)` —
  walks `layout.raw_dir`, parses everything supported, writes a top-level
  `CORPUS.md` and `data/parsed/corpus.json`.
- `caption_image(image_bytes, *, mime, doc_name, nearby_caption, context, provider=None, model=None, layout=None, ...)` →
  `VLMResult`. (requires `[vlm]`)

## Development

```bash
git clone https://github.com/Research-Commons/docparser
cd docparser
python -m venv .venv && source .venv/bin/activate
pip install -e ".[all,dev]"
pytest -ra
ruff check src tests
mypy
python -m build           # produces dist/*.whl + *.tar.gz
twine check dist/*
```

### Publishing

CI runs lint + mypy + tests on Python 3.10-3.12 and builds the distribution on
every push/PR (`.github/workflows/ci.yml`). Pushing a version tag (e.g.
`v0.2.0`) triggers `.github/workflows/publish.yml`, which builds and uploads to
PyPI via **Trusted Publishing** (OIDC, no stored token) — configure a trusted
publisher for the project on PyPI first. To publish manually instead:

```bash
python -m build
twine check dist/*
twine upload dist/*
```

## License

MIT — see [LICENSE](LICENSE).
