Metadata-Version: 2.4
Name: bobine
Version: 0.2.0
Summary: Standalone PDF / Office / text → Markdown ingestion engine (extracted from OKFgraph)
Author-email: opticsWolf <opticswolf@protonmail.com>
License-Expression: Apache-2.0 OR MIT
Project-URL: Homepage, https://github.com/opticsWolf/bobine
Project-URL: Repository, https://github.com/opticsWolf/bobine
Project-URL: Documentation, https://github.com/opticsWolf/bobine/blob/main/README.md
Keywords: pdf,markdown,ocr,onnx,latex,ingestion,rapidocr,pdf-oxide
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Markup :: Markdown
Classifier: Topic :: Text Processing :: Markup :: LaTeX
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: LICENSES/Apache-2.0.txt
License-File: LICENSES/MIT.txt
Requires-Dist: Pillow>=10.0
Provides-Extra: pdf-ingest
Requires-Dist: pdf_oxide>=0.2.1; extra == "pdf-ingest"
Requires-Dist: office_oxide>=0.1.8; extra == "pdf-ingest"
Requires-Dist: rapidocr==3.9.2; extra == "pdf-ingest"
Requires-Dist: rapid_layout==1.2.1; extra == "pdf-ingest"
Requires-Dist: rapid_table==3.0.2; extra == "pdf-ingest"
Requires-Dist: numpy>=2.0; extra == "pdf-ingest"
Provides-Extra: formula
Requires-Dist: onnxruntime>=1.17; extra == "formula"
Requires-Dist: tokenizers>=0.13.2; extra == "formula"
Requires-Dist: opencv-python>=4.5; extra == "formula"
Requires-Dist: chardet>=5.0; extra == "formula"
Requires-Dist: requests>=2.28; extra == "formula"
Requires-Dist: PyYAML>=6.0; extra == "formula"
Provides-Extra: markdown
Requires-Dist: mordant>=0.8; extra == "markdown"
Requires-Dist: python-frontmatter>=1.0; extra == "markdown"
Requires-Dist: pyyaml>=6.0; extra == "markdown"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: ruff>=0.9; extra == "dev"
Requires-Dist: reportlab>=4.0; extra == "dev"
Requires-Dist: numpy>=2.0; extra == "dev"
Dynamic: license-file

# bobine

Standalone **PDF / Office / text → Markdown ingestion engine**, extracted from
the OKFgraph project as a self-contained module. Runs on a single
`onnxruntime` wheel with no CUDA-version coupling — RapidAI family + `pdf_oxide`
for PDFs, pure Python for text documents.

Dual-licensed under the terms of either the MIT License or the Apache License,
Version 2.0 — you may choose either (see [LICENSE](LICENSE)).

## Docs

- [**Quick reference**](docs/quickref.md) — install, API, config, common tasks
- [**Architecture**](docs/architecture.md) — modules, data flow, coordinate
  spaces, vendoring, testing strategy
- [**Implementation plan**](docs/IMPLEMENTATION_PLAN.md) — phases, roadmap, status
- [**Parity audit**](docs/PARITY.md) — OKFgraph extraction fidelity + drift catalogue

## Why bobine?

The ingestion pipeline was entangled with the knowledge-graph project it served.
`bobine` moves the whole pipeline — conversion, image staging, markdown
linting, document normalization — into its own package so any consumer (a
graph, a CLI, an MCP server, a batch tool) can reuse it without importing a
database stack.

## Installation

```bash
pip install -e .                # core (Pillow only)
pip install -e ".[pdf-ingest]"  # + pdf_oxide + RapidAI ONNX passes
pip install -e ".[formula]"     # + formula OCR (vendored RapidLaTeXOCR)
pip install -e ".[markdown]"    # + mordant linting + frontmatter parsing
```

Everything is optional: the package imports with zero dependencies and
degrades gracefully (no-op fast paths, clear `RuntimeError`s when a backend
is missing).

## Quick start

```python
from bobine import ConverterConfig, RoutingMode, ingest_document

# One PDF → staged, linted markdown in ./out (images → ./out/_assets,
# links rewritten to okf-asset://<id>)
result = ingest_document(
    "paper.pdf",
    "out",
    config=ConverterConfig(
        routing_mode=RoutingMode.SURGICAL,
    ),
)
print(result.md_path, result.image_count, result.page_count)

# Text documents need no native deps at all
doc = ingest_document("notes.txt", "out")
```

### PDF conversion with ONNX heavy passes

`HybridConverter` routes pages through four modes:

| Mode      | Behaviour                                                                 |
|-----------|---------------------------------------------------------------------------|
| `NEVER`   | Fast path only (pdf_oxide). No ONNX models loaded.                        |
| `AUTO`    | Heuristics per page → full ONNX layout + OCR on flagged pages.            |
| `SURGICAL`| Formula crops via RapidLaTeXOCR only; full pipeline just for scans.       |
| `ALWAYS`  | Every page through the full ONNX layout + OCR pipeline.                   |

```python
from bobine import HybridConverter, ConverterConfig, RoutingMode

conv = HybridConverter(ConverterConfig(routing_mode=RoutingMode.AUTO))
conv.ensure_models()
md = conv.convert_pdf(
    "paper.pdf", work_dir="work", should_continue=lambda: True, on_page=lambda i, n: None
)
conv.close()
```

### Text-type documents

```python
from bobine import load_markdown_document, wrap_thoughts, lint_markdown

doc = load_markdown_document("note.md")  # frontmatter-aware
thought = wrap_thoughts("raw reasoning…", topic="graphs")
fixed = lint_markdown(doc.body, auto_fix=True)  # mordant, guarded
```

## Module layout

```
bobine/
├── __init__.py      public API
├── config.py        ConverterConfig, RoutingMode
├── engine.py        OnnxRapidEngine (lazy ONNX model manager)
├── converter.py     HybridConverter (core PDF/Office pipeline)
├── tables.py        HTML table → GFM pipe-table converter
├── assets.py        okf-asset:// staging for extracted images
├── versions.py      RapidAI version pins + runtime check
├── documents.py     Document model, frontmatter, thoughts wrapper
├── markdown.py      mordant linting (guarded, no-op without it)
├── pipeline.py      convert_to_markdown / stage_images / ingest_document
└── _vendor/         third-party code, vendored with licenses intact
    └── rapid_latex_ocr/   formula OCR (MIT (c) 2023 RapidAI; numpy-2 fixed)
```

### Formula OCR (SURGICAL mode)

The LaTeX formula recognizer is **vendored** (`bobine/_vendor/rapid_latex_ocr/`,
MIT (c) 2023 RapidAI) with the numpy-2 incompatibility fixed upstream never
addressed — no external package needed. Runtime deps come from the
`[formula]` extra; the ONNX models (~179 MB) auto-download on first use from
`github.com/RapidAI/RapidLaTeXOCR/releases/download/v0.0.0/` into
`bobine/_vendor/rapid_latex_ocr/models/` (git-ignored).

Formula regions come from the text layer (TeX math fonts / unicode math
chars), merged **line-aware** so multi-line display equations become one
crop. For text-layer-hostile PDFs (Word/InDesign/OCR output without math
fonts), set `ConverterConfig(formula_layout_fallback=True)` to ask the
layout model for equation regions instead (pulls the `rapid_layout` stack
into SURGICAL mode — off by default).

## Output contract

`ingest_document` produces a directory that a graph/import layer can consume:

- `<stem>.md` — linted markdown with `okf-asset://<id>` image links
- `_assets/<id>.<ext>` — staged image bytes (deduped, concept-scoped ids)

`bobine` never embeds, indexes, or writes to a database. The consumer owns
embedding and storage (in OKFgraph that is `OKFRouter.import_bundle`).

## Testing

```bash
# unit suite (no native backends needed — fake pdf_oxide objects drive the
# converter's routing/splice/ONNX-assembly paths)
pytest

# integration suite (requires bobine[pdf-ingest] + bobine[formula])
pytest -m integration

# coverage + lint
pytest --cov=bobine --cov-report=term-missing
ruff check . && ruff format --check .
```

Markers: `integration` (real pdf_oxide/office_oxide/RapidAI + the PDF corpus)
and `slow` (ONNX runs over real pages)

### Test-PDF corpus

`tests/fixtures/pdf/` holds **trimmed page ranges** from three CC BY 4.0
arXiv papers (solitons physics, splitting-methods math, trust-ML tables) plus
a generated scanned page — see `tests/fixtures/SOURCES.md` for provenance and
attribution. The full untrimmed PDFs are git-ignored under
`tests/fixtures/full_pdfs/` for local tests. The scanned page is regenerable:

```bash
uv run --with reportlab python tests/fixtures/generate_corpus.py
```

The integration tests self-skip when backends are missing, so the bare install
always stays green. CI (`.github/workflows/ci.yml`) runs the core suite on
Python 3.10–3.13 plus an integration job. **169 tests, 92 % coverage** as of
2026-08-09.
## Version pinning

RapidAI packages move fast; `check_rapid_versions()` warns on first import if
an installed version drifts from the known-good list. Silence with
`BOBINE_INGEST_ALLOW_UNPINNED=1` (the legacy `OKFGRAPH_INGEST_ALLOW_UNPINNED`
is still honoured).
