Metadata-Version: 2.4
Name: cyberdocextractor
Version: 0.2.0
Summary: Production-grade document-to-Markdown conversion library with hexagonal architecture.
Author-email: Cyberdyne Corp <oss@cyberdyne.example>
License: MIT
License-File: LICENSE
Keywords: csv,document,docx,extraction,hexagonal,markdown,pdf,xlsx
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Office/Business
Classifier: Topic :: Text Processing :: Markup :: Markdown
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: jinja2>=3.1
Requires-Dist: typing-extensions>=4.12
Provides-Extra: all
Requires-Dist: defusedxml>=0.7; extra == 'all'
Requires-Dist: docling>=2.57; extra == 'all'
Requires-Dist: httpx>=0.27; extra == 'all'
Requires-Dist: openpyxl>=3.1; extra == 'all'
Requires-Dist: pdfplumber>=0.11; extra == 'all'
Requires-Dist: pillow>=10.0; extra == 'all'
Requires-Dist: pymupdf4llm>=0.0.5; extra == 'all'
Requires-Dist: pymupdf>=1.23; extra == 'all'
Requires-Dist: python-docx>=1.1; extra == 'all'
Requires-Dist: python-dotenv>=1.0; extra == 'all'
Requires-Dist: rich>=13.7; extra == 'all'
Requires-Dist: typer>=0.12; extra == 'all'
Requires-Dist: xlrd>=2.0; extra == 'all'
Provides-Extra: cli
Requires-Dist: rich>=13.7; extra == 'cli'
Requires-Dist: typer>=0.12; extra == 'cli'
Provides-Extra: doc
Requires-Dist: docling>=2.57; extra == 'doc'
Provides-Extra: docx
Requires-Dist: defusedxml>=0.7; extra == 'docx'
Requires-Dist: python-docx>=1.1; extra == 'docx'
Provides-Extra: llm
Requires-Dist: httpx>=0.27; extra == 'llm'
Requires-Dist: pillow>=10.0; extra == 'llm'
Requires-Dist: python-dotenv>=1.0; extra == 'llm'
Provides-Extra: pdf
Requires-Dist: pdfplumber>=0.11; extra == 'pdf'
Requires-Dist: pymupdf4llm>=0.0.5; extra == 'pdf'
Requires-Dist: pymupdf>=1.23; extra == 'pdf'
Provides-Extra: xlsx
Requires-Dist: defusedxml>=0.7; extra == 'xlsx'
Requires-Dist: openpyxl>=3.1; extra == 'xlsx'
Requires-Dist: xlrd>=2.0; extra == 'xlsx'
Description-Content-Type: text/markdown

# CyberDocExtractor

[![PyPI version](https://img.shields.io/pypi/v/cyberdocextractor.svg)](https://pypi.org/project/cyberdocextractor/)
[![CI](https://github.com/CyberdyneCorp/CyberDocExtractor/actions/workflows/ci.yml/badge.svg)](https://github.com/CyberdyneCorp/CyberDocExtractor/actions/workflows/ci.yml)
[![Security](https://github.com/CyberdyneCorp/CyberDocExtractor/actions/workflows/security.yml/badge.svg)](https://github.com/CyberdyneCorp/CyberDocExtractor/actions/workflows/security.yml)
[![Docs](https://github.com/CyberdyneCorp/CyberDocExtractor/actions/workflows/docs.yml/badge.svg)](https://github.com/CyberdyneCorp/CyberDocExtractor/actions/workflows/docs.yml)
[![Coverage](https://img.shields.io/badge/coverage-97%25-brightgreen)](./pyproject.toml)
[![Tests](https://img.shields.io/badge/tests-374%20passing-brightgreen)](./tests)
[![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue)](https://www.python.org/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)
[![Typed: mypy strict](https://img.shields.io/badge/typed-mypy%20strict-informational)](./pyproject.toml)

> **Production-grade** Python library for converting **PDF, DOCX, DOC, XLSX,
> XLS, and CSV** documents to Markdown. Multiple precision levels, automatic
> extractor fallback, LLM-assisted image descriptions, and a strict
> hexagonal core.

```bash
pip install cyberdocextractor==0.1.1
```

## Why another document extractor?

Every upstream library (Docling, PyMuPDF, pdfplumber, python-docx, openpyxl,
xlrd, …) has its own model, error semantics, and quirks. CyberDocExtractor
wraps them behind a **single typed domain model**:

- **Hexagonal architecture** — the domain doesn't know what PyMuPDF is.
  Boundaries are enforced by `import-linter` in CI (three contracts, kept
  on every PR).
- **Pluggable extractors** — swap, reorder, or add adapters without touching
  the core. Heavy dependencies are **lazy-imported**, so installing with no
  extras still works.
- **Automatic fallback** — if the preferred extractor fails, the next
  candidate in the policy chain is tried. A corrupt page doesn't abort a
  batch.
- **Deterministic quality scoring** — every conversion reports a 0.0 – 1.0
  score, so downstream RAG / ETL can gate on confidence.
- **Immutable value objects everywhere** — frozen, slotted dataclasses with
  invariants enforced in `__post_init__`; metadata maps wrapped in
  `MappingProxyType` so templates can't corrupt pipeline state.
- **Strict CI** — mypy strict, ruff with pydocstyle, `import-linter`
  architectural contracts, 90% coverage floor (currently 97%), matrix tests
  on Linux + macOS × Python 3.11 / 3.12 / 3.13.

## Installation

| Command | What you get |
|---|---|
| `pip install cyberdocextractor` | Core + CSV (stdlib) |
| `pip install 'cyberdocextractor[pdf]'` | + PyMuPDF / pymupdf4llm / pdfplumber (PDF, 3 precision tiers) |
| `pip install 'cyberdocextractor[doc]'` | + Docling (PDF / DOCX / XLSX at HIGHEST_QUALITY; also powers DOCX) |
| `pip install 'cyberdocextractor[docx]'` | + python-docx + defusedxml (needed by the `.doc` adapter) |
| `pip install 'cyberdocextractor[xlsx]'` | + openpyxl (XLSX) + xlrd (legacy XLS) + defusedxml |
| `pip install 'cyberdocextractor[llm]'` | + httpx + Pillow (LLM image descriptions) |
| `pip install 'cyberdocextractor[cli]'` | + Typer + Rich (`cyberdoc` command) |
| `pip install 'cyberdocextractor[all]'` | Everything above |

The `.doc` adapter additionally requires **LibreOffice** on your `PATH`
(`soffice` or `libreoffice`) — the adapter shells out for the
CFBF → DOCX conversion, then parses the result with `python-docx`.

## 30-second Python

```python
from pathlib import Path
from cyberdocextractor import PrecisionLevel, build_pipeline, load_document

pipeline = build_pipeline()

doc = load_document(Path("report.pdf"), precision=PrecisionLevel.BALANCED)
report = pipeline.convert(doc)

print(report.markdown.text)
print(f"extractor={report.extractor}")
print(f"attempted={list(report.attempted)}")
print(f"quality={report.markdown.quality_score:.2f}")
```

## 30-second CLI

```bash
# Single document to stdout (or -o FILE)
cyberdoc convert report.pdf --show-score

# Batch with glob + precision override
cyberdoc batch inputs/ outputs/ --pattern '*.pdf' --level 3

# Which adapters are available on this machine?
cyberdoc status

# Size / MIME / auto-selected precision (no conversion)
cyberdoc info report.pdf

# List bundled Jinja2 templates
cyberdoc templates

cyberdoc version
```

| Command | Purpose |
|---|---|
| `convert` | Convert one document. Flags: `--level {1,2,3,4}`, `--template NAME`, `--mime MIME`, `--output FILE`, `--show-score`, `--no-fallback`. |
| `batch` | Walk a directory with a glob pattern and write mirrored `.md` files. Failures are captured, not raised — `exit 1` if any file failed. |
| `status` | Rich table of extractor availability per MIME type (PDF / DOCX / DOC / XLSX / XLS / CSV). |
| `info` | Print size, MIME, classification, and the precision that would be auto-selected. |
| `templates` | List available Jinja2 templates. |
| `version` | Print the installed package version. |

Full [CLI reference](./docs/cli.md).

## Architecture

Four layers, one set of import rules, enforced by `import-linter` in CI:

```mermaid
flowchart TB
    CLI["cli — Typer entrypoints"]
    APP["app — ConversionPipeline + stages"]
    INFRA["infra — PyMuPDF, pdfplumber, Docling, openpyxl, xlrd, Jinja2, httpx, LibreOffice"]
    DOMAIN["domain — pure value objects, ports, rules"]

    CLI --> APP
    CLI --> INFRA
    CLI --> DOMAIN
    APP --> DOMAIN
    INFRA --> APP
    INFRA --> DOMAIN

    classDef pure fill:#eef,stroke:#448,stroke-width:2px;
    classDef app fill:#efe,stroke:#484,stroke-width:2px;
    classDef infra fill:#fee,stroke:#844,stroke-width:2px;
    classDef cli fill:#fef,stroke:#848,stroke-width:2px;

    class DOMAIN pure;
    class APP app;
    class INFRA infra;
    class CLI cli;
```

- **Domain** — pure value objects (`Document`, `Block`, `NormalizedDoc`,
  `Markdown`, `ExtractionOutcome`, `TemplateContext`, `ConversionReport`),
  enums, `Protocol` ports, typed errors, pure rules. **Zero external
  dependencies.**
- **Application** — `ConversionPipeline` + five composable stages, loader +
  batch helpers, default `Clock` / `Scorer`.
- **Infrastructure** — one lazy-imported adapter per upstream library.
- **CLI** — Typer commands that parse arguments and delegate.

Three `import-linter` contracts keep the stack honest on every PR:

1. **Hexagonal layers** (layered stack).
2. **Domain is pure** — domain cannot import from app, infra, or cli.
3. **App depends only on domain** — app cannot import infra or cli.

## Conversion pipeline

```mermaid
flowchart LR
    Input[Document] --> Ex[ExtractionStage]
    Ex --> Meta[MetadataStage]
    Meta --> Img[ImageEnrichmentStage]
    Img --> Tab[TableProfilingStage]
    Tab --> Ren[RenderingStage]
    Ren --> Report[ConversionReport]
```

- **ExtractionStage** walks the policy-selected chain with automatic
  fallback; adapters wrap failures in `ExtractionOutcome.fail` rather than
  raising.
- **MetadataStage**, **ImageEnrichmentStage**, **TableProfilingStage** are
  defensive — individual probe / describer / profiler failures are logged
  and skipped, never allowed to fail the conversion.
- **RenderingStage** wraps Jinja2 errors in `RenderingError` and scores the
  final Markdown.

Full deep-dive: [docs/architecture/pipeline.md](./docs/architecture/pipeline.md).

## Precision levels

| Level | Name | Typical extractor | Trade-off |
|---|---|---|---|
| 1 | `FASTEST` | `pymupdf-chunked` | speed > fidelity; no layout analysis |
| 2 | `BALANCED` (default) | `pymupdf4llm` | good text + tables at moderate speed |
| 3 | `TABLE_OPTIMIZED` | `pdfplumber` | best for structured tables |
| 4 | `HIGHEST_QUALITY` | Docling | OCR, images, full layout; slowest |

When you pass `BALANCED` (the default), the policy **auto-selects** based
on document size:

- `< 2 MiB` → `HIGHEST_QUALITY` (small files are cheap to process at full
  fidelity).
- `> 20 MiB` → `FASTEST` (archival scans need throughput).
- otherwise → stays at `BALANCED`.

`document.metadata["has_tables"] = True` promotes to `TABLE_OPTIMIZED` at
any size. Passing any precision other than `BALANCED` bypasses the
auto-resolution — your choice is respected as-is.

Rationale in [ADR 0005](./docs/adr/0005-size-aware-precision.md); disable
via `build_pipeline(size_aware_policy=False)`.

## Format support

| Format | Extension | Extractor(s) | Required extras |
|---|---|---|---|
| CSV | `.csv` | `csv-stdlib` | none |
| PDF | `.pdf` | `pymupdf-chunked`, `pymupdf4llm`, `pdfplumber`, `docling` | `[pdf]` (one of the first three) or `[doc]` (Docling) |
| DOCX | `.docx` | `docling` | `[doc]` |
| **DOC** (legacy) | `.doc` | `libreoffice-doc` | `[docx]` **plus** LibreOffice on PATH |
| XLSX | `.xlsx` | `openpyxl`, `docling` | `[xlsx]` or `[doc]` |
| **XLS** (legacy) | `.xls` | `xlrd` | `[xlsx]` |

`cyberdoc status` shows the matrix for the adapters actually available in
your environment.

## Releases

| Version | Highlights |
|---|---|
| [**0.1.1**](https://pypi.org/project/cyberdocextractor/0.1.1/) | Added `XlrdExtractor` (legacy `.xls`) and `LibreOfficeDocExtractor` (legacy `.doc` via `soffice`). CLI status table surfaces DOC and XLS columns. |
| [**0.1.0**](https://pypi.org/project/cyberdocextractor/0.1.0/) | First public release. Full hexagonal core, six extractors, Jinja2 renderer, heuristic scorer, PDF + Office metadata probes, OpenAI-compatible LLM image describer, Typer CLI. |

Changelog: [docs/changelog.md](./docs/changelog.md). Roadmap
(including known limitations and post-1.0 plans):
[docs/roadmap.md](./docs/roadmap.md).

## Documentation

- [Quickstart](./docs/quickstart.md) — every install flow, first conversion, batch, CLI.
- [Architecture overview](./docs/architecture/overview.md) — layer rules,
  error routing, sequence diagram.
- [Pipeline deep-dive](./docs/architecture/pipeline.md) — per-stage
  semantics, testing recipes, extensibility points.
- [Domain model](./docs/architecture/domain-model.md) — UML + invariant
  tables per value object.
- [Ports reference](./docs/architecture/ports.md) — every `Protocol` + its
  adapters.
- [Writing an extractor](./docs/guides/writing-an-extractor.md) — step-by-step.
- [Writing a template](./docs/guides/writing-a-template.md) — `TemplateContext`
  schema + sandbox notes.
- [LLM image descriptions](./docs/guides/llm-image-descriptions.md) — OpenAI,
  Azure, Ollama, vLLM recipes.
- [Batch conversion](./docs/guides/batch-conversion.md) — per-file failure
  handling + parallelism guidance.
- [ADRs](./docs/adr/index.md) — seven decisions, each with Context /
  Decision / Consequences.

## Development

Prerequisites: Python 3.11+, [uv](https://docs.astral.sh/uv/),
[just](https://just.systems/).

```bash
git clone https://github.com/CyberdyneCorp/CyberDocExtractor.git
cd CyberDocExtractor
just bootstrap    # uv sync --all-extras --group dev --group docs + pre-commit

just check        # format + lint + mypy + import-linter + tests
just test-unit    # unit tests only
just test-bdd     # pytest-bdd scenarios only
just docs-serve   # mkdocs-material live preview
```

See [CONTRIBUTING.md](./CONTRIBUTING.md) for the full contribution
workflow.

## Security

See [SECURITY.md](./SECURITY.md) for the responsible-disclosure policy.
Highlights:

- XML parsing uses `defusedxml` by default (Office metadata probes).
- Jinja2 environments are sandboxed; `autoescape=False` because the output
  format is Markdown.
- Dependencies scanned weekly (Bandit, `pip-audit`, TruffleHog).

## License

MIT — see [LICENSE](./LICENSE).
