Metadata-Version: 2.4
Name: docx-parse-eval
Version: 0.1.0
Summary: Differential evaluation harness for .docx parsers: reference OOXML extraction vs any document-intelligence output (Docling, ...) on a shared schema
Author-email: Samuel Schmidt <schmidt.l.samuel@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/9s-l-s9/docx-parse-eval
Project-URL: Repository, https://github.com/9s-l-s9/docx-parse-eval
Project-URL: Issues, https://github.com/9s-l-s9/docx-parse-eval/issues
Keywords: docx,docling,document-parsing,evaluation,TEDS,differential-testing
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Markup :: XML
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2
Requires-Dist: python-docx>=1
Requires-Dist: Levenshtein>=0.27
Requires-Dist: apted>=1.0
Requires-Dist: pandas>=2
Requires-Dist: pyarrow>=15
Provides-Extra: test
Requires-Dist: pytest>=8; extra == "test"
Dynamic: license-file

# `docx-parse-eval` — differential evaluation for `.docx` parsers

Differential-testing harness for `.docx` document-intelligence pipelines:
project the source (OOXML reference extraction) and any parser's native output
(Docling `DoclingDocument` JSON, or one adapter per additional model) into one
schema, then diff — conservation metrics, a TEDS/TEDS-Struct table quality
tier, a snapshot drift tier, and a fixture self-test tier beneath them.
Spec: [`evaluation-framework-spec.md`](./evaluation-framework-spec.md).

**Use it when** your `.docx` are too long to eyeball (100+ pages),
confidential (evaluation runs entirely on your machine; the corpus never
enters CI, agents, or third-party services — R8), or parsed *recurringly*
(the snapshot tier answers "did the parser upgrade change my extractions?").
**Skip it** for a few short one-off files, or for scanned PDFs/images — the
approach needs a machine-readable source format, so there is no
gold-by-construction for pixels.

**What a result means:** gold is an independent reading of the same OOXML, so
a fired flag is a guaranteed discrepancy between two readings — a defect on at
least one side, always worth inspecting — not a claim of absolute truth. The
`bless`/`reconcile` steps put a human in the arbitration seat.

## Quick start

The harness never runs Docling itself — produce the `DoclingDocument` JSON
out-of-band, wherever docling is installed (R8/R11):

```sh
docling --to json --output dl/ my_document.docx
```

Install with **uv** (or plain pip in a virtualenv — both verified):

```sh
uv sync                        # dev env: package + pytest/ruff/mypy (uv.lock pinned)
uv run docx-parse-eval --help
pip install path/to/doc-parsing-evaluation-framework   # or classic pip

# 1. reference extraction straight from the OOXML → "silver" draft record
docx-parse-eval bootstrap my_document.docx --out work/

# 2. eyeball the silver record against the document, then bless it
docx-parse-eval bless work/my_document.silver.json --out .   # → gold/my_document.json

# 3. project the Docling output into the same schema; --source binds the
#    prediction to the .docx bytes so compare enforces source identity (R7)
docx-parse-eval predict dl/my_document.json --out pred/ --source my_document.docx

# 4. diff the two sides — non-zero exit iff any defect flag fires
docx-parse-eval compare --gold gold/my_document.json \
                        --pred pred/my_document.docling-adapter.json --out results/

# 5. self-contained HTML page: flagged metrics, side-by-side table grids with
#    per-cell diffs + TEDS, caption pairs (repeat --gold/--pred per doc)
docx-parse-eval report --gold gold/my_document.json \
                       --pred pred/my_document.docling-adapter.json --out report.html
```

Remaining verbs: `reconcile --gold … --draft …` (field-level diff of blessed
gold vs a re-bootstrapped draft, R6), `run --manifest corpus.json --out …`
(compare over a whole corpus, one combined table; manifest is
`[{"doc_id": …, "gold": …, "pred": …}, …]`), and `snapshot <record.json>`
(drift tier: content hash vs stored baseline, non-zero exit on drift).
`compare`/`run` write long-form CSV + Parquet, exit non-zero when any flag
fires (CI gate), and refuse records whose `source_sha256` differ
(`--allow-source-mismatch` overrides). `--mlflow` logs a run when mlflow is
installed, no-op otherwise. Other parsers plug in via `predict --adapter NAME`:
a module with `extract(json_path, **kw) -> EvaluationRecord`, registered under
the `docx_parse_eval.adapters` entry-point group. The same verbs are importable
from Python: `from docx_parse_eval import extract_reference, compare, …`.

## Development environment

The maintained environment is **Guix** (deps pinned in
[`manifest.scm`](./manifest.scm); the profile has `python3`, no bare
`python`); `uv sync` or `pip install .[test]` also work.

```sh
guix shell -m manifest.scm -- python3 -m pytest tests -q            # test suite
guix shell ruff -- ruff check src tests                             # lint
guix shell -m manifest.scm python-mypy -- python3 -m mypy src/docx_parse_eval --ignore-missing-imports
```

## Layout

```
  manifest.scm                 # Guix harness-core deps
  src/docx_parse_eval/
    schema.py                  # EvaluationRecord (Pydantic v2); SCHEMA_VERSION="0.4"
    normalize.py               # shared NFC/whitespace normalisation + tokenisation
    config.py                  # policy decisions + flag thresholds (calibration record)
    io.py                      # JSON records, content hashing, R6 record diff
    teds.py                    # TEDS/TEDS-Struct (PubTabNet metric on python-apted)
    comparator.py              # Script 3 — metrics + flags, model-agnostic
    output.py                  # CSV/Parquet long-form emit + optional MLflow hook
    report.py                  # self-contained HTML failure report
    cli.py                     # bootstrap/bless/reconcile/predict/compare/run/snapshot/report
    adapters/
      ooxml_reference.py       # Script 1 — gold/reference adapter (python-docx)
      docling_adapter.py       # Script 2 — DoclingDocument JSON adapter (zero-dep)
  tests/                       # fixture self-test tier (green/red mutations, hard cases)
```

JSON records are the source of truth (`gold/<doc_id>.json`,
`pred/<doc_id>.<producer>.json`); results emit as CSV + Parquet long-form.

## Status / roadmap

The harness (adapters, comparator, calibration, CLI, report) is complete and
green on fixtures. Applying it to a real corpus is deliberately a local, human
step (R8): bootstrap each document, bless silver → gold, run the parser
out-of-band, `predict`/`compare`, and inspect every flag on the first pass —
any extractor bug found there gets distilled into a new minimal synthetic
fixture rather than debugged on the real document. Flag thresholds in
`config.py` are fixture-calibrated first passes; expect to tune them on first
real-corpus contact. Deferred: HTML export-loss diagnostic, HuggingFace
dataset bundle, per-figure image-description scoring.
