Metadata-Version: 2.5
Name: neurotic-docx-bench
Version: 1.0.0
Summary: A benchmark for manipulating docx files and assessing their fidelity to Microsoft Word(R), the oracle.
Project-URL: Homepage, https://github.com/jandira-tech/neurotic_docx_bench
Project-URL: Repository, https://github.com/jandira-tech/neurotic_docx_bench
Project-URL: Issues, https://github.com/jandira-tech/neurotic_docx_bench/issues
Project-URL: Changelog, https://github.com/jandira-tech/neurotic_docx_bench/blob/main/CHANGELOG.md
Project-URL: Fixtures, https://huggingface.co/datasets/arthrod/neurotic_docx_bench-fixtures
Author-email: Arthur Souza Rodrigues <arthursrodrigues@gmail.com>
License-Expression: AGPL-3.0-only
License-File: LICENSE
Keywords: benchmark,docx,microsoft-word,ooxml,pdf,redline,tracked-changes
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: GNU Affero General Public License v3
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Office/Business :: Office Suites
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Text Processing :: Markup :: XML
Classifier: Typing :: Typed
Requires-Python: >=3.14
Requires-Dist: aiofiles>=25.1.0
Requires-Dist: docx-revisions>=0.1.5
Requires-Dist: huggingface-hub>=2.0
Requires-Dist: lxml>=6.0
Requires-Dist: numpy>=2.0
Requires-Dist: pillow>=10.0
Requires-Dist: polars>=1.44.2
Requires-Dist: pydantic>=2.11
Requires-Dist: pymupdf>=1.24
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: scikit-image<0.28,>=0.24
Requires-Dist: scipy>=1.13
Requires-Dist: typer>=0.12
Provides-Extra: competitors
Requires-Dist: nupunkt==0.6.0; extra == 'competitors'
Requires-Dist: playwright>=1.61.0; extra == 'competitors'
Requires-Dist: redlines[levenshtein,nupunkt]==0.6.1; extra == 'competitors'
Requires-Dist: superdoc-sdk==2.16.0; extra == 'competitors'
Provides-Extra: gpu
Requires-Dist: torch>=2.5; extra == 'gpu'
Description-Content-Type: text/markdown

> **See every page side by side: [jandira-tech.github.io/neurotic_docx_bench](https://jandira-tech.github.io/neurotic_docx_bench/)**  
> DOCX to PDF across engines, and redlines against Word ([/redlines/](https://jandira-tech.github.io/neurotic_docx_bench/redlines/)), scored per page.  
> Navigator: [DOCX to PDF](https://jandira-tech.github.io/neurotic_docx_bench/) ·
> [redlines vs Word](https://jandira-tech.github.io/neurotic_docx_bench/redlines/) (Jubarte, Docxodus, SuperDoc; redlines, accepted, rejected) ·
> [speed](https://jandira-tech.github.io/neurotic_docx_bench/speed/)

# neurotic_docx_bench

Pixel scores of DOCX tools against Microsoft Word oracles. Neurotic: it refuses
to round, to drop failures, or to trust a renderer it has not pinned.

## Who maintains this

Maintained by Arthur Souza Rodrigues (Jandira Technologies), who also writes
Jubarte, one of the ranked tools. Jubarte rows are marked † and follow the same
rules. Every place the harness touches another tool's code is listed in
[`docs/VENDOR_NOTES.md`](docs/VENDOR_NOTES.md).

| | |
| --- | --- |
| **Scores** | 0 to 100 per document |
| **Redline oracle** | Word renders both sides: Word's render of the candidate DOCX against Word's render of Word's own compare. Only Word-rendered rows are ranked; LibreOffice-rendered rows are history |
| **DOCX to PDF oracle** | SHA-pinned Word-export PDFs (`pdf_accepted_word`, `pdf_redlines_randomized`, `pdf_source`, `pdf_source_randomized`) |
| **Second lens** | `docxide_metrics`: docxide-pdf's own Jaccard / text-boundary metrics on the same fixtures, and the same two columns on every redline row |
| **Stores** | `results/bench.jsonl` (fidelity), `results/converters.jsonl` (DOCX to PDF), `results/speed.jsonl` + `results/redline_speed_bench/` (speed), `results/archive/` (history only) |
| **Results** | [`RESULTS.md`](RESULTS.md) (headline) and [`RESULTS_DETAILED.md`](RESULTS_DETAILED.md) (history, paired comparisons, methodology), both generated |
| **Visual report** | `runs/<run>/report.html` |
| **Speed** | [`docs/SPEED.md`](docs/SPEED.md) |

One row per tool in every headline table: its latest eligible run in the current comparability group (same document set, renderer and scorer). No best pin, no best-of-N. Author-affiliated tools (Jubarte) are marked and follow the same rules. Compare rows only within one table.

```bash
uv run bench report            # regenerates RESULTS.md, RESULTS_DETAILED.md and the vendor table below
uv run bench report --check    # exit 1 when the published views are stale (CI)
```

For the ranked redline benchmarks Word renders both the candidate and the oracle DOCX, so the score compares like with like. Legacy rows rendered with LibreOffice 26.2.4.2 stay in the history; for those, re-rendering the same DOCX is byte-identical on one build (`results/noise_floor.json`). The `oracle-identity` calibration row (`bench calibrate`) is Word's own DOCX through the candidate pipeline and must score 100; the `null-baseline` row is the base document unchanged, the floor a redline tool must beat.

Ranked rows today: DOCX to PDF, the docxide metrics, redlines vs Word's compare,
and accepted/rejected changes. Every other benchmark is waiting for a
Word-rendered run and shows history only.

## Limitations

- The score is pixel similarity to Microsoft Word's output. It has not been
  validated against human judgment of redline quality. A redline can be correct
  and still lay out differently from Word's compare. The Lens health table in
  [`RESULTS_DETAILED.md`](RESULTS_DETAILED.md) shows the pixel and functional
  lenses disagreeing on 1% to 29% of documents, depending on the tool.
- The oracle is one build of Word for Mac. Word-rendered rows can only be
  reproduced on a machine with Word; CI cannot regenerate them.
- Adapters for other tools are written by the Jubarte author. A low score can be
  the adapter's fault; [`docs/VENDOR_NOTES.md`](docs/VENDOR_NOTES.md) records
  the known cases, including adapter failures first found by a tool's maintainer.
- Failures count as zero. That mixes "the tool could not run on this file" with
  "the tool ran badly"; both the ITT and scored-only columns are printed for
  that reason.
- Corpus: DOCX files from [superdoc-dev/docx-corpus](https://huggingface.co/datasets/superdoc-dev/docx-corpus),
  collected from the public web and made available under ODC-By 1.0. ODC-By
  covers the collection, not copyright in the individual files. A rights holder
  who wants a document removed can write to contact@arthur.law; it is removed
  from the fixtures and future runs.
- The sealed holdout has never been run ("no holdout runs recorded yet" in
  [`RESULTS_DETAILED.md`](RESULTS_DETAILED.md)).

## Disputing a score

Open an issue. A harness fix that raises a competitor's score ships with the
same urgency as one that raises Jubarte's (standing rule 3 in
[`docs/VENDOR_NOTES.md`](docs/VENDOR_NOTES.md)).

## Results visibility

[`RESULTS.md`](RESULTS.md) is the headline: one table per benchmark, intervals, calibration rows, and the reason for every row that is not ranked. [`RESULTS_DETAILED.md`](RESULTS_DETAILED.md) holds every row in the stores grouped by comparability group with its eligibility verdict, pairwise comparisons, lens-health alarms, and the methodology. Rows that predate provenance stamping, holdout-only runs and retracted runs live in `results/archive/` with a manifest and appear only in the history. The three-way [`renderer corpus`](corpus/no_comments_pdf_was_generated_by_word/renderer_corpus/README.md) contains Word, Jubarte, and docxide-pdf PDFs for the same 398 source DOCX files.

The 2026-10-14 Wordcraft CLI cohort (400 comparisons and 400 conversions against fresh Word 16.115 oracles) is reported on its own in [`results/wordcraft_20261014_fresh/REPORT.md`](results/wordcraft_20261014_fresh/REPORT.md). Besides Wordcraft 0.6.1 and 0.7.0 it covers, on the same fixtures, jubarte 0.11.0 / 0.12.0, docxodus 12.6.5 / 13.0.0, folio-core 0.60.0, stemma 0.6.0 and superdoc-sdk 2.20.0 (comparisons), and LibreOffice, office2pdf, pdfitdown, rdocx, docxide-pdf, libreoffice_convert_rust and PyMuPDF Pro (conversions). Its sample and oracles differ from the stores above, so its numbers are not ranked against them.

## Benchmarks

Compare vendors only within one table. LibreOffice scores and Playwright scores are separate measurements.

| Benchmark | Input | Oracle |
| --- | --- | --- |
| **`script_redlines`** | `base.docx` + `next.docx` → tool redline DOCX | Word redline PDF |
| **`accepted_changes`** | Accept all `w:ins`/`w:del` on the tool redline | Word redline with changes accepted |
| **`roundtrip`** | Self-diff or the tool’s roundtrip path | Original source PDF (identity = 100) |
| **`visual_rendering`** | Source DOCX in the vendor web editor | `pdf_source` |
| **`visual_redlines`** | Word redline DOCX in the vendor web editor | `pdf_redlines_word` |
| **`visual_accepted_changes`** | Accepted Word redline in the vendor web editor | `pdf_accepted_word` |
| **`docx_to_pdf`** | Accepted Word redline DOCX + randomized redline DOCX | SHA-pinned `pdf_accepted_word` + `pdf_redlines_randomized` |
| **`docx_to_pdf_no_redline_docs`** | Source DOCX + randomized source DOCX | SHA-pinned `pdf_source` + `pdf_source_randomized` |
| **`docxide_metrics`** | The `docx_to_pdf_no_redline_docs` inputs, scored with docxide-pdf's Jaccard / text-boundary metrics at 150 DPI | Same SHA-pinned `pdf_source` + `pdf_source_randomized` |

`visual_*` loads Word's DOCX in the editor, not the tool's own redline. Generator package and editor package are separate pins.

Pins: [`bench.yaml`](bench.yaml). Tool identities: [`bench.registry.yaml`](bench.registry.yaml) (the table below is generated from it).

<!-- VENDORS-START -->
| Tool | Role | Engine | Author-affiliated | Note |
| --- | --- | --- | --- | --- |
| [Wordcraft CLI](https://github.com/storytold/wordcraft) | generator | wordcraft-cli |  |  |
| [jubarte 0.11.0](https://github.com/jandira-tech/jubarte-redlines) | generator | jubarte-redlines | yes |  |
| [jubarte 0.12.0](https://github.com/jandira-tech/jubarte-redlines) | generator | jubarte-redlines | yes |  |
| [docxodus 12.6.5](https://github.com/JSv4/docxodus) | generator | docxodus |  |  |
| [Wordcraft CLI 0.7.0](https://github.com/storytold/wordcraft) | generator | wordcraft-cli |  |  |
| [docxodus 13.0.0](https://github.com/JSv4/docxodus) | generator | docxodus |  |  |
| [folio-core 0.60.0](https://github.com/stella/folio) | generator | @stll/folio-core |  |  |
| [stemma 0.6.0](https://github.com/stemma-sh/stemma) | generator | stemma |  |  |
| [superdoc-sdk 2.20.0](https://github.com/Harbour-Enterprises/SuperDoc) | generator | superdoc-sdk |  |  |
| [docxide-pdf 0.18.4](https://github.com/sverrejb/docxide-pdf) | converter | docxide-pdf |  |  |
| [rdocx 0.16.0](https://github.com/tensorbee/rdocx) | converter | rdocx |  |  |
| [office2pdf 0.8.1](https://github.com/developer0hye/office2pdf) | converter | office2pdf (Typst) |  |  |
| [pdfitdown 4.2.0](https://github.com/AstraBert/PdfItDown) | converter | office2pdf (Typst) |  | Delegates Office formats to office2pdf; same engine, listed for completeness. |
| [libreoffice_convert_rust 0.1.0](https://gitcode.com/dnrops/libreoffice_convert_rust) | converter | LibreOffice |  |  |
| [PyMuPDF Pro 1.28.2](https://pymupdf.io/pro) | converter | pymupdf-pro |  | Unlicensed (Restricted Mode): converts only the first 3 pages, so it is scored on the at-most-3-page set. |
| [jubarte (lossless)](https://github.com/jandira-tech/jubarte-redlines) | generator | jubarte-final | yes |  |
| [jubarte (ast)](https://github.com/jandira-tech/jubarte-redlines) | generator | jubarte-final | yes |  |
| [jubarte-rust](https://github.com/jandira-tech/jubarte-redlines) | generator | jubarte-redlines | yes |  |
| [jubarte-wasm](https://github.com/jandira-tech/jubarte-redlines) | generator | jubarte-redlines | yes |  |
| [docxodus](https://github.com/JSv4/docxodus) | generator | docxodus |  |  |
| [docxodus (C#)](https://github.com/JSv4/docxodus) | generator | docxodus |  | C# build of docxodus; speed benchmark only. |
| [folio](https://github.com/stella/folio) | generator | @stll/folio-core |  |  |
| [superdoc](https://github.com/Harbour-Enterprises/SuperDoc) | generator | superdoc-sdk |  |  |
| [redlines](https://github.com/houfu/redlines) | generator | redlines |  |  |
| [superdoc-redlines](https://github.com/yuch85/superdoc-redlines) | generator | superdoc-redlines |  |  |
| [stemma](https://github.com/stemma-sh/stemma) | generator | stemma |  |  |
| [safe-docx](https://github.com/UseJunior/safe-docx) | generator | safe-docx |  |  |
| [docxodus (viewer)](https://github.com/JSv4/react-docxodus-viewer) | editor | react-docxodus-viewer |  |  |
| [folio (viewer)](https://github.com/stella/folio) | editor | @stll/folio-react |  |  |
| [superdoc (editor)](https://github.com/Harbour-Enterprises/SuperDoc) | editor | superdoc |  |  |
| [jubarte](https://github.com/jandira-tech/jubarte-redlines) | converter | jubarte-redlines | yes |  |
| [rdocx](https://github.com/tensorbee/rdocx) | converter | rdocx |  |  |
| [office2pdf](https://github.com/developer0hye/office2pdf) | converter | office2pdf (Typst) |  |  |
| [pdfitdown](https://github.com/AstraBert/PdfItDown) | converter | office2pdf (Typst) |  | Delegates Office formats to office2pdf; same engine, listed for completeness. |
| [doxx](https://github.com/bgreenwell/doxx) | converter | doxx |  | No PDF export; listed as not applicable, never ranked. |
| [libreoffice_convert_rust](https://gitcode.com/dnrops/libreoffice_convert_rust) | converter | LibreOffice |  |  |
| [soffice](https://www.libreoffice.org/) | converter | LibreOffice |  |  |
| [dxpdf](https://github.com/nerdy-pro/dxpdf) | converter | dxpdf |  |  |
| [docxide-pdf](https://github.com/sverrejb/docxide-pdf) | converter | docxide-pdf |  |  |
| [PyMuPDF Pro](https://pymupdf.io/pro) | converter | pymupdf-pro |  | Unlicensed (Restricted Mode): converts only the first 3 pages, so it is scored on the at-most-3-page set. |
<!-- VENDORS-END -->

---

## Install

Version 1.0.0 is built for PyPI as `neurotic-docx-bench` but not uploaded yet; until it is, install
from a checkout (`uv tool install .`, or `pip install .`). macOS and Linux only (the hub download locks
files with `fcntl`).

```bash
pip install neurotic-docx-bench            # once 1.0.0 is on PyPI; or: uv tool install neurotic-docx-bench
bench fixtures list v1                     # the fixture set, its hub commit, cases per test
bench fixtures get v1                      # 14,513 files (2.5 GB) into corpus/word, each sha256-checked
bench fixtures get v1 --test redline       # one test's files only (the corpus tables always come along)
```

Version 1 of the package carries fixture set **v1**: every file on the hub dataset
[`arthrod/neurotic_docx_bench-fixtures`](https://huggingface.co/datasets/arthrod/neurotic_docx_bench-fixtures)
whose oracle Microsoft Word made, for every test the bench runs, pinned to dataset commit
`984e64d869803abea841e66364f7b82bb30b3a70`:

| test | a case is | cases |
|---|---|---:|
| `docx_to_pdf` | a document and Word's PDF of it | 2,925 |
| `redline` | base and next documents, Word's compare of them and Word's PDF of the compare | 3,502 |
| `accept_all` | a Word compare, Word's Accept All of it and Word's PDF of the result | 400 |
| `reject_all` | a Word compare, Word's Reject All of it and Word's PDF of the result | 400 |

The wheel ships the set's manifest (`neurotic_docx_bench/data/fixture_sets/v1.csv.gz`: test, case,
state, role, path, sha256, size, git blob id), never the documents. `bench fixtures get` reads exactly
those files at the pinned commit, checks the hub's listing against the manifest before downloading,
and every downloaded file against its sha256 after. From Python: `neurotic_docx_bench.fixture_sets`
(`load`, `rows`, `files`, `get`). LibreOffice renders, superseded Word renders and the one compare
Word left without a PDF are not in v1. Generators for the competitor tools are the `competitors`
extra (`pip install 'neurotic-docx-bench[competitors]'`).

## Quick start (from a checkout)

Python 3.14 (`uv`), Node 26 + pnpm, LibreOffice 26.2.4.2 on `PATH`.

```bash
uv sync
pnpm install --frozen-lockfile   # workspace: root pin tree + the utils/ engine fallbacks + live-bench frontend
# (only for the visual_* playwright benchmarks:) cd harness/folio-viewer && bun install

uv run bench run --only jubarte-final-lossless --limit 5
uv run bench run
uv run bench docx-to-pdf --tool jubarte --tool rdocx --tool office2pdf --tool pdfitdown --tool doxx
```

Each `bench run`: resolve `tool_version` → generate → render → score → append `results/bench.jsonl` → gate vs snapshot.

```bash
uv run bench run --only docxodus --limit 5 --no-emit
uv run bench accept-scores jubarte
uv run bench render <docx-dir> <work-dir> -b soffice
uv run bench compare <candidate-pdfs> <oracle-pdfs> --tool name
```

### Try one fixture with your own tool

`bench try` scores any tool on one fixture of the tryout set (100 pairs drawn from the Word-exported corpus, `corpus/tryout/tryout_100.csv`) against the Word oracle, next to jubarte's precomputed output for the same pair. Nothing is written under `results/`.

```bash
uv run bench try list
uv run bench try run --random --task redline --tool "mytool {base} {next} -o {out}" --json out/try.json
uv run bench try run --fixture <pair_stem> --task convert --tool "mytool {input} -o {out}" --renderer soffice
uv run bench try fetch <pair_stem> --dest tryout_dl && uv run bench try run --root tryout_dl --fixture <pair_stem> --tool "..."
```

`--renderer passthrough` (default) expects the tool to write the PDF itself; `soffice` and `word` render a `.docx` output first. `--against` takes another command template or a PDF path instead of jubarte. `--seed` repeats a random pick. `scripts/tryout_jubarte.py` regenerates jubarte's outputs for the set (`bench try build-set` redraws it).

### The Word corpus

Word is the source of truth for PDFs. Everything Word produced lives under `corpus/word/`, tracked in this repository (docx and PDFs included) and the one source of truth. It was built by copy (`bench corpus build --origins`) from Word's working folders and, with `--fixtures`, from the jubarte-first `_fixtures` folder; those folders are not part of the repository. Only what Word finished is in: docx Word could not open, blacklisted stems and the compares touching them, rejected compares and, in the sets that are Word render runs, documents Word did not render to PDF are left out and listed per set in `PROVENANCE.json`. A PDF whose producer is not Word (LibreOffice, a tool's own writer) is refused.

The tree is laid out by provider and by the state of the docx, read from its XML:

```
corpus/word/
  clean/                      docx/  pdf/  pdf_prior/
  tracking_without_comments/  docx/  pdf/  pdf_prior/
  with_comments_clean/        docx/  pdf/  pdf_prior/
  with_comments_tracking/     docx/  pdf/  pdf_prior/
  documents.csv  comparisons.csv  pools/  notices/  PROVENANCE.json  README.md  MANIFEST.sha256.json
```

`pdf/` holds the current Word render of each docx, `pdf_prior/` the render an earlier Word build made of the same docx, under the same name. A comparison lives in the state of the compared docx.

Every file carries the id of the docx it represents, the first 10 hex digits of the sha256 of the docx bytes. A document is `<id>_<name>`; a Word compare of two documents is `<idA>_<a>__vs__<idB>_<b>_redline_<idC>`, idC being the compare's own id; the Word PDF shares its docx's stem. A tool's output for either is the Word stem plus `_<tool>`, and the scorer keys a candidate by stripping that suffix (`pipeline.redline_key`, `pipeline.render_key`). Names are lower-cased, folded to `[a-z0-9_-]` and cut at 48 characters; the original names are kept in `notices/RENAMED.csv` and in the `names` column of the tables. Byte-identical docx from several origins are one file with every origin name and set recorded on it; two different docx sharing an id fail the build.

| set | what |
| --- | --- |
| `sources_500` | docx-corpus documents with their Word PDF; the stems a later Word build re-rendered keep the earlier render under `pdf_prior/` |
| `en_pairs_500` | the English base/next documents; the first Word pass is the reference, the second pass only fills the stems the first lacks (`filled` in the provenance) |
| `redlines_a100_b10`, `redlines_en_500` | Word compares of those documents (`<base>__vs__<next>`) with the Word PDF of each compared document |
| `word_based`, `word_based_randomized`, `word_redlines_superdoc` | the documents of those folders and Word's compares of their pairs (resolved through `centralized_mapping.csv`) with the September 2026 Word renders of the compares |
| `word_based_0926`, `word_based_randomized_0926`, `word_redlines_superdoc_0926` | the September 2026 compare run: fresh compares of the same pairs with their Word PDFs, other bytes, so other documents |
| `nocomments`, `nocomments_randomized` | the July 2026 Word run with comments stripped: documents and compares with their Word PDFs |
| `fixtures_originals`, `fixtures_word_compares` | the jubarte-first fixtures and Word's compares of them (no Word PDF exists of either); built only with `--fixtures` |

`documents.csv` and `comparisons.csv` list every entry with its id, state, paths, sets, names, sha256 and PDF producer; `comparisons.csv` also carries `key`, `base_id` and `next_id`. `pools/<set>_pairs.csv` (key, base, next, docx, pdf, state) and `pools/<set>_renders.csv` are what `bench.yaml` points a run at. `notices/` holds the rename record, the naming note, the ODC-By-1.0 license and the origins' own notes and logs. `PROVENANCE.json` records origins, counts, what was left out and why, states and the docset id of each set; the tree is pinned by `MANIFEST.sha256.json`. docx and PDF files are gitignored and travel with `bench fixtures upload`; the tables, pools, notices, provenance and manifest are tracked.

```bash
uv run bench corpus build --dry-run     # plan against the origins, copy nothing
uv run bench corpus build               # copy (a clone on APFS), write tables, pools, notices, provenance, manifest
uv run bench corpus build --fixtures /path/to/jubarte-first/_fixtures   # the fixtures sets too
uv run bench corpus check               # verify corpus/word against its manifest (exit 1 on drift)
uv run bench corpus list                # the sets with counts, docset ids and what was left out
```

A destination file whose bytes changed is refused unless `--force`; `--only <set>` builds a subset (a comparison set needs its sources set in the selection).

---

## Scoring

1. Match candidate PDF to oracle PDF by `<base>_<next>` (redlines) or plain stem (`docx_to_pdf`, roundtrip). The denominator is the oracle document set (`results/docsets.json`); a document with no candidate output enters at 0 (intent-to-treat).
2. Raster each page at 144 DPI.
3. Score each page 0 to 100 as a weighted sum: SSIM 40, ink F1 20, edge IoU 15, colour 15, blob 10. A document scores 0.7 times its page mean plus 0.3 times its worst page; "Perfect (100)" counts documents within 1e-6 of 100.

Scoring core is a verbatim lift of [superdoc-visual-benchmarks](https://github.com/superdoc-dev/superdoc-visual-benchmarks); `tests/test_parity.py` checks byte-identical behaviour. Pages present on only one side enter at 0, ink-weighted (`pagefair-v2`), for script_redlines, accepted_changes and roundtrip; the visual_* benchmarks rank on the raw score because cross-engine repagination is expected there. Rank ties: adjacent rows tie when the paired bootstrap interval of their median difference includes 0.

Regression gate (CI tooling, not methodology): 100 always passes; a per-document drop vs the accepted snapshot warns; an aggregate mean or median drop beyond `eps = max(1e-4, 3 sigma)` from `results/noise_floor.json` fails. Promote with `uv run bench accept-scores <tool>`.

### Content cache

`bench run` reuses renders, page rasters and scored rows across runs from `.bench-cache/` (next to `results/`, git-ignored). Entries are keyed by content, never by path: candidate sha256, oracle sha256 (and base sha256 when a base PDF joins the row), DPI, renderer id, and a scorer fingerprint (sha256 over `score.py`, `score_v2.py`, `page_metrics.py`, `pipeline.py`, `raster.py`, the MuPDF build and the cache schema). Editing any scoring source or upgrading PyMuPDF invalidates every score entry on its own; nothing needs clearing by hand.

A row restored from the cache carries `cached: true` and no `raster_ns`/`score_ns`; a restored render has `cached: true` and no `duration_ns`. Timings (`render_s`, `raster_s`, `score_s`, the `visual_*` render-speed stats) therefore come only from fresh work. Rasters are still written under the run's work dir so galleries and diagnostics read them as before.

```bash
uv run bench run --no-cache        # score everything fresh (BENCH_NO_CACHE=1 does the same)
BENCH_CACHE_DIR=/fast/disk uv run bench run
uv run bench cache                 # root, entry counts, size, scorer fingerprint
uv run bench cache --clear
```

`bench compare` never reads the cache.

### Profiling the pipeline

`bench profile` times every stage (generate, render, raster, score) on a seeded sample of documents and prints, per run and benchmark, each stage's count, total, mean, median, p95, max and share of the run, plus the run's wall time and renderer. It is one uncached pass and never a result: nothing is appended to `results/bench.jsonl`, no gate runs, and recorded runs are not skipped.

```bash
uv run bench profile --run jubarte                     # 10 documents, seed 0
uv run bench profile --run jubarte --sample 25 --seed 3 --json out/profile.json
uv run bench profile --roundtrip --accept-compare      # time those stages too
```

The same `--sample` and `--seed` pick the same documents, so two profiles (before and after a scorer change, or CPU against a torch device) compare like for like.

### Torch backend for the scorer kernels

The scorer's two heaviest kernels, the CIEDE2000 colour distance and SSIM, have a torch port behind `--device` on `bench run`, `bench profile` and `bench compare` (the `gpu` extra: `uv sync --extra gpu`). The default is the skimage path, which the parity tests lock byte for byte; the torch path reproduces it within float32 tolerance and is checked against it by `tests/test_kernels.py`.

```bash
uv run bench profile --run jubarte --device mps      # Apple silicon
uv run bench run --device auto                       # cuda, then mps, else the numpy path
```

`auto` is silent when no GPU is present; `cpu`, `mps` and `cuda` fall back to the numpy path with one warning when torch or that device is unavailable. The device is exported as `BENCH_DEVICE` for the command's duration only, so worker processes inherit it; the backend id (`numpy`, `torch-mps`, ...) is part of the content-cache score key, the profile report (`scorer_backend`) and the hardware stamp on every result. On CPU the torch kernels are no faster per core than skimage (each pool worker is limited to its share of the cores), so `--device cpu` is for parity checks; the gain is on a GPU.

---

## Speed methodology

Data: `results/speed.jsonl`, `results/redline_speed_bench/`. Detail: [`docs/SPEED.md`](docs/SPEED.md).

| Mode | Measures |
| --- | --- |
| Warm in-process (`*-inproc`) | Compare work in one long-lived process |
| CLI | Process spawn + init + compare |
| WASM | In-process after load |
| Microbench | Small N × reps |

Large-N (`scripts/redline_speed_bench.ts`): up to 1000 unique `.docx` → 5000 pairs (Mulberry32 seed 42), warmup, `performance.now()`, failures excluded.

```bash
node --import tsx scripts/speed-bench.ts --pairs 30 --reps 3 --out results/speed.jsonl
pnpm run redline-speed-bench:warm
```

---

## Project map

```
bench.yaml                 # runs, pins, oracles
corpus/word/               # what Word produced, by document state (bench corpus build)
corpus/word_based/         # redline DOCX + LibreOffice oracle PDFs
corpus/no_comments_pdf_was_generated_by_word/  # Word-exported PDFs (docx_to_pdf)
corpus/no_comments_pdf_was_generated_by_word/renderer_corpus/  # Word/Jubarte/docxide PDFs
bench.registry.yaml        # tool identities (one id per spelling in every store)
results/bench.jsonl        # fidelity store (LFS)
results/converters.jsonl   # DOCX to PDF store (one line per report and tool)
results/docsets.json       # document set per benchmark (the ITT denominators)
results/retractions.jsonl  # runs that must never be ranked, with reasons
results/archive/           # legacy rows and their manifest (history only)
src/neurotic_docx_bench/
src/neurotic_docx_bench/ledger/         # registry, rows, policy, stats, tables, build (`bench report`)
scripts/
```

[`AGENTS.md`](AGENTS.md) · [`CONTRIBUTING.md`](CONTRIBUTING.md)

---

## Credits

This bench exists because these projects publish their work. Every tool is measured as published, at the version pinned in [`bench.yaml`](bench.yaml); a score is a measurement, not an endorsement. Licenses are as each project's repository states them on 2026-10-28.

### Tools measured

| Project | Author | License | What the bench uses it for |
| --- | --- | --- | --- |
| [jubarte-redlines](https://github.com/jandira-tech/jubarte-redlines) | Jandira Tech (author-affiliated) | AGPL-3.0 | redline generator (native CLI, WASM, lossless and AST lanes); DOCX→PDF converter |
| [Wordcraft CLI](https://github.com/storytold/wordcraft) | storytold (ArtCraft) | Apache-2.0 | redline generator |
| [docxodus](https://github.com/JSv4/docxodus) | JSv4 | MIT | redline generator (WASM `compareDocuments`, and its C# build in the speed lane) |
| [react-docxodus-viewer](https://github.com/JSv4/react-docxodus-viewer) | JSv4 | MIT | viewer for the `visual_*` benchmarks |
| [folio](https://github.com/stella/folio) (`@stll/folio-core`, `@stll/folio-react`) | stella | Apache-2.0 | redline generator; viewer for the `visual_*` benchmarks |
| [SuperDoc](https://github.com/Harbour-Enterprises/SuperDoc) (`superdoc-sdk`, `superdoc`) | Harbour Enterprises | AGPL-3.0 | redline generator (Python SDK); editor for the `visual_*` benchmarks |
| [superdoc-redlines](https://github.com/yuch85/superdoc-redlines) | yuch85 | Apache-2.0 | redline generator (SuperDoc headless extract→align→apply) |
| [docx-redline-js](https://github.com/AnsonLai/docx-redline-js) | AnsonLai | MIT | dropped in #88 (2026-10-28): migration-kit sibling dependency unavailable; historical results retained |
| [redlines](https://github.com/houfu/redlines) | Houfu Ang | MIT | text-diff redline baseline |
| [stemma](https://github.com/stemma-sh/stemma) | stemma-sh | Apache-2.0 | redline generator (`stemma compare`) |
| [safe-docx](https://github.com/UseJunior/safe-docx) | UseJunior | Apache-2.0 | redline generator (`compareDocuments`) |
| [docxide-pdf](https://github.com/sverrejb/docxide-pdf) | Sverre Johann Bjørke (sverrejb) | Apache-2.0 | DOCX→PDF converter; source of the second scorer's metrics (below) |
| [rdocx](https://github.com/tensorbee/rdocx) | tensorbee | MIT | DOCX→PDF converter |
| [office2pdf](https://github.com/developer0hye/office2pdf) | Yonghye Kwon (developer0hye) | Apache-2.0 | DOCX→PDF converter (Typst) |
| [PdfItDown](https://github.com/AstraBert/PdfItDown) | Clelia (Astra) Bertelli | MIT | DOCX→PDF converter (delegates to office2pdf) |
| [dxpdf](https://github.com/nerdy-pro/dxpdf) | Nerdy Production | MIT | DOCX→PDF converter |
| [doxx](https://github.com/bgreenwell/doxx) | Brandon Greenwell | MIT | listed converter; it has no PDF export, so it is never ranked |
| [libreoffice_convert_rust](https://gitcode.com/dnrops/libreoffice_convert_rust) | dnrops | MIT OR Apache-2.0 | DOCX→PDF converter (drives LibreOffice) |
| [PyMuPDF Pro](https://pymupdf.io/pro) | Artifex | commercial | DOCX→PDF converter, run unlicensed (first 3 pages only) |
| [LibreOffice](https://www.libreoffice.org/) | The Document Foundation | MPL-2.0 | the pinned renderer of the soffice lane and a converter |

### What the bench is built on

| Project | Author | License | What the bench uses it for |
| --- | --- | --- | --- |
| [superdoc-visual-benchmarks](https://github.com/superdoc-dev/superdoc-visual-benchmarks) | SuperDoc | AGPL-3.0 | the scoring core (`score.py`, `diff.py`, `raster.py`, reports), lifted verbatim |
| [docxide-pdf `tests/common/`](https://github.com/sverrejb/docxide-pdf) | sverrejb | Apache-2.0 | Jaccard and text-boundary page metrics, ported to `page_metrics.py` and held to upstream's frozen numbers by `tests/test_page_metrics.py` |
| [docx-revisions](https://github.com/balalofernandez/docx-revisions) | Alvaro Fernandez Moreno | MIT | Accept All / Reject All (`bench accept` / `reject`) |
| [nupunkt](https://github.com/alea-institute/nupunkt) | ALEA Institute | MIT | sentence segmentation for the `redlines` lane |
| [docx-corpus](https://huggingface.co/datasets/superdoc-dev/docx-corpus) | SuperDoc | see the dataset card | source documents for `bench try remote` and the corpus |
| [PyMuPDF](https://github.com/pymupdf/PyMuPDF) | Artifex | AGPL-3.0 | PDF rasterisation for both scorers |
| [scikit-image](https://scikit-image.org/) | the scikit-image team | BSD-3-Clause | CIEDE2000 and SSIM in the scorer |
| Microsoft Word | Microsoft | proprietary | the oracle: every reference redline, Accept/Reject and PDF. Word is a Microsoft trademark |

## License

Scoring core derived from [superdoc-visual-benchmarks](https://github.com/superdoc-dev/superdoc-visual-benchmarks). This repository is **AGPL-3.0-only**. See [`LICENSE`](LICENSE).

Published scores are measurements, not endorsements. Microsoft Word is a trademark of Microsoft.
