Metadata-Version: 2.4
Name: pdf-strikethrough-detect
Version: 0.9.0
Summary: Detect struck-through (deleted) text in PDFs and scanned document images.
Author: Niles Liu
License-Expression: MIT
Project-URL: Homepage, https://github.com/niles-liu/pdf-strikethrough-detect
Project-URL: Repository, https://github.com/niles-liu/pdf-strikethrough-detect
Project-URL: Issues, https://github.com/niles-liu/pdf-strikethrough-detect/issues
Project-URL: Changelog, https://github.com/niles-liu/pdf-strikethrough-detect/blob/main/CHANGELOG.md
Keywords: strikethrough,strikeout,redline,ocr,pdf,document-intelligence,deleted-text,scanned-documents
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Classifier: Topic :: Text Processing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: pillow>=9.0
Requires-Dist: scipy>=1.10
Requires-Dist: onnxruntime>=1.16
Requires-Dist: pymupdf>=1.26
Provides-Extra: markdown
Requires-Dist: pymupdf4llm>=0.0.27; extra == "markdown"
Provides-Extra: rapidocr
Requires-Dist: rapidocr>=3.2; extra == "rapidocr"
Requires-Dist: onnxruntime>=1.16; extra == "rapidocr"
Provides-Extra: tesseract
Requires-Dist: pytesseract>=0.3.10; extra == "tesseract"
Provides-Extra: torch
Requires-Dist: torch>=2.0; extra == "torch"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Dynamic: license-file

# pdf-strikethrough-detect

[![PyPI](https://img.shields.io/pypi/v/pdf-strikethrough-detect.svg)](https://pypi.org/project/pdf-strikethrough-detect/)
[![Python versions](https://img.shields.io/pypi/pyversions/pdf-strikethrough-detect.svg)](https://pypi.org/project/pdf-strikethrough-detect/)
[![CI](https://github.com/niles-liu/pdf-strikethrough-detect/actions/workflows/ci.yml/badge.svg)](https://github.com/niles-liu/pdf-strikethrough-detect/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://github.com/niles-liu/pdf-strikethrough-detect/blob/main/LICENSE)

Detect **struck-through (deleted) text** in PDFs and scanned document images.

Strikethrough detection is a surprisingly unserved niche: most "redline"/diff tools assume clean
born-digital PDFs and fall apart on scans — which is the real-world case. `pdf-strikethrough-detect`
handles both, does the hard part (scanned images) with a tiny CPU model, and makes no cloud calls.

```python
import pdf_strikethrough as st

# born-digital PDF — exact, no OCR
for w in st.strikethroughs_in_pdf("contract.pdf"):
    print(w["page"], repr(w["chars"]), "partial" if w["partial"] else "full")
```

## Install

```bash
pip install pdf-strikethrough-detect
```

Pure pip, no system binaries required: the CNN runs on ONNX Runtime (CPU) and the ~318 KB model
ships inside the wheel. Extras:

```bash
pip install "pdf-strikethrough-detect[markdown]"    # clean_markdown() via pymupdf4llm
pip install "pdf-strikethrough-detect[rapidocr]"    # free scanned-word OCR backend (no binary)
pip install "pdf-strikethrough-detect[tesseract]"   # word-level OCR (also needs the tesseract binary)
pip install "pdf-strikethrough-detect[torch]"       # optional .pt CNN fallback for dev/retraining
```

The CNN ships as ONNX and needs nothing extra. `[torch]` is only for development — the loader
prefers `strike_verdict_cnn.onnx`, falling back to a `strike_verdict_cnn.pt` checkpoint if you
point `PDF_STRIKETHROUGH_MODEL_DIR` at one. Regenerate the shipped ONNX from a trained checkpoint
with [`tools/export_model.py`](tools/export_model.py).

## Native / born-digital PDFs — exact

In a born-digital PDF a strikethrough is a *vector drawing* (a line or thin rect over the text),
so detection is exact ground truth — no OCR, no model, no guessing. Both vector-rule and
filled-rect strikethrough styles are handled.

```python
import pdf_strikethrough as st

for w in st.strikethroughs_in_pdf("contract.pdf"):
    print(w["page"], repr(w["chars"]))          # 'chars' = the struck substring
print(st.clean_markdown("contract.pdf"))        # surviving text, deletions removed (needs [markdown])
```

Each native record: `{page, text, chars, char_span, partial, bbox_frac, tier, coverage, verdict,
final}` (`tier` is `"vector"`, `"flag"`, or `"annot"`). Vector records also carry
`stroke_color`/`stroke_width` (the paint + thickness of the dominant stroke — red = opposing
counsel is evidence); annotation records carry `annot_author`/`annot_created`/`annot_modified`/
`annot_color`/`annot_id` (the redline's "who and when"). Scanned records replace `coverage` with
the evidence that decided them — `score`, `cnn_prob`, `cnn_agrees`, `conf` — and `tier` is
`"auto"`/`"review"`. The keys are documented as `TypedDict`s in `pdf_strikethrough.types`
(`StruckWord`, `DetectResult`, `Passage`); the package ships `py.typed`, so type checkers see them.
Partial strikes (`semi-` of `semi-monthly`) are resolved to a char range. `bbox_frac` is in
fractions of the rendered page (rotation-aware), so it maps directly onto a rendered pixmap.

Three native detectors, all **base-PyMuPDF only** (no pymupdf4llm), selected by `method`:

```python
st.strikethroughs_in_pdf("contract.pdf", method="vector")  # stroke geometry (default) —
                                                           #   precise partial-char spans + color/width
st.strikethroughs_in_pdf("contract.pdf", method="flag")    # MuPDF's FZ_STEXT_STRIKEOUT signal —
                                                           #   also catches font-attribute strikes
st.strikethroughs_in_pdf("contract.pdf", method="annot")   # explicit /StrikeOut annotations —
                                                           #   Acrobat/Preview redlines + forensics
st.strikethroughs_in_pdf("contract.pdf", method="both")    # union of all three — maximum recall
```

**Validated across domains.** On 10 public redline PDFs (federal regulatory redlines from the
Copyright Office, FDIC, CEQ and EPA; California privacy-regulation redlines; a municipal
development code — 54.7k struck words), **99.8% of vector detections are independently confirmed
by MuPDF's strikeout signal** (92.5–100% per document; 8 of 10 ≥99.9%), and the flag method adds
~2.6% more words (font-attribute strikes and edge cases) — use `method="both"` to capture them. `pymupdf4llm` is not used for detection at all; it is only an
optional `[markdown]` extra for richer layout in `clean_markdown()`.

## Any PDF — routed per page, scanned pages use OCR + CNN

```python
import pdf_strikethrough as st
from pdf_strikethrough.ocr import rapidocr_backend
from pdf_strikethrough.scanned import ScanConfig

res = st.detect_pdf("mixed.pdf",
                    ocr=rapidocr_backend(),                 # for scanned pages
                    scan_config=ScanConfig.confidence_free())

struck   = [w for w in res["words"] if w["final"]]          # struck words (boxes, char spans)
markdown = res["markdown"]                                  # deletions as ~~struck~~
clean    = res["clean_text"]                                # surviving text, deletions removed
passages = res["passages"]                                  # grouped deletion sections
```

`detect_pdf` classifies each page native-vs-scanned, runs the exact path on native pages
(`method="vector"|"flag"|"annot"|"both"`) and the geometry→OCR→CNN pipeline on scanned ones, and
assembles `markdown` / `clean_text` / `passages` for **both** page kinds from its own strike
decisions (so the text and the word records always agree — no dependence on an external markdown
engine). Already have an Azure Document Intelligence result? Pass `di_result=...` (the REST JSON
dict, an `{'analyzeResult': ...}` envelope, or `sdk_result.as_dict()`) to skip re-OCR and use
DI's word boxes.

Scanned pages with no OCR backend raise `OcrRequiredError` by default; pass
`on_missing_ocr="skip"` to skip them (with a warning in `res["warnings"]`) and still get
everything from the native pages. Password-protected PDFs raise `EncryptedPdfError`.

> `clean_markdown()` remains a separate, higher-fidelity **native-only** path that borrows
> pymupdf4llm's layout (headings, paragraphs). `detect_pdf`'s `markdown` is layout-plain but works
> uniformly on scanned pages too.

### Choosing an OCR backend

The geometry + CNN carry the detection and are **OCR-independent**; OCR only supplies word boxes
to attribute strikes to, plus a confidence prior. Benchmarked by rasterizing born-digital redline
pages into image-only "scans" and taking the native detector's strikes on the born-digital
originals as ground truth (3 documents, 24 pages, 2,170 known strikes; reproduce with
[`benchmarks/scanned_recovery.py`](benchmarks/scanned_recovery.py)):

| Backend | Setup | Strike recovery | Word granularity |
|---|---|---|---|
| Azure Document Intelligence | cloud, paid | **95%** of known strikes | exact word boxes |
| **RapidOCR** | `pip`, no binary | **97%** of known strikes | ~4× coarser (phrase-level) |
| Tesseract | needs system binary | not benchmarked here | genuine word-level |

Use `ScanConfig.confidence_free()` with RapidOCR (its confidences cluster near 1.0 and don't
separate struck from clean text); the default `ScanConfig()` is calibrated to Azure DI, whose
struck words drop to 0.43–0.94. Across backends the scanned path recovers **95–97% of the exact
native strike set** — the geometry + CNN, not the OCR engine, carry the detection.

## Beyond PDFs — images, Word docs, cloud OCR

The same detection reaches inputs that never were a born-digital PDF:

```python
# Raster image files (.png/.jpg/.tiff, incl. multi-page TIFF) — photos, faxes, scans.
# Every frame is a scanned page, so it needs OCR; DPI comes from image metadata (else 200).
res = st.detect_image_file("scan.tiff", ocr=rapidocr_backend())

# Word .docx — strike formatting (w:strike/w:dstrike) + tracked deletions (w:del, with author/date).
# No OCR, no geometry: records use tier="docx" and a `para` index. Stdlib-only, no extra needed.
for w in st.strikethroughs_in_docx("contract.docx"):
    print(w["para"], repr(w["chars"]), w["docx_change"], w.get("docx_author"))

# AWS Textract / Google Document AI — neither flags strikethrough; add the layer on top.
# Convert a pre-fetched result to {page: [Word]} and feed it as words_by_page (works on images too).
res = st.detect_pdf("doc.pdf", words_by_page=st.words_from_textract(textract_response))
res = st.detect_pdf("doc.pdf", words_by_page=st.words_from_docai(docai_document))
```

**Provenance mode for RAG.** Struck text entering a vector index is a real failure mode — the
retriever surfaces a sentence the document *deleted*. `clean_text` removes it; `provenance_text`
instead keeps each deletion as a marker so it's recorded-as-deleted, not silently dropped:

```python
res = st.detect_pdf("contract.pdf")
res["clean_text"]           # 'The fee is billed monthly.'
st.provenance_text(res)     # 'The fee is [deleted: the old rate of 5%] billed monthly.'
```

## Low-level building blocks

```python
gray = st.render_page_gray(doc[0], dpi=200)     # always returns a HxW uint8 grayscale array

lines = st.strike_lines(gray, dpi=200)          # OCR-free stroke geometry (strike/underline/rule)
                                                # pass the dpi the image was rendered/scanned at

# word boxes are PAGE FRACTIONS in [0,1], origin top-left — not pixels
p = st.score_word(gray, (0.12, 0.34, 0.38, 0.36))   # CNN strike probability (0..1)

from pdf_strikethrough.ocr import Word
# detect_scanned_image accepts your own image; RGB / float arrays are coerced to grayscale uint8
recs = st.detect_scanned_image(gray, [Word("foo", (0.12, 0.34, 0.38, 0.36), 0.6)])

# visual overlay — render each struck page with the strikes boxed (red=full, orange=partial)
for pg in st.render_overlay("contract.pdf", dpi=150):
    pg["image"].save(f"overlay-p{pg['page']}.png")     # or st.save_overlays(src, "out/")
```

**Logging.** Diagnostics (page routing, native method + record counts, OCR/CNN timings) are
emitted at `DEBUG` under the `pdf_strikethrough` logger — silent by default (a `NullHandler` is
attached). Opt in with `logging.getLogger("pdf_strikethrough").setLevel(logging.DEBUG)` and a
handler; `warnings` stay reserved for caller-facing hazards.

## CLI

```bash
pdf-strikethrough detect contract.pdf                       # native pages (scanned pages are
                                                            #   skipped with a warning)
pdf-strikethrough detect scan.pdf --ocr rapidocr            # include scanned pages
pdf-strikethrough detect scan.tiff --ocr rapidocr           # a raster image file (or multi-page TIFF)
pdf-strikethrough detect contract.docx                      # a Word doc (strike + tracked deletions)
pdf-strikethrough detect scan.pdf --di-result di.json       # use a pre-fetched Azure DI result
pdf-strikethrough detect scan.pdf --textract-result t.json  # ...or AWS Textract / Google DocAI
pdf-strikethrough detect doc.pdf --provenance prov.txt      # deletions kept as [deleted: ...] markers
pdf-strikethrough detect doc.pdf --method both              # max-recall native detection
pdf-strikethrough detect doc.pdf --method annot             # explicit /StrikeOut annotations only
pdf-strikethrough detect doc.pdf --pages 1-5,12             # only these pages (1-based)
pdf-strikethrough detect doc.pdf --json out.json            # full struck words + passages + evidence
pdf-strikethrough detect doc.pdf --json -                   # ...or stream JSON to stdout
pdf-strikethrough detect doc.pdf --clean-text clean.txt     # surviving text, deletions removed
pdf-strikethrough detect doc.pdf --markdown marked.md       # deletions as ~~struck~~
pdf-strikethrough detect doc.pdf --overlay out/             # page images with strikes boxed
pdf-strikethrough detect scan.pdf --ocr rapidocr --dump-crops crops/  # export scored crops to label
pdf-strikethrough detect doc.pdf --fail-if-found            # exit 3 if any strike found (CI gate)
cat doc.pdf | pdf-strikethrough detect -                    # read the PDF from stdin
pdf-strikethrough --version
```

**Batch mode.** Pass several files, a directory, or a glob and each is processed independently —
JSONL output (one result object per line), `--jobs N` to spread files across processes, and one
bad file never aborts the run (its line carries an `error` key):

```bash
pdf-strikethrough detect ./contracts/ --jsonl out.jsonl        # every document in a directory
pdf-strikethrough detect *.pdf --jobs 4 --jsonl -              # glob, 4 workers, stream to stdout
pdf-strikethrough detect ./contracts/ --fail-if-found          # CI gate over a whole tree (exit 3)
```

Per-file output flags (`--markdown`/`--overlay`/`--clean-text`/`--provenance`/cloud results/
`--pages`) are single-file only; in batch mode use `--jsonl`.

Other flags: `--dpi` (raster DPI for scanned pages, default 200; read from image metadata for image
files — very-high-dpi input is normalized to 200 internally, see below), `--overlay-dpi` (overlay
render DPI, default 150), `--limit` (max words in plain output), `--jobs` / `--jsonl` (batch),
`--scan-config auto|azure-di|confidence-free`, `--markdown`, `--docai-result`. Exit codes: `0` ok,
`1` usage/file error (batch: ≥1 file errored), `2` encrypted / OCR required, `3` `--fail-if-found`
matched. Full help: `pdf-strikethrough detect -h`.

## Examples

Runnable, dependency-light scripts are in [`examples/`](examples/) — they generate their own
sample PDFs (no assets to download) so every snippet is copy-paste runnable:

```bash
python examples/native_quickstart.py     # build a redline PDF, detect strikes, print clean text
python examples/scanned_quickstart.py    # rasterize it to a "scan", run OCR + CNN  ([rapidocr])
python examples/overlay_quickstart.py     # print strike evidence + write a before/after overlay
python examples/rag_provenance.py         # clean_text vs provenance_text ([deleted: ...] markers)
```

## How it works

- **Native**: merged horizontal vector strokes through a word's middle band (excludes under/over-
  lines); coverage ≥ 50% → struck, partials resolved to a char range.
- **Scanned geometry** (`lines.py`): per-angle morphological opening extracts stroke *fragments*,
  collinear fragments are stitched, then strict filters (spine fill, stroke run-thickness,
  angle/length) separate real strikes from bold crossbars and serif-glyph chains.
- **CNN** (`cnn.py`, StrikeNet, 79k params): resolves pixel-ambiguous cases — a thin strike over
  an ascender-less word is pixel-identical to a glyph chain, and only a learned model tells them
  apart. Ships as ONNX; set `PDF_STRIKETHROUGH_MODEL_DIR` to use your own weights.
- **Attribution** (`scanned.py`): assigns strokes to OCR words with char spans and full/partial
  resolution, plus a visual-row "orphan" pass for words the detector's stroke evidence missed.

**Resolution & resource limits.** The geometry tunables and the CNN's crop are calibrated at
200 dpi; scanned rasters above ~300 dpi are downsampled to 200 internally (accuracy-neutral — extra
resolution buys nothing but cost). Output boxes are page fractions, so this is invisible to results.
A hard raster budget caps a single page at 128 Mpix, so a hostile huge-mediabox page is
auto-downsampled with a warning instead of OOMing the process (see [`SECURITY.md`](SECURITY.md)).

**Operating points.** The CNN's struck/clean decision threshold is an explicit choice, not a
buried constant. `ScanConfig.recall_first()` biases toward catching every deletion (legal / audit
review — never miss a struck word); `ScanConfig.precision_first()` biases toward never dropping
live text as if deleted (RAG / indexing). Calibrate the threshold from your own labeled data with
`pdf_strikethrough.calibration` (`threshold_for_recall`, `threshold_for_precision`,
`conformal_threshold` for a distribution-free recall floor) and pass it as
`recall_first(cnn_p_hi=…)`.

**Improving the model.** `--dump-crops DIR` (or `st.dump_crops`) exports every crop the pipeline
scored plus its verdict as a labeling set; label it, retrain with
[`training/train_strikenet.py`](training/), and load your weights via
`PDF_STRIKETHROUGH_MODEL_DIR`, `cnn.set_model_dir`, or `cnn.ensure_model(url, sha256, …)`
(digest-verified download). A ["contribute a failing page"](.github/ISSUE_TEMPLATE/failing-page.md)
report feeds the same loop, and [`demo/`](demo/) is a drag-and-drop Gradio app for trying it — live
at [**huggingface.co/spaces/niles-liu/strikethrough-demo**](https://huggingface.co/spaces/niles-liu/strikethrough-demo).

**Hosted weights.** The shipped StrikeNet is also published on the Hugging Face Hub
([`niles-liu/strikenet`](https://huggingface.co/niles-liu/strikenet)); the digest is verified before
the graph is ever loaded, so a tampered host can't swap the model:

```python
import pdf_strikethrough as st
st.ensure_model(
    "https://huggingface.co/niles-liu/strikenet/resolve/main/strike_verdict_cnn.onnx",
    "fac2c51baaa75ee782196bdfe7452638cb48c7deddb21163b1ac6a0a72ae4457",
    meta_url="https://huggingface.co/niles-liu/strikenet/resolve/main/strike_verdict_cnn.meta.json",
)
```

**Scope — validated scripts & layouts.** Detection assumes **horizontal, left-to-right** text; a
strike is a near-horizontal stroke through a word's middle band. Reading-order assembly is
column-aware (two-column pages read down each column; tables read across rows).

| Case | Status |
|---|---|
| Latin / Western, horizontal | validated (10-PDF redline benchmark, 54.7k struck words) |
| CJK, **horizontal** set (native + scanned) | detected — strikes are horizontal regardless of script (regression-tested) |
| Two-column / newspaper layout, struck table rows, hyphenated-line-break strikes | handled |
| **Vertical** writing modes (CJK/Mongolian), RTL strike axes | out of scope (roadmap) |
| Handwritten / freehand strikes | untested (geometry assumes near-straight strokes) |

RapidOCR (the recommended scanned backend) is strongest on Chinese, so horizontal-CJK scans work
end to end today; full vertical-text support is future work.

**Handwritten strikes** are the main known gap: the geometry gate looks for a near-straight,
near-horizontal stroke, and the CNN was trained on rendered digital strikes, so a wavy pen
scribble or a heavy cross-out may be missed. If you hit one, a
[failing-page report](.github/ISSUE_TEMPLATE/failing-page.md) with `--dump-crops` output is the
fastest way to turn it into a fix — it becomes both a regression test and training data.

## API surface

The top-level package (`import pdf_strikethrough as st`) exports ~37 names; the most-used:

| Group | Names |
|---|---|
| High-level | `strikethroughs_in_pdf`, `clean_markdown`, `provenance_text`, `detect_pdf`, `detect_image_file`, `detect_scanned_image`, `strikethroughs_in_docx`, `open_pdf`, `render_page_gray`, `render_overlay`, `save_overlays` |
| Native detectors | `native_page_strikes`, `native_flag_strikes`, `native_annot_strikes`, `native_doc_strikes`, `page_strikes`, `native_markdown`, `strip_struck_markdown` |
| Scanned geometry + classifier | `strike_lines`, `ink_mask`, `to_gray_u8`, `analyze_scanned_page`, `ScanConfig`, `classify_page_source`, `apply_cnn_verdict` |
| CNN | `score_word`, `score_crops`, `std_crop`, `word_crop_px`, `verdict_of`, `get_model_meta`, `ensure_model` |
| Active learning + calibration | `dump_crops`, `pdf_strikethrough.calibration` (`threshold_for_recall`, `threshold_for_precision`, `conformal_threshold`, `pr_curve`) |
| OCR (backends + cloud-result adapters) | `Word`, `rapidocr_backend`, `tesseract_backend`, `words_from_azure_di`, `words_from_textract`, `words_from_docai` |
| Errors | `OcrRequiredError`, `EncryptedPdfError` |
| Types | `StruckWord`, `DetectResult`, `Passage` (in `pdf_strikethrough.types`) |

Everything is docstringed; `help(st.detect_pdf)` is the reference.

## Contributing, security, citation

- Development setup, running the tests, and regenerating the model: [`CONTRIBUTING.md`](CONTRIBUTING.md).
- This package parses untrusted PDFs — reporting policy in [`SECURITY.md`](SECURITY.md).
- Citing it in research: [`CITATION.cff`](CITATION.cff).

## License

MIT.
