Metadata-Version: 2.5
Name: autosxtract
Version: 0.4.0
Summary: Cascading PDF text extraction — cheapest step first, stopping at the first one that suffices
Project-URL: Homepage, https://github.com/autosxtract/autosxtract
Project-URL: Documentation, https://github.com/autosxtract/autosxtract#readme
Project-URL: Repository, https://github.com/autosxtract/autosxtract
Project-URL: Issues, https://github.com/autosxtract/autosxtract/issues
Author: AutosXtract
Maintainer: AutosXtract
License: MIT
License-File: LICENSE
Keywords: apple-vision,cascade,ocr,paddleocr,pdf,text-extraction
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Classifier: Topic :: Text Processing
Requires-Python: >=3.11
Requires-Dist: asn1crypto>=1.5
Requires-Dist: numpy>=1.24
Requires-Dist: onnxruntime>=1.17
Requires-Dist: opencv-python-headless>=4.9
Requires-Dist: pillow>=10.0
Requires-Dist: pydantic>=2.0
Requires-Dist: pymupdf>=1.24
Requires-Dist: pyobjc-framework-vision>=10.0; sys_platform == 'darwin'
Requires-Dist: rapidocr>=3.0
Requires-Dist: striprtf>=0.0.26
Provides-Extra: all
Requires-Dist: docling>=2.0; extra == 'all'
Requires-Dist: httpx>=0.27; extra == 'all'
Requires-Dist: ocrmac>=1.0; (sys_platform == 'darwin') and extra == 'all'
Requires-Dist: onnxruntime>=1.17; extra == 'all'
Requires-Dist: onnxtr>=0.6; extra == 'all'
Requires-Dist: paddleocr>=3.7; extra == 'all'
Requires-Dist: pytesseract>=0.3.10; extra == 'all'
Requires-Dist: rapidocr>=3.0; extra == 'all'
Provides-Extra: apple
Requires-Dist: ocrmac>=1.0; (sys_platform == 'darwin') and extra == 'apple'
Provides-Extra: dev
Requires-Dist: mypy>=1.8; extra == 'dev'
Requires-Dist: pre-commit>=3.7; extra == 'dev'
Requires-Dist: pytest-cov>=4.1; extra == 'dev'
Requires-Dist: pytest>=7.4; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: docling
Requires-Dist: docling>=2.0; extra == 'docling'
Provides-Extra: onnx
Requires-Dist: onnxtr>=0.6; extra == 'onnx'
Provides-Extra: paddle
Requires-Dist: onnxruntime>=1.17; extra == 'paddle'
Requires-Dist: rapidocr>=3.0; extra == 'paddle'
Provides-Extra: paddleocr
Requires-Dist: paddleocr>=3.7; extra == 'paddleocr'
Provides-Extra: remote
Requires-Dist: httpx>=0.27; extra == 'remote'
Provides-Extra: veto
Requires-Dist: pytesseract>=0.3.10; extra == 'veto'
Description-Content-Type: text/markdown

# AutosXtract

Cascading text extraction from PDFs: every document descends steps from the
cheapest to the most expensive and stops at the first one that produces
acceptable text.

```bash
pip install autosxtract
```

```python
from autosxtract import Cascade

r = Cascade().extract_file("document.pdf")
print(r.text)
```

That is it. **One command and one class, identical on macOS, Linux and
Windows** — the OCR engine arrives already chosen by the machine, and there is
no extra to memorise and no configuration to write.

> Extraction **does no networking**. No external worker, no SSH tunnel, no
> remote endpoint, no third-party service call. The only connection in the whole
> package is the one-off download of the PP-OCRv6 weights — unnecessary once
> they are on disk.

---

## Installation

`pip` evaluates platform markers **on the installing machine**, so the same
`pip install autosxtract` brings different things on each system:

| | macOS | Linux and Windows |
|---|---|---|
| OCR engine | **Apple Vision**, in-process | **PP-OCRv6 tiny** on ONNX |
| what pip brings | `pyobjc-framework-Vision` | `rapidocr`, `onnxruntime` |
| common to both | `pymupdf`, `pydantic`, `numpy`, `pillow`, `opencv-python-headless`, `striprtf`, `asn1crypto` | same |
| resulting cascade | `native → vision` | `native → paddle` |
| installed size | ~330 MB | ~570 MB |

None of that needs to be known in order to use it. To check what **your**
machine resolved:

```bash
autosxtract diagnose
```

On a Linux box with the default install the output is this, verbatim:

```
autosxtract 0.4.0
machine    Linux (x86_64)
resources  72 usable core(s)
automatic parallelism: 4 page(s) per document, 4 document(s) in flight (aggregate cap 144)

engines:
  [ ] vision       vision requires Darwin  (single queue: ignores threads)
  [ ] ocrmac       ocrmac requires Darwin  (single queue: ignores threads)
  [x] paddle       PP-OCRv6 tiny
  [ ] onnx         onnx unavailable: No module named 'onnxtr'; install with pip install autosxtract[onnx]
  [ ] tesseract    tesseract unavailable: No module named 'pytesseract'; install with pip install autosxtract[veto]

cascade:   native -> paddle

models:    /home/you/.cache/autosxtract  (complete)
```

On a Mac, `vision` is marked as well and the last line reads
`cascade: native -> vision -> paddle` — PP-OCRv6 is not Vision's substitute
there, it is the step BELOW it, and the second independent reading the
agreement gate needs in order to fire at all. The other platforms' engines are the ones that
then show why they are out. If you see **`cascade: native` alone**, no OCR
engine was installed — the diagnosis says why and how to fix it, and it is the
only case where a scanned PDF comes out empty.

### Why Apple gets its own treatment

It is not a platform preference, it is a measurement. `VNRecognizeTextRequest`
runs on-device on a small, specialised model with acceleration that has no x86
equivalent — **92% of the words and 100% of the numeric anchors** preserved at
the median of 60 audited documents, against 53% and 75% for the best local
substitute. Where it exists, it is the right step.

And it runs **inside your process**: no server, no worker, no tunnel. A
previous version of this pipeline reached Vision on a Mac over a reverse SSH
tunnel, and that worker going down silently degraded the text — 488 documents
re-extracted down the worse path and 28,239 characters lost without anyone
noticing. That failure mode does not exist here.

### The choice happens twice

| when | who decides | what |
|---|---|---|
| at `pip install` | a platform marker in the package | what is **installed** |
| at the first extraction | `platform.py` plus the engine registry | what is **available** |

Both are necessary. The second is what stops the library from blowing up when
the marker did not match: a Docker image built on a Mac and run on Linux, an
install with `--no-deps`, a lockfile generated for another system, an
incomplete pyobjc. In any of those the cascade degrades with a warning instead
of breaking.

### Extras

None is needed for ordinary use. They exist to ask for **the other** engine —
your platform's already arrived:

| extra | what for |
|---|---|
| `paddleocr` | the three official PP-OCRv6 tiers, INT8 and a fine-tuned recogniser |
| `paddle` | nothing new — PP-OCRv6 is mandatory everywhere; kept so old commands work |
| `apple` | `ocrmac`, a safety net for Vision's direct path |
| `veto` | Tesseract as the third voice of the vetoes (needs the binary on PATH) |
| `onnx` | OnnxTR, a second cheap engine of a different architecture |
| `remote` | steps that talk to an external service (Docling API, vision model) |
| `docling` | Docling running inside the process, no network |

```bash
pip install 'autosxtract[paddleocr]'      # to pick a model tier
```

PP-OCRv6 is **not** an extra: it installs on every platform, because on a Mac it
is not Vision's substitute but the step below it — `native -> vision -> paddle`.
That second engine is not redundancy either: it is what enables the **agreement
gate**, which needs two independent readings to prove the reading is complete.

### System requirements

- **Python 3.11 or newer**, on macOS, Linux or Windows.
- **macOS**: nothing beyond pip. Vision is part of the system.
- **Linux/Windows**: nothing beyond pip. `opencv-python-headless` avoids the
  graphics libraries the normal variant needs — which matters in a container.
- **The `veto` extra**: besides the Python package, the Tesseract binary on the
  PATH (`apt install tesseract-ocr`, `brew install tesseract`).
- **First run off Apple hardware**: it downloads ~10 MB of PP-OCRv6 weights into
  `~/.cache/autosxtract`. After that the machine can stay offline. For a closed
  environment, run `autosxtract download-models` on a machine with network and
  point `AUTOSXTRACT_MODELS` at the copied directory.

---

## In thirty seconds

```python
from autosxtract import Cascade

cascade = Cascade()          # instantiate ONCE and reuse
r = cascade.extract_file("document.pdf")

r.text          # the extracted text
r.step          # who produced it: 'native', 'vision', 'paddle', ...
r.score         # 0.0 to 1.0
r.provenance    # "vision: native(no text layer) -> vision(ok)"
r.to_dict()     # all of that, ready for JSON
```

Instantiating `Cascade()` once and reusing it matters: loading the models is the
dominant cost of the first call, and one cascade per document pays it on every
file.

Several files at once, with parallelism already bounded by the machine:

```python
results = cascade.extract_batch(["a.pdf", "b.pdf", "c.pdf"])   # {name: Result}
```

And from the terminal:

```bash
autosxtract diagnose              # what steps this machine has
autosxtract extract document.pdf  # text on stdout, provenance on stderr
autosxtract extract *.pdf --json output.json
```

## Why a cascade, and not a single model

The temptation is to use one good model for everything. Measured across two
real archives, it **worsens** the time:

| | time |
|---|---|
| a single model at 2.5 pages/s, 935 documents | 6.2 min |
| the cascade | **4.44 min** |

The reason is the distribution, not the model: **31% of the documents already
have a text layer** and cost 13 ms. A single model would charge ~400 ms on pages
that have no OCR to do.

```
    STEP                        COST/DOC   RESOLVES   QUALITY (60 audited docs)
    -------------------------------------------------------------------------------
    0. unwrap                    ~0.1 ms      —       RTF / BRy / PKCS#7
    1. native (PyMuPDF)          13.4 ms     31%      native text, exact
    2. Apple Vision             ~400 ms      64%      words 92%  anchors 100%
       or PP-OCRv6 tiny         ~500 ms               (off Apple hardware)
    3. vetoes + screening         ~1 s       the rest
    4. remote steps            4 to 47 s     opt-in   only if you instantiate them
```

Step 0 almost never does anything — on a real PDF it costs reading 16 bytes. It
exists for the cases where the extension lies: in a real archive, **128
documents arrive as `.pdf` while being RTF, a BRy envelope or PKCS#7**, and
PyMuPDF raises `Failed to open stream` on them. No OCR recovers them — there is
no image to recognise, there is plain text nobody was reading.

## The gates

What stops the cascade from spending the expensive step for nothing. Each
answers a different question, and every one of them exists because a measurement
asked for it.

| gate | question | why |
|---|---|---|
| **stamp** | is what is left content, or just the conformity banner? | in an audit of 1,339 documents, 403 (30%) had text that *looked* fine; in 227 of them there was only the court stamp — 250 to 600 characters that sail past any size threshold |
| **coverage** | does the text layer cover the sheet? | in a filing that embeds an official letter as an image, the native text is flawless and the attachment is never read |
| **agreement** | did two engines read **the same thing**? | it proves the reading is *complete*, not that the page is short. It costs nothing: both readings are already in hand |
| **consensus** | did all of them read almost nothing? | proof of *absence*. One dissenting engine is enough to escalate |
| **contest** | who wins in the end? | the highest `quality × log(1+volume)`, not the last one to run |

Before any step marked `expensive = True`, five more:

| veto | question | why |
|---|---|---|
| **photograph** | is the page a photo rather than a document? | continuous tone with no text — the expensive step would only return `[SIGNATURE]` |
| **no ink** | is there ink on the sheet, outside the stamp? | calibrated at 1%: it avoids 2 of the 11 useless escalations without losing any of the 9 useful ones |
| **no legible word** | can a local OCR read anything here? | the **only one that measures instead of estimating**. It saves 27.2 minutes across 19 documents, and the largest content lost is a 124-character stamp |
| **sparse page** | is there anything to read? | "NOTICE OF INSPECTION", a signature sheet. Four of them paid for the expensive step to yield 126 to 433 characters |
| **reading confirmed** | have the two already read the same? | the agreement gate above, applied to the expensive step |

Not escalating is **not discarding**: in all five cases the document keeps the
text the cheap layer already read.

After an expensive step runs, the **replacement gate** decides whether it may
take the previous text's place. Length alone gets both of the costliest cases
wrong — partial coverage (a power of attorney transcribed up to page 10 of 15 is
still longer than a bad extraction of all 15) and digit corruption
(`9XXYZ3ZE...` scores exactly like `9XXYZ32E...`). A candidate failed here is
**discarded**, not demoted: volume is usually on the wrong side.

## Provenance

Every result carries the whole path, with the reason for each refusal.

```python
r = cascade.extract_file("document.pdf")

print(r.provenance)
# vision: native(quality 0.42 below 0.75) -> vision(ok)

for a in r.attempts:
    print(a.step, a.accepted, a.reason, a.chars, f"{a.ms:.0f}ms")

r.to_dict()   # ready for JSON, a log or a database
```

"The system extracted the text" is not an auditable answer. "The native step
read 41 characters and was refused on density; Vision read 3,812 and passed" is.

## Configuration

Every threshold in one place, each annotated with the measurement that fixed it.

```python
from autosxtract import Cascade, Config

cascade = Cascade(Config(
    dpi=150,                # at 100 DPI anchor preservation falls to 85.5%
    min_useful_words=12,    # floor outside the stamp
    min_agreement=0.60,     # calibrated on 24 real escalations
    engines=["paddle"],     # explicit order; None = the machine decides
))
```

## Parallelism

The same library runs on a 2-core laptop and a 72-core server, so the three
parallelism fields accept `None` — **decide from the machine** — and that is the
default.

```python
Config(
    page_parallelism=None,      # None = min(4, cores)
    document_parallelism=None,  # None = min(4, cores)
    concurrency_cap=None,       # None = cores × 2; 0 turns it off
)
```

`autosxtract diagnose` shows what your machine resolved:

```
resources  2 usable core(s); affinity limits to 2 of 72
automatic parallelism: 2 page(s) per document, 2 document(s) in flight (aggregate cap 4)
```

Detection is the most restrictive of three sources, because none covers
everything: `sched_getaffinity` (catches `taskset` and cpusets), the cgroup
quota (catches `docker run --cpus=2`) and `os.cpu_count()`. **`os.cpu_count()`
lies inside a container** — it reports the host's cores, so a pod with 2 CPUs on
a 72-core machine would open 72 threads to fight over 2.

### Why those defaults

Measured on PP-OCRv6 tiny, 12 real pages, the same machine restricted with
`taskset`:

| threads | 72 cores | 2 cores |
|---|---|---|
| 1 | 1.36 pages/s | 1.44 pages/s |
| 2 | 1.74 | **1.68** |
| 4 | **1.99** | 1.58 |
| 8 | 2.18 | 1.54 |
| 16 | 2.27 | 1.71 |

On 2 cores throughput plateaus at 2 threads; asking for 8 delivers *less* (1.54
against 1.68) with 4× more pages in flight. On 72, going from 4 to 16 threads
gains 1.14×. One fixed number serves both badly.

### An explicit number is obeyed

Whoever knows what they are doing decides. What it is not, is a promise — the
curve flattens early. Two caveats:

- **The aggregate cap still cuts the batch.** `documents × pages` grows
  unnoticed: 4 × 8 is 32 simultaneous pages, each holding a rendered image and
  the model's activations, and on a small machine the memory limit arrives
  before the CPU one. The cut lands on the *pages*, never on the documents —
  reducing documents raises total time predictably, reducing pages per document
  costs almost nothing. Turn it off with `concurrency_cap=0`.
- **A single-queue engine ignores the number.** Apple's Neural Engine serves one
  request at a time: measured from 1 to 12 threads, constant throughput at ~2.5
  pages/s and latency from 430 ms to 3,492 ms. `VisionEngine` declares
  `scales_with_threads = False` and the cascade uses 1, recording the effective
  value in the provenance so nobody configures 8 and measures the time of 1
  without understanding why. There the useful parallelism is **per document**.

Resolution is a method (`config.pages_in_flight()`), not a computed field: a
preset stored in YAML and used in two environments has to answer differently in
each.

## Swapping the OCR model

The default is **PP-OCRv6 tiny**, and the choice is measured: with the
bottleneck in CPU, what decides is throughput and not benchmark accuracy. But a
default is a default — a different archive wants a different model, and swapping
it does not require touching the cascade.

```python
from autosxtract import Cascade, NativeStep, OCRStep
from autosxtract.engines.paddle import PaddleEngine

PaddleEngine()                              # PP-OCRv6 tiny (default)
PaddleEngine(det="tiny", rec="medium")      # small detector + large recogniser [paddleocr]
PaddleEngine(rec_dir="/my/finetuned_rec")   # your model, trained on your archive
PaddleEngine(quantized=True)                # INT8, if you exported it
PaddleEngine(preprocess="otsu")             # binarise before reading
PaddleEngine(providers=["CoreMLExecutionProvider", "CPUExecutionProvider"])

cascade = Cascade(steps=[NativeStep(), OCRStep(PaddleEngine(rec="medium"))])
```

**Why `det` and `rec` are separate:** accuracy lives in the recogniser. Finding
*where* there is text is an easier task than reading *what* is written, so "small
detector with large recogniser" tends to be the best-returning trade — and it
only exists if the two are distinct parameters.

The engine picks its backend on its own: `paddleocr` when installed (giving the
three official tiers, INT8 and a fine-tuned recogniser), `rapidocr` as the
lightweight alternative — which serves the tiny and nothing else. To pick a
tier, install `pip install 'autosxtract[paddleocr]'`. For an entirely custom
engine, see the next section.

From the command line, to measure a swap without writing code:

```bash
autosxtract extract document.pdf --det tiny --rec medium
autosxtract extract document.pdf --rec-dir /my/finetune --no-layers
```

## Containment layers

A small OCR engine reads the **body** of a document almost perfectly. The error
concentrates in four places: vertical stamps, signatures over printed text,
two-column headers and run-together words. No OCR reads *through* a stamp — so
the strategy is not "read better", it is to **contain the damage, flag where it
is, and recover what is recoverable**.

| layer | what it does | cost |
|---|---|---|
| **1** | classifies each line; junk becomes `[illegible]`, fragments are dropped, the body passes untouched | ~2 ms, no model |
| **1b** | re-segments run-together words against the lexicon | included |
| **1.5** | visual signature detector (optional, see the caveat) | ~35 ms/page |
| **2** | re-reads the targets: rotates the vertical stamp 90°, crops the dirty line tighter | ~30 ms |
| **3** | per-page report: how much is trustworthy and what to do | free |

Measured on 895 pages against the same engine without the layers: median CER
0.132 → 0.129, **entity recall 0.902 → 0.921**, p50 latency 298 → 236 ms. 79
pages improve, **4 get worse** (all already bad), clean pages are untouched. On
the vertical-stamp subset, recall rises from 0.745 to 0.837.

The principle behind the thresholds: `illegible` only for **unambiguous** junk.
Anything doubtful becomes `suspect` — the text passes, but the page loses
confidence and tends to escalate. That way the weak signal is not lost.

The report enters the provenance:

```python
r = cascade.extract_file("document.pdf")
r.details["layers"]
# {'lines_total': 103, 'lines_illegible': 1, 'lines_vertical': 3,
#  'lines_recovered': 1, 'trusted_fraction': 0.975,
#  'needs_escalation': False, 'suggested_action': 'accept_with_holes'}
```

The layers need an engine that exposes **line geometry** (`read_page`). An
engine that only returns running text keeps working — the layers simply do not
run, and the provenance says so.

### The lexicon decides what counts as the language

Classification compares the line's tokens against a lexicon, and its quality
decides the classification's quality. The built-in one is a floor; building your
own from validated texts is measurably better:

```python
from autosxtract import Config
from autosxtract.quality.lexicon import Lexicon

mine = Lexicon.from_texts(Path("validated").glob("*.txt"))
Config(lexicon=mine)
```

With a small lexicon, more correct text falls into `suspect` — the safe side of
the error, but it escalates pages for nothing.

### The signature detector: read before switching it on

Off by default, and not out of generic caution. A `yolo11n` trained on a public
signature dataset (mAP50 0.995 on the public validation set) **did not
transfer**: across 895 pages of a legal archive it produced a detection on 19% of
them, most of which were **false positives** — an authenticity seal at 0.92
confidence, a "DELIVERED" stamp, an ICP-Brasil logo, a QR code, a coat of arms,
and the printed word "SIGNATURES". And it missed the target case: a cursive
signature over a name produced no box even at 0.12 confidence.

The public dataset is contract signatures — thick strokes, isolated, clean
background. The real case is a thin cursive scribble **over** text, on a degraded
scan.

That is why a box **never decides on its own**: it only counts if some
overlapping line is illegible, and it is discarded if any overlapping line is
stamp text. That filter is what makes an imperfect detector usable.

```python
Config(signature_detector="/my/models/sig.onnx")
```

For a stage 2 that works, annotate 200-400 pages of **your** archive, including
NEGATIVES (stamps, seals, logos and QR codes as not-a-signature). Without them
the model repeats the same false positives. The structural rule — a scribble
above a name with a job title nearby — always runs, with or without a detector,
and costs nothing.

## Page routing

Classifies the page from what the cheap engine read, running no model (~1 ms):
`table`, `stamped_digital`, `degraded` or `normal`. Switch it on with
`Config(page_routing=True)`; it does not change the extraction, it is a signal
for whoever consumes it.

**About tables, a measured verdict worth more than the route.** Across 17 pages
classified as tables, the cheap engine with layers recovered **0.797** of the
numeric values against 0.699 for a dedicated structure model (SLANet), which
also cost 6-29 s per page against 0.27 s. On a poor scan the bottleneck **is not
the structure model, it is the cell OCR** — rebuilding the grid does not help if
the recogniser cannot read the cell, and the rebuild still loses the surrounding
prose. Across 895 pages, switching to the table model's output won on **1 page**.

That is why the library classifies the route and does **not** embed a table step.

## How to add an engine

A class with one method, plus the decorator. The cascade finds it on its own —
nothing else changes.

```python
from autosxtract.engines.base import OCREngine, register

@register(name="my_ocr", priority=25, extra="my-ocr")
class MyOCR(OCREngine):
    def _load(self):
        import my_ocr
        return my_ocr.Reader()

    def transcribe_page(self, image: bytes) -> tuple[str, float]:
        r = self.model.read(image)
        return r.text, r.confidence * 100
```

`priority` is preference order (lowest first). The defaults come from
comparative measurement on the same sample: 10 Vision, 20 PP-OCRv6, 90
Tesseract.

Two rules the contract imposes:

- **Confidence does not arbitrate quality.** Across 60 audited documents it did
  not separate a good reading from an unsafe one — there was an unsafe document
  at 100. It enters only as a floor against degenerate output.
- **A missing engine is never an exception.** `available()` returns the reason in
  words, the step goes inert and the cascade moves on. The absence of a tool is
  not evidence about the document.

If your backend exposes line geometry, implement `read_page` as well: that is
what enables the containment layers.

## Remote steps

Docling and vision models exist in the library, but **only through explicit
instantiation**. They are not in the default cascade, there is no
environment-variable discovery, and `Config` has not a single host, URL or
credential field. Whoever wants them builds them:

```python
import os
from autosxtract import Cascade, NativeStep, OCRStep, ScreeningStep, get
from autosxtract.steps.remote import DoclingStep, VLMStep

cascade = Cascade(steps=[
    NativeStep(),
    OCRStep(get("vision")),
    DoclingStep(
        url="http://docling:5001",
        token=os.environ["DOCLING_TOKEN"],
        conversion_timeout=180,
        force_ocr_if_empty=True,
    ),
    ScreeningStep(),
    VLMStep(
        url="https://my-endpoint/v1",
        model="PaddleOCR-VL-0.9B",
        token=os.environ["VLM_TOKEN"],
        dpi=200,
        images_per_batch=2,
        max_tokens_per_page=2000,
        parallelism=8,
    ),
])
```

`pip install 'autosxtract[remote]'` brings `httpx`. Installing turns nothing on —
the steps only exist if you instantiate them. Both are `expensive = True`, so the
five vetoes run before and the replacement gate after. A network failure becomes
a refused attempt with the reason in the provenance, never an exception. The
`token` never appears in `repr`, in a log or in the result.

About the expensive step's model: a 27B model transcribing a sheet of paper is
measured waste — 47.4 s per document. Specialised OCR models in the 0.5-3B range
(GLM-OCR 0.9B, PaddleOCR-VL 0.9B, dots.ocr 1.7B, DeepSeek-OCR 3B, GOT-OCR2.0
0.58B) are built for this case and promise ~100×. That is why `model` is a
parameter, not a constant.

### Docling without a network

The same engine, running **inside the process**. The choice between the two
forms is not obvious:

| form | network | models | when to choose |
|---|---|---|---|
| `DoclingStep` | yes | on the server | several processes share one installation; the client stays light |
| `LocalDoclingStep` | **no** | ~2 GB here | an isolated machine with no egress, or a single process |

```bash
pip install 'autosxtract[docling]'
```

```python
from autosxtract.steps.docling_local import LocalDoclingStep

docling = LocalDoclingStep(workers=2, ocr_engine="rapidocr")   # once
cascade = Cascade(steps=[NativeStep(), OCRStep(get("paddle")), docling])
```

Loading is lazy — building the step loads no model, so assembling a cascade and
finding the PDF is native does not cost the 2 GB for nothing. But **reuse the
instance**: one step per document pays the whole load on every file. Internally
there is a pool of converters, because `DocumentConverter` is not thread-safe.

## How to add a step

The other extension point. A step is any object with `name` and
`run(ctx) -> StepResult`:

```python
from autosxtract import Cascade, NativeStep, OCRStep, get
from autosxtract.steps.base import StepResult
from autosxtract.types import Attempt, Candidate

class MyFormatStep:
    name = "my_format"

    def run(self, ctx) -> StepResult:
        text = unwrap_my_format(ctx.pdf_bytes)
        if not text:
            return StepResult(Attempt(self.name, False, "not my format"))
        return StepResult(
            Attempt(self.name, True, "unwrapped", len(text)),
            Candidate(self.name, text, score=1.0),
        )

cascade = Cascade(steps=[MyFormatStep(), NativeStep(), OCRStep(get("paddle"))])
```

`StepResult` separates two things that are not the same: the **verdict** (does
the cascade stop?) and the **candidate** (does the text enter the contest?). A
refused step may still have produced the best reading the document has.

Declaring `expensive = True` makes the cascade run the five vetoes before it and
submit the result to the replacement gate afterwards.

## Architecture

```
autosxtract/
  cascade.py        the orchestrator — steps, gates, vetoes, contest
  config.py         every threshold, each with the measurement that fixed it
  formats.py        RTF / BRy / PKCS#7 — is the file really a PDF?
  types.py          Line, Page, Transcription, Candidate, Attempt, Result
  platform.py       Apple or not; the only hardware-dependent decision
  resources.py      how many cores there really are (affinity, cgroup, cpu_count)
  image.py          image dimensions from the header, with no dependency
  pdf/              knows only the file: lock, render, profile, coverage, ink
  quality/          knows only text: stamp, metrics, gate, consensus, anchors,
                    prose, screening, markers, vetoes, rejection, lines,
                    lexicon, routing
  engines/          what reads pixels: vision, paddle, onnx, tesseract,
                    signature (YOLO) + the registry
  steps/            native, generic OCR, unwrap, screening, layers,
                    docling_local (no network) and remote/ (explicit url)
```

The layers do not cross each other: `pdf/` does not know what an engine is,
`quality/` does no I/O, `engines/` does not know what a cascade is. That is what
lets each be tested alone — and what makes a new engine cost one file.

## What has been measured and does **not** work

This section exists so nobody spends time again on what has already been refuted.

| tried | result |
|---|---|
| turning off Vision's language correction | doubles throughput (2.5 → 5.4 pages/s) and **loses text**: 102 documents fall to worse engines, −227 anchors and −4,981 characters across 935 documents |
| deduplicating pages | only 2.0% repeat (33 of 1,675) |
| predicting which documents will escalate (bytes/page) | medians of 89k against 109k, almost complete overlap |
| nine families of pixel statistics for "empty page" | an empty page and a dense-but-faded page produce identical statistics — what separates them is consensus between engines |
| Apple Vision's `.fast` mode | rejected on quality |
| grey JPEG at 100 DPI | 85.5% anchor preservation; it loses dates and tax numbers |
| more threads against Vision | constant throughput, linear latency: it is a single hardware queue |
| collapsing the cascade into one model | 6.2 min against 4.44 min |
| a dedicated table structure model | 0.699 value recall against 0.797 for the cheap engine with layers, at 6-29 s/page against 0.27 s |
| a signature detector trained on a public dataset | 19% of pages with a detection, most false positives on seals, stamps, logos and QR codes |

## Known limits

- **Apple Vision does not scale with threads.** A single Neural Engine queue: the
  ceiling is ~2.5 pages/s per machine. The library knows this and uses 1 thread
  there regardless of what you configure. The useful parallelism is **per
  document**.
- **With no OCR engine installed**, a scanned PDF comes out empty — correctly,
  and `autosxtract diagnose` says so in large letters. It happens when the
  platform marker did not match: an image built on a Mac and run on Linux,
  `--no-deps`, or a lockfile generated for another `sys_platform`.
- **PyMuPDF is serialised** by a process lock: it crashes the process under
  concurrency (segfault in `page_get_textpage`). The measured cost of
  serialising is ~4%.

## Development

```bash
make hooks       # pre-commit install (do this first)
make test        # pytest
make lint        # ruff check + ruff format --check
make typecheck   # mypy
make privacy     # scan for sensitive artefacts
make all
```

`pre-commit` runs the privacy scan **before** the style hooks, and the order is
deliberate: a commit blocked on formatting costs thirty seconds; a real document
published has no undo. The scan validates Brazilian tax IDs, company IDs and
case numbers by their **check digit**, not merely by their shape — a scanner
that shouts at every 14-digit number is switched off in the first week.

It has already paid for itself: it caught a real case number that had made its
way into this library's own examples.

## Licence

MIT.
