Metadata-Version: 2.4
Name: pageledger
Version: 0.1.1
Summary: Page-denominated document extraction run ledger
Author: Peter Busscher
License-Expression: MIT
Project-URL: Homepage, https://github.com/peterbussch/pageledger
Project-URL: Documentation, https://github.com/peterbussch/pageledger#readme
Project-URL: PDF/OCR First Run, https://github.com/peterbussch/pageledger/blob/main/docs/pdf-ocr-first-run.md
Project-URL: OCR Options, https://github.com/peterbussch/pageledger/blob/main/docs/ocr-options.md
Project-URL: Issues, https://github.com/peterbussch/pageledger/issues
Project-URL: Source, https://github.com/peterbussch/pageledger
Keywords: document-ai,ocr,vlm,digital-humanities,provenance
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Text Processing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: PyYAML>=6.0.2
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: jsonschema>=4.0; extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: ruff>=0.9; extra == "dev"
Requires-Dist: twine>=6.0; extra == "dev"
Provides-Extra: pdf
Requires-Dist: pypdf>=6; extra == "pdf"
Dynamic: license-file

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/peterbussch/pageledger/main/assets/pageledger-lockup-horizontal-reversed.png">
    <img src="https://raw.githubusercontent.com/peterbussch/pageledger/main/assets/pageledger-lockup-horizontal.png" alt="PageLedger — the tallied page" width="440">
  </picture>
</p>

<p align="center"><em>Record, route, and review document extraction — one page at a time.</em></p>

**PageLedger is for researchers who extract text from archives at scale** —
digital humanities labs, libraries, historians, and social scientists running
OCR/VLM over thousands of scanned pages who must be able to answer, months
later: which pages went through which engine and prompt, what it cost, which
pages are suspect, and what happened when they were re-extracted.

It is a page-denominated **run ledger** — not an OCR model, not a
PDF-to-Markdown converter. You bring the extraction engine (Tesseract,
Docling, Marker, a cloud VLM); PageLedger routes pages to it, enforces
page/token/dollar budgets, and writes the evidence as plain files:
provenance, quality diagnostics, cost with its basis, audit queues, and
executable rerun plans. No service, no database — a run directory you can
grep, cite, and reproduce.

PageLedger grew out of
[soviet-corpus](https://github.com/peterbussch/soviet-corpus), a Soviet census
digitization and corpus-linguistics project where those questions had to be
answerable for source citation and methodological review. The run-ledger
layer was extracted from that working pipeline and rebuilt as a standalone,
domain-neutral tool.

The current alpha covers the run-ledger core: built-in `text`, `pdf_text`
(born-digital), and `pdf_ocr` (scanned, via local Tesseract) adapters, custom
adapters via import strings, dry runs, budgets, quality signals, page-scoped
reruns (`pageledger rerun`), and cross-run diffs (`pageledger compare-runs`).
It does not call network VLMs out of the box, classify pages, align records
to schemas, or compute audit grades — see
[Current Runtime Capabilities](#current-runtime-capabilities).

## Install

```bash
pip install pageledger
```

Optional born-digital PDF text extraction:

```bash
pip install "pageledger[pdf]"
```

## Quickstart

Create a text fixture and a starter config:

```bash
printf 'first page\fsecond page\n' > sample.txt
pageledger init-config --out pageledger.yml
```

Inspect the run without extraction:

```bash
pageledger run sample.txt --config pageledger.yml --out runs/dry --dry-run
```

Run the built-in text adapter, then summarize the run directory:

```bash
pageledger run sample.txt --config pageledger.yml --out runs/text --json
pageledger inspect-run runs/text
```

For a born-digital PDF, install the optional PDF extra and use `pdf_text`:

```bash
pip install "pageledger[pdf]"
cat > pageledger-pdf.yml <<'YAML'
schema_version: "0.1"
taxonomy:
  page_types:
    prose:
      default_action: transcribe_text
run:
  adapter: pdf_text
YAML
pageledger run document.pdf --config pageledger-pdf.yml --out runs/pdf-dry --dry-run
pageledger run document.pdf --config pageledger-pdf.yml --out runs/pdf-text --json
```

`--out` must point to a new empty directory. `pdf_text` extracts an existing
text layer and does not OCR scanned images.

For a scanned PDF with no text layer, use `pdf_ocr`. It shells out to
`pdftoppm` and `tesseract`, so install poppler and Tesseract first
(`pageledger doctor` checks for both):

```bash
pageledger init-config --adapter pdf_ocr --out pageledger-ocr.yml
pageledger run scan.pdf --config pageledger-ocr.yml --out runs/scan-ocr
```

Set DPI and language under `run.adapter_options`:

```yaml
run:
  adapter: pdf_ocr
  adapter_options:
    dpi: 400
    lang: eng+deu
```

The run directory contains `manifest.json`, `route-map.yml`, `audit.json`,
`audit.md`, `provenance.jsonl`, `quality.jsonl`, `cost.json`, `run.log`,
`rerun-manifest.yml`, and per-page raw output under `raw/`.

For a worked example on a real declassified scan, see
[`docs/examples/jfk-scanned-archive.md`](docs/examples/jfk-scanned-archive.md).
For choosing between local, cloud, and hybrid extraction, see
[`docs/ocr-options.md`](docs/ocr-options.md). Custom adapter examples live
under [`examples/`](examples/); load them with `--adapter-path`:

```bash
pageledger run scan.pdf --config custom.yml --out runs/custom \
  --adapter-path examples
```

## How a Run Works

```mermaid
flowchart TD
    A["inputs (text / PDF)"] --> B["paginate<br/>(form-feed or PDF pages)"]
    B --> C["route pages<br/>route-map.yml"]
    C --> D{"budget preflight<br/>max_pages"}
    D -- over cap --> X["refuse: nothing written"]
    D -- ok --> E["extract page via adapter<br/>(retry + backoff)"]
    E --> F["quality signals<br/>quality.jsonl"]
    E --> G["provenance.jsonl<br/>+ cost.json (cost_basis)"]
    E --> H{"budget mid-run<br/>pages / tokens / USD"}
    H -- over cap --> Y["halt: manifest status=failed,<br/>partial artifacts consistent"]
    H -- ok --> E
    F -- warnings --> I["review queue<br/>audit.json / audit.md"]
    I --> J["rerun-manifest.yml"]
    J -- "pageledger rerun<br/>(stronger adapter)" --> E
    G --> K["pageledger compare-runs<br/>(parent vs rerun)"]
```

Every box on the right is a plain file in the run directory; nothing needs a
service or database.

## Current Runtime Capabilities

**Built-in and tested:**

- `pageledger run` for text fixtures (form-feed pagination), born-digital PDF
  text layers (via `pageledger[pdf]`), and scanned PDFs (`pdf_ocr`, using
  locally installed `pdftoppm` + `tesseract`).
- Adapter options (`run.adapter_options`) passed to built-in and custom
  adapter constructors, and `--adapter-path` for loading custom adapter
  modules without touching PYTHONPATH.
- Dry-run mode that generates full run artifacts without calling extractors.
- Per-page provenance (`provenance.jsonl`), quality diagnostics
  (`quality.jsonl`), cost rollups (`cost.json`), structured run logs
  (`run.log`), and audit/review queues (`audit.json`, `audit.md`).
- Page-denominated budget enforcement (pages, tokens, dollars) with preflight
  and per-page caps.
- Retry with configurable `max_retries` and optional exponential backoff
  (`retry.backoff: exponential`).
- `pageledger rerun` — re-extracts exactly the pages listed in a previous
  run's rerun manifest (typically with a stronger adapter), preserving page
  ids, recording parent lineage, enforcing `max_rerun_depth`, and warning if
  a source file changed since the parent run.
- `pageledger compare-runs` — page-by-page diff of two runs: character/word
  deltas, warnings resolved or introduced, adapters, and cost.
- Cost provenance: `cost.json` records `cost_basis` (`adapter_reported`,
  `configured_rate`, `mixed`, or `none`) so derived accounting rates are
  never mistaken for provider-billed spend, plus runner-measured
  `extraction_seconds` per page and in total.
- `pageledger doctor` diagnostics for optional PDF/OCR/cloud tooling.
- Configurable custom adapters via `module.path:object` import strings.

**Adapter-supported but user-supplied:**

- OCRmyPDF preprocessing, or any external engine wrapped as a custom adapter.
- Cloud OCR/VLM adapters (user provides API keys, adapter code, and pricing).
- Local document-conversion engines (Docling, Marker, Surya) through custom
  adapters.
- PDF page counting for custom adapters that expose `page_count(source)`.

**Documented future work (not yet implemented):**

- Automatic page classification (the alpha routes every page to `review` in
  dry-run mode; no classifier ships).
- Schema alignment execution (the schema config section is parsed but the
  aligner does not yet produce normalized records).
- Audit grading (review queues are populated but no grades are computed).
- Multi-adapter routing chains, `rerun_if`/`quarantine_if` policy enforcement,
  and staged CLI commands (`classify`, `extract`, `align`, `audit`).

## Known Limits

- **`pdf_text` reads existing text layers.** It does not perform OCR. For
  scanned PDFs use `pdf_ocr` (local Tesseract) or wrap a stronger engine
  (OCRmyPDF, Docling, Marker, Surya, cloud OCR) as a custom adapter.
- **`pdf_ocr` needs poppler and Tesseract installed.** PageLedger never
  installs OCR engines; it shells out to `pdftoppm` and `tesseract` and fails
  with an install hint (`pageledger doctor`) when they are missing. OCR
  quality is Tesseract's, at the DPI and language you configure.
- **Quality signals are diagnostic, not calibrated.** `quality.jsonl` records
  per-page evidence: character/word counts, replacement characters, control
  characters, suspicious symbol density, lexical shape (token counts, mean
  token length), and embedded-text deltas. A seven-item warning taxonomy
  (empty_text, short_text, replacement_characters, control_characters,
  suspicious_symbol_density, fragmented_text, suspicious_embedded_text_delta)
  flags pages for review. These signals are NOT OCR accuracy scores — they are
  evidence a human should weigh. In particular, no heuristic here detects
  word-level misrecognition ("matericl" for "material"); that takes a
  dictionary, a model, or a human, and PageLedger ships none of them.
- **Quality-warning pages appear in the audit review queue.** For execution
  runs, every page with one or more quality warnings is added to
  `audit.json` → `review_queue` with reason `"quality_warning"`. Dry-run
  review entries use route-based reasons (`"no_classifier_available"`).
- **No automatic page routing.** Every page routes to the configured
  `default_action` (or `review` in dry-run mode). Projects that need
  page-type-aware routing must classify pages outside PageLedger or wait for
  a future classifier.
- **Schema alignment does not execute.** The schema config section is validated
  and preserved in the config snapshot, but the runner does not yet align
  extractor output to declared columns, types, or checks.
- **Reruns re-extract listed pages; they do not merge results.**
  `rerun-manifest.yml` lists pages from `audit.json` → `review_queue`;
  `pageledger rerun` re-extracts exactly those pages into a new run directory
  with parent lineage. `rerun_status` is `"executable"`, `"empty_queue"`, or
  `"no_further_generations"` (depth cap reached). Combining parent and rerun
  outputs into one corpus remains the project's decision —
  `pageledger compare-runs` shows the per-page evidence for making it.

## Tested Scale

The 0.1.x alpha has been tested locally on:

- **5,000 synthetic text pages** (2.4s, ~2,100 pages/sec, artifact counts
  verified — raw files = provenance lines = quality lines = pages extracted).
- **1,000 synthetic text pages** (0.25s, ~4,000 pages/sec).
- **72-page born-digital PDF** (via `pageledger[pdf]`).
- **5 small checked-in text fixtures** covering clean, multipage, OCR-noisy,
  blank, and short text.

This is evidence of the envelope, not a universal performance guarantee.
Stress tests are marked `@pytest.mark.stress` and skipped in default CI.
Run them with:

```bash
python -m pytest tests/pageledger/ -m stress
```

## The Canonical Unit: Pages

PageLedger's defining decision is that **the page is the canonical unit of
work.** It is the only unit every extraction backend shares:

- Cloud OCR (Textract, Azure Document Intelligence, Google Document AI, Mistral
  OCR) bills and reports **per page**.
- VLM/LLM extractors expose **tokens** — but only on model-backed paths; tokens
  are meaningless for classical OCR.
- Self-hosted engines (e.g. Docling) have **no dollar cost at all**, only compute
  time.

Because pages are the common denominator, routing, budgeting, and audit in
pages lets you compare a Textract run against a Mistral run against a local
model with one number. Tokens, compute-seconds, and dollars ride on top as
optional, provider-conditional signals.

Every adapter reports a usage record where **`pages` is required** and
everything else is optional:

```python
usage = {
    "pages": 1,             # REQUIRED — the portable unit
    "tokens": None,         # VLM/LLM paths only
    "compute_seconds": None, # self-hosted engines
    "cost_usd": None,       # optional adapter-reported passthrough
}
```

Dollar cost is **derived by PageLedger, never required of the adapter**, in
priority order: (1) adapter-reported `cost_usd`, (2) configured unit rates
(`cost_per_page` / `cost_per_1k_tokens`), (3) otherwise `null` — the run still
reports raw page counts. Budgets cap on **pages, tokens, or dollars**, whichever
the config sets, because the page count is the only value always present.

## Why Not Just Use OCR?

OCR and document-AI tools are improving quickly. Mistral OCR, Google Document
AI, Azure Document Intelligence, AWS Bedrock Data Automation, Docling, Marker,
Surya, olmOCR, OCR-D, and Unstructured all cover parts of extraction,
conversion, layout analysis, validation, or workflow orchestration.

That makes PageLedger's scope deliberately narrow: it is useful when you need a
local, provider-neutral ledger around those tools, not when you only need one
tool to convert a document once.

| Tool | Strong At | PageLedger Difference |
|---|---|---|
| Mistral OCR | Hosted OCR/document understanding with Markdown and table reconstruction. | PageLedger records why pages went to Mistral, what it cost, and which pages need review or rerun. |
| Google Document AI / Gemini layout parser | Managed OCR, layout parsing, table structure, and RAG-oriented chunking. | PageLedger keeps provider-neutral run artifacts and project-local audit queues outside a cloud workflow. |
| Azure Document Intelligence | Managed text, key-value, table, and field extraction. | PageLedger can track Azure runs beside local or other-provider runs with the same page-denominated manifest. |
| AWS Bedrock Data Automation | Managed classification, extraction, validation, HITL review, and business-rule workflows. | PageLedger stays lighter and filesystem-native for research projects that do not want a cloud stack. |
| Docling | Converting PDFs and documents into structured output. | PageLedger records which pages went to Docling, the usage and cost evidence, and which pages need review or rerun. |
| Marker | High-quality PDF-to-Markdown conversion. | PageLedger handles routing, run manifests, cost controls, and review queues around conversion. |
| Surya | OCR, layout analysis, table recognition. | PageLedger can call Surya as an extractor, then audit the result across a full archive run. |
| olmOCR | LLM-based PDF extraction with document-oriented output. | PageLedger adds page routing, quality diagnostics, cost evidence, and rerun manifests. |
| OCR-D | Mature OCR workflow model with METS/PAGE/ALTO conventions. | PageLedger aims to be lighter, VLM-aware, and friendlier to local research workflows that need JSON/YAML artifacts before full library infrastructure. |
| Unstructured | Document partitioning and preprocessing for downstream use. | PageLedger focuses on extraction run control: what ran, what passed, what failed, what cost money, and what should be rerun. |

Non-goal: PageLedger is not an OCR engine, PDF converter, or layout detector.
If a project already has clean structured text from a single tool and does not
need routing, schema checks, review queues, cost tracking, or reruns, it likely
does not need PageLedger.

For practical provider-agnostic choices, see
[`docs/ocr-options.md`](docs/ocr-options.md). The recommended workflow: run cheap
local extraction first, inspect `quality.jsonl` warnings to identify weak pages,
re-extract just those pages with a stronger adapter via
`pageledger rerun runs/first/ --config stronger.yml --out runs/second/`, then
`pageledger compare-runs runs/first/ runs/second/` to see what improved.

## What PageLedger Cannot Do

PageLedger does not make OCR or VLM output correct by itself. It cannot
calibrate confidence scores across unrelated extractors, guarantee text
accuracy, infer a project's source citation, or make a right-to-left,
mixed-script, tabular, or handwritten collection work without explicit adapter
and schema configuration.

Its job is narrower: preserve enough evidence that a researcher can see what
ran, what failed, what is uncertain, and what should be reviewed or rerun.

## Why This Exists

Modern OCR and document-AI tools can produce impressive text, tables, and
Markdown, but real archive work fails in smaller and more frustrating ways:

- Some pages are title pages, indexes, maps, appendices, blanks, or prose,
  not data pages.
- Expensive VLM calls should be routed deliberately instead of sprayed across
  every page.
- Model-emitted tables often have variant headers, shifted columns, missing
  rows, merged cells, or inconsistent numeric formats.
- Confidence scores are weak unless they are checked against schema,
  arithmetic, page totals, and model agreement.
- Long extraction runs need cost tracking, retry logic, checkpoints, and
  provenance that survives reruns.

PageLedger is for projects where "the model returned JSON" is only the
beginning of the work. The current alpha is for proving local run evidence and
adapter contracts before trusting larger extraction workflows.

## Audience

- Digital humanities labs processing archival scans.
- Libraries and archives building reproducible extraction pipelines.
- Historians and social scientists extracting structured data from source
  books, statistical tables, reports, and registers.
- Civic, legal, nonprofit, and local-government document projects.
- Document-AI engineers who need a lightweight local control plane around
  Docling, Marker, Surya, olmOCR, Tesseract, or API VLMs.

## Architecture and Roadmap

The alpha implements the **run controller** (budgets, retry, provenance,
quality signals, audit queues, rerun execution, cross-run comparison) and
the **adapter protocol** (thin wrappers that accept a routed page and return
serializable content, warnings, confidence hints, and usage metadata — see
[`docs/adapter-protocol.md`](docs/adapter-protocol.md)).

Automatic page routing, schema alignment, audit grading, rerun execution, and
staged CLI commands are design targets, documented with examples in
[`docs/design.md`](docs/design.md).

## CLI

Six commands ship in the alpha; `run` and `rerun` are the ones that extract:

```bash
pageledger run scans/ --config pageledger.yml --out runs/run-001/
pageledger run scans/ --config pageledger.yml --out runs/run-001/ --dry-run
pageledger run scans/ --config pageledger.yml --out runs/run-001/ --json --log-level INFO
pageledger rerun runs/run-001/ --config stronger.yml --out runs/run-002/
                                              # re-extract only the flagged pages
pageledger compare-runs runs/run-001/ runs/run-002/
                                              # page-by-page diff of two runs
pageledger init-config --out pageledger.yml   # write a minimal valid config
pageledger inspect-run runs/run-001/          # summarize a run directory
pageledger doctor --json                      # check optional dependencies
```

`--dry-run` generates the route map and planning artifacts without calling
extractors, so you can inspect routing before spending money or trusting
output. `--json` makes command status scriptable (parseable stdout, errors on
stderr), and `--log-level` controls the detail written to the run log.

Three adapters ship built in. `text` writes each page of a UTF-8 text input
to `raw/` (pages split on the form-feed character). `pdf_text` reads
born-digital PDF text layers through `pypdf` (install `pageledger[pdf]`).
`pdf_ocr` OCRs scanned PDFs through locally installed `pdftoppm` and
`tesseract`. For richer layout extraction, point `run.adapter` at a project
adapter such as `my_project.adapters:TesseractCliAdapter` (use
`--adapter-path` to make the module importable); PageLedger keeps the same
manifest, cost, log, and provenance envelope around any engine.

The doctor output reports optional packages, common OCR commands, and whether
cloud OCR/VLM keys are present without printing their values.

Starter config examples live in `docs/examples/`:

- `pageledger.yml` (recommended first-use combined config)
- `page-taxonomy.yml`
- `table-schema.yml`
- `run-policy.yml`

The split files are useful for documentation and larger projects. The
recommended v0.1 user experience should start with one `pageledger.yml` that
contains `taxonomy`, `schema`, and `run` sections.

## Run Artifacts

A run should be inspectable without a database or service:

```text
runs/run-001/
├── manifest.json
├── config-snapshot.yml
├── route-map.yml
├── raw/
│   └── doc_0001_page_0002.txt
├── normalized/
├── audit.json
├── audit.md
├── provenance.jsonl
├── quality.jsonl
├── cost.json
├── run.log
└── rerun-manifest.yml
```

The important artifact is not only the extracted data, but the evidence around
it: which pages were skipped, which model or engine was used, what usage and
cost evidence was observed, which pages failed or need review, and what should
be rerun or reviewed.

`manifest.json` is the canonical run artifact. `audit.md` is a human rendering
of `audit.json`, not a second independent source of truth. Quarantined pages
live in `audit.json` under `quarantine_queue`. `config-snapshot.yml` preserves
the exact user config that produced the run. `run.log` is JSONL — one line per
extractor call with timestamp, `page_id`, adapter, status, and any error — so
partial or failed runs stay greppable. Every artifact validates against the
JSON Schemas in [`schemas/`](schemas/). The artifact directory keeps the name
`normalized/` for stability, but the current alpha does not write normalized
records yet; schema alignment remains a documented design target.

## Design Principles

- Record uncertainty; do not silently fix it.
- Treat heuristic confidence as evidence, not probability. Uncalibrated
  extractors should not imply certainty.
- Every run produces inspectable artifacts on disk.
- Adapters are thin: PageLedger does not own extraction, it owns the process
  around extraction.
- The manifest, route map, and provenance files should be useful without a
  running service or database.
- Preserve separate citations for software and source data.

## Related Documents

- [`schemas/`](schemas/) — JSON Schema files, the machine-readable authority
  for artifact contracts
- [`docs/design.md`](docs/design.md) — design targets: router, schema aligner,
  rerun policies, staged CLI, open research questions
- [`docs/comparison.md`](docs/comparison.md) — positioning against the 2026
  extraction ecosystem
- [`docs/run-manifest-spec.md`](docs/run-manifest-spec.md),
  [`docs/route-map-spec.md`](docs/route-map-spec.md),
  [`docs/audit-spec.md`](docs/audit-spec.md),
  [`docs/provenance-spec.md`](docs/provenance-spec.md),
  [`docs/rerun-manifest-spec.md`](docs/rerun-manifest-spec.md) — per-artifact
  field specs
- [`docs/adapter-protocol.md`](docs/adapter-protocol.md) — how to wrap an
  OCR/VLM tool as an adapter
- [`AGENTS.md`](AGENTS.md) — orientation for AI coding agents working in this
  repository
- [`CITATION.cff`](CITATION.cff) — how to cite PageLedger in research output
