Metadata-Version: 2.5
Name: pagebound
Version: 0.1.0
Summary: Shared PDF ingestion cache: structured markdown and JSON with page and bbox provenance, content-addressed and invalidated by hash.
Project-URL: Homepage, https://github.com/hearthresearch/pagebound
Project-URL: Repository, https://github.com/hearthresearch/pagebound
Project-URL: Issues, https://github.com/hearthresearch/pagebound/issues
Author: José Fernandes
License: MIT
License-File: LICENSE
Keywords: docling,document-parsing,pdf-to-markdown,provenance,research-tools,zotero
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing :: Markup :: Markdown
Requires-Python: >=3.11
Requires-Dist: pypdfium2<6.0.0,>=4.30.0
Requires-Dist: typer>=0.12.0
Provides-Extra: zotero
Requires-Dist: pyzotero>=1.14.0; extra == 'zotero'
Description-Content-Type: text/markdown

# pagebound

Shared PDF ingestion cache. Converts a PDF once, into structured markdown
and JSON that carry page and bounding-box provenance for every block, and
serves that result to every tool that needs it, and to every agent: the
CLI is written so that an LLM agent can find a paper, open its full
text and cite a passage by page from `--help` alone.

## Why

Converting a research PDF into something a machine can reason about is
expensive: the Docling model set is one to two gigabytes, and a warm
conversion of a fifteen-page paper takes fifteen to thirty seconds. A
cost that high, paid repeatedly and independently by every tool on the
same machine, is the classic case for a shared cache.

Every tool that reads papers ends up solving a slice of the same
problem: how to invoke the converter, where to cache the result, how to
keep images out of the text, how to carry page and bounding-box
provenance, what to do when the converter crashes, how to skip a scan
that has no text. pagebound is the union of those slices, written once.

## What it is not

Not a converter. [Docling](https://github.com/docling-project/docling)
does the conversion, and does it well. pagebound invokes it as an
external tool, so the heavy pipeline never enters any consumer's
dependency tree.

Not a Zotero plugin. A Zotero library is one way to find PDFs; a
directory is another. Both are source adapters.

## What is actually scarce

Not the conversion. The *contract*: a cached artifact that records what
produced it, so a consumer can tell whether it still matches the PDF on
disk. Existing tools in this space cache by filename, which fails
silently the moment a PDF is replaced.

pagebound keys every artifact on the source hash *and* the recipe that
produced it: converter, resolved version, and options. A Docling upgrade
produces a new entry rather than silently changing the meaning of an old
one.

## For agents and LLM tools

The most common reader of a cached paper is not a program but an agent
in a conversation: "what does Bandara et al. 2015 say about rigour?",
"is this quote really in the paper?", "brief me on this before I read
it". pagebound is shaped for that reader.

- **A handle you can say.** Every paper has a twelve-character sha256
  prefix, such as `6cbb126970f0`. It names the object directory, needs
  no state beyond the hash, and stops resolving exactly when the PDF
  behind it is replaced. `pagebound path 6cbb126970f0` prints the
  directory; `pagebound path BAK9F4R4` (a Zotero item key) and
  `pagebound path 10.1108/eb024320` (a DOI) do the same.
- **Find, then read.** `pagebound list --json` is the whole index as
  JSON: title, authors, year, DOI, sha256, and whether the text is
  cached. An agent greps it for a surname and a year, takes the sha256,
  and reads `document.md` in the directory `pagebound path` prints.
  Reading never runs Docling, so it costs milliseconds and works on a
  machine without it.
- **Cite by page.** `document.json` carries every block with its page
  and bounding box, and `md_start`/`md_end` offsets into `document.md`,
  so a passage found in the markdown maps back to "page 4, this box"
  for a citation or a highlight, with no fuzzy matching.
- **Help that stands alone.** Every command's `--help` states what it
  prints, what it needs running, its exit codes (0 found, 1 not in the
  cache or gave up, 2 usage error) and the fields of any JSON output,
  so an agent can drive the CLI without reading this file.
- **Honest text.** The markdown carries the source characters, never
  `&amp;` or `\_`, so a verbatim quote check is a string search. Text
  that came out of OCR is marked as such in `document.json`
  (`source.text_layer` false), so an agent can weigh it.

A minimal agent skill is three commands:

    pagebound list --json | jq '.[] | select(.authors[] | test("Bandara")) | select(.year == 2015)'
    pagebound path <sha256 from above>
    cat "$(pagebound path <sha256>)/document.md"

## Installing

    uv add pagebound                    # as a dependency of your project
    uv add "pagebound[zotero]"          # adds pyzotero, for `pagebound sync`
    uv tool install "pagebound[zotero]" # the CLI on its own, on PATH

uv is required either way: Docling is never a dependency, pagebound runs
it through `uv tool run`, so the first conversion on a machine downloads
it. Not on PyPI yet; until then, point uv at the repository.

## Artifacts

Per `(source hash, recipe)`:

| File | For | Mean per paper |
|---|---|---|
| `document.json` | the stable contract; blocks with page, bbox, heading path | 0.17 MB |
| `docling.json` | `DoclingDocument` with page renders stripped, for docling-core consumers | 0.47 MB |
| `document.md` | markdown, with offsets pointing back into the blocks | 0.08 MB |
| `images/` | the paper's figures as PNGs, referenced from `docling.json`, never base64 | 0.4 MB (about 7 figures of 54 KB) |

Measured over 113 papers: about 1.1 MB per paper, so a 5,900-paper
library costs about 6.5 GB per recipe. Every Docling upgrade is a new
recipe and converts again beside the old one, so the cache grows by
that much per upgrade until `pagebound clean` drops the superseded
conversions; `pagebound stats` shows the cost per recipe at any time.

Two things Docling emits are not stored, and they were most of the
bytes. Its JSON carries a base64 render of every page; stripping those
removes 97.9% of it and costs nothing, since the result still validates
as a `DoclingDocument` and still chunks with `HybridChunker`. And
`--image-export-mode referenced` also writes a PNG of every page beside
the figures; nothing references them, and on a real library they were
94% of the whole cache, about 12 MB per paper. pagebound versions before
0.1.0 stored them; `pagebound clean` removes them.

Objects are built in a dot-prefixed staging directory and renamed into
place, so a reader sees a whole object or none; skip dot-prefixed
directories when listing. `document.json` is the completeness sentinel:
a directory without it is a tombstone or a torn write, and must not be
read. `tombstone.json` is `{"reason": ..., "tombstoned_at": ...}`, the
time in ISO 8601 UTC; tombstones written before the field existed have
only the reason. When one source has
several recipe directories, read the one whose `conversion.converted_at`
is newest; `sync` always writes under the recipe that runs now, so that
is the current recipe's artifact whenever one exists.

## Finding a paper: the index

Artifacts are keyed on `(source sha256, recipe)`, which is file identity,
not paper identity. The bridge is `index.sqlite` at the cache root,
filled by `pagebound sync`: one row per Zotero item, carrying the item's
identity, the PDF's hash and location, and the bibliographic metadata
Zotero curates. The row is written before the conversion is attempted,
so an item whose PDF failed to convert is still findable by title or
DOI, and the store says it is tombstoned rather than the index saying
nothing.

    items (
        library_key TEXT,  -- 'user' or 'group:<id>'
        item_key    TEXT,  -- Zotero item key
        sha256      TEXT,  -- the artifact key under objects/
        path        TEXT,  -- the PDF on disk
        size        INTEGER,
        mtime       REAL,
        title       TEXT,
        authors     TEXT,  -- JSON array of names, Zotero order
        year        INTEGER,  -- NULL when Zotero has no date
        doi         TEXT   -- lowercase, prefix-stripped; NULL when absent
    )
    -- PRIMARY KEY (library_key, item_key); indexes on sha256 and doi.

    item_collections (
        library_key TEXT,
        item_key    TEXT,
        collection  TEXT   -- Zotero collection name, flat
    )
    -- PRIMARY KEY (library_key, item_key, collection); index on collection.
    -- Rewritten per item on every sync that sees it, so removals in
    -- Zotero propagate; like the metadata, a snapshot until the next sync.

## Using it from Python

`pagebound.get_document(pdf_path)` returns an `Artifact`: `document`,
`markdown`, `path` (the object directory, where `docling.json` and
`images/` live) and `converted`, true only when this call ran the
converter. It resolves the current recipe, which costs one
`docling --version` per process (about 5 s), unless you pass
`recipe=Recipe(...)` yourself; an explicit recipe also pins that Docling
version on the converter, so the provenance recorded is the Docling that
ran. `timeout=` bounds one Docling run in seconds.

`get_cached_document(pdf_path)` returns the newest complete artifact
across every recipe, or `None`, and never converts: one hash, no
Docling, so it works on a machine without Docling. `cache_status(pdf_path)`
reports `cached`, `tombstoned` (with the `Tombstone`'s reason and time)
or `absent`, for one recipe or, with `recipe=None`, across all of them,
so a batch can say "12 cached, 3 to convert, 1 gave up" before paying
for anything.

All three take `fallback_stores=[Store(), ...]`: read-only caches
consulted after the writable `store`. A hit there is copied into `store`
(about 1 MB per paper), a tombstone there replays, and nothing is
ever written to a fallback. A tool that may write only under its own
project directory points `store` there and lists the shared cache as a
fallback.

Failures are all `PageboundError`. A tombstone replays as
`TombstonedError`, a `ConversionFailedError` carrying `reason` and
`tombstoned_at`; a tombstoned scan raises `TombstonedNoTextLayerError`,
which is also a `NoTextLayerError`, so `except NoTextLayerError` keeps
catching scans whether the verdict is fresh or recorded; a PDF that
cannot be read raises `SourceUnreadableError`; a missing uv or Docling
raises `DoclingUnavailableError`. `force=True` retries a tombstone.

A PDF with no text layer is refused and tombstoned unless
`ocr_scans=True` (`--ocr-scans` on the CLI), which lets Docling OCR it
under the same recipe and retries an earlier no-text-layer tombstone.
Every artifact records the probe's verdict in `document.source`:
`text_layer` (false means the text came out of OCR), `probe_pages` and
`probe_chars`; older artifacts read them as `None`.

`document.md` is a contract: it is rendered here from the JSON's raw
text, never through Docling's markdown serializer, so it carries no
`&amp;`, `\_` or other escapes; every body block with a bounding box
and non-empty text is in it verbatim, tables as rendered tables; and
each such block's `md_start`/`md_end` are Python string indices into
exactly that markdown, with `markdown[md_start:md_end] == block.text`.
Running headers and footers (`layer == "furniture"`) are in
`document.blocks` but not in the markdown, so their offsets are `None`.

`get_document_for_item(library_key, item_key)` skips hashing when a
stat shows the PDF unchanged, and `get_document_for_doi(doi)` reaches an
item through the index, preferring the user library when a DOI is held
in two; both go through the index, a snapshot of the last sync. The
index itself is `pagebound.index.Index` (`lookup`, `lookup_doi`,
`lookup_sha`, `lookup_sha_prefix`, `iter_items`), and `pagebound.sync`
fills it: `sync_items` takes an iterable of `ZoteroItem` (build them
with `iter_items` and a pyzotero client from `zotero_client`, or by
hand), so a consumer with its own Zotero access can sync a collection or
one item without the CLI. `get_document(path)` on its own writes nothing
to the index: the sync is the only writer the index trusts.

A consumer that only reads uses `pagebound.store.Store`:
`find_sha_prefix` expands a handle, `artifacts` lists the recipes holding
a complete artifact, and `latest_artifact` picks the newest conversion,
none of which touches Docling. Non-Python consumers open the
index directly, read-only: the journal mode
is WAL, `PRAGMA user_version` is the schema version (currently 1; any
other value means delete the file and run `pagebound sync`), and
DOIs are stored normalised, so normalise before matching (lowercase,
strip `https://doi.org/`, `http://doi.org/`, `https://dx.doi.org/`, `http://dx.doi.org/` and `doi:` prefixes).

Lookups are exact. Fuzzy search, and metadata for PDFs that never came
through a sync, are deliberately out of scope: matching policy belongs
to consumers, and the sync is the only writer the index trusts.

The CLI mirrors this: `pagebound list [--json] [--collection NAME]`
shows what the index holds and each item's state, `cached`, `tombstoned`
(tried and given up, no artifact) or `absent`; `pagebound path REF`
resolves a paper to its object directory. On a terminal, `list`, `stats`
and `status` render as tables and coloured lines for people; `--json`,
and the paths `path` and `convert` print, are plain text and never
change shape in a pipe.

`REF` is read by shape, so one handle works in conversation and on the
command line: a sha256 prefix (lowercase hex, six or more characters,
git style), a Zotero item key (eight uppercase characters, looked up in
the user library), or a DOI. A prefix that matches more than one object
is an error, never a guess, and one that names an object written by
`pagebound convert` resolves even though that object has no index row.
`pagebound path LIBRARY ITEM` still reaches items in group libraries.

    pagebound path 6cbb126970f0          # sha256 prefix
    pagebound path MH23L252              # Zotero item key
    pagebound path 10.1016/j.jsis.2016.05.001
    pagebound path group:12345 ITEM0002

The twelve-character prefix is the recommended handle: it is what the
object directory is named after, it needs no state beyond the hash, and
it stops resolving exactly when the PDF behind it is replaced.

`path` reads the cache only, so it needs neither Zotero nor Docling. When
a PDF has artifacts under more than one recipe it prints the newest
conversion, judged by the artifact's own `converted_at`; since `sync`
always writes under the recipe that runs now, that is the current
recipe's artifact whenever one exists. A paper that is indexed but has
no converted text exits 1 with the tombstone's reason when there is one.

## License

MIT.
