Metadata-Version: 2.4
Name: pdf-jsonl-smith
Version: 0.2.0
Summary: Convert born-digital PDF text blocks to BigQuery-ready JSONL.
Project-URL: Homepage, https://github.com/yeiichi/pdf-jsonl-smith
Project-URL: Documentation, https://pdf-jsonl-smith.readthedocs.io/
Project-URL: Repository, https://github.com/yeiichi/pdf-jsonl-smith
Project-URL: Issues, https://github.com/yeiichi/pdf-jsonl-smith/issues
Author-email: Eiichi YAMAMOTO <info@yeiichi.com>
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: pymupdf
Provides-Extra: dev
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: python-semantic-release; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Provides-Extra: docs
Requires-Dist: furo; extra == 'docs'
Requires-Dist: sphinx; extra == 'docs'
Provides-Extra: gcp
Requires-Dist: google-cloud-documentai; extra == 'gcp'
Requires-Dist: google-cloud-storage; extra == 'gcp'
Description-Content-Type: text/markdown

# pdf-jsonl-smith

`pdf-jsonl-smith` converts PDF-derived text into compact BigQuery-ready
newline-delimited JSON.

The `local` app is a thin, boring wrapper around PyMuPDF for born-digital PDFs.
The `docai` app runs Google Document AI Form Parser, but still emits compact
purpose-built records instead of full Document AI payloads.

## Install

```bash
uv pip install -e ".[dev]"
```

For the Google Document AI app:

```bash
uv pip install -e ".[dev,gcp]"
```

## CLI

```bash
pdf-jsonl-smith local input.pdf -o text_units.jsonl
```

Optional metadata:

```bash
pdf-jsonl-smith local input.pdf \
  --output text_units.jsonl \
  --source-uri gs://bucket/input.pdf \
  --schema-version 0.1
```

By default, the CLI prints a short completion summary to stdout:

```text
Wrote 42 records to text_units.jsonl from input.pdf.
```

Use `--quiet` to suppress the summary.

The older `pdf2jsonl` command remains available as a compatibility alias for
the local PyMuPDF extractor:

```bash
pdf2jsonl input.pdf -o text_units.jsonl
```

The `docai` app uploads a local PDF to Cloud Storage, runs Google Document AI,
and writes compact JSONL records:

```bash
pdf-jsonl-smith docai input.pdf \
  --output compact.jsonl \
  --project my-project \
  --location us \
  --processor-id PROCESSOR_ID \
  --staging-uri gs://bucket/pdf-jsonl-smith/input/
```

By default, `docai` emits compact `text`, `key-values`, `tables`, and `lists`
records. Use `--include` to narrow the output:

```bash
pdf-jsonl-smith docai input.pdf \
  --output compact.jsonl \
  --project my-project \
  --location us \
  --processor-id PROCESSOR_ID \
  --staging-uri gs://bucket/pdf-jsonl-smith/input/ \
  --include text,key-values
```

Use `--jsonl-gcs-uri` to upload the compact JSONL output after writing it
locally.

For a beginner-friendly explanation of Document AI, buckets, and authentication,
see the Document AI page in the published docs.

The `bq` app is reserved for the BigQuery workflow:

```bash
pdf-jsonl-smith bq
```

The `docai` app policy is to extract from rich Google Document AI payloads, but
emit only compact, purpose-built records. Full Document AI payloads are not part
of the final JSONL output.

Useful `jq` checks:

```bash
jq -r '.unit_type' compact.jsonl | sort | uniq -c
```

```bash
jq -r 'select(.unit_type=="text") | .text' compact.jsonl
```

```bash
jq -r 'select(.unit_type=="key_value") | [.key, .value, .confidence] | @tsv' \
  compact.jsonl
```

Because the output is one unified JSONL stream, filter by `unit_type` when you
want only main text, form key-value pairs, table cells, or list items.

## Python API

```python
from pdf_jsonl_smith import extract_pdf_to_jsonl

count = extract_pdf_to_jsonl(
    input_path="input.pdf",
    output_path="text_units.jsonl",
    source_uri="gs://bucket/input.pdf",
)
```

The function returns the number of JSONL records written.

## Output

The output is UTF-8 JSONL: one standalone JSON object per text unit. Records are
flat and BigQuery-friendly.

For the local extractor, extraction starts from PyMuPDF text blocks. Adjacent blocks on the
same page may be merged when their geometry looks like wrapped lines from the
same paragraph. This is intentionally conservative and does not try to infer
document semantics.

Fields:

```text
schema_version STRING
doc_id STRING
source_uri STRING
filename STRING
page_number INT64
unit_type STRING
unit_index INT64
text STRING
bbox_x0 FLOAT64
bbox_y0 FLOAT64
bbox_x1 FLOAT64
bbox_y1 FLOAT64
char_count INT64
word_count INT64
extractor STRING
extracted_at TIMESTAMP
warning STRING
```

## Supported

* Born-digital PDFs.
* PyMuPDF block-based text extraction.
* Conservative same-page wrapped-line merging.
* Google Document AI Form Parser compact output.
* UTF-8 JSONL output.
* BigQuery-friendly flat records.
* One unified JSONL stream with `unit_type` filters.

## Not Supported

For the local extractor:

* Scanned PDFs / OCR.
* Forms.
* Image text.

For all extractors:

* Full Document AI payload mirroring.
* Perfect reading order.
* Cross-page paragraph reconstruction.
* Semantic heading detection.
* Lossless table reconstruction.

## Development

```bash
uv sync --dev
uv run pytest
uv run ruff check
uv run sphinx-build -b html docs/source docs/build/html
uv build
```

## Project Infrastructure

This project is maintained as a `uv`-based Python package.

Releases use Python Semantic Release with zero-version releases enabled for the
initial `0.x` series. The GitHub Actions release workflow runs on `main`,
creates releases from conventional commits, builds distributions with `uv`, and
publishes to PyPI using trusted publishing.

Documentation sources live in `docs/source`, build output goes under
`docs/build`, and the site uses Sphinx with the Furo theme. It is intended to be
built by Read the Docs through the RTD GitHub App integration.

## License

MIT
