Metadata-Version: 2.5
Name: brewdoc
Version: 0.3.0
Summary: Brew any document into LLM-ready Markdown: PDF tables, spreadsheets, docx - no ML, deterministic, with a JSON receipt
Project-URL: Homepage, https://github.com/kochetkov-ma/brewdoc
Project-URL: Repository, https://github.com/kochetkov-ma/brewdoc
Project-URL: Issues, https://github.com/kochetkov-ma/brewdoc/issues
Author-email: Maksim Kochetkov <apmatypa88@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: docx,llm,markdown,pdf,rag,table-extraction,xlsx
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Text Processing :: Markup :: Markdown
Requires-Python: >=3.12
Requires-Dist: pdfplumber==0.11.10
Requires-Dist: python-calamine==0.8.2
Requires-Dist: selectolax==0.4.13; python_version < '3.15' and platform_python_implementation == 'CPython'
Provides-Extra: render-js
Requires-Dist: quickjs-ng==0.17.0.1; (python_version < '3.15' and platform_python_implementation == 'CPython') and extra == 'render-js'
Description-Content-Type: text/markdown

# brewdoc

Document to Markdown for agent systems: PDF, Word, PPTX presentations, spreadsheets, EPUB and static
HTML in, one Markdown file plus a JSON receipt out. No models or OCR. Local conversion
is offline and deterministic (same file, same bytes). Explicit `--url` acquires a page.
The receipt says what was rendered,
what was dropped and what the route structurally cannot carry.

## Install

Version 0.3.0 is prepared for publication. These registry commands apply only after
0.3.0 is published on PyPI. Original 0.2.0 publication is tracked separately.

| channel | command |
|---|---|
| pip | `pip install brewdoc==0.3.0` |
| uvx | `uvx brewdoc==0.3.0 file.pdf` |
| uv tool | `uv tool install brewdoc==0.3.0` |
| Docker | `RUN pip install brewdoc==0.3.0` |

The existing original source tag can be installed now. It predates EPUB and URL acquisition:

```
uvx --from git+https://github.com/kochetkov-ma/brewdoc@v0.2.0 brewdoc file.pdf
```

Requires Python >= 3.12. Base runtime dependencies: `pdfplumber` (pulls pypdfium2, Pillow,
cryptography) and `python-calamine`. HTML requires CPython 3.12-3.14 and the native Lexbor
parser from this exact conditional dependency:

```text
selectolax==0.4.13; python_version < '3.15' and platform_python_implementation == 'CPython'
```

Other runtimes retain the `.html` route and Python API but explicitly refuse HTML. On supported
CPython, a missing or broken parser is an environment failure and `--self-check` exits 1.
On an unsupported HTML runtime, self-check verifies HTML refusal and checks all other formats.

Version 0.3.0 includes URL acquisition and optional JavaScript capture. After publication,
install the extra with `pip install 'brewdoc[render-js]==0.3.0'`. It uses QuickJS-ng 0.17.0.1
and bundled LinkeDOM 0.18.13. From this source checkout, install the locked extra
with `uv sync --locked --extra render-js`. Run it with
`uv run --locked --extra render-js brewdoc --url URL --render-js`.
No Node, browser engine or invocation-time component download is used.

## Usage

```
brewdoc file.pdf --out file.md
brewdoc slides.pptx --out slides.md
brewdoc book.epub --out book.md
brewdoc page.html --out page.md
brewdoc --url https://example.com/article --out article.md
brewdoc --url https://example.com/article --render-js --timeout 30 --out article.md
brewdoc file.xlsx                # no --out: receipt line, then the Markdown, on stdout
brewdoc file.xlsx --sheet Calc --sheet Inputs
brewdoc file.xlsm --artifact formula/sheet/000002=formulas.json
brewdoc file.xlsm --artifact vba/project/000001=vbaProject.bin
brewdoc --self-check             # renders synthetic fixtures twice, compares sha256, exit 0 when green
```

`--sheet` is repeatable. Names are exact and case-sensitive. Output follows caller order while
keeping each sheet's full-workbook ordinal. `--artifact` is also repeatable. Use keys listed in the
Markdown artifact table. Both options are workbook-only. brewdoc validates every request and
output collision before writing.

URL mode accepts one public HTTP(S) HTML page with strict UTF-8 decoding. It uses
GET, verified TLS and default ports only. Private destinations, URL credentials
and HTTPS downgrades are refused; proxy environment settings are ignored. Each redirect and
resource is validated. Acquisition allows at most five redirects and an 8 MiB
body limit. `--timeout SECONDS` sets one maximum waiting budget for loading and
optional JS, default 30 seconds. It accepts positive finite seconds within the
supported clock range and returns earlier when ready. Increase it for pages
that need longer loading or JS waiting.
Bounded cleanup and local conversion can add time after waiting ends.
No login, cookie profile or bot bypass is offered.
Local files and default static URL capture do not execute JavaScript.

`--render-js` explicitly executes the selected page's scripts locally in a separate
process with bounded APIs and resources. No filesystem or shell APIs are exposed,
and captures do not reuse browser state. This is not a general operating-system
security sandbox. It supports DOM mutations, bounded jobs/timers and guarded GET
requests. Async XHR also supports the public Algolia search endpoint used by
hn.algolia.com, with its original string body and no cookies or user credentials.
Other POST requests remain unsupported; search redirects are refused without replay.
Supported module loading uses the native engine. Temporary `localStorage`
starts empty for each capture; accessing it marks the result partial.
Each capture starts with a writable `innerWidth` of 1024 CSS pixels.
Width changes stay within that capture.
It does not provide browser layout, iframe execution,
workers, WebSockets, media or browser-wide compatibility. Native JS requires a
verified QuickJS public ABI on macOS ARM64 or 64-bit glibc Linux (x86_64/aarch64).
Windows, macOS Intel, musl Linux and 32-bit systems refuse JS. Missing components or unsupported
runtimes refuse before fetching.
See [URL acquisition limits](FORMATS.md#url-acquisition).

URL receipts use `brewdoc.receipt/2` with redacted acquisition provenance and the
frozen snapshot identity. Timing-only expiry preserves available valid content
with `acquisition.capture_status=partial` and a diagnostic.
The receipt distinguishes the current DOM from an explicitly marked fallback to
the already acquired original HTML, where JS completion is unknown. No HTML is
returned from an incomplete response. Hard network, security, heap, native fault,
tamper or resource failures exit 1 and preserve an existing output file.
Exit 0 confirms conversion, not script completion or article fidelity.
Live acquisition can change; conversion of the same frozen snapshot remains
deterministic. No 90% coverage or
global JavaScript compatibility is claimed.

Combining `--url` with a local path or `--self-check`, or using `--render-js` without
`--url`, exits 2 before acquisition. Invalid timeout values and an explicit
`--timeout` without `--url` also exit 2. URL requests with `--sheet` or `--artifact`
exit 1 with an HTML refusal receipt before HTTP.

Every successful document uses `brewdoc.markdown/2`. A quoted source title is followed by Metadata,
Artifacts, Known omissions, Contents, then anchored content units. PDF keys are `page/000001`,
workbook keys are `sheet/000001`, DOCX keys are `chapter/000001`, PPTX keys are `slide/000001`,
EPUB keys follow declared spine ordinals as `chapter/000001`, and HTML has one `chapter/000001`.
Metadata contains deterministic source identity, selection, content hashes, conversion tallies, and
artifact capabilities. It excludes paths, timestamps, permissions, host data, and unsupported author
metadata. EPUB adds source-declared publication titles, creators, languages and identifiers.

For a conversion request, one JSON receipt line goes to stdout first (wrapped here, `unit_keys` omitted):

```json
{"artifacts": [], "broken_ligature_words": 0, "columns_split": 1,
 "dropped": {"cid_survivors": 0, "control_chars": 0, "ligatures": 0, "nbsp": 0,
             "non_ascii_replaced": 0, "page_numbers": 3, "pua_glyphs": 0,
             "running_heads": 0, "soft_hyphens": 0},
 "file_ok": true, "markdown_schema": "brewdoc.markdown/2",
 "not_carried": ["images, figures and the text drawn inside them",
                 "a table that spans a page break",
                 "text rotated out of the horizontal reading order"],
 "out": "file.md",
 "reason": "pdf rendered: 7 pages, 1 tables, 26 text regions, 1 column splits",
 "receipt_schema": "brewdoc.receipt/1",
 "route": "pdf", "source": "nist-sp800-145.pdf", "tables": 1,
 "text_regions": 26, "unit_kind": "page", "units": 7}
```

| key | meaning |
|---|---|
| `file_ok`, `reason` | `false` when the file was refused (no text layer, unreadable, unsupported suffix); `reason` names why, or summarises what was rendered |
| `route` | which reader ran: `pdf`, `doc`, `presentation`, `sheet`, `epub` or `html` |
| `source` | the local filename, or the redacted requested URL in URL mode |
| `unit_kind`, `units` | the route's content unit and how many were rendered; one pair for every format. `page` for PDF, `chapter` for DOCX, EPUB and HTML, `slide` for PPTX, `sheet` for a workbook, and `none` on a suffix brewdoc does not read |
| `tables`, `text_regions`, `columns_split` | what else was rendered |
| `receipt_schema` | the receipt key contract; `brewdoc.receipt/1` replaced the per-format `pages` and `sheets` counters |
| `broken_ligature_words` | words with a ligature the font mapped to a stray code point, folded to ASCII, not repaired |
| `dropped` | sanitised items removed on purpose, counted by kind (running heads, page numbers, ...) |
| `not_carried` | what this route structurally cannot represent - read it before concluding a fact is absent |
| `out` | the Markdown path, or `null` when it went to stdout |
| `markdown_schema`, `unit_keys` | the contract and ordered full-source content keys; success only |
| `artifacts` | available keyed formula or opaque VBA outputs and requested output paths; success only |
| `selected_sheets` | the `--sheet` names in caller order; present only when sheets were selected |

Paths resolve against the current working directory; `--out` creates parent directories.
Exit code is non-zero on refusal; the receipt line is still printed.

## Formats

| in | reader | notes |
|---|---|---|
| `.pdf` | pdfplumber | needs a text layer; route chosen per region, never per page |
| `.docx` | stdlib zip + XML | paragraphs and tables |
| `.pptx` | stdlib zip + XML | slides in declared order; text, tables, notes and picture metadata |
| `.epub` | stdlib zip + XML | EPUB 2/3 static XHTML in declared spine order; Unicode metadata and navigation |
| `.xlsx` `.xlsm` `.xls` `.xlsb` `.ods` | python-calamine | selected sheets keep full-source ordinals; numbers as stored (`85.0`), dates as ISO days |
| `.html` | selectolax Lexbor | full static body or fragment, complete tables and source-declared controls; CPython 3.12-3.14 |

`.xlsx` and `.xlsm` expose original OOXML formulas as one deterministic JSON artifact per
formula-bearing selected sheet. Markdown still contains cached values. `.xlsm` and `.xlsb` can expose
one complete related `vbaProject.bin` as opaque bytes. brewdoc does not execute macros, decompile VBA,
report source-module counts, evaluate formulas, or infer dependency graphs.

HTML reads local strict UTF-8, with an optional BOM. It preserves meaningful Unicode, navigation,
main content, sidebars and footers in recovered DOM order. It never fetches links or resources,
executes scripts, submits forms or infers interactive values. Links use a restricted scheme policy;
form actions remain inert metadata. Inline code outside tables becomes a literal fenced block;
surrounding emphasis and links remain balanced. Safe links in literal code and around code/images
retain destination records without changing the source literals. Table-cell code uses labelled
escaped text. Quotes and list items containing standalone child blocks use explicit nesting
boundaries in place of visual Markdown
containers. Plain list paragraphs retain their item marker and continuation indentation.
See [HTML scope and limits](FORMATS.md#static-html).

EPUB reads one package with static XHTML in EPUB 2 or 3. Each declared spine item becomes
a chapter, including empty and non-linear items, front matter and back matter. Publication
navigation retains source hierarchy and links to generated chapter and fragment anchors.
Declared metadata and meaningful Unicode are preserved. Images remain inert alt/title/target
records; no pixels or external resources are read. EPUB adds no parser dependency.
Malformed required content, unsafe archives, unavailable navigation targets, encrypted reading
text, declared fixed layout and unsupported required media without an XHTML fallback refuse
the whole book before output writes. Multiple rootfiles are outside this supported profile.
See [EPUB scope and limits](FORMATS.md#epub).

The public Python API exposes EPUB, HTML and presentation rendering and generic workbook artifact access:

```python
import brewdoc

markdown, tally = brewdoc.render_epub("book.epub")
code, receipt, markdown = brewdoc.run("book.epub", out="book.md")
markdown, tally = brewdoc.render_html("page.html")
code, receipt, markdown = brewdoc.run("page.html", out="page.md")
markdown, tally = brewdoc.render_presentation("slides.pptx")
refs = brewdoc.list_book_artifacts("book.xlsm", sheets=("Calc",))
payload = brewdoc.read_book_artifact("book.xlsm", refs[0].key, sheets=("Calc",))
code, receipt, markdown = brewdoc.run(
    "book.xlsm", out="book.md", sheets=("Calc",),
    artifact_outputs={refs[0].key: "formulas.json"},
)
```

`artifact_outputs` maps artifact keys to paths. The CLI builds it from repeated `--artifact KEY=PATH`.

CSV and TSV are intentionally unsupported and are not planned. LLMs can read their text tables
directly, so brewdoc does not convert them to Markdown.

Planned formats, excluded ones, deferred ones and the never-list: `FORMATS.md`.
A planned format is refused by name today. The adapter contract and the ordered checklist for
adding one are in [`docs/architecture.md`](docs/architecture.md).

PDF regions, in the order tried:

| region looks like | rendered as |
|---|---|
| ruled on all sides | Markdown table from the drawn ruling |
| ruling unfinished (cells never closed) | source baselines recovered as logical rows on the inferred structural grid |
| ruled across only (head and foot rules) | column grid from the rule segment ends |
| unruled, tabular band | layout-preserved text |
| unruled, captioned `Table N` | the band is bounded first (caption to next blank band), then read as a table |
| two-column prose | split at the gutter (relative density on the character x-histogram), left then right |
| plain prose | lines |

## What it does not do

| gap | behaviour |
|---|---|
| OCR | a PDF with no text layer is refused by name, never rendered empty |
| PDF and DOCX images, figures, text drawn inside them | not carried; listed in `not_carried` |
| PPTX pictures | represented by deterministic name, size, alt text, caption, source and byte-count placeholders; pixels are never emitted |
| EPUB layout, scripts, media and graphics | CSS, pagination, font rendering, scripts, active media, image pixels and unsupported SVG/MathML semantics are omitted |
| PPTX charts, SmartArt, audio, video and embedded objects | not carried; listed in `not_carried` |
| a table spanning a page break | columns recovered, wrapped rows stay separate rows |
| text rotated out of horizontal reading order | not carried |
| sub/superscripts | folded back into their line (`NH4H2PO4`, not `NHHPO` over `4 2 4`) |
| broken font CMaps (ligatures mapped to wrong code points) | counted (`broken_ligature_words`), folded to ASCII, never guessed back |
| formula calculation | cached values in Markdown; passive source formulas only for `.xlsx` and `.xlsm` |
| readable VBA source | unavailable; `.xlsm` and `.xlsb` preserve a related project only as opaque bytes |

## Determinism

No timestamps or unordered iteration. Two renders of one file have the same sha256;
`--self-check` pins it. PDF, DOCX, PPTX and workbook output remains ASCII. HTML and EPUB Markdown use
UTF-8 without BOM and LF newlines, with the same bytes in the API, file output and CLI stdout.
Receipt JSON keeps ASCII escaping.

## Size and speed

These measurements cover the existing PDF/workbook baseline before the HTML parser addition.
They do not measure HTML or EPUB conversion or the new installed footprint.

| measure | value |
|---|---|
| installed footprint | ~45 MB (pdfplumber, pypdfium2, Pillow, cryptography, python-calamine) |
| import | 0.08 s |
| 130-page PDF | ~5.5 s |
| 14-sheet xlsx | ~0.9 s |

## Benchmarks

Browse the [corpus catalog](benchmarks/CATALOG.md), follow the
[measurement method](benchmarks/METHODOLOGY.md), and inspect the
[full-run results](benchmarks/results/README.md). Completed runs include actual
outputs and known failures; capture completion does not mean content quality passed.

## Releasing

Tag-triggered, see `RELEASING.md`.

## License

MIT, see `LICENSE`.
