Metadata-Version: 2.5
Name: lumberjack-py
Version: 0.5.0
Summary: Structure-aware Markdown, HTML, and DOCX lumber pipeline for RAG preprocessing
Project-URL: Homepage, https://github.com/ouyangfeng2022/lumberjack
Project-URL: Documentation, https://ouyangfeng2022.github.io/lumberjack/
Project-URL: Changelog, https://github.com/ouyangfeng2022/lumberjack/blob/main/CHANGELOG.md
Project-URL: Source, https://github.com/ouyangfeng2022/lumberjack
Project-URL: Repository, https://github.com/ouyangfeng2022/lumberjack
Project-URL: Issues, https://github.com/ouyangfeng2022/lumberjack/issues
Project-URL: Security, https://github.com/ouyangfeng2022/lumberjack/security/policy
Author: tianleG
License-Expression: MIT
License-File: LICENSE
Keywords: chunking,document-processing,docx,html,markdown,rag,splitter,text-splitting
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Text Processing :: Markup :: Markdown
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: markdown-it-py[linkify,plugins]>=4.0.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: all
Requires-Dist: cachetools>=7.1.1; extra == 'all'
Requires-Dist: fastapi>=0.115.0; extra == 'all'
Requires-Dist: haystack-ai>=2.0; extra == 'all'
Requires-Dist: langchain-core>=0.3; extra == 'all'
Requires-Dist: langchain-text-splitters>=0.3; extra == 'all'
Requires-Dist: llama-index-core>=0.12; extra == 'all'
Requires-Dist: openpyxl>=3.1; extra == 'all'
Requires-Dist: python-docx>=1.1.0; extra == 'all'
Requires-Dist: python-multipart>=0.0.18; extra == 'all'
Requires-Dist: starlette<1.0.0; extra == 'all'
Requires-Dist: tiktoken>=0.9.0; extra == 'all'
Requires-Dist: tomli>=2.0; (python_version < '3.11') and extra == 'all'
Requires-Dist: transformers>=4.41.0; extra == 'all'
Requires-Dist: tree-sitter-bash>=0.25; extra == 'all'
Requires-Dist: tree-sitter-c-sharp>=0.23; extra == 'all'
Requires-Dist: tree-sitter-c>=0.24; extra == 'all'
Requires-Dist: tree-sitter-cpp>=0.23; extra == 'all'
Requires-Dist: tree-sitter-go>=0.25; extra == 'all'
Requires-Dist: tree-sitter-java>=0.23; extra == 'all'
Requires-Dist: tree-sitter-javascript>=0.25; extra == 'all'
Requires-Dist: tree-sitter-kotlin>=1.1; extra == 'all'
Requires-Dist: tree-sitter-lua>=0.2; extra == 'all'
Requires-Dist: tree-sitter-php>=0.23; extra == 'all'
Requires-Dist: tree-sitter-python>=0.25; extra == 'all'
Requires-Dist: tree-sitter-ruby>=0.23; extra == 'all'
Requires-Dist: tree-sitter-rust>=0.24; extra == 'all'
Requires-Dist: tree-sitter-swift>=0.7; extra == 'all'
Requires-Dist: tree-sitter-typescript>=0.23; extra == 'all'
Requires-Dist: tree-sitter-zig>=1.1; extra == 'all'
Requires-Dist: tree-sitter>=0.25; extra == 'all'
Requires-Dist: uvicorn>=0.34.0; extra == 'all'
Provides-Extra: code-parsing
Requires-Dist: tree-sitter-bash>=0.25; extra == 'code-parsing'
Requires-Dist: tree-sitter-c-sharp>=0.23; extra == 'code-parsing'
Requires-Dist: tree-sitter-c>=0.24; extra == 'code-parsing'
Requires-Dist: tree-sitter-cpp>=0.23; extra == 'code-parsing'
Requires-Dist: tree-sitter-go>=0.25; extra == 'code-parsing'
Requires-Dist: tree-sitter-java>=0.23; extra == 'code-parsing'
Requires-Dist: tree-sitter-javascript>=0.25; extra == 'code-parsing'
Requires-Dist: tree-sitter-kotlin>=1.1; extra == 'code-parsing'
Requires-Dist: tree-sitter-lua>=0.2; extra == 'code-parsing'
Requires-Dist: tree-sitter-php>=0.23; extra == 'code-parsing'
Requires-Dist: tree-sitter-python>=0.25; extra == 'code-parsing'
Requires-Dist: tree-sitter-ruby>=0.23; extra == 'code-parsing'
Requires-Dist: tree-sitter-rust>=0.24; extra == 'code-parsing'
Requires-Dist: tree-sitter-swift>=0.7; extra == 'code-parsing'
Requires-Dist: tree-sitter-typescript>=0.23; extra == 'code-parsing'
Requires-Dist: tree-sitter-zig>=1.1; extra == 'code-parsing'
Requires-Dist: tree-sitter>=0.25; extra == 'code-parsing'
Provides-Extra: docx
Requires-Dist: python-docx>=1.1.0; extra == 'docx'
Provides-Extra: haystack
Requires-Dist: haystack-ai>=2.0; extra == 'haystack'
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.3; extra == 'langchain'
Requires-Dist: langchain-text-splitters>=0.3; extra == 'langchain'
Provides-Extra: llama-index
Requires-Dist: llama-index-core>=0.12; extra == 'llama-index'
Provides-Extra: spreadsheets
Requires-Dist: openpyxl>=3.1; extra == 'spreadsheets'
Provides-Extra: tokenizers
Requires-Dist: cachetools>=7.1.1; extra == 'tokenizers'
Requires-Dist: tiktoken>=0.9.0; extra == 'tokenizers'
Requires-Dist: transformers>=4.41.0; extra == 'tokenizers'
Provides-Extra: toml
Requires-Dist: tomli>=2.0; (python_version < '3.11') and extra == 'toml'
Provides-Extra: web
Requires-Dist: fastapi>=0.115.0; extra == 'web'
Requires-Dist: python-multipart>=0.0.18; extra == 'web'
Requires-Dist: starlette<1.0.0; extra == 'web'
Requires-Dist: uvicorn>=0.34.0; extra == 'web'
Description-Content-Type: text/markdown

<p align="center">
  <img src="assets/lumberjack-logo.svg" width="160" alt="Lumberjack logo: an axe, document, trees, and a tree ring" />
</p>

# Lumberjack

[![PyPI](https://img.shields.io/pypi/v/lumberjack-py.svg)](https://pypi.org/project/lumberjack-py/)
[![Python](https://img.shields.io/pypi/pyversions/lumberjack-py.svg)](https://pypi.org/project/lumberjack-py/)
[![Docs](https://img.shields.io/badge/docs-GitHub%20Pages-4051b5)](https://ouyangfeng2022.github.io/lumberjack/)
[![CI](https://github.com/ouyangfeng2022/lumberjack/actions/workflows/ci.yml/badge.svg)](https://github.com/ouyangfeng2022/lumberjack/actions/workflows/ci.yml)

**Turn documents and flat records into retrieval-ready chunks with structure and provenance intact.**

Lumberjack is a Python library and CLI for RAG preprocessing. It reads a document tree instead of cutting plain text, preserves headings and source metadata, respects tables and fenced code, and keeps every chunk within a token budget whenever its content can be split safely.

[中文说明](README.zh-CN.md) · [Documentation](https://ouyangfeng2022.github.io/lumberjack/) · [PyPI](https://pypi.org/project/lumberjack-py/)

## Why Lumberjack?

- **Context survives splitting.** Chunks separate heading metadata from body content, so retrieval can retain the section that introduced an answer.
- **Structure stays honest.** Markdown/HTML/DOCX become heading trees; CSV/TSV, JSONL, and logs remain ordered records rather than fabricated sections.
- **Budgets are practical.** The default splitter plans with fast incremental estimates, then records an authoritative final token count.
- **Topology is your choice.** Pack sibling sections, preserve subtrees, or process each section body independently.

## Input formats: now and planned

Lumberjack currently supports the formats below. A “planned” entry is a design
commitment, not an installation promise: do not pass it to `format` or rely on
automatic detection until it is marked supported in a release.

| Format family | Formats | Status | Intended structural model |
| --- | --- | --- | --- |
| Markup documents | Markdown (`.md`, `.markdown`), HTML (`.html`, `.htm`) | Supported | Headings, blocks, tables, lists, code, and source lines. |
| Word-processing documents | DOCX (`.docx`) | Supported | Heading styles, paragraphs, tables, lists, and document properties. |
| Plain and rich text | TXT (`.txt`, `.text`), text logs (`.log`) | Supported | TXT paragraphs/lines; logs are atomic ordered records. RTF remains planned. |
| OpenDocument and legacy word processing | ODT, DOC | Planned | Headings and blocks when the source format exposes them. |
| Spreadsheets and delimited data | CSV (`.csv`), TSV (`.tsv`), XLSX (`.xlsx`, `spreadsheets` extra) | Supported | Header schema plus atomic rows with sheet/row/column provenance. XLS and ODS remain planned. |
| Semi-structured data | JSON (`.json`), JSONL (`.jsonl`, `.ndjson`), XML (`.xml`), YAML (`.yaml`, `.yml`), TOML (`.toml`) | Supported | Scalar key paths, record lines, and XML leaf-element paths; all remain ordered records. |
| Analytical and database exports | SQLite (`.sqlite`, `.sqlite3`, `.db`), SQL dump (`.sql`) | Supported | SQLite table/row provenance and ordered SQL statements. Parquet, Avro, and ORC are deferred. |
| Source code and notebooks | Python, JavaScript/TypeScript, Bash, C/C++, C#, Go, Java, Kotlin, Lua, PHP, Ruby, Rust, Swift, Zig, and Jupyter notebooks | Supported | Install `code-parsing` for Tree-sitter declaration boundaries, syntax-error recovery, and source locations. Other languages are deferred. |
| Presentations and ebooks | PPTX, PPT, ODP, EPUB | Planned | Slides/pages, titles, notes, and ordered content blocks. |
| Messages and archives | EML, MSG, MBOX | Planned | Message headers, body, attachments, and thread provenance. |
| PDFs and images | PDF, PNG, JPG/JPEG, TIFF, WebP | Planned | Native PDF text/layout where available; OCR/layout blocks and page provenance otherwise. |

Flat data is a first-class design case, not a disguised heading tree. CSV,
TSV, JSONL, Parquet, and record-oriented JSON will use ordered record/row units
with schema and field-path provenance. Heading-oriented policies such as
sibling section packing are not meaningful for those inputs; their adapters
must select row/record-aware packing, preserve complete protected rows when
configured, and report logical locations such as row number, column name, JSON
path, page, or sheet. This behavior is planned; it is not implemented by the
current three parsers.

## Inspect every pipeline stage

Today, `Lumberjack.saw()` returns `SplitResult(document, chunks)`, and the
public parser, splitter, and finalizer can be called individually when an
integration needs an intermediate `DocTree` or `ChunkDraft`.

The planned pipeline trace API will make every built-in and optional parsing
stage inspectable through one stable result: raw `Document`, extraction output
(for example OCR/layout), normalized `DocTree`, `ChunkDraft`s, rendered text,
normalized/transformed text, and final `Chunk`s. Visual PDF parsers such as
MinerU, Docling, PaddleOCR-VL, and dots.mocr will be optional integrations—not
core dependencies—and will retain page, bounding-box, and parser provenance.

## Install

```bash
pip install lumberjack-py

# Exact tokenizers, DOCX/XLSX, Tree-sitter code parsing, and the Web API
pip install "lumberjack-py[tokenizers,docx,spreadsheets,code-parsing,web]"
```

Requires Python 3.10 or newer.

## Split your first document

```python
from lumberjack import Lumberjack

result = Lumberjack(max_tokens=500).saw("""
# Deployment

Deploy the service through the approved release workflow.

## Rollback

Keep the previous image available until health checks pass.
""")

for chunk in result.chunks:
    print(chunk.own_heading, chunk.body, chunk.token_count)
```

`result.document` contains the parsed `DocTree`, title, metadata, and source provenance. Each `Chunk` includes its body, final `token_count`, heading context, and source line range when available. See the [five-minute guide](https://ouyangfeng2022.github.io/lumberjack/getting-started/quickstart/) for file inputs, output handling, and optional dependencies.

## Use the CLI

```bash
# Infer a supported format from the file extension.
lumber handbook.md --max-tokens 1200

# Preserve CSV rows as atomic records (also use this for TSV, JSONL, and logs).
lumber people.csv --splitter record --max-tokens 1200

# Emit JSON suitable for an ingestion job.
lumber report.docx --tokenizer tiktoken --splitter subtree > chunks.json
```

The [CLI reference](https://ouyangfeng2022.github.io/lumberjack/reference/cli/) lists every option and its default.

## Choose a splitter

| Splitter | Choose it when you need |
| --- | --- |
| `sibling` | Well-filled chunks that may pack adjacent sibling sections with shared context. |
| `subtree` | A whole section subtree to stay together whenever it fits. |
| `section` (default) | Each section's direct body to be considered independently. |

| Counting mode | Behavior |
| --- | --- |
| Unprefixed / `incremental-*` | Fast running estimates while planning, followed by a final authoritative count. |
| `exact-*` | Fully recount each rendered candidate during planning. |

Tokenizer choice and counting mode are independent. For example, `tiktoken` works with `incremental-sibling`. Read the [splitter decision guide](https://ouyangfeng2022.github.io/lumberjack/concepts/splitting/) before tuning a production pipeline.

## Web API

```bash
lumberjack-serve --reload
```

With the `web` extra installed, the service exposes `POST /lumber/api/split/text` for UTF-8 Markdown, HTML, TXT, LOG, CSV/TSV, JSON/JSONL, XML, and YAML requests, and `POST /lumber/api/split/file` for those formats plus DOCX and XLSX (install `spreadsheets` for XLSX). Select `splitter: "record"` for LOG, CSV/TSV/XLSX, JSON/JSONL, XML, and YAML. The remaining planned formats above are not accepted yet. FastAPI serves interactive OpenAPI documentation at [`/docs`](http://127.0.0.1:9612/docs) while the server is running.

## Learn more

- [Core concepts](https://ouyangfeng2022.github.io/lumberjack/concepts/pipeline/) — `Document` to `Chunk`, rendering, budgets, and source metadata.
- [Configuration guide](https://ouyangfeng2022.github.io/lumberjack/guides/configuration/) — block policies and budget controls.
- [Custom components](https://ouyangfeng2022.github.io/lumberjack/guides/custom-components/) — parsers, tokenizers, splitters, and post-processing stages.
- [Python API reference](https://ouyangfeng2022.github.io/lumberjack/reference/python/)
- [Contributing](CONTRIBUTING.md) · [Security](SECURITY.md) · [License](LICENSE)

Benchmark reporting is under construction; this project does not make unverified quality or performance claims.
