Metadata-Version: 2.4
Name: markdown-chunkify
Version: 0.3.0
Summary: Split Markdown into hierarchical sections with ids, parent links and source spans.
Keywords: markdown,chunking,text-splitter,rag,embeddings,llm
Author: Gal Gilor
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Text Processing :: Markup :: Markdown
Classifier: Typing :: Typed
Requires-Dist: pydantic>=2.0.2
Requires-Python: >=3.12
Project-URL: Homepage, https://github.com/Gal-Gilor/markdown-chunkify
Project-URL: Changelog, https://github.com/Gal-Gilor/markdown-chunkify/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/Gal-Gilor/markdown-chunkify/issues
Description-Content-Type: text/markdown

[![Lint, test and build](https://github.com/Gal-Gilor/markdown-chunkify/actions/workflows/lint-test-build.yaml/badge.svg)](https://github.com/Gal-Gilor/markdown-chunkify/actions/workflows/lint-test-build.yaml) [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/Gal-Gilor/markdown-chunkify)

# markdown-chunkify

A Python library that splits Markdown into hierarchical sections. It keeps code blocks intact and links each section to its parent section.

## Features

- Lines inside fenced code blocks (```` ``` ```` or `~~~`, including fences in list items), HTML comment blocks, `<pre>`, `<script>`, `<style>` and `<textarea>` blocks and YAML front matter are never treated as headers. Section text keeps them exactly as written
- Splits on headers from H1 to H6. `MarkdownSplitter(max_level=3)` keeps deeper headers as text of the enclosing section
- Gives each section an id and its parent section's id, so the header hierarchy can be rebuilt as a tree or loaded into a graph. Headers with the same text are kept as separate sections
- Text before the first header, or a document without headers, is returned as a level-0 section with an empty header
- `section.breadcrumb()` returns the section's header path, such as `Docker > Installation`, to send with the section text to an embedding model or LLM
- Records where each section is in the source text. `text[start:end]` with `start, end = section.metadata.span` returns the section's header line and text as written. `metadata.char_count` is the length of that span, not of `content` or `to_markdown()`

## Installation

```bash
pip install markdown-chunkify
```

The splitter depends on Pydantic.

## Usage

```python
from markdown_chunkify import MarkdownSplitter

splitter = MarkdownSplitter()

# Split from text
sections = splitter.split_text(markdown_text)

# Split from file
sections = splitter.split_file("document.md")
```

`MarkdownSplitter` is immutable, so one instance can be reused.

### Section ids and parents

`metadata.id` is the section's position in document order, starting at 0, so `sections[i].metadata.id == i`. `metadata.parent_id` is the id of the enclosing section, or `None` for top-level sections and text before the first header.
```python
text = "# Docker\n## Installation\n## Usage\n# Kubernetes"
sections = MarkdownSplitter().split_text(text)

[(s.header, s.metadata.id, s.metadata.parent_id) for s in sections]
# [('Docker', 0, None), ('Installation', 1, 0), ('Usage', 2, 0), ('Kubernetes', 3, None)]

sections[sections[1].metadata.parent_id].header  # 'Docker'
```

The other relationships follow from these two fields:
- A section's children are the sections whose `parent_id` is its `id`.
- Sections with the same `parent_id` are siblings, whatever their header levels. In `# A`, `### C`, `## B`, both C and B are children of A. A skipped level adds no placeholder section.
- The previous and next sections in reading order have ids `id - 1` and `id + 1`.

Ids are unique within one document. To use them across documents, combine them with your own document key, such as `f"{doc_key}:{section.metadata.id}"`. Ids are positions, not content versions: inserting or removing a section renumbers the sections after it, and editing a section's text keeps its id. To detect changed sections between runs, for example to skip re-extracting entities from unchanged text, hash `content`. Changing `max_level` changes the tree and the ids.

### Breadcrumbs

`section.breadcrumb()` joins the parent headers and the section's own header. Sent with the section text, it tells an embedding model or LLM what the section is about when the text itself says "it" or "this step".
```python
text = "# Docker\n## Installation\nRun the installer."
section = MarkdownSplitter().split_text(text)[1]

section.breadcrumb()  # 'Docker > Installation'
section.breadcrumb(separator=" / ")  # 'Docker / Installation'

prompt_text = f"{section.breadcrumb()}\n\n{section.content}"
```

Text before the first header returns an empty string. The breadcrumb is computed from `metadata.parents`, so it is not part of the serialized section.

### Cleaning text from PDF parsers

PDF parsers often produce ligatures (`ﬁ`, `ﬂ`), non-breaking spaces and fullwidth characters. A chunk with `ﬁnd` doesn't match a search for `find`, and entity extraction can miss the word. The splitter doesn't clean text. Clean it before calling `split_text`, so the spans index into the cleaned string you pass in.
```python
import unicodedata

from markdown_chunkify import MarkdownSplitter

clean = unicodedata.normalize("NFKC", pdf_markdown)
clean = clean.replace("\u00ad", "").replace("\u200b", "")  # Soft hyphen, zero-width space
sections = MarkdownSplitter().split_text(clean)
```

NFKC normalization from the standard library replaces ligatures, non-breaking spaces and fullwidth characters, and keeps accents:

| Input | NFKC output |
|---|---|
| `ﬁnd ﬂow` | `find flow` |
| `a\xa0b` (non-breaking space) | `a b` |
| `ＡＢＣ１２３` (fullwidth) | `ABC123` |
| `Müller` | `Müller` |
| `x²` | `x2` |
| `co\xadoperate` (soft hyphen) | unchanged |
| `zero\u200bwidth` (zero-width space) | unchanged |

NFKC also folds superscripts and subscripts, so `x²` becomes `x2`. It leaves soft hyphens and zero-width spaces in the text, so remove them with `str.replace`.

### Limitations

- Only ATX headers (`#` to `######`) are recognized. Setext headers, underlined with `===` or `---`, stay in the section text.
- Headers and fences inside blockquotes (`> # Title`) are not recognized.
- A line indented by 4 or more spaces that starts with ```` ``` ```` or `~~~` is treated as a fence, even inside an indented code block. If that fence is never closed, the headers after it are treated as text and a warning is logged.
- An HTML comment or a `<pre>`, `<script>`, `<style>` or `<textarea>` block is only recognized when it starts the line. One that opens mid-line (`text <!--` or `text <pre>`) does not hide the `#` lines inside it.
- Headers inside `<details>` are split, as GitHub renders Markdown after a blank line inside `<details>`. The `<details>` and `</details>` tags stay in the text of the sections around them.
- A header with no text (`#`, `## ##`, or only whitespace after the `#`) is kept as text of the enclosing section, not an empty section. CommonMark renders it as an empty heading.
- A document that starts with `---` is treated as having YAML front matter only if the line after it is a `key:` entry and every line up to the next `---` or `...` looks like YAML (`key: value`, `- item`, indented lines, comments or blank lines). Otherwise the `---` is a horizontal rule and the headers after it are split as usual.
- Sections have no size limit. A document without headers is returned as one section and a long section stays whole, so split long sections before embedding them.

## Data Models

`Section` and `SectionMetadata` are exported from the package root:
```python
from markdown_chunkify import Section
from markdown_chunkify import SectionMetadata
```

The models are frozen, so to change a section, call `section.model_copy(update={"content": "..."})`, which returns a changed copy. The freezing is shallow: `metadata.parents` is a plain dict, so treat it as read-only.

### Section

A section returned by the `split_text` and `split_file` methods. It holds the section's header, text, header level and metadata.
```python
class Section(BaseModel):
    model_config = ConfigDict(frozen=True)

    # The header of the section (without #), empty for text before the first header
    header: str
    # The content of the section
    content: str
    # Number of # symbols (1-6), 0 for text before the first header
    level: int
    # Hierarchy and position information
    metadata: SectionMetadata

    # Convert the section back to Markdown
    def to_markdown(self) -> str: ...

    # Join the parent headers and the section's header, such as "Docker > Installation"
    def breadcrumb(self, separator: str = " > ") -> str: ...
```

### SectionMetadata

A section's id, its parent's id, its parent headers and where it is in the source text.
```python
class SectionMetadata(BaseModel):
    model_config = ConfigDict(frozen=True)

    # Position in document order, starting at 0
    id: int
    # Id of the enclosing section, None for top-level sections
    parent_id: int | None
    # Parent headers by level, such as {"h1": "Guide", "h2": "Install"}
    parents: dict[str, str]
    # Start (inclusive) and end (exclusive) offsets in the source text
    span: tuple[int, int]
    # Number of characters in the span, computed from it
    char_count: int
```

The span starts at the header line and ends at the last non-whitespace character of the section text. It does not include subsections, leading blank lines or trailing whitespace. Offsets count Python characters (code points), not UTF-8 bytes or UTF-16 units, in the string passed to `split_text`, including a byte order mark. `split_file` offsets index into the file's contents as decoded with its `encoding`, with line endings kept as written.

## Requirements

Python 3.12 or later.

## Contributing

Report bugs and request features in [GitHub issues](https://github.com/Gal-Gilor/markdown-chunkify/issues). For a large change, open an issue to discuss it before you start.

Set up a development environment with [uv](https://docs.astral.sh/uv/):
```bash
git clone https://github.com/Gal-Gilor/markdown-chunkify.git
cd markdown-chunkify
uv sync
```

Run the tests and Ruff before opening a pull request:
```bash
uv run pytest -vv
uv run ruff check .
uv run ruff format .
```

Open pull requests against `main`. The [Lint, test and build](https://github.com/Gal-Gilor/markdown-chunkify/blob/main/.github/workflows/lint-test-build.yaml) workflow runs Ruff, the tests on Python 3.12, 3.13 and 3.14, and a wheel build on each pull request. Add tests for changed behavior, and add user-facing changes to [CHANGELOG.md](https://github.com/Gal-Gilor/markdown-chunkify/blob/main/CHANGELOG.md).

To release a version, update `version` in `pyproject.toml` and the changelog, then publish a GitHub release tagged `v<version>`, such as `v0.3.0`. The [Release](https://github.com/Gal-Gilor/markdown-chunkify/blob/main/.github/workflows/release.yaml) workflow builds the package and publishes it to PyPI with trusted publishing.

## License

Apache License 2.0. See [LICENSE](https://github.com/Gal-Gilor/markdown-chunkify/blob/main/LICENSE).
