Metadata-Version: 2.5
Name: split_markdown4gpt
Version: 2.7.7
Summary: Split large Markdown files into token-bounded chunks for LLM context windows
Project-URL: Homepage, https://github.com/twardoch/split-markdown4gpt
Project-URL: Documentation, https://twardoch.github.io/split-markdown4gpt/
Project-URL: Source, https://github.com/twardoch/split-markdown4gpt
Project-URL: Bug Tracker, https://github.com/twardoch/split-markdown4gpt/issues
Author-email: Adam Twardoch <adam+github@twardoch.com>
License: Apache-2.0
License-File: AUTHORS.md
License-File: LICENSE.txt
Keywords: chunking,context-window,data-preprocessing,gpt,gpt-3,gpt-4,llm,markdown,mistletoe,natural-language-processing,nlp,openai,openai-gpt,python,split-text,summarization,text-analysis,text-processing,text-summarization,text-tokenization
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Text Processing
Classifier: Topic :: Text Processing :: Markup :: Markdown
Requires-Python: >=3.10
Requires-Dist: fire>=0.5.0
Requires-Dist: mistletoe>=1.1.0
Requires-Dist: python-frontmatter>=1.0.0
Requires-Dist: pyyaml>=5.1
Requires-Dist: regex>=2023.6.3
Requires-Dist: syntok>=1.4.4
Requires-Dist: tiktoken>=0.4.0
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.0; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.20; extra == 'docs'
Provides-Extra: testing
Requires-Dist: pytest-cov>=4.0; extra == 'testing'
Requires-Dist: pytest>=7.0; extra == 'testing'
Description-Content-Type: text/markdown

# split-markdown4gpt: split large Markdown for LLM context windows

LLMs have finite context windows. A 50,000-word documentation site won't fit in one API call. `split-markdown4gpt` breaks Markdown files into token-bounded chunks that respect document structure — so each chunk stays coherent rather than cutting mid-sentence or mid-section.

## Why splitting matters

Every LLM API call has a hard token limit (e.g. 4,096 tokens for GPT-3.5, 8,192 for GPT-4). Exceeding it raises an error. Naive character-based chunking cuts through sentences, code blocks, and headings — destroying context. This tool:

1. Parses the Markdown AST (via `mistletoe`)
2. Counts tokens using the same tokenizer the model uses (via `tiktoken`)
3. Packs as many complete sections as fit within the limit
4. When a single section exceeds the limit, falls back to sentence-level splitting (via `syntok`)

The result: chunks that respect heading hierarchy, never split a sentence across chunks, and are as large as possible without exceeding the token budget.

## Install

```bash
pip install split_markdown4gpt
```

## CLI

```bash
mdsplit4gpt large_doc.md --model gpt-3.5-turbo --limit 4096 --separator "=== SPLIT ==="
```

Outputs all sections joined by the separator. Redirect to a file or pipe to another tool.

| Flag | Default | Description |
|------|---------|-------------|
| `--model` | `gpt-3.5-turbo` | Tokenizer model (determines token counting) |
| `--limit` | model max | Max tokens per chunk |
| `--separator` | `=== SPLIT ===` | String inserted between chunks in output |

## Python API

```python
from split_markdown4gpt import split

# Returns a list of Markdown strings, each within the token limit
chunks = split("large_doc.md", model="gpt-3.5-turbo", limit=4096)

for i, chunk in enumerate(chunks):
    response = openai_client.chat.completions.create(
        model="gpt-3.5-turbo", messages=[{"role": "user", "content": chunk}]
    )
```

Also accepts a string or file-like object:

```python
chunks = split(markdown_string, model="gpt-4", limit=8000)
chunks = split(open("doc.md"), limit=2000)
```

## How it works

```
split(md)
  ├── meta_data()        strip YAML front matter
  ├── MarkdownLLMSplitter.build_md_dict()
  │     ├── mistletoe Document  parse Markdown AST
  │     └── build dict tree keyed by heading level
  ├── calculate_sizes()  count tokens recursively (cached with lru_cache)
  └── get_sections_from_md_dict_by_limit()
        ├── walk tree, accumulate blocks until limit reached
        └── oversized blocks → sentence-split via syntok
```

### `MarkdownLLMSplitter` class

```python
from split_markdown4gpt import MarkdownLLMSplitter

splitter = MarkdownLLMSplitter(gptok_model="gpt-4", gptok_limit=6000)
splitter.load_md("doc.md")
splitter.build()

for section in splitter.gen_section_dicts():
    print(section["gptok_size"], "tokens")
    print(section["md"])
```

Methods: `load_md()`, `load_md_path()`, `load_md_str()`, `build()`, `list_section_texts()`, `list_section_dicts()`, `gen_section_texts()`, `gen_section_dicts()`, `split()`.

## Supported models

Any model supported by `tiktoken`, including all `gpt-3.5-turbo` and `gpt-4` variants. The limit defaults to the model's published context window.

## Dependencies

- `tiktoken` — OpenAI-compatible token counting
- `mistletoe` — Markdown AST parser
- `syntok` — sentence segmentation for oversized sections
- `frontmatter` — YAML front matter stripping
- `fire` — CLI
- `regex`, `PyYAML` — utilities

## License

Apache License 2.0. Copyright © 2023 Adam Twardoch.
