Metadata-Version: 2.4
Name: tinychunk
Version: 0.1.0
Summary: A tiny, zero-dependency text chunking library for Python.
Project-URL: Homepage, https://github.com/mmexu/tinychunk
Project-URL: Repository, https://github.com/mmexu/tinychunk
Project-URL: Issues, https://github.com/mmexu/tinychunk/issues
Project-URL: Changelog, https://github.com/mmexu/tinychunk/blob/main/CHANGELOG.md
Author: mmexu
License-Expression: MIT
License-File: LICENSE
Keywords: chunking,llm,nlp,rag,split,text
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: mypy>=1.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Description-Content-Type: text/markdown

# tinychunk

A tiny, zero-dependency text chunking library for Python.

Split text into manageable pieces by characters, words, sentences, paragraphs, or lines — with optional overlap, boundary-aware modes, and metadata. Pure Python standard library, nothing else.

## Why tinychunk?

Most chunking libraries pull in additional dependencies such as tokenizers, embedding models, or larger framework toolkits. `tinychunk` focuses on a single task: splitting text into chunks. It is implemented using only the Python standard library and keeps the API intentionally small and predictable.

Use it when you need simple, predictable text segmentation without adding runtime dependencies.

## Installation

```bash
pip install tinychunk
```

Requires Python 3.10 or newer. No runtime dependencies.

## Quick Start

```python
from tinychunk import chunk

with open("article.txt", encoding="utf-8") as f:
    text = f.read()

chunks = chunk(text, size=500, mode="chars")

for c in chunks:
    print(c)
```

## Features

* Five chunking modes: `chars`, `words`, `sentences`, `paragraphs`, `lines`
* Configurable chunk size and overlap
* Boundary-aware chunking for words, sentences, paragraphs, and lines
* Optional metadata: chunk index, start position, and end position
* Iterator mode to process chunks one by one without building the final result list
* Custom literal separators for domain-specific splitting
* Zero runtime dependencies
* Fully type-hinted public API
* MIT licensed

## Quality

* Extensive pytest test suite covering normal usage, edge cases, Unicode, and error handling
* Type-hinted public API, passes mypy in strict mode and ruff with extended rules
* Small, dependency-free implementation
* Public API kept intentionally minimal

## Usage

### Basic chunking by characters

```python
from tinychunk import chunk

text = "The quick brown fox jumps over the lazy dog. " * 50
chunks = chunk(text, size=200, mode="chars")
```

In `chars` mode, the text is split by character count. This mode may split inside words because it works at character level.

### Chunking by words

```python
from tinychunk import chunk

text = "The quick brown fox jumps over the lazy dog"
chunks = chunk(text, size=3, mode="words")

print(chunks)
# ['The quick brown', 'fox jumps over', 'the lazy dog']
```

Word mode keeps words intact and preserves the original whitespace between words inside each chunk.

### Chunking with overlap

Overlap is useful when consecutive chunks should share some context.

```python
chunks = chunk(text, size=3, overlap=1, mode="words")

print(chunks)
# ['The quick brown', 'brown fox jumps', 'jumps over the', 'the lazy dog']
```

The `overlap` value is measured in the unit of the selected mode:

* characters for `mode="chars"`
* words for `mode="words"`
* sentences for `mode="sentences"`
* paragraphs for `mode="paragraphs"`
* lines for `mode="lines"`

### Chunking by sentences

```python
from tinychunk import chunk

text = "First sentence. Second sentence. Third sentence. Fourth sentence."
chunks = chunk(text, size=2, mode="sentences")

print(chunks)
# ['First sentence. Second sentence.', 'Third sentence. Fourth sentence.']
```

Sentence splitting is simple and rule-based. It detects sentence boundaries after `.`, `!`, or `?` followed by whitespace. It does not try to handle all linguistic edge cases such as abbreviations.

### Chunking by paragraphs

```python
from tinychunk import chunk

text = "First paragraph.\n\nSecond paragraph.\n\nThird paragraph."
chunks = chunk(text, size=1, mode="paragraphs")

print(chunks)
# ['First paragraph.', 'Second paragraph.', 'Third paragraph.']
```

Paragraphs are separated by blank lines.

### Chunking by lines

```python
from tinychunk import chunk

text = "line 1\nline 2\nline 3"
chunks = chunk(text, size=2, mode="lines")

print(chunks)
# ['line 1\nline 2', 'line 3']
```

Lines are split on newline characters.

### Getting metadata

When you need to know where each chunk came from in the original text, use `with_metadata=True`.

```python
from tinychunk import chunk

text = "The quick brown fox"
chunks = chunk(text, size=2, mode="words", with_metadata=True)

for c in chunks:
    print(f"Chunk {c.index}: chars {c.start}-{c.end}, text={c.text!r}")
```

This returns `Chunk` objects instead of plain strings.

Each `Chunk` contains:

* `text`: the chunk content
* `index`: zero-based chunk index
* `start`: start position in the original text
* `end`: end position in the original text

### Iterator mode

Use `chunk_iter()` when you want to iterate over chunks one by one instead of building the final list of chunks.

```python
from tinychunk import chunk_iter

text = "The quick brown fox jumps over the lazy dog"

for c in chunk_iter(text, size=3, mode="words"):
    print(c)
```

Note: `tinychunk` operates on strings. The input text is still held in memory. `chunk_iter()` avoids building the final list of chunks, but it is not a full streaming parser for files.

### Custom separators

For domain-specific splitting, such as code, logs, or structured documents, you can provide custom literal separators.

```python
from tinychunk import chunk

source_code = """import x

def foo():
    pass

def bar():
    pass
"""

chunks = chunk(source_code, size=1, separators=["\n\ndef "])
```

When `separators` is provided, it overrides `mode`.

Separators are matched as literal strings, not as regular expressions. This keeps the behavior predictable and safe for use with untrusted input.

If multiple separators are provided, the first separator that appears in the text is used.

```python
separators = ["\n\nclass ", "\n\ndef ", "\n\n"]
chunks = chunk(source_code, size=1000, separators=separators)
```

## API Reference

### `chunk(text, size, *, mode="chars", overlap=0, with_metadata=False, separators=None)`

Splits a string into chunks and returns a list.

**Parameters:**

* `text` (`str`): The input text to split.
* `size` (`int`): Target size per chunk. The unit depends on `mode`.
* `mode` (`str`): One of `"chars"`, `"words"`, `"sentences"`, `"paragraphs"`, `"lines"`. Default: `"chars"`.
* `overlap` (`int`): Number of units shared between consecutive chunks. Must be greater than or equal to `0` and smaller than `size`. Default: `0`.
* `with_metadata` (`bool`): If `True`, returns `Chunk` objects instead of strings. Default: `False`.
* `separators` (`list[str] | tuple[str, ...] | None`): Custom literal separators. Overrides `mode` when provided.

**Returns:**

* `list[str]` if `with_metadata=False`
* `list[Chunk]` if `with_metadata=True`

**Raises:**

* `TypeError`: If a parameter has the wrong type.
* `ValueError`: If a parameter has an invalid value.

### `chunk_iter(text, size, *, mode="chars", overlap=0, with_metadata=False, separators=None)`

Same parameters as `chunk()`, but returns an iterator instead of a list.

Use this when you want to consume chunks one at a time.

### `Chunk`

An immutable dataclass with positional metadata.

Fields:

* `text` (`str`): The chunk content.
* `index` (`int`): Zero-based position in the chunk sequence.
* `start` (`int`): Start position in the original text.
* `end` (`int`): End position in the original text.

Example:

```python
from tinychunk import chunk

chunks = chunk("hello world", size=5, mode="chars", with_metadata=True)

first = chunks[0]
print(first.text)
print(first.index)
print(first.start)
print(first.end)
```

## Behavior Notes

### Character mode

`mode="chars"` splits by character count. It can split inside words.

```python
chunk("hello world", size=5, mode="chars")
# ['hello', ' worl', 'd']
```

### Word mode

`mode="words"` splits on whitespace and keeps words intact.

```python
chunk("a b c d e", size=2, mode="words")
# ['a b', 'c d', 'e']
```

### Sentence mode

`mode="sentences"` uses a simple punctuation-based rule. It is intentionally small and dependency-free, not a full natural-language sentence tokenizer.

### Empty input

Empty input returns an empty list.

```python
chunk("", size=5)
# []
```

## Requirements

* Python 3.10 or newer
* No runtime dependencies

## License

MIT — see LICENSE.
