Metadata-Version: 2.2
Name: ragkit-chunkwise
Version: 0.1.0
Summary: Token-aware, boundary-respecting text/document chunking for RAG ingestion (stdlib-only).
Author-email: Meet2147 <meetjethwa3@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/Meet2147/pythonLibraries/tree/main/chunkwise
Project-URL: Repository, https://github.com/Meet2147/pythonLibraries
Project-URL: Issues, https://github.com/Meet2147/pythonLibraries/issues
Keywords: rag,chunking,nlp,text-splitting,tokens,embeddings,genai,llm
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: tokenizers
Requires-Dist: tiktoken; extra == "tokenizers"

<p align="center">
  <img src="https://raw.githubusercontent.com/Meet2147/pythonLibraries/main/chunkwise/assets/logo.png" alt="chunkwise" width="460">
</p>

<p align="center">
  <a href="https://pypi.org/project/ragkit-chunkwise/"><img src="https://img.shields.io/pypi/v/ragkit-chunkwise.svg" alt="PyPI"></a>
  <img src="https://img.shields.io/pypi/pyversions/ragkit-chunkwise.svg" alt="Python versions">
  <img src="https://img.shields.io/badge/license-MIT-green.svg" alt="License: MIT">
</p>

# chunkwise

**Token-aware, boundary-respecting text and document chunking for RAG ingestion.**

> Part of the **ragkit** suite. Install with `pip install ragkit-chunkwise`, then `import chunkwise`.

Naive character slicing shreds sentences and paragraphs and ignores model token
limits. `chunkwise` splits text along real structural boundaries (paragraphs →
lines → sentences → words → characters), packs the pieces up to a token budget,
and carries a configurable overlap between chunks — so your retrieval index gets
clean, self-contained passages instead of arbitrary fragments.

- Pure Python standard library at runtime. No required third-party dependencies.
- Python 3.8+.
- Pluggable token counting — plug in `tiktoken`, a HuggingFace tokenizer, or any
  `callable(str) -> int`.
- Accurate character offsets back into the original text.

## Install

```bash
pip install ragkit-chunkwise
```

Optional tokenizer extra (pulls in `tiktoken`):

```bash
pip install "ragkit-chunkwise[tokenizers]"
```

Local development (from `chunkwise/`):

```bash
pip install -e .
```

## Quick Start

```python
from chunkwise import chunk_text

paragraph = (
    "Retrieval-augmented generation grounds a language model in your own data. "
    "You first split documents into chunks, embed them, and store the vectors. "
    "At query time you retrieve the most relevant chunks and feed them to the model."
)

for c in chunk_text(paragraph, chunk_size=15, chunk_overlap=4):
    print(c.index, c.token_count, repr(c.text))
```

Each result is a `Chunk` with the text, its position, character offsets, and a
token count.

### The effect of `chunk_size` and `chunk_overlap`

`chunk_size` is the maximum length of a chunk, measured by your
`length_function` (words by default). `chunk_overlap` is how much of the tail of
one chunk is repeated at the head of the next — overlap preserves context that
would otherwise be cut off at a boundary, which improves retrieval recall.

```python
from chunkwise import RecursiveChunker

text = " ".join(f"word{i}" for i in range(60))

# Bigger chunks, no overlap → fewer, disjoint chunks.
print(len(RecursiveChunker(chunk_size=30, chunk_overlap=0).split_text(text)))   # ~2

# Smaller chunks with overlap → more chunks that share context.
print(len(RecursiveChunker(chunk_size=15, chunk_overlap=5).split_text(text)))   # more
```

`chunk_overlap` must be strictly less than `chunk_size` or a `ValueError` is
raised.

## API Reference

### `RecursiveChunker`

```python
RecursiveChunker(
    chunk_size=512,
    chunk_overlap=64,
    separators=None,                 # default: ["\n\n", "\n", ". ", " ", ""]
    length_function=word_token_counter,
    keep_separator=True,
)
```

Recursively splits text using an ordered list of separators, trying the largest
structural boundary first. If a piece still exceeds `chunk_size`, it recurses
with the next separator; the final `""` separator splits by characters as a last
resort. Adjacent small pieces are greedily merged up to `chunk_size`, and
`chunk_overlap` tokens from the previous chunk's tail are carried forward.

Methods:

- `.split_text(text) -> List[str]` — return chunk strings.
- `.chunk(text, metadata=None) -> List[Chunk]` — return `Chunk` objects with
  accurate character offsets, token counts, sequential indexes, and `metadata`
  merged into each chunk.

### `chunk_text(text, chunk_size=512, chunk_overlap=64, **kwargs) -> List[Chunk]`

Convenience wrapper around `RecursiveChunker`. Extra keyword arguments
(`separators`, `length_function`, `keep_separator`) are forwarded to the
chunker. An optional `metadata=` keyword is attached to every chunk.

### `SentenceChunker`

```python
SentenceChunker(chunk_size=512, chunk_overlap=64, length_function=word_token_counter)
```

Splits text into sentences with a lightweight regex (handles `.`, `!`, `?`
followed by whitespace or end-of-string), then packs whole sentences into chunks
up to `chunk_size` with sentence-level overlap. Sentences are never cut mid-way.
Exposes `.split_text(text)` and `.chunk(text, metadata=None)`.

### `chunk_markdown(text, chunk_size=512, chunk_overlap=64, length_function=word_token_counter) -> List[Chunk]`

Splits markdown into sections at headings (lines starting with `#`). The trail of
active headings (outermost → innermost) is attached to each chunk's metadata
under the `"headings"` key, and oversized sections are further split with
`RecursiveChunker`.

```python
from chunkwise import chunk_markdown

md = "# Guide\nIntro.\n\n## Setup\nInstall the package and configure it."
for c in chunk_markdown(md, chunk_size=50):
    print(c.metadata["headings"], "->", repr(c.text))
# ['Guide'] -> '# Guide\nIntro.\n'
# ['Guide', 'Setup'] -> '## Setup\nInstall the package and configure it.'
```

### Custom `length_function`

Any `callable(str) -> int` works. Built-ins:

- `word_token_counter(text)` — whitespace-split word count (default).
- `char_token_counter(text)` — `len(text)`.

To chunk by *real* model tokens, plug in a tokenizer:

```python
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")

from chunkwise import RecursiveChunker
chunker = RecursiveChunker(
    chunk_size=256,
    chunk_overlap=32,
    length_function=lambda t: len(enc.encode(t)),
)
chunks = chunker.chunk(my_document)
```

### The `Chunk` dataclass

| Field         | Type   | Meaning                                                        |
|---------------|--------|---------------------------------------------------------------|
| `text`        | `str`  | The chunk text.                                               |
| `index`       | `int`  | Position in the output sequence (0-based).                    |
| `start`       | `int`  | Character offset of the chunk's primary span in the original. |
| `end`         | `int`  | End character offset (exclusive).                             |
| `token_count` | `int`  | Length of `text` per the `length_function`.                   |
| `metadata`    | `dict` | User metadata (defaults to `{}`).                             |

`len(chunk)` returns `token_count`.

**Offsets and overlap:** for `RecursiveChunker` (and the sentence/markdown
chunkers built on the same logic), `original_text[chunk.start:chunk.end] ==
chunk.text` holds even when overlap is used — successive chunks simply share an
overlapping character range. `start`/`end` always describe the contiguous
primary span a chunk covers.

## Design notes / correctness

- No chunk exceeds `chunk_size` by more than a single indivisible unit (e.g. one
  very long word or a single sentence longer than the budget).
- Empty or whitespace-only input returns `[]`.
- Overlap never produces an infinite loop; an indivisible piece larger than the
  budget is isolated rather than re-seeding subsequent chunks.

## Running the tests

```bash
python -m unittest discover -s tests -v
```

## License

MIT
