Metadata-Version: 2.5
Name: chunkguard
Version: 0.1.1
Summary: PIC document chunking for RAG with any LLM and embedding model
Project-URL: Homepage, https://github.com/zakariaefakhri/chunkguard
Project-URL: Issues, https://github.com/zakariaefakhri/chunkguard/issues
Project-URL: Paper, https://aclanthology.org/2025.findings-acl.422/
Author: Zakariae Fakhri
License-Expression: MIT
License-File: LICENSE
Keywords: chunking,embeddings,llm,rag,retrieval,text-splitter
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: pydantic>=2.0
Requires-Dist: sentencex>=1.0.30
Provides-Extra: litellm
Requires-Dist: litellm>=1.83.0; extra == 'litellm'
Requires-Dist: pydantic-settings>=2.2; extra == 'litellm'
Description-Content-Type: text/markdown

# chunkguard

[![CI](https://github.com/zakariaefakhri/chunkguard/actions/workflows/ci.yml/badge.svg)](https://github.com/zakariaefakhri/chunkguard/actions/workflows/ci.yml)

Document chunking for RAG with **PIC** (Pseudo-Instruction for document Chunking), with any
LLM and any embedding model. From
[*Document Segmentation Matters for Retrieval-Augmented Generation*](https://aclanthology.org/2025.findings-acl.422/)
(Wang et al., Findings of ACL 2025). This is an independent implementation, not the authors' code.

When you build a knowledge base, you don't know the questions users will ask. PIC stands in
for them with an LLM summary of the document. Every sentence is embedded and compared with
the summary; the threshold is the mean similarity. Neighbouring sentences at or above it
form a chunk of core content, neighbouring sentences below it a chunk of side details. So
chunk boundaries sit exactly where the text turns towards or away from its topic, and every
sentence is judged against the whole document rather than just its neighbour, as in
semantic chunking.

Order is kept, no sentence is cut, nothing is dropped. Cost per document: one LLM call and
one embedding pass over its sentences. No training.

## Install

```bash
pip install chunkguard              # core: bring your own model functions (Pydantic + sentencex)
pip install "chunkguard[litellm]"   # plus .env configuration for 100+ providers via LiteLLM
```

## Choose your models

chunkguard needs two things, and you pick both: an LLM that writes one summary per document,
and an embedding model. They can come from different providers.

**Option 1: plain functions.** `PICChunker` takes any `complete(prompt) -> str` and any
`embed(texts) -> list[vector]`. No provider SDK is involved:

```python
from chunkguard import PICChunker
from sentence_transformers import SentenceTransformer  # local embeddings, as an example
import anthropic  # any LLM, as an example

claude = anthropic.Anthropic()
encoder = SentenceTransformer("BAAI/bge-m3")


def complete(prompt: str) -> str:
    message = claude.messages.create(
        model="claude-sonnet-5-5", max_tokens=512, messages=[{"role": "user", "content": prompt}]
    )
    return message.content[0].text


chunker = PICChunker(complete=complete, embed=lambda texts: encoder.encode(texts).tolist())
chunks = chunker.split(doc_id="report-7", text=cleaned_text, language="en")
```

Options: `max_chars` (default 2000), `min_chars` (default 0, see below), `summary_max_chars`,
`summary_prompt`, `sentence_splitter`.

**Option 2: `.env` with LiteLLM.** Install the extra, copy [`.env.example`](.env.example) to
`.env` and set the two models as [LiteLLM strings](https://docs.litellm.ai/docs/providers):

```bash
CHUNKGUARD_LLM_MODEL=anthropic/claude-sonnet-5-5
CHUNKGUARD_EMBEDDING_MODEL=cohere/embed-multilingual-v3.0
```

Put keys in `.env` as `CHUNKGUARD_LLM_API_KEY` / `CHUNKGUARD_EMBEDDING_API_KEY`. If those are
unset, LiteLLM uses the provider's usual variables (`ANTHROPIC_API_KEY`, `COHERE_API_KEY`,
`AWS_*`, ...) from the process environment. Azure, Bedrock, Vertex, Ollama, vLLM and any
OpenAI-compatible server work the same way. Then:

```python
from chunkguard import chunker_from_env

chunker = chunker_from_env()
chunks = chunker.split(doc_id="report-7", text=cleaned_text, language="en")

for chunk in chunks:
    print(chunk.index, chunk.relevant, round(chunk.similarity, 3), chunk.text[:60])
```

`split` takes plain text that extraction has already cleaned (line breaks inside sentences
joined). Each `Chunk` carries:

| Field | Meaning |
|---|---|
| `doc_id`, `index` | Document and position of the chunk in it |
| `text` | Exact slice `input[start:end]` |
| `start`, `end` | Character offsets into the input (end exclusive) |
| `sentence_start`, `sentence_end` | Sentence range (end exclusive) |
| `relevant` | Core content (`True`) or side details (`False`) |
| `similarity` | Mean cosine similarity of the chunk's sentences to the summary |
| `summary` | The document summary, for use as metadata in the index |

`Chunk` is a Pydantic model; `chunk.model_dump()` gives a dict for your vector store.

### Async

```python
from chunkguard import async_chunker_from_env

chunker = async_chunker_from_env()
chunks = await chunker.split("report-7", cleaned_text, "en")
```

`AsyncPICChunker` takes async `complete` and `embed` functions in the same way.

### Your own summary prompt

The default is the paper's prompt (`chunkguard.SUMMARY_PROMPT`) plus an instruction to answer
in the document's language. Pass `summary_prompt` with `{text}` and `{language}` placeholders
to change it.

### Your own sentence splitter

Sentences come from [sentencex](https://github.com/wikimedia/sentencex) (see [Limits](#limits)).
To use another splitter, pass a function
that returns `(start, end)` character spans. For example, NLTK's Punkt, which the paper uses
for sentence splitting:

```python
from nltk.tokenize.punkt import PunktSentenceTokenizer

punkt = PunktSentenceTokenizer()


def nltk_sentences(text, language):
    return list(punkt.span_tokenize(text))


chunker = PICChunker(complete, embed, sentence_splitter=nltk_sentences)
```

## How it works

1. **Sentences.** Split into sentences with sentencex, keeping character offsets. Every line
   break also ends a sentence, so headings and list items stand alone.
2. **Summary.** One LLM call with the summarizer prompt from the paper (Appendix C), plus an
   instruction to answer in the document's language.
3. **Similarity.** Embed the summary and every sentence; cosine similarity of each sentence
   to the summary.
4. **Threshold.** The mean of those similarities. A sentence is relevant at or above it.
5. **Grouping.** Each run of neighbouring sentences with the same flag becomes one chunk.

Steps 1 to 5 are the method as published. chunkguard adds three things for production use:

6. **Long chunks** (over `max_chars`, default 2000) are split at sentence boundaries into
   near-equal pieces. `max_chars=None` switches this off.
7. **Short chunks** merge into the neighbour with the closest mean similarity when you set
   `min_chars` (off by default), unless that would exceed `max_chars`. A merged chunk is
   `relevant` when its relevant sentences make up at least half of its text. Note that with
   merging, a short document can end up as a single chunk.
8. **Long documents** reach the summarizer as eight evenly spread excerpts totalling
   `summary_max_chars`, so the call fits the context window and every part of the document
   still shapes the summary.

## Configuration

With plain functions, pass the options to `PICChunker` (`max_chars`, `min_chars`,
`summary_max_chars`, `summary_prompt`, `sentence_splitter`). With the LiteLLM extra, they come
from environment variables or `.env`, prefixed `CHUNKGUARD_`:

| Variable | Default | Meaning |
|---|---|---|
| `LLM_MODEL` | required | LiteLLM model string, e.g. `gemini/gemini-2.5-flash` |
| `LLM_API_KEY` | provider variable | API key |
| `LLM_API_BASE` | – | Endpoint, e.g. Azure resource, Ollama or any OpenAI-compatible server |
| `LLM_API_VERSION` | – | Azure only |
| `EMBEDDING_MODEL`, `EMBEDDING_API_*` | | Same settings for the embedding model |
| `EMBEDDING_BATCH_SIZE` | `64` | Texts per embedding request |
| `REQUEST_TIMEOUT` | `120` | Seconds per request |
| `MAX_RETRIES` | `2` | Retries per request on provider errors |
| `MAX_CHUNK_CHARS` | `2000` | Split longer chunks; `0` = no limit |
| `MIN_CHUNK_CHARS` | `0` | Merge shorter chunks into a neighbour; `0` = off |
| `SUMMARY_MAX_CHARS` | `100000` | Excerpt longer documents for the summary; `0` = send all |

Embedding requests stay below 400,000 characters each. A sentence longer than 8,000
characters is truncated for embedding only; its chunk keeps the full text.

LiteLLM versions 1.82.7 and 1.82.8 were compromised on PyPI in March 2026; the extra requires
1.83.0 or later.

## Limits

- **Sentence splitting is rule-based.** sentencex carries punctuation and abbreviation rules
  for many languages and falls back to related ones. chunkguard adds two language-neutral
  rules: no cut before a lowercase letter, and no cut after a number of up to three digits
  ("im 18. Jahrhundert"), which merges the rare English "We sold 3. Then ..." into one
  sentence. Some boundaries are still missed (a sentence ending in "etc."), and languages
  without sentence punctuation (Thai) are not split. Pass your own splitter if that matters.
- **Sizes are in characters, not tokens.** Pick `MAX_CHUNK_CHARS` with your embedding model's
  token limit in mind.
- **The relevance flag is relative to the document.** Nearly every document has relevant and
  irrelevant chunks, however on-topic it is overall. Use it to weight or filter retrieval,
  not as an absolute quality score.
- **Retrieval gains are the paper's results** on English Wikipedia and open-domain QA. This
  repository does not reproduce them.

## Development

```bash
uv sync
uv run pytest
uv run ruff check . && uv run ruff format --check .
```

Releases: bump `version` in `pyproject.toml`, then push a matching tag (`v0.1.0`). The
release workflow publishes to PyPI via trusted publishing.

## Citation

If you use PIC in research, cite the paper:

```bibtex
@inproceedings{wang-etal-2025-document,
  title     = {Document Segmentation Matters for Retrieval-Augmented Generation},
  author    = {Wang, Zhitong and Gao, Cheng and Xiao, Chaojun and Huang, Yufei and Si, Shuzheng and
               Luo, Kangyang and Bai, Yuzhuo and Li, Wenhao and Duan, Tangjian and Lv, Chuancheng and
               Lu, Guoshan and Chen, Gang and Qi, Fanchao and Sun, Maosong},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2025},
  pages     = {8063--8075},
  year      = {2025},
  url       = {https://aclanthology.org/2025.findings-acl.422/}
}
```

## License

[MIT](LICENSE)
