Metadata-Version: 2.4
Name: chunk-factory
Version: 0.1.5
Summary: An efficient chunking library that integrates traditional and advanced methods, with real-time evaluation of chunking results.
Home-page: https://github.com/hjandlm/Chunk-Factory
Author: Jie Huang
Author-email: hjie97bi@gmail.com
License: MIT
Keywords: python,windows,mac,linux,text chunk,chunker,eval text chunk,LLM chunk,openai,gemini
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Framework :: Jupyter
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: Microsoft :: Windows :: Windows 10
Classifier: Operating System :: Microsoft :: Windows :: Windows 11
Classifier: Operating System :: Unix
Requires-Python: >=3.10.0
Description-Content-Type: text/markdown
Requires-Dist: transformers>=4.49.0
Requires-Dist: tiktoken>=0.9.0
Requires-Dist: seaborn>=0.13.2
Requires-Dist: matplotlib>=3.10.1
Requires-Dist: sentence-transformers>=4.0.2
Requires-Dist: pandas>=2.2.3
Requires-Dist: openai>=1.68.2
Requires-Dist: google-genai>=1.9.0
Requires-Dist: jieba>=0.42.1
Requires-Dist: nltk>=3.9.1
Requires-Dist: tqdm>=4.67.1
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary


<div align='center'>

![Chunk-Fcatory Logo](./chunk_factory/assets/logo.png)

# ✨Chunk-Factory ✨
[![PyPI](https://img.shields.io/badge/pypi-v0.1.0-blue)](https://pypi.org/project/chunk-factory/)
[![MIT License](https://img.shields.io/badge/license-MIT-blue)](https://opensource.org/licenses/MIT)
[![Documentation](https://img.shields.io/badge/docs-Usage.md-blue.svg)](Usage.md)
[![GitHub stars](https://img.shields.io/github/stars/hjandlm/Chunk-Factory?style=social)](https://github.com/hjandlm/Chunk-Factory/stargazers)

_Chunk-Factory is a fast, efficient text chunking library with real-time evaluation._

[Instroduction](#Instroduction) •
[Installation](#Installation) •
[Usage](#Usage) •
[Supported Methods](#supported-methods) •
[TODO](#TODO) •
[References](#References) •
[Citation](#Citation) •

</div>

## Instroduction
**Chunk-Factory** is a Python library that offers various text chunking methods, including both traditional approaches and state-of-the-art techniques. It not only provides efficient text chunking but also offers real-time evaluation metrics, allowing immediate assessment of chunking results. These features are crucial for retrieval-augmented tasks, helping to optimize context extraction and utilization in the retrieval process.

With Chunk-Factory, users can easily chunk text and evaluate its effectiveness, making it suitable for a wide range of natural language processing applications, particularly in scenarios that require fine-grained retrieval and document segmentation.

Note: Every time I do RAG, I have to chop up semantically coherent text into chunks and then have no clue whether it’s good or not. I can only guess based on the retrieval results, but can’t tell if it’s the retriever’s fault or the chunking’s fault. This library is here to solve that problem by evaluating the quality of the chunking first. Hopefully, it can help some people out of their misery—so annoying!

## Installation
To install Chunk-Factory, simply run:

```bash
pip install chunk-factory
```

## Usage

Here's a basic example to get you started:

```python
from chunk_factory import Chunker

text = 'Chunk-Factory is a Python library that offers various text chunking methods, including both traditional approaches and state-of-the-art techniques. It not only provides efficient text chunking but also offers real-time evaluation metrics, allowing immediate assessment of chunking results. These features are crucial for retrieval-augmented tasks, helping to optimize context extraction and utilization in the retrieval process.'
language = 'en'

ck = Chunker(text,language)
text_chunks = ck.basechunk(chunk_size=20,chunk_overlap=5)
for i,chunk in enumerate(text_chunks):
    print(f'Number {i+1}: ', chunk)
```

## Supported Methods
Chunk-Factory provides several chunkers to help you efficiently split your text for RAG tasks or other natural language processing tasks (such as information extraction). Here's a quick overview of the available chunkers:

- **BaseChunker**: Splits text based on words or tokens.

- **SegmentChunker**: Splits text into chunks based on sentence or paragraph boundaries.

- **DensexChunker**: Splits text into propositions.

- **LumberChunker**: Splits text based on semantics using LLM.

- **MspChunker**: Splits text based on label probabilities from a small model to determine chunking.

- **PPLChunker**: Splits text based on perplexity.

## TODO
- [ ] Add traditional text chunking methods 

  - [✔️] Add segment chunking method

  - [ ] Add semantic chunking method  

- [ ] Add retrieval evaluation methods


## References
* MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented Generation System [[Paper]](https://arxiv.org/abs/2503.09600)![](https://img.shields.io/badge/arXiv-2025.03-red)
* Dense X Retrieval: What Retrieval Granularity Should We Use? [[Paper]](https://openreview.net/forum?id=WO0WM0xrJo)![](https://img.shields.io/badge/EMNLP-2024-blue)
* LumberChunker: Long-Form Narrative Document Segmentation [[Paper]](https://aclanthology.org/2024.findings-emnlp.377/)![](https://img.shields.io/badge/EMNLP-2024-blue)
* Meta-chunking: Learning efficient text segmentation via logical perception [[Paper]](https://arxiv.org/abs/2410.12788)![](https://img.shields.io/badge/arXiv-2024.11-red)



## Citation
If you use Chunk-Factory in your research, please cite it as follows:

```
@misc{chunkfactory2025,
  author = {Jie H},
  title = {Chunk-Factory: A toolkit with a variety of text chunking methods},
  year = {2025},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/hjandlm/Chunk-Factory}},
}
```



