Metadata-Version: 2.4
Name: chunkrank
Version: 1.0.0
Summary: Model-aware text chunking and answer re-ranking for LLM pipelines. Automatically adapts chunk size to tokenizer and context window, then consolidates and ranks answers across chunks.
License: MIT
License-File: LICENCE
Author: Amit
Requires-Python: >=3.14,<4.0
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.14
Provides-Extra: all
Provides-Extra: cohere-embed
Provides-Extra: dev
Provides-Extra: openai-embed
Provides-Extra: semantic
Requires-Dist: anthropic (>=0.25) ; extra == "all"
Requires-Dist: cohere (>=5.0) ; extra == "all"
Requires-Dist: cohere (>=5.0) ; extra == "cohere-embed"
Requires-Dist: deptry (>=0.24.0,<0.25.0)
Requires-Dist: mypy (>=1.19.1,<2.0.0)
Requires-Dist: numpy (>=1.26)
Requires-Dist: openai (>=1.0) ; extra == "all"
Requires-Dist: openai (>=1.0) ; extra == "openai-embed"
Requires-Dist: pytest (>=8.0) ; extra == "dev"
Requires-Dist: pytest-asyncio (>=0.23) ; extra == "dev"
Requires-Dist: rank-bm25 (>=0.2.2)
Requires-Dist: regex (>=2023.10.3)
Requires-Dist: ruff (>=0.14.10,<0.15.0)
Requires-Dist: scikit-learn (>=1.5)
Requires-Dist: sentence-transformers (>=2.7,<3.0) ; extra == "all"
Requires-Dist: sentence-transformers (>=2.7,<3.0) ; extra == "semantic"
Requires-Dist: tiktoken (>=0.7) ; extra == "all"
Requires-Dist: transformers (>=4.40) ; extra == "all"
Project-URL: Documentation, https://github.com/AmitoVrito/chunkrank#readme
Project-URL: Issue Tracker, https://github.com/AmitoVrito/chunkrank/issues
Project-URL: Source Code, https://github.com/AmitoVrito/chunkrank
Description-Content-Type: text/markdown

# ChunkRank: Model-Aware Chunking + Answer Ranking
```
Used internally for long-document QA and evaluation pipelines handling 1,000+ PDFs.
```
```
ChunkRank is a lightweight Python library that automatically chunks 
text based on an LLM’s tokenizer and context window, then consolidates
and ranks answers across chunks. In short ChunkRank is a model-aware text 
chunking and answer re-ranking library for LLM pipelines.
```

🔗 PyPI : https://pypi.org/project/chunkrank/

---

## Why ChunkRank?

When working with LLMs, long documents must be split into chunks, but:
- Every model has **different tokenizers and context limits**
- Chunk sizes are usually **hard-coded and error-prone**
- Answer quality drops when responses come from **multiple chunks**
- Existing RAG frameworks are **heavy** when you only need chunking + ranking

**ChunkRank solves this gap.**

---

## What It Does

✅**Model-aware chunking**  
- Pass a model name (`gpt-4o-mini`, `claude-3.5-sonnet`, `Llama-3.1-8B` etc.)   
- ChunkRank automatically:
  - Selects the correct tokenizer
  - Applies the correct context window
  - Reserves token space for prompts and responses

No manual token math. No trial-and-error.
  
✅**Answer consolidation & ranking**  
- Query runs across multiple chunks
- Multiple candidate answers are produced
- ChunkRank **re-ranks** them to return the best answer
Works standalone — no full RAG stack required.

---

## Installation

```bash
pip install chunkrank
```
or for development:
```bash
poetry install

```
## Quick Example

```python
import chunkrank

text = open("document.txt").read()
question = "What is the main topic of this document?"

chunks = chunkrank.split(text, model="gpt-4o-mini")
answers = chunkrank.answer(question, chunks)
best = chunkrank.rank(answers)

print(best)
```

---

## Core API

```python
import chunkrank

# 1. Split text into model-aware chunks
chunks = chunkrank.split(text, model="gpt-4o-mini")

# 2. Answer the question across all chunks
#    Default: local extractive (no API key required)
answers = chunkrank.answer(question, chunks)

#    With OpenAI:
answers = chunkrank.answer(question, chunks, provider="openai", api_key="sk-...")

#    With Anthropic:
answers = chunkrank.answer(question, chunks, provider="anthropic", api_key="sk-ant-...")

# 3. Rank and return the best answer
best_answer = chunkrank.rank(answers)
```

---

## Pipeline API

```python
from chunkrank import ChunkRankPipeline

# Local (no LLM)
pipe = ChunkRankPipeline(model="gpt-4o-mini")

# With OpenAI
pipe = ChunkRankPipeline(model="gpt-4o-mini", provider="openai", api_key="sk-...")

# With Anthropic
pipe = ChunkRankPipeline(model="gpt-4o-mini", provider="anthropic", api_key="sk-ant-...")

answer = pipe.process(question="What is the main topic?", text=text)
```

---


## Supported Capabilities

- Automatic model → tokenizer → context resolution
- Token, sentence, and paragraph chunking strategies
- Cross-encoder based answer re-ranking
- Works with OpenAI, Anthropic, HF, Llama-based models
- Drop-in utility for QA, summarization, extraction

---

## How It Fits

| Tool | What it does |
|------|-------------|
| LangChain / LlamaIndex | Full RAG pipelines |
| Haystack | End-to-end retrieval frameworks |
| **ChunkRank** | Focused, model-aware chunking + answer ranking |

**ChunkRank complements RAG frameworks — it doesn’t replace them.**

---
## Roadmap
1. Build the **model registry** (model → context window + tokenizer).  
2. Implement **chunking strategies** (tokens, sentences, paragraphs).  
3. Integrate a **re-ranking engine** (start with Hugging Face cross-encoder).  
4. Package and release to PyPI with a simple API.  
---

## Community

- [Contributors](CONTRIBUTORS.md)
- [Maintainers](MAINTAINERS.md)

---

