Metadata-Version: 2.4
Name: ragprepkit
Version: 0.1.4
Summary: Document preprocessing toolkit for RAG and LLM pipelines: text cleaning, chunking strategies, metadata extraction, and token counting.
Project-URL: Homepage, https://www.mindrops.com/open-source/ragprepkit
Project-URL: Documentation, https://github.com/mindropsrepo/ragprepkit#readme
Project-URL: Repository, https://github.com/mindropsrepo/ragprepkit
Project-URL: Issues, https://github.com/mindropsrepo/ragprepkit/issues
Project-URL: Changelog, https://github.com/mindropsrepo/ragprepkit/blob/main/CHANGELOG.md
Author-email: Mindrops <info@mindrops.com>
License: MIT
License-File: LICENSE
Keywords: chunking,document-processing,embeddings,llm,nlp,rag,text-preprocessing
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.9
Provides-Extra: dev
Requires-Dist: build>=1.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.4.0; extra == 'dev'
Requires-Dist: twine>=5.0; extra == 'dev'
Provides-Extra: tokenizers
Requires-Dist: tiktoken>=0.5.0; extra == 'tokenizers'
Description-Content-Type: text/markdown

# RagPrepKit

[![PyPI](https://img.shields.io/pypi/v/ragprepkit.svg)](https://pypi.org/project/ragprepkit/)
[![Python Versions](https://img.shields.io/pypi/pyversions/ragprepkit.svg)](https://pypi.org/project/ragprepkit/)
[![License](https://img.shields.io/pypi/l/ragprepkit.svg)](https://pypi.org/project/ragprepkit/)

Document preprocessing toolkit for RAG (retrieval-augmented generation) and
LLM pipelines. `ragprepkit` handles the unglamorous but high-leverage work that
sits between "raw document" and "ready to embed": cleaning noisy text,
splitting it into retrieval-sized chunks, pulling out lightweight structural
metadata, and estimating token counts before you ever call a model.

It has zero required dependencies. Everything works out of the box on
Python 3.9+; installing the optional `tiktoken` extra upgrades token counting
from a heuristic estimate to an exact count.

## Why this exists

Most RAG bugs aren't in the retrieval or the prompt — they're in the
preprocessing step nobody looked at closely: boilerplate leaking into
embeddings, sentences getting cut in half at chunk boundaries, or token
budgets blowing up because nobody counted before sending. `ragprepkit` is a
small, inspectable, dependency-light layer for that step, not a framework
that owns your whole pipeline.

## Features

- **Text cleaning** — unicode normalization, control-character stripping,
  configurable boilerplate-line removal (cookie notices, copyright footers,
  newsletter prompts), and whitespace normalization.
- **Three chunking strategies** — fixed-size character windows, sentence-safe
  chunking that never splits mid-sentence, and a recursive
  paragraph → sentence → fixed-size fallback for mixed long-form documents.
- **Lightweight metadata extraction** — word/sentence counts, estimated
  reading time, markdown headings, extracted URLs, and a coarse script-based
  language guess.
- **Token counting** — exact counts via an optional `tiktoken` integration,
  with a dependency-free heuristic fallback so the library always works.
- **Zero required dependencies** — the core package has no runtime
  dependencies; `tiktoken` is opt-in via an extra.
- **Fully typed** — type hints throughout, with a `py.typed` marker for
  PEP 561 compatibility with type checkers.

## Project structure

```
ragprepkit/
├── src/
│   ├── ragprepkit/
│   │    ├── __init__.py     # public API exports, __version__
│   │    ├── cleaning.py     # clean_text, normalize_whitespace, strip_boilerplate
│   │    ├── chunking.py     # Chunk, fixed_size_chunks, sentence_chunks, recursive_chunks
│   │    ├── metadata.py     # DocumentMetadata, extract_metadata
│   │    ├── tokens.py       # count_tokens, estimate_cost
│   │    └── py.typed 
│   └──tests                 # pytest test suite (mirrors src/ragprepkit modules)                 
├── pyproject.toml           # build config, metadata, dependencies
├── LICENSE
└── README.md
```

## Installation

```bash
  pip install ragprepkit
```

```python
from ragprepkit import clean_text
```

Requires Python 3.9 or later.

## Quick start

```python
from ragprepkit import clean_text, recursive_chunks, extract_metadata, count_tokens

raw = """
# Quarterly Report

Cookie Policy applies to this site.

Revenue grew 12% year over year, driven primarily by expansion
in the enterprise segment. Customer churn declined for the third
consecutive quarter.

## Outlook

Management expects continued growth into next year, though macro
headwinds remain a risk. See https://example.com/full-report for
the full filing.

All rights reserved 2026.
"""

text = clean_text(raw)
chunks = recursive_chunks(text, chunk_size=300, overlap=30)
meta = extract_metadata(text)

print(f"{len(chunks)} chunks, {meta.word_count} words, {meta.estimated_reading_time_minutes} min read")
for chunk in chunks:
  print(f"[chunk {chunk.index}] ({count_tokens(chunk.text)} tokens) {chunk.text[:60]}...")
```

## Usage examples

### 1. Cleaning scraped or exported documents

```python
from ragprepkit import clean_text

raw_html_text = """
Sign up for our newsletter
This is the main content of the article.

It contains    extra      whitespace.
"""

cleaned = clean_text(
    raw_html_text,
    remove_boilerplate=True,
    extra_boilerplate_patterns=[r"^Sign up for our newsletter.*$"],
)

print(cleaned)
```

`clean_text` runs unicode normalization, control-character stripping,
boilerplate-line removal, and whitespace normalization in one pass. Each
step is also exposed individually (`normalize_whitespace`,
`strip_boilerplate`) if you want to compose your own pipeline.

### 2. Choosing a chunking strategy

```python
from ragprepkit import fixed_size_chunks, sentence_chunks, recursive_chunks

text = """
# Quarterly Report

Revenue grew 12% year over year, driven by strong enterprise demand.

The company expanded into three new markets during the quarter.

Management expects continued growth into next year, though macroeconomic
conditions remain uncertain.
"""

# Uniform windows — fastest, ignores structure. Good for short, dense text.
fixed = fixed_size_chunks(text, chunk_size=80, overlap=10)

# Never splits a sentence — good when exact quotes/citations matter.
by_sentence = sentence_chunks(text, max_chars=80, overlap_sentences=1)

# Paragraph-first, falling back to sentence- then fixed-size splitting.
# The best default for mixed long-form documents (reports, articles, docs).
by_structure = recursive_chunks(text, chunk_size=80, overlap=10)

print(f"Fixed: {len(fixed)} chunks")
print(f"Sentence: {len(by_sentence)} chunks")
print(f"Recursive: {len(by_structure)} chunks")
```

Each strategy returns a list of `Chunk` objects carrying `text`,
`start_char`, `end_char`, `index`, and an open `metadata` dict you can
populate with your own fields (source document ID, page number, etc.)
before handing chunks to your embedding step.

```python
for chunk in by_structure:
    chunk.metadata["source"] = "quarterly_report.pdf"
    chunk.metadata["page"] = estimate_page_number(chunk.start_char)
```

### 3. Extracting metadata for filtering and routing

```python
from ragprepkit import extract_metadata

document_text = """
# Quarterly Report

Revenue grew 12% year over year.

For more information visit:
https://example.com

## Outlook

Management expects continued growth into next year.
"""

meta = extract_metadata(document_text)

print(meta.word_count)
print(meta.headings)  # ['Quarterly Report', 'Outlook']
print(meta.urls)      # ['https://example.com']
print(meta.estimated_reading_time_minutes)
print(meta.likely_language)

```

Useful for building filters ("only index docs over 200 words"), surfacing
a table of contents from headings, or routing documents to a
language-specific pipeline before deeper processing.

### 4. Token counting and cost estimation

```python
from ragprepkit import count_tokens, estimate_cost

text = """
Revenue grew 12% year over year, driven by strong enterprise demand.
Management expects continued growth into next year.
"""

# Count tokens in the text.
tokens = count_tokens(text)
print(tokens)

# Estimate processing cost.
cost = estimate_cost(text, price_per_1k_tokens=0.003)
print(cost)
```

### 5. End-to-end pipeline sketch

```python
from ragprepkit import clean_text, recursive_chunks, count_tokens


def prepare_for_embedding(raw_text: str, source_id: str, max_tokens_per_chunk: int = 300):
    cleaned = clean_text(raw_text)
    chunks = recursive_chunks(cleaned, chunk_size=1200, overlap=120)

    prepared = []
    for chunk in chunks:
        if count_tokens(chunk.text) > max_tokens_per_chunk:
            # Fall back to a tighter split for oversized chunks.
            chunk_pieces = recursive_chunks(chunk.text, chunk_size=600, overlap=60)
        else:
            chunk_pieces = [chunk]

        for piece in chunk_pieces:
            prepared.append({
                "text": piece.text,
                "source_id": source_id,
                "tokens": count_tokens(piece.text),
            })

    return prepared


text = """
# Quarterly Report

Revenue grew 12% year over year, driven by strong enterprise demand.
Management expects continued growth into next year.
"""

prepared = prepare_for_embedding(text, source_id="quarterly_report.pdf")
print(prepared[0])
```

## API reference

| Function | Module                | Purpose |
|---|-----------------------|---|
| `clean_text(text, **opts)` | `ragprepkit.cleaning` | Full cleaning pipeline |
| `normalize_whitespace(text)` | `ragprepkit.cleaning` | Collapse spaces/blank lines |
| `strip_boilerplate(text, extra_patterns=None)` | `ragprepkit.cleaning` | Remove boilerplate lines |
| `fixed_size_chunks(text, chunk_size, overlap)` | `ragprepkit.chunking` | Character-window chunking |
| `sentence_chunks(text, max_chars, overlap_sentences)` | `ragprepkit.chunking` | Sentence-safe chunking |
| `recursive_chunks(text, chunk_size, overlap)` | `ragprepkit.chunking` | Paragraph → sentence → fixed fallback |
| `extract_metadata(text, words_per_minute=200)` | `ragprepkit.metadata` | Word/sentence counts, headings, URLs, language guess |
| `count_tokens(text, encoding_name="cl100k_base")` | `ragprepkit.tokens`   | Token count (exact w/ tiktoken, else estimate) |
| `estimate_cost(text, price_per_1k_tokens, encoding_name)` | `ragprepkit.tokens`   | Rough cost estimate for a given price point |

## Development

```bash
 git clone https://github.com/mindropsrepo/ragprepkit.git
 cd ragprepkit
 pip install -e ".[dev,tokenizers]"
 pytest
 ruff check .
```

## Design notes / limitations

- `extract_metadata`'s language detection is a coarse latin vs. non-latin
  script heuristic, not real language identification — pair with a proper
  library (e.g. `langdetect`, `fasttext`) if you need accurate ISO codes.
- `extract_metadata`'s sentence count is a simple punctuation-based
  heuristic and can overcount on things like decimal numbers or
  abbreviations — treat it as an estimate, not an exact linguistic count.
- `count_tokens` falls back to a `len(text) // 4` heuristic without
  `tiktoken` installed. This is a commonly cited rough average for English
  text, not an exact count for every tokenizer or language — install the
  `tokenizers` extra for precision.
- `estimate_cost` intentionally takes price as an argument rather than
  embedding a pricing table, since provider pricing changes frequently.


## About Mindrops

**Mindrops** is a software company that develops AI, cloud, web, mobile, and enterprise software solutions.

**ragprepkit** is an open-source library developed and maintained by Mindrops for document preprocessing in Retrieval-Augmented Generation (RAG) and large language model (LLM) workflows. The project focuses on practical utilities for text cleaning, chunking, metadata extraction, and token counting.

For more information, visit [Mindrops](https://www.mindrops.com).

## Links

- 🌐 **Mindrops Open Source:** [mindrops.com/open-source](https://www.mindrops.com/open-source)
- 📦 **RagPrepKit:** [Project page](https://www.mindrops.com/open-source/ragprepkit)
- 🐙 **GitHub:** [Repository](https://github.com/mindropsrepo/ragprepkit)
- 📧 **Contact:** [info@mindrops.com](mailto:info@mindrops.com)

## License

Ragprepkit is released under the MIT License.

MIT — see [LICENSE](LICENSE).