Metadata-Version: 2.4
Name: preembed
Version: 0.1.1
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Environment :: Console
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing
Requires-Dist: hypothesis>=6 ; extra == 'dev'
Requires-Dist: maturin>=1.5,<2 ; extra == 'dev'
Requires-Dist: pytest>=8 ; extra == 'dev'
Requires-Dist: pytest-cov>=5 ; extra == 'dev'
Requires-Dist: ruff>=0.4 ; extra == 'lint'
Requires-Dist: mypy>=1.9 ; extra == 'lint'
Provides-Extra: dev
Provides-Extra: lint
License-File: LICENSE
Summary: High-performance document preparation toolkit for LLM and RAG pipelines.
Keywords: llm,rag,chunking,document-processing,embeddings
Home-Page: https://github.com/AdityaMunot/preembed
Author: Aditya Munot
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Changelog, https://github.com/AdityaMunot/preembed/blob/main/CHANGELOG.md
Project-URL: Documentation, https://github.com/AdityaMunot/preembed/tree/main/docs
Project-URL: Homepage, https://github.com/AdityaMunot/preembed
Project-URL: Issues, https://github.com/AdityaMunot/preembed/issues
Project-URL: Repository, https://github.com/AdityaMunot/preembed

# preembed

<p align="center">
  <picture>
    <img src="https://raw.githubusercontent.com/AdityaMunot/preembed/main/assets/logo.png" alt="preembed" width="400">
  </picture>
</p>

> Clean, chunk, deduplicate, and score documents before embedding.

**preembed** is a high-performance toolkit that sits between your raw documents and your vector database. It turns messy HTML, markdown, logs, and exports into high-quality, LLM-ready chunks — so your RAG pipeline retrieves better context and hallucinates less.

[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-blue.svg)](https://www.python.org)
[![Rust](https://img.shields.io/badge/Rust-Powered-orange.svg)](https://www.rust-lang.org)

## Why

Most RAG pipelines embed documents directly — noise, duplicates, and all. The result: wasted tokens, bloated vector stores, and retrieval that returns boilerplate instead of answers.

preembed fixes the data before it enters the pipeline:

```
raw documents → clean → chunk → dedupe → score → embed
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                        preembed handles this
```

## Install

```bash
pip install preembed
```

For development (requires Rust toolchain):

```bash
pip install -e ".[dev]"
maturin develop --release
```

## Quickstart

### Pipeline (recommended)

```python
from preembed import Pipeline

result = Pipeline(chunk_size=512, overlap=64).run("path/to/documents/")
result.save_report("report.json")

print(f"{result.stats.total_chunks} chunks, {result.stats.duplicate_ratio:.0%} duplicates removed")
```

### Individual primitives

```python
from preembed import clean_text, chunk_text, dedupe_chunks, score_chunks

text = clean_text("<html><script>tracking()</script><p>Actual content here.</p></html>")
# → "Actual content here."

chunks = chunk_text(text, chunk_size=512, overlap=64)
chunks = dedupe_chunks(chunks)
scored = score_chunks(chunks)
# Each chunk gets a 0-1 quality score + warning labels
```

### Streaming (memory-efficient)

```python
for batch in Pipeline().stream("huge_corpus/", batch_size=50):
    upload_to_vector_db(batch.scored_chunks)
```

Cross-batch deduplication included. Processes one batch at a time — constant memory regardless of corpus size.

### Custom scoring

```python
from preembed import register_scorer

@register_scorer("domain_relevance", weight=0.15)
def domain_scorer(chunk: str) -> float:
    return 1.0 if "patient" in chunk.lower() else 0.3
```

Blends with the built-in scoring engine. Errors default to neutral (0.5) — won't crash your pipeline.

## How it works

| Stage | What it does |
|---|---|
| **Clean** | Strip HTML, scripts, styles. Normalize whitespace and unicode. |
| **Chunk** | Structure-aware splitting. Preserves headings, code blocks, paragraphs. Configurable size + overlap. |
| **Dedupe** | Exact, normalized, and near-duplicate removal via fingerprint similarity. |
| **Score** | Quality score (0–1) based on length, density, structure, duplicates. Warning labels for boilerplate, too-short/long, low-density. |
| **Report** | JSON, Markdown, or HTML diagnostics with per-chunk scores and recommendations. |

## API

```python
# Core primitives
clean_text(text: str) -> str
chunk_text(text: str, chunk_size=512, overlap=64, mode="word", tokenizer=None) -> list[str]
dedupe_chunks(chunks, return_stats=False, near_duplicate_threshold=0.9) -> list[str] | DedupeResult
score_chunks(chunks) -> list[Chunk]

# Pipeline
Pipeline(chunk_size=512, overlap=64, dedupe=True, score=True)
Pipeline.run(source, progress=None) -> PipelineResult
Pipeline.stream(source, batch_size=50, progress=None) -> Iterator[PipelineResult]

# Extensibility
register_scorer(name, weight=0.1)(fn)
clear_scorers()
```

## Development

```bash
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest
```

## License

[MIT](LICENSE)

