Metadata-Version: 2.4
Name: aerochunk
Version: 0.1.2
Summary: A zero-framework, low-latency semantic chunker with visual HTML debugging.
Author: Harsh Rana
Project-URL: Homepage, https://github.com/HarshStats/aerochunk
Project-URL: Bug Tracker, https://github.com/HarshStats/aerochunk/issues
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: sentence-transformers>=2.0.0
Requires-Dist: numpy>=1.21.0

<!-- Shields/Badges -->
![PyPI Version](https://img.shields.io/pypi/v/aerochunk.svg)
![Python Version](https://img.shields.io/pypi/pyversions/aerochunk.svg)
![License](https://img.shields.io/pypi/l/aerochunk.svg)

# AeroChunk: A Zero-Framework Semantic Chunker for RAG

## 1. Executive Summary

AeroChunk is a low-latency, zero-framework semantic text chunker engineered for Retrieval-Augmented Generation (RAG) pipelines. The efficacy of RAG systems is critically dependent on the quality of text segmentation. Prevailing methodologies often employ naive, fixed-size chunking (e.g., recursive character splitting), which can lead to significant semantic loss by arbitrarily severing conceptual units. Conversely, solutions that do preserve semantic integrity are frequently encumbered by heavy frameworks (e.g., LangChain, LlamaIndex) or reliant on costly, high-latency cloud APIs.

AeroChunk addresses this dichotomy by providing a high-fidelity semantic chunking mechanism that operates entirely on local machine resources. It leverages the computational efficiency of `sentence-transformers` and `numpy` to eliminate external dependencies, API costs, and framework bloat.

## 2. Key Features

- **High-Fidelity Semantic Chunking:** Preserves conceptual integrity by segmenting text based on semantic relatedness.
- **Zero-Framework & Local Execution:** Operates without requiring frameworks like LangChain or LlamaIndex and runs entirely on local resources, ensuring low latency and zero API costs.
- **Low Memory Footprint:** Engineered for efficiency, consuming minimal RAM even with large documents.
- **Visual HTML Debugger:** An industry-first tool that generates an HTML report for a transparent, interpretable view of the semantic boundaries identified during the chunking process.

## 3. Empirical Benchmarks

### Standard Document Analysis
AeroChunk was first benchmarked against two standard LangChain text splitters on a repetitive text block to evaluate baseline performance and chunking quality.

| Method                  | Execution Time (s) | Chunks Generated | Outcome         |
| ----------------------- | ------------------ | ---------------- | --------------- |
| **AeroChunk**           | **7.51**           | **72**           | **Optimal**     |
| LangChain (Recursive)   | 0.00               | 51               | Semantic Loss   |
| LangChain (Semantic)    | 7.53               | 1                | Failed to Split |

### High-Load Stress Test
To evaluate performance under load, AeroChunk was benchmarked against LangChain's `SemanticChunker` on a 15,000-word document. The results demonstrate that AeroChunk not only maintains superior segmentation accuracy but also exhibits significantly greater computational efficiency, consuming approximately 69% less peak memory and executing over 40% faster.

| Method                  | Peak RAM (MB) | Execution Time (s) | Chunks Generated | Outcome         |
| ----------------------- | ------------- | ------------------ | ---------------- | --------------- |
| **AeroChunk (Batch 32)**| **6.62**      | **11.18**          | **1500**         | **Optimal**     |
| LangChain (Semantic)    | 21.56         | 18.73              | 4                | Failed to Split |

## 4. Architectural Methodology

The AeroChunk pipeline is a four-stage process designed for computational efficiency and semantic accuracy.

1.  **Regex Tokenization:** The input text is first segmented into individual sentences using a regular expression that identifies sentence-terminating punctuation (`.`, `!`, `?`).
2.  **Vectorization:** Each sentence is then converted into a 384-dimensional vector embedding using a local `sentence-transformers` model (`all-MiniLM-L6-v2` by default).
3.  **Cosine Similarity Analysis:** The system computes the pairwise cosine similarity between the vector embeddings of adjacent sentences. This score quantifies the semantic relatedness between them.
4.  **Threshold Bounding:** A semantic boundary is declared wherever the cosine similarity between two consecutive sentences drops below a predefined threshold (default: `0.5`). Sentences are aggregated into a chunk until such a drop is detected, at which point a new chunk begins.

This methodology ensures that chunks are formed from contiguous, semantically related sentences, thereby preserving the conceptual integrity of the source text.

## 5. Installation

Install the package using pip:
```bash
pip install aerochunk
```

## 6. Usage

### Basic Chunking

Instantiate the `AeroChunker` and pass text to the `chunk_text` method.

```python
from aerochunk import AeroChunker

# Your text document
text = """Machine learning (ML) is a field of study in artificial intelligence concerned with the development and study of statistical algorithms. These algorithms can learn from data and generalize to unseen data. Recently, artificial neural networks have been able to surpass many previous approaches in performance."""

# Initialize the chunker
chunker = AeroChunker()

# Process the text
chunks = chunker.chunk_text(text)

# Print the resulting chunks
for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1}: {chunk}\n")
```

### Visual Debugging

After chunking, call the `export_debug_html()` method to generate a visual report of the chunking decisions. This is useful for fine-tuning the similarity threshold.

```python
# (Continuing from the previous example)

# Generate the HTML report
# This file will show where semantic boundaries were drawn and why.
debug_file = chunker.export_debug_html(output_file="aero_debug_report.html")

print(f"Visual debug report saved to: {debug_file}")
```

## 7. License

This project is licensed under the **MIT License**.
