Metadata-Version: 2.4
Name: token-reducer
Version: 0.2.0
Summary: Intelligent token reduction library for LLM applications with context-aware compression
Author-email: Usama Tufail <usamabhutta1250@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/UsamaTufail31/token-reducer
Project-URL: Documentation, https://github.com/UsamaTufail31/token-reducer#readme
Project-URL: Repository, https://github.com/UsamaTufail31/token-reducer
Project-URL: Issues, https://github.com/UsamaTufail31/token-reducer/issues
Keywords: llm,tokens,compression,nlp,optimization,rag,context-reduction
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tiktoken>=0.5.0
Requires-Dist: regex>=2023.0.0
Provides-Extra: transformers
Requires-Dist: transformers>=4.30.0; extra == "transformers"
Requires-Dist: torch>=2.0.0; extra == "transformers"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.18.0; extra == "anthropic"
Provides-Extra: nlp
Requires-Dist: spacy>=3.5.0; extra == "nlp"
Provides-Extra: similarity
Requires-Dist: sentence-transformers>=2.2.0; extra == "similarity"
Requires-Dist: scikit-learn>=1.3.0; extra == "similarity"
Provides-Extra: code
Requires-Dist: libcst>=1.0.0; extra == "code"
Provides-Extra: all
Requires-Dist: token-reducer[anthropic,code,nlp,similarity,transformers]; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.4.0; extra == "dev"
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: black>=23.7.0; extra == "dev"
Requires-Dist: ruff>=0.0.285; extra == "dev"
Requires-Dist: mypy>=1.5.0; extra == "dev"
Requires-Dist: isort>=5.12.0; extra == "dev"
Requires-Dist: pre-commit>=3.3.0; extra == "dev"
Provides-Extra: docs
Requires-Dist: mkdocs>=1.5.0; extra == "docs"
Requires-Dist: mkdocs-material>=9.1.0; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.22.0; extra == "docs"
Dynamic: license-file

# Token Reducer

**Intelligent token reduction library for LLM applications with context-aware compression**

[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

## Overview

Token Reducer is a Python library designed to reduce token counts in text and code inputs for Large Language Model (LLM) applications while preserving semantic meaning, logical structure, and task-relevant information. Achieve **50-70% token reduction** without distorting facts or breaking code logic.

### Key Features

- 🎯 **Context-Aware Compression**: Task-specific strategies for summarization, RAG, extraction, reasoning, and more
- 📊 **Multi-Level Compression**: Choose between light (5-15%), moderate (20-40%), or aggressive (50-70%) reduction
- 🔄 **Multi-Pass Pipeline**: Specialized passes for optimal compression (normalize → prune → compress → summarize → repack)
- 💻 **Text & Code Support**: Domain-specific compression for both natural language and source code
- 🔌 **Tokenizer Agnostic**: Works with OpenAI, Anthropic, HuggingFace, and custom tokenizers
- 🛡️ **Fail-Safe Mode**: Automatic quality validation with semantic similarity checking
- ⚡ **High Performance**: <100ms per 1000 tokens for text, <200ms for code
- 📴 **Offline Operation**: No cloud dependencies, works standalone

## Installation

### Basic Installation

```bash
pip install token-reducer
```

### With Optional Dependencies

```bash
# For HuggingFace tokenizers
pip install token-reducer[transformers]

# For Anthropic tokenizers
pip install token-reducer[anthropic]

# For NLP features (entity extraction, sentence segmentation)
pip install token-reducer[nlp]

# For semantic similarity checking
pip install token-reducer[similarity]

# For code compression
pip install token-reducer[code]

# Install everything
pip install token-reducer[all]
```

## Quick Start

### Text Compression

```python
from token_reducer import compress_text, TaskContext, CompressionLevel

# Compress text for summarization task
result = compress_text(
    text="Your long article text here...",
    task=TaskContext.SUMMARIZATION,
    level=CompressionLevel.MODERATE,
    tokenizer="gpt-4"
)

print(f"Original: {result.original_tokens} tokens")
print(f"Compressed: {result.compressed_tokens} tokens")
print(f"Reduction: {result.reduction_percentage}%")
print(f"\nCompressed text:\n{result.compressed_text}")
```

### Code Compression

```python
from token_reducer import compress_code, TaskContext, CompressionLevel

# Compress code for LLM context
result = compress_code(
    code="""
    def calculate_total_price(items, tax_rate=0.1):
        # Calculate subtotal
        subtotal = sum(item['price'] for item in items)
        # Apply tax
        tax = subtotal * tax_rate
        # Return total
        return subtotal + tax
    """,
    task=TaskContext.CODE_COMPLETION,
    level=CompressionLevel.AGGRESSIVE,
    language="python"
)

print(f"Compressed code:\n{result.compressed_code}")
```

### Advanced Configuration

```python
from token_reducer import CompressionConfig, compress_text

config = CompressionConfig(
    task=TaskContext.RAG,
    level=CompressionLevel.MODERATE,
    tokenizer="claude-3",
    preserve_entities=True,
    preserve_numbers=True,
    quality_threshold=0.90,
    enable_fail_safe=True
)

result = compress_text(text, config=config)
```

## Compression Strategies

### Task Types

Token Reducer adapts compression strategies based on your use case:

- **SUMMARIZATION**: Preserves causal links and chronological order
- **RAG**: Optimizes for retrieval context (entities, facts, key phrases)
- **EXTRACTION**: Keeps only fields relevant to extraction target
- **REASONING**: Preserves premises, key details, and logical connections
- **TRANSLATION**: Bypasses compression entirely
- **CODE_COMPLETION**: Preserves function signatures and interfaces
- **DEBUGGING**: Maintains variable names and error-relevant context
- **QUESTION_ANSWERING**: Preserves facts and entities for potential questions

### Compression Levels

| Level | Token Reduction | Semantic Similarity | Use Case |
|-------|----------------|---------------------|----------|
| **Light** | 5-15% | >98% | Maximum safety, minimal loss |
| **Moderate** | 20-40% | >90% | Balanced compression and quality |
| **Aggressive** | 50-70% | >80% | Maximum savings, acceptable loss |

## How It Works

### Text Compression Pipeline

1. **Normalize**: Fix spacing, remove HTML, standardize quotes
2. **Prune**: Remove duplicates, redundancy, verbose explanations
3. **Compress**: Extract entities/facts, compact phrasing, reduce adjectives
4. **Summarize**: Apply task-specific tightening
5. **Repack**: Shorten sentences, optimize structure

### Code Compression Pipeline

1. **Remove Noise**: Strip comments, blank lines, logging, debug prints
2. **Rename Identifiers**: Shorten variable/function/class names
3. **Remove Unused**: Eliminate dead code, unused imports/functions
4. **Optimize Expressions**: Simplify boolean/arithmetic expressions
5. **Summarize Functions**: Replace bodies with summaries (aggressive mode)
6. **Reformat**: Minimize whitespace, compact structure

## Performance

- **Text**: <100ms per 1000 tokens (excluding quality checks)
- **Code**: <200ms per 1000 tokens (excluding quality checks)
- **Quality Check**: <50ms for semantic similarity validation

## Use Cases

### Reduce LLM API Costs

```python
# Before: 10,000 tokens × $0.03/1K = $0.30 per request
# After (60% reduction): 4,000 tokens × $0.03/1K = $0.12 per request
# Savings: 60% cost reduction
```

### Fit More Context in Token Limits

```python
# Compress multiple documents to fit in context window
from token_reducer import batch_compress_text

results = batch_compress_text(
    texts=[doc1, doc2, doc3, doc4, doc5],
    task=TaskContext.RAG,
    level=CompressionLevel.MODERATE,
    parallel=True
)

# Combine compressed documents within token limit
combined = "\n\n".join(r.compressed_text for r in results)
```

### RAG Pipeline Optimization

```python
# Compress retrieved documents before sending to LLM
retrieved_docs = vector_store.similarity_search(query, k=10)

compressed_docs = [
    compress_text(
        doc.page_content,
        task=TaskContext.RAG,
        level=CompressionLevel.MODERATE
    )
    for doc in retrieved_docs
]

# Use compressed docs in prompt
context = "\n\n".join(d.compressed_text for d in compressed_docs)
```

## Documentation

- **Repository**: [https://github.com/UsamaTufail31/token-reducer](https://github.com/UsamaTufail31/token-reducer)
- **Issues**: [https://github.com/UsamaTufail31/token-reducer/issues](https://github.com/UsamaTufail31/token-reducer/issues)
- **Examples**: [https://github.com/UsamaTufail31/token-reducer/tree/main/examples](https://github.com/UsamaTufail31/token-reducer/tree/main/examples)

## Development

### Setup Development Environment

```bash
# Clone repository
git clone https://github.com/UsamaTufail31/token-reducer.git
cd token-reducer

# Install with development dependencies
pip install -e ".[dev,all]"

# Install pre-commit hooks
pre-commit install
```

### Run Tests

```bash
# Run all tests
pytest

# Run with coverage
pytest --cov=token_reducer --cov-report=html

# Run specific test file
pytest tests/test_text_compression.py
```

### Code Quality

```bash
# Format code
black src/ tests/

# Sort imports
isort src/ tests/

# Lint code
ruff src/ tests/

# Type check
mypy src/
```

## Contributing

Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Citation

If you use Token Reducer in your research, please cite:

```bibtex
@software{token_reducer,
  title = {Token Reducer: Intelligent Token Reduction for LLM Applications},
  author = {Tufail, Usama},
  year = {2024},
  url = {https://github.com/UsamaTufail31/token-reducer}
}
```

## Acknowledgments

- Inspired by research in context compression and semantic similarity
- Built with modern NLP libraries (spaCy, sentence-transformers, tiktoken)
- Designed for the LLM application development community

## Support

- **Issues**: [GitHub Issues](https://github.com/UsamaTufail31/token-reducer/issues)
- **Discussions**: [GitHub Discussions](https://github.com/UsamaTufail31/token-reducer/discussions)

---

**Created by Usama Tufail** | [GitHub](https://github.com/UsamaTufail31)
