Metadata-Version: 2.4
Name: isage-refiner
Version: 0.1.0.2
Summary: Intelligent context compression algorithms for LLM systems
Home-page: https://github.com/intellistream/sageRefiner
Author: SAGE Team
Author-email: IntelliStream Team <shuhao_zhang@hust.edu.cn>
License: Apache-2.0
Project-URL: Homepage, https://github.com/intellistream/sageRefiner
Project-URL: Documentation, https://intellistream.github.io/SAGE/
Project-URL: Repository, https://github.com/intellistream/sageRefiner
Project-URL: Bug Tracker, https://github.com/intellistream/sageRefiner/issues
Keywords: rag,llm,compression,context,nlp,ai,ml
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch<3.0.0,>=2.0.0
Requires-Dist: transformers<4.54.0,>=4.52.0
Requires-Dist: numpy<3.0.0,>=1.24.0
Requires-Dist: pyyaml<7.0,>=6.0
Requires-Dist: json-repair<1.0.0,>=0.30.0
Requires-Dist: tqdm>=4.65.0
Requires-Dist: rich<14.0.0,>=13.0.0
Requires-Dist: isage-common
Requires-Dist: isage-kernel
Requires-Dist: isage-libs
Requires-Dist: isage-middleware
Provides-Extra: vllm
Requires-Dist: vllm<0.13,>=0.9.2; extra == "vllm"
Provides-Extra: reranker
Requires-Dist: FlagEmbedding>=1.0.0; extra == "reranker"
Provides-Extra: benchmark
Requires-Dist: isage-common[embedding]; extra == "benchmark"
Requires-Dist: isage-benchmark[longbench]; extra == "benchmark"
Requires-Dist: scipy>=1.10.0; extra == "benchmark"
Requires-Dist: matplotlib>=3.7.0; extra == "benchmark"
Requires-Dist: pandas>=2.0.0; extra == "benchmark"
Requires-Dist: datasets>=2.14.0; extra == "benchmark"
Requires-Dist: pyarrow>=10.0.0; extra == "benchmark"
Provides-Extra: full
Requires-Dist: vllm<0.13,>=0.9.2; extra == "full"
Requires-Dist: FlagEmbedding>=1.0.0; extra == "full"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: scipy>=1.10.0; extra == "dev"
Requires-Dist: matplotlib>=3.7.0; extra == "dev"
Requires-Dist: pandas>=2.0.0; extra == "dev"
Requires-Dist: datasets>=2.14.0; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest>=7.0.0; extra == "test"
Requires-Dist: pytest-cov>=4.0.0; extra == "test"
Requires-Dist: scipy>=1.10.0; extra == "test"
Requires-Dist: matplotlib>=3.7.0; extra == "test"
Requires-Dist: pandas>=2.0.0; extra == "test"
Requires-Dist: datasets>=2.14.0; extra == "test"
Requires-Dist: isage-common[embedding]; extra == "test"
Requires-Dist: isage-benchmark[longbench]; extra == "test"
Dynamic: author
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# sageRefiner

**Intelligent Context Compression Library for LLM Systems**

sageRefiner provides state-of-the-art context compression algorithms to reduce token usage while
maintaining semantic quality for Large Language Model applications.

## Features

- **8 Compression Algorithms**

  - **LongRefiner**: LLM-based selective compression with importance scoring
  - **REFORM**: Attention-based compression with KV cache optimization
  - **Provence**: Sentence-level pruning using DeBERTa reranker
  - **LLMLingua2**: Fast BERT-based token classification
  - **LongLLMLingua**: Question-aware perplexity-based compression
  - **RECOMP-Abstractive**: T5-based summarization
  - **RECOMP-Extractive**: BERT-based sentence selection
  - **EHPC**: Evaluator Heads based efficient compression

- **High Compression Ratios**: 2-10x compression while preserving key information

- **Flexible Configuration**: YAML/dict-based configuration

- **Production Ready**: Battle-tested in the SAGE framework

## Installation

```bash
# Basic installation
pip install isage-refiner

# With vLLM support (for LongRefiner)
pip install isage-refiner[vllm]

# Development mode
git clone https://github.com/intellistream/sageRefiner.git
cd sageRefiner
pip install -e .
```

## Quick Start

```python
from sage_refiner import LLMLingua2Compressor

# Initialize compressor
compressor = LLMLingua2Compressor()

# Compress context
result = compressor.compress(
    context="Your long document text here...",
    question="What is the main topic?",
    target_token=500,
)

print(f"Compression rate: {result['compression_rate']:.2%}")
print(f"Compressed: {result['compressed_context']}")
```

## Algorithms

| Algorithm         | Model             | Best For                          |
| ----------------- | ----------------- | --------------------------------- |
| **LongRefiner**   | Qwen/Llama + LoRA | High-quality semantic compression |
| **REFORM**        | Any Llama/Qwen    | Fast attention-based selection    |
| **Provence**      | DeBERTa           | Document-level filtering          |
| **LLMLingua2**    | BERT              | Speed-critical applications       |
| **LongLLMLingua** | GPT-2/Llama       | Long document scenarios           |
| **RECOMP-Abst**   | T5                | Summarization-style compression   |
| **RECOMP-Extr**   | BERT              | Sentence extraction               |
| **EHPC**          | Llama-8B          | Evaluator heads selection         |

## Configuration

```python
from sage_refiner import RefinerConfig

config = RefinerConfig(
    algorithm="llmlingua2",
    target_token=500,
    force_tokens=["important", "keyword"],
)
```

## Examples

```bash
# Basic compression
python examples/basic_compression.py

# Compare algorithms
python examples/algorithm_comparison.py
```

## Requirements

- Python 3.11+
- PyTorch 2.0+
- Transformers 4.43+

## Benchmarking

The `benchmarks` module provides a comprehensive evaluation framework for all compression
algorithms:

```bash
# Quick comparison of multiple algorithms
pip install isage-refiner[benchmark]
sage-refiner-bench compare \
    --algorithms baseline,longrefiner,reform,provence \
    --samples 100

# Detailed evaluation with budget sweep
sage-refiner-bench sweep \
    --algorithm longrefiner \
    --budgets 512,1024,2048,4096
```

For detailed benchmarking documentation, see [benchmarks/README.md](benchmarks/README.md) and
[benchmarks/STRUCTURE.md](benchmarks/STRUCTURE.md).

## Citation

```bibtex
@software{sageRefiner2025,
  title = {sageRefiner: Context Compression for LLM},
  author = {SAGE Team},
  year = {2025},
  url = {https://github.com/intellistream/sageRefiner}
}
```

## License

Apache License 2.0

## Links

- **SAGE Framework**: https://github.com/intellistream/SAGE
- **Issues**: https://github.com/intellistream/sageRefiner/issues

## Documentation & Development

- **[Development Guide](docs/README.md)** - Setup, workflow, and guidelines
- **[Pre-commit Hooks](docs/PRE_COMMIT.md)** - Setup and usage guide
- **[CI/CD Pipeline](docs/HOOKS_SETUP.md)** - Automated checks details

For quick setup:

```bash
bash utils/installation/quickstart.sh    # Full installation
bash utils/hooks/setup-hooks.sh          # Setup pre-commit hooks
```
