Metadata-Version: 2.4
Name: tiny-turboquant
Version: 0.3.0
Summary: Low-bit vector and KV-cache compression research toolkit for PyTorch
Author: Pradeep Boopathy
License-Expression: MIT
Project-URL: Homepage, https://github.com/pradeepboopathy/tiny-turboquant
Project-URL: Repository, https://github.com/pradeepboopathy/tiny-turboquant
Project-URL: Issues, https://github.com/pradeepboopathy/tiny-turboquant/issues
Keywords: quantization,kv-cache,llm,compression,vector-search,pytorch,rag,transformers
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.1
Requires-Dist: numpy>=1.24
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Provides-Extra: demos
Requires-Dist: matplotlib>=3.7; extra == "demos"
Requires-Dist: faiss-cpu>=1.7.4; extra == "demos"
Requires-Dist: sentence-transformers>=2.6; extra == "demos"
Provides-Extra: llm
Requires-Dist: transformers>=4.40; extra == "llm"
Requires-Dist: accelerate>=0.30; extra == "llm"
Provides-Extra: all
Requires-Dist: matplotlib>=3.7; extra == "all"
Requires-Dist: faiss-cpu>=1.7.4; extra == "all"
Requires-Dist: sentence-transformers>=2.6; extra == "all"
Requires-Dist: transformers>=4.40; extra == "all"
Requires-Dist: accelerate>=0.30; extra == "all"
Dynamic: license-file

# Tiny TurboQuant

Tiny TurboQuant is a lightweight PyTorch research toolkit for low-bit vector compression and KV-cache compression experiments.

Version **0.3.0** adds quality-hardening and integration-oriented research features:

- optional **per-head calibration** in `HybridTurboQuantKVCache`
- separate **Key/Value recent windows** for stronger Key preservation
- safer outlier clamping for small head dimensions
- paged/streaming attention utility that avoids one giant dense cache reconstruction in research tests
- `CompressedVectorIndex` API with optional full-precision reranking
- serving-engine integration scaffolding for vLLM/TensorRT-LLM style experiments

## Important limitation

This package demonstrates packed memory compression and memory-quality benchmarking. It is **not** a production compressed-attention engine. Hugging Face generation still receives dense K/V tensors. The paged attention utility dequantizes page-by-page and avoids one full dense cache tensor, but it is not a fused CUDA/Triton kernel. Real latency gains require fused kernels or serving-engine integration.

Do not use this package to claim training acceleration, fine-tuning memory reduction, production legal/medical QA readiness, drop-in vLLM replacement, exact nearest-neighbor search, or faster LLM inference.

## Install

```bash
pip install tiny-turboquant
```

## Hybrid KV-cache usage

```python
from tiny_turboquant import HybridTurboQuantKVCache

cache = HybridTurboQuantKVCache(
    key_bits=6,
    value_bits=4,
    key_outlier_bits=8,
    value_outlier_bits=8,
    n_key_outliers=32,
    n_value_outliers=16,
    key_recent_window=128,
    value_recent_window=64,
    per_layer_calibration=True,
    per_head_calibration=True,
)
```

## Compressed vector index usage

```python
import torch
from tiny_turboquant import CompressedVectorIndex

emb = torch.randn(10_000, 384)
index = CompressedVectorIndex(bits=4, store_original_for_rerank=True).add(emb)
results = index.search(emb[0], top_k=5, rerank_top_k=100)

print(index.compression_ratio())
print(results[0])
```

## Paged attention utility

```python
import torch
from tiny_turboquant import HybridTurboQuantKVCache

cache = HybridTurboQuantKVCache(key_bits=6, value_bits=4, recent_window=16)
k = torch.randn(1, 4, 64, 32)
v = torch.randn(1, 4, 64, 32)
cache.update(k, v, layer_idx=0)

q = torch.randn(1, 4, 1, 32)
out = cache.paged_attention(q, layer_idx=0)
```

## Serving integration scaffolding

```python
from tiny_turboquant import PagedKVCacheSpec, VLLMExperimentAdapter

spec = PagedKVCacheSpec(key_bits=6, value_bits=4, page_size=128)
plan = VLLMExperimentAdapter(spec).integration_plan()
print(plan)
```

## Scope

Use this package for:

- compressed vector-search experiments
- RAG embedding compression experiments
- KV-cache memory/quality tradeoff experiments
- educational or research benchmarking
- testing hybrid recent-token retention strategies
- studying separate K/V compression settings
- testing per-layer/per-head calibration

Current focus:

- memory compression
- quality measurement
- cache-format experiments
- research scaffolding for serving integration

Future direction:

- actual Triton/CUDA fused dequant + attention kernels
- stronger compressed-domain attention estimators
- engine-specific paged-cache integration experiments
