Metadata-Version: 2.4
Name: tiny-turboquant
Version: 0.5.0
Summary: Low-bit vector and KV-cache compression research toolkit for PyTorch
Author: Pradeep Boopathy
License-Expression: MIT
Project-URL: Homepage, https://github.com/pradeepboopathy/tiny-turboquant
Project-URL: Repository, https://github.com/pradeepboopathy/tiny-turboquant
Project-URL: Issues, https://github.com/pradeepboopathy/tiny-turboquant/issues
Keywords: quantization,kv-cache,llm,compression,vector-search,pytorch,rag,transformers
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.1
Requires-Dist: numpy>=1.24
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Provides-Extra: demos
Requires-Dist: matplotlib>=3.7; extra == "demos"
Requires-Dist: faiss-cpu>=1.7.4; extra == "demos"
Requires-Dist: sentence-transformers>=2.6; extra == "demos"
Provides-Extra: llm
Requires-Dist: transformers>=4.40; extra == "llm"
Requires-Dist: accelerate>=0.30; extra == "llm"
Provides-Extra: all
Requires-Dist: matplotlib>=3.7; extra == "all"
Requires-Dist: faiss-cpu>=1.7.4; extra == "all"
Requires-Dist: sentence-transformers>=2.6; extra == "all"
Requires-Dist: transformers>=4.40; extra == "all"
Requires-Dist: accelerate>=0.30; extra == "all"
Dynamic: license-file

# Tiny TurboQuant

## v0.5.0: real LLM KV-cache benchmarking

This release adds repeatable KV-cache benchmark utilities for real Hugging Face causal language models.

New capabilities:

- `tiny-tq kv-bench` for DynamicCache vs `HybridTurboQuantKVCache` comparison
- prompt modes: `short`, `medium`, `long`, and `stress`
- safe / balanced / aggressive / quality-headwise KV-cache presets
- automatic outlier count selection with `"auto"`
- KV-cache memory estimator via `tiny-tq kv-estimate`
- JSON and Markdown benchmark reports
- baseline vs compressed output comparison
- generation-drift diagnostics such as first divergence and optional logit KL

This release continues to focus on memory compression and quality measurement. It does **not** claim production inference acceleration.

Example:

```bash
tiny-tq kv-estimate --layers 24 --kv-heads 8 --head-dim 128 --seq-len 4096 --batch-size 4

tiny-tq kv-bench \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --preset safe \
  --prompt-mode short \
  --max-new-tokens 8 \
  --report-json kv_report.json \
  --report-md kv_report.md
```


Tiny TurboQuant is a lightweight PyTorch research toolkit for low-bit vector compression, compressed RAG retrieval, and KV-cache compression experiments.

Version **0.4.0** focuses on the RAG/vector-index roadmap:

- `RAGCompressedIndex` for compressed retrieval experiments
- document chunking helpers
- JSONL / folder text ingestion utilities
- compressed vector index save/load
- RAG index save/load
- retrieval metrics: recall@k, precision@k, MRR, nDCG, overlap, label-match ratio
- CLI entry point: `tiny-tq rag-bench`
- existing hybrid KV-cache and paged-attention research utilities from v0.3.x

## Important limitation

This package demonstrates packed memory compression and memory-quality benchmarking. It is **not** a production compressed-attention engine. Hugging Face generation still receives dense K/V tensors. The paged attention utility dequantizes page-by-page and avoids one full dense cache tensor, but it is not a fused CUDA/Triton kernel. Real latency gains require fused kernels or serving-engine integration.

Do not use this package to claim training acceleration, fine-tuning memory reduction, production legal/medical QA readiness, drop-in vLLM replacement, exact nearest-neighbor search, or faster LLM inference.

## Install

```bash
pip install tiny-turboquant
```

Optional demo dependencies:

```bash
pip install "tiny-turboquant[demos]"
```

## Compressed RAG index from precomputed embeddings

```python
import torch
from tiny_turboquant import RAGCompressedIndex

texts = [
    "Embedding compression can reduce vector-store memory in RAG systems.",
    "KV cache stores Key and Value tensors during LLM generation.",
]
embeddings = torch.randn(len(texts), 384)

index = RAGCompressedIndex.from_embeddings(
    texts,
    embeddings,
    bits=4,
    store_original_for_rerank=True,
)

results = index.search(embeddings[0], top_k=1, rerank_top_k=2)
print(index.memory_report())
print(results[0].text)
```

## Compressed RAG index from documents

Requires `sentence-transformers`:

```python
from tiny_turboquant import RAGCompressedIndex

docs = [
    "RAG systems retrieve relevant document chunks and pass them to an LLM.",
    "Compressed vector indexes reduce embedding memory usage.",
]

index = RAGCompressedIndex.from_documents(
    docs,
    embedding_model="sentence-transformers/all-MiniLM-L6-v2",
    bits=4,
    chunk_size=500,
    overlap=50,
)

results = index.search("How do we reduce vector-store memory?", top_k=3)
```

## Save and load

```python
index.save("rag_index.ttq")
loaded = RAGCompressedIndex.load("rag_index.ttq")
```

## Retrieval metrics

```python
from tiny_turboquant import recall_at_k, mrr_at_k, ndcg_at_k

retrieved = ["doc-1", "doc-2", "doc-3"]
relevant = {"doc-2", "doc-5"}

print(recall_at_k(retrieved, relevant, k=3))
print(mrr_at_k(retrieved, relevant, k=3))
print(ndcg_at_k(retrieved, relevant, k=3))
```

## CLI

```bash
tiny-tq version
```

Synthetic RAG benchmark:

```bash
tiny-tq rag-bench --synthetic --bits 4 --top-k 10 --rerank-top-k 50
```

JSONL benchmark:

```bash
tiny-tq rag-bench \
  --input-jsonl docs.jsonl \
  --text-field text \
  --query "How can we reduce vector-store memory?" \
  --bits 4 \
  --top-k 10 \
  --rerank-top-k 50
```

## Hybrid KV-cache usage

```python
from tiny_turboquant import HybridTurboQuantKVCache

cache = HybridTurboQuantKVCache(
    key_bits=6,
    value_bits=4,
    key_outlier_bits=8,
    value_outlier_bits=8,
    n_key_outliers=32,
    n_value_outliers=16,
    key_recent_window=128,
    value_recent_window=64,
    per_layer_calibration=True,
    per_head_calibration=True,
)
```

## Compressed vector index usage

```python
import torch
from tiny_turboquant import CompressedVectorIndex

emb = torch.randn(10_000, 384)
index = CompressedVectorIndex(bits=4, store_original_for_rerank=True).add(emb)
results = index.search(emb[0], top_k=5, rerank_top_k=100)

print(index.compression_ratio())
print(results[0])
```

## Project position

Current focus:

- memory compression
- retrieval quality measurement
- RAG/vector-index experiments
- KV-cache compression research

Future direction:

- real workload RAG benchmarks
- FAISS/vector database integration
- long-context real-model KV-cache benchmarks
- fused dequant + attention kernels
- serving-engine integration experiments
