Metadata-Version: 2.4
Name: tiny-turboquant
Version: 0.12.0
Summary: Pip-installable compressed KV serving backend and diagnostics for PyTorch
Author: Pradeep Boopathy
License-Expression: MIT
Project-URL: Homepage, https://github.com/pradeepboopathy/tiny-turboquant
Project-URL: Repository, https://github.com/pradeepboopathy/tiny-turboquant
Project-URL: Issues, https://github.com/pradeepboopathy/tiny-turboquant/issues
Keywords: quantization,kv-cache,llm,compression,vector-search,pytorch,rag,transformers
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.1
Requires-Dist: numpy>=1.24
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Provides-Extra: demos
Requires-Dist: matplotlib>=3.7; extra == "demos"
Requires-Dist: faiss-cpu>=1.7.4; extra == "demos"
Requires-Dist: sentence-transformers>=2.6; extra == "demos"
Provides-Extra: hf
Requires-Dist: transformers>=4.40; extra == "hf"
Requires-Dist: accelerate>=0.30; extra == "hf"
Requires-Dist: safetensors>=0.4; extra == "hf"
Provides-Extra: server
Requires-Dist: transformers>=4.40; extra == "server"
Requires-Dist: accelerate>=0.30; extra == "server"
Requires-Dist: safetensors>=0.4; extra == "server"
Provides-Extra: llm
Requires-Dist: transformers>=4.40; extra == "llm"
Requires-Dist: accelerate>=0.30; extra == "llm"
Requires-Dist: safetensors>=0.4; extra == "llm"
Provides-Extra: all
Requires-Dist: matplotlib>=3.7; extra == "all"
Requires-Dist: faiss-cpu>=1.7.4; extra == "all"
Requires-Dist: sentence-transformers>=2.6; extra == "all"
Requires-Dist: transformers>=4.40; extra == "all"
Requires-Dist: accelerate>=0.30; extra == "all"
Requires-Dist: safetensors>=0.4; extra == "all"
Dynamic: license-file

# tiny-turboquant

`tiny-turboquant` is a pip-installable compressed KV-cache serving backend foundation for Hugging Face decoder models.

The package now has two layers:

1. **Runtime path**: run generation while storing the KV cache compressed between decode steps.
2. **Diagnostic path**: scan real model KV tensors and validate which K/V policies are safe.

The main direction is no longer “add more benchmarks.” The main direction is:

```text
serve first
validate second
scan third
experimental last
```

## Current status

`v0.12.0` introduces the first compressed-KV backend foundation:

```text
Hugging Face model -> generate token-by-token -> store KV compressed between steps -> dequantize before stock HF attention
```

This is a real compressed-storage decode loop. It is **not yet** a fused compressed-attention accelerator.

## What this package does now

- Runs a Hugging Face causal LM through a tiny compressed-KV generation engine.
- Stores `past_key_values` compressed between decode steps.
- Supports conservative KV policies such as `q8k-q8v` and `q8k-int4v`.
- Provides an OpenAI-compatible local HTTP server.
- Provides baseline-vs-compressed generation validation.
- Keeps the previous real-KV scanner for policy discovery.

## What this package does not claim yet

- It does not claim production vLLM speedup.
- It does not claim llama.cpp parity.
- It does not claim fused compressed attention.
- It does not claim TurboQuant+ kernel parity.
- It does not claim that low-bit V is safe for every model.

The current backend stores KV compressed, then dequantizes before calling the stock Hugging Face attention path. Fused dequantization plus attention is a later milestone.

## Install

Base package:

```bash
pip install tiny-turboquant
```

For Hugging Face serving:

```bash
pip install "tiny-turboquant[hf]"
```

For server usage:

```bash
pip install "tiny-turboquant[server]"
```

For development:

```bash
pip install "tiny-turboquant[dev]"
```

## Quick start

### 1. List available KV policies

```bash
tiny-tq bench --list-policies
```

Current policies:

| Policy | Meaning |
|---|---|
| `fp16` | Dense Hugging Face KV cache baseline |
| `q8k-q8v` | Conservative compressed K and V storage |
| `q8k-int4v` | q8 keys, int4 values, boundary-layer protection |
| `q8k-turbo4v` | serving-facing alias for q8 keys + 4-bit value storage |

Start with `q8k-q8v`. Move to `q8k-int4v` only after validation.

### 2. Run one compressed-KV generation benchmark

```bash
tiny-tq bench \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --kv-policy q8k-q8v \
  --prompt "Explain why KV-cache memory grows during long-context decoding." \
  --max-new-tokens 64
```

The output includes:

```text
input_tokens
output_tokens
kv_policy
elapsed_seconds
tokens_per_second
cache_report.compressed_storage_bytes
cache_report.dense_fp16_bytes
cache_report.memory_saved_pct
```

### 3. Compare dense KV against compressed KV

```bash
tiny-tq validate \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --baseline fp16 \
  --candidate q8k-q8v \
  --prompt "Give a short explanation of KV-cache compression." \
  --max-new-tokens 48 \
  --report-json validate_qwen_q8.json
```

The validation report compares:

```text
baseline generated text
candidate generated text
same prefix token count
exact token match
baseline cache report
candidate cache report
```

### 4. Run the local OpenAI-compatible server

```bash
tiny-tq serve \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --kv-policy q8k-q8v \
  --host 127.0.0.1 \
  --port 8000 \
  --max-new-tokens 128
```

Health check:

```bash
curl http://127.0.0.1:8000/health
```

Chat completion:

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "messages": [
      {"role": "user", "content": "Explain KV-cache compression in two sentences."}
    ],
    "max_tokens": 64
  }'
```

Completion:

```bash
curl http://127.0.0.1:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "KV-cache compression helps because",
    "max_tokens": 64
  }'
```

## Python API

```python
from tiny_turboquant.backends.hf import TinyTurboHFEngine

engine = TinyTurboHFEngine(
    "Qwen/Qwen2.5-0.5B-Instruct",
    kv_policy="q8k-q8v",
    device="auto",
    dtype="auto",
).load()

result = engine.generate(
    "Explain KV-cache compression in simple terms.",
    max_new_tokens=64,
)

print(result.generated_text)
print(result.cache_report)
```

## How the v0.12 backend works

The backend runs a controlled Hugging Face decode loop:

```text
1. Run the prompt through the model with use_cache=True.
2. Compress the returned past_key_values into TinyTurboKVCache.
3. Pick the next token.
4. Dequantize the compressed cache back to a dense legacy past_key_values tuple.
5. Run the next token through the model.
6. Compress the updated cache again.
7. Repeat until max_new_tokens or EOS.
```

This means the cache is actually stored compressed between decode steps. The current speed is not the final goal because dense materialization still happens before attention.

## Why the default is conservative

Real model-query validation on Qwen showed that random-query diagnostics were too optimistic. Lower-bit V could look good under random queries and fail under real model queries. The safer initial backend policy is therefore asymmetric and conservative:

```text
start: q8k-q8v
then validate: q8k-int4v
later: fused/sparse/turbo policies
```

The package keeps `real-model-kv-scan` so policy decisions can be measured instead of guessed.

Example scan:

```bash
tiny-tq real-model-kv-scan \
  --preset safe \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --prompt "Long context text..." \
  --max-prompt-tokens 256 \
  --page-size 64 \
  --summary-table \
  --report-json qwen_scan.json \
  --report-md qwen_scan.md
```

## CLI surface

Stable commands:

```text
tiny-tq serve
tiny-tq bench
tiny-tq validate
tiny-tq real-model-kv-scan
tiny-tq kv-estimate
tiny-tq version
```

Older research commands are still present for compatibility, but they should not define the public product direction. A future cleanup release will move old experiments under an experimental namespace.

## Backend roadmap

### v0.12.x

- Stabilize the HF compressed-KV backend.
- Improve generation validation.
- Add per-layer fallback explanations.
- Add q8k-int4v and q8k-turbo4v validation reports.

### v0.13.x

- Add fused dequantization plus attention experiments.
- Add long-context sparse-V prototype.
- Add perplexity and prompt-suite validation.

### v0.14.x

- Explore vLLM adapter.
- Explore llama.cpp/GGUF bridge or export path.
- Keep Python package install simple while making native acceleration optional.

## Development

```bash
git clone https://github.com/pradeepboopathy/tiny-turboquant
cd tiny-turboquant
pip install -e ".[dev,hf]"
pytest
```

Build:

```bash
python -m build
python -m twine check dist/*
```

Upload:

```bash
python -m twine upload dist/*
```

## Design rule

`tiny-turboquant` should be a clean compressed-KV backend, not a pile of experiments.

Experiments are allowed internally. The public path should remain:

```text
serve -> bench -> validate -> scan
```
