Metadata-Version: 2.1
Name: kvcrush
Version: 0.1.0
Summary: Crush KV cache 6x. 1-4 bits per value, near-zero loss. Based on Google TurboQuant.
Home-page: UNKNOWN
License: UNKNOWN
Platform: UNKNOWN
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: numpy
Requires-Dist: numpy ; extra == 'numpy'

# TurboQuant

Extreme compression for numerical data. 10-16x smaller. Inspired by [Google's TurboQuant research](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/).

## Install

```bash
pip install turboquant  # coming soon — for now, copy turboquant.py
```

## Usage

```python
from turboquant import TurboQuant

# Your data — prices, sensors, metrics, anything numerical
data = [99.5, 100.2, 100.8, 101.1, 100.7, ...]

# Compress
tq = TurboQuant(bits=4)           # 4-bit = ~10x compression
compressed = tq.compress(data)     # returns bytes

# Decompress
restored = tq.decompress(compressed)

# Check
print(f"Size: {len(data)*8:,}B -> {len(compressed):,}B ({len(data)*8/len(compressed):.0f}x)")
print(f"Error: {tq.relative_error(data, restored):.2f}%")
```

## Compression vs Accuracy

| Bits | Compression | Best For |
|------|-------------|----------|
| 4-bit | ~10x | Time-series storage, dashboards, approximate queries |
| 3-bit | ~13x | Pattern search, similarity matching, trend analysis |
| 2-bit | ~16x | Rough storage, directional analysis |
| 1-bit | ~32x | Sign-only — up/down classification |

## How it works

1. **Delta encoding** — stores differences between consecutive values (makes data near-zero-mean)
2. **Randomized Hadamard rotation** — spreads variance evenly across dimensions
3. **Uniform quantization** — maps rotated values to N-bit integers
4. **Bit packing** — stores efficiently

The rotation is the key: most data has uneven variance across dimensions. Rotation makes it uniform, so simple quantization works without per-block scaling overhead.

## When to use it

**Good for:**
- Storing months/years of time-series in memory instead of days
- Compressing database columns (metrics, logs, sensor readings)
- Reducing Redis/cache memory usage
- Shipping less data over the network (IoT, microservices)
- Approximate nearest-neighbor search on compressed vectors

**Not for:**
- Financial trading (use exact data for execution)
- Scientific computation requiring full precision
- Data where every decimal matters

## Honest benchmarks

On correlated data (prices, sensors):
- 4-bit: 10.6x compression, ~3% MAPE
- 3-bit: 12.7x compression, ~2% MAPE

On uncorrelated data (random):
- 4-bit: 10.6x compression, ~6% MAPE
- Use `compress(data, use_delta=False)` for uncorrelated data

## API

```python
# Full control
tq = TurboQuant(bits=4, block_size=32, seed=42)
compressed = tq.compress(data, use_delta=True)
restored = tq.decompress(compressed)
ratio = tq.ratio(data, compressed)
error = tq.error(data, restored)       # Mean absolute error
pct_error = tq.relative_error(data, restored)  # MAPE %

# One-liners
from turboquant import compress, decompress
compressed = compress(data, bits=4)
restored = decompress(compressed)

# Files
from turboquant import compress_file, decompress_file
compress_file("data.csv", "data.tq", bits=4)
decompress_file("data.tq", "data_restored.json")
```

## License

MIT


