Metadata-Version: 2.1
Name: hypertensor
Version: 1.0.0
Summary: Per-weight precision + CUDA kernels for PyTorch
Author: CoolPuzzler
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch
Provides-Extra: dev
Requires-Dist: ninja ; extra == 'dev'

# HyperTensor

**Per-weight mixed-precision for PyTorch. Faster inference, lower VRAM, same accuracy.**

HyperTensor is a precision-adaptive quantization framework that assigns the *lowest safe numerical precision to each individual weight* in a neural network.

---

## Why HyperTensor?

Modern neural networks waste compute.

Most models run entirely in FP32 (or globally in FP16/BF16), even though:
- many weights work fine in INT8 or INT4
- some benefit from FP16/BF16
- only a small fraction truly require FP32

HyperTensor solves this by **measuring error per weight** and selecting the minimum precision that preserves stability.

---

## What it does

HyperTensor:

- analyzes each weight independently
- assigns the lowest safe precision
- builds mixed-precision execution buffers
- executes using **INT4 / INT8 / FP16 / BF16 / FP32 — within the same model**

Result:
- faster inference
- significantly lower VRAM usage
- predictable, stable accuracy

---

## Key Features

- Per-weight and per-block precision selection
- INT4 / INT8 / FP16 / BF16 / FP32 execution
- Flexible precision planner:
  LOWEST, HIGHEST, ALLOW, CLOSEST, AVERAGE, BEST, PER_WEIGHT, PER_BLOCK
- Drop-in HyperLinear layer
- HyperAttention module for LLaMA-style architectures
- Training + inference support
- Inference-only mode (removes FP32 master weights)
- Custom CUDA kernels:
  - INT4 / INT8 GEMM
  - FP16 / BF16 backward
- Serialization of quantized models and precision plans

---

## Design Philosophy

HyperTensor is built to be:

- **Accuracy-first** — every decision is based on measured error
- **Granular** — operates at the weight level, not just per tensor
- **Composable** — use the planner, modules, or full stack independently
- **Training-aware** — STE, backward kernels, and caching are first-class
- **Deployment-focused** — inference-only mode minimizes VRAM
- **Format-minimal** — INT4, INT8, FP16, BF16, FP32 only

The system is designed to be extended with new planner modes, kernels, and modules.

---

## Benchmarks

All benchmarks were run on an NVIDIA RTX 4080 Laptop GPU (12GB VRAM), reflecting realistic consumer hardware performance.

The main benchmark suite (`benchmark.py`) evaluates training and inference on a 4096-dim MLP (batch size 16).

---

## Training Benchmarks

### Training Speed
<img src="benchmarks/training_speed.png" width="420">

Training speed remains close to FP32 (~17–18 ms/step). Most planner modes introduce minimal overhead.

### Training Memory Usage
<img src="benchmarks/training_memory.png" width="420">

Memory usage increases due to quantization buffers and error tracking. PER_WEIGHT and PER_BLOCK are the most expensive; FP32 remains lowest.

### Training Drift vs FP32
<img src="benchmarks/drift_training.png" width="420">

All modes maintain stable accuracy (~0.31–0.33 MAE), including aggressive configurations.

---

## Inference Benchmarks

### Inference Speed
<img src="benchmarks/inference_speed.png" width="420">

Inference improves across most modes. AVERAGE is the fastest, but all benefit from reduced precision.

### Inference Memory Usage
<img src="benchmarks/inference_memory.png" width="420">

Memory drops significantly:
- FP32: >500 MB
- HyperTensor: ~210 MB typical
- PER_BLOCK: ~147 MB (lowest)

### Inference Drift vs FP32
<img src="benchmarks/drift_inference.png" width="420">

Accuracy remains stable (~0.32 MAE) even under aggressive quantization.

---

## TinyLLaMA Benchmark

An end-to-end LLM benchmark is included in `TinyLLaMA_benchmark.py`.

Results:

=== HyperTensor LLM Test ===

[1] Loading FP32 model...
Running FP32 benchmark...
  FP32 inference: 108.732 ms

[2] Unloading FP32 model...
[3] Converting to HyperTensor...
Scanning weights (CPU)...
Applying mixed precision (mode = best)...
Moving to GPU...
Running HyperTensor benchmark...
  HyperTensor inference: 49.795 ms

=== Summary ===
FP32:        108.732 ms
HyperTensor: 49.795 ms
Speedup:     2.18x

HyperTensor reduces TinyLLaMA inference latency by over 2× while maintaining stable accuracy.

---

## Combined Results

<img src="benchmarks/benchmark_results.png" width="600">

Summary:

- Training speed ≈ FP32
- Training memory increases (expected)
- Inference speed improves
- Inference memory drops significantly
- Accuracy remains stable across all modes

HyperTensor delivers consistent accuracy, strong inference performance, and substantial VRAM savings on a 12GB GPU.

---

## Documentation

See `DOCUMENTATION.md` for:
- usage examples
- architecture details
- advanced configuration

---

## Installation
```bash
git clone https://github.com/CoolPuzzler/HyperTensor.git HyperTensor
cd HyperTensor
pip install -e .
```

---

## Requirements

- Python ≥ 3.12
- PyTorch with CUDA
- NVIDIA GPU (sm_89 target by default)
- Optional: ninja for faster builds

CUDA extensions are compiled on first import via torch.utils.cpp_extension.load.

---

## License

This project is licensed under the **MIT** License.
See the **LICENSE** file for full details.
