Metadata-Version: 2.1
Name: hypertensor
Version: 1.0.1
Summary: Per-weight precision + CUDA kernels for PyTorch
Author: CoolPuzzler
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch
Provides-Extra: dev
Requires-Dist: ninja; extra == "dev"

# HyperTensor

**Per-weight mixed-precision for PyTorch — faster inference, lower VRAM, near-FP32 accuracy**

HyperTensor enables mixed-precision execution inside a single PyTorch model by assigning numerical precision at the individual weight level (INT4, INT8, FP16, BF16, or FP32) based on measured reconstruction error.

This means a single model can dynamically mix precisions across weights in the same forward pass while remaining fully compatible with standard PyTorch.

In (`TinyLLaMA_benchmark.py`), converting a model from FP32 to HyperTensor achieved ~2× inference speedup with minimal accuracy loss, alongside significant VRAM reduction on consumer GPUs.

Under the hood, a precision planner builds a per-weight format map, which is compiled into optimized execution buffers and executed using custom CUDA kernels and fused mixed-precision operators.

HyperTensor is modular: the planner, runtime, and model wrappers can be used independently or combined for research and deployment of adaptive precision neural networks.

---

## Install (PyPI)

```bash
pip install hypertensor
```

---

## Why HyperTensor?

Modern neural networks waste compute.

Most models run entirely in FP32 (or globally in FP16/BF16), even though:
- many weights are safe in INT8 or INT4
- some benefit from FP16/BF16
- only a small fraction require FP32

HyperTensor addresses this by **measuring reconstruction error per weight** and selecting the lowest precision that preserves stability.

---

## What it does

HyperTensor:

- analyzes each weight independently
- assigns the lowest safe precision
- builds mixed-precision execution buffers
- executes INT4 / INT8 / FP16 / BF16 / FP32 within a single model

Result:
- reduced VRAM usage
- faster inference
- stable accuracy relative to FP32 baselines

---

## Usage Example

```python
import torch
import torch.nn as nn
import hypertensor
from HyperTensor import WeightOption

class BenchmarkMLP(nn.Module):
    def __init__(self, dim=4096, depth=4, refresh_every=8):
        super().__init__()
        layers = []
        for _ in range(depth):
            layers.append(
                hypertensor.HyperLinear(
                    dim,
                    dim,
                    refresh_every=refresh_every,
                    prefer_bf16=True,
                )
            )
            layers.append(nn.ReLU())
        self.net = nn.Sequential(*layers)

    def forward(self, x):
        return self.net(x)
```

---

## Convert model

```python
plan = hypertensor.scan_model(
    model,
    eps_fp16=0.0,
    eps_bf16=0.0,
    prefer_bf16=True,
)

model_ht = hypertensor.wrap_model_with_hypertensor(
    model,
    plan,
    option=WeightOption.BEST,
    inference_only=True,
)
```

---

## Key Features

- Per-weight and per-block precision selection
- INT4 / INT8 / FP16 / BF16 / FP32 execution
- Precision planner modes:
  LOWEST, HIGHEST, ALLOW, CLOSEST, AVERAGE, BEST, PER_WEIGHT, PER_BLOCK
- Drop-in `HyperLinear` replacement for `nn.Linear`
- `HyperAttention` for transformer architectures
- Training + inference support
- Inference-only mode (removes FP32 master weights)
- Custom CUDA kernels for INT4/INT8/FP16/BF16
- Serializable quantized models + precision plans

---

## Design Philosophy

HyperTensor is built to be:

- **Accuracy-first** — decisions based on measured reconstruction error
- **Granular** — operates at weight or block level
- **Composable** — planner, runtime, and modules are independent
- **Training-aware** — supports STE + custom backward kernels
- **Deployment-focused** — inference-only mode minimizes memory
- **Format-minimal** — only INT4, INT8, FP16, BF16, FP32

---

## Benchmarks

All benchmarks were run on an NVIDIA RTX 4080 Laptop GPU (12GB VRAM) using a fixed-seed experimental setup.

The benchmark suite evaluates a 4096-dimensional MLP under both training and inference workloads across all precision planning modes.

---

## TinyLLaMA Benchmark

End-to-end transformer evaluation:

- FP32 baseline measured
- Model converted to HyperTensor
- Inference repeated under identical inputs

Result:
- ~2× inference speedup
- Stable output behavior relative to FP32

---

## Key Results

- Training speed: ~FP32 equivalent
- Training memory: increased (expected due to buffers)
- Inference speed: improved across most modes
- Inference memory: significantly reduced
- Accuracy: stable within evaluated drift bounds

---

## Installation (from source)

```bash
git clone https://github.com/CoolPuzzler/HyperTensor.git
cd HyperTensor
pip install -e .
```

---

## Requirements

- Python ≥ 3.12
- PyTorch with CUDA
- NVIDIA GPU (sm_89 target default)
- Optional: ninja (faster CUDA builds)

CUDA extensions compile automatically on first import.

---

## Citation / Project Status

HyperTensor is an active research system for fine-grained mixed-precision execution in deep neural networks.

---

## License

This project is licensed under the **MIT** License. See the **LICENSE** file for full details.
