Metadata-Version: 2.4
Name: fastllm-turbo
Version: 2.0.0
Summary: ONA-guided mixed-precision: faster LLM with same output
Home-page: https://fast-llm.dev
Author: Kasish
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: torch>=2.0.0
Requires-Dist: transformers>=4.40.0
Dynamic: author
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# Fast LLM

Run any AI model **faster** on CPU — same output, zero training.

```
pip install fastllm-turbo
fast-llm calibrate Qwen/Qwen2.5-0.5B
```

## How it works

Most of a neural network's weights don't need full precision. We use **ONA's reconstruction loss** to identify which layers are sensitive to precision loss and which are robust. Sensitive layers stay FP32. Robust layers use INT8 dynamic quantization.

The result: mixed-precision model that outputs **the same tokens** as the original, but runs faster because ~60% of computation uses 4× smaller weights.

| Method | Speed | Output matches original? |
|--------|-------|--------------------------|
| INT8 all layers | ~2× | No (quality degrades on small models) |
| ONA mixed-precision | ~1.6× | **Yes** |
| FP32 (original) | 1× | — |

## Install

**pip**
```bash
pip install fastllm-turbo
```

**Linux / macOS**
```bash
curl -sSf https://fast-llm.dev/install.sh | sh
```

**Windows (PowerShell)**
```powershell
powershell -c "iex ((New-Object Net.WebClient).DownloadString('https://fast-llm.dev/install.ps1'))"
```

## Usage

```bash
# Step 1: Analyze which layers can be INT8
fast-llm calibrate Qwen/Qwen2.5-0.5B

# Step 2: Save optimized model
fast-llm save Qwen/Qwen2.5-0.5B

# Step 3: Run
fast-llm run Qwen/Qwen2.5-0.5B --benchmark

# Interactive chat
fast-llm chat Qwen/Qwen2.5-0.5B

# Desktop GUI
fast-llm gui
```

Works with any HuggingFace causal LM: Qwen, Llama 3, Phi-3, Mistral, Gemma, DeepSeek.

## How ONA fits in

ONA (Omni Neural Architecture) is a neural network built from scratch in NumPy with per-neuron attention and forward-pass learning. Its reconstruction loss — originally designed for self-supervised learning without backprop — doubles as a **precision saliency metric**. We reused this metric to solve a completely different problem: deciding which layers of a transformer model can be safely quantized.

```
ONA reconstruction loss  →  layer sensitivity analysis  →  mixed-precision model
```

## Requirements

- Python 3.8+
- PyTorch 2.0+ (CPU)
- 1–4 GB RAM per billion parameters

## License

MIT
