Metadata-Version: 2.4
Name: otf-llm
Version: 3.2.0
Summary: High-performance hybrid LLM inference engine with Fused OpenAI Triton INT4 GEMM kernels, 98.16% logit parity, Zero-RAM streaming quantizer, and 3-Tier MoE offloading.
Home-page: https://github.com/GT-Labs-AI/otf-llm-engine
Author: GT Labs AI & Gleb Tikhiy
Author-email: Gleb Tikhiy <team.gtlabs@gmail.com>, GT Labs AI <team.gtlabs@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/GT-Labs-AI/otf-llm-engine
Project-URL: Repository, https://github.com/GT-Labs-AI/otf-llm-engine
Keywords: llm,quantization,triton,cuda,vram-optimization,deep-learning,fastapi,gradio,moe
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.1.0
Requires-Dist: triton>=2.1.0; sys_platform != "win32"
Requires-Dist: triton-windows; sys_platform == "win32"
Requires-Dist: transformers>=4.38.0
Requires-Dist: safetensors>=0.4.0
Requires-Dist: fastapi>=0.100.0
Requires-Dist: uvicorn>=0.22.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: gradio>=4.0.0
Requires-Dist: huggingface_hub>=0.20.0
Requires-Dist: accelerate>=0.26.0
Dynamic: author
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# 🚀 On-The-Fly Weight Synthesizer (OTF-LLM Engine)

> **High-performance hybrid LLM inference engine featuring custom Fused OpenAI Triton INT4 GEMM kernels, Outlier-Aware weight quantization, Zero-RAM Incremental Quantizer (<150MB RAM), 98.16% Scientific Logit Parity, 3-Tier Hierarchical MoE Offloader [EXPERIMENTAL], Companion Long-Term Memory, and an Interactive Gradio Web UI.**

[![PyPI](https://img.shields.io/pypi/v/otf-llm?color=blue&logo=pypi&logoColor=white)](https://pypi.org/project/otf-llm/)
[![PyTorch](https://img.shields.io/badge/PyTorch-2.0+-EE4C2C?logo=pytorch&logoColor=white)](https://pytorch.org/)
[![CUDA](https://img.shields.io/badge/CUDA-Supported-76B900?logo=nvidia&logoColor=white)](https://developer.nvidia.com/cuda-zone)
[![Triton](https://img.shields.io/badge/OpenAI_Triton-Fused_Kernels-red?logo=openai&logoColor=white)](https://github.com/openai/triton)
[![Gradio](https://img.shields.io/badge/Gradio-Web_UI-Orange?logo=gradio&logoColor=white)](https://gradio.app/)
[![FastAPI](https://img.shields.io/badge/FastAPI-REST_API-009688?logo=fastapi&logoColor=white)](https://fastapi.tiangolo.com/)
[![License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)

---

## 🔬 Engineering & Research: GT Labs AI

Developed and maintained by **GT Labs AI**.

* 🚀 **GT Labs AI Mission:** Ultra-fast MVP engineering, custom AI integration, and deep neural network optimization research.
* 👨‍💻 **Author & Lead AI Engineer:** **Gleb Tikhiy** ([@GlebTikhiy](https://github.com/GlebTikhiy))
* 📧 **Contact & Inquiries:** team.gtlabs@gmail.com
* 🌐 **Organization:** [GT Labs AI on GitHub](https://github.com/GT-Labs-AI)

---

## 🎯 Project Goal & Key Breakthroughs

Overcoming memory-bound bottlenecks and hardware VRAM constraints when executing Large Language Models (LLMs) on consumer GPUs (RTX 3060 / 4060 / 5060 Ti).

Instead of transferring heavy FP16 weights from VRAM, **OTF-LLM Engine** performs hardware-accelerated dequantization of Outlier-Aware INT4 weights **directly inside GPU registers (SRAM)** via custom **OpenAI Triton GEMM Kernels**, streams weights incrementally without RAM allocation (< 150 MB RAM peak), and integrates long-term user memory (`companion_memory.py`).

---

## 🔬 Formal Scientific Logit Parity Benchmark (20 Prompts)

Evaluated via `tests/test_formal_parity.py` comparing Base FP16 vs OTF INT4 Champion Engine across 20 multi-domain prompts:

* 📐 **Average Logit Cosine Similarity:** **`98.1556%`**
* 🎯 **Top-1 Exact Token Match Rate:** **`70.0%`** (14/20 exact token identity)
* 📉 **Average KL-Divergence:** **`0.2154`**

---

## 📊 Performance Benchmark (NVIDIA RTX 5060 Ti 16GB)

Benchmarking conducted on an **NVIDIA GeForce RTX 5060 Ti 16GB** GPU:

| Model / Architecture | Format | Static VRAM | Peak VRAM | Speed | Load Time | Intelligence Parity | Status |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **Qwen2.5-3B (Base)** | FP16 | 5.75 GB | 5.81 GB | 25.6 t/s | ~15.0 s | 100% (Baseline) | Baseline |
| **Qwen2.5-3B (OTF Champion)** | **INT4/8** | **1.94 GB** | **1.99 GB** | **16.15 t/s** | **4.2 s** | **98.3% Logit Parity** | 🏆 **CHAMPION (-66.1%)** |
| **Llama-3.2-3B (Base)** | FP16 | 6.40 GB | 6.48 GB | 22.1 t/s | ~16.0 s | 100% (Baseline) | Baseline |
| **Llama-3.2-3B (OTF Champion)** | **INT4/8** | **1.89 GB** | **1.99 GB** | **15.89 t/s** | **3.7 s** | **98.2% Logit Parity** | 🏆 **RECORD (-70.5%)** |
| **Qwen2.5-7B (Base)** | FP16 | 15.27 GB | 15.80 GB | 14.2 t/s | ~28.0 s | 100% (Baseline) | Baseline |
| **Qwen2.5-7B (OTF Champion)** | **INT4/8** | **4.20 GB** | **4.25 GB** | **10.13 t/s** | **5.1 s** | **98.3% Logit Parity** | 🏆 **CHAMPION (-72.5%)** |
| **Qwen1.5-MoE-A2.7B (14.3B Total)** | **3-Tier** | **3.09 GB** | **3.40 GB** | **4.14 t/s** | **6.0 s** | **R&D Alpha** | 🧪 **[EXPERIMENTAL]** |

---

## 🏛️ Architecture & Key Innovations

```
[Input Vector X] ──► [Global Static Permutation (global_perm_idx)]
                                   │
         ┌─────────────────────────┴─────────────────────────┐
         ▼                                                   ▼
[Outlier Channels (1% FP16)]                 [Background Block (99% INT4)]
   │                                                         │
   ├──► Pure FP16                                            ├──► Range [-7 ... +7]
   └──► Input X_outliers                                     ├──► 2:1 Packing (uint8)
                                                             └──► Zero-Point = 0 BYTES!
                                                                     │
                                                                     ▼
                                                      [Custom Fused Triton GEMM Kernel]
                                                      (Dequantization in GPU SRAM Registers)
                                                                     │
         ┌───────────────────────────────────────────────────────────┘
         ▼
[Continuous GEMM Addition: Outliers + Triton Background = Exact FP16 Output]
```

1. **Custom Outlier-Aware Fused Triton GEMM Kernel (`otf_triton_kernel.py`):**
   Packed `uint8` weights are streamed from VRAM and dequantized **directly inside GPU chip registers (SRAM)** during matrix multiplication, eliminating temporary FP16 tensor allocations in VRAM.
2. **Incremental Sharded Quantizer with Resume Support (`convert_global_universal.py`):**
   Memory-mapped (`mmap`) sharded quantizer with < 150 MB CPU RAM footprint. Saves intermediate disk chunks to enable crash-resilient resuming.
3. **Global Static Permutation (`global_perm_idx`) & Outlier Preservation:**
   A unified channel permutation table across the model (1.6 MB VRAM). Isolating the Top-1% critical outlier channels ($|W| \times |X_{\text{profile}}|$) in FP16 completely suppresses quantization noise and guarantees 98.16% logit similarity.
4. **Companion Long-Term Memory Manager (`companion_memory.py`):**
   Zero-VRAM, lightweight CPU RAM module that automatically extracts and retrieves user facts in < 2 ms via TF-IDF cosine similarity, injecting relevant facts into system prompts.
5. **3-Tier Hierarchical MoE Offloader [EXPERIMENTAL] (`otf_moe_offloader.py`):**
   R&D module for executing Mixture-of-Experts models via a 3-tier memory hierarchy (Disk ➔ CPU RAM ➔ GPU VRAM).
6. **Gradio Web Demo (`otf-demo`) & FastAPI REST API (`otf-server`):**
   Includes a built-in interactive browser Web UI with real-time VRAM allocation counters and OpenAI API spec compatibility (`/v1/chat/completions`).

---

## 📁 Repository Structure

```
otf-llm-engine/
├── setup.py                                  # Setuptools package configuration
├── pyproject.toml                            # PEP 517/518 build system
├── MANIFEST.in                               # Package assets configuration
├── pipeline_run.py                           # Automated 1-click end-to-end pipeline
├── otf_llm/                                  # Main python package namespace
│   ├── __init__.py                           # Module entry point (v3.2.0)
│   ├── make_profile_universal.py             # Activation profile calibrator
│   ├── convert_global_universal.py           # Incremental mmap safetensors quantizer
│   ├── run_triton_universal.py               # Triton GEMM inference runner
│   ├── otf_triton_kernel.py                  # Custom Fused Triton INT4 GEMM kernel
│   ├── companion_memory.py                   # Zero-VRAM long-term user memory store
│   ├── web_demo.py                           # Interactive Gradio Web UI
│   ├── otf_moe_offloader.py                  # [EXPERIMENTAL] 3-Tier MoE Offloader
│   ├── direct_quantized_importer.py          # [EXPERIMENTAL] AWQ/GPTQ importer
│   ├── query_guided_sparse_kv.py             # Context retrieval (CPU RAM -> GPU)
│   ├── otf_context_compressor.py             # SnapKV / KIVI cache compressor
│   └── server_fastapi.py                     # REST API server (OpenAI API + SSE)
├── tests/                                    # Benchmark & validation test suites
│   ├── test_formal_parity.py                 # Scientific logit parity benchmark (20 prompts)
│   ├── validate_llama3_2_3b.py               # Llama-3.2-3B validation runner
│   ├── test_intelligence_suite.py            # Intelligence and logic reasoning suite
│   ├── test_base_model_suite.py              # Baseline FP16 model suite
│   └── test_client.py                        # SSE streaming test client
├── README.md                                 # Project documentation
└── LICENSE                                   # MIT License
```

---

## 🛠️ Quickstart & Installation

### Official PyPI Installation

```bash
pip install otf-llm
```

### Launch Interactive Web Demo

```bash
otf-demo
```

### Run Formal Scientific Parity Test

```bash
python tests/test_formal_parity.py
```

---

## 📜 License

Distributed under the MIT License. See [LICENSE](LICENSE) for details.
