Metadata-Version: 2.4
Name: ternary-quant
Version: 0.1.6
Summary: Post-training ternary quantization for HuggingFace generative models
Author: Asad Ismail
License: Apache-2.0
Project-URL: Repository, https://github.com/Asad-Ismail/ternary-quant
Project-URL: Issues, https://github.com/Asad-Ismail/ternary-quant/issues
Keywords: quantization,ternary,llm,vlm,post-training-quantization,generative-models
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.1.0
Requires-Dist: transformers>=4.46.0
Requires-Dist: accelerate>=0.28.0
Requires-Dist: datasets>=2.18.0
Requires-Dist: safetensors>=0.4.0
Requires-Dist: tqdm
Requires-Dist: numpy
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Provides-Extra: paper
Requires-Dist: hqq>=0.2.8.post1; extra == "paper"
Requires-Dist: matplotlib>=3.7.0; extra == "paper"
Provides-Extra: multimodal
Requires-Dist: pillow; extra == "multimodal"
Requires-Dist: num2words; extra == "multimodal"
Requires-Dist: torchvision; extra == "multimodal"
Provides-Extra: eval
Requires-Dist: lm-eval>=0.4.0; extra == "eval"
Provides-Extra: serving
Requires-Dist: vllm>=0.4.0; extra == "serving"
Dynamic: license-file

<div align="center">

```
 _____     ___                _   
|_   _|__ / _ \ _  _ __ _ _ _| |_ 
  | ||___| (_) | || / _` | ' \  _|
  |_|     \__\_\\_,_\__,_|_||_\__|
```

**Component-first post-training ternary quantization for HuggingFace models.
HF-native PTQ for VLMs, seq2seq, and audio — with packed runtime modes for selected 7B-class models.**

[![CI](https://github.com/Asad-Ismail/ternary-quant/actions/workflows/ci.yml/badge.svg)](https://github.com/Asad-Ismail/ternary-quant/actions/workflows/ci.yml)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/)
[![PyPI version](https://img.shields.io/pypi/v/ternary-quant.svg)](https://pypi.org/project/ternary-quant/)
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Asad-Ismail/ternary-quant/blob/main/notebooks/quickstart.ipynb)
[![Demo](https://img.shields.io/badge/🤗%20Space-Live%20Demo-blue)](https://huggingface.co/spaces/AsadIsmail/ternary-quant-demo)

**[Try the live demo](https://huggingface.co/spaces/AsadIsmail/ternary-quant-demo)**  &nbsp;|&nbsp;  Pre-quantized: [Qwen3-1.7B](https://huggingface.co/AsadIsmail/Qwen3-1.7B-ternary) &middot; [Qwen2-VL-2B](https://huggingface.co/AsadIsmail/Qwen2-VL-2B-ternary) &middot; [Gemma4-E2B](https://huggingface.co/AsadIsmail/Gemma4-E2B-ternary)

</div>

GGUF/llama.cpp is the strongest default for local deployment of supported models,
and current llama.cpp builds support selected multimodal models through
`libmtmd` (including image input and experimental audio input). `ternary-quant`
targets a different gap: HuggingFace/Python-native, component-first
post-training ternary quantization of existing FP16 checkpoints. Decoder-LLM
benchmarks retain 95–99% of FP16 quality (97.7% mean across seven models);
multimodal/audio quality is model-card and validation-suite dependent.

```
Qwen2.5-7B:   15.2 GB FP16  →  9.6 GB triton_memory  ✓ fits 12 GB GPU (RTX 4070/3080)
Phi-3.5-mini:  7.6 GB FP16  →  4.6 GB triton_memory  ✓ fits  6 GB GPU (RTX 3060/GTX 1060)
Mistral-7B:   14.0 GB FP16  →  5.0 GB stored           (98.7% ARC quality, stays in Python/HF)
FLAN-T5-small:154 MB FP16  →  125 MB stored            (5.1× decoder component)
```

---

## Quickstart

```bash
pip install ternary-quant
# or with uv
uv pip install ternary-quant

# Vision-Language Model
ternary-quant quantize-broad Qwen/Qwen2-VL-2B-Instruct \
  --output ./qwen2vl-quant \
  --components text_backbone vision_backbone \
  --scheme tritplane3

# Causal LM
ternary-quant quantize-broad Qwen/Qwen3-4B \
  --output ./qwen3-4b-quant --components text_backbone --scheme tritplane3

# Seq2seq / Audio
ternary-quant quantize-broad openai/whisper-small \
  --output ./whisper-quant --components decoder

# Generate
ternary-quant generate ./qwen2vl-quant --prompt "Describe this image" --runtime-mode cached
```

Or use a pre-quantized model directly:

```python
from ternary_quant.inference import load_ternary_model, generate_text

model, tokenizer = load_ternary_model("AsadIsmail/Qwen3-1.7B-ternary", device="cuda", runtime_mode="cached")
print(generate_text(model, tokenizer, "What is ternary quantization?", max_new_tokens=100))
```

---

## What it does

`ternary-quant` is **post-training quantization (PTQ)** — it takes supported
existing FP16 HuggingFace checkpoints and pushes selected transformer weights
toward {-1, 0, +1} (ternary). No retraining required.

**PTQ ternary vs trained ternary (BitNet):** BitNet-style models are trained
from scratch to operate with ternary weights, which is different from converting
an existing FP16 checkpoint after training. `ternary-quant` does PTQ: take an
existing FP16 model (Qwen, Mistral, Whisper, VLMs) and quantize selected
components without retraining. GGUF/llama.cpp supports trained ternary formats
for compatible models, but not this repo's HF-native component-first PTQ
workflow for arbitrary checkpoints.

---

## Key results

**95–99% decoder-LLM quality retention** across 7 models (0.5B-7B), measured with lm-eval 0-shot (ARC, HellaSwag, WinoGrande):

| Model | Avg retain | Compression |
|---|---|---|
| Phi-2 (2.7B) | **99.2%** | 1.8x |
| Qwen2.5-0.5B | **99.1%** | 1.8x |
| Mistral-7B | **98.7%** | 2.8x |
| Qwen3-4B | **98.1%** | 1.3x |
| Qwen3-1.7B | **97.5%** | 2.7x |

Full results: [docs/KNOWN_GOOD_MODELS.md](docs/KNOWN_GOOD_MODELS.md)

Validated paths cover decoder LLMs, selected VLMs (Qwen2-VL, Gemma-4,
SmolVLM), seq2seq (FLAN-T5), and audio (Whisper).

---

## How it compares

| Tool | LLMs | VLMs | Seq2seq / Audio | PTQ ternary | Component selection | Stays in Python/HF |
|---|---|---|---|---|---|---|
| GGUF / llama.cpp | ✓✓✓ | ✓ (selected models via libmtmd) | partial / experimental | trained only | partial | ✗ |
| bitsandbytes | ✓✓ | partial | ✗ | ✗ | ✗ | ✓ |
| GPTQ / AWQ | ✓✓ | partial | ✗ | ✗ | ✗ | ✓ |
| PT²-LLM / PTQTP | ✓✓ | ✗ | ✗ | ✓ | ✗ | ✗ |
| **ternary-quant** | **✓✓** | **✓✓** | **✓✓** | **✓** | **✓** | **✓** |

**Choose the right tool:**
- Standard decoder LLM, want max speed locally → **GGUF** (llama.cpp is faster on CPU, better ecosystem)
- PTQ ternary on decoder LLMs, care about peak quality → **PT²-LLM** (ICLR 2026, strongest ternary PTQ results on decoder LLMs)
- Need HF-native component-first ternary PTQ for a VLM, seq2seq, audio, or custom model → **ternary-quant**
- 4-bit LLM with best quality-per-bit → **HQQ-4** (outperforms ternary-quant on standard LLMs)

---

## Runtime modes

| Mode | Hardware | Weights | Speed | Use case |
|---|---|---|---|---|
| `cached` | CPU / GPU / MPS | FP16 in memory | **fastest** | Default — dequantizes once at load |
| `triton_memory` | NVIDIA GPU | **packed 2-bit** | ~3-6x slower | Fit oversized models on small GPUs |
| `metal` | Apple Silicon | **packed 2-bit** | ~3-4x slower | Fit oversized models on Mac |

`triton_memory` and `metal` keep weights as packed 2-bit ternary codes throughout inference — no FP16 weight matrix is ever materialized. This trades speed for memory, enabling models that don't fit otherwise.

> **`metal` mode requires torch ≥ 2.7** (for `torch.mps.compile_shader`). On older torch it degrades to `cached`. Apple-Silicon MPS generation also needs torch ≥ 2.7 to avoid a GQA matmul crash present in 2.6.

| Model | FP16 VRAM | triton_memory | Fits on |
|---|---|---|---|
| Phi-3.5-mini (3.8B) | 7.6 GB | **4.6 GB** | RTX 3060 6GB |
| Qwen2.5-7B (7.6B) | 15.2 GB | **9.6 GB** | RTX 4070 12GB |

| Model | Cached (Mac) | Metal (Mac) | Memory saved |
|---|---|---|---|
| Qwen3-1.7B | 10.4 GB / 46 tok/s | 4.6 GB / 12 tok/s | 2.3x |
| Qwen2-VL-2B | 16.8 GB / 52 tok/s | 7.4 GB / 13 tok/s | 2.3x |

Full details: [docs/RUNTIME_MODES.md](docs/RUNTIME_MODES.md)

---

## Learn more

- [Known good models](docs/KNOWN_GOOD_MODELS.md) — measured results and commands for validated architecture families
- [Runtime modes](docs/RUNTIME_MODES.md) — triton_memory, metal, CPU inference details
- [Research track](docs/RESEARCH_ADVANCED_TERNARY.md) — RAST + CCRE for true 1.58-bit compression
- [Examples](examples/README.md) — text, seq2seq, VLM walkthroughs
- [Paper](paper/paper.tex) — full experimental results and method description (LaTeX)
- [Release notes](releases/v0.1.6.md) — what changed in this version

---

## Honest positioning

- **HQQ-4 beats ternary-quant on quality-per-bit** for standard LLMs — use it when quality is the priority
- **PT²-LLM and PTQTP** achieve strong PTQ ternary results on decoder-only LLMs — ternary-quant's algorithmic contribution on that axis is parallel, not superior
- The broad path is a ternary PTQ research/runtime path, not a general replacement for strong INT4 deployments
- The research track achieves true near-ternary weights with a real quality cost — this is the active research frontier, not a production-ready path
- GGUF is better for local deployment of supported standard LLMs and selected multimodal models; ternary-quant's value is component-first ternary PTQ in the HuggingFace ecosystem
