Metadata-Version: 2.4
Name: nearbits
Version: 0.3.0
Summary: Nearbits ternary-residual compression for Hugging Face models on low-RAM devices.
Author: subham
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: transformers>=4.40.0
Requires-Dist: accelerate>=0.28.0
Requires-Dist: sentencepiece>=0.1.99
Requires-Dist: psutil>=5.9.0

# Nearbits

Nearbits is a Python package for compressing Hugging Face models with a ternary-residual strategy tuned for the lowest practical RAM footprint.

The default math is a post-training `1.58-bit` style path for ordinary linear weights: a grouped ternary base `{-1, 0, 1}` with blended absmean/RMS scaling, plus a tiny VIP residual slice kept in `4-bit` or `8-bit`. Protected modules such as embeddings, norms-adjacent outputs, and heads stay in `int8` for stability.

## New Tech

Nearbits now uses a `bitnet158` formula for ordinary projection layers:

```text
W_hat = s_g * T(W_g, tau_g) + R_vip
```

Where:

- `W_g` is a weight group
- `T(W_g, tau_g)` is ternary quantization to `{-1, 0, 1}` with threshold `tau_g`
- `s_g = alpha * mean(|W_g|) + (1 - alpha) * rms(W_g)` is the blended group scale
- `R_vip` is a tiny residual correction made from only the top VIP weights, stored in `4-bit` or `8-bit`

Default values in this release:

```text
alpha = 0.7
	au_g = 0.9 * mean(|W_g|)
vip_fraction = 0.001
max_vip_fraction = 0.01
vip_bits = 4
activation_bits = 8
```

This is designed to keep RAM low on arbitrary Hugging Face checkpoints while still producing usable output. It is inspired by BitNet-style ternary math, but this package is still a post-training approximation rather than native BitNet training.

## What Nearbits does

- Compresses ordinary `nn.Linear` and Hugging Face `Conv1D` layers to a `bitnet158`-style ternary code path plus small VIP residuals
- Keeps only protected layers such as embeddings and heads in `int8`
- Applies optional per-token activation quantization in `8-bit` during inference-style forward passes
- Works directly on existing Hugging Face models without retraining
- Exposes simple chat and prompt benchmarking helpers

## Install

Install PyTorch first for your platform, then install Nearbits.

```bash
pip install torch
pip install nearbits
```

For local development from this repo:

```bash
pip install torch
pip install -e .
```

## Quick Start

```python
from nearbits import HybridLowBitConfig, chat, compress_hf_model, load_hf_model, model_report

model, tokenizer, model_kind = load_hf_model("Qwen/Qwen2.5-1.5B-Instruct")
print(model_report(model))

config = HybridLowBitConfig(
    group_size=128,
    bitnet_scale_blend=0.7,
    bitnet_threshold_factor=0.9,
    vip_fraction=0.001,
    max_vip_fraction=0.01,
    vip_bits=4,
    activation_bits=8,
)

compressed_model = compress_hf_model(model, config)
print(model_report(compressed_model))

reply = chat(compressed_model, tokenizer, "Who are you?")
print(reply)
```

## Google Colab Usage

Use `examples/google_collab_hybrid_lowbit.py` as the runnable Colab example.

Typical Colab cells:

```python
!pip install torch
!pip install nearbits
```

```python
from nearbits import HybridLowBitConfig, benchmark_prompts, compress_hf_model, load_hf_model, model_report

MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"
PROMPTS = [
    "hi",
    "hello",
    "who are you?",
    "what is 2 + 2?",
    "write one sentence about India.",
    "explain model compression in simple words.",
]

model, tokenizer, model_kind = load_hf_model(MODEL_ID)
print("Before:", model_report(model))

compressed_model = compress_hf_model(
    model,
    HybridLowBitConfig(
        vip_fraction=0.001,
        max_vip_fraction=0.01,
        vip_bits=4,
        activation_bits=8,
    ),
)
print("After:", model_report(compressed_model))

rows = benchmark_prompts(compressed_model, tokenizer, PROMPTS)
for row in rows:
    print(row["prompt"])
    print(row["response"])
    print(row["latency_sec"])
```

## Public API

- `load_hf_model(model_id, trust_remote_code=False)`
- `compress_hf_model(model, config=None)`
- `model_size_mb(model)`
- `model_report(model)`
- `chat(model, tokenizer, prompt, ...)`
- `benchmark_prompts(model, tokenizer, prompts, ...)`
- `HybridLowBitConfig(...)`

## Notes

- The public package name is `nearbits`
- The default ordinary-layer strategy is `bitnet158`, not whole-layer `int8`
- This is a post-training approximation for arbitrary Transformer checkpoints, not native BitNet training
- The strongest RAM reduction comes from the ternary base plus bounded VIP residuals

## Publish New Version

Build and upload the new version to PyPI with:

```bash
python -m pip install --upgrade build twine
python -m build
python -m twine upload dist/*
```

If you want to publish only this version's fresh artifacts, clean `dist` first and then upload:

```bash
rmdir /s /q dist
python -m build
python -m twine upload dist/*
```
