Metadata-Version: 2.4
Name: nearbits
Version: 0.2.0
Summary: Hybrid low-bit compression for Hugging Face models on low-RAM devices.
Author: Codex
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: transformers>=4.40.0
Requires-Dist: accelerate>=0.28.0
Requires-Dist: sentencepiece>=0.1.99
Requires-Dist: psutil>=5.9.0

﻿# NearBit

NearBit is a Python package for compressing Hugging Face models with a single strategy: `hybrid_lowbit`.

The package keeps high-impact parts in `int8`, compresses the bulk of linear layers to `binary` or `ternary`, rescues outlier weights in `int8`, and targets the lowest practical RAM footprint without retraining.

## What NearBit does

- Compresses `nn.Linear` layers with `binary` or `ternary` storage plus group-wise scales
- Keeps sensitive linear layers in `int8`
- Compresses `nn.Embedding` tables to `int8`
- Works directly on existing Hugging Face models
- Exposes simple chat and prompt benchmarking helpers

## Install

Install PyTorch first for your platform, then install NearBit.

```bash
pip install torch
pip install nearbit
```

For local development from this repo:

```bash
pip install torch
pip install -e .
```

## Quick Start

```python
from nearbit import HybridLowBitConfig, chat, compress_hf_model, load_hf_model, model_report

model, tokenizer, model_kind = load_hf_model("Qwen/Qwen2.5-1.5B-Instruct")
print(model_report(model))

config = HybridLowBitConfig(
    group_size=128,
    binary_outlier_fraction=0.002,
    ternary_outlier_fraction=0.001,
    binary_nmse_threshold=0.045,
    ternary_nmse_threshold=0.02,
)

compressed_model = compress_hf_model(model, config)
print(model_report(compressed_model))

reply = chat(compressed_model, tokenizer, "Who are you?")
print(reply)
```

## Google Colab Usage

Use `examples/google_collab_hybrid_lowbit.py` as the runnable Colab example.

Typical Colab cells:

```python
!pip install torch
!pip install nearbit
```

```python
from nearbit import HybridLowBitConfig, benchmark_prompts, compress_hf_model, load_hf_model, model_report

MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"
PROMPTS = [
    "hi",
    "hello",
    "who are you?",
    "what is 2 + 2?",
    "write one sentence about India.",
    "explain model compression in simple words.",
]

model, tokenizer, model_kind = load_hf_model(MODEL_ID)
print("Before:", model_report(model))

compressed_model = compress_hf_model(model, HybridLowBitConfig())
print("After:", model_report(compressed_model))

rows = benchmark_prompts(compressed_model, tokenizer, PROMPTS)
for row in rows:
    print(row["prompt"])
    print(row["response"])
    print(row["latency_sec"])
```

## Public API

- `load_hf_model(model_id, trust_remote_code=False)`
- `compress_hf_model(model, config=None)`
- `model_size_mb(model)`
- `model_report(model)`
- `chat(model, tokenizer, prompt, ...)`
- `benchmark_prompts(model, tokenizer, prompts, ...)`
- `HybridLowBitConfig(...)`

## Notes

- NearBit is centered on one executable path: `hybrid_lowbit`
- There is no public `int8` fallback API in the package surface
- Quality depends on the model architecture and compression thresholds
- The strongest RAM reduction comes from the hybrid path, not from pure `int8`
