Metadata-Version: 2.4
Name: bigrig
Version: 0.2.0
Summary: Run large Mixture-of-Experts models on an Apple Silicon Mac that does not have room for them, and know what it cost
Author: BigRig
License: Apache-2.0
Project-URL: Homepage, https://github.com/arjvnv/bigrig
Project-URL: Source, https://github.com/arjvnv/bigrig
Project-URL: Issues, https://github.com/arjvnv/bigrig/issues
Keywords: mlx,apple-silicon,mixture-of-experts,llm,inference
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS :: MacOS X
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: huggingface_hub>=0.20
Requires-Dist: mlx>=0.20; sys_platform == "darwin" and platform_machine == "arm64"
Requires-Dist: mlx-lm>=0.20; sys_platform == "darwin" and platform_machine == "arm64"
Provides-Extra: engine
Requires-Dist: mlx>=0.20; extra == "engine"
Requires-Dist: mlx-lm>=0.20; extra == "engine"
Requires-Dist: huggingface_hub>=0.20; extra == "engine"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Dynamic: license-file

<div align="center">

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="assets/logo-dark.svg">
  <source media="(prefers-color-scheme: light)" srcset="assets/logo-light.svg">
  <img alt="bigrig" src="assets/logo-light.svg" width="46%">
</picture>

<br>

![Apple Silicon](https://img.shields.io/badge/platform-Apple%20Silicon-1b1a18?style=flat-square)
![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-2f6f5e?style=flat-square)
![795 assertions](https://img.shields.io/badge/tests-795%20assertions-2f6f5e?style=flat-square)
![Bit-exact](https://img.shields.io/badge/streaming-bit--exact-2f6f5e?style=flat-square)
![Apache 2.0](https://img.shields.io/badge/licence-Apache--2.0-2f6f5e?style=flat-square)

**[Quickstart](docs/quickstart.md)** ·
[Install](docs/install.md) ·
[Models](docs/models.md) ·
[CLI](docs/cli.md) ·
[How it works](docs/HOW-IT-WORKS.md) ·
[Measurements](docs/MEASUREMENTS.md) ·
[Contributing](CONTRIBUTING.md)

</div>

---

## About

A 30B MoE model uses only a handful of its experts for any given word, but every runtime keeps
all of them in memory. bigrig keeps a bounded set in RAM and reads the rest from SSD, or shrinks
the model to fit — **your choice, asked before anything changes** — while a quality meter watches
the output and tells you if the trade went too far.

- **Three strategies, chosen for you.** Run it untouched, shrink it to fit, or stream it from
  disk. Picked in increasing order of what each costs you: nothing, then accuracy, then speed.
- **It asks before changing your model.** Shrinking is the only path that alters weights, so it
  is the only one that needs your agreement — and every run afterwards still prints which mode
  it is serving.
- **Streaming is bit-exact.** Not "close". The same model resident and streamed at a 36.75%
  cache-miss rate scored identical perplexity to ten decimal places.
- **A quality meter, running live.** No reference answer, no second forward pass. It separates a
  healthy model from a damaged one at 0.0% vs 36.7% of tokens flagged.
- **Nothing extra on disk.** Experts are read straight out of the downloaded safetensors, so
  preparing a model writes nothing at all.
- **Speaks OpenAI and Anthropic.** Including a one-command hook into Claude Code, and a web
  interface for people who would rather not use a terminal.

---

## Getting started

### Install

```bash
git clone https://github.com/arjvnv/bigrig.git && cd bigrig
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[engine]"
```

`rig` and `bigrig` are the same command. Full details in [install.md](docs/install.md).

### Run a model

```bash
rig prepare mlx-community/OLMoE-1B-7B-0125-Instruct-4bit
rig serve   OLMoE-1B-7B-0125-Instruct-4bit
```

`prepare` downloads the model and writes **nothing else**. `serve` tells you exactly what it is
about to serve, then where to find it:

```
  bigrig serving OLMoE-1B-7B-0125-Instruct-4bit on http://127.0.0.1:8080
  running EXACT at 4-bit, fully in RAM (untouched)
  quality monitor: on

  open  http://127.0.0.1:8080  in a browser to chat and watch quality
  api   127.0.0.1:8080/v1/chat/completions   (OpenAI)
        127.0.0.1:8080/v1/messages           (Anthropic -- Claude Code)
  agent bigrig launch OLMoE-1B-7B-0125-Instruct-4bit
```

That second line is printed on **every** run. Whatever mode you are in, the precision being
served is never a surprise.

### Then pick whichever suits you

**A browser** — open `http://127.0.0.1:8080`. A chat box, streaming replies, and along the top:
the model, the mode it is running in, memory held, live tokens/second, and a **quality indicator
that turns amber mid-generation** if the model starts looping or losing coherence. One
self-contained page, no CDN, nothing to install.

**Claude Code** — one command, which starts the server and points the agent at it:

```bash
rig launch OLMoE-1B-7B-0125-Instruct-4bit
```

Nothing on disk is changed: the environment variables are set on the agent's process only.

**Your own code** — both APIs are served on the same port, so any client library works by
changing its base URL:

```
POST /v1/chat/completions      OpenAI
POST /v1/messages              Anthropic
```

**A terminal** — `rig run <model>`.

---

## What happens when a model doesn't fit

bigrig stops and asks, because the two ways forward cost different things:

```
    [1] Shrink it to fit     full speed, but THE WEIGHTS CHANGE
    [2] Keep it exact        bit-for-bit identical, streamed from disk, slower
```

Your answer is remembered, and every run still prints which one it is serving. In a script with
neither `--compress` nor `--exact`, it **refuses** rather than guessing — quietly serving a
degraded model is worse than stopping.

**It will also tell you not to use it.** If a model already fits, bigrig loads it normally and
says so; putting the engine in front of a model that doesn't need it only makes it slower.

---

## Measurements

On an M4 MacBook Air (24 GB). Nothing here is projected.

**Shrinking vs streaming, at the same memory** (OLMoE-1B-7B, wikitext-2):

| memory | shrink (all in RAM) | stream (exact) | which wins |
|---|---|---|---|
| 2.82 GB | 3-bit, 92 tok/s, +17% perplexity | 50 tok/s, exact | shrink |
| 2.62 GB | 3-bit, **111 tok/s**, +18% perplexity | 25 tok/s, exact | **shrink, 4.4×** |
| 2.01 GB | 2-bit, 110 tok/s, **+83%** perplexity | 17 tok/s, exact | stream |

Below 3 bits is a cliff, not a bargain — so **3 bits is the floor**, and going lower takes an
explicit `--min-bits 2`.

Full write-up, including the negative results: [docs/MEASUREMENTS.md](docs/MEASUREMENTS.md).

---

## The quality meter on its own

`bigrig_layer` is independent of the engine. It watches any model's output distribution and
flags degradation with no reference answer and no second forward pass — and it works against
Ollama, llama.cpp and the OpenAI API too:

```python
from bigrig_layer import AdaptiveMeter

m = AdaptiveMeter()
for step in generation:
    m.observe(probs)
    m.observe_token(tok)
    if m.is_degraded():
        print("quality problem:", m.reason())
```

---

## Repository layout

```
bigrig_engine/     the engine — strategy, streaming, precision, serving, web UI, CLI
bigrig_layer/      the quality meter (standalone, engine-agnostic)
tests/               ./run-tests.sh runs every assertion
docs/                install, quickstart, models, cli, design, measurements
src/  notes/         research scripts and lab notebook — not shipped
```

See [CONTRIBUTING.md](CONTRIBUTING.md) for how the tests are written and why.

## Requirements

Apple Silicon (M1 or later), macOS, Python 3.10+.

## Licence

[Apache License 2.0](LICENSE). Use it, modify it, ship it commercially — keep the notice and
don't sue us over patents.
