Metadata-Version: 2.4
Name: experts4bit-qlora
Version: 0.54.0
Summary: Fine-tune and serve large Mixture-of-Experts models on one GPU, with 4-bit experts in GPU memory, RAM or SSD.
Author-email: Jordan Anderson <paul.jordan.anderson@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://cerinamroth.com/ml/experts4bit-qlora/
Project-URL: Documentation, https://cerinamroth.com/ml/experts4bit-qlora/
Project-URL: Source, https://github.com/pjordanandrsn/experts4bit-qlora
Project-URL: Issues, https://github.com/pjordanandrsn/experts4bit-qlora/issues
Project-URL: Changelog, https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/CHANGELOG.md
Project-URL: Release Notes, https://github.com/pjordanandrsn/experts4bit-qlora/releases/latest
Project-URL: Status, https://cerinamroth.com/ml/status/
Project-URL: Capabilities, https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/capabilities.json
Project-URL: Solutions, https://cerinamroth.com/ml/solutions/
Project-URL: Benchmarks, https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/claims.json
Project-URL: Kernel: grouped-nf4-gemm, https://pypi.org/project/grouped-nf4-gemm/
Project-URL: Upstream (bitsandbytes#1965), https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1965
Project-URL: Tracking (bitsandbytes#1849), https://github.com/bitsandbytes-foundation/bitsandbytes/issues/1849
Keywords: mixture-of-experts,moe,qlora,lora,bitsandbytes,transformers,pytorch,quantization,nf4,mxfp4,int4,fp8,cuda,triton,gpu-offload,nvme,llm-inference,llm-training,fine-tuning,consumer-gpu
Classifier: Environment :: GPU :: NVIDIA CUDA
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: THIRD_PARTY_NOTICES.md
Requires-Dist: torch>=2.2
Requires-Dist: bitsandbytes>=0.43
Provides-Extra: fast
Requires-Dist: grouped-nf4-gemm>=0.30.0; extra == "fast"
Provides-Extra: train
Requires-Dist: transformers>=5.0; extra == "train"
Requires-Dist: datasets>=2.14; extra == "train"
Requires-Dist: accelerate>=0.30; extra == "train"
Requires-Dist: safetensors>=0.4; extra == "train"
Requires-Dist: huggingface_hub>=0.23; extra == "train"
Provides-Extra: serve
Requires-Dist: fastapi>=0.110; extra == "serve"
Requires-Dist: uvicorn>=0.29; extra == "serve"
Requires-Dist: pydantic>=2; extra == "serve"
Requires-Dist: transformers>=5.0; extra == "serve"
Requires-Dist: accelerate>=0.30; extra == "serve"
Requires-Dist: safetensors>=0.4; extra == "serve"
Requires-Dist: huggingface_hub>=0.23; extra == "serve"
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Requires-Dist: transformers>=5.0; extra == "test"
Requires-Dist: accelerate>=0.30; extra == "test"
Requires-Dist: safetensors>=0.4; extra == "test"
Requires-Dist: fastapi>=0.110; extra == "test"
Requires-Dist: httpx>=0.27; extra == "test"
Requires-Dist: grouped-nf4-gemm>=0.30.0; extra == "test"
Requires-Dist: jsonschema>=4.18; extra == "test"
Requires-Dist: compressed-tensors==0.18.0; extra == "test"
Dynamic: license-file

# experts4bit-qlora

### Fine-tune large MoE models on a single GPU.

[![PyPI](https://img.shields.io/pypi/v/experts4bit-qlora)](https://pypi.org/project/experts4bit-qlora/)
[![CI](https://github.com/pjordanandrsn/experts4bit-qlora/actions/workflows/ci.yml/badge.svg)](https://github.com/pjordanandrsn/experts4bit-qlora/actions/workflows/ci.yml)

**Quantize the experts. Train adapters. Move weights to RAM or SSD when GPU memory runs out.**

**On a 12 GB card:** with `OFFLOAD_EXPERTS=1`, Qwen3-30B-A3B QLoRA-trains at 7.16 GB peak GPU memory while the other
~13–15 GB of packed experts stay in pinned CPU RAM (`e4b.offload.fits-30b-class`). [The command](#load-a-model-and-check-it) ·
[how to read the number](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/STORAGE-MODES.md#reading-the-headline-memory-numbers)

Mixture-of-Experts (MoE) models keep many expert networks but activate only a few per token.
Their fused expert weights can escape ordinary `load_in_4bit=True` loading and stay in full precision.
This runtime loads those weights in 4-bit, trains them with QLoRA, and serves the model.

**New here? [Start with Loggetta](https://github.com/pjordanandrsn/loggetta).**
`pip install loggetta` installs this runtime and its kernels. Use `loggetta inspect` to check your machine
and `loggetta plan` to plan supported training.
Use this package directly when you need control over loading, training, serving, or memory placement.

> **Limits.** GPU execution needs Linux and NVIDIA CUDA; the fast kernels need an sm_80+ GPU. Support varies by model
> family and execution path. 4-bit saves memory but is not automatically faster than bf16, and vLLM was faster in the
> measured Qwen decode comparison. [Check the fit](#check-the-fit).

[Install](#install) · [Results](#what-is-measured) · [Model support](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/ARCHITECTURE_SUPPORT.md) · [Task guides](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/SOLUTIONS.md) · [Hugging Face](https://huggingface.co/spaces/pjordanandrsn/research)

<!-- release-block:start -->
<!-- generated by `python scripts/check_readme_claims.py --write-release-block` from CHANGELOG.md's latest release heading; do not edit by hand -->
**Latest released package:** [`experts4bit-qlora` 0.54.0](https://pypi.org/project/experts4bit-qlora/0.54.0/) · **Current development status:** [`docs/STATUS.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/STATUS.md) on `main` (this README describes `main`) · **Released documentation for 0.54.0:** [`docs/`](https://github.com/pjordanandrsn/experts4bit-qlora/tree/v0.54.0/docs) · [`README.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/v0.54.0/README.md) · [`CHANGELOG.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/v0.54.0/CHANGELOG.md)
<!-- release-block:end -->

## Install

```bash
pip install "experts4bit-qlora[train,fast]"
```

GPU execution needs **Linux + NVIDIA CUDA**. Fast kernels need an **sm_80+ GPU**,
Triton ≥ 3.4; the streaming loader needs transformers ≥ 5.0. The base runtime needs torch ≥ 2.2;
the fast extra applies the kernel package’s newer PyTorch floor.
CI tests Python 3.11. Start with a CUDA-compatible PyTorch installation for your GPU.

For a smaller install, the base package provides quantization and adapters; `[train]` adds the trainer;
`[fast]` adds [grouped-nf4-gemm](https://github.com/pjordanandrsn/grouped-nf4-gemm).
Exact dependency floors live in [pyproject.toml](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/pyproject.toml).

## Load a model and check it

This downloads Qwen3-30B-A3B and keeps its frozen experts in host RAM. Allow enough RAM and disk space for the checkpoint.

```python
import torch
from experts4bit_qlora import load_moe_4bit_streaming, verify_moe_4bit

model, config = load_moe_4bit_streaming(
    "Qwen/Qwen3-30B-A3B", "cuda", torch.bfloat16,
    r=8, alpha=16, quant_type="nf4", offload=True,
)
verify_moe_4bit(model, strict=True)  # raises if an expert stack stayed high precision
```

To train the built-in Alpaca example:

```bash
MODEL=Qwen/Qwen3-30B-A3B OFFLOAD_EXPERTS=1 STEPS=150 R=8 TRAIN_EXPERTS=1 OUT=./out python -m experts4bit_qlora.train
```

For your own dataset and reusable native adapters, see [Loggetta's training guide](https://github.com/pjordanandrsn/loggetta/blob/main/docs/TRAINING.md).

## Choose the path you need

| Task | Start here |
| :--- | :--- |
| Plan before loading weights | [Loggetta](https://github.com/pjordanandrsn/loggetta) |
| Train fused MoE experts with QLoRA | [Training guide](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/solutions/qlora-fused-moe-experts.md) |
| Speed up training | `enable_fast_train(model, dgrad=True)`; assert its return count is positive |
| Fit experts in RAM or NVMe storage | [Offload guide](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/solutions/offload-moe-experts-to-cpu-or-nvme.md) |
| Fit the rest of the model | `enable_dense_offload` — freeze base weights before adding adapters |
| Serve on a consumer GPU | [Serving guide](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/solutions/serve-large-moe-on-a-consumer-gpu.md) |
| Work with native MXFP4 checkpoints | [MXFP4 guide](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/solutions/mxfp4-moe-training-and-residency.md) |

## What is measured

These are specific workloads, with public results and controls. The table is checked against
[claims.json](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/claims.json) in CI.
**Measured** means a public run record; the full register also labels private, superseded, and retired results.

| Workload and evidence | Result | Status |
| :--- | :--- | :--- |
| Qwen3-30B-A3B QLoRA · RTX 5090 · TC1's field recipe, both frameworks on one stack, e4b at every default (`e4b.train.h2h.unsloth.qwen3.5090.2026-10-08.field-device-ratio`) | Unsloth spends **1.92×** e4b's GPU time per step; wall-clock **2.80×** on an AMD EPYC 7713. Comparable held-out loss. Peak **26.16 vs 24.27 GB** (Unsloth lower). | measured |
| Packed 4,096-token Qwen3 training · RTX 5090 · e4b at its defaults (`e4b.train.h2h.unsloth.qwen3.5090.2026-10-08.packed-4k-device-ratio`) | Unsloth spends **1.160×** e4b's GPU time per step; the wall-clock ratio varies with the host (below). | measured |
| Expert-offload training on a 12 GB card (`e4b.offload.fits-30b-class`) | Qwen3-30B-A3B: **7.16 GB** peak; Gemma-4-26B-A4B: **8.47 GB**. | measured |

The speed rows profile both frameworks on one stack (torch 2.12.1). The GPU-time ratio is the host-independent figure; the
wall-clock ratio grows on a slower host, because Unsloth runs 14× e4b's CPU ops per step at the field recipe. On packed rows
the wall ratio read 1.287, 1.472 and 1.773 on three hosts. The earlier field-recipe readings, 2.352 and 2.468, were wall
ratios from before the reentrant checkpoint.
The offload row uses a different training setup.

[Qwen comparison](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/h2h-2026-10-02/tc1/RESULTS-tc1-pos69.md) · [Packed 4K result](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/h2h-2026-10-02/tc1/RESULTS-tc1-pos68.md) · [All current results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/STATUS.md)

<details>
<summary>How the two Unsloth rows were measured</summary>

- **Hardware:** one rented RTX 5090 per reading (a Vast.ai verified host). The field-recipe row ran on an AMD EPYC 7713
  host; the packed row on an AMD engineering-sample CPU with a 30.7-CPU quota.
- **Software:** torch 2.12.1+cu130 for both frameworks; Unsloth 2026.9.14; e4b and grouped-nf4-gemm at the commits in each
  run record.
- **Method:** matched fp32 adapters and initialization, two draws per arm, three profiled steps before the timed window.
  Validity and scoring by [`bench/tc1/tc1_reduce.py`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/tc1/tc1_reduce.py), under the
  [pre-registration](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/tc1/TC1-PREREG.md).
- **Rerun:** the [TC1 harness](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/tc1/README.md) on one RTX 5090, token `qwen3pos69` (field recipe) or `qwen3pos68`
  (packed rows).
- **Run records** (versions, host and every arm): [field recipe](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/h2h-2026-10-02/tc1/receipts/tc1-5090-137/RESULTS-tc1.md)
  · [packed rows](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/h2h-2026-10-02/tc1/receipts/tc1-5090-136/RESULTS-tc1.md)

</details>

## New in 0.54.0

- **Faster Qwen3-MoE prefill by default in `serve_paged`.** The prefill's norms and rotary run decode's fused kernels, and
  K19's prefill rows take the lean dispatch. On one RTX 5090 (Qwen3-30B-A3B int4), read separately, they take 4.0 ms and
  1.15 ms off a 38.2 ms 512-token prefill. The first changes prefill arithmetic, at parity on the quality instrument; the
  second is bit-identical. `E4B_FUSE_PREFILL_GLUE=0` / `E4B_PREFILL_LEAN_DISPATCH=0` restore the old path.
- **The training estimate prices the double-quantized absmax the run stores,** so it falls by 1.35 GB on Qwen3-30B-A3B.
- **EAGLE-3 speculative decode was measured and closed at B = 1:** a verify costs about twice a decode step, so it ran
  at ×0.76–0.92 of plain decode. `E4B_PAGED_SPEC` stays opt-in with no speed claim.
- **Offline loads of a pinned, complete cache no longer need the Hub,** and the offload loader fits a 12 GB card at
  Qwen3-235B-A22B shapes.

## Check the fit

- **Models:** use the [architecture support matrix](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/ARCHITECTURE_SUPPORT.md). Support and test coverage vary by family and execution path.
- **Memory:** CPU/NVMe offload trades transfer time and host storage for GPU memory. Measure your workload.
- **Speed:** 4-bit saves weight memory; it is not automatically faster or more energy-efficient than bf16 when both fit.
- **Serving:** vLLM was faster in the measured Qwen decode comparison. This runtime focuses on fitting and controlling low-bit MoEs; see the [serving results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/SERVING-THROUGHPUT.md).
- **Platforms:** CUDA execution is the tested path. ROCm, macOS, Windows and other non-CUDA GPU execution are outside this package's scope.

## Documentation and machine-readable data

[Task guides](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/SOLUTIONS.md) ·
[API and capability map](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/capabilities.json) ·
[Claims and evidence](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/claims.json) ·
[Package compatibility](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/system-manifest.json) ·
[Documentation index](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/INDEX.md) ·
[llms.txt](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/llms.txt) ·
[Contributing](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/AGENTS.md)

## License

MIT. The vendored bitsandbytes code is also MIT; see [third-party notices](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/THIRD_PARTY_NOTICES.md).
For integration or research work: [jordananderson.work](https://jordananderson.work).
