Metadata-Version: 2.4
Name: experts4bit-qlora
Version: 0.56.1
Summary: Fine-tune and serve large Mixture-of-Experts models on one GPU, with 4-bit experts in GPU memory, RAM or SSD.
Author-email: Jordan Anderson <paul.jordan.anderson@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://cerinamroth.com/ml/experts4bit-qlora/
Project-URL: Documentation, https://cerinamroth.com/ml/experts4bit-qlora/
Project-URL: Source, https://github.com/pjordanandrsn/experts4bit-qlora
Project-URL: Issues, https://github.com/pjordanandrsn/experts4bit-qlora/issues
Project-URL: Changelog, https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/CHANGELOG.md
Project-URL: Release Notes, https://github.com/pjordanandrsn/experts4bit-qlora/releases/latest
Project-URL: Status, https://cerinamroth.com/ml/status/
Project-URL: Capabilities, https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/capabilities.json
Project-URL: Solutions, https://cerinamroth.com/ml/solutions/
Project-URL: Benchmarks, https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/claims.json
Project-URL: Kernel: grouped-nf4-gemm, https://pypi.org/project/grouped-nf4-gemm/
Project-URL: Upstream (bitsandbytes#1965), https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1965
Project-URL: Tracking (bitsandbytes#1849), https://github.com/bitsandbytes-foundation/bitsandbytes/issues/1849
Keywords: mixture-of-experts,moe,qlora,lora,bitsandbytes,transformers,pytorch,quantization,nf4,mxfp4,int4,fp8,cuda,triton,gpu-offload,nvme,llm-inference,llm-training,fine-tuning,consumer-gpu
Classifier: Environment :: GPU :: NVIDIA CUDA
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: THIRD_PARTY_NOTICES.md
Requires-Dist: torch>=2.2
Requires-Dist: bitsandbytes>=0.43
Provides-Extra: fast
Requires-Dist: grouped-nf4-gemm>=0.43.0; extra == "fast"
Provides-Extra: train
Requires-Dist: transformers>=5.0; extra == "train"
Requires-Dist: datasets>=2.14; extra == "train"
Requires-Dist: accelerate>=0.30; extra == "train"
Requires-Dist: safetensors>=0.4; extra == "train"
Requires-Dist: huggingface_hub>=0.23; extra == "train"
Provides-Extra: serve
Requires-Dist: fastapi>=0.110; extra == "serve"
Requires-Dist: uvicorn>=0.29; extra == "serve"
Requires-Dist: pydantic>=2; extra == "serve"
Requires-Dist: transformers>=5.0; extra == "serve"
Requires-Dist: accelerate>=0.30; extra == "serve"
Requires-Dist: safetensors>=0.4; extra == "serve"
Requires-Dist: huggingface_hub>=0.23; extra == "serve"
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Requires-Dist: transformers>=5.0; extra == "test"
Requires-Dist: accelerate>=0.30; extra == "test"
Requires-Dist: safetensors>=0.4; extra == "test"
Requires-Dist: fastapi>=0.110; extra == "test"
Requires-Dist: httpx>=0.27; extra == "test"
Requires-Dist: grouped-nf4-gemm>=0.43.0; extra == "test"
Requires-Dist: jsonschema>=4.18; extra == "test"
Requires-Dist: compressed-tensors==0.18.0; extra == "test"
Dynamic: license-file

# experts4bit-qlora

### Fine-tune large MoE models on a single GPU.

[![PyPI](https://img.shields.io/pypi/v/experts4bit-qlora)](https://pypi.org/project/experts4bit-qlora/)
[![CI](https://github.com/pjordanandrsn/experts4bit-qlora/actions/workflows/ci.yml/badge.svg)](https://github.com/pjordanandrsn/experts4bit-qlora/actions/workflows/ci.yml)

**Quantize the experts. Train adapters. Move weights to RAM or SSD when GPU memory runs out.**

**On a 12 GB card:** with `OFFLOAD_EXPERTS=1`, Qwen3-30B-A3B QLoRA-trains TC1's matched set (fp32 adapters, seq 2048,
r 16) at micro-batch 1 × accum 8 at 10.463 GB peak allocated on an RTX A2000 12 GB, while the rest of its packed experts
stay in pinned CPU RAM (`e4b.train.frontier.qwen3.a2000-12gb.2026-10-02`, one draw; micro-batch 2 ran out of memory on
that shared card). [The command](#load-a-model-and-check-it) ·
[how to read the number](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/STORAGE-MODES.md#reading-the-headline-memory-numbers)

Mixture-of-Experts (MoE) models keep many expert networks but activate only a few per token.
Their fused expert weights can escape ordinary `load_in_4bit=True` loading and stay in full precision.
This runtime loads those weights in 4-bit, trains them with QLoRA, and serves the model.

**New here? [Start with Loggetta](https://github.com/pjordanandrsn/loggetta).**
`pip install loggetta` installs this runtime and its kernels. Use `loggetta inspect` to check your machine
and `loggetta plan` to plan supported training.
Use this package directly when you need control over loading, training, serving, or memory placement.

> **Limits.** GPU execution needs Linux and NVIDIA CUDA; the fast kernels need an sm_80+ GPU. Support varies by model
> family and execution path. 4-bit saves memory but is not automatically faster than bf16, and vLLM was faster in the
> measured Qwen decode comparison. [Check the fit](#check-the-fit).

[Install](#install) · [Results](#what-is-measured) · [Model support](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/ARCHITECTURE_SUPPORT.md) · [Task guides](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/SOLUTIONS.md) · [Hugging Face](https://huggingface.co/spaces/pjordanandrsn/research)

<!-- release-block:start -->
<!-- generated by `python scripts/check_readme_claims.py --write-release-block` from CHANGELOG.md's latest release heading; do not edit by hand -->
**Latest released package:** [`experts4bit-qlora` 0.56.1](https://pypi.org/project/experts4bit-qlora/0.56.1/) · **Current development status:** [`docs/STATUS.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/STATUS.md) on `main` (this README describes `main`) · **Released documentation for 0.56.1:** [`docs/`](https://github.com/pjordanandrsn/experts4bit-qlora/tree/v0.56.1/docs) · [`README.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/v0.56.1/README.md) · [`CHANGELOG.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/v0.56.1/CHANGELOG.md)
<!-- release-block:end -->

## Install

```bash
pip install "experts4bit-qlora[train,fast]"
```

GPU execution needs **Linux + NVIDIA CUDA**. Fast kernels need an **sm_80+ GPU**,
Triton ≥ 3.4; the streaming loader needs transformers ≥ 5.0. The base runtime needs torch ≥ 2.2;
the fast extra applies the kernel package’s newer PyTorch floor.
CI tests Python 3.11. Start with a CUDA-compatible PyTorch installation for your GPU.

For a smaller install, the base package provides quantization and adapters; `[train]` adds the trainer;
`[fast]` adds [grouped-nf4-gemm](https://github.com/pjordanandrsn/grouped-nf4-gemm).
Exact dependency floors live in [pyproject.toml](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/pyproject.toml).

## Load a model and check it

This downloads Qwen3-30B-A3B and keeps its frozen experts in host RAM. Allow enough RAM and disk space for the checkpoint.

```python
import torch
from experts4bit_qlora import load_moe_4bit_streaming, verify_moe_4bit

model, config = load_moe_4bit_streaming(
    "Qwen/Qwen3-30B-A3B", "cuda", torch.bfloat16,
    r=8, alpha=16, quant_type="nf4", offload=True,
)
verify_moe_4bit(model, strict=True)  # raises if an expert stack stayed high precision
```

To train the built-in Alpaca example:

```bash
MODEL=Qwen/Qwen3-30B-A3B OFFLOAD_EXPERTS=1 STEPS=150 R=8 TRAIN_EXPERTS=1 OUT=./out python -m experts4bit_qlora.train
```

For your own dataset and reusable native adapters, see [Loggetta's training guide](https://github.com/pjordanandrsn/loggetta/blob/main/docs/TRAINING.md).

## Choose the path you need

| Task | Start here |
| :--- | :--- |
| Plan before loading weights | [Loggetta](https://github.com/pjordanandrsn/loggetta) |
| Train fused MoE experts with QLoRA | [Training guide](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/solutions/qlora-fused-moe-experts.md) |
| Speed up training | `enable_fast_train(model, dgrad=True)`; assert its return count is positive |
| Fit experts in RAM or NVMe storage | [Offload guide](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/solutions/offload-moe-experts-to-cpu-or-nvme.md) |
| Fit the rest of the model | `enable_dense_offload` — freeze base weights before adding adapters |
| Serve on a consumer GPU | [Serving guide](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/solutions/serve-large-moe-on-a-consumer-gpu.md) |
| Work with native MXFP4 checkpoints | [MXFP4 guide](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/solutions/mxfp4-moe-training-and-residency.md) |

## What is measured

These are specific workloads, with public results and controls. The table is checked against
[claims.json](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/claims.json) in CI.
**Measured** means a public run record; the full register also labels private, superseded, and retired results.

| Workload and evidence | Result | Status |
| :--- | :--- | :--- |
| Qwen3-30B-A3B QLoRA · RTX 5090 · TC1's field recipe, both frameworks on one stack, e4b at every default (`e4b.train.h2h.unsloth.qwen3.5090.2026-10-08.field-device-ratio`) | Unsloth spends **1.92×** e4b's GPU time per step; wall-clock **2.80×** on an AMD EPYC 7713. Comparable held-out loss. Peak **26.16 vs 24.27 GB** (Unsloth lower). | measured |
| Packed 4,096-token Qwen3 training · RTX 5090 · e4b at its defaults (`e4b.train.h2h.unsloth.qwen3.5090.2026-10-08.packed-4k-device-ratio`) | Unsloth spends **1.160×** e4b's GPU time per step; the wall-clock ratio varies with the host (below). | measured |
| Expert-offload training on a 12 GB card (`e4b.train.frontier.qwen3.a2000-12gb.2026-10-02`) | Qwen3-30B-A3B, TC1's matched set (fp32 adapters) at micro-batch 1 × accum 8: **10.463 GB** peak allocated on an RTX A2000 12 GB, held-out 1.9533 → 0.8483 at N=20. An older, lighter reading gave 7.16 GB, and 8.47 GB for Gemma-4-26B-A4B (`e4b.offload.fits-30b-class`); it was not reproduced ([how to read these](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/STORAGE-MODES.md#reading-the-headline-memory-numbers)). | measured |

The speed rows profile both frameworks on one stack (torch 2.12.1). The GPU-time ratio is the host-independent figure; the
wall-clock ratio grows on a slower host, because Unsloth runs 14× e4b's CPU ops per step at the field recipe. On packed rows
the wall ratio read 1.472 (`e4b.train.h2h.unsloth.qwen3.5090.2026-10-08.packed-4k-device-ratio`) and 1.773
(`e4b.train.memory.packed-4k-position-defaults.5090.2026-10-07`) on two hosts. The earlier field-recipe readings, 2.352 and 2.468, were wall
ratios from before the reentrant checkpoint.
The offload row uses a different training setup.

[Qwen comparison](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/h2h-2026-10-02/tc1/RESULTS-tc1-pos69.md) · [Packed 4K result](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/h2h-2026-10-02/tc1/RESULTS-tc1-pos68.md) · [All current results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/STATUS.md)

<details>
<summary>How the two Unsloth rows were measured</summary>

- **Hardware:** one rented RTX 5090 per reading (a Vast.ai verified host). The field-recipe row ran on an AMD EPYC 7713
  host; the packed row on an AMD engineering-sample CPU with a 30.7-CPU quota.
- **Software:** torch 2.12.1+cu130 for both frameworks; Unsloth 2026.9.14; e4b and grouped-nf4-gemm at the commits in each
  run record.
- **Method:** matched fp32 adapters and initialization, two draws per arm, three profiled steps before the timed window.
  Validity and scoring by [`bench/tc1/tc1_reduce.py`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/tc1/tc1_reduce.py), under the
  [pre-registration](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/tc1/TC1-PREREG.md).
- **Rerun:** the [TC1 harness](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/tc1/README.md) on one RTX 5090, token `qwen3pos69` (field recipe) or `qwen3pos68`
  (packed rows).
- **Run records** (versions, host and every arm): [field recipe](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/h2h-2026-10-02/tc1/receipts/tc1-5090-137/RESULTS-tc1.md)
  · [packed rows](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/h2h-2026-10-02/tc1/receipts/tc1-5090-136/RESULTS-tc1.md)

</details>

## New in 0.56.1

- **Sixteen requests decode 1.4654× as fast on Qwen3-30B-A3B, one request unchanged, every token identical.** K25, the
  batched NF4 decode GEMM, now runs at tile plan `c1` on `qwen3_moe` where its decode resolved to K39's `prmt`
  (grouped-nf4-gemm 0.50.0 or later, on NVIDIA). Elsewhere it stays on `today`, recorded in `info["k25_plan"]`. The
  plan changes tiling and staging only, so the values are bit-identical to `today`'s. Measured on one RTX 5090 against
  the `today` plan, both under K39's decode: the smallest of four ABBA pairs. `E4B_K25_PLAN=today` opts out.
  ([P161](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/p161/RESULTS-p161.md), `e4b.serve.p161.k25-plan-c1.qwen3.5090.2026-10-20`)

## New in 0.56.0

Serving defaults on `qwen3_moe`. Each was measured alone on Qwen3-30B-A3B on one RTX 5090 (170 SMs); their combined
effect has not been read. Two need a newer grouped-nf4-gemm than `[fast]` installs (0.43.0 or later): K25's prmt decode
needs 0.50.0 and the int8 head needs 0.48.0. Below those each resolves off, recorded, and nothing breaks. To get them,
run `pip install -U grouped-nf4-gemm`.

- **Sixteen requests decode 1.0925× as fast, one request unchanged, every token identical.** K25, the batched NF4 decode
  GEMM, now decodes its codebook with grouped-nf4-gemm K39's byte-permute decode (`lut="prmt"`), bit-identical to the
  select tree it replaces. It needs grouped-nf4-gemm 0.50.0 or later and an NVIDIA device; otherwise it resolves to the
  tree, recorded in `info["k25_lut"]`. `E4B_K25_LUT=tree` opts out.
  ([P158](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/p158/RESULTS-p158.md), `e4b.serve.p158.k25-prmt.qwen3.5090.2026-10-20`)
- **One request decodes 1.052× as fast from the lm_head on its per-row int8 grid** (grouped-nf4-gemm K36; at
  Qwen3-30B-A3B's 151936 x 2048 head only; needs grouped-nf4-gemm 0.48.0 or later, else it resolves off). **Outputs
  change:** the logits differ from the bf16 head's, inside the quality gates read on Qwen3-30B-A3B @ad44e777. Sixteen
  requests read 0.992, a cost not yet attributed. `E4B_INT8_HEAD_BW=0` restores the bf16 head.
  ([P153](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/p153/RESULTS-p153.md), `e4b.serve.p153.int8-head-bw.qwen3.5090.2026-10-19`)
- **One request decodes 1.044× as fast with the paged FP8 attention decode capped at 16 splits** (buckets 1, 2 and 4
  move; bitwise-identical to the library's plan at contexts up to 1024). Sixteen requests are unchanged.
  `E4B_ATTN_SPLIT_CAP=0` opts out. ([P143](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/p143/RESULTS-p143.md),
  `e4b.serve.p143.attn-split-cap.qwen3.5090.2026-10-19`)

Training, opt-in. Each was measured on the card named, against the baseline its bullet names.

- **Layer-major training under expert offload steps 2.365× as fast on Qwen3-30B-A3B on one RTX 4090.** `layer_major_step`
  (`experts4bit_qlora.engines.layer_major`) runs a step's micro-batches layer by layer, so each offloaded layer is staged
  once per step instead of once per micro-batch. It replicates at 2.311× on an RTX 5090 and holds on Mixtral-8x7B at
  2.118× on an RTX 4090. Opt-in. ([P137](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/p137/RESULTS-p137.md),
  `e4b.train.p137.layer-major-offload.qwen3.4090.2026-10-18`; [P140](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/p140/RESULTS-p140.md),
  `e4b.train.p140.layer-major-offload.qwen3.5090.2026-10-18`; [P144](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/p144/RESULTS-p144.md),
  `e4b.train.p144.layer-major-offload.mixtral.4090.2026-10-18`)
- **The CLI trainer can take the fused training path.** In 0.55.0, `python -m experts4bit_qlora.train` always ran the
  reference loop. `FAST_TRAIN=1` now puts it on the fused path, and `LAYER_MAJOR=1` implies it. At the CLI's own default
  recipe on Qwen3-30B-A3B with `OFFLOAD_EXPERTS=1`, `LAYER_MAJOR=1` steps 1.455× as fast as `FAST_TRAIN=1` alone on one
  RTX 4090. An explicit `FAST_TRAIN=0` keeps the reference loop.
  ([P146](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/p146/RESULTS-p146.md), `e4b.train.p146.layer-major-cli.qwen3.4090.2026-10-18`)

Mixtral serving.

- **Mixtral-8x7B decodes with its B=1 fold stack by default.** The T1 glue, its second round and the router epilogue now
  resolve to `auto` on Mixtral; fused q/k/v does not engage on this family. A one-request step takes 0.9393 of
  its time without them (about 20.2-20.5 ms to 19.0 ms) on one RTX 5090. Opt out per knob: `E4B_FUSE_T1_GLUE=0`,
  `E4B_FUSE_T1_GLUE_R2=0`, `E4B_FUSE_ROUTER_EPI=0`.
  ([P135](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/p135/RESULTS-p135.md), `e4b.serve.p135.fold-speed.mixtral.5090.2026-10-18`)
- **`E4B_SERVE_ATTN_INT8=1`: int8 weight-only attention projections on Mixtral, opt-in.** q/k/v/o are held in int8
  with a per-row fp32 scale, and decode calls of 1-16 rows run grouped-nf4-gemm's K37 kernels (from 0.49.0). On one
  RTX 5090 the served read decodes one request 1.161x as fast and sixteen 1.018x, with decode quality inside its
  registered bound. It stays opt-in until a routing-matched read: the default-flip PR is held.
  ([P149](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/p149/RESULTS-p149.md),
  `e4b.serve.p149.int8-attn.mixtral.5090.2026-10-20`)

Scale.

- **One full-depth QLoRA optimizer step on Qwen3-235B-A22B on one GPU.** All 94 decoder layers and every routed expert,
  two optimizer steps plus a fresh-process repeat of step 0, on one RTX PRO 6000 Blackwell Workstation Edition.
  ([Q235](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/q235/RESULTS-q235.md), run `q235-pro6000-8`, MILESTONE PASS)

## Check the fit

- **Models:** use the [architecture support matrix](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/ARCHITECTURE_SUPPORT.md). Support and test coverage vary by family and execution path.
- **Memory:** CPU/NVMe offload trades transfer time and host storage for GPU memory. Measure your workload.
- **Speed:** 4-bit saves weight memory; it is not automatically faster or more energy-efficient than bf16 when both fit.
- **Serving:** vLLM was faster in the measured Qwen decode comparison. This runtime focuses on fitting and controlling low-bit MoEs; see the [serving results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/SERVING-THROUGHPUT.md#throughput-census-bo7-measured-in-repo).
- **Platforms:** CUDA execution is the tested path. ROCm, macOS, Windows and other non-CUDA GPU execution are outside this package's scope.

## Documentation and machine-readable data

[Task guides](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/SOLUTIONS.md) ·
[API and capability map](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/capabilities.json) ·
[Claims and evidence](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/claims.json) ·
[Package compatibility](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/system-manifest.json) ·
[Documentation index](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/INDEX.md) ·
[llms.txt](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/llms.txt) ·
[Contributing](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/AGENTS.md)

## License

MIT. The vendored bitsandbytes code is also MIT; see [third-party notices](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/THIRD_PARTY_NOTICES.md).
For integration or research work: [jordananderson.work](https://jordananderson.work).
