Metadata-Version: 2.4
Name: experts4bit-qlora
Version: 0.58.0
Summary: Fine-tune and serve large Mixture-of-Experts models on one GPU, with 4-bit experts in GPU memory, RAM or SSD.
Author-email: Jordan Anderson <paul.jordan.anderson@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://cerinamroth.com/ml/experts4bit-qlora/
Project-URL: Documentation, https://cerinamroth.com/ml/experts4bit-qlora/
Project-URL: Source, https://github.com/pjordanandrsn/experts4bit-qlora
Project-URL: Issues, https://github.com/pjordanandrsn/experts4bit-qlora/issues
Project-URL: Changelog, https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/CHANGELOG.md
Project-URL: Release Notes, https://github.com/pjordanandrsn/experts4bit-qlora/releases/latest
Project-URL: Status, https://cerinamroth.com/ml/status/
Project-URL: Capabilities, https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/capabilities.json
Project-URL: Solutions, https://cerinamroth.com/ml/solutions/
Project-URL: Benchmarks, https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/claims.json
Project-URL: Kernel: grouped-nf4-gemm, https://pypi.org/project/grouped-nf4-gemm/
Project-URL: Upstream (bitsandbytes#1965), https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1965
Project-URL: Tracking (bitsandbytes#1849), https://github.com/bitsandbytes-foundation/bitsandbytes/issues/1849
Keywords: mixture-of-experts,moe,qlora,lora,bitsandbytes,transformers,pytorch,quantization,nf4,mxfp4,int4,fp8,cuda,triton,gpu-offload,nvme,llm-inference,llm-training,fine-tuning,consumer-gpu
Classifier: Environment :: GPU :: NVIDIA CUDA
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: THIRD_PARTY_NOTICES.md
Requires-Dist: torch>=2.2
Requires-Dist: bitsandbytes>=0.43
Provides-Extra: fast
Requires-Dist: grouped-nf4-gemm>=0.43.0; extra == "fast"
Provides-Extra: train
Requires-Dist: transformers>=5.0; extra == "train"
Requires-Dist: datasets>=2.14; extra == "train"
Requires-Dist: accelerate>=0.30; extra == "train"
Requires-Dist: safetensors>=0.4; extra == "train"
Requires-Dist: huggingface_hub>=0.23; extra == "train"
Provides-Extra: serve
Requires-Dist: fastapi>=0.110; extra == "serve"
Requires-Dist: uvicorn>=0.29; extra == "serve"
Requires-Dist: pydantic>=2; extra == "serve"
Requires-Dist: transformers>=5.0; extra == "serve"
Requires-Dist: accelerate>=0.30; extra == "serve"
Requires-Dist: safetensors>=0.4; extra == "serve"
Requires-Dist: huggingface_hub>=0.23; extra == "serve"
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Requires-Dist: transformers>=5.0; extra == "test"
Requires-Dist: accelerate>=0.30; extra == "test"
Requires-Dist: safetensors>=0.4; extra == "test"
Requires-Dist: fastapi>=0.110; extra == "test"
Requires-Dist: httpx>=0.27; extra == "test"
Requires-Dist: grouped-nf4-gemm>=0.43.0; extra == "test"
Requires-Dist: jsonschema>=4.18; extra == "test"
Requires-Dist: compressed-tensors==0.18.0; extra == "test"
Dynamic: license-file

# experts4bit-qlora

### Fine-tune large MoE models on a single GPU.

[![PyPI](https://img.shields.io/pypi/v/experts4bit-qlora)](https://pypi.org/project/experts4bit-qlora/)
[![CI](https://github.com/pjordanandrsn/experts4bit-qlora/actions/workflows/ci.yml/badge.svg)](https://github.com/pjordanandrsn/experts4bit-qlora/actions/workflows/ci.yml)

**Quantize the experts. Train adapters. Move weights to RAM or SSD when GPU memory runs out.**

**On a 12 GB card:** with `OFFLOAD_EXPERTS=1`, Qwen3-30B-A3B QLoRA-trains TC1's matched set (fp32 adapters, seq 2048,
r 16) at micro-batch 1 × accum 8 at 10.463 GB peak allocated on an RTX A2000 12 GB, while the rest of its packed experts
stay in pinned CPU RAM (`e4b.train.frontier.qwen3.a2000-12gb.2026-10-02`, one draw; micro-batch 2 ran out of memory on
that shared card). [The command](#load-a-model-and-check-it) ·
[how to read the number](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/STORAGE-MODES.md#reading-the-headline-memory-numbers)

Mixture-of-Experts (MoE) models keep many expert networks but activate only a few per token.
Their fused expert weights can escape ordinary `load_in_4bit=True` loading and stay in full precision.
This runtime loads those weights in 4-bit, trains them with QLoRA, and serves the model.

**New here? [Start with Loggetta](https://github.com/pjordanandrsn/loggetta).**
`pip install loggetta` installs this runtime and its kernels. Use `loggetta inspect` to check your machine
and `loggetta plan` to plan supported training.
Use this package directly when you need control over loading, training, serving, or memory placement.

> **Limits.** GPU execution needs Linux and NVIDIA CUDA; the fast kernels need an sm_80+ GPU. Every measured training
> and serving result uses one GPU per process; running one model across two or more GPUs is untested. Support varies by
> model family and execution path. 4-bit saves memory but is not automatically faster than bf16, and vLLM was faster in the measured Qwen
> decode comparison. [Check the fit](#check-the-fit).

[Install](#install) · [Results](#what-is-measured) · [Model support](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/ARCHITECTURE_SUPPORT.md) · [Task guides](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/SOLUTIONS.md) · [Hugging Face](https://huggingface.co/spaces/pjordanandrsn/research)

<!-- release-block:start -->
<!-- generated by `python scripts/check_readme_claims.py --write-release-block` from CHANGELOG.md's latest release heading; do not edit by hand -->
**Latest released package:** [`experts4bit-qlora` 0.58.0](https://pypi.org/project/experts4bit-qlora/0.58.0/) · **Current development status:** [`docs/STATUS.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/STATUS.md) on `main` (this README describes `main`) · **Released documentation for 0.58.0:** [`docs/`](https://github.com/pjordanandrsn/experts4bit-qlora/tree/v0.58.0/docs) · [`README.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/v0.58.0/README.md) · [`CHANGELOG.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/v0.58.0/CHANGELOG.md)
<!-- release-block:end -->

## Install

```bash
pip install "experts4bit-qlora[train,fast]"
```

GPU execution needs **Linux + NVIDIA CUDA**. Fast kernels need an **sm_80+ GPU**,
Triton ≥ 3.4; the streaming loader needs transformers ≥ 5.0. The base runtime needs torch ≥ 2.2;
the fast extra applies the kernel package’s newer PyTorch floor.
CI tests Python 3.11. Start with a CUDA-compatible PyTorch installation for your GPU.

For a smaller install, the base package provides quantization and adapters; `[train]` adds the trainer;
`[fast]` adds [grouped-nf4-gemm](https://github.com/pjordanandrsn/grouped-nf4-gemm).
Exact dependency floors live in [pyproject.toml](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/pyproject.toml).

## Load a model and check it

This downloads Qwen3-30B-A3B and keeps its frozen experts in host RAM. Allow enough RAM and disk space for the checkpoint.

```python
import torch
from experts4bit_qlora import load_moe_4bit_streaming, verify_moe_4bit

model, config = load_moe_4bit_streaming(
    "Qwen/Qwen3-30B-A3B", "cuda", torch.bfloat16,
    r=8, alpha=16, quant_type="nf4", offload=True,
)
verify_moe_4bit(model, strict=True)  # raises if an expert stack stayed high precision
```

To train the built-in Alpaca example:

```bash
MODEL=Qwen/Qwen3-30B-A3B OFFLOAD_EXPERTS=1 STEPS=150 R=8 TRAIN_EXPERTS=1 OUT=./out python -m experts4bit_qlora.train
```

For your own dataset and reusable native adapters, see [Loggetta's training guide](https://github.com/pjordanandrsn/loggetta/blob/main/docs/TRAINING.md).

## Choose the path you need

| Task | Start here |
| :--- | :--- |
| Plan before loading weights | [Loggetta](https://github.com/pjordanandrsn/loggetta) |
| Train fused MoE experts with QLoRA | [Training guide](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/solutions/qlora-fused-moe-experts.md) |
| Speed up training | `enable_fast_train(model, dgrad=True)`; assert its return count is positive |
| Fit experts in RAM or NVMe storage | [Offload guide](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/solutions/offload-moe-experts-to-cpu-or-nvme.md) |
| Fit the rest of the model | `enable_dense_offload` — freeze base weights before adding adapters |
| Serve on a consumer GPU | [Serving guide](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/solutions/serve-large-moe-on-a-consumer-gpu.md) |
| Work with native MXFP4 checkpoints | [MXFP4 guide](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/solutions/mxfp4-moe-training-and-residency.md) |

## What is measured

These are specific workloads, with public results and controls. The table is checked against
[claims.json](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/claims.json) in CI.
**Measured** means a public run record; the full register also labels private, superseded, and retired results.

| Workload and evidence | Result | Status |
| :--- | :--- | :--- |
| Qwen3-30B-A3B QLoRA · RTX 5090 · TC1's field recipe, both frameworks on one stack, e4b at every default (`e4b.train.h2h.unsloth.qwen3.5090.2026-10-08.field-device-ratio`) | Unsloth spends **1.92×** e4b's GPU time per step (medians of two draws); wall-clock **2.80×** on an AMD EPYC 7713. Comparable held-out loss. Peak **26.16 vs 24.27 GB** (Unsloth lower). | measured |
| Packed 4,096-token Qwen3 training · RTX 5090 · e4b at its defaults (`e4b.train.h2h.unsloth.qwen3.5090.2026-10-08.packed-4k-device-ratio`) | Unsloth spends **1.160×** e4b's GPU time per step (medians of two draws); the wall-clock ratio varies with the host (below). | measured |
| Expert-offload training on a 12 GB card (`e4b.train.frontier.qwen3.a2000-12gb.2026-10-02`) | Qwen3-30B-A3B, TC1's matched set (fp32 adapters) at micro-batch 1 × accum 8: **10.463 GB** peak allocated on an RTX A2000 12 GB, held-out 1.9533 → 0.8483 at N=20. An older, lighter reading gave 7.16 GB, and 8.47 GB for Gemma-4-26B-A4B (`e4b.offload.fits-30b-class`); it was not reproduced ([how to read these](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/STORAGE-MODES.md#reading-the-headline-memory-numbers)). | measured |

The speed rows profile both frameworks on one stack (torch 2.12.1). The GPU-time ratio is the host-independent figure; the
wall-clock ratio grows on a slower host, because Unsloth runs 14× e4b's CPU ops per step at the field recipe. On packed rows
the wall ratio read 1.472 (`e4b.train.h2h.unsloth.qwen3.5090.2026-10-08.packed-4k-device-ratio`) and 1.773
(`e4b.train.memory.packed-4k-position-defaults.5090.2026-10-07`) on two hosts. The earlier field-recipe readings, 2.352 and 2.468, were wall
ratios from before the reentrant checkpoint.
The offload row uses a different training setup.

[Qwen comparison](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/h2h-2026-10-02/tc1/RESULTS-tc1-pos69.md) · [Packed 4K result](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/h2h-2026-10-02/tc1/RESULTS-tc1-pos68.md) · [All current results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/STATUS.md)

<details>
<summary>How the two Unsloth rows were measured</summary>

- **Hardware:** one rented RTX 5090 per reading (a Vast.ai verified host). The field-recipe row ran on an AMD EPYC 7713
  host; the packed row on an AMD engineering-sample CPU with a 30.7-CPU quota.
- **Software:** torch 2.12.1+cu130 for both frameworks; Unsloth 2026.9.14; e4b and grouped-nf4-gemm at the commits in each
  run record.
- **Method:** matched fp32 adapters and initialization, two draws per arm, three profiled steps before the timed window.
  Validity and scoring by [`bench/tc1/tc1_reduce.py`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/tc1/tc1_reduce.py), under the
  [pre-registration](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/tc1/TC1-PREREG.md).
- **Rerun:** the [TC1 harness](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/tc1/README.md) on one RTX 5090, token `qwen3pos69` (field recipe) or `qwen3pos68`
  (packed rows).
- **Run records** (versions, host and every arm): [field recipe](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/h2h-2026-10-02/tc1/receipts/tc1-5090-137/RESULTS-tc1.md)
  · [packed rows](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/h2h-2026-10-02/tc1/receipts/tc1-5090-136/RESULTS-tc1.md)

</details>

## New in 0.58.0

- **Qwen3-30B-A3B's int8 lm_head serves 2 to 8 requests by default.** With the int8 head on (its default at the
  151936 x 2048 head), the head's 2- to 8-row calls run grouped-nf4-gemm K37's `linear_i8` over the head's own int8
  codes instead of `F.linear` on their bf16 rounding: 2, 4 and 8 requests decode 1.0311x, 1.0243x and 1.0165x as fast
  (the smallest of four ABBA pairs at each, inside P153's quality bars;
  `e4b.serve.p232.int8-head-rows.qwen3.5090.2026-10-24`). **Outputs change by default** at those request counts:
  greedy text can differ (76 of 112 greedy rows matched in P232); one request
  and 9 or more are unchanged, and `E4B_INT8_HEAD_ROWS=1` restores the previous output. Other heads and families stay
  at 1 until a read of their own.
- **TC1's field recipe fits a 24 GiB RTX 4090** with three opt-in levers (`NF4_QLORA_COMPACT_DELTA=1`,
  `NF4_QLORA_PAD_BUCKETS=1`, `E4B_CHUNKED_LM_LOSS=256`) and grouped-nf4-gemm 0.51.0: two RTX 4090s on two machines
  trained all 60 steps with no out-of-memory error, 586 MiB of the card's total left at the fullest step
  (`e4b.train.4090-fit.field.4090.2026-10-24`). FIT covers that fixture only (Qwen3-30B-A3B, sequence 2048, micro-batch
  2 x accumulation 4, r 16); the levers stay opt-in, and no speed is claimed.
- **A served adapter's base is verified, not reported unverified.** The paged server compares each qfp-v1 fingerprint
  a `backend-adapter/2` manifest records with the base it serves: the attention projections as loaded, and the expert
  stacks read from the served arena. A match serves with no opt-in; an expert mismatch is refusal 12
  (`E4B_PAGED_ADAPTER_ACCEPT_BASE_MISMATCH=experts` serves it anyway, reported in `/health`), and an attention mismatch
  is refusal 11, with no override.
- **A merged NF4 arena, opt-in.** `experts4bit_qlora.merged_arena.write` bakes a trained expert delta offline into a
  new NF4 arena, and `E4B_PAGED_MERGED_ARENA=1` serves it with the adapter it was merged from; any other pairing is
  refusal 13. It stays off by default until its registered served read gates it.
- **A shared KV pool, opt-in.** With `E4B_PAGED_KV_POOL=shared`, slots claim KV blocks as they write them, the
  scheduler admits by reserved blocks, and the pool is sized from the device's free memory, so a request waits for
  blocks rather than running the pool dry. `/health` reports the sizing. No speed is claimed.
- **Small-batch serving levers, each opt-in until a served read of its own:** `E4B_NF4_SMALLM_BW` (2- to 8-token
  steps on the bandwidth GEMV), `E4B_DENSE_LT_PIN` (a pinned cuBLASLt GEMM for named dense projections at 2 to 8 rows),
  `E4B_K25_PLAN=m1` and `E4B_ATTN_SPLIT_FIXED`; also `E4B_PAGED_PREFILL_EVERY` and
  `E4B_PAGED_LOADED_CHUNK_TOKENS`, which space and shorten prefill while requests decode.
- **Deterministic training, opt-in.** `E4B_TRAIN_DETERMINISTIC=1` turns on the deterministic token gather and
  deterministic algorithms for each backward, and refuses at trainer start unless `CUBLAS_WORKSPACE_CONFIG` is
  `:4096:8` or `:16:8`. No default changes, and no cost is stated.

## New in 0.57.0

- **The paged server serves a trained adapter.** `E4B_PAGED_ADAPTER=<dir>` loads one adapter trained with bf16
  attention (one trained on 4-bit attention is refused). The server
  verifies the directory before it reads any weight (paths, sizes, sha256, tensors, shapes, dtypes, finiteness), merges
  the attention LoRA into the bf16 projections at load, so serving it adds no per-token work, and refuses to start,
  naming each reason, rather than serve the base model in the adapter's place. Expert LoRA is refused for now. `/health`
  reports the adapter's id, manifest hash and merge report.
- **An adapter can carry a manifest the server checks.** `backend-adapter/2` records a canonical fingerprint of the
  quantized base (scheme `qfp-v1`: the same base fingerprints alike whether its NF4 absmax is stored nested or
  expanded), the training facts that decide whether serving is exact, and the LoRA scale actually applied; the server
  checks every field and refuses by name a manifest it cannot accept. Comparing the expert base's fingerprint at serve
  time is the next step; until then that check stays an explicit opt-in refusal.
  [`docs/ADAPTER_PORTABILITY.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/ADAPTER_PORTABILITY.md)
  states, from code, which training and serving pairings are exact, bounded behind a registered gate, or refused.
- **gpt-oss expert LoRA, opt-in.** `E4B_GPTOSS_EXPERT_LORA=1` wraps gpt-oss's experts for LoRA on the unfused
  reference loop, graded against an independent oracle on CPU. It refuses `offload`, and its speed against a fused
  route is unmeasured.
- **The plan prices the loss the run takes.** When the chunked LM loss refuses an LM head whose width differs from the
  configured vocabulary, the planner now prices the stock loss, as the run does.

## New in 0.56.1

- **Sixteen requests decode 1.4654× as fast on Qwen3-30B-A3B, one request unchanged, every token identical** (the
  smallest of four ABBA pairs). K25, the batched NF4 decode GEMM, now runs at tile plan `c1` on `qwen3_moe` where its
  decode resolved to K39's `prmt` (grouped-nf4-gemm 0.50.0 or later, on NVIDIA). Elsewhere it stays on `today`, recorded
  in `info["k25_plan"]`. The plan changes tiling and staging only, so the values are bit-identical to `today`'s.
  Measured on one RTX 5090 against the `today` plan, both under K39's decode: the smallest of four ABBA pairs.
  `E4B_K25_PLAN=today` opts out.
  ([P161](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/p161/RESULTS-p161.md), `e4b.serve.p161.k25-plan-c1.qwen3.5090.2026-10-20`)

## New in 0.56.0

Serving defaults on `qwen3_moe`. Each was measured alone on Qwen3-30B-A3B on one RTX 5090 (170 SMs); their combined
effect has not been read. Two need a newer grouped-nf4-gemm than `[fast]` installs (0.43.0 or later): K25's prmt decode
needs 0.50.0 and the int8 head needs 0.48.0. Below those each resolves off, recorded, and nothing breaks. To get them,
run `pip install -U grouped-nf4-gemm`.

- **Sixteen requests decode 1.0925× as fast, one request unchanged, every token identical.** (The smallest of four ABBA
  pairs.) K25, the batched NF4 decode GEMM, now decodes its codebook with grouped-nf4-gemm K39's byte-permute decode
  (`lut="prmt"`), bit-identical to the select tree it replaces. It needs grouped-nf4-gemm 0.50.0 or later and an NVIDIA
  device; otherwise it resolves to the tree, recorded in `info["k25_lut"]`. `E4B_K25_LUT=tree` opts out.
  ([P158](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/p158/RESULTS-p158.md), `e4b.serve.p158.k25-prmt.qwen3.5090.2026-10-20`)
- **One request decodes 1.052× as fast from the lm_head on its per-row int8 grid** (the smallest of four ABBA pairs;
  grouped-nf4-gemm K36; at
  Qwen3-30B-A3B's 151936 x 2048 head only; needs grouped-nf4-gemm 0.48.0 or later, else it resolves off). **Outputs
  change:** the logits differ from the bf16 head's, inside the quality gates read on Qwen3-30B-A3B @ad44e777. Sixteen
  requests read 0.992, a cost not yet attributed. `E4B_INT8_HEAD_BW=0` restores the bf16 head.
  ([P153](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/p153/RESULTS-p153.md), `e4b.serve.p153.int8-head-bw.qwen3.5090.2026-10-19`)
- **One request decodes 1.044× as fast with the paged FP8 attention decode capped at 16 splits** (the smallest of four
  ABBA pairs; buckets 1, 2 and 4 move; bitwise-identical to the library's plan at contexts up to 1024). Sixteen requests
  are unchanged. `E4B_ATTN_SPLIT_CAP=0` opts out.
  ([P143](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/p143/RESULTS-p143.md),
  `e4b.serve.p143.attn-split-cap.qwen3.5090.2026-10-19`)

Training, opt-in. Each was measured on the card named, against the baseline its bullet names.

- **Layer-major training under expert offload steps 2.365× as fast on Qwen3-30B-A3B on one RTX 4090** (the smaller of
  two ABBA pairs of median step times). `layer_major_step` (`experts4bit_qlora.engines.layer_major`) runs a step's
  micro-batches layer by layer, so each offloaded layer is staged once per step instead of once per micro-batch. It
  replicates at 2.311× on an RTX 5090 and holds on Mixtral-8x7B at 2.118× on an RTX 4090, each the smaller of two ABBA
  pairs of median step times. Opt-in.
  ([P137](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/p137/RESULTS-p137.md),
  `e4b.train.p137.layer-major-offload.qwen3.4090.2026-10-18`;
  [P140](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/p140/RESULTS-p140.md),
  `e4b.train.p140.layer-major-offload.qwen3.5090.2026-10-18`;
  [P144](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/p144/RESULTS-p144.md),
  `e4b.train.p144.layer-major-offload.mixtral.4090.2026-10-18`)
- **The CLI trainer can take the fused training path.** In 0.55.0, `python -m experts4bit_qlora.train` always ran the
  reference loop. `FAST_TRAIN=1` now puts it on the fused path, and `LAYER_MAJOR=1` implies it. At the CLI's own default
  recipe on Qwen3-30B-A3B with `OFFLOAD_EXPERTS=1`, `LAYER_MAJOR=1` steps 1.455× as fast as `FAST_TRAIN=1` alone on one
  RTX 4090 (the smaller of two ABBA pairs of median step times). An explicit `FAST_TRAIN=0` keeps the reference loop.
  ([P146](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/p146/RESULTS-p146.md), `e4b.train.p146.layer-major-cli.qwen3.4090.2026-10-18`)

Mixtral serving.

- **Mixtral-8x7B decodes with its B=1 fold stack by default.** The T1 glue, its second round and the router epilogue now
  resolve to `auto` on Mixtral; fused q/k/v does not engage on this family. A one-request step takes 0.9393 of its time
  without them (median step time, the larger of two interleaved blocks) (about 20.2-20.5 ms to 19.0 ms) on one RTX 5090.
  Opt out per knob: `E4B_FUSE_T1_GLUE=0`, `E4B_FUSE_T1_GLUE_R2=0`, `E4B_FUSE_ROUTER_EPI=0`.
  ([P135](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/p135/RESULTS-p135.md), `e4b.serve.p135.fold-speed.mixtral.5090.2026-10-18`)
- **`E4B_SERVE_ATTN_INT8=1`: int8 weight-only attention projections on Mixtral, opt-in.** q/k/v/o are held in int8 with
  a per-row fp32 scale, and decode calls of 1-16 rows run grouped-nf4-gemm's K37 kernels (from 0.49.0). On one RTX 5090
  the served read decodes one request 1.161x as fast (the smallest of four ABBA pairs) and sixteen 1.018x, with decode
  quality inside its registered bound. It stays opt-in until a routing-matched read: the default-flip PR is held.
  ([P149](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/p149/RESULTS-p149.md),
  `e4b.serve.p149.int8-attn.mixtral.5090.2026-10-20`)

Scale.

- **One full-depth QLoRA optimizer step on Qwen3-235B-A22B on one GPU.** All 94 decoder layers and every routed expert,
  two optimizer steps plus a fresh-process repeat of step 0, on one RTX PRO 6000 Blackwell Workstation Edition.
  ([Q235](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/q235/RESULTS-q235.md), run `q235-pro6000-8`, MILESTONE PASS)

## Check the fit

- **Models:** use the [architecture support matrix](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/ARCHITECTURE_SUPPORT.md). Support and test coverage vary by family and execution path.
- **Memory:** CPU/NVMe offload trades transfer time and host storage for GPU memory. Measure your workload.
- **Speed:** 4-bit saves weight memory; it is not automatically faster or more energy-efficient than bf16 when both fit.
- **Serving:** vLLM was faster in the measured Qwen decode comparison. This runtime focuses on fitting and controlling low-bit MoEs; see the [serving results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/SERVING-THROUGHPUT.md#throughput-census-bo7-measured-in-repo).
- **Platforms:** CUDA execution is the tested path. ROCm, macOS, Windows and other non-CUDA GPU execution are outside this package's scope.

## Documentation and machine-readable data

[Task guides](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/SOLUTIONS.md) ·
[API and capability map](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/capabilities.json) ·
[Claims and evidence](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/claims.json) ·
[Package compatibility](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/system-manifest.json) ·
[Documentation index](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/INDEX.md) ·
[llms.txt](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/llms.txt) ·
[Contributing](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/AGENTS.md)

## License

MIT. The vendored bitsandbytes code is also MIT; see [third-party notices](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/THIRD_PARTY_NOTICES.md).
For integration or research work: [jordananderson.work](https://jordananderson.work).
