Metadata-Version: 2.4
Name: turbo-dllm
Version: 0.1.1
Summary: Highly optimized distributed training for diffusion language models.
Author: Turbo-dLLM contributors
License-Expression: Apache-2.0
Project-URL: Documentation, https://github.com/ScalingIntelligence/Turbo-dLLM/tree/main/docs
Project-URL: Issues, https://github.com/ScalingIntelligence/Turbo-dLLM/issues
Project-URL: Source, https://github.com/ScalingIntelligence/Turbo-dLLM
Keywords: diffusion-language-models,distributed-training,pytorch
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: PyYAML<7,>=6.0
Requires-Dist: tomli>=2.0; python_version < "3.11"
Requires-Dist: numpy<3,>=1.26
Requires-Dist: einops<1,>=0.8
Requires-Dist: torch<3,>=2.8
Requires-Dist: transformers<6,>=5.0
Requires-Dist: huggingface-hub<2,>=0.34
Requires-Dist: accelerate<2,>=1.0
Requires-Dist: safetensors<1,>=0.4
Provides-Extra: data
Requires-Dist: datasets==4.8.5; extra == "data"
Provides-Extra: capture
Requires-Dist: specforge==0.2.0; (sys_platform == "linux" and python_version >= "3.11" and python_version < "3.14") and extra == "capture"
Requires-Dist: flash-attn-4==4.0.0b15; (sys_platform == "linux" and python_version >= "3.11" and python_version < "3.14") and extra == "capture"
Provides-Extra: plots
Requires-Dist: wandb>=0.19; extra == "plots"
Requires-Dist: matplotlib>=3.8; extra == "plots"
Provides-Extra: logging
Requires-Dist: wandb>=0.19; extra == "logging"
Provides-Extra: deepspeed
Requires-Dist: deepspeed>=0.19.2; extra == "deepspeed"
Provides-Extra: sglang
Requires-Dist: sglang==0.5.20; (sys_platform == "linux" and python_version < "3.14") and extra == "sglang"
Requires-Dist: cuda-tile==1.6.0rc5; (sys_platform == "linux" and python_version < "3.14") and extra == "sglang"
Requires-Dist: flash-attn-4==4.0.0b19; (sys_platform == "linux" and python_version < "3.14") and extra == "sglang"
Provides-Extra: vllm
Requires-Dist: vllm==0.29.0; (sys_platform == "linux" and python_version < "3.14") and extra == "vllm"
Provides-Extra: gpu
Requires-Dist: torch==2.10.0; extra == "gpu"
Requires-Dist: transformers==5.13.0; extra == "gpu"
Requires-Dist: accelerate==1.14.0; extra == "gpu"
Requires-Dist: deepspeed==0.19.2; extra == "gpu"
Requires-Dist: transformer-engine[core_cu12,pytorch]==2.13.0; extra == "gpu"
Requires-Dist: cuda-bindings==12.9.4; extra == "gpu"
Requires-Dist: cuda-python==12.9.4; extra == "gpu"
Requires-Dist: apache-tvm-ffi==0.1.12; extra == "gpu"
Requires-Dist: nvidia-cutlass-dsl==4.5.2; extra == "gpu"
Requires-Dist: nvidia-cutlass-dsl-libs-base==4.5.2; extra == "gpu"
Requires-Dist: torch-c-dlpack-ext==0.1.5; extra == "gpu"
Requires-Dist: quack-kernels==0.5.0; extra == "gpu"
Provides-Extra: qwen3-8
Requires-Dist: transformers==5.13.0; extra == "qwen3-8"
Requires-Dist: transformer-engine[core_cu12,pytorch]==2.13.0; extra == "qwen3-8"
Requires-Dist: flash-linear-attention[conv1d,cuda,tilelang]==0.5.2; extra == "qwen3-8"
Requires-Dist: tilelang==0.1.13; extra == "qwen3-8"
Provides-Extra: kernel-build
Requires-Dist: build==1.5.0; extra == "kernel-build"
Requires-Dist: ninja==1.13.0; extra == "kernel-build"
Requires-Dist: setuptools-scm==9.2.2; extra == "kernel-build"
Requires-Dist: nvidia-cutlass-dsl==4.5.2; extra == "kernel-build"
Requires-Dist: nvidia-cutlass-dsl-libs-base==4.5.2; extra == "kernel-build"
Requires-Dist: torch-c-dlpack-ext==0.1.5; extra == "kernel-build"
Requires-Dist: quack-kernels==0.5.0; extra == "kernel-build"
Provides-Extra: test
Requires-Dist: pytest>=8; extra == "test"
Requires-Dist: omegaconf>=2.3; extra == "test"
Provides-Extra: dev
Requires-Dist: build==1.5.0; extra == "dev"
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: omegaconf>=2.3; extra == "dev"
Requires-Dist: wandb>=0.19; extra == "dev"
Requires-Dist: matplotlib>=3.8; extra == "dev"
Requires-Dist: ruff>=0.12; extra == "dev"
Dynamic: license-file

# Turbo-dLLM

Turbo-dLLM is a highly optimized distributed training library for diffusion
language models. It includes the official implementation of context-sharded
block parallelism for scaling block-diffusion language model training to large
contexts, plus typed configuration, prepared-data runtimes, checkpointing, and
optimized CUDA kernels.

## Install

Install the portable package for configuration, data preparation, APIs, and
CPU-safe validation:

```bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install turbo-dllm
dllm doctor
```

GPU training uses a bundle matched to the host's Python, CUDA, and GPU
architecture:

```bash
dllm bundle install --release v0.1.1 --auto
```

The installer downloads only an exact supported bundle and verifies its native
artifacts. See [installation](docs/getting-started/installation.md) for source
installs, offline mirrors, and model-specific extras.

## Start training

Create an editable starter project:

```bash
dllm init ./my-run
cd ./my-run
```

Add JSONL records such as `{"text": "A training document."}` to
`data/train.jsonl`, then prepare and launch:

```bash
dllm data prepare --config prepare.yaml
dllm data validate data/prepared --config train.yaml
dllm launch --config train.yaml --dry-run
dllm doctor --config train.yaml
dllm launch --config train.yaml --nproc-per-node 1
```

The launcher validates the config and runtime before starting workers. For
distributed training, change the topology in `train.yaml` and set the matching
process count:

```bash
dllm launch --config train.yaml --nproc-per-node 8
```

For speculative training and deployment, follow the
[DFlash2 train-to-serve guide](docs/getting-started/dflash2-training-and-serving.md).

## Data and recipes

The preparation frontend accepts Hugging Face datasets, JSONL, Parquet, text,
and token IDs. It supports text, chat messages, prompt/completion records, and
token-level supervision while keeping tokenization outside the GPU training
loop.

```bash
python -m pip install 'turbo-dllm[data]'
dllm data prepare --config prepare.yaml
dllm data inspect data/prepared
dllm data stats data/prepared
```

Packaged recipes provide small validation runs and focused examples:

```bash
dllm recipe list
dllm recipe show smoke/cuda-fast-dllm-v2
dllm recipe copy examples/fast-dllm-v2-qwen3 ./run.yaml
dllm config validate --config ./run.yaml
```

## Supported training

- Models: generic causal LMs, DFlash, DiffusionGemma, Nemotron Labs Diffusion,
  and Qwen3.8.
- Objectives: standard block diffusion, Fast-dLLM v2, DFlash distillation, and
  DiffusionGemma native SFT.
- Parallelism: data, context, block, tensor, sequence, FSDP, and supported
  DiffusionGemma expert parallelism.
- Operations: deterministic data artifacts, checkpoint/resume, profiling, and
  optional W&B logging.

Unsupported combinations fail during validation instead of silently falling
back.

## Documentation

- [Quickstart](docs/getting-started/quickstart.md)
- [Train and serve DFlash2](docs/getting-started/dflash2-training-and-serving.md)
- [Data preparation](docs/configuration/data-preparation.md)
- [RunSpec configuration](docs/configuration/run-spec.md)
- [Supported models](docs/models/supported.md)
- [Parallelism](docs/parallelism/topologies.md)
- [GPU bundles](docs/operations/gpu-bundles.md)
- [API](docs/api/index.md)
- [Contributing](CONTRIBUTING.md) and [security](SECURITY.md)

## License

First-party code is Apache-2.0. Vendored components retain their upstream
licenses; see [NOTICE](NOTICE) and
[FlashAttention provenance](third_party/flash-attention/PROVENANCE.md).
