Metadata-Version: 2.4
Name: trl
Version: 1.18.0
Summary: Train transformer language models with reinforcement learning.
Author-email: Leandro von Werra <leandro.vonwerra@gmail.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/huggingface/trl
Keywords: transformers,huggingface,language modeling,post-training,rlhf,sft,dpo,grpo
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: accelerate>=1.11.0
Requires-Dist: datasets>=4.7.0
Requires-Dist: huggingface_hub>=0.34.0
Requires-Dist: jinja2
Requires-Dist: numpy>=1.26.0
Requires-Dist: packaging>20.0
Requires-Dist: pandas>=2.1.1
Requires-Dist: psutil>=5.9.4
Requires-Dist: pyarrow>=21.0.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: torch>=2.6.0
Requires-Dist: transformers>=4.56.2
Provides-Extra: deepspeed
Requires-Dist: deepspeed>=0.18.6; extra == "deepspeed"
Requires-Dist: transformers!=5.1.0; extra == "deepspeed"
Provides-Extra: harbor
Requires-Dist: harbor>=0.13.0; python_version >= "3.12" and extra == "harbor"
Provides-Extra: kernels
Requires-Dist: transformers[kernels]; extra == "kernels"
Requires-Dist: transformers[hub-kernels]; extra == "kernels"
Provides-Extra: liger
Requires-Dist: liger-kernel>=0.8.2; extra == "liger"
Provides-Extra: math-verify
Requires-Dist: math-verify>=0.8.0; extra == "math-verify"
Provides-Extra: openreward
Requires-Dist: openreward>=0.1.109; extra == "openreward"
Provides-Extra: peft
Requires-Dist: peft>=0.17.0; extra == "peft"
Provides-Extra: quality
Requires-Dist: pre-commit; extra == "quality"
Requires-Dist: hf-doc-builder; extra == "quality"
Provides-Extra: quantization
Requires-Dist: bitsandbytes>=0.45.2; extra == "quantization"
Provides-Extra: test
Requires-Dist: pytest-cov; extra == "test"
Requires-Dist: pytest-datadir>=1.7.0; extra == "test"
Requires-Dist: pytest-rerunfailures>=16.7; extra == "test"
Requires-Dist: pytest-xdist; extra == "test"
Requires-Dist: pytest; extra == "test"
Provides-Extra: vllm
Requires-Dist: vllm<0.34.0,>=0.23.0; extra == "vllm"
Requires-Dist: aiohttp>=3.13.3; extra == "vllm"
Requires-Dist: requests; extra == "vllm"
Provides-Extra: vlm
Requires-Dist: Pillow; extra == "vlm"
Requires-Dist: torchvision; extra == "vlm"
Requires-Dist: num2words==0.5.14; extra == "vlm"
Provides-Extra: dev
Requires-Dist: deepspeed>=0.18.6; extra == "dev"
Requires-Dist: transformers[kernels]; extra == "dev"
Requires-Dist: transformers[hub-kernels]; extra == "dev"
Requires-Dist: math-verify>=0.8.0; extra == "dev"
Requires-Dist: openreward>=0.1.109; extra == "dev"
Requires-Dist: peft>=0.17.0; extra == "dev"
Requires-Dist: pre-commit; extra == "dev"
Requires-Dist: hf-doc-builder; extra == "dev"
Requires-Dist: bitsandbytes>=0.45.2; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: pytest-datadir>=1.7.0; extra == "dev"
Requires-Dist: pytest-rerunfailures>=16.7; extra == "dev"
Requires-Dist: pytest-xdist; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: aiohttp>=3.13.3; extra == "dev"
Requires-Dist: Pillow; extra == "dev"
Requires-Dist: torchvision; extra == "dev"
Requires-Dist: num2words==0.5.14; extra == "dev"
Requires-Dist: jmespath; extra == "dev"
Dynamic: license-file

# TRL - Transformers Reinforcement Learning

<div style="text-align: center">
    <picture>
        <source media="(prefers-color-scheme: light)" srcset="https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl_banner_light.png">
        <img src="https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl_banner_dark.png" alt="TRL Banner">
    </picture>
</div>

<hr> <br>

<h3 align="center">
    <p>A comprehensive library to post-train foundation models</p>
</h3>

<p align="center">
    <a href="https://github.com/huggingface/trl/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/github/license/huggingface/trl.svg?color=blue"></a>
    <a href="https://huggingface.co/docs/trl/index"><img alt="Documentation" src="https://img.shields.io/website?label=documentation&url=https%3A%2F%2Fhuggingface.co%2Fdocs%2Ftrl%2Findex&down_color=red&down_message=offline&up_color=blue&up_message=online"></a>
    <a href="https://github.com/huggingface/trl/releases"><img alt="GitHub release" src="https://img.shields.io/github/release/huggingface/trl.svg"></a>
    <a href="https://huggingface.co/trl-lib"><img alt="Hugging Face Hub" src="https://img.shields.io/badge/🤗%20Hub-trl--lib-yellow"></a>
</p>

## 🎉 What's New

**⚡ AsyncGRPOTrainer is now stable:** [`AsyncGRPOTrainer`](https://huggingface.co/docs/trl/async_grpo_trainer) graduates to the stable API. It runs GRPO much faster by decoupling generation from training: a vLLM server streams rollouts while the trainer updates the model.

## Overview

TRL is a cutting-edge library designed for post-training foundation models using advanced techniques like Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), and Direct Preference Optimization (DPO). Built on top of the [🤗 Transformers](https://github.com/huggingface/transformers) ecosystem, TRL supports a variety of model architectures and modalities, and can be scaled-up across various hardware setups.

## Highlights

- **Trainers**: Various fine-tuning methods are easily accessible via trainers like [`SFTTrainer`](https://huggingface.co/docs/trl/sft_trainer), [`GRPOTrainer`](https://huggingface.co/docs/trl/grpo_trainer), [`AsyncGRPOTrainer`](https://huggingface.co/docs/trl/async_grpo_trainer), [`DPOTrainer`](https://huggingface.co/docs/trl/dpo_trainer), [`KTOTrainer`](https://huggingface.co/docs/trl/kto_trainer) and more.

- **Efficient and scalable**:
  - Leverages [🤗 Accelerate](https://github.com/huggingface/accelerate) to scale from single GPU to multi-node clusters using methods like [DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) and [DeepSpeed](https://github.com/deepspeedai/DeepSpeed).
  - Full integration with [🤗 PEFT](https://github.com/huggingface/peft) enables training on large models with modest hardware via quantization and LoRA/QLoRA.
  - Integrates [🦥 Unsloth](https://github.com/unslothai/unsloth) for accelerating training using optimized kernels.

- **Command Line Interface (CLI)**: A simple interface lets you fine-tune with models without needing to write code.

## Installation

### Python Package

Install the library using `pip`:

```bash
pip install trl
```

### From source

If you want to use the latest features before an official release, you can install TRL from source:

```bash
pip install git+https://github.com/huggingface/trl.git
```

### Repository

If you want to use the examples you can clone the repository with the following command:

```bash
git clone https://github.com/huggingface/trl.git
```

## Quick Start

For more flexibility and control over training, TRL provides dedicated trainer classes to post-train language models or PEFT adapters on a custom dataset. Each trainer in TRL is a light wrapper around the 🤗 Transformers trainer and natively supports distributed training methods like DDP, DeepSpeed ZeRO, and FSDP.

### `SFTTrainer`

Here is a basic example of how to use the [`SFTTrainer`](https://huggingface.co/docs/trl/sft_trainer):

```python
from trl import SFTTrainer
from datasets import load_dataset

dataset = load_dataset("trl-lib/Capybara", split="train")

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=dataset,
)
trainer.train()
```

### `GRPOTrainer`

[`GRPOTrainer`](https://huggingface.co/docs/trl/grpo_trainer) implements the [Group Relative Policy Optimization (GRPO) algorithm](https://huggingface.co/papers/2402.03300) that is more memory-efficient than PPO and was used to train [Deepseek AI's R1](https://huggingface.co/deepseek-ai/DeepSeek-R1). `accuracy_reward` needs math-verify, which you can install with `pip install trl[math_verify]`.

```python
from datasets import load_dataset
from trl import GRPOTrainer
from trl.rewards import accuracy_reward

dataset = load_dataset("trl-lib/DeepMath-103K", split="train")

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    reward_funcs=accuracy_reward,
    train_dataset=dataset,
)
trainer.train()
```

> [!NOTE]
> For reasoning models, use the `reasoning_accuracy_reward()` function for better results.

### `AsyncGRPOTrainer`

[`AsyncGRPOTrainer`](https://huggingface.co/docs/trl/async_grpo_trainer) runs the same algorithm as [`GRPOTrainer`](https://huggingface.co/docs/trl/grpo_trainer), much faster: a vLLM server streams rollouts while the trainer updates the model, so generation and training overlap instead of alternating. The trainer needs vLLM and kernels, and `accuracy_reward` needs math-verify, which you can install with `pip install trl[vllm,kernels,math_verify]`.

```python
# train_async_grpo.py
from datasets import load_dataset
from trl import AsyncGRPOTrainer
from trl.rewards import accuracy_reward

if __name__ == "__main__":
    dataset = load_dataset("trl-lib/DeepMath-103K", split="train")

    trainer = AsyncGRPOTrainer(
        model="Qwen/Qwen3.5-2B",
        reward_funcs=accuracy_reward,
        train_dataset=dataset,
    )
    trainer.train()
```

Generation runs on a vLLM server, on a different GPU than the trainer. Start it on GPU 0:

```bash
CUDA_VISIBLE_DEVICES=0 VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3.5-2B \
    --weight-transfer-config '{"backend": "nccl"}' --logprobs-mode processed_logprobs --max-model-len 4096
```

Then run the training on GPU 1:

```bash
CUDA_VISIBLE_DEVICES=1 python train_async_grpo.py
```

### `DPOTrainer`

[`DPOTrainer`](https://huggingface.co/docs/trl/dpo_trainer) implements the popular [Direct Preference Optimization (DPO) algorithm](https://huggingface.co/papers/2305.18290) that was used to post-train [Llama 3](https://huggingface.co/papers/2407.21783) and many other models. Here is a basic example of how to use the `DPOTrainer`:

```python
from datasets import load_dataset
from trl import DPOTrainer

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

trainer = DPOTrainer(
    model="Qwen/Qwen3-0.6B",
    train_dataset=dataset,
)
trainer.train()
```

### `KTOTrainer`

[`KTOTrainer`](https://huggingface.co/docs/trl/kto_trainer) implements the [Kahneman-Tversky Optimization (KTO) algorithm](https://huggingface.co/papers/2402.01306), which aligns models from simple binary (desirable / undesirable) feedback rather than paired preferences. Here is a basic example of how to use the `KTOTrainer`:

```python
from datasets import load_dataset
from trl import KTOTrainer

dataset = load_dataset("trl-lib/kto-mix-14k", split="train")

trainer = KTOTrainer(
    model="Qwen/Qwen3-0.6B",
    train_dataset=dataset,
)
trainer.train()
```

### `RewardTrainer`

Here is a basic example of how to use the [`RewardTrainer`](https://huggingface.co/docs/trl/reward_trainer):

```python
from trl import RewardTrainer
from datasets import load_dataset

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

trainer = RewardTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    train_dataset=dataset,
)
trainer.train()
```

## Command Line Interface (CLI)

You can use the TRL Command Line Interface (CLI) to quickly get started with post-training methods like Supervised Fine-Tuning (SFT) or Direct Preference Optimization (DPO):

**SFT:**

```bash
trl sft --model_name_or_path Qwen/Qwen2.5-0.5B \
    --dataset_name trl-lib/Capybara \
    --output_dir Qwen2.5-0.5B-SFT
```

**DPO:**

```bash
trl dpo --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
    --dataset_name argilla/Capybara-Preferences \
    --output_dir Qwen2.5-0.5B-DPO
```

**KTO:**

```bash
trl kto --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
    --dataset_name trl-lib/kto-mix-14k \
    --output_dir Qwen2.5-0.5B-KTO
```

Read more about CLI in the [relevant documentation section](https://huggingface.co/docs/trl/clis) or use `--help` for more details.

## Development

If you want to contribute to `trl` or customize it to your needs make sure to read the [contribution guide](https://github.com/huggingface/trl/blob/main/CONTRIBUTING.md) and make sure you make a dev install:

```bash
git clone https://github.com/huggingface/trl.git
cd trl/
pip install -e .[dev]
```

## Experimental

A minimal incubation area is available under `trl.experimental` for unstable / fast-evolving features. Anything there may change or be removed in any release without notice.

Example:

```python
from trl.experimental.new_trainer import NewTrainer
```

Read more in the [Experimental docs](https://huggingface.co/docs/trl/experimental_overview).

## Citation

```bibtex
@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin and Dirhoussi, Amine},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2020}
}
```

## License

This repository's source code is available under the [Apache-2.0 License](LICENSE).
