Metadata-Version: 2.4
Name: TrackmaniaRL
Version: 1.1.0
Summary: Reinforcement-learning library for training agents in Trackmania 2020
Author: Jakub Szulc
License-Expression: MIT
Project-URL: Homepage, https://github.com/Palamabron/TrackmaniaRL
Project-URL: Repository, https://github.com/Palamabron/TrackmaniaRL
Project-URL: Documentation, https://github.com/Palamabron/TrackmaniaRL/tree/main/readme
Project-URL: Issues, https://github.com/Palamabron/TrackmaniaRL/issues
Project-URL: Changelog, https://github.com/Palamabron/TrackmaniaRL/blob/main/CHANGELOG.md
Project-URL: Security, https://github.com/Palamabron/TrackmaniaRL/blob/main/SECURITY.md
Keywords: reinforcement learning,robot learning,trackmania,self driving,roborace
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Games/Entertainment
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: <3.13,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: pydantic>=2.0
Requires-Dist: numpy>=1.24
Requires-Dist: torch>=2.4
Requires-Dist: tensordict>=0.6
Requires-Dist: gymnasium>=0.29
Requires-Dist: PyYAML>=6.0
Requires-Dist: zstandard>=0.23
Provides-Extra: orchestrator
Requires-Dist: google-genai>=2.2.0; extra == "orchestrator"
Requires-Dist: optuna>=3.0; extra == "orchestrator"
Provides-Extra: trackmania
Requires-Dist: libevdev>=0.13; sys_platform == "linux" and extra == "trackmania"
Requires-Dist: vgamepad>=0.1.0; (sys_platform == "win32" or sys_platform == "linux") and extra == "trackmania"
Provides-Extra: wandb
Requires-Dist: wandb>=0.15.8; extra == "wandb"
Provides-Extra: distributed
Requires-Dist: grpcio>=1.66; extra == "distributed"
Requires-Dist: protobuf>=5.27; extra == "distributed"
Requires-Dist: safetensors>=0.4.5; extra == "distributed"
Provides-Extra: mamba
Requires-Dist: causal-conv1d>=1.4; sys_platform == "linux" and extra == "mamba"
Requires-Dist: mamba-ssm>=2.2; sys_platform == "linux" and extra == "mamba"
Provides-Extra: all
Requires-Dist: google-genai>=2.2.0; extra == "all"
Requires-Dist: optuna>=3.0; extra == "all"
Requires-Dist: libevdev>=0.13; sys_platform == "linux" and extra == "all"
Requires-Dist: vgamepad>=0.1.0; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
Requires-Dist: wandb>=0.15.8; extra == "all"
Requires-Dist: grpcio>=1.66; extra == "all"
Requires-Dist: protobuf>=5.27; extra == "all"
Requires-Dist: safetensors>=0.4.5; extra == "all"
Requires-Dist: causal-conv1d>=1.4; sys_platform == "linux" and extra == "all"
Requires-Dist: mamba-ssm>=2.2; sys_platform == "linux" and extra == "all"
Dynamic: license-file

<p align="center">
  <img src="https://raw.githubusercontent.com/Palamabron/TrackmaniaRL/main/docs/assets/trackmaniarl-logo.png" alt="TrackmaniaRL logo" width="520">
</p>

# TrackmaniaRL

[![PyPI](https://img.shields.io/pypi/v/TrackmaniaRL?color=blue)](https://pypi.org/project/TrackmaniaRL/)
[![Python](https://img.shields.io/pypi/pyversions/TrackmaniaRL?color=3776AB)](https://pypi.org/project/TrackmaniaRL/)
[![CI](https://github.com/Palamabron/TrackmaniaRL/actions/workflows/ci.yml/badge.svg)](https://github.com/Palamabron/TrackmaniaRL/actions/workflows/ci.yml)
[![License](https://img.shields.io/pypi/l/TrackmaniaRL?color=green)](https://github.com/Palamabron/TrackmaniaRL/blob/main/LICENSE)
[![Status](https://img.shields.io/pypi/status/TrackmaniaRL?color=orange)](https://pypi.org/project/TrackmaniaRL/)

TrackmaniaRL is a reinforcement-learning library for training agents in
Trackmania 2020. It combines ready-to-use algorithms, replay buffers, model
families and Trackmania telemetry with explicit interfaces for replacing any
component in an experiment.

This checkout is TrackmaniaRL `1.1.0` and supports Python 3.12. This package
release introduces the breaking RunSpec 2.0 runtime and checkpoint contract;
follow the [migration guide](https://github.com/Palamabron/TrackmaniaRL/blob/main/readme/migration-2.0.md)
before reusing a 1.0 project or checkpoint. Pin `trackmaniarl==1.1.0` exactly
during this transition; a compatible 1.x dependency range cannot express this
intentional breaking boundary.

## What you get

- asynchronous local or distributed actor/learner training;
- Standard Q, QR-DQN, IQN and FQF through one distributional learner, plus
  SAC, REDQ-SAC, TQC, PPO, BC and stable discrete SAC;
- uniform, prioritized, sequence and demonstration-mixing replay;
- typed configuration, transitions and training batches;
- Trackmania telemetry, lidar and track-geometry feature pipelines;
- durable rollout journals, safe policy transfer and resumable checkpoints;
- local JSONL observability with optional W&B, Gemini and Optuna
  integrations;
- an installable extension project generated by `trackmaniarl init`.

TrackmaniaRL has no global runtime configuration and no mandatory external
tracker. A run is described by `run.yaml` and explicit `module:attribute`
component paths.

## Documentation

| If you want to... | Start here |
| --- | --- |
| install the released library and create an agent | [Quick start](#install-and-create-an-agent) |
| run this repository from source | [Development setup](https://github.com/Palamabron/TrackmaniaRL/blob/main/readme/development.md#repository-setup) |
| understand processes, data flow, security boundaries and package ownership | [Architecture and editable diagrams](https://github.com/Palamabron/TrackmaniaRL/blob/main/readme/architecture.md) |
| configure RunSpec, Trackmania, evaluation or distributed execution | [Configuration reference](https://github.com/Palamabron/TrackmaniaRL/blob/main/readme/configuration.md) |
| replace a learner, model, replay strategy or game adapter | [SDK and extension guide](https://github.com/Palamabron/TrackmaniaRL/blob/main/readme/sdk.md) |
| choose an algorithm and verify its support contract | [Algorithm support matrix](https://github.com/Palamabron/TrackmaniaRL/blob/main/readme/algorithms.md) |
| configure PER, n-step returns or recurrent replay | [Replay and sequence guide](https://github.com/Palamabron/TrackmaniaRL/blob/main/readme/replay-and-sequences.md) |
| understand and tune every Trackmania reward component | [Reward reference](https://github.com/Palamabron/TrackmaniaRL/blob/main/readme/rewards.md) |
| prepare Trackmania and OpenPlanet | [Trackmania workflow](https://github.com/Palamabron/TrackmaniaRL/blob/main/readme/trackmania.md) |
| record demonstrations, train BC and hand off to RL | [Imitation-learning workflow](https://github.com/Palamabron/TrackmaniaRL/blob/main/readme/imitation-learning.md) |
| migrate a 1.x run or checkpoint | [2.0 migration guide](https://github.com/Palamabron/TrackmaniaRL/blob/main/readme/migration-2.0.md) |
| design a useful W&B workspace or diagnose a run | [Observability guide](https://github.com/Palamabron/TrackmaniaRL/blob/main/readme/observability.md) |
| diagnose low GPU utilization or compare throughput | [Performance guide](https://github.com/Palamabron/TrackmaniaRL/blob/main/readme/performance.md) |
| report a security issue or review trust boundaries | [Security policy](https://github.com/Palamabron/TrackmaniaRL/blob/main/SECURITY.md) |

## Install and create an agent

Install the published CLI with [uv](https://docs.astral.sh/uv/):

```bash
uv tool install --index https://download.pytorch.org/whl/cpu --with "torch==2.11.0+cpu" "trackmaniarl==1.1.0"
trackmaniarl init my-trackmania-agent --template trackmania
cd my-trackmania-agent
uv sync
uv run trackmaniarl validate run.yaml
```

The `trackmania` template creates an installable agent project with a validated
reference RunSpec and the Trackmania, algorithm and distributed extras declared
for you. W&B remains
opt-in: add the `wandb` extra and an explicit `WandbTracker` component only when
you want remote logging. Omit `--template trackmania` to generate the smaller,
game-free starter project.
`trackmaniarl validate` checks imports, contracts and a synthetic learner update
without starting the game or contacting an external tracker.

The generated directory is the application layer of your project. Keep custom
models, rewards and adapters there and treat the installed `trackmaniarl`
package as the reusable library. `run.yaml` is executable configuration because
its `class_path` entries import Python objects; only run configurations and
extension packages you trust.

To add the SDK to an existing Python project instead, choose only the extras you
need:

```bash
uv add "trackmaniarl==1.1.0"
uv add "trackmaniarl[distributed]==1.1.0"
uv add "trackmaniarl[trackmania,distributed]==1.1.0" "vgamepad @ git+https://github.com/Palamabron/vgamepad@5f3435df3f8a0e658feb58b207d9137cdb5183cd"
```

The CPU Torch override keeps the short-lived scaffolding tool lightweight; the
generated Trackmania project selects the tested CUDA index independently. For
an existing Trackmania project, add the Trackmania extra and vetted `vgamepad`
source in the same resolver transaction shown above, then retain that direct
source until its required installation fix is released upstream. The `all`
extra cannot carry repository-local uv source pins into another project.

| Extra | Adds |
| --- | --- |
| `all` | every optional integration and model dependency |
| `trackmania` | Trackmania environment and Windows/Linux virtual-gamepad support |
| `distributed` | authenticated gRPC rollouts and safetensors policy transfer |
| `wandb` | Weights & Biases logging |
| `orchestrator` | Gemini and Optuna experiment strategies |
| `mamba` | optional native Mamba kernel; the Pure PyTorch backend needs no extension |

## Run Trackmania

Live collection requires Trackmania 2020 on Windows, Openplanet School Mode,
the signed [**TrackmaniaRL Connect** (`SAC_GetData`)](https://openplanet.dev/plugin/sac_getdata)
plugin installed through Plugin Manager, and a prepared map/geometry asset. The bundled source is a
developer-reference snapshot, not the normal installation path. Follow the
[Trackmania workflow](https://github.com/Palamabron/TrackmaniaRL/blob/main/readme/trackmania.md)
or the concrete
[OpenPlanet guide](https://github.com/Palamabron/TrackmaniaRL/blob/main/trackmaniarl/project/openplanet/README.md)
before starting the game integration.

The generated Trackmania project pins the patched
[Palamabron/vgamepad](https://github.com/Palamabron/vgamepad/tree/5f3435df3f8a0e658feb58b207d9137cdb5183cd)
revision containing the unreleased Windows installation fix from
[vgamepad PR #47](https://github.com/yannbouteiller/vgamepad/pull/47). Keep
that source pin until the fix is included in an upstream vgamepad release.

With Trackmania and the OpenPlanet plugin running:

```bash
uv run trackmaniarl track check --config run.yaml
uv run trackmaniarl smoke run.yaml --transitions 100
uv run trackmaniarl train run.yaml
```

The connection check validates three exact 33-field frames, session protocol 2,
the active map UID and player readiness. The bounded smoke test uses the same
asynchronous learner/actor path as training, verifies a live policy refresh and
writes a checkpoint. Start a fresh
run directory when the run API or immutable configuration changes; the current
schema is RunSpec `2.0`.

Generated Trackmania projects select the tested CUDA PyTorch wheels on Windows
and Linux. CPU-only Linux and ROCm users must replace that generated Torch
source with the index matching their host; macOS uses the normal PyPI wheel and
can use MPS. `device: auto` resolves CUDA, ROCm, MPS or CPU from the installed
Torch build.

## Off-policy runtime model

<p align="center">
  <img src="https://raw.githubusercontent.com/Palamabron/TrackmaniaRL/main/docs/diagrams/runtime-architecture-preview.png" alt="TrackmaniaRL runtime architecture: configuration creates actors and learner; actors send durable rollouts to replay, and learner updates publish policy snapshots" width="900">
</p>

The [architecture guide](https://github.com/Palamabron/TrackmaniaRL/blob/main/readme/architecture.md)
contains the full explanation and editable Excalidraw sources for the runtime,
local/remote deployment, checkpoint recovery and Trackmania integration.

`trackmaniarl train` starts a coordinator/learner and one local actor as
independent, Windows-safe `spawn` processes. This is the off-policy runtime used
by the value-based and actor-critic learners: collection continues while the
learner updates replay and periodically publishes policy snapshots. PPO is an
on-policy exception and uses the local `trackmaniarl.Trainer` API with
`OnPolicySequenceSampler`; it is not supported by the distributed
learner/actor commands.

Read the diagram from top to bottom: `run.yaml` selects and validates
components, the actor collects game transitions and spools them durably, and
the learner ingests, samples, updates and checkpoints. The feedback arrow is
an immutable policy snapshot, so an actor never receives a pickled learner
object. Mamba belongs inside the selected model as an opt-in temporal encoder;
it does not change the actor/learner boundary or the rollout protocol.

### Distributed security and durability

For multiple machines, set the same `TRACKMANIARL_DISTRIBUTED_TOKEN` on every
participant and expose the learner through an encrypted tunnel. The learner
binds to loopback so its bearer token and rollout data are not sent over the
network in clear text:

```bash
# Generate once, then put the value in an ignored .env on both machines.
uv run python -c "import secrets; print(secrets.token_urlsafe(32))"

# training machine
uv run trackmaniarl learner run.yaml --bind 127.0.0.1:8787

# Trackmania machine: create the tunnel first
ssh -N -L 8787:127.0.0.1:8787 TRAINING_MACHINE
uv run trackmaniarl actor run.yaml --connect 127.0.0.1:8787 --actor-id PC-1
```

The handshake rejects mismatched run fingerprints, map UIDs, geometry and pace
reference contents, custom component package source, and feature/action
contracts. Rollouts use Protobuf/gRPC with Zstandard compression, and policy
state is transferred with safetensors rather than pickle.

The token authenticates participants but does not encrypt traffic. Never expose
the gRPC port directly; keep the listener on loopback and use SSH, WireGuard or
another authenticated encrypted tunnel.

<p align="center">
  <img src="https://raw.githubusercontent.com/Palamabron/TrackmaniaRL/main/docs/diagrams/distributed-security-preview.png" alt="Distributed security and durability: an actor spools rollouts, an encrypted tunnel reaches loopback gRPC, then token and contract checks precede WAL ingestion" width="900">
</p>

Read this diagram from left to right. An actor persists a rollout before it is
sent, the encrypted tunnel terminates at the learner's loopback listener, and
the learner checks identity, run compatibility and payload limits before the
contiguous WAL/replay commit. Portable policy snapshots and checkpoints leave
the shared learner runtime. The [editable source](https://github.com/Palamabron/TrackmaniaRL/blob/main/docs/diagrams/distributed-security.excalidraw)
is available for architecture reviews.

## Components and extension API

Components are selected through stable descriptive module paths. The unified
value learner and composite model factory are configured directly, for example:

```yaml
components:
  learner:
    class_path: trackmaniarl.algorithms.value_based:DiscreteValueLearner
  model_factory:
    class_path: trackmaniarl.models.factory:CompositeValueModelFactory
```

Start a new component in the generated extension project. Keep it there when
it is project-specific; move it to the owning library package only when it is
reusable and has passed deterministic contract, configuration and, where
applicable, live Trackmania checks. The [SDK guide](https://github.com/Palamabron/TrackmaniaRL/blob/main/readme/sdk.md)
lists the contract and release gates; the Trackmania connection check and
bounded smoke test apply only to game-facing components.

The stable contracts in `trackmaniarl.core` include `Learner`,
`OfflineSupervisedLearner`, `Policy`, `ModelFactory`, `ReplayStore`, `Sampler`,
`FeaturePipeline`, `Evaluator`, `RunLogger` and `CheckpointCodec`. Game-specific implementations belong in the
generated extension project, so offline validation does not require Trackmania
or optional game dependencies.

Every run writes a redacted immutable config manifest, per-attempt environment
and execution provenance, local JSONL events, checkpoints and bounded compressed
episode artifacts. Only the learner needs W&B
credentials; `WANDB_API_KEY` can be supplied through the environment or project
`.env`.

See the [SDK guide](https://github.com/Palamabron/TrackmaniaRL/blob/main/readme/sdk.md)
for the full component schema and a built-in run example. Release history is in
the [changelog](https://github.com/Palamabron/TrackmaniaRL/blob/main/CHANGELOG.md).

## Development

Clone the repository and install the development group:

```bash
git clone https://github.com/Palamabron/TrackmaniaRL.git
cd TrackmaniaRL
uv sync --group dev
uv run poe fmt
uv run poe types
uv run poe test
```

The commands are intentionally identical on Windows, Linux, WSL and CI. See
[CONTRIBUTING.md](https://github.com/Palamabron/TrackmaniaRL/blob/main/CONTRIBUTING.md)
and [SECURITY.md](https://github.com/Palamabron/TrackmaniaRL/blob/main/SECURITY.md)
before opening a contribution or reporting a vulnerability.

For the repository layout, change workflow, test levels and rules for adding a
public component, read the
[development guide](https://github.com/Palamabron/TrackmaniaRL/blob/main/readme/development.md).

## Project status and attribution

TrackmaniaRL is beta software. The project originated from TMRL and has since
been substantially redesigned. It is not affiliated with or endorsed by
Ubisoft, Nadeo or the TMRL maintainers. Trackmania is a trademark of
Nadeo/Ubisoft. See [NOTICE](https://github.com/Palamabron/TrackmaniaRL/blob/main/NOTICE)
for attribution.
