TD3+BC Implementation Plan¶

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Add the TD3+BC algorithm with offline training, checkpoint evaluation, prediction, and resume support for continuous-control tasks.

Architecture: Reuse the existing MLPTD3Model actor/twin-critic stack and add a dedicated TD3BC algorithm plus offline train_td3_bc(...) trainer. Reuse the same deterministic offline dataset recipe pattern introduced for IQL so the first version supports reproducible offline training without adding a broader external dataset ingestion surface yet.

Tech Stack: Python, PyTorch, Gymnasium, pytest

Task 1: Define TD3+BC update behavior with failing tests¶

Files: - Create: tests/test_td3_bc_update.py - Reference: tests/test_td3_update.py - Reference: tests/test_iql_update.py

Step 1: Write the failing test

Add tests that define: - td3_bc_loss(...) returns named metrics - invalid bc_alpha / policy_delay values fail fast - TD3BC.update(...) returns one gradient step and named metrics

Step 2: Run test to verify it fails

Run: PYTHONPATH=src pytest tests/test_td3_bc_update.py -q

Expected: FAIL because the TD3+BC algorithm module does not exist yet.

Step 3: Write minimal implementation

Create the TD3+BC algorithm by reusing MLPTD3Model and adding the actor BC regularization term.

Step 4: Run test to verify it passes

Run: PYTHONPATH=src pytest tests/test_td3_bc_update.py -q

Expected: PASS

Task 2: Add offline TD3+BC trainer coverage¶

Files: - Create: src/rl_training/runtime/td3_bc_trainer.py - Create: tests/test_td3_bc_trainer_smoke.py - Reference: src/rl_training/runtime/td3_trainer.py - Reference: src/rl_training/runtime/iql_trainer.py

Step 1: Write the failing test

Add a smoke test that trains TD3+BC for a short offline Pendulum-v1 run using the deterministic dataset recipe, then asserts checkpoint creation and basic metrics.

Step 2: Run test to verify it fails

Run: PYTHONPATH=src pytest tests/test_td3_bc_trainer_smoke.py -q

Expected: FAIL because train_td3_bc(...) does not exist yet.

Step 3: Write minimal implementation

Implement the offline trainer loop where total_timesteps means gradient steps, and reuse the deterministic offline dataset recipe pattern.

Step 4: Run test to verify it passes

Run: PYTHONPATH=src pytest tests/test_td3_bc_trainer_smoke.py -q

Expected: PASS

Task 3: Wire TD3+BC into registry, API, CLI, examples, and checkpoint workflows¶

Files: - Create: configs/td3_bc/pendulum.yaml - Create: examples/td3_bc_pendulum_reference.py - Create: tests/test_td3_bc_reference_script.py - Modify: src/rl_training/algorithms/__init__.py - Modify: src/rl_training/api/algorithms.py - Modify: src/rl_training/api/__init__.py - Modify: src/rl_training/__init__.py - Modify: src/rl_training/experiment/registry.py - Modify: tests/test_cli.py - Modify: tests/test_public_api.py - Modify: tests/test_package_api_exports.py - Modify: tests/test_checkpoint_workflows.py - Modify: tests/test_experiment_manager.py

Step 1: Write the failing tests

Add focused coverage for: - CLI train command with a TD3+BC YAML config - managed public API learn/evaluate/predict flow - package export visibility - checkpoint evaluation and resume workflow - experiment manager spec registration - example script smoke execution

Step 2: Run tests to verify they fail

Run: PYTHONPATH=src pytest tests/test_cli.py tests/test_public_api.py tests/test_package_api_exports.py tests/test_checkpoint_workflows.py tests/test_experiment_manager.py tests/test_td3_bc_reference_script.py -q

Expected: FAIL because TD3+BC is not yet registered end-to-end.

Step 3: Write minimal implementation

Register TD3+BC in all public surfaces and add a compact reference config/example using the deterministic offline dataset recipe.

Step 4: Run tests to verify they pass

Run: PYTHONPATH=src pytest tests/test_cli.py tests/test_public_api.py tests/test_package_api_exports.py tests/test_checkpoint_workflows.py tests/test_experiment_manager.py tests/test_td3_bc_reference_script.py -q

Expected: PASS

Task 4: Final verification¶

Files: - Verify only

Step 1: Run targeted TD3+BC suite

Run: PYTHONPATH=src pytest tests/test_td3_bc_update.py tests/test_td3_bc_trainer_smoke.py tests/test_td3_bc_reference_script.py tests/test_cli.py tests/test_public_api.py tests/test_package_api_exports.py tests/test_checkpoint_workflows.py tests/test_experiment_manager.py -q

Expected: PASS

Step 2: Run full test suite

Run: PYTHONPATH=src pytest -q

Expected: PASS