Metadata-Version: 2.4
Name: ccpl-rl
Version: 0.7.6
Summary: Causal Consequence-Penalized Learning for delayed constrained reinforcement learning
Project-URL: Homepage, https://github.com/sciencebanda09/ccpl
Project-URL: Repository, https://github.com/sciencebanda09/ccpl
Project-URL: Issues, https://github.com/sciencebanda09/ccpl/issues
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: scipy>=1.10
Requires-Dist: matplotlib>=3.7
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Provides-Extra: safety
Requires-Dist: gymnasium>=0.26; extra == "safety"
Requires-Dist: safety-gymnasium>=0.3; extra == "safety"
Requires-Dist: mujoco>=2.3; extra == "safety"
Dynamic: license-file

# CCPL

## Causal Consequence-Penalized Learning

CCPL is a research implementation for constrained reinforcement learning when
constraint consequences can be delayed. Standard constrained RL can attribute a
delayed consequence to the wrong current action, confuse correlation with
causal contribution, and introduce Bellman-target non-stationarity when the
penalty multiplier changes.

The implementation combines four components:

1. A delay-corrected Bellman operator for stochastic consequence delays.
2. A state-conditioned multiplier `lambda(s)` for the constrained policy.
3. An interventional Consequence Net for action-level causal attribution when
   interventional labels are available from the controlled SCM.
4. Separate reward and constraint Q-functions so multiplier changes do not
   alter either critic's TD target.

This is research software. The causal attribution and state-conditioned
multiplier claims are conditional on the assumptions documented in
[`docs/MATHEMATICAL_SPEC.md`](docs/MATHEMATICAL_SPEC.md); they are not claims
of universal superiority or automatic causal identification in arbitrary
environments.

## Results integrity

**Paper-reported benchmark results.** Results reported in the associated paper
belong to the paper's stated experimental protocol and should be cited from
that paper.

**Current repository results.** The repository contains code, configurations,
tests, and result-generation scripts. Benchmark outputs are generated locally
and are not represented here as paper results. Use the exact configuration,
seed list, environment versions, and command recorded in
[`REPRODUCIBILITY.md`](REPRODUCIBILITY.md).

Quick or reduced runs, including short Safety Gymnasium-style runs, are
preliminary checks and must not be presented as the full paper benchmark.

## Installation

From a clean clone:

```bash
python -m pip install -e ".[dev]"
```

The core implementation requires Python 3.10+, NumPy, SciPy, and Matplotlib.
Optional Safety Gymnasium dependencies are available with:

```bash
python -m pip install -e ".[dev,safety]"
```

## Quick start

Install the package and use the public API from another project:

```bash
python -m pip install ccpl-rl
```

```python
from ccpl import make_ccpl, make_env, run_episode

agent = make_ccpl(state_dim=6, action_dim=5, seed=42)
env = make_env("standard", seed=42)
result = run_episode(agent, env, train=False)
print(result)
```

The distribution name on PyPI is `ccpl-rl`; the Python import namespace remains
`ccpl`. Run the repository smoke configuration separately:

```bash
python run_ccpl.py theory
python run_benchmark_v7.py --episodes 200 --eval-eps 30
```

For modern MuJoCo-backed Safety Gymnasium testing with CCPL and CPO-FO:

```bash
python -m pip install -e ".[dev,safety]"
python ccpl_experiments.py --exp E8 --tasks SafetyPointGoal1 --episodes 500 --eval-episodes 100 --seeds 3
```

E8 records per-task reward and constraint metrics and writes visualization
figures for reward, constraint satisfaction, reward-versus-cost, and learning
curves. These are external-environment results and must be reported with task,
MuJoCo, Safety Gymnasium, and seed metadata.

### Application integration

For a custom environment, use `agent.fit(env, episodes=...)` for training and
`agent.predict(observation)` for deterministic inference. The environment must
provide `reset()` and `step(action)`, where `step` returns
`(next_state, reward, consequence, done, info)`. For Gymnasium environments,
wrap the environment and expose the safety cost in `info["cost"]`:

```python
from ccpl import GymnasiumCCPLEnv, make_ccpl

env = GymnasiumCCPLEnv(gym_env, consequence_key="cost", consequence_delay=2)
agent = make_ccpl(state_dim=12, action_dim=4, constraint_d=10.0)
history = agent.fit(env, episodes=1000)
action = agent.predict(observation)
agent.save("checkpoints/ccpl.pkl")
agent = type(agent).load("checkpoints/ccpl.pkl")
```

CCPL currently supports discrete actions and NumPy observations. Checkpoint
files are Python pickle files: load only files from trusted sources. Validate
safety budgets, latency, and behavior across multiple seeds before production
deployment; the algorithm is research software and does not provide safety
certification.

For runtime enforcement, wrap the policy with `SafetyPolicy`. Supply a
conservative application-specific validator and configure a fail-closed budget:

```python
from ccpl import SafetyPolicy

safe_agent = SafetyPolicy(
    agent,
    action_is_safe=lambda observation, action: action != emergency_action,
    action_dim=4,
    fallback_action=0,
    consequence_budget=10.0,
    audit_path="logs/safety.jsonl",
)
action = safe_agent.predict(observation)
safe_agent.observe_consequence(cost, done=done)
```

This is a runtime safety control and audit mechanism, not a certification.

The canonical full synthetic benchmark is configured in
[`configs/main_v7.yaml`](configs/main_v7.yaml). Generated outputs should be
archived with the command and environment metadata when used for a research
claim.

## Reproduction

- [Reproducibility guide](REPRODUCIBILITY.md)
- [Data specification](DATA_SPEC.md)
- [Research protocol](docs/RESEARCH_PROTOCOL.md)
- [Mathematical specification](docs/MATHEMATICAL_SPEC.md)
- [Data and artifact policy](docs/DATA_AND_ARTIFACTS.md)
- [CCPL and CPO-FO comparison protocol](docs/CPO_COMPARISON.md)
- Modern MuJoCo-backed Safety Gymnasium comparisons are run by the E8 command
  documented in the CPO protocol.
- [Primary configuration](configs/main_v7.yaml)

The contraction result requires its stated assumptions, including a positive
minimum delay condition. Unknown stochastic delays do not by themselves imply
contraction. Likewise, the state-conditioned multiplier result is conditional
on the assumptions in the mathematical specification and is not a universal
dominance claim.

## Repository structure

```text
ccpl/                 Installable Python package
  algorithms/         CCPL, baselines, networks, theory utilities
  environments/       Synthetic and Safety Gymnasium-style environments
configs/               Versioned experiment configurations
docs/                  Research protocol and artifact documentation
scripts/               Convenience entry points
tests/                 Numerical, theoretical, and regression tests
run_ccpl.py            Legacy-compatible experiment entry point
run_benchmark_v7.py   Benchmark runner
```

## Tests

```bash
python -m pytest -q
```

The current verified suite contains 37 tests: 37 passed, 0 failed, and 0
skipped.

## Citation

If you use CCPL in academic or research work, please cite the associated
paper. Citation metadata is maintained in [`CITATION.cff`](CITATION.cff).
Bibliographic details that are not present in this repository are intentionally
not invented here.

## License

CCPL is source-available research software with separate commercial
licensing. Academic, educational, and non-commercial research users may
inspect, run, modify, and use the software for experiments and publication,
provided they retain the copyright and license notices and cite the associated
paper. Production deployment, commercial products or services, SaaS/API
offerings, proprietary integrations, paid services based substantially on
CCPL, and commercial redistribution require a separate commercial license.
See [`LICENSE`](LICENSE). This is not an OSI-approved open-source license.

## GitHub metadata

Suggested repository description:

> Causal RL for safety constraints under delayed consequences, combining delay-corrected Bellman targets, causal consequence attribution, state-conditioned Lagrange multipliers, and dual Q-functions.

Suggested topics: `reinforcement-learning`, `safe-reinforcement-learning`,
`constrained-reinforcement-learning`, `causal-reinforcement-learning`,
`causal-inference`, `deep-reinforcement-learning`, `machine-learning`,
`artificial-intelligence`, `neurips`, `research`.
