Metadata-Version: 2.4
Name: pymimir-rl
Version: 0.3.0b4
Summary: Reinforcement Learning (RL) package for Mimir.
Author-email: Simon Stahlberg <simon.stahlberg@gmail.com>
License-Expression: GPL-3.0-or-later
Project-URL: Homepage, https://github.com/simon-stahlberg/mimir-rl
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.6.0
Requires-Dist: pymimir==0.14.0b5
Requires-Dist: networkx>=3.5
Provides-Extra: dev
Requires-Dist: pymimir-rgnn==0.3.0b4; extra == "dev"
Requires-Dist: pytest>=6.0; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# Mimir-RL

Mimir-RL is a Python library that implements RL algorithms using PyTorch and PyTorch RL that are tightly integrated with Mimir.

## Dead-end detection

Pass an optional state/goal detector callable to a trajectory sampler:

```python
import pymimir as mm
from pymimir_rl import BoltzmannTrajectorySampler, CachedDeadEndDetector

detector = CachedDeadEndDetector(mm.H2DeadEndDetector)
sampler = BoltzmannTrajectorySampler(
    model, reward_function, temperature=0.5, dead_end_detector=detector,
)
```

For h², construct problems with `generator="grounded"`. The cache creates one
native detector per problem. `Trajectory` calls it after sampling, in state order,
and propagates each proof forward for the same goal. Existing reward-function
proofs and actionless non-goal states also provide dead-end evidence.

Detected states with applicable actions do not terminate or prune rollouts.
`Transition.successor_is_dead_end` is separate from `is_terminal`, and ordinary
rewards are preserved. Optimizers assign fixed dead-end targets without
bootstrapping from those successors. Hindsight cloning recomputes labels for its
new goal using the same detector cache.

`OffPolicyAlgorithm` accepts `dead_end_replay_buffer` and `hindsight_replay_buffer`.
The former receives a suffix of each trajectory containing a proven dead state.
By default it starts with the transition entering the first proven dead state.
An earlier state whose recorded maximum Q-value is at most
`-10000 * dead_end_q_factor` moves the cutoff to the transition entering that
state. `dead_end_q_factor` defaults to `0.25` (threshold `-2500`) and must be in
`(0, 1]`. Nonfinite predictions do not move the cutoff. Selection uses the
recorded maximum over all applicable actions and requires no additional inference.

The Q-based cutoff only selects replay experience; it does not create dead-end
labels or change targets. All sampled trajectories remain intact for hindsight
refinement. Unproven horizon cutoffs do not qualify for the dead-end buffer,
even if their Q-values are low.
