Metadata-Version: 2.4
Name: rlprobe
Version: 0.1.0
Summary: Flight recorder + doctor for RL fine-tuning runs (GRPO/PPO): record everything, diagnose failures early.
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/Nick-2908/rlprobe
Project-URL: Documentation, https://nick-2908.github.io/rlprobe/
Project-URL: Repository, https://github.com/Nick-2908/rlprobe
Project-URL: Changelog, https://github.com/Nick-2908/rlprobe/blob/main/CHANGELOG.md
Keywords: reinforcement-learning,rlhf,grpo,ppo,llm,fine-tuning,diagnostics,observability,trl
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.5
Requires-Dist: pyarrow>=14
Requires-Dist: numpy>=1.24
Requires-Dist: typer>=0.9
Requires-Dist: rich>=13
Provides-Extra: trl
Requires-Dist: trl<1.8,>=1.7; extra == "trl"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5; extra == "docs"
Dynamic: license-file

# rlprobe

**Flight recorder + doctor for RL fine-tuning runs (GRPO / PPO and variants).**

RL fine-tuning fails in ways that are slow, silent, and expensive: a run can look
healthy on its loss curves for hours while the policy quietly collapses (entropy → 0),
reward-hacks (padding answers for length), or escapes its reference model (KL blowup).
rlprobe records a complete, replayable account of a training run at near-zero
overhead, then diagnoses it against a catalog of 14 known failure signatures —
live during training (with optional auto-halt) or post-hoc.

```python
from rlprobe.integrations.trl import attach

trainer = GRPOTrainer(...)        # your existing setup
attach(trainer, "runs/my-run")    # ← one line
trainer.train()
```

```console
$ rlprobe doctor runs/my-run
CRITICAL ENTROPY_COLLAPSE | onset ~step 212 | confidence 90%
entropy fell from ~2.31 to ~0.048 — the policy is (near-)deterministic and
exploration is dead, while reward has plateaued (classic collapse pair)

try:
  • add/increase an entropy bonus
  • raise the sampling temperature during rollouts
  • check whether the reward is saturated (nothing left to explore for)
```

## Install

```bash
pip install rlprobe[trl]      # with the TRL integration
pip install rlprobe           # core only (manual API + doctor; no torch needed)
```

## What it does

- **Flight recorder** — per-step scalar metrics, per-sample forensics (prompts,
  completions, rewards, advantages, GRPO group ids), discrete events, and full run
  config, written to a self-contained run directory. Async writer, never blocks
  training, flushes on crash/SIGTERM, &lt;2% step-time overhead, verified at
  100k-step scale.
- **Doctor** — deterministic detectors (trend tests, sustained thresholds,
  changepoints — no ML magic) for 14 failure signatures: KL blowup, entropy
  collapse, length reward-hacking, format gaming, clip saturation, degenerate GRPO
  groups, advantage collapse, value divergence, NaN/grad pathology, truncation
  starvation, repetition spirals, reward-std death, positional KL asymmetry,
  throughput rot. Ranked findings with onset step, evidence, and remediations.
- **Live guardian** — `rlprobe watch` tails a growing run and alerts (webhook
  support); `attach(trainer, run_dir, halt=True)` stops training automatically on
  critical, high-confidence findings.
- **Post-mortem suite** — `rlprobe report` (self-contained HTML), `rlprobe diff`
  (where did run B diverge from A?), `rlprobe samples` / `rlprobe show` (forensic
  queries down to individual completions).
- **Framework-agnostic core** — the TRL callback is a thin layer over a manual API
  (`Recorder.log_step / log_samples / log_event`) that any training loop can call.

No GPU needed to try it — the synthetic run generator fabricates realistic
pathological recordings:

```bash
rlprobe synth runs/demo --pathology entropy_collapse
rlprobe doctor runs/demo
rlprobe report runs/demo
```

## Documentation

Full docs live in [`docs/`](docs/index.md) (mkdocs; `pip install -e ".[docs]" &&
mkdocs serve`): [quickstart](docs/quickstart.md), the
[failure-signature catalog](docs/signatures/index.md) — each signature explained as
RL-debugging documentation in its own right — the
[recording format](docs/recording-format.md), and the
[API & CLI reference](docs/api.md).

## Development

```bash
pip install -e ".[dev,trl]"
pytest
ruff check src tests examples benchmarks
```

Design history and phase-by-phase implementation notes: [HANDOFF.md](HANDOFF.md).

## License

Apache-2.0
