Metadata-Version: 2.4
Name: closed-loop-training
Version: 0.4.0
Summary: ReMiT + RLPT closed-loop flywheel: continuous self-improving LLM training
Home-page: https://github.com/KingLabsA/llm-training-platform
Author: KingLabsA
Author-email: KingLabsA <kinglabsa@users.noreply.github.com>
License: Apache 2.0
Project-URL: Homepage, https://github.com/KingLabsA/llm-training-platform
Project-URL: Source, https://github.com/KingLabsA/llm-training-platform
Project-URL: Bug Tracker, https://github.com/KingLabsA/llm-training-platform/issues
Project-URL: ReMiT Paper, https://arxiv.org/abs/2602.03075
Project-URL: RLPT Paper, https://aclanthology.org/2026.acl-long
Keywords: llm,training,reinforcement-learning,remit,rlpt,self-improvement,closed-loop,deep-learning
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: torch>=2.0.0
Requires-Dist: rich>=13.0.0
Requires-Dist: nltk>=3.8
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: black>=23.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Provides-Extra: transformers
Requires-Dist: transformers>=4.30.0; extra == "transformers"
Requires-Dist: datasets>=2.10.0; extra == "transformers"
Provides-Extra: tui
Requires-Dist: rich>=13.0.0; extra == "tui"
Dynamic: license-file

# Closed-Loop Training System — ReMiT + RLPT Flywheel

```
╔══════════════════════════════════════════════════════════╗
║     ██████╗██╗     ██████╗ ███████╗██████╗ ██╗          ║
║    ██╔════╝██║    ██╔═══██╗██╔════╝██╔══██╗██║          ║
║    ██║     ██║    ██║   ██║█████╗  ██║  ██║██║          ║
║    ██║     ██║    ██║   ██║██╔══╝  ██║  ██║██║          ║
║    ╚██████╗██████╗╚██████╔╝███████╗██████╔╝███████╗     ║
║     ╚═════╝╚═════╝ ╚═════╝ ╚══════╝╚═════╝ ╚══════╝     ║
║                                                          ║
║         L  O  O  P        T  R  A  I  N  I  N  G         ║
║                                                          ║
║         ReMiT + RLPT  |  Continuous Self-Improvement    ║
╚══════════════════════════════════════════════════════════╝
```

**Author:** KingLabsA  
**GitHub:** https://github.com/KingLabsA/llm-training-platform  
**HuggingFace:** https://huggingface.co/King3Djbl | https://huggingface.co/fableforge-ai  
**Ollama:** https://ollama.com/FableForge-AI  
**License:** Apache 2.0

**Continuous self-improving training: ReMiT token reweighting + RLPT next-segment reasoning, chained in iterative flywheel.**

---

## Quickstart

```bash
pip install closed-loop-training  # or: pip install -e .
closedloop train --model meta-llama/Llama-3.2-1B --corpus data.jsonl
# Try dummy mode first:
closedloop train --model dummy --corpus data.json --iterations 3 --tui
```

---

## Features

- **ReMiT Mid-Training** — Dynamic token reweighting using frozen RL reference. Tokens where the RL model assigns higher probability get upweighted. Soft sigmoid weighting clipped to [0.8, 1.2]. No token discarding.
- **RLPT Post-Training** — Next-segment reasoning on pre-training data. GRPO-style optimization with generative reward model (GRM). No human annotations needed.
- **Closed-Loop Flywheel** — Post-trained model becomes the RL reference for the next cycle. Each iteration bootstraps from the previous. Convergence detection stops when improvement < 0.5%.
- **Live TUI** — Rich terminal dashboard with real-time ReMiT/RLPT metrics, progress bars, and iteration history table.
- **Dummy Mode** — Test the full pipeline without GPU or real models. Use `--model dummy`.
- **Checkpoint & Resume** — Save & resume from any iteration. History saved as JSON.
- **Convergence Detection** — Automatic early stopping when per-iteration improvement falls below threshold.

---

## Architecture: Closed-Loop Flywheel

```
                    ┌─────────────────────────────────────┐
                    │         Closed-Loop Iteration        │
                    │                                     │
                    │  ┌──────────┐    ┌──────────────┐   │
                    │  │  ReMiT   │    │     RLPT     │   │
                    │  │  Mid-    │───>│  Post-       │   │
                    │  │ Training │    │  Training    │   │
                    │  └────┬─────┘    └──────┬───────┘   │
                    │       │                 │           │
                    │       v                 v           │
                    │  ┌─────────────────────────────┐    │
                    │  │     RL Reference Model      │    │
                    │  │  (becomes teacher next cycle)│    │
                    │  └──────────────┬──────────────┘    │
                    │                 │                   │
                    │                 v                   │
                    │  ┌─────────────────────────────┐    │
                    │  │     Gap Analysis +          │    │
                    │  │     Domain Targeting        │    │
                    │  └─────────────────────────────┘    │
                    │                 │                   │
                    └─────────────────┼───────────────────┘
                                      │
                                      v
                          ┌─────────────────────┐
                          │  Improved Base Model │
                          │  (fed to next cycle) │
                          └─────────────────────┘
```

---

## How It Works

1. **Initialize** — Load a base model and training corpus. If no RL reference exists yet, skip ReMiT weighting on the first iteration.
2. **ReMiT Mid-Training** — For each batch, compute per-token loss discrepancy between the base model and the frozen RL reference. Generate importance weights via centered sigmoid clipped to [0.8, 1.2]. Train on weighted NLL.
3. **RLPT Post-Training** — Segment corpus text into sentence units. For each segment, prompt the model to predict the next segment. Score predictions with a Generative Reward Model (GRM). Optimize via GRPO (group-relative advantages with KL penalty).
4. **Reference Update** — Set the post-trained model as the new RL reference (frozen). The next ReMiT cycle will compare against this improved reference.
5. **Convergence Check** — Measure ReMiT loss improvement. If below threshold, stop.
6. **Iterate** — Repeat. Each cycle the model learns from its own improvements.

---

## CLI Reference

### `closedloop train`

| Flag | Default | Description |
|------|---------|-------------|
| `--model` | `meta-llama/Llama-3.2-1B` | HuggingFace model name or `dummy` |
| `--corpus` | (required) | Path to JSON, JSONL, or text corpus |
| `--iterations` | `3` | Number of closed-loop flywheel iterations |
| `--mid-train-steps` | `500` | ReMiT steps per iteration |
| `--rl-train-steps` | `500` | RLPT steps per iteration |
| `--remit-batch-size` | `8` | Batch size for ReMiT |
| `--rlpt-batch-size` | `4` | Batch size for RLPT |
| `--max-samples` | `10000` | Max corpus samples to load |
| `--domain` | `None` | Target domain name |
| `--max-seq-length` | `2048` | Maximum sequence length |
| `--lr` | `3e-4` | Base learning rate |
| `--remit-lr` | `3e-4` | ReMiT learning rate |
| `--rlpt-lr` | `1e-6` | RLPT learning rate |
| `--output-dir` | `./closed_loop_output` | Output directory |
| `--log-interval` | `10` | Steps between logging |
| `--device` | `auto` | `auto`, `cuda`, or `cpu` |
| `--tui` | (flag) | Enable live TUI display |

### `closedloop resume`

| Flag | Default | Description |
|------|---------|-------------|
| `--checkpoint` | (required) | Path to `.pt` checkpoint |
| `--output-dir` | checkpoint dir | Output directory |
| `--iterations` | from checkpoint | Override remaining iterations |
| `--tui` | (flag) | Enable live TUI display |

### TUI Mode vs Non-TUI Mode

- **TUI mode** (`--tui`): Rich live dashboard with real-time ReMiT/RLPT metrics, progress bars, iteration history table, phase indicators, and elapsed time. Ideal for demos, monitoring, and debugging.
- **Non-TUI mode** (default): Print-based logging. Each step logs loss/metrics. Compact output suitable for scripts, CI/CD, and headless environments.

### Environment Variables

| Variable | Purpose |
|----------|---------|
| `PYTHONPATH` | Should include project root for imports |

---

## Citation

```bibtex
@misc{kinglabsa2026closedloop,
  author = {KingLabsA},
  title = {Closed-Loop Training System: ReMiT + RLPT Flywheel},
  year = {2026},
  publisher = {GitHub},
  url = {https://github.com/KingLabsA/llm-training-platform}
}

@article{huang2026remit,
  title={ReMiT: Reinforcement Learning Guided Mid-Training for Large Language Models},
  author={Huang et al.},
  journal={arXiv preprint arXiv:2602.03075},
  year={2026}
}

@inproceedings{li2026rlpt,
  title={RLPT: Reinforcement Learning on Pre-Training Data for Next-Segment Reasoning},
  author={Li et al.},
  booktitle={ACL 2026},
  year={2026}
}
```

---

## License

Apache 2.0 — See LICENSE file.

---

*Built by KingLabsA. ReMiT paper: arXiv 2602.03075. RLPT paper: ACL 2026.*
