Metadata-Version: 2.4
Name: ConMedRL
Version: 1.1.0
Summary: An Offline Constrained Reinforcement Learning Toolkit for Medical Decision Making
Author-email: Maotong Sun <maotong.sun@tum.de>, Jingui Xie <jingui.xie@tum.de>
Maintainer-email: Maotong Sun <maotong.sun@tum.de>, Jingui Xie <jingui.xie@tum.de>
License-Expression: MIT
Project-URL: Homepage, https://github.com/smt970913/ConMED-RL
Project-URL: Documentation, https://github.com/smt970913/ConMED-RL#readme
Project-URL: Repository, https://github.com/smt970913/ConMED-RL
Project-URL: Bug Tracker, https://github.com/smt970913/ConMED-RL/issues
Keywords: reinforcement learning,constrained reinforcement learning,offline reinforcement learning,clinical decision support,healthcare,ICU,critical care,machine learning,artificial intelligence
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Healthcare Industry
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: flask==2.2.2
Requires-Dist: scikit-learn>=1.2.1
Requires-Dist: torch>=1.8.0
Requires-Dist: numpy>=1.23.5
Requires-Dist: pandas>=1.5.3
Requires-Dist: tqdm==4.64.1
Requires-Dist: celery==5.3.6
Requires-Dist: gunicorn==20.1.0
Requires-Dist: pillow==9.4.0
Requires-Dist: requests==2.31.0
Requires-Dist: dask[complete]>=2023.1.0
Requires-Dist: threadpoolctl>=3.1.0
Requires-Dist: joblib>=1.2.0
Requires-Dist: jupyter==1.0.0
Requires-Dist: jupyterlab==4.0.5
Requires-Dist: notebook==7.0.2
Requires-Dist: matplotlib==3.7.1
Requires-Dist: seaborn==0.12.2
Requires-Dist: plotly==5.17.0
Provides-Extra: dev
Requires-Dist: pytest>=6.0; extra == "dev"
Requires-Dist: pytest-cov>=2.0; extra == "dev"
Requires-Dist: black>=21.0; extra == "dev"
Requires-Dist: flake8>=3.8; extra == "dev"
Requires-Dist: jupyter>=1.0; extra == "dev"
Requires-Dist: notebook>=6.0; extra == "dev"
Provides-Extra: models
Requires-Dist: torch>=1.8.0; extra == "models"
Requires-Dist: torchvision>=0.9.0; extra == "models"
Provides-Extra: viz
Requires-Dist: matplotlib>=3.3.0; extra == "viz"
Requires-Dist: seaborn>=0.11.0; extra == "viz"
Requires-Dist: plotly>=5.0.0; extra == "viz"
Provides-Extra: data
Requires-Dist: pyarrow>=10.0; extra == "data"
Requires-Dist: d3rlpy>=2.0; extra == "data"
Provides-Extra: llm
Requires-Dist: openai>=1.0; extra == "llm"
Requires-Dist: anthropic>=0.20; extra == "llm"
Dynamic: license-file

# ConMED-RL: Offline Constrained RL for Critical-Care Research

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![PyPI version](https://badge.fury.io/py/conmedrl.svg)](https://badge.fury.io/py/conmedrl)

**ConMED-RL** is an **Offline Constrained Reinforcement Learning (OCRL)**
toolkit for retrospective critical-care research. It combines ICU data
processing, discrete and continuous constrained policy learning, separate
Fitted Q Evaluation (FQE) models for objectives and constraints, and
research-interface examples.

This toolkit builds upon our research on OCRL applications in critical care: a published study in *IISE Transactions on Healthcare Systems Engineering* addressing ICU discharge decision-making, and ongoing work under revision in *Health Care Management Science* on ICU extubation decision-making.

## 🚀 Quick Start

### Installation

Install ConMED-RL using pip:

```bash
pip install conmedrl

# Optional d3rlpy/Parquet or LLM integrations
pip install "conmedrl[data]"
pip install "conmedrl[llm]"
```

### Brief Usage Instruction
- Set Hyparameter
```python
from ConMedRL import RLConfigurator, RLConfig_custom 

dm_configuration = RLConfigurator()
dm_configuration.choose_config_method()
dm_configuration.config.memory_capacity
```

- Load data into the training procedure (discharge decision-making case)

```python
from ConMedRL import FQE, FQI, TrainDataLoader, ValTestDataLoader

# Load your clinical data
train_data_loader = TrainDataLoader(cfg = dm_configuration.config, 
                                    outcome_table = outcome_table_train, 
                                    state_var_table = state_var_table, 
                                    terminal_state = terminal_state)

train_data_loader.data_buffer_train(action_name = 'discharge_action', 
                                    done_condition = None,
                                    num_constraint = 2)

val_data_loader = ValTestDataLoader(cfg = dm_configuration.config, 
                                    outcome_table_select = outcome_table_val_select, 
                                    state_var_table_select = state_var_table_val_select, 
                                    outcome_table = outcome_table_val, 
                                    state_var_table = state_var_table_val, 
                                    terminal_state = terminal_state)

val_data_loader.data_buffer(action_name = 'discharge_action', 
                            done_condition = None,
                            num_constraint = 2)
```

- Initialize the training of OCRL-based policy learning framework (discharge decision-making case)
```python
# Set the training of the model
ocrl_training = RLTraining(cfg = dm_configuration.config, 
                           state_dim = state_var_table.shape[1], 
                           action_dim = 2, 
                           train_data_loader = train_data_loader.data_torch_loader_train,
                           val_data_loader = val_data_loader.data_torch_loader)

# Building the FQI agent
fqi_agent = ocrl_training.fqi_agent_config(...) 

# Building the FQE agents
fqe_agent_obj = ocrl_training.fqe_agent_config(...) 

fqe_agent_con_0 = ocrl_training.fqe_agent_config(...) 

fqe_agent_con_1 = ocrl_training.fqe_agent_config(...) 

ocrl_training.train(agent_fqi = fqi_agent, 
                    agent_fqe_obj = fqe_agent_obj, 
                    agent_fqe_con_list = [fqe_agent_con_0, fqe_agent_con_1], 
                    constraint = True,
                    save_num = 100,
                    z_value = 1.96)
```

## 📦 Core Components

### Offline Constrained Reinforcement Learning (OCRL) Algorithms

- **Fitted Q-Evaluation (FQE)**: Policy evaluation method for offline data
- **Fitted Q-Iteration (FQI)**: Value-based offline RL algorithm 
- **Replay Buffer**: Efficient data management for training
- **Custom RL Configurator**: Flexible configuration for different clinical scenarios

### Data Processing

- **TrainDataLoader**: Handles training data preparation and batch generation
- **ValTestDataLoader**: Manages validation and testing data processing
- Support for custom done conditions and constraint cost functions
- Unified MIMIC-IV/SICdb preprocessing and an approved declarative adapter for
  MIMIC-like datasets such as NWICU
- Review-first, metadata-only LLM planning for custom discrete or continuous
  clinical OCRL tasks; generated code is never executed
- CSV, Parquet, d3rlpy, and de-identified FHIR R4 NDJSON exchange outputs
- Auditable patient withdrawal with dataset/model version tracking,
  stale-model rejection, and caller-controlled fresh retraining

See
[`DATA_PROCESSING.md`](https://github.com/smt970913/ConMED-RL/blob/main/DATA_PROCESSING.md)
and the generic-data, FHIR, and exact-unlearning example notebooks for the full
safety and interoperability contract.

## 🏥 Key Features

- **Offline Learning**: Train models on historical clinical data without online interaction
- **Constraint Handling**: Built-in support for clinical safety/efficiency constraints
- **Flexible Architecture**: Easy integration with existing clinical datasets
- **Medical Focus**: Specifically designed for critical care decision-making scenarios
- **Research-Backed**: Based on peer-reviewed methodologies

## 📊 Use Cases

ConMED-RL has been successfully applied to:

- **ICU Discharge Decision-Making**: Optimizing timing and safety of patient discharge
- **ICU Mechanical Ventilation Weaning**: Supporting extubation decisions with constraint satisfaction
- **Multi-Constraint Clinical Decisions**: Balancing multiple clinical objectives requirements

## 🔧 Hyperparameter Configuration

```python
from ConMedRL import RLConfigurator

configuration = RLConfigurator()
configuration.choose_config_method()
rl_config = configuration.config
```

For a non-interactive, fully specified configuration and a short training run,
see `Experiment Notebook/Example_ConMedRL_End_to_End_Workflow.ipynb`.

## Data Preprocessing

ConMED-RL expects data in MDP format suitable for offline RL training:
- **State Table**: Physiological measurements and clinical variables
- **Outcome Table**: Actions, costs/rewards, and terminal indicators

See the [full documentation](https://github.com/smt970913/ConMED-RL) for data preprocessing examples.

## 📖 Documentation and Examples

For comprehensive guides, tutorials, and examples:

- **GitHub Repository**: [https://github.com/smt970913/ConMED-RL](https://github.com/smt970913/ConMED-RL)
- **Example Notebooks**: Interactive Jupyter notebooks for MIMIC-IV datasets
- **Web Application Demo**: Research interface for model-integration checks

## 🔬 Research and Citation

This toolkit is based on research published in academic journals. If you use ConMED-RL in your research, please cite:

```bibtex
@misc{sun2025comedRL,
  author       = {Maotong Sun and Jingui Xie},
  title        = {ConMED-RL: An OCRL-Based Toolkit for Medical Decision Support},
  year         = {2025},
  howpublished = {\url{https://github.com/smt970913/ConMED-RL}},
  note         = {Accessed: xxxx-xx-xx},
}
```

## 🛠️ Requirements

- Python 3.8 or higher
- PyTorch
- NumPy
- Pandas
- scikit-learn

All dependencies are automatically installed with the package.

## 📝 License

This project is licensed under the MIT License - see the [LICENSE](https://github.com/smt970913/ConMED-RL/blob/main/LICENSE) file for details.

## 👥 Authors and Contact

- **Maotong Sun** - maotong.sun@tum.de
- **Jingui Xie** - jingui.xie@tum.de

School of Management, Technical University of Munich

## 🤝 Contributing

We welcome contributions! For major changes, please open an issue first to discuss what you would like to change.

For development setup and contributing guidelines, visit the [GitHub repository](https://github.com/smt970913/ConMED-RL).

## 🔗 Links

- **PyPI**: [https://pypi.org/project/conmedrl/](https://pypi.org/project/conmedrl/)
- **GitHub**: [https://github.com/smt970913/ConMED-RL](https://github.com/smt970913/ConMED-RL)
- **Issues**: [https://github.com/smt970913/ConMED-RL/issues](https://github.com/smt970913/ConMED-RL/issues)
- **Documentation**: [https://github.com/smt970913/ConMED-RL#readme](https://github.com/smt970913/ConMED-RL#readme)

## ⚠️ Disclaimer

This toolkit is intended for retrospective research and software evaluation.
It is not a medical device and must not be used to direct patient care without
appropriate local validation, governance, regulatory review, and clinician
oversight.

---

**Keywords**: reinforcement learning, constrained reinforcement learning, offline reinforcement learning, clinical decision support, healthcare, ICU, critical care, machine learning, artificial intelligence
