Metadata-Version: 2.4
Name: OMOP-MEDS
Version: 0.3.0
Summary: An ETL to convert OMOP data to the MEDS format.
Author-email: Robin van de Water <robin.vandewater@hpi.de>, Matthew McDermott <mattmcdermott8@gmail.com>
Project-URL: Homepage, https://github.com/rvandewater/OMOP_MEDS
Project-URL: Issues, https://github.com/rvandewater/OMOP_MEDS/issues
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: <3.14,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: MEDS-extract~=0.6.2
Requires-Dist: requests
Requires-Dist: beautifulsoup4
Requires-Dist: hydra-core
Requires-Dist: loguru
Requires-Dist: polars>=1.26
Requires-Dist: omop_schema
Requires-Dist: meds~=0.4.1
Requires-Dist: tqdm
Provides-Extra: dev
Requires-Dist: pre-commit<4; extra == "dev"
Requires-Dist: mdformat-admon; extra == "dev"
Provides-Extra: tests
Requires-Dist: pytest; extra == "tests"
Requires-Dist: pytest-cov; extra == "tests"
Requires-Dist: hydra-joblib-launcher; extra == "tests"
Provides-Extra: local-parallelism
Requires-Dist: hydra-joblib-launcher; extra == "local-parallelism"
Provides-Extra: slurm-parallelism
Requires-Dist: hydra-submitit-launcher; extra == "slurm-parallelism"
Provides-Extra: large-data
Requires-Dist: polars[rt64]; extra == "large-data"
Dynamic: license-file

<div align="center">
  <img src="https://github.com/rvandewater/OMOP_MEDS/blob/main/static/omop_to_meds_logo.svg?raw=true"
alt="omop-to-meds logo" height="300">
</div>

# MEDS OMOP ETL

[![PyPI - Version](https://img.shields.io/pypi/v/OMOP_MEDS)](https://pypi.org/project/OMOP_MEDS/)
[![codecov](https://codecov.io/gh/rvandewater/OMOP_MEDS/graph/badge.svg?token=RW6JXHNT0W)](https://codecov.io/gh/rvandewater/OMOP_MEDS)
[![tests](https://github.com/rvandewater/OMOP_MEDS/actions/workflows/tests.yaml/badge.svg)](https://github.com/rvandewater/OMOP_MEDS/actions/workflows/tests.yml)
[![code-quality](https://github.com/rvandewater/OMOP_MEDS/actions/workflows/code-quality-main.yaml/badge.svg)](https://github.com/rvandewater/OMOP_MEDS/actions/workflows/code-quality-main.yaml)
![python](https://img.shields.io/badge/-Python_3.12%20%7C%203.13-blue?logo=python&logoColor=white)
[![license](https://img.shields.io/badge/License-MIT-green.svg?labelColor=gray)](https://github.com/rvandewater/OMOP_MEDS#license)
[![PRs](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/rvandewater/OMOP_MEDS/pulls)
[![contributors](https://img.shields.io/github/contributors/rvandewater/OMOP_MEDS.svg)](https://github.com/rvandewater/OMOP_MEDS/graphs/contributors)
[![DOI](https://zenodo.org/badge/940565218.svg)](https://doi.org/10.5281/zenodo.19398854)
[![MEDS v0.4](https://img.shields.io/badge/MEDS-0.4-blue)](https://medical-event-data-standard.github.io/)

An ETL pipeline for transforming Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM)
datasets into MEDS using MEDS-Extract.
We gratefully acknowledge the developers of the first OMOP MEDS ETL, from which we took inspiration,
which can be found here: https://github.com/Medical-Event-Data-Standard/meds_etl.

We currently support OMOP 5.3 and 5.4 datasets. Earlier versions might work but are not tested and are perhaps (?)
not used in practice anymore. Please open pull requests if you want to add support for earlier versions.

- More information about OMOP can be found here: https://ohdsi.github.io/CommonDataModel/
- More information about MEDS can be found here: https://medical-event-data-standard.github.io/

For reading in OMOP in the right schema, we use the `omop-schema` package (developed for this ETL but can be used
elswehere), which can be found here:
https://github.com/rvandewater/omop_schema.
If your OMOP schema is non-standard (but still approximately OMOP), you should be able to use the `omop-schema` package to define your own schema and use it in this ETL.

## Setup

Requires Python `>=3.12,<3.14` (`MEDS-extract 0.6.2` does not support 3.11).

First install the package:

```bash
pip install OMOP_MEDS
```

Then:

```bash
export DATASET_NAME="Your_OMOP_Dataset_Name" # e.g. MIMIC_IV_OMOP
export OMOP_VERSION="5.3" # or 5.4
export RAW_INPUT_DIR="path/to/your/input"
export ROOT_OUTPUT_DIR="/path/to/your/output"
OMOP_MEDS raw_input_dir=$RAW_INPUT_DIR root_output_dir=$ROOT_OUTPUT_DIR
```

To try with the MIMIC-IV OMOP demo dataset (this downloads a version to your local machine), you can run:

```bash
OMOP_MEDS raw_input_dir=path/to/your/input root_output_dir=/path/to/your/output do_download=True ++do_demo=True
```

Example config for an OMOP dataset:

```yaml
dataset_name: MIMIC_IV_OMOP
raw_dataset_version: 1.0
omop_version: 5.3

urls:
  dataset:
    - https://physionet.org/content/mimic-iv-demo-omop/0.9/
    - url: EXAMPLE_CONTROLLED_URL
      username: ${oc.env:DATASET_DOWNLOAD_USERNAME}
      password: ${oc.env:DATASET_DOWNLOAD_PASSWORD}
  demo:
    - https://physionet.org/content/mimic-iv-demo-omop/0.9/
  common:
    - EXAMPLE_SHARED_URL # Often used for shared metadata files
```

Run this with:

```bash
OMOP_MEDS ++DATASET_CFG=your_config.yaml raw_input_dir=path/to/your/input root_output_dir=/path/to/your/output \
do_download=True
```

## Differences with the original meds_etl_omop

This package is designed as a more flexible and configurable alternative to the original `meds_etl_omop` package.
We make a few important choices that have impact on your downstream training and task definitions:

- We use the mapped concepts by default, which are more standardized across datasets and, for large, health systems can
    be more clean, especially if you are working with a limited tokenizer on a large dataset.
    You can still use the source concepts by setting `++prefer_source=True`.
- We use more tables than in the original `meds_etl_omop` package, which can lead to more complete patient histories.
    Watch for potential information leakage. You can change your table configs in pre_MEDS.yaml and event_configs.yaml
- This package is more resource intensive, please adjust your `n_shards` and watch your memory usage.

## Different package versions

We have different versions of the ETL pipeline, which are designed for different use cases:

```bash
pip install OMOP_MEDS[dev] # for development and testing
pip install OMOP_MEDS[tests] # for running tests
pip install OMOP_MEDS[local_parallelism] # for local parallelization with hydra-joblib-launcher
pip install OMOP_MEDS[slurm_parallelism] # for parallelization with slurm using hydra-submitit-launcher
pip install OMOP_MEDS[large_data] # for handling large datasets with polars (rt64), more than 3.2B rows
```

Run routine tests (without the downloaded e2e test makes it a lot faster, simply remove the `-k` flag to run all tests):

```bash
pytest -q -k 'not test_e2e'
```

## Pre-MEDS settings

The following settings can be used to configure the pre-MEDS steps.

```bash
OMOP_MEDS \
	root_output_dir=/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/small_demo \
	raw_input_dir=/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/full_omop \
	do_download=False ++do_overwrite=True ++limit_subjects=50
```

- `root_output_dir`: Set the root output directory.
- `raw_input_dir`: Path to the raw input directory.
- `do_download`: Set to `False` to skip downloading the dataset.
- `++do_overwrite`: Set to `True` to overwrite existing files.
- `++limit_subjects`: Limit the number of subjects to process.
- `++prefer_source`: Set to `True` to prefer source concepts over mapped concepts.
- `++join_on_visit`: Set to `True` to join person table on having any associated visits.
- `++process_notes`: Set to `True` to .

Pre-meds batching settings. This is relevant if (some of) your input tables are very large, and you want to process
them in batches. This can be useful to reduce memory usage, but it also increases the runtime, so use with caution.
The batching settings are as follows:

- `++pre_meds_chunked_tables`: Tables eligible for batched pre-MEDS processing.
- `++pre_meds_batching_row_threshold`: Row count threshold for batching.
- `++pre_meds_batch_mode`: `auto`, `per_shard`, `by_shards`, or `by_rows`.
- `++pre_meds_batch_size_shards`: Shards per batch in `by_shards` mode.
- `++pre_meds_batch_input_rows`: Max rows per batch in `by_rows` mode.

Also check out the `main.yaml` config file for more default settings and details on how to configure the pre-MEDS steps,
which can be found here:
src/OMOP_MEDS/configs/main.yaml

## MEDS-Extract settings

If you want to convert a large dataset, you can use parallelization with MEDS-transforms
(the MEDS-transformation step that takes the longest).

Using local parallelization with the `hydra-joblib-launcher` package, you can set the number of workers:

```
pip install hydra-joblib-launcher --upgrade
```

Then, you can set the number of workers as environment variable:

```bash
export N_WORKERS=8
```

Moreover, you can set the number of subjects per shard to balance the parallelization overhead based on how many
subjects you have in your dataset:

```bash
export N_SUBJECTS_PER_SHARD=100000
```

For large datasets, the pipeline keeps high `shard_events` limits by default:

```yaml
row_chunksize: 20000000000
infer_schema_length: 999999999
```

Optional local parallelism is still available via `OMOP_MEDS[local_parallelism]` and `N_WORKERS`.

## The MIMIC-IV OMOP Dataset

We use the demo dataset for MIMIC-IV in the OMOP format, which is a subset of the MIMIC-IV dataset.
This dataset downloaded from Physionet does not include the standard dictionary linking definitions but should otherwise
be functional. We also include a small sample of synthetic (LLM-generated) data in the test/demo_resources directory, which can be used for testing and development purposes.

## Advanced datetime resolution

To handle datetime resolution in a more flexible way, you can configure the `datetime_resolver` settings in your config file.
This allows you to specify which columns to use for determining the datetime of events, with support for both primary and override columns.
The primary purpose of this is to prevent information leakage by using the last edit datetime (of notes) instead of the original note datetime, if available.
The following block can be found in the pre_MEDS.yaml config file, and you can adjust the column names based on your dataset's schema:

```yaml
datetime_resolver:
  primary_datetime_col: note_datetime     # preferred; full timestamp
  primary_date_col: note_date     # fallback; promoted to 23:59:59
  override_datetime_col: xtn_note_last_edit_datetime     # optional XTN column
  override_date_col: xtn_note_last_edit_date     # optional XTN column
```

The following is an example of how the data then looks like:

| note_date  | note_datetime       | xtn_note_last_edit_date | xtn_note_last_edit_datetime |
| ---------- | ------------------- | ----------------------- | --------------------------- |
| 2023-01-01 | 2023-01-01 08:00:00 | 2023-01-01              | 2023-01-01 08:05:00         |
| 2023-01-02 | 2023-01-02 09:30:00 | 2023-01-02              | 2023-01-02 10:00:00         |

In this case it will prefer the `xtn_note_last_edit_datetime` column for the datetime of the event (if it is in fact later in time),
which can help to prevent information leakage in some cases, but it will fall back to `note_datetime`/`note_date` if the override column is not available.
See `build_preferred_event_datetime` function in the code for more details on how this works.

## Particularities

- Care site is added to the visit as text
- Add support for care_site table (visit_detail)

## Citation

If you use this ETL for your research, please use the citation link in Github, which points to the Zenodo DOI,
and which is also included below:

```
@software{van_de_Water_OMOP_MEDS_ETL_2025,
author = {van de Water, Robin Philippus},
doi = {10.5281/zenodo.15132444},
license = {MIT},
month = feb,
title = {{OMOP\_MEDS ETL}},
url = {https://github.com/rvandewater/OMOP_MEDS},
year = {2025}
}
```
