Metadata-Version: 2.4
Name: imperandi
Version: 0.1.0
Summary: Build analysis-ready CT and MR cohorts from heterogeneous DICOM data
Author: Diana Mandache
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/dmandache/IMPERANDI
Project-URL: Repository, https://github.com/dmandache/IMPERANDI
Project-URL: Documentation, https://imperandi.readthedocs.io/
Project-URL: Issues, https://github.com/dmandache/IMPERANDI/issues
Keywords: medical-imaging,dicom,nifti,medical-image-processing,healthcare-ai,mri,ct
Requires-Python: <3.13,>=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: scipy
Requires-Dist: pandas
Requires-Dist: unidecode
Requires-Dist: tqdm
Requires-Dist: pydicom
Requires-Dist: dicom2nifti
Requires-Dist: nibabel
Requires-Dist: pyyaml
Provides-Extra: segment
Requires-Dist: TotalSegmentator; extra == "segment"
Requires-Dist: scikit-image; extra == "segment"
Requires-Dist: xgboost; extra == "segment"
Requires-Dist: SimpleITK; extra == "segment"
Provides-Extra: viewer
Requires-Dist: ipython; extra == "viewer"
Requires-Dist: matplotlib; extra == "viewer"
Requires-Dist: ipykernel; extra == "viewer"
Requires-Dist: ipympl; extra == "viewer"
Requires-Dist: ipywidgets; extra == "viewer"
Requires-Dist: panel; extra == "viewer"
Requires-Dist: param; extra == "viewer"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: black[jupyter]; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Provides-Extra: slow
Requires-Dist: idc-index; extra == "slow"
Provides-Extra: all
Requires-Dist: imperandi[segment,viewer]; extra == "all"
Provides-Extra: all-dev
Requires-Dist: imperandi[all,dev,slow]; extra == "all-dev"
Dynamic: license-file

# **IM**aging **PRE**processing **A**nd **N**ormalization for **D**iagnostic **I**nteroperability

![image](https://raw.githubusercontent.com/dmandache/IMPERANDI/main/static/imperandi-logo.png)

![Python](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12-blue)
[![Documentation](https://readthedocs.org/projects/imperandi/badge/?version=latest)](https://imperandi.readthedocs.io/en/latest/)
[![Code style](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
![Linting](https://img.shields.io/badge/lint-ruff-red)
![Tests](https://github.com/dmandache/IMPERANDI/actions/workflows/tests.yml/badge.svg?branch=main)
[![codecov](https://codecov.io/gh/dmandache/IMPERANDI/branch/main/graph/badge.svg)](https://codecov.io/gh/dmandache/IMPERANDI)

IMPERANDI is a Python framework and CLI for transforming heterogeneous DICOM collections into analysis-ready CT and MR imaging cohorts. It standardizes identifiers, curates imaging series and volume-level metadata, converts selected volumes to NIfTI, and supports downstream segmentation, contrast-phase identification, radiomics extraction, and quality control within a coherent, reproducible pipeline.

## Disclaimers

### Research use only

IMPERANDI is research software and is not a medical device. Outputs are not intended for clinical diagnosis or treatment decisions without independent validation.

### DICOM privacy

IMPERANDI should not be assumed to de-identify input DICOM data. Users are responsible for appropriate anonymization and data-governance procedures before processing or sharing data.

### Funding

This work performed under the RHU OPERANDI project was supported in part by the French National Research Agency (Agence Nationale de la Recherche, ANR) as its 3rd PIA, integrated to France 2030 plan under reference _ANR-21-RHUS-0012._

## Why IMPERANDI matters

- Reduces manual data wrangling by turning raw DICOM trees into structured cohort tables.
- Improves reproducibility with explicit CSV outputs at every stage and deterministic ID logic.
- Improves reliability on real hospital exports with archive support, failure tracking, and resumable workflows.
- Keeps adoption practical in secure environments with a lightweight Python-first toolchain.

## Current framework functionalities

### 1) Ingest and harmonize imaging metadata (`parse` + `clean` = `ingest`)

- Scans DICOM files from folders, globbed roots, and nested archives (`.zip`, `.tar`, `.tar.gz`, `.tgz`).
- Extracts selected DICOM header tags into a raw metadata table (`dicom_index.csv`).
- Builds stable patient/study/series identifiers from tags, folder structure, or hybrid fallback rules.
- Applies manifest-driven hooks for patient-key standardization and derived columns.
- Cleans and curates cohorts by filtering modality/noise patterns, localizers, non-target anatomy, non-axial acquisitions, and implausible scan geometry.
- Aggregates slices into robust volume-level records and computes exam/acquisition ordering.

Impact: turns fragmented acquisition data into a consistent cohort backbone that downstream models and analytics can trust.

### 2) Convert DICOM volumes to NIfTI (`convert`)

- Converts curated DICOM volume rows to NIfTI in parallel using `dicom2nifti`.
- Preserves source-to-output traceability in a CSV (`nifti_path` per row).
- Handles archive-backed DICOM paths transparently via on-demand materialization.
- Writes explicit conversion error tables without aborting the whole run.

Impact: creates a standardized imaging representation for model training, segmentation, and feature extraction at scale.

### 3) Configurable segmentation (`segment`)

- Runs configurable task pipelines (default backend: TotalSegmentator).
- Dispatches CT and MRI volumes to separate manifest-defined
  TotalSegmentator model lists.
- Adds optional post-processing (mask merge, closing, hole filling, largest connected component).
- Uses multiprocessing with timeout controls and produces warning/error tracking CSVs.

Impact: converts CT and MR volumes into ready-to-use anatomical/tumor masks
with operational safeguards for large cohort processing.

### 4) Contrast phase curation (`phase`)

- Resolves an ordered YAML-defined fallback chain across explicit ontology,
  metadata rules, and TotalSegmentator prediction.
- Restricts the bundled TotalSegmentator phase predictor to CT by default.
- Appends canonical `phase` provenance and `totalseg_*` prediction fields when
  model inference is needed.
- Captures per-row failures into dedicated error outputs.

Impact: enables phase-aware stratification and analysis without manual review of every study.

### 5) Radiomics feature extraction (`radiomics`)

- Extracts PyRadiomics features for organ and tumor regions from CT/MRI + masks.
- Includes a organ-minus-tumor extraction path for cleaner parenchyma characterization.
- Supports optional cohort filtering controls and error-aware output generation.
- Supports PyRadiomics parameterization from either `--pyradiomics_settings /path/to/Params.yaml` or manifest `radiomics` settings.

Impact: accelerates feature exctraction for prognostic and response modeling pipelines.

### 6) Interactive quality control viewer (Jupyter)

- Provides a 3D CT/MRI + mask viewer for cohort navigation and quick visual QA.
- Uses HU windows for CT and percentile windows (1st–99th by default) for MRI.
- Supports patient/date/modality/phase exploration, mask overlays, window presets, and keyboard navigation.

Impact: shortens the feedback loop between pipeline outputs and clinical/imaging validation.

![image](https://raw.githubusercontent.com/dmandache/IMPERANDI/main/static/viewer-demo.png)

## CLI overview

IMPERANDI ships a single CLI with these subcommands:

- `parse`: scan DICOMs and build metadata index tables.
- `clean`: filter and normalize parsed metadata.
- `ingest`: run `parse` then `clean`.
- `convert`: convert indexed DICOM volumes to NIfTI.
- `segment`: run configurable segmentation on NIfTI volumes (requires _TotalSegmentator_, install with `.[segment]`).
- `phase`: resolve canonical contrast phase; a TotalSegmentator fallback requires
  `.[segment]`.
- `radiomics`: extract radiomics features from NIfTI volumes and masks (requires a separate PyRadiomics installation; see below).

Get help:

```bash
imperandi --help
imperandi parse --help
imperandi clean --help
imperandi ingest --help
imperandi convert --help
imperandi segment --help
imperandi phase --help
imperandi radiomics --help
```

## Install

Base install:

```bash
python -m pip install -e .
```

Segmentation dependencies:

```bash
python -m pip install -e ".[segment]"
```

Radiomics dependencies (requires Git):

PyRadiomics is installed separately from Git to keep IMPERANDI's package metadata
compatible with PyPI. This extra step also applies to `[all]` and `[all-dev]`.

```bash
python -m pip install "pyradiomics @ git+https://github.com/AIM-Harvard/pyradiomics.git@master"
```

Notebook and web quality-control viewers:

```bash
python -m pip install -e ".[viewer]"
```

Development and test tooling:

```bash
python -m pip install -e ".[dev]"
```

Enable tracked git hooks (recommended):

```bash
git config core.hooksPath .githooks
```

With hooks enabled, `git push` strips output/execution state from changed `*.ipynb` files, stages those changes, and stops once so you can commit the cleaned notebooks.

Install all runtime features:

```bash
python -m pip install -e ".[all]"
python -m pip install "pyradiomics @ git+https://github.com/AIM-Harvard/pyradiomics.git@master"
```

Install all runtime features, development tools, and slow-test dependencies:

```bash
python -m pip install -e ".[all-dev]"
python -m pip install "pyradiomics @ git+https://github.com/AIM-Harvard/pyradiomics.git@master"
```

Optional Jupyter kernel setup:

```bash
python -m ipykernel install --user --name imperandi310 --display-name "IMPERANDI (Python 3.10)"
```

## Quickstart

Run ingest (parse + clean):

```bash
imperandi ingest \
  --root_path /path/to/dicom \
  --output_dir /path/to/output \
  --manifest generic
```

Convert to NIfTI:

```bash
imperandi convert \
  --csv_path /path/to/output/dicom_index_clean.csv \
  --output_dir /path/to/nifti_root \
  --csv_path_out /path/to/output/nifti_index.csv
```

Run segmentation:

```bash
imperandi segment \
  --csv_path /path/to/output/nifti_index.csv \
  --csv_path_out /path/to/output/nifti_index_segmented.csv
```

Curate contrast phase:

```bash
imperandi phase \
  --csv_path /path/to/output/nifti_index_segmented.csv \
  --csv_path_out /path/to/output/nifti_index_phased.csv \
  --manifest generic
```

Extract radiomics:

```bash
imperandi radiomics \
  --csv_path /path/to/output/nifti_index_phased.csv \
  --csv_path_out /path/to/output/nifti_index_radiomics.csv
```

Extract radiomics with explicit PyRadiomics YAML settings:

```bash
imperandi radiomics \
  --csv_path /path/to/output/nifti_index_phased.csv \
  --pyradiomics_settings /path/to/Params.yaml \
  --csv_path_out /path/to/output/nifti_index_radiomics.csv
```

Use manifest-defined radiomics settings:

```bash
imperandi radiomics \
  --csv_path /path/to/output/nifti_index_phased.csv \
  --manifest generic \
  --csv_path_out /path/to/output/nifti_index_radiomics.csv
```

If both `--manifest` and `--pyradiomics_settings` are provided, IMPERANDI warns and
prefers manifest `radiomics` settings when that section exists.

## Core outputs

- `parse`:
  - `dicom_index.csv` (resolved IDs and selected DICOM tags)
  - optional `dicom_tags_snapshot.ndjson` (full recursive tags on a sampled subset, via `--snapshot_tags`)
- `clean`:
  - cleaned cohort table (default `<input>_clean.csv`)
- `convert`:
  - NIfTI-enriched cohort table (`nifti_index.csv` by default)
  - conversion failures (`conv_errors.csv` by default)
- `segment`, `phase`, `radiomics`:
  - enriched cohort table + command-specific error CSV

## Configuration

IMPERANDI configuration is done through YAML dataset manifests and optional hooks.
Manifests hold the declarative settings, while hooks provide Python-based
customization when a static file is not enough.

Generic manifests shipped with the package live in:

- `src/imperandi/builtin_datasets_config/manifests/*.yaml`

Their built-in hooks live in:

- `src/imperandi/builtin_datasets_config/hooks/`

Repository-specific OPERANDI configuration lives in `dataset_configs/` and
is intentionally not included in installed packages. Load it only by its
explicit YAML path, for example
`--manifest ./dataset_configs/manifests/operandi.yaml`.

You can pass a built-in manifest name (such as `generic`) or a custom manifest
path. The usual customization flow is to copy a built-in YAML file, edit
`id_extraction`, `id_standardization`, `derived_columns`,
`phase_curation`, `segmentation`, and `radiomics`, then run with
`--manifest ./site-a.yaml`. JSON manifests are not accepted.

`phase_curation.strategies` is an ordered fallback chain. It can contain an
explicit site ontology, IMPERANDI's metadata rules, TotalSegmentator phase
prediction, or any combination. The canonical result is written to `phase`,
with its provenance in `phase_source`, `phase_confidence`, and `phase_reason`.

Hooks are normal Python callables referenced by manifest keys
`hook_module` and `function`: `id_standardization` hook rewrites
`patient_key`, while `derived_columns` hook can add fields based on an existing column.

For configuring radiomic extraction, manifest key `radiomics` can directly contain a PyRadiomics-style
settings object (same structure as `Params.yaml` content).
Official PyRadiomics parameter guide:
[PyRadiomics customization docs](https://pyradiomics.readthedocs.io/en/latest/customization.html).

For configuring segmentation, define CT and/or MR task lists under
`segmentation.modalities`. CT uses TotalSegmentator tasks such as `total`, while
MR uses the corresponding `_mr` models such as `total_mr`. Logical output names
can stay consistent across modalities in the cohort CSV.
Official TotalSegmentator task guide:
[TotalSegmentator subtasks guide](https://github.com/wasserth/TotalSegmentator#subtasks).

Full IMPERANDI configuration guide: [Documentation](https://imperandi.readthedocs.io/en/latest/manifests.html).

## Performance and reliability notes

- Parallel execution controls are available for heavy stages (`parse`, `convert`, `segment`).
- Long-running stages (`parse`, `convert`, `segment`, `phase`, `radiomics`) use a unified checkpoint interface:
  `--checkpoint_every_rows`, `--checkpoint_every_sec`, `--no_resume`, `--strict_resume`.
- Resume is enabled by default; pass `--no_resume` to disable it.
- `parse` reads tags from defaults (`DEFAULT_DICOM_TAGS`) plus `--tags`; use `--snapshot_tags` for full recursive tag snapshots on sampled data.
- `parse` auto-detects archive-heavy inputs from a deterministic root sample (`--archive_detect_sample_size`) and can switch to archive-aware mode at runtime when needed.
- Archive workflows are bounded by depth and include path-safety protections.
- Most commands support `--dry-run` for pipeline planning and CI smoke checks.

## Testing

Fast tests are organized under `tests/unit` and run in normal CI:

```bash
python -m pytest -m "not slow"
```

Dataset-backed tests run the complete ingest, convert, segment, phase, and
radiomics workflow on small IRCAD and TCGA-LIHC cohorts. Downloaded inputs and
generated outputs stay outside version control:

```bash
python -m pip install -e '.[all-dev]'
python -m pip install "pyradiomics @ git+https://github.com/AIM-Harvard/pyradiomics.git@master"
python tests/slow/ircad/download.py
python tests/slow/tcga_lihc/download.py
python -m pytest tests/slow -m slow
```

Each dataset can also run independently:

```bash
bash tests/slow/ircad/pipeline.sh
bash tests/slow/tcga_lihc/pipeline.sh
```

See [`tests/slow/README.md`](tests/slow/README.md) for layout, environment
overrides, and dataset-specific instructions.

## Use Case on [IRCAD Dataset](https://www.ircad.fr/research/data-sets/liver-segmentation-3d-ircadb-01/)

Download the dataset (~800MB):

```bash
wget https://cloud.ircad.fr/index.php/s/JN3z7EynBiwYyjy/download -O ircad.zip
```

Unzip the archive:

```bash
unzip ircad.zip -d ircad_dicom
```

After extraction, your structure should look similar to:
```
ircad_dicom/
└── 3Dircadb1/
    ├── 3Dircadb1.1/
    │   ├── PATIENT_DICOM.zip/
    │   ├── MASKS_DICOM.zip/
    │   └── ...
```

Install package:

```bash
conda create -n imperandi310 python=3.10
conda activate imperandi310
pip install -e .[all]
python -m pip install "pyradiomics @ git+https://github.com/AIM-Harvard/pyradiomics.git@master"
```

Execute pipeline:
```bash
imperandi ingest "ircad_dicom/3Dircadb1/**/PATIENT_DICOM*" . --snapshot_tags
imperandi convert dicom_index_clean.csv ircad_nifti/
imperandi segment nifti_index.csv
imperandi phase nifti_index.csv
imperandi radiomics nifti_index.csv
```

Inspect results with dashboards:
- explore images & segmentations with the interactive viewer
- inspect DICOM tags
- basic radiomics statistics
