Metadata-Version: 2.4
Name: neuromaestro
Version: 1.0.0
Summary: A meta-pipeline for automated multi-modal neuroimaging processing on HPC clusters
Author-email: Qiuyu Yu <qiuyuyu.psych@gmail.com>
Maintainer-email: Qiuyu Yu <qiuyuyu.psych@gmail.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/GeorgiaCenterforDevelopmentalScience/NeuroMaestro
Project-URL: Documentation, https://georgiacenterfordevelopmentalscience.github.io/NeuroMaestro/
Project-URL: Repository, https://github.com/GeorgiaCenterforDevelopmentalScience/NeuroMaestro
Project-URL: Issues, https://github.com/GeorgiaCenterforDevelopmentalScience/NeuroMaestro/issues
Project-URL: Changelog, https://github.com/GeorgiaCenterforDevelopmentalScience/NeuroMaestro/blob/main/CHANGELOG.md
Keywords: NeuroMaestro,neuroimaging,fMRI,BIDS,HPC,Slurm,pipeline,fMRIPrep,MRIQC,QSIPrep
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: typer>=0.12
Requires-Dist: PyYAML>=6.0
Requires-Dist: dash>=4.0
Requires-Dist: dash-cytoscape>=1.0
Requires-Dist: dash-bootstrap-components>=2.0
Requires-Dist: plotly>=6.0
Requires-Dist: pandas>=2.2
Requires-Dist: bids-validator-deno>=3.0.2
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Dynamic: license-file

<p align="center">
  <img src="https://raw.githubusercontent.com/GeorgiaCenterforDevelopmentalScience/NeuroMaestro/main/docs/images/neuromaestro.jpg" alt="NeuroMaestro" width="240">
</p>

# NeuroMaestro

A modular neuroimaging preprocessing pipeline for HPC clusters, with both a GUI and CLI. Pipeline parameters and analysis workflows are configured through YAML files, allowing flexible customization without modifying the underlying code.

Full handbook: https://georgiacenterfordevelopmentalscience.github.io/NeuroMaestro/

---

## Supported Pipelines

The pipeline is organized into four stages. The software used at each stage is configured per-project and can be customized; the examples below reflect our defaults.

| Stage | Description | Example software |
|-------|-------------|-----------------|
| **Data preparation** | Unzip raw data, convert DICOM to BIDS | dcm2bids |
| **Intermediate steps** | Any intermediate step required before modality pipelines (staged pipelines only) | AFNI `sswarper2` |
| **BIDS pipelines** | Modality preprocessing that runs directly on BIDS data | fMRIPrep, XCP-D, QSIPrep, QSIRecon |
| **Quality control** | Image quality metrics | MRIQC |

The key distinction between pipeline types:
- **Staged pipelines** (`--staged-prep`) require an intermediate step before modality preprocessing, by default skull stripping and nonlinear registration of the T1w image. Designed for workflows like AFNI that operate on locally processed data, but can be adapted to any pipeline requiring intermediate steps.
- **BIDS pipelines** (`--bids-prep`) run directly on BIDS-formatted data without intermediate steps (e.g. fMRIPrep, QSIPrep).

---

## Installation

**Prerequisites:** Python 3.10+, HPC cluster with SLURM, neuroimaging software installed on the cluster.

```bash
pip install neuromaestro

# Verify
neuromaestro --help
neuromaestro-gui --help
```

To install from source in development mode:

```bash
git clone https://github.com/GeorgiaCenterforDevelopmentalScience/NeuroMaestro.git
cd NeuroMaestro
pip install -e '.[dev]'
```

---

## Configuration

Each project requires a `{project}_config.yaml` in the `project_config/` folder of your config directory, the one `--config-dir` points at. Run `neuromaestro generate-config {project}` to create one from the template, then fill in paths, HPC modules, and pipeline options.

Modalities available under `--bids-prep` and `--staged-prep`, and the output folder of each task (`output_pattern`), are declared in `config.yaml`.

Expected output files for `--resume` are defined in `config/results_check/{project}_checks.yaml`. Edit this file to specify which files should exist (and optionally their minimum size) for a subject to be considered complete.

HPC scheduler, resource profiles (memory, walltime, CPU), and submission flags are configured in `hpc_config.yaml`. Recommended values are pre-filled; update partition names and resource limits to match your cluster.

---

## Quick Start

**First-time setup:** scaffold config and script templates into your study directory.

```bash
neuromaestro init /path/to/my_study
# Optionally export to ~/.bashrc so you can omit --config-dir on every command.
# If you skip this, pass --config-dir /path/to/my_study/config explicitly instead.
export NEUROMAESTRO_CONFIG_DIR=/path/to/my_study/config
```

This copies `config.yaml`, `hpc_config.yaml`, `project_config/`, `results_check/`, and script templates. Edit them to match your cluster and project before running.

---

```bash
# Launch the GUI
neuromaestro-gui
```

Open the link it prints at startup (`http://127.0.0.1:8050/?token=...`). Requests without the token are rejected, and a new token is made at each launch.

Or use the CLI. Start with a dry-run on one subject to verify the plan:

```bash
neuromaestro run \
  --subjects 001 \
  --input /data/BIDS \
  --output /data/processed \
  --work /data/work \
  --config-dir /data/config \
  --project my_study \
  --session 01 \
  --bids-prep rest \
  --dry-run
```

Full pipeline (all stages, multiple subjects):

```bash
neuromaestro run \
  --subjects subjects.txt \
  --input /data/raw \
  --output /data/processed \
  --work /data/work \
  --config-dir /data/config \
  --project my_study \
  --session 01 \
  --prep unzip_recon \
  --intermed volume \
  --staged-prep cards,kidvid \
  --staged-post cards,kidvid \
  --bids-prep rest,dwi \
  --bids-post rest,dwi \
  --mriqc all
```

Dependencies are enforced automatically by the scheduler.

---

## CLI Reference

### Core options

| Option | Description |
|--------|-------------|
| `--subjects` | Comma-separated subject IDs or path to a `.txt` file |
| `--input` | Input directory (raw data with `--prep`, BIDS data otherwise) |
| `--output` | Output directory |
| `--work` | Work directory (logs, database, intermediate files) |
| `--config-dir` | Path to config directory (contains `config.yaml`, `hpc_config.yaml`, `project_config/`). Optional if `$NEUROMAESTRO_CONFIG_DIR` is exported. |
| `--project` | Project name (must match a `{project}_config.yaml`) |
| `--session` | Session label (e.g. `01`) |

### Pipeline options

| Option | Description |
|--------|-------------|
| `--prep` | Data preparation: `unzip`, `recon`, `unzip_recon` |
| `--intermed <tasks>` | Intermed tasks to run (comma-separated, e.g. `volume`). Staged prep tasks depend on these when given. Omit it only when the intermed outputs already exist from an earlier run. |
| `--bids-prep <modalities>` | BIDS pipeline preprocessing (comma-separated, e.g. `rest,dwi`) |
| `--bids-post <modalities>` | BIDS pipeline postprocessing (comma-separated, e.g. `rest,dwi`) |
| `--staged-prep <modalities>` | Staged pipeline preprocessing (comma-separated, e.g. `cards,kidvid`) |
| `--staged-post <modalities>` | Staged pipeline postprocessing (comma-separated, e.g. `cards,kidvid`) |
| `--mriqc` | Quality control: `individual`, `group`, `all` |

### Run options

| Option | Description |
|--------|-------------|
| `--dry-run` | Preview the execution plan without submitting jobs |
| `--resume` | Skip subjects whose expected outputs already exist |
| `--skip-bids-validation` | Skip pre-run BIDS validation |
| `--skip-preflight` | Skip pre-flight config checks |
| `--wait` | Wait for submitted jobs to finish before exiting |
| `--polling-interval` | Seconds between job status checks when `--wait` is set. Default 60. |

---

## Resume a Partial Run

```bash
neuromaestro run \
  --subjects subjects.txt \
  --input /data/BIDS \
  --output /data/processed \
  --work /data/work \
  --config-dir /data/config \
  --project my_study \
  --session 01 \
  --bids-prep rest \
  --resume
```

`--resume` checks each subject's expected output files (defined in `config/results_check/{project}_checks.yaml`) before submitting. Subjects that already have valid outputs are skipped, and the skipped IDs are printed per task.

---

## Directory Structure

### Input (BIDS)

```
input_directory/
├── sub-001/
│   └── ses-01/
│       ├── anat/
│       ├── func/
│       └── ...
└── sub-002/
    └── ...
```

### Output Example

```
output_directory/
└── project_name/
    ├── raw/
    ├── BIDS/
    ├── AFNI_derivatives/
    ├── BIDS_derivatives/
    │   ├── fmriprep/
    │   ├── xcpd/
    │   ├── qsiprep/
    │   └── qsirecon/
    ├── postprocessing/
    └── quality_control/
        └── mriqc/

work_directory/
├── database/        # default db_path, shared by projects with the same --work
└── project_name/
    └── log/
```

---

For questions or issues, contact [QiuyuYu](https://github.com/QiuyuYu3) or open a repository issue.
