Metadata-Version: 2.4
Name: dtiplayground
Version: 0.8.15
Summary: NIRAL pipeline software for diffusion MRI: preprocessing and quality control, fiber profile analysis, atlas building
Home-page: https://github.com/niraluser/dtiplayground
Author: Martin Styner, SK Park, NIRAL, University of North Carolina @ Chapel Hill
Author-email: styner@unc.edu
License: MIT
Keywords: dtiplayground,dmriprep,dmriatlas,dmriautotract,dmrifiberprofile,nrrd,nifti,dwi,dti,qc,quality control
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: wheel
Requires-Dist: cmake>=3.24.1
Requires-Dist: pynrrd<2,>=1.0.0
Requires-Dist: dipy<1.10,>=1.6.0
Requires-Dist: pyyaml>=5.3.1
Requires-Dist: nibabel<6,>=5.0.0
Requires-Dist: opencv-python-headless
Requires-Dist: simpleitk>=2.1.1
Requires-Dist: xhtml2pdf
Requires-Dist: flask
Requires-Dist: flask_cors
Requires-Dist: flask_jwt_extended
Requires-Dist: numpy<2,>=1.21
Requires-Dist: fury<0.13,>=0.10.0
Requires-Dist: vtk>=9.1
Requires-Dist: torch>=2.1
Requires-Dist: matplotlib>=3.7
Requires-Dist: scikit-image>=0.21
Requires-Dist: scikit-learn>=1.3
Requires-Dist: scipy>=1.10
Requires-Dist: markdown
Requires-Dist: reportlab
Requires-Dist: pypdf2
Requires-Dist: pandas<3,>=1.4
Requires-Dist: dmri-amico>=2.1.1
Dynamic: author
Dynamic: author-email
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: license-file
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# DTI Playground

DTI Playground is python based NIRAL pipeline software for diffusion MRI: preprocessing and quality control of
diffusion weighted images, fiber profile extraction and analysis, and atlas building. It ships a web UI
(DTIPlaygroundLab) and these command line tools:

| Tool | Purpose |
|---|---|
| [`dmriprep`](#dmriprep--preprocessing-and-qc-of-diffusion-weighted-images) | Preprocessing and quality control of DWIs, as a pipeline of modules (artifact checks, susceptibility and eddy current correction, denoising, brain masking, tensor estimation, registration, reports); single scans or a whole cohort |
| [`dmrifiberprofile`](#dmrifiberprofile--fiber-profiles-and-their-analysis) | Along-tract profiles of diffusion properties, and the tools to gather, impute and QC them |
| [`dmriatlas`](#dmriatlas--atlas-building) | Atlas building from several diffusion tensor images |
| `dmriplayground` | Shared configuration and tool installation (`install-tools`, `init`) |
| `dmriplaygroundlab` | The web UI for all of the above |

Detailed documentation: <https://niraluser.github.io/DTIPlayground/>, built from `docs/` on every change, with a
reference page for every module and option of dmriprep and dmrifiberprofile. It is also published at
<https://dtiplayground.readthedocs.io/en/latest/>. The change log is in [CHANGELOG.md](https://github.com/NIRALUser/DTIPlayground/blob/master/CHANGELOG.md).

## Installation

Python 3.9 - 3.12 is required (3.11 recommended). On Windows, use WSL2 with a Linux Python.

### Using pip

```
$ pip install dtiplayground
$ dmriplaygroundlab
```

### Using conda

```
$ conda create -n dtienv-py311 python=3.11
$ conda activate dtienv-py311
$ pip install --upgrade dtiplayground
$ dmriplaygroundlab
```

### Using Docker

```
$ docker pull niraluser/dtiplayground
```

The image contains all the necessary tools (FSL / DTIPlaygroundTools) and an initialized configuration. Run bash in
it, or:

```
$ docker run -it --rm --user $(id -u):$(id -g) -e HOME=$HOME -v $HOME:$HOME -v <WORKINGDIR>:<WORKINGDIR> niraluser/dtiplayground
```

It launches DTIPlaygroundLab; open the link it prints in a browser. Add `--gpus all` (NVIDIA container toolkit) to
use a GPU, e.g. for `dmrifiberprofile impute`.

### FSL and DTIPlaygroundTools

The command line tools need FSL and DTIPlaygroundTools, which pip does not install:

```
$ dmriplayground install-tools
```

The default output directory is `$HOME/.niral-dti/`.

- `--clean-install` removes existing software packages and temporary files first
- `--no-remove` keeps the temporary build files
- `--nofsl` does not install FSL
- `--install-only` does not update the software paths of the current configuration
- `--build` builds DTIPlaygroundTools with docker (docker required)

Once installed, `$HOME/.niral-dti/global_variables.yml` holds the root paths of the packages, and the software paths
of the current dmriprep version are set unless `--install-only` was given.

Administrators who provide FSL and dtiplayground-tools on the network can point users at them with the environment
variables `FSLDIR` (the FSL root directory) and `DTIPLAYGROUNDTOOLS` (a dtiplayground-tools directory holding
`info.yml`); each user is then asked whether to use them when initializing.

**NOTE** If the installation fails, remove `$HOME/.niral-dti` and install again, or re-initialize with
`$ dmriplayground init`. The bundled FSL installer runs under python2, which has to be on the path
(e.g. `ln -s /usr/bin/python2.7 /usr/bin/python2`); nothing else in DTI Playground needs it.

### Optional: MRtrix3

The `mrtrix3` option of the MULTI_SHELL_Estimate module needs MRtrix3, which is not installed by pip. In a conda
environment:

```
$ conda install -c mrtrix3 mrtrix3
```

## dmriprep — preprocessing and QC of diffusion weighted images

dmriprep performs quality control over diffusion weighted images. Quality control is an essential preprocessing step
in DTI research, in which the bad gradients with artifacts are excluded or corrected by various computational
methods. The software and library provide a module based package with which users can build their own QC pipeline as
well as new pipeline modules.

The modules and their protocol options: [README about Preprocessing](https://github.com/NIRALUser/DTIPlayground/blob/master/dtiplayground/dmri/preprocessing/README.md).

### Graphical interface

```
$ dmriplaygroundlab
```

From the menu, go to the DMRIPrep application.

### Command line

*FSL / DTIPlaygroundTools required*

1. **init** - Initialize configuration (default: `$HOME/.niral-dti/dmriprep-<version>`)

**init** command generates the configuration directory and files with following command. One just needs to execute this command only once unless a different configuration is needed. If you want to reset the initial configuration directory, you can run init again.
```
    $ dmriprep init
```
If you want to set different directory other than default one :
```
    $ dmriprep --config-dir my/config/dir init
```
Once run, `config.yml` and `environment.yml` will be in the directory.

You can manually specify the tool directory (which is generated by `install-tools` command) by `--tools-dir` option.
```
    $ dmriprep init --tools-dir <path/to/tool_dir>
```

2. **update** - Update if `config.yml` has been changed (e.g. in case of adding user module directory).
Changing `config.yml` file should be followed by updating `environment.yml` with running update command :
```
    $ dmriprep [--config-dir my/config/dir] update
```
This will update module-specific informations such as binary locations or package location used by the corresponding module. It simply updates `environment.yml`

3. **make-protocols** - Generating a default protocol file

The first thing to do QC is to generate default protocol file that has pipeline information.
```
    $ dmriprep [base options] make-protocols -i IMAGE_FILENAME [-o OUTPUT_FILENAME_] [-d MODULE1 MODULE2 ... ]
```
if `-o` option is omitted, the output protocol will be printed on terminal.`-d` option specifies the list of modules for the QC, with which command will generate the default pipeline and protocols of the sequence. Same module can be used redundantly. If `-d` option is not specified, the default pipeline will be generated from the file `protocol_template.yml` . You can change the default pipeline in `protocol_template.yml` file. With two input images (opposite phase encodings), the default pipeline also has SUSCEPTIBILITY_Correct, before EDDYMOTION_Correct.

4. **run** - Run pipeline
To run with default protocol generated from `protocol_template.yml`:

```
    $ dmriprep [base options] run -i IMAGE_FILES -o OUTPUT_DIR -d [ MODULE1 MODULE2 ... ]
```
`-d` option (default protocol) works as described in **make-protocols** command. But you need to specify `"-d"` for the default pipeline from the template.  If `-o` option is omitted, default directory will be set to `Image filename_QC`. IMAGE_FILES may be a list of files to process. In case of susceptibility correction, IMAGE_FILES needs to have counterparts for the polarities. `dmriprep` automatically process qc for all the input images before the susceptibility correction stage.

To run with existing protocol file:
```
    $ dmriprep run -i IMAGE_FILES -p PROTOCOL_FILE -o output/directory/
```

`-p` option cannot be used with `-d` option.

**Age appropriate registration target:** with `referenceNormativeModel` (a folder written by `dmrifiberprofile qc-registration --build-normative`), DTI_Register registers to the mean tensor of the age bin of the subject instead of `referenceImage`. The age is the protocol `age` (or the global variable `age`), else the row of the scan in `ageCSV` (the table of `qc-registration --age-csv`: subject/session/age columns are detected, `ageColumn` and `ageUnits` override that; the subject and session are read from the path as `sub-<id>`/`ses-<id>`), else `ageRegex` on the path (default `ses-(\d+)m`). Without any of them the reference image is used, with a warning.

**Rerunning into an existing output directory:** a module with a result from a previous run is not recomputed, unless its protocol or the global variables given with `-g` changed since that run (compared with the `settings.yml` stored in the module's folder; the module and the following ones are then recomputed). `--overwrite` recomputes all modules. Results of versions before 0.7.11 have no `settings.yml`: they are reused (with a warning) until the run uses `--overwrite`. Global variables given with `-g` take precedence over those stored by a previous run (`global_variables.yml`).

**Denoising and Gibbs ringing removal:** DWI_Denoise (DIPY MP-PCA, the method of MRtrix `dwidenoise`, or Patch2Self)
and GIBBS_Correct (DIPY, the method of MRtrix `mrdegibbs`, for full Fourier acquisitions) are not in the default
pipeline. Both assume raw, uninterpolated data: put DWI_Denoise first and GIBBS_Correct right after it, e.g.
`-d DWI_Denoise GIBBS_Correct SLICE_Check INTERLACE_Check EDDYMOTION_Correct QC_Report`. MP-PCA also writes the noise
level map (`<base>_DWI_noise_sigma.nii.gz`), Patch2Self the RMS of the removed signal (`<base>_DWI_noise_residual.nii.gz`).

**QC outputs** (next to the QCed image, `<base>_<name>`), also columns of the QC_Report CSV and of the batch QC table.
The per volume tables carry an `original_index`, the index of the volume in the input of the pipeline (`volume` is its
position in the image the module worked on; the two differ once a volume has been excluded):
- `DENOISE_QC.tsv` (DWI_Denoise): noise level (MP-PCA sigma) and b=0 SNR in the brain, or the median of the Patch2Self
  residual map, standard deviation of the removed signal. `GIBBS_QC.tsv` (GIBBS_Correct): mean absolute change in the brain.
- `EDDY_motion.tsv` (EDDYMOTION_Correct): per volume translations (mm), rotations (degrees), framewise displacement
  (Power et al. 2012, 50 mm head radius), RMS movement and eddy outlier slices (`interpolateBadData`/`--repol`).
  `EDDY_QC.tsv`: mean/max framewise displacement, maximum translation and rotation (relative to the first volume and
  between consecutive volumes), outlier slices, b=0 SNR and CNR of each shell (mean of eddy `--cnr_maps` in the mask).
- `DTI_fit.tsv` (DTI_Estimate): per volume R² and correlation of the signal predicted by a WLS tensor fit (in the mask,
  without the voxels that have a non-positive value in some volume, e.g. thresholded at 0 after eddy), and the number
  of poorly fitted slices (slice R² more than 4 scaled MADs below that of the same slice in the other volumes of the
  shell). `DTI_fit_QC.tsv`: their summary; `DTI_fit_carpet.png`: slice x volume R² (in the QC report).
- `IMAGE_QC.tsv` (QC_Report, protocol `imageQC`): the same numbers on the raw input (`raw_`) and on the preprocessed
  image (`qced_`), so preprocessing can be checked against its input. Neighboring DWI correlation (`ndc`, as DSI
  Studio: each volume with the volume of the same shell in the closest direction, over the mask; also per shell), bad
  slices (`bad_slices`: correlation with the slice below, or mean intensity relative to the median slice of the volume,
  more than 3.5 scaled MADs below the same slice position in the other volumes of the shell — the first catches a
  corrupted slice, the second a signal dropout), and with `bTableCheck` the fiber coherence index of the b-table
  (`coherence`, Schilling et al. 2019: stepping along the principal direction of a tensor fit reaches a similar
  direction; `coherence_best_flip` names the b-vector axis whose sign would raise it, `none` if the b-table as given is
  the most coherent). `IMAGE_ndc.tsv`: per volume; `IMAGE_QC_plot.png`: both stages by original volume index (in the
  QC report).

**[NOTE]** when using 2 image files for SUSCEPTIBILITY_Correct and other multi input modules, order of files can be important. For the SUSCEPTIBILITY_Correct, AP(FH), RL, SI phased file comes first. (e.g. `$ dmriprep -i AP_img.nrrd PA_img.nrrd ...`)

5. **run-dir** Run output directory having protocol file

```
    $ dmriprep run-dir output/directory [--overwrite]
```

6. **bids** / **run-batch** - Process a cohort (locally or on a SLURM cluster)

`bids` processes the DWIs of a BIDS dataset (`sub-*/[ses-*/]dwi/*_dwi.nii[.gz]` with `.bval`/`.bvec`), with the BIDS-App
interface (`bids_dir output_dir participant|group`):
```
    $ dmriprep bids /data/study /data/study/derivatives/dmriprep participant -p protocol.yml -j 4 -t 2
    $ dmriprep bids /data/study /data/study/derivatives/dmriprep group
```
- **Datasets**: each DWI run is a dataset, unless the protocol needs two images (SUSCEPTIBILITY_Correct): the runs of a
  session that differ only by `dir`/`run` and have opposite phase encoding (`PhaseEncodingDirection` of the sidecar, or
  the `dir` label with the image orientation) are then processed as pairs, the run encoded towards posterior (AP, RL,
  SI) first. The phase encoding axis and value (`TotalReadoutTime`) of SUSCEPTIBILITY_Correct are set from the
  sidecars for each pair. Runs that can't be paired are listed as skipped.
- **Protocols**: `-p protocol.yml`, or one protocol per acquisition with shell patterns on the run names, the first
  match being used: `-p '*acq-dir79*=dir79.yml' '*acq-shells06*=shells06.yml'`. Or `-d [MODULE ...]`: the default
  protocol with these modules (as `run -d`; without modules the default pipeline of `protocol_template.yml`:
  SLICE_Check, INTERLACE_Check, EDDYMOTION_Correct, QC_Report, with SUSCEPTIBILITY_Correct before EDDYMOTION_Correct
  for the runs that have an opposite phase encoded run, which are then processed as pairs), generated for each
  acquisition since some default parameters depend on the image (`<output_dir>/batch/default_protocols/`); `-b` sets
  its b0 threshold.
  `--participant-label` and `--session-label` restrict the subjects and sessions: only their datasets are processed,
  and the datasets of the other subjects and sessions of earlier calls on the same BIDS dataset stay in the batch
  (`batch-status`, `group`, and SLURM tasks of earlier calls still in the queue).
- **Outputs**: `<output_dir>/sub-<label>/[ses-<label>/]dwi/<dataset id>/` holds the usual output of `dmriprep run` (same
  file names, e.g. `<scan>_dwi_QCed.nii.gz`; for a pair `sub-.._ses-.._acq-.._dwi_QCed.nii.gz`), with
  `dataset_description.json` at the top. `<output_dir>/batch/` holds the manifest (`manifest.tsv`), the protocol of
  each dataset (`protocols/`), the skipped runs (`skipped.tsv`) and the state of each dataset (`status.tsv`).
- `--dry-run` writes the batch folder and lists the datasets per protocol, grouped by acquisition (dimensions, voxel
  size, volumes, shells), so that a mixed cohort is noticed before processing.
- **Resuming**: running the same command again processes only the datasets that are not done: new, failed or
  interrupted datasets, and datasets done with another protocol (only the modules whose settings changed are then
  recomputed). `--rerun` processes all datasets again, `--only ID ...` some of them, `--overwrite` recomputes all their
  modules. `dmriprep batch-status <output_dir>` lists the datasets that are not done (`--all`: all of them). A dataset is processed by one task at a time: a task that starts while another one is processing the same dataset (e.g. a SLURM array submitted twice) stops without touching it (exit code 75, `skipped` in a local run); the lock is `batch_task.lock` in the dataset folder, and the lock of a task that is gone is taken over.
- **Local execution**: `-j` datasets are processed at the same time, each in its own process with `-t` threads; its
  log is `batch_log.txt` (and the usual `log.txt`) in its folder.
- **SLURM** (e.g. Longleaf): `--slurm` writes `<output_dir>/batch/slurm_array.sh`, a job array with one task per
  dataset to process (its dataset ids are in its own `batch/slurm_ids_<time>.txt`, so a later call does not change the
  datasets of an array that is still queued), to submit with `sbatch`; `--slurm-submit` submits it. Options: `--slurm-time` (default
  24:00:00), `--slurm-mem` (16G), `--slurm-partition`, `--slurm-max-parallel`, `--slurm-setup` (a shell line run
  before dmriprep, e.g. `'module load fsl'`), `--slurm-option=--gres=gpu:1` (any `#SBATCH` option); `-t` sets the
  CPUs per task. The tasks run the dmriprep (Python environment and configuration directory) the script was written
  with, so write it on the cluster. Run the command again after the job to see which datasets failed and resubmit
  them.
- **group**: `<output_dir>/batch/qc_table.tsv` and `qc_report.html` (state, run time, volumes in/out, excluded
  volumes, mask volume, mean FA per dataset, and the noise, motion, SNR/CNR and tensor fit summaries of the modules
  that ran, see **QC outputs**; values far from the cohort median are marked: excluded volumes, and in the bad
  direction mean framewise displacement, outlier slices, poorly fitted slices and mean tensor fit R²), and
  `fiberprofile_datasheet.csv`, the datasheet for `dmrifiberprofile` (columns `id`, `DTI`, and `FW DTI`, `Deformation
  field`, `FWF`, `NDI`, `ODI` when the pipeline writes them). `dmriprep batch-report <output_dir>` does the same for
  any batch.

`run-batch` processes datasets listed in a datasheet (TSV or CSV), e.g. NRRD files outside of BIDS, with the same
options:
```
    $ dmriprep run-batch -m cohort.tsv -o /data/cohort_QC -p protocol.yml -j 4
```
Datasheet columns: `id` and `image_1` (required), `image_2` (second image, e.g. the opposite phase encoding, in the
order of **run**), `protocol` (else `-p`, whose patterns match the id, or `-d`), `output_dir` (relative to `-o`, default: the
id), `output_file_base`, `subject`, `session`, and `overrides`: module parameters of the dataset as JSON, e.g.
`{"SUSCEPTIBILITY_Correct": {"phaseEncodingValue": 0.0945}}`. Relative paths are relative to the datasheet.

### Writing a module

#### Adding a module

Once initialized, users can add their custom module from scratch or existing system/user modules by following command
```
$ dmriprep add-module <module-name> [--base-module <base-module-name>] [--edit]
```
Following command will generate initial skeletal files of module
```
$ dmriprep add-module HELLO_World
```
Then you can test if the module can be loaded properly with
```
$ dmriprep update
```
You can use your module right in protocol file.

if `-b` , `--base-module` is specified, new model will copy existing code and data from the base module.
e.g.
```
$ dmriprep add-module MYFIRST_Module -b SLICE_Check
```
MYFIRST_Module will have same codes and data (module definition yaml file) from SLICE_Check module with new classname and filenames.

#### Developer

Once module is developed and tested in the user module directory, one can just move that directory in `dtiplayground/dmri/preprocessing/modules` and commit. Make sure the custom module is not existing both in system module directory and user module directory.

#### Removing user module
User module can be removed by
```
$ dmriprep remove-module <module-name>
```
e.g.
```
$ dmriprep remove-module MYFIRST_Module
```

**NOTE** System module cannot be removed by this command. Only user module can be removed.

#### Modules in other directory
You can just copy module directory to `$HOME/.niral-dti/modules/dmriprep` and check with `$ dmriprep update` command. Same applies for removal of user modules.

## dmrifiberprofile — fiber profiles and their analysis

Besides running the EXTRACT_Profile pipeline (`dmrifiberprofile run`), `dmrifiberprofile` provides tools for the analysis and QC of fiber profiles (ported from the FiberProfileAnalysis scripts). Use `dmrifiberprofile <command> --help` for all options.

| Command | Purpose |
|---|---|
| `flip-tensor` | Reflect the tensor frame of a DTI NRRD along axes (fixes tensor orientation / LPS-RAS sign mismatches, e.g. a flip found by `qc-registration`); `--voxel-frame` rotates components stored in the voxel frame into the space of the header |
| `detect-tensor-flip` | Find the correction of the tensor frame (all combinations of flips of x, y, z, with the components in the header frame or in the voxel frame of oblique grids) that orients a DTI correctly, by the coherence of the principal directions along the tracts and, with `--reference`, their agreement with an atlas tensor; apply the result with `flip-tensor` before registering |
| `parametrize-fibers` | Resample fiber tracts on the arc length grid (one point per arc length bin) with point data `FiberLocationIndex` and `SamplingDistance2Origin`; replaces `dtitractstat -f` |
| `compute-axis` | 1D axis of fiber tracts (average curve per arc length bin) as `<tract>_axis.vtk`; optionally with the profiles mapped onto the axis |
| `gather` | Collect subject profiles into `<tract>/<tract>_<metric>.csv` (rows: arc length, columns: datasets), from `.fvp` trees and/or EXTRACT_Profile outputs |
| `impute` | Fill missing profile values with a per-dataset SIREN on the (x, y, z, arc length) of the tract axes |
| `qc-registration` | QC of the registration to the atlas (`sub-*/ses-*/AtlasReg/*_Deformed<METRIC>.nii.gz` + `*_DeformedDTI.nrrd`, or the dmriprep DTI_Register outputs `<scan>_Registered_<METRIC>.nii.gz` + `<scan>_DTI_Registered.nrrd` in any folder below `--data-dir`; missing FA/MD/AD/RD maps are computed from the registered tensor): similarity (MAE, SSIM, NCC), angular error, contiguity of disagreement, CSF check, age-conditional normative model, combined outlier flag. `--build-normative` builds the age-binned normative model from a reference dataset: mean/std/count of every deformed metric map (FA, MD, RD, AD, ...), the angular model and the log-Euclidean mean tensor (`DTI_mean.nrrd`) per age bin. `--tensor-flip` (default auto) corrects the tensor frame of the subjects as `detect-tensor-flip` (flips, voxel frame); every subject is also checked on its own (`TENSOR_frame_best`, `TENSOR_frame_gain_deg`, and a warning when another correction fits the atlas better) |
| `qc-profiles` | Age-binned profile statistics (`_agebinstats.csv`, ages from `--age-csv` or the column names) and plots; profile QC against prior (normative) statistics with value and shape outliers; cleaned profile tables. `--profiles-dir` takes the gathered tables (`<tract>/<tract>_<metric>.csv`) or the profiles of a run as EXTRACT_Profile writes them (`00_EXTRACT_Profile/<metric>/<tract>_<metric>.csv`, either orientation), so `gather` is not needed to QC one run |

### Typical workflow

```
$ dmrifiberprofile parametrize-fibers Atlas/FibersRaw -o Atlas/FibersParam
$ dmrifiberprofile gather --profiles-dir Output_Profiles --fibers-dir Atlas/FibersParam --out-dir Profiles
$ dmrifiberprofile compute-axis Atlas/FibersParam -o FiberAxis
$ dmrifiberprofile impute --profiles-dir Profiles --axis-dir FiberAxis --out-dir Profiles_Imputed
$ dmrifiberprofile qc-registration --data-dir Data --atlas-dir Atlas --normative-dir Atlas/normativeModel --out-dir RegistrationQC
$ dmrifiberprofile qc-profiles --profiles-dir Profiles_Imputed --prior-stats-dir Atlas/normProfiles --registration-qc RegistrationQC --clean-dir Profiles_Clean -o ProfileQC
```

`parametrize-fibers` computes the plane of origin and arc lengths like EXTRACT_Profile and stores the arc lengths in the fibers, so the atlas defines them once: EXTRACT_Profile (protocol option `arcLength: stored`, the default) and `compute-axis` (`--arc-source auto`, the default) use the stored arc lengths of parametrized fibers and compute them only for fibers without (`arcLength: compute` / `--arc-source dtiplayground` always compute them). Parametrized fibers from the older C++ tools store different arc lengths; re-parametrize the raw tracts with `parametrize-fibers`. `impute` uses a GPU when available (`--device`).

Normative profiles (the `normProfiles` of an atlas) are the age-bin statistics of the profiles of a reference dataset, computed with the atlas' parametrized fibers:

```
$ dmrifiberprofile parametrize-fibers Atlas/FibersRaw -o Atlas/FibersParam
$ dmrifiberprofile run -i reference.csv -p protocol.yml -o ReferenceProfiles     # EXTRACT_Profile, atlas: Atlas/FibersParam
$ dmrifiberprofile gather --profiles-dir ReferenceProfiles --fibers-dir Atlas/FibersParam --out-dir normProfiles
$ dmrifiberprofile qc-profiles --profiles-dir normProfiles --plots-dir normProfiles_plots
```

### Datasheet from a folder

Instead of a datasheet, `dmrifiberprofile run -i <folder> -p protocol.yml -o <output>` takes a
folder: the files of every scan below it are detected, grouped by case id (the part of the file names before `_dwi`),
written to `<output>/datasheet_detected.csv`, and the `parameterToColumnHeaderMap` of the protocol is set to its
columns. The files needed follow from the protocol (`propertiesToProfile`, `inputIsDTI`, `useDisplacementField`):

| | native space (`useDisplacementField: true`) | atlas space (`false`) |
|---|---|---|
| tensor (FA, MD, AD, RD) | `<id>_dwi[_QCed]_tensor.nrrd`, `<id>_dwi[_QCed]_DTI.nrrd` | `<id>_dwi*_DeformedDTI.nrrd`, `<id>_dwi*_DTI_Registered.nrrd` |
| displacement field | `<id>_dwi*_GlobalDisplacementField.nrrd`, `<id>_dwi*_DTI_DisplacementField.nrrd` (not the inverse) | - |
| free-water tensor (FWFA, ...) | `<id>_dwi*_FWtensor.nrrd`, `<id>_dwi*_FWDTI.nrrd`; without, the maps below | - (maps) |
| map of property P | `<id>_dwi*_P`, `<id>_dwi*_DTI_P`, `<id>_dwi*_NODDI_P` `.nii[.gz]` | `<id>_dwi*_DeformedP`, `<id>_dwi*_Registered_[DTI_\|NODDI_]P` `.nii[.gz]` |

The tensor and the displacement field are required; a scan without them, or with several files for one column (e.g.
copies of a scan in the folder), is left out and reported, and the run stops with an error listing the known file names
when no scan matches. Property names are case sensitive: `FWF` is the NODDI free-water fraction (`_NODDI_FWF`), `FWf`
the fraction of the free-water DTI model (`_FWf`). `dmrifiberprofile make-datasheet <folder> -p protocol.yml -o
sheet.csv` only writes the datasheet (and `sheet_protocol.yml`, the protocol with its columns) to check it first;
`--id-regex` changes how the case id is taken from the file names.

### Running without a protocol

Without a protocol, `run` uses the defaults of EXTRACT_Profile (FA, MD, AD, RD from the tensors, native space);
`--atlas <folder>` sets the atlas, and a protocol without `tracts` profiles all the tracts (`.vtk`) of the atlas
folder, e.g. `dmrifiberprofile run -i <data folder> --atlas Atlas/FibersParam -o Profiles`. The default
`supportBandwidth` is 3 mm (before 0.7.22: 1 mm), as in the example protocol; compare only profiles made with the same
value.

### Rerunning

With `cleanup: noCleanup` (the default since 0.7.22) the profile of each scan, tract and property is
kept (`<output>/<datasheet>/00_EXTRACT_Profile/<property>/<tract>/<id>_<tract>.fvp`) with the settings and input files
it was computed from (`.json` next to it). A later run into the same output folder reuses the profiles whose protocol
settings (inputIsDTI, useDisplacementField, tensorInterpolation, supportBandwidth, stepSize, arcLength, planeOfOrigin,
noNaN, maskThreshold), tract, mask and input files (path, size, modification time of the image and displacement field)
are unchanged, and computes the others, e.g. only the scans added to a folder or datasheet; the tables of all scans
are written again. The module option `overwrite: true` of EXTRACT_Profile in the protocol recomputes everything. With
`duringProcessing` / `endOfProcessing` the profiles of each scan are deleted and a later run recomputes all of them.
The output folder is named after the datasheet (`datasheet_detected` for a folder input), so rerun with the same one.

### Example: normative profiles

An example protocol, datasheet and datasheet script for the DTI_IBISEP_Feb26 reference dataset are in `examples/normative_profiles`. The case ids of the datasheet must contain the age as `ses-<months>m` (e.g. `sub-011228_ses-012m`). The images are best sampled in native space with the deformation field of each scan (`useDisplacementField: true`), so tensors don't need to be deformed to the atlas. With `inputIsDTI: true`, FA, MD, AD, RD are computed from the tensors of `Original DTI Image`, and `<prefix>FA`, ... from the tensors of `<prefix> DTI Image` in `parameterToColumnHeaderMap` (e.g. FWFA from free-water corrected tensors, `FW DTI Image`); other properties (e.g. NDI, ODI) are sampled from their own image column. An empty datasheet cell leaves that property out for the scan (e.g. no free water / NODDI for single-shell scans).

### Profile QC against normative statistics

`qc-profiles` writes `<tract>/<tract>_<metric>_agebinstats.csv` next to the tables it read; the folder is then used as `--prior-stats-dir` for the QC of new datasets. It reads the gathered tables or, without gathering them first, the profiles of a single run (`--profiles-dir <output>/<datasheet>/00_EXTRACT_Profile`); the metrics are named as `gather` writes them (`fa`, `md`, `ad`, `rd` in lower case), so the same `--prior-stats-dir` fits both, and `--prior-stats-dir` itself takes either layout (the stats are written next to the tables they were computed from). `gather` is still what collects several runs into one set of tables.

`-o <folder>` puts the output folders of `qc-profiles` in that folder (`--plots-dir`, `--stats-dir`, `--clean-dir` and `--clean-plots-dir` are folders in it unless they are absolute paths); without it they are written in the working directory. The tables of the QC go to `--stats-dir` (default `Profiles_Stats`): `summary_mean.csv` and `summary_median.csv`, the whole-tract mean and median of every metric per tract and dataset of the cleaned data (with `--clean-dir`), and `summary_mean_all.csv` / `summary_median_all.csv` before the profile outliers are removed; `excluded_cases.csv` (one row per registration, preprocessing, no age (`--exclude-no-age`) or profile exclusion), `profile_qc_mean.csv` (every profile compared with the prior, with the outlier decision of its tract and scan) and `profile_qc_mean_clean.csv` (on the cleaned data), and `batch_shifts_mean.csv` with `--envelope-batch-correct` (`mean` is `--corr-reference`). `--clean-dir` holds the cleaned tables with their age-bin statistics, and `--plots-dir` / `--clean-plots-dir` the plots (`Plots/`) of all the data and of the cleaned data. Every table gives the age of each scan as a float in `--age-units` (column `age_<units>`, `age_months` by default), as does `registration_qc.csv` of `qc-registration`. Every table and column of `Profiles_Stats` is described in the documentation ([The tables of Profiles_Stats](https://niraluser.github.io/DTIPlayground/DMRIFiberProfile.html#the-tables-of-profiles-stats)).

The age of each profile, which decides its age bin, is the row of its scan in `--age-csv` (a participants/sessions table, the same as `qc-registration --age-csv`, with `--age-column` and `--age-units`), else `--age-regex` on the column name (`ses-(\d+)m` by default); profiles with neither are counted and named at the end of the run instead of quietly falling out of every bin.

A profile location that a metric which cannot be 0 in tissue (FA, MD, RD, AD, NDI, ODI, ...) reports as 0 was sampled outside the brain mask, so it is read as missing on every metric of that tract and case, including the ones where 0 is valid (`--zero-valid-metrics`, default `FWF`; `--keep-outside-brain` keeps them); with `--clean-dir` those cells are written empty in the cleaned tables. A profile with less than `--min-valid-frac` (default 0.75) of its positions left is flagged as an outlier, since too little of it is there to judge: losing a quarter of a tract to locations outside the brain is itself a sign that the registration or the processing went wrong.

## dmriatlas — atlas building

DMRIAtlas builds an atlas from multiple diffusion tensor images. It performs affine and diffeomorphic registrations
and generates the atlas for all the reference images.

### Command line

*DTIPlaygroundTools required. NIfTI DTI is not supported yet.*

The configured directory can be generated from DTIPlaygroundLab (UI):

```
$ dmriatlas build-dir configured/directory
```

### Graphical interface

In DTIPlaygroundLab (`$ dmriplaygroundlab`), go to DMRI AtlasBuilder -> configure -> Generate Output Directory or
Execute.

## Supported image formats

- NRRD
- NIfTI (dmriprep and dmrifiberprofile; dmriatlas does not read NIfTI tensors yet)

## Requirements

- Python 3.9 - 3.12 and its development package (`python-dev` / `python-devel`)
- FSL >= 6.0 and DTIPlaygroundTools, installed with `dmriplayground install-tools` (see
  [FSL and DTIPlaygroundTools](#fsl-and-dtiplaygroundtools))
- Optionally MRtrix3, for the `mrtrix3` option of MULTI_SHELL_Estimate

The Python dependencies are installed by pip. They are declared in `setup.py`, and `requirements.txt` pins the exact
versions the releases are tested with (currently numpy 1.26, dipy 1.9, nibabel 5.2, pandas 2.2, pynrrd 1.0, and
SimpleITK, VTK, fury, torch, matplotlib, scikit-image, scikit-learn, scipy, dmri-amico, flask and the report
libraries). Consult those two files rather than this list. numpy stays below 2.x.

## Contributors

### Developers

- Styner, Martin (main developer) - Neuro Image Research and Analysis Laboratory , University of North Carolina @ Chapel Hill, U.S.
- Park, Sang Kyoon -  Neuro Image Research and Analysis Laboratory , University of North Carolina @ Chapel Hill, U.S.
- Johanna Dubos - Neuro Image Research and Analysis Laboratory , University of North Carolina @ Chapel Hill, U.S. / CPE Lyon, France
- Teyssier, Timothée - Neuro Image Research and Analysis Laboratory, University of North Carolina @ Chapel Hill, U.S. / CPE Lyon, France

### Discussion / test

- Foster, Mark - Neuro Image Research and Analysis Laboratory , University of North Carolina @ Chapel Hill, U.S.
- Vladisova, Roza - Neuro Image Research and Analysis Laboratory , University of North Carolina @ Chapel Hill, U.S.
- Hong, Yoonmi - Neuro Image Research and Analysis Laboratory , University of North Carolina @ Chapel Hill, U.S.

## References

- [Quality Control of Diffusion Weighted Images - Zhexing Liu, et al](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3864968/)

## Acknowledgements

This software has been supported by the following NIH grants: R01HDO55741, U54HDO79124, R01EB021391,
P50HD103573.

## License

MIT

## Roadmap

- Multi node computing with Kubernetes
- `dmriautotract` (automatic tractography), planned. The command and the module scaffolding are in place
  (`dtiplayground/dmri/tractography`, with the same `init` / `add-module` / `make-protocols` / `run` commands as
  dmriprep and a module template); the processing modules are still to be written, so the tool does nothing yet.
  Add a module to `dtiplayground/dmri/tractography/modules` as described in
  [Writing a module](#writing-a-module)

## Change log

See [CHANGELOG.md](https://github.com/NIRALUser/DTIPlayground/blob/master/CHANGELOG.md).
