Metadata-Version: 2.5
Name: fw-gear-file-classifier
Version: 0.10.0
Summary: Generic File Classifier
Project-URL: Repository, https://gitlab.com/flywheel-io/flywheel-apps/file-classifier.git
Author-email: Flywheel <support@flywheel.io>
License-Expression: MIT
License-File: LICENSE
Keywords: Classification,Flywheel,Gears
Classifier: Topic :: Scientific/Engineering
Requires-Python: <4,>=3.12
Requires-Dist: backoff<3,>=2.2.1
Requires-Dist: flywheel-sdk>=21.1
Requires-Dist: fw-classification>=0.6.3
Requires-Dist: fw-file<4,>=3.0
Requires-Dist: fw-gear>=0.3.5
Requires-Dist: nibabel<6,>=5.2
Description-Content-Type: text/markdown

# File Classifier (file-classifier)

## Overview

The File Classifier gear, maintained by Flywheel and part of the Utility suite,
classifies files based on 1) their associated metadata and 2) a classification profile
containing declarative rules that act on that metadata. The profile is highly
customizable and can act on any metadata field available on the input file. The gear
ships with default profiles for CT, MR, PT and MG DICOM images and is built on top of
the [fw-classification](https://gitlab.com/flywheel-io/scientific-solutions/lib/fw-classification)
toolkit. It is designed to run after the
[file-metadata-importer](https://gitlab.com/flywheel-io/scientific-solutions/gears/file-metadata-importer)
gear, typically as a gear rule on file upload.

### Summary

Classifies a file by applying a profile of declarative rules against its existing
metadata, such as DICOM header fields. Ships with default profiles for CT, MR, PT, and
MG modalities, and accepts a custom profile for other file types or classification
schemes. Writes classification fields to the file's metadata. Requires metadata to
already be populated on the file in advance, for example by file-metadata-importer.

### Cite

N/A

### License

[MIT](LICENSE)

### Classification

*Category:* *Utility*

*Gear Level:*

- [X] Project
- [X] Subject
- [X] Session
- [X] Acquisition
- [X] Analysis

### Inputs

- *file-input*
  - __Name__: *file-input*
  - __Type__: *file*
  - __Optional__: *False*
  - __Classification__: *DICOM, DICOM ZIP archive or NIfTI (see
    [File Specifications](#file-specifications))*
  - __Description__: *File to classify.*
  - __Notes__: *DICOM files must already have their header extracted by
    file-metadata-importer. NIfTI files are classified from a matching JSON sidecar.*
- *profile*
  - __Name__: *profile*
  - __Type__: *file*
  - __Optional__: *True*
  - __Classification__: *source code*
  - __Description__: *Optional classification profile.*
  - __Notes__: *A YAML profile that replaces the default profiles. See
    [Custom Classification Profiles](#custom-classification-profiles).*

### Config

- *debug*
  - __Name__: *debug*
  - __Type__: *boolean*
  - __Description__: *Log debug messages.*
  - __Default__: *false*
- *remove_existing*
  - __Name__: *remove_existing*
  - __Type__: *boolean*
  - __Description__: *Clear existing classification.*
  - __Default__: *false*
- *tag*
  - __Name__: *tag*
  - __Type__: *string*
  - __Description__: *The tag to be added on input file upon run completion.*
  - __Default__: *file-classifier*
- *validate*
  - __Name__: *validate*
  - __Type__: *boolean*
  - __Description__: *If true the gear will validate the classification schema.*
  - __Default__: *false*

### Outputs

#### Files

This gear does not produce output files.

#### Metadata

- __File classification__: the classification set by the profile is written to the
  input file's classification (`file.classification`), in accordance with the
  classification schema defined on the Flywheel instance. For example, the
  classification appended might appear similar to:

  ```json
  {
      "file": {
          "classification": {
              "Intent": "Structural",
              "Measurement": "T1"
          }
      }
  }
  ```

- __Custom metadata__: custom classification keys (e.g. `file.classification.Custom`)
  and other file or parent container fields can be set by actions defined in the
  classification profile.
- __NIfTI companions__: when classifying a NIfTI, the same classification and
  modality are also applied to its JSON sidecar and to any matching `.bval`/`.bvec`
  files.
- __QC__: a `classification` QC result is added to the input file. It is `PASS`, with
  the list of profile blocks that evaluated to true, when at least one block matched,
  and `FAIL` otherwise.
- __Tag__: the value of the `tag` config option is added to the input file.

### Pre-requisites

#### Prerequisite Gear Runs

- *file-metadata-importer*: DICOM files are classified from the DICOM header metadata
  that the
  [file-metadata-importer](https://gitlab.com/flywheel-io/scientific-solutions/gears/file-metadata-importer)
  gear stores under `file.info.header.dicom`. It must be run on DICOM files before this
  gear.

#### Prerequisite Files

For NIfTI inputs, a JSON sidecar matching the NIfTI must be present in the same
container (see [Classifying NIfTI Files from a JSON Sidecar](#use-case-3-classifying-nifti-files-from-a-json-sidecar)).
Otherwise, this gear has no prerequisite files.

#### Prerequisite Metadata

The file must carry the metadata that the classification profile rules act on (e.g.
`file.info.header.dicom` for the default DICOM profiles). The Flywheel instance must
be version 21.6 or later; the gear exits with an error on older instances.

## Usage

### Description

The gear loads a classification profile, either the default `main.yaml` profile bundled
with the gear, which covers the CT, MR, PT and MG modalities, or the custom profile
provided as the `profile` input. Any contextual classification rules stored in the
project custom information are appended to the end of the profile. The profile is then
evaluated against the input file's metadata, and the resulting classification and
modality are written back to the file.

By default, the provided classification profiles classify medical DICOM images with
the following modalities:

- Computed Tomography (CT)
- Magnetic Resonance (MR)
- Positron Emission Tomography (PT)
- Mammography (MG)

Custom classification profiles can be created to classify other types of files.

If `remove_existing` is enabled, the existing file classification is cleared before
classifying. If `validate` is enabled, the gear checks that the file modality is
defined on the instance and that the instance's modality classification schema matches
the schema bundled with the gear, and fails otherwise. See
[how to modify the classification schema](https://gitlab.com/flywheel-io/scientific-solutions/lib/fw-classification-profiles/-/blob/main/README.md?ref_type=heads#how-to-modify-the-classification-schema).

#### File Specifications

##### *file-input*

Currently the gear supports classification of the following file types:

- DICOM and DICOM ZIP archive, via the `file.info.header.dicom` namespace which is
  populated using the
  [file-metadata-importer](https://gitlab.com/flywheel-io/scientific-solutions/gears/file-metadata-importer)
  gear.
- NIfTI, classified from a JSON sidecar that shares the NIfTI's basename and is in the
  same container as the input. See
  [NIfTI Classification via JSON Sidecar](#nifti-classification-via-json-sidecar).

##### *profile*

A YAML classification profile. Detailed documentation for writing classification
profiles can be found in the
[profile](https://flywheel-io.gitlab.io/scientific-solutions/lib/fw-classification/fw-classification/profile/)
section of the fw-classification library documentation. See
[Custom Classification Profiles](#custom-classification-profiles).

#### NIfTI Classification via JSON Sidecar

DICOM files are classified from the DICOM header metadata that the
file-metadata-importer gear stores under `file.info.header.dicom`. NIfTI files do not
carry that metadata, so the gear classifies a NIfTI from a JSON sidecar instead.

When the input file is a NIfTI, the gear locates a JSON sidecar in the same container
whose basename matches the NIfTI (for example `REST_10m_bold.json` for
`REST_10m_bold.nii.gz`) and whose Flywheel file type is `source code`. It reads the
sidecar, converts BIDS timing fields (`RepetitionTime`, `EchoTime`, `InversionTime`,
`ExposureTime`, `FrameDuration`) from seconds to milliseconds, and uses those values
as the input to classification. The resulting classification and modality are written
to the NIfTI and mirrored to the sidecar `.json` and any matching `.bval`/`.bvec`
files.

> __Note__: The sidecar's contents are used to *derive* the classification; they are
> not stored as metadata on the NIfTI. The gear does not import the JSON's fields into
> the file's metadata. If you need the sidecar's values imported as metadata, use a
> curation gear such as file-curator with a curation script.

See
[Classifying NIfTI Files from a JSON Sidecar](#use-case-3-classifying-nifti-files-from-a-json-sidecar)
for the setup requirements and a full walkthrough.

#### Custom Classification Profiles

Detailed documentation for creating classification profiles can be found in the
[profile](https://flywheel-io.gitlab.io/scientific-solutions/lib/fw-classification/fw-classification/profile/)
section of the fw-classification library documentation.

Custom classification profiles can be included several ways.

##### 1. Custom Profile File (Recommended)

You can create a custom classification profile and include it as the `profile` input
file when configuring the gear. This completely overrides the default classification
profiles, so you may want to include the default profiles, or sections of the profiles
in your custom profile if you wish to retain some of the default behavior. You can find
the default profiles in the
[fw-classification-profiles](https://gitlab.com/flywheel-io/scientific-solutions/lib/fw-classification-profiles)
library.

The custom profile can reference the default profiles bundled with the gear using
relative paths in the `includes` section. At runtime, relative includes are resolved
against the gear's built-in `classification_profiles/` directory, so you do not need to
provide absolute paths or git URLs.

###### Example: Extend all default profiles with a custom rule

This profile loads all default classification profiles (MR, CT, PT, MG) via
`main.yaml`, then adds a custom block that flags DICOM files as `Deleted` when the
`ProtocolName` header field is set to `Deleted`.

```yaml
name: Custom classifier
includes:
  - main.yaml  # Loads all default profiles (MR, CT, PT, MG)

profile:
  - name: set_custom_deleted
    description: |
      Set custom deleted classification if ProtocolName was deleted
    rules:
      - match_type: 'all'
        match:
          - key: file.type
            is: dicom
          - key: file.info.header.dicom.ProtocolName
            is: 'Deleted'
        action:
          - key: file.classification.Custom
            add: 'Deleted'
```

You can also include only a specific modality profile instead of all defaults by
referencing it directly (e.g., `MR/main.yaml`, `CT/main.yaml`, `PT.yaml`, `MG.yaml`).

###### Example: Include a profile from a git URL

You can include profiles directly from a git repository using the format
`git_repo_url$path/to/file` or `git_repo_url$path/to/file$branch`. The profile is
cloned locally at runtime. This is useful when you want to reference a specific version
of the default profiles or a profile hosted in a different repository.

<!-- markdownlint-disable MD013 -->
```yaml
name: Custom classifier with remote include
includes:
  - https://gitlab.com/flywheel-io/scientific-solutions/lib/fw-classification-profiles$classification_profiles/MR/main.yaml

profile:
  # ... custom blocks go here
```
<!-- markdownlint-enable MD013 -->

##### 2. Contextual Classifications via Project Custom Information

You can also include custom classification blocks by adding them to the __project
custom information__ under the `classifications` key. This is useful when you want to
extend the current classification with a small set of rules for a given project. In
this case, the rules are appended after the profiles have been selected (either default
or by the custom profile method above). This can be done via the UI or the SDK.

> __Note__: This method appends rules to the classification profile after profile
> selection, which means that they have the highest priority.

###### Example: Adding a ProtocolName block via the SDK

```python
import flywheel
fw = flywheel.Client()
proj = fw.get_project(<proj_id>) # or use lookup()
existing_info = proj.info

# Initialize context classifications if they don't exist
existing_info.setdefault('classifications', [])
existing_info['classifications'].append(
    {
        'match': [
            {
                'key': 'file.type',
                'is': 'dicom',
            },
            {
                'key': 'file.info.header.dicom.ProtocolName',
                'is': 'deleted',
            }
        ],
        'action': [
            {'key': 'file.classification.Custom', 'add': 'Deleted'},
        ]
    }
)
proj.replace_info(existing_info)
```

###### Example: Adding a ProtocolName block via the UI

Navigate to the project "Information" tab for your project, then add the following
block to the "Custom Information" section:

![Custom classification block in project custom information](docs/assets/images/custom-classifications-ui.png)

### Workflow

```mermaid
flowchart LR
    A[Input file] --> B[Load profile:<br/>default or custom]
    B --> C[Append project<br/>context rules]
    C --> D[Evaluate rules<br/>on metadata]
    D --> E[Write classification,<br/>modality and QC]
    E --> F[Tag input file]
```

1. The gear checks the Flywheel instance version and, if enabled, clears the existing
   classification and validates the modality classification schema.
2. The classification profile is loaded: the `profile` input if provided, otherwise the
   default `main.yaml` profile.
3. Contextual classification rules from `project.info.classifications` are appended to
   the profile.
4. The profile is evaluated against the file metadata (for NIfTI, against the matching
   JSON sidecar).
5. The classification, modality and any other fields set by the profile are written
   back to the file (and NIfTI companions), and a `classification` QC result is
   recorded.
6. The input file is tagged with the `tag` config value.

### Use Cases

#### Use Case 1: Synergy with other gears

The `file-classifier` gear is part of a suite of gears designed to work synergistically
to help you curate and manage your files. The gear is designed to run after the
`file-metadata-importer` gear to classify files and augment the metadata extracted by
the `file-metadata-importer` gear.

#### Use Case 2: Gear Rules

The `file-classifier` gear is a good candidate to be run as a gear rule automatically
on file upload (triggered after the `file-metadata-importer` gear). This also allows for
more advanced workflows to run via gear rules triggered on specific classifications.
For example, if you have a rule that classifies a file as a T1-weighted MRI image, you
can then trigger a gear rule to automatically run a gear that processes T1-weighted MRI
images.

#### Use Case 3: Classifying NIfTI Files from a JSON Sidecar

Unlike DICOM, a NIfTI file does not store rich acquisition metadata in its header. To
classify a NIfTI, the gear reads a JSON sidecar (for example, a BIDS sidecar produced
by `dcm2niix`) that sits alongside the NIfTI in the same container.

##### Requirements

For the gear to find and use the sidecar, all of the following must be true:

- __Same container.__ The NIfTI and its JSON sidecar must be in the same acquisition
  (or other container). The gear only looks for the sidecar among the files of the
  NIfTI's own parent container.
- __Matching basename.__ The sidecar must share the NIfTI's basename, differing only
  by extension. `REST_10m_bold.json` pairs with `REST_10m_bold.nii.gz` (or
  `REST_10m_bold.nii`); a differently named sidecar such as `REST_bold.json` will not
  pair with `REST_10m_bold.nii.gz`.
- __`source code` file type.__ The sidecar must have the Flywheel file type
  `source code`, which is the default type assigned to `.json` files. If the JSON has
  been reclassified as another type, the gear will not recognize it as a sidecar.
- __Run on the NIfTI.__ Run the gear (or configure the gear rule) with the NIfTI as
  the input file. The gear finds the sidecar itself; running it on the JSON file does
  not trigger the sidecar lookup.

##### Example

Consider an acquisition that contains a NIfTI and its matching sidecar:

- `REST_10m_bold.nii.gz`, with Flywheel file type `nifti`
- `REST_10m_bold.json`, with Flywheel file type `source code`

![Acquisition containing a NIfTI and its matching JSON sidecar](docs/assets/images/nifti-json-acquisition.png)

When the gear runs on `REST_10m_bold.nii.gz`, it:

1. Finds `REST_10m_bold.json` in the same acquisition by matching basename.
2. Reads the sidecar and converts BIDS timing fields (`RepetitionTime`, `EchoTime`,
   `InversionTime`, `ExposureTime`, and `FrameDuration`) from seconds to milliseconds.
3. Classifies the NIfTI from the sidecar values using the active profile and writes the
   result to the file classification. The same classification and modality are also
   applied to the sidecar `.json` and to any matching `.bval`/`.bvec` files.

A second pair, such as `Coronal_Baseline_T1.nii.gz` with `Coronal_Baseline_T1.json`,
works the same way.

> __Note__: The gear classifies the NIfTI *from* the sidecar; it does not copy the
> sidecar's fields onto the NIfTI as metadata. See the [FAQ](#faq) for what is and
> isn't imported.

__Warning__: If the sidecar is not yet present in the container when the gear runs,
the gear waits and retries for a short time (about 30 seconds) before failing with
"Could not find NIfTI sidecar." When using a gear rule, make sure the sidecar lands in
the same container as the NIfTI.

### Logging

This gear uses standard Python logging via the `fw-gear` library. Set the `debug`
config option to `true` for verbose debug output. The logs report which blocks of the
classification profile evaluated to true, which helps troubleshoot unexpected
classifications.

## FAQ

__Q: How do I create a custom classification profile?__

A: Follow the instructions in the
[profile](https://flywheel-io.gitlab.io/scientific-solutions/lib/fw-classification/fw-classification/profile/)
section of the fw-classification library documentation, and see
[Custom Classification Profiles](#custom-classification-profiles).

__Q: I ran the classifier gear but my file is not classified. What's wrong?__

A: The first thing you should check is the logs of the gear job to see if there were
any obvious errors that would cause the gear to fail. If there are no errors, the logs
also produce messages which indicate which portions of the classification profile
evaluated to true. If the issue is still unclear, check whether the file has the
necessary metadata for the profiles to work. If the file does not have the necessary
metadata, the classifier gear will not be able to classify the file. Often, this is
because either the `file-metadata-importer` has not been run on the file, or the file
has been stripped of metadata in some way (e.g. anonymized/de-identified).

__Q: What resources are needed to run the gear?__

A: This gear can be run on any instance type and does not require any specialized
hardware.

__Q: How long does it take to run the gear?__

A: This gear usually takes a few seconds (<10s) to run.

__Q: How are NIfTI files classified, and is my JSON imported?__

A: NIfTI files are classified from a JSON sidecar (for example a BIDS sidecar) that
shares the NIfTI's basename, lives in the same container, and has the Flywheel file
type `source code`, for example `REST_10m_bold.json` next to `REST_10m_bold.nii.gz`.
The gear reads the sidecar to derive the classification and writes the result to the
file classification. It does __not__ copy the sidecar's fields onto the NIfTI as
metadata. To import the JSON's values as metadata, use a curation gear such as
file-curator with a curation script. See
[Classifying NIfTI Files from a JSON Sidecar](#use-case-3-classifying-nifti-files-from-a-json-sidecar)
for details.

## Contributing

For information on how to contribute to this gear, please see
[CONTRIBUTING.md](CONTRIBUTING.md).
