Metadata-Version: 2.4
Name: migenpro
Version: 1.0.0
Summary: Microbial Genome Prospecting (MiGenPro) combines phenotype and genomic linked data. Migenpro serves as a framework for the generation of machine learning models that predict microbial traits from genome sequences. 
License: MIT
License-File: LICENSE
Keywords: machine-learning,genomics,phenotype-prediction,microbial,bioinformatics
Author: Mike Loomans
Author-email: mike.loomans@wur.nl
Maintainer: Mike Loomans
Maintainer-email: mike.loomans@wur.nl
Requires-Python: >=3.10,<3.15
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Provides-Extra: taxonomy
Requires-Dist: PyQt5
Requires-Dist: bio (==1.7.1)
Requires-Dist: cwltool (>=3.1.2)
Requires-Dist: dask
Requires-Dist: docker
Requires-Dist: ete4 (==4.3.0) ; extra == "taxonomy"
Requires-Dist: imbalanced-learn (>=0.13.0)
Requires-Dist: importlib_resources
Requires-Dist: ipython
Requires-Dist: jinja2
Requires-Dist: matplotlib
Requires-Dist: notebook (>=7.4.4,<8.0.0)
Requires-Dist: numpy (>=2.1.3)
Requires-Dist: pandas (>=2.2.3)
Requires-Dist: pytest (>=8.4.1,<9.0.0)
Requires-Dist: requests
Requires-Dist: scikit-learn (>=1.5.2,<1.6)
Requires-Dist: seaborn (==0.13.2)
Requires-Dist: shap (>=0.48.0)
Requires-Dist: xgboost (>=2.0.0)
Project-URL: Documentation, https://gitlab.com/wurssb/migenpro/-/blob/dev/README.md
Project-URL: Homepage, https://gitlab.com/wurssb/migenpro
Project-URL: Repository, https://gitlab.com/wurssb/migenpro
Description-Content-Type: text/markdown

![Coverage](https://migenpro-f51bc2.gitlab.io/coverage.svg)
![codequality](https://migenpro-f51bc2.gitlab.io/pylint.svg)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

# MiGenPro - Microbial Genome Processing Toolkit
MiGenPro: A flexible linked data framework for phenotype-genotype prediction of microbial traits using machine learning.

## Functionalities
- Genome annotation of genome accessions (downloaded automatically) or your own FASTA files.
- Data formatting and cleaning for microbial genome datasets.
- Conversion of raw query data into structured feature and phenotype matrices.
- Advanced filtering options to remove low-frequency features or phenotypes.
- Feature selection via variance thresholding and mutual information ranking.
- Automatic resampling with SMOTE (numerical), SMOTEN (categorical), or SMOTENC (mixed).
- Parallel processing support for efficient handling of large datasets.
- Easy training and prediction with machine learning models on microbial characteristics.

## Quickstart

### Prerequisites
- **Docker** is required for the genome annotation step (`--annotation`). Ensure the Docker daemon is running and your user has permission:
  ```bash
  sudo usermod -aG docker $USER
  # Log out and back in for group changes to take effect
  ```
- **Java** (JRE 11+) is required for SAPP-based SPARQL queries. The SAPP jar (v2.2.0) is included in the repository at `binaries/SAPP-2.0.jar`.
- **Git LFS** is required to check out the large files in this repository: the BacDive phenotype graph (`data/bacdive.hdt.gz`), the SAPP jar, the manuscript matrices and the manuscript models. Install it (e.g. `conda install -c conda-forge git-lfs` or your package manager) and run `git lfs install` once **before** cloning. If you cloned without it, run `git lfs pull` inside the repository afterwards. Without Git LFS these files are small text pointers and the commands below fail.

### Installation
MiGenPro targets Python >= 3.10. Install from source using Poetry (recommended) or pip:

```bash
git lfs install
git clone https://gitlab.com/wurssb/migenpro.git
cd migenpro
conda create -n migenpro python=">=3.10,<3.15"
conda activate migenpro
# Using Poetry (recommended)
poetry install
# Or using pip
pip install -r installation/requirements.txt
pip install -e .
```

### Full demo (end-to-end)
This command queries phenotypes, downloads and annotates 20 genomes, queries the annotations, then trains and predicts:
```bash
migenpro --df --gq --ml --annotation \
  --sapp_jar ./binaries/SAPP-2.0.jar \
  --phenotype_query_file sparql_phenotype:demo_gram.sparql \
  --phenotype_hdt_file ./data/bacdive.hdt.gz \
  --genome_query_file sparql_genome:DomainCopyNumber.sparql \
  --abs_frequency 1 \
  --threads 20 \
  --sampling_type auto --train --predict --output ./demo_output
```

**Note:** The `--annotation` step downloads genomes from ENA/NCBI and runs the full CWL annotation workflow (Bakta, InterProScan, etc.) in Docker containers. With 20 demo genomes this typically takes 30-60 minutes depending on hardware and network.

### ML only (skip annotation, provide your own matrices)
If you already have feature and phenotype matrices, you can skip the annotation and genome querying steps:
```bash
migenpro --ml \
  --feature_matrix ./output/feature_matrix.tsv \
  --phenotype_matrix ./output/phenotype_matrix.tsv \
  --sampling_type auto --train --predict --output ./output/
```

### Sampling strategies
| Flag | Use when |
|------|----------|
| `--sampling_type SMOTE` | All features are numerical (e.g. protein domain counts) |
| `--sampling_type SMOTEN` | All features are categorical/nominal |
| `--sampling_type SMOTENC` | Mixed numerical and categorical features |
| `--sampling_type auto` | Let MiGenPro detect feature types automatically |

### Feature selection
Features with zero variance (constant across all samples) are always removed before training.
Additional filtering options:

| Flag | Description |
|------|-------------|
| `--min_variance 0.01` | Remove features with variance below the given threshold |
| `--top_percentile 20` | Keep only the top N% of features ranked by mutual information |

When `--top_percentile` is used, variance filtering runs first to strip stale features, then mutual information is computed on the reduced set for speed.

The `--param_grids` flag can be used to optimise the parameters of the machine learning models.
An example json file is available at: `tests/resources/param_grid.json`

The individual steps:
1. Querying phenotype graphs
2. Annotating genomes
3. Querying the annotated genomes
4. Training machine learning models
5. Predicting phenotypes with existing models
6. Feature importance analysis
7. Summarising the results


#### 1. Querying phenotype graphs
```bash
migenpro --df \
    --phenotype_query_file sparql_phenotype:demo_gram.sparql \
    --phenotype_hdt_file ./data/bacdive.hdt.gz \
    --abs_frequency 1 \
    --sapp_jar binaries/SAPP-2.0.jar \
    --output ./output/
```

#### 2. Annotating genomes
Genome annotation is done by default using the workflow: https://workflowhub.eu/workflows/1170/
this can be changed using the `--cwl_file` flag with a workflow of your choice granted that it takes a fasta file as input.

**Note on `--threads`:** genomes are annotated **one after another** (sequentially). `--threads` controls the number of threads *each* genome's workflow may use (Bakta, InterProScan, ...), not the number of genomes annotated in parallel. Annotating many genomes therefore takes a multiple of the single-genome time; to speed things up you can start several `migenpro --annotation` runs in parallel (finished genomes are skipped on restart).

By default the genomes listed in the first column of the phenotype matrix from step 1 are downloaded from ENA and annotated.
Genome accessions are recognised with or without a version suffix (`GCA_001038485` or `GCA_001038485.1`).
For more than ~100 genomes, pass the [NCBI datasets CLI](https://www.ncbi.nlm.nih.gov/datasets/docs/v2/command-line-tools/download-and-install/) with `--dataset_bin` to download from NCBI instead.
```bash
migenpro --annotation \
    --phenotype_matrix ./output/phenotype_matrix.tsv \
    --threads 8 \
    --output ./output/
```
Annotations are written to `<output>/genomes/` (or `<genome_dir>/genomes/` with `--genome_dir`); genomes that already have an annotation (`*.SAPP.hdt.gz`) are skipped, so an interrupted run can simply be restarted.
Add `--yaml_only` to only write the workflow input files without running the workflow.

#### Annotating your own genomes (FASTA)
To annotate your own assemblies instead of downloading genomes, pass them with `--fasta`. It accepts:
- a single FASTA file (`.fasta`, `.fna` or `.fa`, optionally gzipped, e.g. `isolate_01.fasta.gz`),
- a directory: every FASTA file directly inside it is annotated,
- a text file with one FASTA path or URL per line.

```bash
migenpro --annotation \
    --fasta ./my_genomes/ \
    --threads 8 \
    --output ./output/

migenpro --gq \
    --genome_query_file sparql_genome:DomainCopyNumber.sparql \
    --output ./output/
```

**Genome identifiers.** Each genome is named after its file without the FASTA and gzip extensions: `my_genomes/isolate_01.fasta.gz` becomes `isolate_01`.
If the file name contains an assembly accession (e.g. `GCA_000005845.2_ASM584v2_genomic.fna.gz`), that accession is used instead.
This identifier is the row name in `feature_matrix.tsv`, so use the same identifiers in the first column of your phenotype matrix:

```
# genome	phenotype
isolate_01	positive
isolate_02	negative
```

Feature and phenotype matrices are matched on the **first 13 characters** of the identifier (the length of an accession such as `GCA_000005845`), so keep your file names unique within their first 13 characters (`isolate_01`, `isolate_02`, ... is fine; `my_sequencing_run_A` and `my_sequencing_run_B` are not).

You can then train on your genomes (step 4) or predict with an existing model (step 5).

#### Predicting phenotypes for your own genome(s)
The repository ships pre-trained models (trained on the manuscript data) under
`migenpro/data/manuscript_models/` for the phenotypes `gram`, `motility`, `oxygen`,
`temperature` and `spore`. To predict these phenotypes for one of your own genomes:

1. Annotate the genome and build its feature matrix (steps 2 and 3), either from an
   accession or your own FASTA:

   ```bash
   migenpro --annotation --gq \
       --fasta ./my_genome.fasta.gz \
       --genome_query_file sparql_genome:DomainCopyNumber.sparql \
       --sapp_jar binaries/SAPP-2.0.jar \
       --output ./my_prediction/
   ```

2. Write a phenotype matrix that lists your genome with a placeholder label (labels are
   ignored in prediction-only mode):

   ```
   # genome	phenotype
   my_genome	query
   ```

3. Combine your genome's feature matrix with the feature matrix the model was trained on
   (the domain columns must be aligned; columns missing from your genome's matrix are
   treated as zero), then load the model and predict:

   ```bash
   migenpro --ml \
       --load_model migenpro/data/manuscript_models/migenpro_xgb_models/motility_XGBClassifier.pkl \
       --predict \
       --feature_matrix ./combined_feature_matrix.tsv \
       --phenotype_matrix ./my_phenotype_matrix.tsv \
       --output ./my_prediction/predictions/
   ```

The prediction and its per-class confidence are in
`./my_prediction/predictions/protein_domains/XGBClassifier/XGBClassifier-predictions.tsv`.

#### 3. Querying the annotated genomes
Runs a SPARQL query on every annotation (`*.hdt.gz`) under `<output>/genomes/` (or under `--genome_dir`) and combines the results into `<output>/feature_matrix.tsv`.

```bash
migenpro --gq \
    --genome_query_file sparql_genome:DomainCopyNumber.sparql \
    --sapp_jar binaries/SAPP-2.0.jar \
    --output ./output/
```

#### 4. Training machine learning models
We will now use the default parameters for training the models.
If you wish to optimise the parameters you can do this using the `--param_grids` flag.
```bash
migenpro --ml \
      --feature_matrix ./output/feature_matrix.tsv \
      --phenotype_matrix ./output/phenotype_matrix.tsv \
      --output ./output/
```

With feature selection and resampling:
```bash
migenpro --ml \
      --feature_matrix ./output/feature_matrix.tsv \
      --phenotype_matrix ./output/phenotype_matrix.tsv \
      --min_variance 0.01 \
      --top_percentile 20 \
      --sampling_type auto \
      --output ./output/
```

#### 5. Predicting phenotypes with existing models
Trained models are saved as pickle files under
`<output>/protein_domains/<Classifier>/<Classifier>_protein_domains.pkl`.
To predict with such a model instead of training a new one, pass it with `--load_model` together with `--predict`.
The feature matrix must contain the same protein domain features the model was trained on (see steps 2 and 3).

**Prediction-only mode.** When `--load_model` and `--predict` are given without `--train`,
MiGenPro skips the train/test split and predicts on **every genome** listed in the
phenotype matrix that is also present in the feature matrix. The phenotype labels are
not used, so a single row with a placeholder label is enough — this is the way to
predict phenotypes for new genomes with pre-trained models. Results are written to
`<output>/protein_domains/<Classifier>/<Classifier>-predictions.tsv`:


```bash
migenpro --ml \
      --load_model ./output/protein_domains/RandomForestClassifier/RandomForestClassifier_protein_domains.pkl \
      --predict \
      --feature_matrix ./output/feature_matrix.tsv \
      --phenotype_matrix ./output/phenotype_matrix.tsv \
      --output ./output/predictions/
```

With `--train` (or without `--load_model`), the previous behaviour is kept: predictions
are made on the train/test split of the intersected genomes and scored against the
observed phenotypes (`<Classifier>-{train,test}.tsv`).


#### 6. Feature importance analysis
```bash
migenpro --fi \
        --models path/to/models \
        --feature_matrix path/to/features.tsv \
        --phenotype_matrix path/to/phenotype.tsv \
        --output ./output/
```

#### 7. Summarising the results

Wait for the script to finish and retrieve the results of your prediction from the output directory.
There the predictions are given in the following format:

| Genome | Phenotype | Prediction | Confidence |
|--------|-----------|------------|------------|
| GCA123 | Temperature | mesophilic | 0.96 |

```bash
migenpro --summarise \
        --output ./output/
```

## Contributing
Pull requests are welcome. Please run the test suite before submitting:
```bash
python -m pytest tests/              # fast tests
python -m pytest --run-main tests/   # also runs the resource-intensive integration tests
```

### Clone the repository
```bash
git clone https://gitlab.com/wurssb/migenpro.git
cd migenpro
```

### Installing the development dependencies
```bash
conda create -n migenpro python=">=3.10,<3.15"
conda activate migenpro
poetry install
```

## Recreating the results from the study
The input matrices are included in this repository under `migenpro/data/manuscript_matrices/`, and the complete study output is archived on [Zenodo](https://zenodo.org/records/16995284). For step-by-step instructions to reproduce every phenotype and experiment track, see **[REPRODUCING_RESULTS.md](REPRODUCING_RESULTS.md)**.

## Citation
If you use MiGenPro in your research, please cite the MiGenPro publication. The study output is archived on [Zenodo](https://zenodo.org/records/16995284).

## License
MiGenPro is released under the [MIT License](LICENSE). Data files under `data/`, `output/`, and `tests/resources/genomes` are released under CC0 1.0 Universal.

## Maintainers
Jasper J. Koehorst (@jjkoehorst) and Mike Loomans (@MikeLoomans1999)

