Metadata-Version: 2.4
Name: pg-sui
Version: 1.8.7
Summary: Python machine and deep learning API to impute missing genotypes
Author-email: "Drs. Bradley T. Martin and Tyler K. Chafin" <evobio721@gmail.com>
Maintainer-email: "Dr. Bradley T. Martin" <evobio721@gmail.com>
License: GNU General Public License v3 (GPLv3)
Project-URL: Homepage, https://github.com/btmartin721/PG-SUI
Project-URL: Documentation, https://pg-sui.readthedocs.io/en/latest/
Project-URL: Source, https://github.com/btmartin721/PG-SUI.git
Project-URL: BugTracker, https://github.com/btmartin721/PG-SUI/issues
Keywords: impute,imputation,AI,deep learning,machine learning,neural network,vae,autoencoder,ubp,nlpca,population genetics,unsupervised,supervised,bioinformatics,snp,genomics,genotype,missing data,data analysis,data science,statistics,data visualization,python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Scientific/Engineering :: Visualization
Classifier: Operating System :: MacOS
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Unix
Classifier: Operating System :: POSIX
Classifier: Natural Language :: English
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: matplotlib<4.0,>=3.8
Requires-Dist: numpy<3.0,>=2.2
Requires-Dist: pandas<3.0,>=2.2
Requires-Dist: scikit-learn<2.0,>=1.4
Requires-Dist: scipy<2.0,>=1.13
Requires-Dist: seaborn<0.14,>=0.13.2
Requires-Dist: snpio>=1.7.4
Requires-Dist: optuna<5.0,>=4.6
Requires-Dist: rich
Requires-Dist: rich[jupyter]
Requires-Dist: torch<3.0,>=2.12
Requires-Dist: toytree
Requires-Dist: tqdm
Requires-Dist: fastapi[standard]>=0.110
Requires-Dist: uvicorn[standard]>=0.23
Provides-Extra: intel
Requires-Dist: scikit-learn-intelex; extra == "intel"
Provides-Extra: docs
Requires-Dist: sphinx; extra == "docs"
Requires-Dist: sphinx-rtd-theme; extra == "docs"
Requires-Dist: sphinxcontrib-napoleon; extra == "docs"
Requires-Dist: sphinxcontrib-programoutput; extra == "docs"
Provides-Extra: dev
Requires-Dist: ruff; extra == "dev"
Requires-Dist: twine; extra == "dev"
Requires-Dist: wheel; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: sphinx; extra == "dev"
Requires-Dist: sphinx-rtd-theme; extra == "dev"
Requires-Dist: sphinxcontrib-napoleon; extra == "dev"
Requires-Dist: sphinxcontrib-programoutput; extra == "dev"
Requires-Dist: requests; extra == "dev"
Provides-Extra: optional
Requires-Dist: PyObjC; extra == "optional"
Provides-Extra: gui
Requires-Dist: fastapi>=0.110; extra == "gui"
Requires-Dist: uvicorn[standard]>=0.23; extra == "gui"
Dynamic: license-file

# PG-SUI

![PG-SUI Logo: Stylized blue and purple gradient design with faded appearance representing PG-SUI - Population Genomic Supervised and Unsupervised Imputation](https://github.com/btmartin721/PG-SUI/blob/master/img/pgsui-logo-faded.png)

Population Genomic Supervised and Unsupervised Imputation.

## About PG-SUI

PG-SUI is a Python 3 API that uses machine learning to impute missing values from population genomic SNP data. There are several supervised and unsupervised machine learning algorithms available to impute missing data, as well as some non-machine learning imputers that are useful.

Below is some general information and a basic tutorial. For more detailed information, see our [API Documentation](https://pg-sui.readthedocs.io/en/latest/).

## How PG-SUI works

PG-SUI fills missing SNP genotypes using unsupervised neural networks, supervised machine-learning models, or deterministic methods. Its neural models learn to reconstruct genotype calls using observed values as training targets. To evaluate an imputer, PG-SUI temporarily masks some observed genotypes, predicts them, and compares the predictions with their known values. After fitting, it fills the genotypes that were missing in the original input. Those calls have no known truth, so they are excluded from accuracy scores.

The following schematic animation illustrates the variational autoencoder (VAE) genotype imputation workflow. Its predictions are illustrative, not measured results.

![VAE workflow showing genotype input and validation masking, a neural encoder and decoder, and imputed output](./img/impute_vae_workflow.gif)

### Unsupervised Imputation Methods

Unsupervised imputers include four neural network models:

+ Variational Autoencoder (VAE) [1](#1)
  + Encodes genotypes as a latent distribution, then decodes them to reconstruct observed calls. Training excludes originally missing calls from the reconstruction loss; the trained model predicts those calls.
+ Autoencoder [2](#2)
  + Learns a lower-dimensional representation of the genotypes and reconstructs observed calls. Its predictions fill originally missing positions.
+ Nonlinear PCA (NLPCA)
  + Learns per-sample latent representations and a decoder network to reconstruct genotypes.
+ Unsupervised backpropagation (UBP)
  + Initializes sample representations with PCA, then refines the decoder and representations in stages.

The animations below show the four neural imputation workflows.

![Autoencoder workflow showing genotype input, encoder and decoder networks, and imputed output](./img/impute_autoencoder_workflow.gif)

![VAE workflow showing genotype input, a latent distribution, neural decoder, and imputed output](./img/impute_vae_workflow.gif)

![NLPCA workflow showing genotype input, sample embeddings, a neural decoder, and imputed output](./img/impute_nlpca_workflow.gif)

![UBP workflow showing genotype input, staged decoder training, projection, and imputed output](./img/impute_ubp_workflow.gif)

### Supervised Imputation Methods

Supervised methods use scikit-learn's `IterativeImputer`, which is based on multivariate imputation by chained equations (MICE) [3](#3). It predicts missing genotypes at each SNP site using other sites as features. Users can configure how many other SNP features are considered for each prediction. PG-SUI supports these classifiers:

+ ImputeRandomForest
+ ImputeHistGradientBoosting

See the [scikit-learn documentation](https://scikit-learn.org) for more information on IterativeImputer and each of the classifiers.

The animations below illustrate the supervised imputation models.

![Random forest imputation workflow showing genotype input, tree votes, and filled calls](./img/impute_random_forest_workflow.gif)

![Histogram gradient boosting workflow showing genotype input, binned features, boosted trees, and filled calls](./img/impute_hist_gradient_boosting_workflow.gif)

### Non-Machine Learning (Deterministic) Methods

We also include several deterministic options for imputing missing data, including:

+ Per-population mode per SNP site
+ Overall mode per SNP site
+ Reference allele (REF genotype)

The animations below illustrate each deterministic model.

![Reference-allele imputation workflow showing missing calls filled with the REF genotype](./img/impute_ref_allele_workflow.gif)

![Most-frequent imputation workflow showing per-locus genotype counts and filled calls](./img/impute_most_frequent_workflow.gif)

## Installing PG-SUI

PG-SUI supports both pip and conda distributions. Both are kept current with up-to-date releases.

### Installation with Pip

To install PG-SUI with pip, do the following. It is strongly recommended to install pg-sui in a virtual environment.

``` shell
python3 -m venv .pgsui-venv
source .pgsui-venv/bin/activate
pip install pg-sui
```

### Installation with Anaconda

To install PG-SUI with Anaconda, do the following:

``` shell
conda create -n pgsui-env --strict-channel-priority --override-channels \
    -c btmartin721 -c conda-forge -c bioconda \
    python=3.12 pg-sui
conda activate pgsui-env
```

### Docker Container

We also maintain a Docker image that comes with PG-SUI preinstalled. This can be useful for automated workflows such as Nextflow or Snakemake.

``` shell
docker pull btmartin721/pg-sui:latest
docker run -it --rm btmartin721/pg-sui:latest pg-sui --help
```

### Optional MacOS GUI

PG-SUI ships an optional Electron GUI (Graphical User Interface) wrapper around the Python CLI. Currently for the GUI, only MacOS is supported.

1. Install the Python-side extras (FastAPI/ uvicorn helper) if you want to serve from Python:
   `pip install pg-sui[gui]`
2. Install [Node.js](https://nodejs.org) and fetch the app dependencies:
   `pgsui-gui-setup`
3. Launch the graphical interface:
   `pgsui-gui`

The GUI shells out to the same CLI underneath, so presets, overrides, and YAML configs behave identically.

## Input Data

You can read your input files as a GenotypeData object from the [SNPio](https://snpio.readthedocs.io/en/latest/) package. SNPio supports the VCF, PHYLIP, STRUCTURE, and GENEPOP input file formats.

``` python
# Import snpio. Automatically installed with pg-sui.
from snpio import VCFReader

# Read in VCF alignment.
# SNPio also supports PHYLIP, STRUCTURE, and GENEPOP input file formats.
data = VCFReader(
    filename="pgsui/example_data/phylogen_subset14K.vcf.gz",
    popmapfile="pgsui/example_data/popmaps/phylogen_nomx.popmap", # optional
    force_popmap=True, # optional
)
```

## Supported Imputation Methods

There are several supported algorithms PG-SUI uses to impute missing data. Each one can be run by calling the corresponding class. You must provide a GenotypeData instance as the first positional argument.

You can import all the supported methods with the following:

``` python
from pgsui import ImputeVAE, ImputeAutoencoder, ImputeRefAllele, ImputeMostFrequent, ImputeRandomForest, ImputeHistGradientBoosting
```

### Unsupervised Imputers

The four unsupervised imputers can be run by initializing them with the SNPio ``GenotypeData`` object and then calling ``fit()`` and ``transform()``.

``` python
# Initialize the models, then fit and impute
vae = ImputeVAE(data) # Variational autoencoder
vae.fit()
vae_imputed = vae.transform()

ae = ImputeAutoencoder(data) # standard autoencoder
ae.fit()
ae_imputed = ae.transform()
```

The ``*_imputed`` objects are NumPy arrays of IUPAC single-character codes that are compatible with SNPio's ``GenotypeData`` objects.

### Supervised Imputers

Various supervised imputation options are supported, and these use the same API design.

``` python
# Supervised IterativeImputer classifiers

# Random Forest
rf = ImputeRandomForest(data)
rf.fit()
imputed_rf = rf.transform()

# HistGradientBoosting
hgb = ImputeHistGradientBoosting(data)
hgb.fit()
imputed_hgb = hgb.transform()
```

### Non-machine learning methods

The following deterministic methods are supported. ``ImputeMostFrequent`` supports the mode-per-population or overall (global) mode options to inform imputation.

``` python
# Per-population, per-locus mode
pop_mode = ImputeMostFrequent(data, by_populations=True)
pop_mode.fit()
imputed_pop_mode = pop_mode.transform()

# Per-locus mode
mode = ImputeMostFrequent(data, by_populations=False)
mode.fit()
imputed_mode = mode.transform()
```

Or, always replace missing values with the reference allele.

``` python
ref = ImputeRefAllele(data)
ref.fit()
imputed_ref = ref.transform()
```

## Command-Line Interface

Run the PG-SUI CLI with ``pg-sui`` (installed alongside the library). The CLI follows the same precedence model as the Python API:

``code defaults  <  preset (--preset)  <  YAML (--config)  <  explicit CLI flags  <  --set key=value``.

Recent releases add explicit switches for the simulated-missingness workflow shared by the neural and supervised models:

+ ``--sim-strategy`` selects one of ``random``, ``random_weighted``, ``random_weighted_inv``, ``nonrandom``, ``nonrandom_weighted``.
+ ``--sim-prop`` sets the proportion of observed calls to temporarily mask when building the evaluation set.

Example:

``` shell
pg-sui \
  --input data.vcf.gz \
  --popmap pops.popmap \
  --models ImputeVAE ImputeAutoencoder \
  --preset balanced \
  --sim-strategy random_weighted_inv \
  --sim-prop 0.3 \
  --prefix ae_and_vae \
  --n-jobs 4 \
  --tune-n-trials 100 \
  --set tune.enabled=True
```

CLI overrides cascade into every selected model, so a single invocation can evaluate multiple imputers with a consistent simulation strategy and output prefix.

STRUCTURE inputs accept a few extra flags for parsing metadata:

``` shell
pg-sui \
  --input data.str \
  --format structure \
  --structure-has-popids \
  --structure-allele-start-col 2 \
  --structure-allele-encoding '{"1":"A","2":"C","3":"G","4":"T","-9":"N"}'
```

## References

1. Kingma, D.P. & Welling, M. (2013). Auto-encoding variational bayes. In: Proceedings  of  the  International Conference on Learning Representations (ICLR). arXiv:1312.6114 [stat.ML].

2. Hinton, G.E., & Salakhutdinov, R.R. (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786), 504-507.

3. Stef van Buuren, Karin Groothuis-Oudshoorn (2011). mice: Multivariate Imputation by Chained Equations in R. Journal of Statistical Software 45: 1-67.
