Metadata-Version: 2.4
Name: G2PInsight
Version: 1.0.3
Summary: Genome-wide association analysis toolkit
Author-email: chenrf <12024128035@stu.ynu.edu.cn>
License: MIT
Project-URL: Homepage, https://github.com/chenrf0407/G2P_tool
Project-URL: Repository, https://github.com/chenrf0407/G2P_tool
Project-URL: Issues, https://github.com/chenrf0407/G2P_tool/issues
Keywords: GWAS,genotype,phenotype,machine-learning,SHAP,bioinformatics
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Requires-Python: <3.13,>=3.8
Description-Content-Type: text/markdown
Requires-Dist: numpy<2.0,>=1.20
Requires-Dist: pandas<2.0,>=1.3
Requires-Dist: scikit-learn>=1.0
Requires-Dist: scipy<2.0,>=1.7
Requires-Dist: joblib>=1.0
Requires-Dist: matplotlib>=3.5
Requires-Dist: Pillow<12.3,>=8
Requires-Dist: numba>=0.57
Requires-Dist: lightgbm<4.0.0,>=3.3
Requires-Dist: xgboost<2.1,>=1.6
Requires-Dist: catboost<2.0,>=1.0
Requires-Dist: shap>=0.40
Requires-Dist: plotly>=5.0
Requires-Dist: kaleido>=0.2
Requires-Dist: seaborn>=0.11
Provides-Extra: bed
Requires-Dist: bed-reader>=0.2; extra == "bed"
Provides-Extra: monitor
Requires-Dist: psutil>=5.0; extra == "monitor"
Provides-Extra: all
Requires-Dist: bed-reader>=0.2; extra == "all"
Requires-Dist: psutil>=5.0; extra == "all"

# G2PInsight

**What it is:** G2PInsight is a command-line pipeline for **genotype-to-phenotype (G2P)** analysis. You provide genotype data (VCF or PLINK) and a phenotype file; the tool prepares a training matrix, fits machine-learning models, explains SNP importance with SHAP, and can score new samples — no coding required.

**What it is for:** breeding and genetics studies where you want to predict traits, compare ML methods, find important genomic regions, and generate publication-ready figures.

**How to use it:** run four stages in order (predict is optional):

```text
preprocess  →  train or train-all  →  visualize  →  predict (optional)
```

Supports **classification** and **regression**, optional **GWAS/LD** SNP filtering in preprocess, and bundled **PLINK** / **GEMMA** binaries.

---

## Table of Contents

- [What you get](#what-you-get)
- [Quick Start](#quick-start)
- [Pipeline](#pipeline)
- [Installation](#installation)
- [Commands](#commands)
  - [preprocess](#1-preprocess)
  - [train](#2-train)
  - [train-all](#3-train-all)
  - [predict](#4-predict)
  - [visualize](#5-visualize)
- [Output layout](#output-layout)
- [FAQ](#faq)
- [Developer guide](#developer-guide)
- [License](#license)

---

## What you get

| Stage | What it does | Main outputs |
|-------|----------------|--------------|
| **preprocess** | Match samples, QC SNPs, optional GWAS/LD, build training matrix | `*_train_data.txt` (tab-delimited), `*_metadata.json`, phenotype plot |
| **train** | Fit **one** model: tuning, CV, test metrics, SHAP | `{Model}_model.pkl`, metrics, SHAP files |
| **train-all** | Fit **all** applicable models, pick the best on the test set | Best model folder + comparison JSON |
| **visualize** | Genome-wide importance, performance curves, SHAP plots | PNG / HTML under `visualize/` |
| **predict** | Score new samples with a saved model | `*_predictions.tsv` |

**Models:** LightGBM, RandomForest, XGBoost, SVM, CatBoost, Logistic (classification only). **train-all** skips Logistic for regression.

**Important conventions**

- **train** / **train-all** need preprocess `*_metadata.json` (`-j`) — not a raw matrix path alone.
- GWAS/LD runs in **preprocess only**; training does not re-filter SNPs.
- Wait for preprocess to finish (`Metadata file generated successfully`) before training. In batch jobs, use `set -e`.
- **Large SNP counts:** use preprocess `-f 2`–`4`, **train-all** with `--parallel_models 1`; metadata guardrails **block `--parallel_models > 1` when ≥ 500k SNPs**.

---

## Quick Start

```bash
# 1. Preprocess
G2PInsight preprocess --vcf data/genotype.vcf -p data/phenotype.txt -o results/

# 2. Train (check metadata exists first)
test -f results/preprocess/results_metadata.json && echo "OK"
G2PInsight train -j results/preprocess/results_metadata.json -m LightGBM -o results/

# 3. Visualize
G2PInsight visualize \
  -i results/train/LightGBM/LightGBM_shap_values.txt \
  -I results/train/LightGBM/LightGBM_plotting_data.json \
  -o results/plot/lightgbm
```

**Whole-genome or millions of SNPs** — preprocess with filtering, then train conservatively:

```bash
G2PInsight preprocess --bfile genotype -p pheno.txt -o results/ -f 4
G2PInsight train-all -j results/preprocess/results_metadata.json -o results/ \
  --parallel_models 1 --threads 4 --no_hyperparameter_search --n_folds 3
```

---

## Pipeline

```text
  genotype + phenotype
         │
         ▼
   preprocess  ──►  *_train_data.txt + *_metadata.json
         │
         ▼
   train / train-all  ──►  model.pkl, metrics, SHAP, plotting JSON
         │
         ├──► visualize  ──►  PNG / HTML
         └──► predict    ──►  predictions.tsv
```

Use `-o my_project/` as the project folder. Preprocess writes `my_project/preprocess/my_project_metadata.json` — that is the path for **train `-j`**.

---

## Installation

**Requirements:** Python 3.8–3.12, Linux (recommended) or macOS. PLINK and GEMMA are bundled.

```bash
pip install G2PInsight
pip install "G2PInsight[bed]"   # strongly recommended for large preprocess jobs
# pip install "G2PInsight[all]"  # bed-reader + psutil
```

From source:

```bash
git clone https://github.com/chenrf0407/G2P_tool.git
cd G2P_tool && pip install -e ".[bed]"
```

```bash
G2PInsight --version
G2PInsight -h
```

Without `bed-reader`, preprocess falls back to slower PLINK `--recodeA` (`--genotype-backend raw`).

---

## Commands

### 1. preprocess

#### What it is

Turns VCF / PLINK genotypes plus a two-column phenotype file into a **sample × SNP training matrix** and a **metadata JSON** that downstream stages require.

#### What you get

| File | Purpose |
|------|---------|
| `{prefix}_train_data.txt` | Tab-delimited matrix (`sample` index, SNP columns, `phenotype`); may be `.txt.gz` if large |
| `{prefix}_metadata.json` | Paths, sample list, SNP count, task type — **required for train** |
| `phenotype_distribution_pie.png` | Classification: class proportions |
| `phenotype_distribution_histogram.png` | Regression: trait distribution (after sample matching) |

SNP columns are named `chr_position` (e.g. `1_123456`).

#### How to run

```bash
G2PInsight preprocess (--bfile|--file|--vcf) <genotype> -p <phenotype> -o <output> [options]
```

**Typical run** (GWAS + LD feature selection):

```bash
G2PInsight preprocess --vcf data.vcf -p pheno.txt -o results/ -f 4
```

**Large genome** (fast bed read, lower memory):

```bash
G2PInsight preprocess --bfile genotype -p pheno.txt -o results/ -f 4 --no-cache
```

#### Key options

| Option | Default | What it controls |
|--------|---------|------------------|
| `-f` | `1` | `1` none · `2` GWAS · `3` LD · `4` GWAS→LD |
| `--no-filter-snps` | off | Skip MAF / missingness QC (not for whole-genome) |
| `--genotype-backend` | `auto` | `auto`/`bed` = direct `.bed` read; `raw` = PLINK `--recodeA` |
| `--no-cache` | off | Less memory on huge jobs |
| `--parallel-chr-recode` | off | Parallel recode (**raw** backend only) |

**Inputs:** exactly one of `--bfile`, `--file`, `--vcf`. Phenotype: two columns, no header (sample ID + value).

---

### 2. train

#### What it is

Trains **one** ML model from `*_metadata.json`: load matrix → split samples → tune hyperparameters (optional) → K-fold CV on training split → final fit → test metrics → SHAP.

#### What you get

| File | Purpose |
|------|---------|
| `{Model}_model.pkl` | Saved model for **predict** |
| `{Model}_metrics.json` | **Held-out test set** metrics (main result) |
| `{Model}_cv_results.json`, `{Model}_cv_oof.tsv` | CV on **training split** only |
| `{Model}_shap_values.txt` | Genome-wide importance (tab-delimited, `.txt` extension) |
| `{Model}_shap_dependence.tsv` | Per-sample SHAP for dependence plots |
| `{Model}_plotting_data.json` | Index for **visualize -I** |

#### How to run

```bash
G2PInsight train -j <metadata.json> -m <model> -o <output_dir> [options]
```

```bash
# Default
G2PInsight train -j results/preprocess/results_metadata.json -m LightGBM -o results/

# Large SNP set
G2PInsight train -j results/preprocess/results_metadata.json -m LightGBM -o results/ \
  --no_hyperparameter_search --n_folds 3 --threads 4
```

**Evaluation (default):** 80% train / 20% test. Tuning and K-fold CV use the 80% only; `{Model}_metrics.json` reports the 20% test set.

**Flow:** hyperparameter search (internal 80/20 of train) → K-fold CV on full train split → final fit → SHAP.

#### Key options

| Option | Default | Notes |
|--------|---------|-------|
| `-m` | — | `LightGBM`, `RandomForest`, `XGBoost`, `SVM`, `CatBoost`, `Logistic` |
| `--n_folds` | `5` | CV folds on training split |
| `--no_hyperparameter_search` | off | Faster; recommended for huge matrices |
| `--threads` | `1` | Per-model CPU threads |
| `--shap_dependence_top` | `0` | Cap SNPs in dependence file (`0` = all) |
| `--ignore-warnings` | off | Hide `[WARNING]` logs and Python warnings (errors still shown) |
| `--group-file` | — | Per-group 70/15/15 train/val/test |
| `--train-ids-file` / `--test-ids-file` | — | Custom hold-out split |

Before loading data, **train** checks metadata and logs memory guidance.

---

### 3. train-all

#### What it is

Trains **every model** for your task on the **same** data split. The parent reads the matrix once; each **worker process** runs the full pipeline for one model. After all finish, the **best model** (by held-out test performance) is kept; other model folders are removed.

#### What you get

| File | Purpose |
|------|---------|
| `{BestModel}/` | Winning model: `.pkl`, metrics, SHAP, plotting files |
| `best_model_info.json` | Which model won and the metric value |
| `model_comparison_report.json` | Success/fail, timings, settings |
| `all_models_cv_results.json` | All models’ CV results (saved before cleanup) |

**Best model rule:** classification → AUC (else accuracy); regression → Pearson *r* on test set.

#### How to run

```bash
G2PInsight train-all -j <metadata.json> -o <output_dir> [options]
```

```bash
# Recommended default
G2PInsight train-all -j results/preprocess/results_metadata.json -o results/ \
  --parallel_models 1 --threads 4

# Millions of SNPs
G2PInsight train-all -j results/preprocess/results_metadata.json -o results/ \
  --parallel_models 1 --threads 4 --no_hyperparameter_search --n_folds 3
```

Shares **train** options except `-m`. Extra: `--parallel_models` (default `1`), `--ignore-warnings`. **≥ 500k SNPs + `parallel_models > 1` is rejected** before load.

---

### 4. predict

#### What it is

Applies a trained `.pkl` model to **new samples**. Aligns features to training SNPs (missing → zero-filled).

#### What you get

`<output_dir>/predict/{Model}_predictions.tsv` — columns `sample`, `prediction` (plus `prob_class_*` for classification).

#### How to run

```bash
G2PInsight predict -i <input> -m <model.pkl> -o <output_dir> [--task_type <type>]
```

| Input (`-i`) | Notes |
|--------------|-------|
| Tab-delimited matrix `.txt` / `.txt.gz` | Same format as preprocess output |
| PLINK binary prefix | `.bed/.bim/.fam` — **training SNPs extracted first** |
| VCF `.vcf` / `.vcf.gz` | Converted internally; **training SNPs extracted first** |

```bash
G2PInsight predict -i new_genotype -m results/train/LightGBM/LightGBM_model.pkl -o results/ --task_type regression
```

Use the same genome build and SNP naming as training (`*_training_features.json`).

---

### 5. visualize

#### What it is

Builds **figures** from training outputs — genome-wide SHAP, top-SNPs bar chart, performance curves, CV diagnostics. Training writes data files; **visualize** renders PNG/HTML.

#### What you get

Under `<parent_of_output_prefix>/visualize/`:

| With `-i` (SHAP) | With `-I` (plotting JSON) |
|------------------|---------------------------|
| Genome-wide importance (static + interactive) | Held-out test performance curves |
| Top-SNPs bar chart | Test metrics bar chart |
| SHAP summary & dependence (needs `*_shap_dependence.tsv`) | CV fold metrics, grouped hold-out bars |

#### How to run

```bash
G2PInsight visualize [-i <shap_values.txt>] [-I <plotting_data.json>] -o <prefix> [--top_snps N]
```

At least one of `-i` or `-I` is required.

```bash
G2PInsight visualize -I results/train/LightGBM/LightGBM_plotting_data.json -o results/plot/perf
G2PInsight visualize -i results/train/LightGBM/LightGBM_shap_values.txt -o results/plot/shap --top_snps 20
```

**`*_shap_values.txt` columns:** `feature`, `importance_abs`, `effect` (`1` / `-1`).

For huge SNP sets, set `--shap_dependence_top N` during training, then `--top_snps` ≤ *N* here.

---

## Output layout

```text
results/
├── preprocess/
│   ├── results_train_data.txt[.gz]
│   ├── results_metadata.json       ← train -j
│   └── phenotype_distribution_*.png
├── train/
│   ├── LightGBM/                   ← or only BestModel/ after train-all
│   │   ├── LightGBM_model.pkl
│   │   ├── LightGBM_metrics.json
│   │   ├── LightGBM_shap_values.txt
│   │   ├── LightGBM_plotting_data.json
│   │   └── ...
│   ├── best_model_info.json        ← train-all
│   ├── model_comparison_report.json
│   └── all_models_cv_results.json
├── predict/
│   └── LightGBM_predictions.tsv
└── plot/visualize/
    └── *.png / *.html
```

Temp files under `{output}/tmp/` are removed after successful runs.

---

## FAQ

### train requires metadata — why?

**train** and **train-all** need `*_metadata.json` from preprocess. It records the matrix path, samples, SNP count, and task type.

### Where is the metadata file?

`<output>/preprocess/{prefix}_metadata.json`. Example: `-o results/` → `results/preprocess/results_metadata.json`.

### train-all keeps only one model folder

By design: best model by **test set** metrics. See `best_model_info.json` and `all_models_cv_results.json` for the full comparison.

### train-all fails or workers crash

**Symptom:** `Worker process crashed`, all models `failed`, or error about `--parallel_models` and SNP count.

**What to do:**

1. **≥ 500k SNPs:** use `--parallel_models 1` only (higher values are blocked).
2. Re-run preprocess with **`-f 2` or `-f 4`** if too many SNPs were kept.
3. Use `--parallel_models 1 --no_hyperparameter_search --n_folds 3 --threads 4`.
4. Increase job memory; or test one model: `G2PInsight train -m LightGBM ...`.

### Training or load is very slow

1. Fewer SNPs in preprocess (`-f 2`–`4`).
2. Uncompressed `*_train_data.txt` loads faster than `.gz`.
3. `--no_hyperparameter_search`, lower `--n_folds`.

### SHAP dependence file is huge

Pass `--shap_dependence_top N` in train / train-all (e.g. `50`). Genome-wide `*_shap_values.txt` is unchanged.

### Preprocess slow or killed

1. Do not use `--no-filter-snps` on whole-genome data; use `-f 2`–`4`.
2. Install with `pip install "G2PInsight[bed]"` (or `pip install bed-reader`) for fast `auto` backend.
3. Add `--no-cache`; for `raw` backend try `--parallel-chr-recode`.

### predict: constant values or no feature match

1. Same genome build and `chr_pos` naming as training.
2. Check sample IDs match.
3. For VCF/PLINK, training SNPs are extracted automatically — ensure those SNPs exist in the new data.

### visualize: missing `effect` column

Use training-generated `*_shap_values.txt` with `feature`, `importance_abs`, `effect`.

### Which preprocess backend was used?

Log lines: `Genotype conversion backend: bed` (fast) or `raw` (PLINK `--recodeA`).

---

## Developer guide

```text
assocG2P/G2PInsight/
├── main.py
└── bin/
    ├── preprocess.py
    ├── modeltraining.py
    ├── visualization.py
    ├── gemma_gwas.py
    └── plink_ld.py
```

```bash
pip install -e ".[bed]"
G2PInsight preprocess -h
```

---

## License

[MIT License](LICENSE)
