Metadata-Version: 2.4
Name: phetk
Version: 0.3.5
Summary: The Phenotype Toolkit
Author-email: Tam Tran <PheTK@mail.nih.gov>
License: GPL-3.0
Project-URL: Homepage, https://github.com/nhgritctran/PheTK
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: adjusttext
Requires-Dist: connectorx
Requires-Dist: duckdb
Requires-Dist: gcsfs
Requires-Dist: google-cloud-bigquery
Requires-Dist: google-cloud-storage
Requires-Dist: lifelines<0.30.1,>=0.28.0
Requires-Dist: lxml
Requires-Dist: matplotlib
Requires-Dist: numpy<2.0,>=1.21.0
Requires-Dist: pandas>=1.3.0
Requires-Dist: polars
Requires-Dist: psutil
Requires-Dist: pyarrow>=10.0.1
Requires-Dist: pysam
Requires-Dist: statsmodels
Requires-Dist: tqdm
Requires-Dist: firthmodels>=0.7.0
Provides-Extra: hail
Requires-Dist: hail; extra == "hail"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-mock; extra == "dev"
Dynamic: license-file

# PheTK - The Phenotype Toolkit

[![Tests](https://github.com/nhgritctran/PheTK/actions/workflows/tests.yml/badge.svg)](https://github.com/nhgritctran/PheTK/actions/workflows/tests.yml)
[![PyPI version](https://img.shields.io/pypi/v/phetk)](https://pypi.org/project/phetk/)
[![Python versions](https://img.shields.io/pypi/pyversions/phetk)](https://pypi.org/project/phetk/)
[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
[![DOI](https://img.shields.io/badge/DOI-10.1093%2Fbioinformatics%2Fbtae719-blue)](https://doi.org/10.1093/bioinformatics/btae719)

The official repository of PheTK, a fast python library for Phenome Wide Association Studies (PheWAS) utilizing both phecode 1.2 and phecodeX 1.0.

__Reference__: Tam C Tran, David J Schlueter, Chenjie Zeng, Huan Mo, Robert J Carroll, Joshua C Denny, PheWAS analysis on large-scale biobank data with PheTK, Bioinformatics, Volume 41, Issue 1, January 2025, btae719, https://doi.org/10.1093/bioinformatics/btae719

__Contact__: [PheTK@mail.nih.gov](mailto:PheTK@mail.nih.gov)

__Releases__: check [GitHub Releases](https://github.com/nhgritctran/PheTK/releases) for the latest versions and changelogs.

## CURRENT MAJOR FUNCTIONALITIES (v0.3 — June 2026)

- **[Variant selection](docs/clinvar-module.md)** - Retrieve ClinVar variants for a gene symbol or genomic region with [`ClinVar.search()`](docs/clinvar-module.md#clinvarsearch), filtering by clinical significance, review star rating, variant type, and allele frequency. Output columns feed directly into [`by_genotype()`](docs/cohort-module.md#by_genotype). Works cross-platform.
- **[Cohort generation](docs/cohort-module.md)** - Build genotype-based cohorts from VCF or Hail data with [`by_genotype()`](docs/cohort-module.md#by_genotype), add demographic and genetic covariates with [`add_covariates()`](docs/cohort-module.md#add_covariates). Cohort generation and covariate retrieval require the _All of Us_ Researcher Workbench (CDR v7-v9); custom platforms are supported with user-provided data paths and BigQuery datasets.
- **[Phecode mapping](docs/phecode-module.md)** - Map ICD codes to phecodes (phecode 1.2, phecodeX 1.0) with [`count_phecode()`](docs/phecode-module.md#count_phecode), and compute [age-at-first-event](docs/phecode-module.md#add_age_at_first_event) and [time-to-event](docs/phecode-module.md#add_phecode_time_to_event). ICD extraction from OMOP requires _All of Us_ or a compatible BigQuery dataset; phecode mapping and time-to-event computation work cross-platform with local files. The bundled mapping tables are also directly loadable via [`get_phecode_map()`](docs/phecode-module.md#get_phecode_map).
- **[PheWAS analysis](docs/phewas-module.md)** - Run phenome-wide association studies with [logistic](docs/phewas-module.md#running-phewas), [Cox](docs/phewas-module.md#cox-regression-parameters), [Firth logistic, and Firth Cox](docs/phewas-module.md#firth-penalized-regression) regression. Works cross-platform on any pre-built dataset.
- **[Visualization](docs/plot-module.md)** - Generate [Manhattan](docs/plot-module.md#manhattan-plot), [Miami](docs/plot-module.md#miami-plot), and [forest](docs/plot-module.md#forest-plot) plots. Works cross-platform.
- **CLI support** - Full command-line interface for all modules (`phetk clinvar`, `phetk phewas`, `phetk cohort`, `phetk phecode`)
- **Distributed computing** - Built-in dsub integration for large-scale analyses on Google Cloud on the _All of Us_ Researcher Workbench.

[**📋 View full changelog**](https://github.com/nhgritctran/PheTK/releases)

***

## QUICK LINKS
- [Installation](#1-installation)
- [1-minute PheWAS demo](#2-1-minute-phewas-demo)
- [PheTK description](#3-descriptions)
- [Usage examples](#4-usage)
  - [ClinVar module](docs/clinvar-module.md)
  - [Cohort module](docs/cohort-module.md)
  - [Phecode module](docs/phecode-module.md)
  - [PheWAS module](docs/phewas-module.md)
  - [Plot module](docs/plot-module.md)
- [System requirements & computing resources](#5-system-requirements)
- Platform specific tutorial(s):
  - ___All of Us___: [Tutorial notebooks](docs/tutorials/README_FIRST.md) - Interactive Jupyter notebooks demonstrating PheTK usage on the _All of Us_ Researcher Workbench with various analysis examples.
Please note that all examples require _All of Us_ registered user access.
- [Changelogs and releases](https://github.com/nhgritctran/PheTK/releases): 
from v0.1.45, please use [GitHub Releases](https://github.com/nhgritctran/PheTK/releases) for the latest versions and changelogs. 
Legacy changelogs were archived in [CHANGELOG.md](legacy/CHANGELOG.md).
- Resource to learn about PheWAS and phecode: [The PheWAS Catalog](https://phewascatalog.org/).

***

## 1. INSTALLATION

### Using pip

The latest version (v0.3+) of PheTK can be installed using the pip install command in the terminal
(note that the lowercase package name "phetk" starts from version 0.2+):

```
pip install phetk --upgrade
```

Users can also specify a version, e.g., for the last stable version of version 0.1:

```
pip install phetk==0.1.47
```

To check current installed version:
```
pip show phetk | grep Version
```

### Using Docker

Please refer to https://hub.docker.com/r/phetk/phetk/tags for the latest docker images.
```bash
docker pull phetk/phetk:latest
```

## 2. 1-MINUTE PHEWAS DEMO

User can run the quick 1-minute PheWAS demo with the following command in a terminal:

```
phetk demo
```

Or in Jupyter Notebook:

```
from phetk import demo

demo.run()
```

The example files (`example_cohort.tsv`, `example_phecode_counts.tsv`, and `example_phewas_results.tsv`) 
generated in this Demo should be in users' current working directory. 
New-to-PheWAS users could explore these files to get a sense of what data are used or generated in PheWAS with PheTK.

## 3. DESCRIPTIONS
PheTK is a fast python library for Phenome Wide Association Studies (PheWAS) utilizing both phecode 1.2 and phecodeX 1.0.

![PheWAS workflow and PheTK modules](img/readme/PheTK_flowchart.png)
Standard PheWAS workflow. Green italicized texts are PheTK module names. 
Black components are supported while gray ones are not supported by PheTK currently.

_All of Us_: the _All of Us_ Research Program (https://allofus.nih.gov/)

## 4. USAGE

For detailed usage examples and documentation for each module, please refer to the individual module documentation:

- **[ClinVar module](docs/clinvar-module.md)** - Retrieve ClinVar variants by gene or genomic region
- **[Cohort module](docs/cohort-module.md)** - Generate genetic cohorts and add covariates
- **[Phecode module](docs/phecode-module.md)** - Map ICD codes to phecodes and generate counts
- **[PheWAS module](docs/phewas-module.md)** - Run PheWAS analysis with logistic or Cox regression
- **[Plot module](docs/plot-module.md)** - Generate Manhattan, Miami, and forest plots

## 5. SYSTEM REQUIREMENTS

PheTK was developed for efficient processing of large data while being resource-friendly. 
It was tested on different platforms from laptops to different cloud environments.

### General Requirements
PheTK's resource requirements vary by usage context. The information in this section is tailored towards cloud computing platforms where large biobanks are often hosted.
- All PheTK functions run on standard machines. The `by_genotype()` Cohort function supports VCF (default, no Spark needed) and Hail (requires a Spark cluster) data formats.
- Both logistic regression and Cox regression scale with CPU counts for faster processing. See figure S2 below from PheTK publication for more information.
In our experience, 4 CPU machines are the most cost-efficient, especially for large-scale analyses.
- For an end-to-end pipeline, the system requirements should be based on the most demanding steps,
which is usually phecode mapping. Since v0.2.7, the phecode module uses DuckDB by default, which brought the memory requirement
for the full _All of Us_ v8 cohort down substantially: a **4 vCPU / 26GB RAM** VM is sufficient for the entire pipeline.
If users only need to run PheWAS analysis, it can be run at a much lower configuration as shown in figure S2.

![PheTK Performance Benchmarks](img/readme/FigureS2.png)
**Figure S2**: Logistic regression performance benchmarks from PheTK publication showing scalability with different CPU configurations and cohort sizes.

#### PheWAS Module - Logistic Regression
- **Minimal resources required** - Can run efficiently on lightweight configurations
- **Minimum tested configuration**: GCP `X-highcpu-4` (4 vCPUs, 8GB RAM, X=GCP machine type, e.g., c2d) or equivalent
- Uses multithreading for parallel processing with lower memory overhead

#### PheWAS Module - Cox Regression  
- **Slightly higher resources required** - Uses multiprocessing which demands more memory
- **Minimum tested configuration**: GCP `X-standard-4` (4 vCPUs, 16GB RAM, X=GCP machine type, e.g., c2d) or equivalent
- The additional memory accommodates the multiprocessing overhead for survival analysis

#### Phecode Module (ICD Code Mapping)
- **Memory requirements scale with cohort size**, but the default `engine="duckdb"` keeps memory bounded via a streaming, spill-capable pipeline
- **Recommended**: For _All of Us_ database v8 with over 500k participants, `count_phecode()` has been tested and runs successfully on a **4 vCPU / 26GB RAM** machine with the default DuckDB engine
- `engine="polars"` runs fully in memory and is somewhat faster, but needs considerably more RAM — roughly **16 vCPU / 104GB RAM** for the full v8 cohort
- `memory_limit` sets the DuckDB memory ceiling explicitly; it defaults to ~90% of currently available RAM. See the [phecode module docs](docs/phecode-module.md#count_phecode)

