Metadata-Version: 2.1
Name: topas-pipeline
Version: 0.1.0
Summary: Automated (phospho)proteomics processing pipeline for large patient cohorts providing cohort as well as patient-specific insights.
Home-page: https://github.com/kusterlab/topas-pipeline
License: Apache-2.0
Keywords: mass spectrometry,protein inference,proteomics
Author: Cecilia Jensen
Author-email: cecilia.jensen@tum.de
Requires-Python: >=3.9,<3.11
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Requires-Dist: XlsxWriter (>=3.0.3,<4.0.0)
Requires-Dist: job-pool (>=0.3.1)
Requires-Dist: matplotlib (>=3.5.2,<4.0.0)
Requires-Dist: networkx (>=2.8,<3.0)
Requires-Dist: numpy (>=1.22.3)
Requires-Dist: openpyxl (>=3.1.2,<4.0.0)
Requires-Dist: pandas (>=2.2.0)
Requires-Dist: picked-group-fdr (>=0.7.6)
Requires-Dist: ppca (>=0.0.4,<0.0.5)
Requires-Dist: psite-annotation (>=0.7.1)
Requires-Dist: pyarrow (>=16.0.0)
Requires-Dist: requests (>=2.28.1,<3.0.0)
Requires-Dist: scikit-learn (>=1.3.0,<2.0.0)
Requires-Dist: scipy (>=1.8.0,<2.0.0)
Requires-Dist: seaborn (>=0.13.2)
Requires-Dist: simsi-transfer (>=0.8.0)
Requires-Dist: statsmodels (>=0.14.2,<0.15.0)
Requires-Dist: toml (>=0.10.2)
Requires-Dist: triqler (>=0.8.1)
Requires-Dist: umap-learn (>=0.5.3,<0.6.0)
Project-URL: Repository, https://github.com/kusterlab/topas-pipeline
Description-Content-Type: text/markdown

# TOPAS-pipeline

Automated (phospho)proteomics processing pipeline for large patient cohorts providing cohort as well as patient-specific insights. 

The results of the pipeline can be explored on the web-based TOPAS portal: https://github.com/kusterlab/TOPAS-portal.git
A public instance of the portal can be found here: https://topas-portal.kusterlab.org/

## Supported inputs

- MaxQuant (TMT, LFQ)
- SIMSI-Transfer (TMT)

## Runtime overview

Runtimes exclude processing time of SIMSI-Transfer.

| Dataset | #channels | #samples | #patients | #cores | runtime (h) | max memory (GB) |
| --- | --- | --- | --- | --- | --- | --- |
| CPTAC UCEC | 170 | 153 | 95 | 8 | 1 | 8 |
| CPTAC BRCA | 170 | 153 | 122 | 8 | 1.1 | 10 |
| CPTAC LUAD | 250 | 225 | 110 | 8 | 1.7 | 11 |
| MTB cohort | 2068 | 1504 | 1284 | 8 | 8.3 | 50 |

## Configuration

The pipeline needs configurations input from a JSON file format. 
Examples can be found in `config.json` (full version) and `config_minimal.json` (only required configs).
Both relative and absolute paths are allowed.

Input parameters:

| Parameter | Required | Description | Example | Default |
| --- | --- | --- | --- | --- |
| results_folder | **yes** | Path to the folder where results will be written. | `"results/example_run"` | N/A |
| sample_annotation | **yes** | Path to the sample annotation file (CSV). | `"example/annotation.csv"` | N/A |
| metadata_annotation | **yes** | Path to the metadata annotation file (Excel). | `"example/METADATA_UCEC.xlsx"` | N/A |
| raw_file_folders | **yes** | List of raw file folders for proteomics and phosphoproteomics data. | `["example/raw_fp", "example/raw_pp"]` | N/A |
| data_types | | List of data types to process: "fp" for proteome and "pp" for phosphoproteome. | `["fp", "pp"]` | `["fp", "pp"]` |
| **simsi** |  |  |  |  |
| run_simsi | | Boolean indicating whether to run SIMSI analysis. | `true` | `true` |
| simsi_folder | **yes** | Path to the folder for writing SIMSI-Transfer results. | `"results/SIMSI"` | N/A |
| tmt_ms_level | | MS level for TMT quantification. | `"ms2"` | `"ms2"` |
| stringencies | | Stringency value for MaRaCluster. | `10` | `10` |
| tmt_requantify | | Boolean indicating whether to requantify TMT data. | `false` | `false` |
| maximum_pep | | Maximum posterior error probability in percent for peptide ID propagation. | `1` | `1` |
| num_threads | | Number of threads to use for SIMSI-Transfer. | `8` | `8` |
| correction_factor_mapping_file | | Path to file mapping experiment names to correction factor files (TSV). | `"example/correction_factor_mapping_file.tsv` | `""` |
| **preprocessing** |  |  |  |  |
| raw_data_location | **yes** | Path to the folder containing MaxQuant search result folders. | `"example/CPTAC_searches"` | N/A |
| fasta_file | **yes** | Path to the FASTA file for protein sequences. | `"example/uniprot_human.fasta"` | N/A |
| picked_fdr | | False discovery rate threshold for protein groups. | `0.01` | `0.01` |
| fdr_num_threads | | Number of threads to use in MaxLFQ computation. | `8` | `8` |
| imputation | | Perform data imputation within batch on phosphoproteome level. | `true` | `true` |
| debug | | Run in debug mode. | `false` | `false` |
| run_lfq | | Input is from LFQ experiments. | `false` | `false` |
| normalize_to_reference | | Normalize channel intensities to the reference channel. | `false` | `false` |
| **clinic_proc** |  |  |  |  |
| pspFastaFile | **yes** | Path to the PSP FASTA file. | `"PSP_annotations/Phosphosite_seq.fasta"` | N/A |
| pspKinaseSubstrateFile | **yes** | Path to the PSP kinase-substrate dataset. | `"PSP_annotations/Kinase_Substrate_Dataset"` | N/A |
| pspAnnotationFile | **yes** | Path to the PSP phosphorylation site dataset. | `"PSP_annotations/Phosphorylation_site_dataset"` | N/A |
| pspRegulatoryFile | **yes** | Path to the PSP regulatory sites file. | `"PSP_annotations/Regulatory_sites"` | N/A |
| prot_baskets | **yes** | Path to the annotation file for TOPAS scores and proteins of interest. | `"TOPASscores_POI_AS_250307.xlsx"` | N/A |
| extra_kinase_annot | | Path to the annotation file with custom kinase-substrate relations. | `""` | `""` |
| **report** |  |  |  |  |
| samples_for_report | | Which samples to include in the report. | `"all"` | `"all"` |
| **portal** |  |  |  |  |
| update | | Automatically update the TOPAS portal once the run has finished | `false` | `false` |
| cohort | | Specifies the cohort name that should be updated in the TOPAS portal. | `""` | `""` |
| url | | URL of the TOPAS portal. | `""` | `""` |
| config | | Configuration file for the TOPAS portal. | `""` | `""` |
| **slack** |  |  |  |  |
| webhook_url | | URL for the Slack webhook. | `"https://hooks.slack.com/services/xxx/yyy/zzz"` | `""` |
| channel | | Name of the slack channel to post pipeline updates. | `"#pipeline_channel"` | `""` |

### Install webhook for slack (optional)

If you want the pipeline to post update messages (finished runs, error messages) to 
your slack channel, follow these steps:

1. Create a new slack app here: https://api.slack.com/apps?new_app=1, use the `from scratch` option.
2. Select your slack workspace and pick an appropriate name for the app, e.g. `topas-pipeline`.
3. Navigate to `Incoming webhooks` in the left menu.
4. Set `Activate Incoming webhooks` to `On` if this was not already the case.
5. Click on `Add New Webhook to Workspace` at the bottom of the page.
6. Select the channel you want to post messages in and add it to your config file as the `slack.channel` property (N.B. do not forget to include the `#` symbol before the channel name!).
7. Copy the generated `Webhook URL` to your config file as the `slack.webhook_url` property.

Source: https://api.slack.com/messaging/webhooks


## Running the pipeline

### With Docker (recommended)

Requirements:

- git
- docker
- make


1. Clone this repository
    ```
    git clone https://github.com/kusterlab/TOPAS-pipeline.git
    ```
2. Build the docker image
    ```
    make build
    ```
3. Create a config file named `config_patients.json` in the repository with your configurations (see section `Configuration`)
4. Start the pipeline:
    ```
    make docker_all 
    ```
   You can also use a custom config file (works only with relative paths) and adjust the memory and cores (default: 100GB, 8 cores):
    ```
    CONFIG_FILE=./path/to/config.json MEMORY_LIMIT=300gb CPU_LIMIT=16 make docker_all
    ```

### With conda and poetry

Requirements:

- git
- Python ">=3.9, <=3.11"
- poetry
- make
- conda

1. Create environment and install required packages from poetry.lock file:
    ```
    conda create --name topas-pipeline python=3.9.12
    conda activate topas-pipeline
    ```
2. Clone this repository
    ```
    git clone https://github.com/kusterlab/TOPAS-pipeline.git
    ```
3. Install dependencies and start a poetry shell
    ```
    poetry install
    poetry shell
    ```
4. Adjust the file paths in `config_minimal.json` and run:
    ```
    make all
    ```


Note that it is also possible to run individual pipeline modules, e.g.:
```
# run simsi
python -m topas_pipeline.simsi -c config.json

# run whole pipeline following simsi
python -m topas_pipeline.main -c config.json 

# run clinical annotation
python -m topas_pipeline.clinical_annotation -c config.json
```


## Example

An example of the project folder setup and configuration file can be found in the `/example` folder.
Check the ReadMe in the `/example` folder for details.

<!--
## Integration tests

If problems arise with running the pipeline, check if the integration tests pass.
```
pytest ./tests/integration_tests/test_simsi.py
pytest ./tests/integration_tests/test_picked_group.py
pytest ./tests/integration_tests/test_clinical_tools.py
```
-->

## Pipeline result files

The pipeline creates a folder with multiple output files:

| Output file | Description | Used on portal |
| --- | --- | :---: |
| configs.json | Copy of the configuration file in JSON format used for this pipeline run as described above. |  |
| sample_annot_filtered.tsv | Subset of sample annotation/metadata after filtering out QC failed samples |  |
| meta_input_file_{data_type}.tsv | Location per batch of search folder input, raw files and TMT correction factor file |  |
| {data_type}_qc_numbers.csv | Per sample count of peptides, median intensities and summed intensities |  |
| {data_type}_qc_batch_wise.csv | Per batch median and summed intensities |  |
| {data_type}_in_batch_correction_factors.csv | Per sample correction factors for in-batch median centering |  |
| {data_type}_ms1_correction_factors.csv | Per batch correction factors for MS1 median centering |  |
| evidence.txt | Precursor level input to Picked Protein Group FDR in MaxQuant evidence.txt format |  |
| pickedGeneGroups.txt | Gene-level output from Picked Protein Group FDR |  |
| pickedGeneGroups_with_quant.txt | Gene-level output from Picked Protein Group FDR including quantification with MaxLFQ |  |
| preprocessed_{data_type}.csv | Data matrix with patients as columns and normalized abundances of proteins or phosphopeptides as rows |  |
| annot_{data_type}.csv | Same as preprocessed_{data_type}.csv but with gene and p-site functional annotations |  |
| {data_type}_measures_rank.tsv | Data matrix with patients as columns and in-cohort rank per protein or phosphopeptide as rows |  |
| {data_type}_measures_fc.tsv | Same as {data_type}_measures_rank.tsv but with fold changes |  |
| {data_type}_measures_z.tsv | Same as {data_type}_measures_rank.tsv but with z-scores | X |
| {data_type}_measures_p.tsv | Same as {data_type}_measures_rank.tsv but with p-values derived from the z-scores |  |
| basket_scores_4th.tsv | Data matrix with patients as rows and TOPAS scores as columns | X |
| basket_scores_4th_gen_zscored.tsv | Data matrix with patients as rows and Z-scored (across kinases) TOPAS scores as columns | X |
| subbasket_scores_{topas_rtk}.tsv | Data matrix with patients as rows and TOPAS subscores as columns for each TOPAS RTK |  |
| kinase_results/kinase_scores.tsv | Data matrix with patients as columns and TOPAS substrate phosphorylation scores as rows | X |
| kinase_results/scored_peptides.tsv | Data matrix with patients as columns and weights and weighted z-scores for phosphopeptides as rows |   |
| protein_results/protein_scores.tsv | Data matrix with patients as columns and TOPAS protein phosphorylation scores as rows | X |
| Reports/{patient_id}_proteomics_results.xlsx | Patient-specific reports | (X) |
| Pipeline_log.txt | Log messages printed by the pipeline |   |

