Metadata-Version: 2.4
Name: hifi_solves_run_humanwgs
Version: 3.1.0
Summary: Upload required files and run [PacBio's Human WGS workflow](https://github.com/PacificBiosciences/HiFi-human-WGS-WDL) via [DNAstack's Workbench](https://omics.ai/workbench/)
Author: DNAstack
Author-email: bioinformatics@dnastack.com
Maintainer: Heather Ward
Maintainer-email: heather@dnastack.com
License: GPLv2
Requires-Python: >=3.13
Description-Content-Type: text/markdown
Requires-Dist: pandas>=2.2.2
Requires-Dist: boto3>=1.34.0
Requires-Dist: dnastack-client-library>=3.1.238
Requires-Dist: azure-storage-blob>=12.20.0
Requires-Dist: azure-core>=1.38.0
Requires-Dist: google-cloud-storage>=2.17.0
Requires-Dist: regex>=2024.11.6
Dynamic: author
Dynamic: author-email
Dynamic: description
Dynamic: description-content-type
Dynamic: license
Dynamic: maintainer
Dynamic: maintainer-email
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# HiFi Solves Human WGS data uploader

As part of [the HiFi Solves Consortium](https://hifisolves.org/collections), organizations will run their sequencing data through [PacBio's Human Whole Genome Sequencing (WGS) pipeline](https://github.com/PacificBiosciences/HiFi-human-WGS-WDL).

This package handles uploading all required raw data to the organization's cloud and staging the sample metadata that describes it. Output files are automatically ingested into Publisher and made available on [hifisolves.org](https://hifisolves.org/collections).

Submitting workflow runs is **not** part of this package. Once your data is uploaded, submit runs from Instruments. The workflow itself, along with its version, defaults and transformations, is registered in your namespace by DNAstack rather than by this tool, so there is nothing to configure here.

## Requirements

- python3.13+
- Credentials for the relevant backend (supported backends: AWS, Azure, GCP)

## Installation

`python3 -m pip install hifi-solves-run-humanwgs`

### Command name

The command is `hifisolves-upload`.

Two earlier names, `hifisolves-ingest` and `run-humanwgs`, still work and do exactly the same thing, so existing scripts do not need changing. Both predate the move of workflow submission to Instruments, and `run-humanwgs` in particular now names something this tool no longer does. Invoking either prints a notice pointing at `hifisolves-upload`; prefer the new name in anything written from here on.

# Script usage

## Arguments

```bash
usage: hifisolves-upload [-h] [-v] -s SAMPLE_INFO -b {AWS,GCP,AZURE}
                         -o ORGANIZATION [--verbose]

Upload genomics data and sample metadata in preparation for running PacBio's
official Human WGS pipeline. Workflow runs are submitted from Instruments, not
from this tool.

options:
  -h, --help            show this help message and exit
  -v, --version         Program version
  -s, --sample-info SAMPLE_INFO
                        Path to sample info CSV or TSV. This file should have
                        columns [sample_id, hifi_reads]. Any other columns are
                        ignored. See documentation for more information on the
                        format of this file.
  -b, --backend {AWS,GCP,AZURE}
                        Backend where infrastructure is set up
  -o, --organization ORGANIZATION
                        Organization identifier; used to infer bucket names
  --verbose             Use verbose logging
```

## Sample info file

The sample info file defines the set of samples that will be uploaded.

This information is organized into a CSV or TSV file with the following columns:

| Column       | Description                                                                                                  |
| :----------- | :------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `sample_id`  | Unique identifier for this sample.                                                                           |
| `hifi_reads` | Path to a BAM of HiFi reads for this sample. Both unaligned movie BAMs and aligned HiFi BAMs are accepted. †  |

† There can be more than one BAM for a sample. If so, add a row per BAM, repeating the `sample_id`. `movie_bams` is still accepted as a column name in place of `hifi_reads`, but not both in the same file.

### Example sample info files

#### Example - single sample

Here we have a single sample, HG005, with two associated BAMs found at the local paths `bams/HG005_1.hifi_reads.bam` and `bams/HG005_2.hifi_reads.bam`.

```csv
sample_id,hifi_reads
HG005,bams/HG005_1.hifi_reads.bam
HG005,bams/HG005_2.hifi_reads.bam
```

#### Example - several samples

Here we have three samples. HG006 and HG007 each have several BAMs, so there are multiple rows for each.

```csv
sample_id,hifi_reads
HG005,bams/HG005_1.hifi_reads.bam
HG006,bams/HG006_1.hifi_reads.bam
HG006,bams/HG006_2.hifi_reads.bam
HG007,bams/HG007_1.hifi_reads.bam
HG007,bams/HG007_2.hifi_reads.bam
HG007,bams/HG007_3.hifi_reads.bam
```

# Running the script

The script uploads input files and stages sample metadata. It does not submit workflow runs; submit those from Instruments once the upload completes and your samples have been indexed.

The script is idempotent, so it is safe to re-run. Any BAM already present at the destination is skipped rather than uploaded again, so a re-run after an interruption transfers only what is still missing.

`--upload-only` and `--region` are still accepted but ignored, so existing invocations do not break. They are hidden from `--help` and should be dropped from new callers: uploading is now the only mode, and the region was only ever needed to build the workflow inputs that this tool no longer generates. Uploading itself does not use a region.

## Environment Variables

### WORKBENCH_URL

By default, the script will use `workbench.omics.ai` as the Workbench URL. You can override this by setting the `WORKBENCH_URL` environment variable:

```bash
export WORKBENCH_URL="custom-workbench.example.com"
```

## Example upload command - AWS

```bash
# AWS credentials
export AWS_ACCESS_KEY_ID=""
export AWS_SECRET_ACCESS_KEY=""
export AWS_SESSION_TOKEN=""

# Used for naming upload and output buckets
ORGANIZATION=""

hifisolves-upload \
    --sample-info sample_info.csv \
    --backend aws \
    --organization "${ORGANIZATION}"
```

## Example upload command - Azure

```bash
# Azure credentials; needs Read, Add, Write, Create, Delete, List
export AZURE_STORAGE_SAS_TOKEN=""

# Used for naming upload and output buckets; this is going to be == the storage account name
ORGANIZATION=""

hifisolves-upload \
    --sample-info sample_info.csv \
    --backend Azure \
    --organization "${ORGANIZATION}"
```

If you have files already uploaded in the target storage account, their paths may be referenced in the format `/<storage_account>/rawdata/path/to/file`.

### Copying Azure <> Azure

If source files are currently in cloud storage, they can be copied into the target storage account rather than copying from local -> cloud.

BAM URLs in the `sample_info` CSV file should be in the format `/<src_storage_account>/<src_storage_container>/path/to/movie.bam`.

An additional env variable, `SOURCE_CONTAINER_SAS_TOKEN`, should be defined. This SAS token should have Read and List permissions on the source container.

```bash
# SAS token for the source bucket (R/L)
export SOURCE_CONTAINER_SAS_TOKEN=""

# SAS token for the destination bucket (R/A/W/C/D/L)
export AZURE_STORAGE_SAS_TOKEN=""

ORGANIZATION=""

hifisolves-upload \
    --sample-info sample_info.csv \
    --backend Azure \
    --organization "${ORGANIZATION}"
```

## Example upload command - GCP

```bash
# GCP credentials - GOOGLE_APPLICATION_CREDENTIALS should point towards a JSON file containing service account information
export GOOGLE_APPLICATION_CREDENTIALS=""

# Used for naming upload and output buckets
ORGANIZATION=""

hifisolves-upload \
	--sample-info sample_info.csv \
	--backend gcp \
	--organization "${ORGANIZATION}"
```

# Development

## Tests

_Note that you will need access to have the active cloud-specific credentials below set for the various cloud backends to run the tests._

See [this secret](https://start.1password.com/open/i?a=7NPE3CP55BHD5FCTKGQYWMT2XU&v=tyykwlqgisiv33exjxnnln3434&i=lltbwiedpo7igirwlhmo27jz3e&h=team-dnastack.1password.com) for the values you'll need to set here.

```bash
# Required AWS credentials
export AWS_ACCESS_KEY_ID=""
export AWS_SECRET_ACCESS_KEY=""
export AWS_SESSION_TOKEN=""
# alternatively - just AWS_PROFILE
# export AWS_PROFILE=""

# Required Azure credentials
_Note that these credentials will eventually expire_
# R/A/W/C/D/L on destination container
export AZURE_STORAGE_SAS_TOKEN=""
# R/L on src container
export SOURCE_CONTAINER_SAS_TOKEN=""

# Path to service account JSON
export GOOGLE_APPLICATION_CREDENTIALS=""
```

`python3 -m unittest discover -b -s tests`

## Building Packages

A `Makefile` resides in the root directory of the package. Rules have been created to clean, build, release and push a package release. The following sections provide details on building and pushing the code to a repository

### Building the package

```
make build
```

This creates a Python package to be uploaded to the Python Package Index (PyPI: pypi.org).

### Building and pushing the Docker image

```
make docker-build
```

This rule builds a Docker image and pushes it to a container registry.
