Metadata-Version: 2.5
Name: altar-popfreq
Version: 0.3.0rc1
Summary: gnomAD population allele-frequency annotation-source binding for Altar
Project-URL: Documentation, https://kundajelab.github.io/altar/
Author: Riya Sinha
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: altar[gcs]<0.4,>=0.3.0rc1
Provides-Extra: build
Requires-Dist: duckdb>=1.0.0; extra == 'build'
Requires-Dist: pyarrow>=17.0.0; extra == 'build'
Provides-Extra: test
Requires-Dist: pytest-asyncio>=0.24; extra == 'test'
Requires-Dist: pytest>=8; extra == 'test'
Description-Content-Type: text/markdown

# Population allele-frequency binding (gnomAD)

`altar-popfreq` exposes gnomAD v4 population allele frequencies through Altar's
`AnnotationSource` contract. gnomAD is a lookup source here: the frequencies are
published before an Altar analysis begins, and the binding runs no model.

This tool includes data from the gnomAD v4.1.1 release, read by default from a
derived copy the Kundaje lab publishes, or from a release its operator builds.
The package ships no gnomAD data; see [Where the data comes from](#where-the-data-comes-from)
and [Data terms](#data-terms). It is not affiliated with or endorsed by the gnomAD
project or the Broad Institute.

The package is not called `gnomad` because gnomAD's
[policies](https://gnomad.broadinstitute.org/policies) say: "to avoid confusion
and misattribution, we ask that you refrain from incorporating "gnomAD" or
"Genome Aggregation Database" into the name of your tool". Names that describe the
data keep gnomAD's name: the contract ID `org.kundajelab.altar.annotation.gnomad`,
the `gnomad_*` columns, `GnomADRelease`, `GnomADSource`, `GNOMAD_ANNOTATIONS`,
and the source's `name`, `gnomad`, which is also its `altar.annotation_sources`
entry-point key because Altar's registry requires the two to match. The package,
the `altar_popfreq` module, and the `altar-popfreq-build` command name the tool.

## Columns

Every column is prefixed `gnomad_`, so gnomAD values never collide with another
source's columns, such as the Open Targets `af_*` frequencies in Altar's region
annotations.

| Column | Type | Meaning |
| --- | --- | --- |
| `gnomad_af` | float | Alternate allele frequency, AC / AN. Null when AN is 0. |
| `gnomad_ac` | int | Alternate allele count. |
| `gnomad_an` | int | Total number of called alleles. |
| `gnomad_nhomalt` | int | Number of homozygous-alternate individuals. |
| `gnomad_af_grpmax` | float | gnomAD's grpmax AF: the greatest AF over the groups gnomAD includes in grpmax. |
| `gnomad_grpmax` | str | The group with that AF, such as `nfe`. |
| `gnomad_faf95_max` | float | gnomAD's maximum filtering allele frequency: the greatest per-group FAF (Poisson 95% CI) over the groups in the table below. Null when no such group has more than one alternate allele. See [Filtering on frequency](#filtering-on-frequency). |
| `gnomad_af_max_ancestry` | float | The greatest non-null AF over the ten ancestry columns below. Null if all are null. |
| `gnomad_af_afr`, `_ami`, `_amr`, `_asj`, `_eas`, `_fin`, `_mid`, `_nfe`, `_sas`, `_remaining` | float | AF within each genetic ancestry group. |
| `gnomad_filters` | str | The site's VCF FILTER value: `PASS`, or the `;`-joined filters it failed, such as `AC0;AS_VQSR`. |

`gnomad_af_grpmax` and `gnomad_faf95_max` differ from `gnomad_af_max_ancestry` on
purpose. gnomAD leaves its bottlenecked groups out of grpmax and the filtering
allele frequency, and which ones depends on the dataset:

| Dataset | Groups grpmax and the FAF cover | Left out |
| --- | --- | --- |
| genomes | `afr`, `amr`, `eas`, `nfe`, `sas` | `ami`, `asj`, `fin`, `mid`, `remaining` |
| exomes | `afr`, `amr`, `eas`, `mid`, `nfe`, `sas` | `asj`, `fin`, `remaining` (exomes have no `ami`) |
| joint | `afr`, `amr`, `eas`, `mid`, `nfe`, `sas` | `ami`, `asj`, `fin`, `remaining` |

This follows gnomAD's v4 documentation and matches the groups seen in v4.1 chr21
data; check gnomAD's release notes for another release. `gnomad_af_max_ancestry`
covers all ten groups in every dataset, so a filter like "common in any ancestry"
can test one scalar column without array logic.

A column the release does not publish is null. For example, the v4 exomes VCFs
have no `AF_ami`, so `gnomad_af_ami` is null in an exomes table.

### Absent variants

A variant that gnomAD has no row for is left out of `annotate()`'s result, and
Altar's merge gives it no gnomAD columns. This is deliberate: an unknown frequency
is not evidence that a variant is rare. Filters must decide what an absent
variant means; see [Filtering on frequency](#filtering-on-frequency).

## Release

`GnomADRelease` names the data a backend serves:

```python
from altar_popfreq import GnomADRelease

release = GnomADRelease(version="4.1.1", dataset="genomes")  # genome_build defaults to "hg38"
release.release  # "4.1.1-genomes"
```

`version` is gnomAD's exact release string, as it appears in the file names
(`gnomad.genomes.v4.1.1.sites.chr1.vcf.bgz` is `4.1.1`). Use `4.1.1` for the
v4.1.1 files, not `4.1`: a patch release changes values. `4.<minor>` and
`4.<minor>.<patch>` are accepted. `dataset` is `genomes`, `exomes`, or `joint`.
`release` is the canonical label of the data, `<version>-<dataset>`, since
frequencies from different datasets or versions are different quantities.

The columns form the published `GNOMAD_ANNOTATIONS` contract
(`org.kundajelab.altar.annotation.gnomad`, version 1.0.0). `source.identity()`
and `release.identity()` return that contract with the release label and the
genome build, so bumping the version or switching the dataset changes the
identity. Rows carry no release column; a host table that holds the same columns
can declare the same contract.

Only gnomAD v4 on hg38 is accepted. gnomAD v2.1.1 is on GRCh37 but uses other
INFO names and ancestry groups (`popmax` instead of `grpmax`, `oth` instead of
`remaining`, and no `ami` or `mid`), so its values would not mean the same thing
under these column names. Supporting it would need its own column contract.

## Where the data comes from

The source reads one gnomAD release in one of three ways. All three serve the
same data under the same identity, `source.identity()`: the contract, the
release label, and the genome build. Where the data is read from, and with which
credentials, is never part of it, so switching between them keeps every stored
score and annotation a cache hit.

| Mode | Spec options | Reads |
| --- | --- | --- |
| Published (default) | none, or `location` | The latest release the Kundaje lab publishes, read in place, anonymously. |
| Owned Parquet | `dataset` | Your copy of the builder's Parquet: a path, or a URI in your bucket. |
| Owned BigQuery | `table` | Your BigQuery table of the builder's columns, such as one loaded from the published Parquet. |

### Published

The lab builds the latest gnomAD release with `altar-popfreq-build` and
publishes it in a public-read bucket, mirrored byte for byte in a public
Hugging Face dataset
([altar-genomics/gnomad-4.1.1-genomes](https://huggingface.co/datasets/altar-genomics/gnomad-4.1.1-genomes)):

| Release | Genome build | Location name | URI |
| --- | --- | --- | --- |
| `4.1.1-genomes` | hg38 | `gcs` (default) | `gs://kundajelab-altar/popfreq/gnomad-4.1.1-genomes` |
| `4.1.1-genomes` | hg38 | `hf` | `hf://datasets/altar-genomics/gnomad-4.1.1-genomes` |

Only the latest release is published. A source with no `dataset` and no `table`
reads it in place, with no credentials, and checks that its first file records
that release; `altar pull` checks every file of a copy it makes. `altar-popfreq`
depends on `altar[gcs]`, so `pip install altar-popfreq` is enough to read it:

```toml
[sources.gnomad]
type = "gnomad"
```

A lookup reads only the chromosome partitions and row groups its variants fall
in, not the whole release. To bill the reads to your project, or to read through
other credentials, name a [connection](https://kundajelab.github.io/altar/guides/connections/)
whose storage options replace the anonymous ones.

`location` reads the Hugging Face mirror instead of the bucket. It holds the
same files, so the source has the same identity and every stored score stays a
cache hit. Reading `hf://` needs the `altar[hf]` extra
(`pip install "altar[hf]"`), which `altar-popfreq` does not install; without it,
the source fails with an error that names the extra. `location` applies only to
the published copy, not with `dataset` or `table`:

```toml
[sources.gnomad]
type = "gnomad"
options = { location = "hf" }
```

In code, `GnomADSource.from_options()` builds the same source
(`from_options(location="hf")` for the mirror), and
`GnomADSource.published_dataset()` (also `altar_popfreq.PUBLISHED_DATASET`)
returns the published locations and identity.

### Owned Parquet

`dataset` names a copy of the builder's output: the published dataset copied to
your bucket or disk, or one you build from gnomAD's VCFs (see
[Build the dataset](#build-the-dataset)). One command copies the published
dataset, file by file, and checks that every copied file records its release:

```console
altar pull gnomad --to gs://my-bucket/gnomad-4.1.1-genomes
altar pull gnomad --from hf --to ./gnomad-4.1.1-genomes
```

`--from hf` copies from the Hugging Face mirror (with `altar[hf]` installed)
instead of the bucket; the copy is the same release either way. `--to` also takes
a local directory. It writes with your Application Default
Credentials; `--project` or `--impersonate-service-account` choose them (see
[Copy a published dataset](https://kundajelab.github.io/altar/guides/published-datasets/)).
The release comes from the files' metadata. A connection gives the storage
options for the URI:

```toml
[connections.lab]
type = "gcp"
options = { project = "my-project" }

[sources.gnomad]
type = "gnomad"
options = { dataset = "gs://my-bucket/gnomad-4.1.1-genomes" }
connection = "lab"
```

A local path needs no connection. Without one, `storage_options` passes non-secret
fsspec options directly, such as `{ token = "anon" }`; a source refuses
`storage_options` together with `connection`.

### Owned BigQuery

`table` names a BigQuery table with the builder's columns, for example the
published Parquet loaded into your project. One command loads it, with one
BigQuery load job into a table clustered on `variant_id`:

```console
altar pull gnomad --bigquery my-project.reference.gnomad_4_1_1_genomes --project my-project
```

The dataset `reference` must exist in a location that can load from the
published bucket. Otherwise, pull the files to a bucket in the dataset's region
with `--to`, and load that copy with `--source-uri gs://my-bucket/gnomad-4.1.1-genomes`.
The source queries the table with the connection's BigQuery client, so it needs
a `gcp` connection with credentials:

```toml
[connections.lab]
type = "gcp"
options = { project = "my-project" }

[sources.gnomad]
type = "gnomad"
options = { table = "my-project.reference.gnomad_4_1_1_genomes" }
connection = "lab"
```

A table carries no Parquet metadata. The source declares the published latest
release, `4.1.1-genomes`, unless `release` names another, such as
`release = "4.1-joint"`. `altar pull --bigquery` records the release as table
labels, and the source reads them with one metadata request (no query, no bytes
billed) when it is built:

| Table labels | The source |
| --- | --- |
| name the release and build it declares | reads the table |
| name another release or build | refuses it, naming both; set `release` to the release the table holds |
| none (a table loaded by hand) | reads it, with a warning naming the release it assumes |

The labels are `altar_source`, `altar_release` and `altar_genome_build`. A label
value holds only lowercase letters, digits, `_` and `-`, so each is the field
lowercased with every other character replaced by `_`: release `4.1.1-genomes`
is `4_1_1-genomes`, and the source is `org_kundajelab_altar_annotation_gnomad`
(`altar.sources.bigquery_release_labels`). To label a table you loaded yourself:

```console
bq update --set_label altar_source:org_kundajelab_altar_annotation_gnomad \
  --set_label altar_release:4_1-joint --set_label altar_genome_build:hg38 \
  my-project:reference.gnomad_4_1_joint
```

The labels say which release was loaded, not that the rows were never edited
since; the identity is only as accurate as the release you name.

The source reads the table in either of two ways, chosen by the store. A
`BigQueryScoreStore` joins it into its own materialize and export queries in
SQL, through `bigquery_relation(table)`, so a run spec with a
[BigQuery store](https://kundajelab.github.io/altar/guides/connections/#keep-scores-in-bigquery) and
this source reads gnomAD in the same query as the scores. Any other store looks
variants up through the Python source in batches of queries. The two paths check
rows differently; see [What each path checks](#what-each-path-checks).

## Build the dataset

Build a release yourself to own a copy of a release the lab does not publish,
such as the exomes or the joint dataset, or to build from the upstream files.
`altar-popfreq-build` streams gnomAD sites VCFs into Parquet, one row per allele,
keyed by the canonical `chr:pos:ref:alt`:

```console
altar-popfreq-build --dataset genomes --version 4.1.1 --workers 8 \
  --input gnomad.genomes.v4.1.1.sites.chr*.vcf.bgz \
  --output ./gnomad-4.1.1-genomes
```

### Publish from a bucket to a bucket

`--input` and `--output` also take URIs (`gs://`, `s3://`, `https://`, `hf://`,
...), so publishing a release is one command on any machine, with no download
and no copy step:

```console
altar-popfreq-build \
  --input gs://gcp-public-data--gnomad/release/4.1.1/vcf/genomes/gnomad.genomes.v4.1.1.sites.chr21.vcf.bgz \
  --dataset genomes --version 4.1.1 \
  --output gs://my-bucket/gnomad-4.1.1-genomes/
```

List every chromosome's URL as an input, with `--workers`, to build them all.
`altar-popfreq` installs `altar[gcs]`; another scheme needs its provider's Altar
extra, such as `altar[http]` for `https://`, and a missing one is named in the
error.
Each input is streamed through fsspec in blocks; its name, the last segment of
its path, names its output file and is checked against the release as a file
name is. When an input's stream fails or ends early (a bgzipped stream without
bgzip's end-of-file block, or one that stops inside a gzip member), the builder
discards what it wrote for that input and reads it again from the start, up to
`--input-attempts` times (3 by default), then fails naming the input.

With an `--output` URI, each input is built in a local scratch directory (under
`TMPDIR`, which needs room for one input's files per worker), its keys are
checked there, and each finished file is uploaded under a temporary name and
committed, so a reader never sees a partial file; the local copy is then
deleted. Each worker uploads its own inputs' files as they finish. Until the
build finishes, the output holds `_altar_build/`, which records the inputs
already committed: run the same command again after a crash, and it skips those
inputs (logging each) and builds the rest. Without `--overwrite`, a URI output
must be empty or absent, or hold such an unfinished build of the same release;
a finished build, an unfinished build of another release, or anything a build
does not write is refused. With `--overwrite`, a previous build there, finished
or not, is deleted first, and anything else is still refused. When the build
finishes, `_altar_build/` is removed. Publish to a new prefix rather than
overwriting one that readers use.

Credentials are the provider's ambient ones (application-default credentials,
`AWS_*` variables, ...). `--input-storage-option` and `--output-storage-option`
pass non-secret fsspec options, such as `project=my-project` or `token=anon`,
one `KEY=VALUE` each and repeatable; a value that reads as JSON takes its type,
so `anon=true` is a boolean. A key that would carry a credential is refused. In
Python, `build_dataset` takes `input_connection` and `output_connection` instead
(see [connections](https://kundajelab.github.io/altar/guides/connections/)).

### What the builder writes and checks

It writes `<output>/chr=<chromosome>/<input name>.parquet`. Each file has
`variant_id`, the key columns `chr`, `pos`, `ref`, `alt`, and the columns above,
and records `altar_popfreq.version`, `altar_popfreq.dataset`,
`altar_popfreq.genome_build`, and `altar_popfreq.builder_version` in its Parquet
metadata. `--workers` builds that many inputs at once in separate processes; each
input is read in one stream, so building per-chromosome files in parallel is the
way to use several cores. The builder needs PyArrow and DuckDB, which `altar`
already requires; `pip install 'altar-popfreq[build]'` states them explicitly.

Rows are sorted by position and written with zstd in row groups of 32,000 rows,
so a point lookup reads one small row group (about 1 MB) per variant it finds,
plus each file's footer once per backend.

The builder:

- requires an empty or absent `--output`. `--overwrite` first deletes a previous
  build there, and refuses if the directory holds anything a build does not write.
  A URI output may also hold an unfinished build of the same release, which is
  resumed (see above);
- checks inputs named like gnomAD's files, `gnomad.<dataset>.v<version>.sites.*`,
  against `--dataset` and `--version`. The VCF header records neither, so a
  renamed file is checked only by its INFO names;
- reads bgzipped, gzipped, or plain VCF, telling them apart by their first bytes,
  and rejects a bgzipped input that does not end with bgzip's end-of-file block,
  so a truncated bgzip download or stream is not built short (plain or plain-gzip
  input has no such marker). A URI input cut short is read again instead;
- reads the INFO names of the chosen dataset. The joint dataset uses `_joint`
  names (`AF_joint`, `AF_grpmax_joint`, `AF_joint_afr`, ...); genomes and exomes
  use bare names (`AF`, `AF_grpmax`, `AF_afr`, ...). An input whose header lacks
  the dataset's AC, AN, AF, or nhomalt is rejected;
- rejects a header that declares a `chr1` length other than GRCh38's;
- keeps gnomAD's already-split multi-allelic records as separate rows, and splits
  any record that is not split, taking each per-allele value by its index;
- trims each allele to its minimal representation (gnomAD's already are) and
  writes chromosomes with the canonical `chr` spelling, so `1` becomes `chr1`;
- skips symbolic, `*`, and non-ACGT alleles, which have no canonical key, and
  reports how many;
- reads numbers from the VCF text, so values are not rounded through 32-bit floats;
- rejects input that is not sorted by position, and any variant key that occurs
  twice across all inputs, including one that trimming produces (`100 AC>AT` and
  `101 C>T` are both `chr1:101:C:T`). A repeated key would make every join fan out;
- deletes everything it wrote to a local `--output` if any input fails. A URI
  output keeps the files of the inputs already committed, so the build resumes.

### Read standard input

To read from object storage, name the URI as an `--input` (see above): the
builder streams it, retries a stream cut short, and checks its name. `--input -`
reads one VCF from standard input instead, for a VCF that another program
produces, while Altar opens nothing but the pipe:

```console
bcftools view -Oz gnomad.genomes.v4.1.1.sites.chr21.vcf.bgz \
  | altar-popfreq-build --dataset genomes --version 4.1.1 --input - \
      --stdin-name gnomad.genomes.v4.1.1.sites.chr21.vcf.bgz \
      --output ./gnomad-4.1.1-genomes-chr21
```

Standard input has no file name, so the builder cannot check it against
`--dataset` and `--version` the way it checks a file named like gnomAD's.
`--stdin-name` names the output file (`stdin.parquet` without it) and, when it
follows gnomAD's naming, is checked like a file name; it is only as reliable as
the command line that passes it. Either way, the header checks are required:
the dataset's INFO names must be present, and the header must declare `chr1`
with GRCh38's length, which a file may omit. These checks tell the joint
dataset from the others and GRCh38 from GRCh37, but not genomes from exomes or
one v4 release from another; those rest on `--dataset` and `--version`. The
other safeguards apply unchanged, including the end-of-file check, but a pipe
cannot be read again, so a stream cut short fails the build. That check
catches a stream cut short only for bgzip input: plain text cut at a line
boundary, or gzip that is not bgzip cut between members, ends cleanly and builds
short without an error. Stream gnomAD's own `.vcf.bgz` files as they are, and
bgzip anything else before piping it. Standard input can be read once per build, and
it is read in the main process alongside `--workers` file and URI inputs.

Files built before the binding was renamed from `altar-gnomad` record the same
keys under `altar_gnomad.`; the source reads those too.
`GnomADSource` over a `ParquetAllelicRecordBackend` reads that metadata and refuses
a dataset built for another version, dataset, or build, or one that records no
release at all. It reads the first file in sorted order only, so opening a
remote dataset costs one footer; `altar pull` checks every file of a copy. A
host's own export must carry the same keys;
`altar_popfreq.RELEASE_METADATA.write(release_fields(release))` returns them.

### Normalize the cohort first

The keys are gnomAD's representation: split, trimmed, and left-aligned. Altar's
VCF ingestion keeps each record's REF when it splits ALTs and does not trim or
left-align, so a multi-allelic cohort record misses gnomAD even for an SNV. The
record `chr1 100 CA C,TA` becomes `chr1:100:CA:C` and `chr1:100:CA:TA`; gnomAD
holds the second allele as the SNV `chr1:100:C:T`. That variant finds no gnomAD
row, so it has no gnomAD columns, and a filter in which null passes keeps it as
rare however common it is.

Normalize any cohort with multi-allelic records, or with indels, before
ingestion:

```console
bcftools norm -m -any -f <reference.fa> cohort.vcf.gz -Oz -o cohort.norm.vcf.gz
```

## Use the source

```python
from altar.sources import ParquetAllelicRecordBackend
from altar_popfreq import GNOMAD_KEY_COLUMNS, GnomADRelease, GnomADSource

backend = ParquetAllelicRecordBackend("./gnomad-4.1.1-genomes", key_columns=GNOMAD_KEY_COLUMNS)
source = GnomADSource(backend, release=GnomADRelease(version="4.1.1", dataset="genomes"))
annotations = await source.annotate(["chr1:10001:A:G"])
```

The dataset can also be read in place where it is hosted. Pass a URI instead of
a path (`gs://`, `s3://`, `az://`, `https://`, `hf://`), with any non-secret
fsspec options as `storage_options`; credentials stay ambient. `gs://` works
with the binding alone; another provider needs its extra, such as `altar[s3]` or
`altar[http]`:

```python
source = GnomADSource.from_dataset(
    "gs://BUCKET/gnomad-4.1.1-genomes",
    storage_options={"token": "anon"},
)
```

Any `AllelicRecordBackend` works, including `SqliteAllelicRecordBackend` and
`BigQueryAllelicRecordBackend` over a table with the same columns. `GnomADSource`
checks every row it reads and raises `ValueError` on a row that breaks the column
contract (a non-null AF with AN 0, or a `gnomad_af_max_ancestry` that is not the
maximum of the ancestry AFs) and on two rows for one variant. The BigQuery
relation below checks differently; see [What each path checks](#what-each-path-checks).

### BigQuery at scale

Most of a cohort's variants are in gnomAD, so gnomAD is a dense source.
`staged_annotation_source` runs a Python source for each job and loads its rows
into a scratch table; it is meant for sparse sources. Instead, load the
published Parquet into BigQuery and read it in SQL:

```console
altar pull gnomad --bigquery project.reference.gnomad_4_1_1_genomes --project project
```

A release you built yourself is copied to Cloud Storage and loaded the same way
`altar pull` loads one:

```console
gcloud storage cp -r ./gnomad-4.1-joint gs://BUCKET/
bq load --source_format=PARQUET --clustering_fields=variant_id \
  project:reference.gnomad_4_1_joint 'gs://BUCKET/gnomad-4.1-joint/*.parquet'
```

```python
from altar_popfreq import GnomADRelease, bigquery_source

gnomad = bigquery_source(
    client,
    "project.reference.gnomad_4_1_1_genomes",
    release=GnomADRelease(version="4.1.1", dataset="genomes"),
)
# pass `gnomad` in annotation_sources to BigQueryScoreStore's materialize and export calls
```

In a run spec, `table` with a connection builds the same join (see
[Owned BigQuery](#owned-bigquery)); `bigquery_source` is for code that holds its
own client. It returns a `BigQueryAnnotationSource` with this binding's name,
columns, and identity (`release.identity()`). Its `relation=` comes from
`bigquery_relation(table_id)`, which selects the contract columns and keeps the
`{variant_filter}` token, so the store limits the gnomAD read to the job's
variants inside its one materialize query.
Use `bigquery_relation` directly to configure the source yourself, passing
`identity=release.identity()`.

### What each path checks

The Python path is `GnomADSource.annotate()` over any backend: a Parquet
dataset, SQLite, or a BigQuery table read by a store other than
`BigQueryScoreStore`. The SQL path is `bigquery_relation`, which
`bigquery_source` and a `table` source in a `BigQueryScoreStore`'s queries use.
The two enforce the column contract differently. On a table the builder wrote,
including one `altar pull --bigquery` loaded from the published Parquet, they
return the same values, because the builder already guarantees every rule below.
A table you made or edited yourself holds only what you checked.

| Rule | Python path (`GnomADSource.annotate`) | SQL path (`bigquery_relation`) |
| --- | --- | --- |
| `gnomad_af` is null when `gnomad_an` is 0 | rejects the row | computes it: `CASE WHEN gnomad_an = 0 THEN NULL ELSE gnomad_af END` |
| `gnomad_af_max_ancestry` is the greatest non-null ancestry AF | rejects the row | computes it from the ten ancestry columns and never reads the stored value |
| one row per variant | rejects the result | not checked: a repeated `variant_id` fans out the store's join |
| a NaN frequency | reads it as null | not checked: the table is trusted to hold no NaN, so a NaN reaches `GREATEST` and comparisons |

The relation computes the maximum with the null-ignoring form of
`altar.predicates.Greatest` (each `GREATEST` argument is a `COALESCE` over the
groups), because BigQuery's `GREATEST` is null when any argument is. Computing both
columns costs a few scalar expressions per row and scans no extra columns. A
uniqueness check would need a query-wide aggregation, so the relation relies on the
builder's guarantee, and the builder never writes NaN; a host's own table might.
The release check is the same on both paths: a `table` source checks the table's
labels once, when it is built. Check a hand-made table once after loading it:

```sql
SELECT variant_id, COUNT(*) FROM `project.reference.gnomad_4_1_1_genomes`
GROUP BY variant_id HAVING COUNT(*) > 1 LIMIT 10
```

## Filtering on frequency

The source has no prioritization predicate: a frequency says how common a variant
is, not that it is interesting. Apply frequency as a filter on materialized or
exported rows instead.

### Filter on the filtering allele frequency

For rare-disease filtering, gnomAD recommends the filtering allele frequency
(FAF), `gnomad_faf95_max`, which it describes for comparison "against a
disease-specific threshold" such as the BA1 criterion of the ACMG/AMP guidelines
([gnomAD help: FAF](https://gnomad.broadinstitute.org/help/faf);
[Whiffin et al. 2017](https://www.nature.com/articles/gim201726)). Write the
filter with Altar's predicate DSL, which renders the same rule to Python and to
SQL:

```python
from altar.predicates import Col, IsNull, Le, Lit, Or

rare = Or(IsNull(Col("gnomad_faf95_max")), Le(Col("gnomad_faf95_max"), Lit(1e-4)))
rare.to_python()(row)  # over a materialized or fetched row
rare.to_sql()  # ((gnomad_faf95_max IS NULL) OR gnomad_faf95_max <= 0.0001)
```

Set the threshold to the maximum credible allele frequency of the disease: if the
FAF is above it, the observed allele count is not compatible with pathogenicity.

What the FAF is, in gnomAD v4:

- **Per group, a lower confidence bound.** For each group, gnomAD computes the
  highest true AF at which the group's observed allele count would still be
  unlikely, from a Poisson 95% confidence interval. It is a conservative lower
  bound on the group's AF, so a few alleles in a small group do not make a
  variant look common.
- **The maximum over groups.** `gnomad_faf95_max` is the greatest per-group FAF.
  It comes from the group with the highest FAF, which need not be the
  `gnomad_grpmax` group, the one with the highest AF.
- **Non-bottlenecked groups only.** It covers the same groups as grpmax, in the
  table under [Columns](#columns): `afr`, `amr`, `eas`, `nfe`, and `sas` in
  genomes, plus `mid` in exomes and joint. It never covers `ami`, `asj`, `fin`,
  or `remaining`.
- **Null for one allele or fewer.** gnomAD sets a group's FAF to 0 when the group
  has at most one alternate allele, and reports the maximum only over groups
  whose FAF is above 0. So `gnomad_faf95_max` is null for a singleton, for a
  variant with at most one allele in each covered group, for a variant seen only
  in groups the FAF leaves out, and for an `AC0` site.

A variant gnomAD lacks has no gnomAD columns, so its FAF is also null. Use
`IsNull(Col("gnomad_ac"))` to tell an absent variant from a present one with a
null FAF.

### Bottlenecked groups: `gnomad_af_max_ancestry`

The FAF and grpmax leave out bottlenecked groups because founder effects make
some variants that are rare elsewhere common there. A founder variant common in
Ashkenazi Jewish or Finnish individuals, and rare in every covered group, passes
a FAF filter. `gnomad_af_max_ancestry` is the greatest AF over all ten groups,
those included. Add it when:

- the cohort is drawn from a bottlenecked group, or a variant common in any
  group should be removed;
- the FAF is null because the variant was seen only in groups it leaves out.

```python
from altar.predicates import And, Col, IsNull, Le, Lit, Or

rare_everywhere = And(
    Or(IsNull(Col("gnomad_faf95_max")), Le(Col("gnomad_faf95_max"), Lit(1e-4))),
    Or(IsNull(Col("gnomad_af_max_ancestry")), Le(Col("gnomad_af_max_ancestry"), Lit(1e-3))),
)
```

`gnomad_af_max_ancestry` is a point estimate with no confidence bound. In a small
group, such as `ami`, one or two alleles give a high AF, so give it a looser
threshold than the FAF.

### Unknown frequencies

In both rules, null passes. Without `IsNull`, `Le(...)` alone treats an unknown
frequency as not matching and drops the variants gnomAD lacks. Null passes only
if the cohort is normalized as described in
[Normalize the cohort first](#normalize-the-cohort-first); otherwise a common
variant spelled differently from gnomAD also passes.

Sites that failed gnomAD's filters are stored as reported. An `AC0` site (no
high-quality alternate allele) has `gnomad_af` 0.0 and a null FAF, so a rarity
filter treats it as observed and rare. If an observed, passing frequency is what
you mean, add `Eq(Col("gnomad_filters"), Lit("PASS"))`, or
`Gt(Col("gnomad_ac"), Lit(0))`, or treat a failed site's frequency as unknown.

## Data terms

gnomAD's [terms of use](https://gnomad.broadinstitute.org/terms) release the
primary exome and genome data under the
[Creative Commons Zero Public Domain Dedication](https://creativecommons.org/publicdomain/zero/1.0/),
with no restrictions on publishing results. The gnomAD team asks users to:

- acknowledge and cite gnomAD and the release used, and link back to it;
- not attempt to re-identify participants;
- contact the consortium before large-scale analyses that may overlap ongoing work.

Some annotations inside the gnomAD VCFs carry other licences (for example,
SpliceAI scores are CC BY-NC 4.0). The builder reads only the frequency, count,
and filter fields listed above, so the dataset the Kundaje lab publishes holds
only CC0 data. The package ships no gnomAD data. By default its source reads
the lab's published copy; operators who build their own download the VCFs
themselves and accept gnomAD's terms.
A tool built on it should state the release it includes, as gnomAD asks, for
example: "This tool includes data from the gnomAD v4.1.1 release."
gnomAD is a trademark of The Broad Institute, Inc.; this binding is not
affiliated with or endorsed by the gnomAD project.
