Metadata-Version: 2.5
Name: altar-clinvar
Version: 0.2.0
Summary: ClinVar germline-classification annotation-source binding for Altar
Project-URL: Documentation, https://kundajelab.github.io/altar/
Author: Riya Sinha
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: altar[gcs]<0.3,>=0.2.0
Provides-Extra: build
Requires-Dist: duckdb>=1.0.0; extra == 'build'
Requires-Dist: pyarrow>=17.0.0; extra == 'build'
Provides-Extra: test
Requires-Dist: pytest-asyncio>=0.24; extra == 'test'
Requires-Dist: pytest>=8; extra == 'test'
Description-Content-Type: text/markdown

# ClinVar binding

`altar-clinvar` exposes ClinVar's aggregate germline classifications through
Altar's `AnnotationSource` contract. ClinVar is a lookup source here: its
classifications are curated before an Altar analysis begins, and the binding
runs no model.

This tool includes data from ClinVar's 20260928 release, read by default from a
derived copy the Kundaje lab publishes, or from a release its operator builds.
The package ships no ClinVar data; see
[Where the data comes from](#where-the-data-comes-from) and
[Data terms](#data-terms). It is not affiliated with or endorsed by NCBI or
ClinVar.

## Columns

Every column is prefixed `clinvar_`, so ClinVar values never collide with
another source's columns.

| Column | Type | From | Meaning |
| --- | --- | --- | --- |
| `clinvar_variation_id` | int | `ID` | The ClinVar Variation ID. Never null. |
| `clinvar_allele_id` | int | `ALLELEID` | The ClinVar Allele ID. |
| `clinvar_classification` | str | `CLNSIG` | The aggregate germline classification, such as `Pathogenic/Likely_pathogenic` or `Pathogenic\|risk_factor`. Null when the record has no germline classification; see [below](#records-without-a-classification). |
| `clinvar_pathogenic` | bool | derived | Whether any term of the classification is `Pathogenic` or `Likely_pathogenic`; see [Classification terms](#classification-terms). Never null. |
| `clinvar_low_penetrance` | bool | derived | Whether any term is `Pathogenic,_low_penetrance` or `Likely_pathogenic,_low_penetrance`. Never null. |
| `clinvar_review_status` | str | `CLNREVSTAT` | The germline review status, such as `criteria_provided,_single_submitter`. |
| `clinvar_review_stars` | int | derived | ClinVar's gold stars, 0 to 4, for that review status. Null when the status is missing or not one the binding knows. |
| `clinvar_conditions` | repeated str | `CLNDN` | ClinVar's preferred names for the conditions, such as `not_provided`. |
| `clinvar_conflicting_classifications` | repeated str | `CLNSIGCONF` | For a conflicting classification, each submitted classification and its count, such as `Likely_benign_(2)`. |
| `clinvar_molecular_consequences` | repeated str | `MC` | ClinVar's molecular consequence names, such as `missense_variant` or `nonsense`, without their SO IDs. |
| `clinvar_variant_type` | str | `CLNVC` | The variant type, such as `single_nucleotide_variant` or `Deletion`. |
| `clinvar_origin` | int | `ORIGIN` | The allele-origin bitmask as ClinVar reports it: 1 germline, 2 somatic, 32 de novo, and so on; 0 is unknown. |
| `clinvar_origins` | repeated str | derived | The origins that bitmask combines, such as `["germline", "de_novo"]` for 33, or `["unknown"]` for 0. A bit ClinVar has not defined stays in `clinvar_origin` but has no label here. |
| `clinvar_genes` | repeated struct | `GENEINFO` | The genes ClinVar reports for the variant, each `{symbol: str, gene_id: int}` with its NCBI Gene ID, in ClinVar's order. |
| `clinvar_oncogenicity`, `clinvar_oncogenicity_review_status` | str | `ONC`, `ONCREVSTAT` | The aggregate oncogenicity classification and its review status. |
| `clinvar_somatic_impact`, `clinvar_somatic_impact_review_status` | str | `SCI`, `SCIREVSTAT` | The aggregate somatic clinical impact and its review status. |

Text keeps ClinVar's VCF spelling, with underscores for spaces, so a value
matches what other tools that read the ClinVar VCF report. VCF percent-escapes
are decoded: `%3B` in a condition name becomes `;`. Repeated columns are lists
and are never null; a field the record lacks is an empty list.

The gene and condition columns are what ClinVar reports for the variant. They
are display and filter values, not a gene or disease model: joining them to gene
tables or ontologies is left to the host.

### Classification terms

`CLNSIG` joins the germline classifications of one variant with `/` and adds
other terms after `|`. HFE C282Y (VariationID 9) is
`Pathogenic/Pathogenic,_low_penetrance|risk_factor`. The binding splits a value
on both separators and compares each term; a comma stays inside its term, so
`Pathogenic,_low_penetrance` is one term.

- `clinvar_pathogenic` is true when any term is `Pathogenic` or
  `Likely_pathogenic`. HFE C282Y, `Pathogenic/Likely_pathogenic`, and
  `Pathogenic/Likely_risk_allele` are pathogenic.
- `clinvar_low_penetrance` is true when any term is `Pathogenic,_low_penetrance`
  or `Likely_pathogenic,_low_penetrance`. A classification that is only low
  penetrance, such as `Pathogenic,_low_penetrance` alone, is **not**
  pathogenic: `clinvar_pathogenic` is false and only `clinvar_low_penetrance` is
  true. A host that wants such variants opts in through this scalar; see
  [Prioritization is opt-in](#prioritization-is-opt-in). HFE C282Y sets both.
- Risk alleles (`Likely_risk_allele`, `Uncertain_risk_allele`, `risk_factor`),
  `Conflicting_classifications_of_pathogenicity`, and every other term set
  neither flag.

### Records without a classification

`clinvar_classification` and `clinvar_review_status` are null, and
`clinvar_pathogenic` is false, for every record that has no germline
classification. That covers variants ClinVar classifies only for somatic
clinical impact or oncogenicity, variants classified only inside a haplotype or
genotype (ClinVar's "included" variants), and records with no classification at
all. Together they are a noticeable share of the file: 5 to 9% of the records in
samples of the 20260928 release. Such a record still has its Variation ID,
genes, consequences, and any somatic or oncogenicity classification.

### Review stars

`clinvar_review_stars` follows ClinVar's
[review status](https://www.ncbi.nlm.nih.gov/clinvar/docs/review_status/) table:

| Stars | Review status |
| --- | --- |
| 4 | `practice_guideline` |
| 3 | `reviewed_by_expert_panel` |
| 2 | `criteria_provided,_multiple_submitters,_no_conflicts` |
| 1 | `criteria_provided,_single_submitter`, `criteria_provided,_conflicting_classifications` |
| 0 | `no_assertion_criteria_provided`, `no_classification_provided`, `no_classification_for_the_single_variant`, `no_classifications_from_unflagged_records` |

`REVIEW_STARS` also maps the names ClinVar used before 2024
(`criteria_provided,_conflicting_interpretations`, `no_assertion_provided`,
`no_interpretation_for_the_single_variant`). The builder rejects VCFs that old,
so these entries exist only so that a table a host built from an older release
can still be read. A status not in the table gets null stars; the builder counts
and reports such statuses rather than failing.

### Absent variants

A variant that ClinVar has no record for is left out of `annotate()`'s result,
and Altar's merge gives it no ClinVar columns. The absence of a submission is not
evidence that a variant is benign.

## Release

`ClinVarRelease` names the data a backend serves:

```python
from altar_clinvar import ClinVarRelease

release = ClinVarRelease(release_date="20260928", genome_build="hg38")
release.release  # "20260928"
```

`release_date` is the date in ClinVar's file names, `clinvar_<YYYYMMDD>.vcf.gz`,
for weekly and monthly files alike. It is the identity's `release`, so each new
release is a different identity and cached rows of different releases never mix.

`genome_build` is `hg38`, for ClinVar's `vcf_GRCh38` files, or `hg19`, for
`vcf_GRCh37`. Both are supported: ClinVar generates both files from the same
records each week, so the classification columns mean the same thing on either
assembly and only the coordinates differ. The identity carries the build, so an
hg19 table never satisfies an hg38 analysis.

An hg19 build has no mitochondrial rows. ClinVar's GRCh37 `MT` records use the
revised Cambridge Reference Sequence (NC_012920), but UCSC hg19's `chrM` is an
older mitochondrial sequence (NC_001807) with different coordinates, so an hg19
`chrM` cohort variant cannot be matched to them safely. The builder skips them
and counts the skips. On hg38, `chrM` is the rCRS and `MT` records are kept.

The columns form the published `CLINVAR_ANNOTATIONS` contract
(`org.kundajelab.altar.annotation.clinvar`, version 1.0.0). `source.identity()`
and `release.identity()` return that contract with the release date and the
genome build. A host table that holds the same columns can declare the same
contract.

### Older releases

Until January 2024, ClinVar's `CLNSIG` held a single "clinical significance"
that mixed germline and somatic submissions. Starting with the `20240127` file,
`CLNSIG` and `CLNREVSTAT` describe the germline classification only, and the
oncogenicity (`ONC`) and somatic clinical impact (`SCI`) classifications have
fields of their own. The contract's columns mean the newer quantities, so
`ClinVarRelease` rejects dates before `20240127` and the builder rejects a
header without the `ONC` and `SCI` fields. The `archive_1.0` files (September
2017 and earlier) use another format altogether.

## Where the data comes from

The source reads one ClinVar release in one of three ways. All three serve the
same data under the same identity, `source.identity()`: the contract, the
release date, and the genome build. Where the data is read from, and with which
credentials, is never part of it, so switching between them keeps every stored
score and annotation a cache hit. `prioritize_pathogenic` works the same way in
all three (see [Prioritization is opt-in](#prioritization-is-opt-in)).

| Mode | Spec options | Reads |
| --- | --- | --- |
| Published (default) | none, or `location` | The release the Kundaje lab publishes, read in place, anonymously. |
| Owned Parquet | `dataset` | Your copy of the builder's Parquet: a path, or a URI in your bucket. |
| Owned BigQuery | `table` | Your BigQuery table of the builder's columns, such as one loaded from the published Parquet. |

### Published

The lab builds ClinVar's GRCh38 VCF with `altar-clinvar-build` and publishes it
in a public-read bucket, mirrored byte for byte in a public Hugging Face dataset
([altar-genomics/clinvar-20260928-hg38](https://huggingface.co/datasets/altar-genomics/clinvar-20260928-hg38)):

| Release | Genome build | Location name | URI |
| --- | --- | --- | --- |
| `20260928` | hg38 | `gcs` (default) | `gs://kundajelab-altar/clinvar/clinvar-20260928-hg38` |
| `20260928` | hg38 | `hf` | `hf://datasets/altar-genomics/clinvar-20260928-hg38` |

ClinVar publishes a new VCF every week, and a monthly release about once a
month, so the published copy is soon not the newest ClinVar has. The binding
pins one release, the latest the lab has published
(`altar_clinvar.PUBLISHED_RELEASE`); its catalog lists no older release, and the
hg19 build is not published. A newer release is a different identity, so a
binding version that moves to it reads it as new data rather than mixing it with
cached rows of the old one. To use a release the binding does not pin yet, or
hg19, build it yourself (see [Build the dataset](#build-the-dataset)) and name it
with `dataset`.

A source with no `dataset` and no `table` reads the published release in place,
with no credentials, and checks that its first file records that release;
`altar pull` checks every file of a copy it makes. `altar-clinvar` depends on
`altar[gcs]`, so `pip install altar-clinvar` is enough to read it:

```toml
[sources.clinvar]
type = "clinvar"
```

To let reviewed pathogenic classifications set `prioritized`:

```toml
[sources.clinvar]
type = "clinvar"
options = { prioritize_pathogenic = true }
```

A lookup reads only the chromosome partitions and row groups its variants fall
in, not the whole release. To bill the reads to
your project, or to read through other credentials, name a
[connection](https://kundajelab.github.io/altar/guides/connections/) whose
storage options replace the anonymous ones.

`location` reads the Hugging Face mirror instead of the bucket. It holds the
same files, so the source has the same identity and every stored score stays a
cache hit. Reading `hf://` needs the `altar[hf]` extra
(`pip install "altar[hf]"`), which `altar-clinvar` does not install; without it,
the source fails with an error that names the extra. `location` applies only to
the published copy, not with `dataset` or `table`:

```toml
[sources.clinvar]
type = "clinvar"
options = { location = "hf" }
```

In code, `ClinVarSource.from_options()` builds the same source
(`from_options(location="hf")` for the mirror), and
`ClinVarSource.published_dataset()` (also `altar_clinvar.PUBLISHED_DATASET`)
returns the published locations and identity.

### Owned Parquet

`dataset` names a copy of the builder's output: the published dataset copied to
your bucket or disk, or one you build from ClinVar's VCF (see
[Build the dataset](#build-the-dataset)), which may be a newer release than the
published one. One command copies the published dataset, file by file, and
checks that every copied file records its release:

```console
altar pull clinvar --to gs://my-bucket/clinvar-20260928-hg38
altar pull clinvar --from hf --to ./clinvar-20260928-hg38
```

`--from hf` copies from the Hugging Face mirror (with `altar[hf]` installed)
instead of the bucket; the copy is the same release either way. `--to` also
takes a local directory. It writes with your Application Default Credentials;
`--project` or `--impersonate-service-account` choose them (see
[Copy a published dataset](https://kundajelab.github.io/altar/guides/published-datasets/)).
The release comes from the files' metadata. A connection gives the storage
options for the URI:

```toml
[connections.lab]
type = "gcp"
options = { project = "my-project" }

[sources.clinvar]
type = "clinvar"
options = { dataset = "gs://my-bucket/clinvar-20260928-hg38" }
connection = "lab"
```

A local path needs no connection. Without one, `storage_options` passes
non-secret fsspec options directly, such as `{ token = "anon" }`; a source
refuses `storage_options` together with `connection`.

### Owned BigQuery

`table` names a BigQuery table with the builder's columns, for example the
published Parquet loaded into your project. One command loads it, with one
BigQuery load job into a table clustered on `variant_id`:

```console
altar pull clinvar --bigquery my-project.reference.clinvar_20260928_hg38 --project my-project
```

The load infers Parquet lists, so the repeated columns become arrays:
`clinvar_conditions`, `clinvar_conflicting_classifications`,
`clinvar_molecular_consequences` and `clinvar_origins` are `ARRAY<STRING>`, and
`clinvar_genes` is `ARRAY<STRUCT<symbol STRING, gene_id INT64>>`. A field the
record lacks is an empty array. Without list inference (a `bq load` without
`--parquet_enable_list_inference`), BigQuery would keep Parquet's encoding,
`STRUCT<list ARRAY<STRUCT<element ...>>>`, which the source rejects.

The dataset `reference` must exist in a location that can load from the
published bucket. Otherwise, pull the files to a bucket in the dataset's region
with `--to`, and load that copy with
`--source-uri gs://my-bucket/clinvar-20260928-hg38`. The source queries the table
with the connection's BigQuery client, so it needs a `gcp` connection with
credentials:

```toml
[connections.lab]
type = "gcp"
options = { project = "my-project" }

[sources.clinvar]
type = "clinvar"
options = { table = "my-project.reference.clinvar_20260928_hg38" }
connection = "lab"
```

A table carries no Parquet metadata. The source declares the published release,
`20260928` on hg38, unless `release` names another release date, such as
`release = "20261005"`, or `genome_build = "hg19"` names the other assembly.
`altar pull --bigquery` records the release as table labels
(`altar_source`, `altar_release`, `altar_genome_build`), and the source reads
them with one metadata request (no query, no bytes billed) when it is built: a
table labelled with another release or build is refused, and a table with no
labels, such as one loaded by hand, is read with a warning naming the release it
assumes. To label a table you loaded yourself:

```console
bq update --set_label altar_source:org_kundajelab_altar_annotation_clinvar \
  --set_label altar_release:20261005 --set_label altar_genome_build:hg38 \
  my-project:reference.clinvar_20261005_hg38
```

The source reads the table in either of two ways, chosen by the store. A
`BigQueryScoreStore` joins it into its own materialize and export queries in
SQL, through `bigquery_relation(table)`, so a run spec with a
[BigQuery store](https://kundajelab.github.io/altar/guides/connections/#keep-scores-in-bigquery)
and this source reads ClinVar in the same query as the scores. Any other store
looks variants up through the Python source in batches of queries. The two paths
check rows differently; see [What each path checks](#what-each-path-checks).

## Build the dataset

`altar-clinvar-build` streams ClinVar's VCF into Parquet, one row per allele,
keyed by the canonical `chr:pos:ref:alt`:

```console
altar-clinvar-build --release-date 20260928 --genome-build hg38 \
  --input clinvar_20260928.vcf.gz \
  --output ./clinvar-20260928-hg38
```

Pass the release's `clinvar_<date>_papu.vcf.gz` file as a second input to add
the variants ClinVar places on the chrY pseudoautosomal regions; its records on
patches, alternate loci, and unplaced contigs are skipped.

`--input` and `--output` also take URIs (`https://`, `gs://`, `s3://`, `hf://`,
...), so publishing a release is one command, with no download and no copy step:

```console
altar-clinvar-build --release-date 20260928 --genome-build hg38 \
  --input https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/clinvar_20260928.vcf.gz \
  --output gs://my-bucket/clinvar-20260928-hg38/
```

A URI needs its provider's Altar extra, such as `altar[http]` for `https://`
or `altar[gcs]` for `gs://`; a missing one is named in the error. An input URI
is streamed through fsspec; its name, the last segment of its
path, names its output file and is checked against `--release-date`. A stream
that fails or ends early is read again from the start, discarding what was
written for it, up to `--input-attempts` times (3 by default). With an
`--output` URI, each input is built in a local scratch directory (under
`TMPDIR`), its keys are checked there, and each finished file is uploaded under a
temporary name and committed, so a reader never sees a partial file. Until the
build finishes, the output holds `_altar_build/`, which records the inputs
already committed, so running the same command again after a crash skips them
and builds the rest; without `--overwrite`, a URI output must be empty or hold
such an unfinished build of the same release. A partition that receives files
from two inputs (chrY, from the release file and its `_papu` file) is checked for
repeated keys again once both are uploaded. Credentials are the provider's
ambient ones; `--input-storage-option` and `--output-storage-option` pass
non-secret fsspec options as repeatable `KEY=VALUE` arguments, and refuse a key
that would carry a credential. The gnomAD builder shares these rules; see
"Publish from a bucket to a bucket" in the
[`altar-popfreq` README](https://pypi.org/project/altar-popfreq/) for the details.

It writes `<output>/chr=<chromosome>/<input name>.parquet`. Each file has
`variant_id`, the key columns `chr`, `pos`, `ref`, `alt`, and the columns above,
and records `altar_clinvar.release_date`, `altar_clinvar.genome_build`, and
`altar_clinvar.builder_version` in its Parquet metadata. The builder needs
PyArrow and DuckDB, which `altar` already requires; `pip install
'altar-clinvar[build]'` states them explicitly.

Rows are sorted by position and written with zstd in row groups of 32,000 rows,
so a point lookup reads one small row group per variant it finds, plus each
file's footer once per backend.

The builder:

- requires an empty or absent `--output`. `--overwrite` first deletes a previous
  build there, and refuses if the directory holds anything a build does not write.
  A URI output may also hold an unfinished build of the same release, which is
  resumed;
- checks inputs named like ClinVar's files, `clinvar_<YYYYMMDD>.vcf.gz` and
  `clinvar_<YYYYMMDD>_papu.vcf.gz`, against `--release-date`;
- reads bgzipped, gzipped, or plain VCF, telling them apart by their first bytes,
  and rejects a bgzipped input that does not end with bgzip's end-of-file block,
  so a truncated bgzip download or stream is not built short (plain or plain-gzip
  input has no such marker). A URI input cut short is read again instead;
- checks each header: `##source=ClinVar`, a `##fileDate` equal to the release
  date, a `##reference` equal to the build's assembly (`GRCh38` or `GRCh37`), and
  every INFO field it reads, including the `ONC` and `SCI` fields of the current
  format;
- writes chromosomes with the canonical `chr` spelling, so `1` becomes `chr1` and
  `MT` becomes `chrM`, and skips `MT` on an hg19 build (see [Release](#release));
- trims each allele to its minimal representation (ClinVar's already are);
- skips and counts records with no ALT (ALT `.`, as for ClinVar's
  `no_sequence_alteration` records), a symbolic ALT, or a non-ACGT base (ClinVar
  writes every IUPAC ambiguity code as `N`), and records on contigs other than
  chr1-22, X, Y, and M;
- counts review statuses and `ORIGIN` bits it does not know, and keeps the rows:
  such a status gets null stars, and such a bit stays in `clinvar_origin` without
  a label;
- rejects a record with more than one ALT; ClinVar records describe one allele;
- reads numbers from the VCF text, and splits `CLNDN`, `CLNSIGCONF`, and
  `GENEINFO` on `|` only, because a comma is part of a condition name and of a
  review status;
- rejects a POS that is not a positive integer, naming the file and line;
- rejects input that is not sorted by position, and any variant key that occurs
  twice across all inputs, including one that trimming produces. A repeated key
  would make every join fan out. Records on skipped contigs are not held to the
  sort order;
- rejects a build that writes no rows, such as one from a header-only file;
- deletes everything it wrote to a local `--output` if any input fails. A URI
  output keeps the files of the inputs already committed, so the build resumes.

It prints how many records it skipped for each reason, and warns about any review
status or `ORIGIN` bit it does not know.

To read from storage, name the URI as an `--input` (see above). `--input -`
reads one VCF from standard input instead, for a VCF another program produces:

```console
bcftools view -Oz clinvar_20260928.vcf.gz \
  | altar-clinvar-build --release-date 20260928 --genome-build hg38 --input - \
      --stdin-name clinvar_20260928.vcf.gz --output ./clinvar-20260928-hg38
```

ClinVar's header records the release date and assembly, so standard input gets
the same header checks as a file: the build fails unless the piped VCF is the
20260928 release. A pipe cannot be read again, so a stream cut short fails the
build. `--stdin-name` names the output file
(`stdin.parquet` without it) and is checked like a file name when it follows
ClinVar's naming. Standard input can be read once per build. The end-of-file
check catches a cut-short stream only for bgzip input, such as ClinVar's own
`.vcf.gz`; plain text or plain gzip cut at a line or member boundary builds short
without an error, so pipe the bgzipped file as published.

`ClinVarSource` over a `ParquetAllelicRecordBackend` reads that metadata and
refuses a dataset built for another release date or build, or one that records
no release at all. It reads the first file in sorted order only, so opening a
remote dataset costs one footer. A host's own export must carry the same keys;
`altar_clinvar.RELEASE_METADATA.write(release_fields(release))` returns them.

### Normalize the cohort first

The keys are ClinVar's representation, trimmed and left-aligned. Altar's VCF
ingestion keeps each record's REF when it splits ALTs and does not trim or
left-align, so a multi-allelic cohort record can miss ClinVar even for an SNV.
The record `chr1 100 CA C,TA` becomes `chr1:100:CA:C` and `chr1:100:CA:TA`;
ClinVar holds the second allele as the SNV `chr1:100:C:T`. That variant finds
no ClinVar row and so has no ClinVar columns, however it is classified.

Normalize any cohort with multi-allelic records, or with indels, before
ingestion:

```console
bcftools norm -m -any -f <reference.fa> cohort.vcf.gz -Oz -o cohort.norm.vcf.gz
```

## Use the source

```python
from altar.sources import ParquetAllelicRecordBackend
from altar_clinvar import CLINVAR_KEY_COLUMNS, ClinVarRelease, ClinVarSource

backend = ParquetAllelicRecordBackend("./clinvar-20260928-hg38", key_columns=CLINVAR_KEY_COLUMNS)
source = ClinVarSource(backend, release=ClinVarRelease(release_date="20260928", genome_build="hg38"))
annotations = await source.annotate(["chr1:1022368:C:A"])
```

The dataset can also be read in place where it is hosted. Pass a URI instead of
a path (`gs://`, `s3://`, `az://`, `https://`, `hf://`), with any non-secret
fsspec options as `storage_options`; credentials stay ambient. The provider's
extra must be installed, such as `altar[gcs]` or `altar[http]`:

```python
source = ClinVarSource.from_dataset(
    "gs://BUCKET/clinvar-20260928-hg38",
    storage_options={"token": "anon"},
)
```

Any `AllelicRecordBackend` works, including `SqliteAllelicRecordBackend` and
`BigQueryAllelicRecordBackend` over a table with the same columns. SQLite has no
arrays, so store the repeated columns there as JSON array text; the source
decodes it. The source rejects a table that breaks the column contract: a
`clinvar_pathogenic`, `clinvar_low_penetrance`, or `clinvar_review_stars` that
does not follow from the classification and review status, `clinvar_origins` that
do not decode `clinvar_origin`, a `clinvar_genes` record without both a symbol and
a gene ID, or two rows for one variant.

### Prioritization is opt-in

By default `prioritize_predicate()` returns `None`. ClinVar is curated evidence,
not a model prediction, and many hosts use it as a filter or a display column
rather than as a reason to prioritize a variant. A host that wants it to set
`prioritized` passes `prioritize_pathogenic=True`:

```python
source = ClinVarSource(backend, release=release, prioritize_pathogenic=True)
source.prioritize_predicate()  # PATHOGENIC_PREDICATE
```

`PATHOGENIC_PREDICATE` is
`And(Eq(Col("clinvar_pathogenic"), Lit(True)), Ge(Col("clinvar_review_stars"), Lit(1)))`:
a pathogenic or likely pathogenic germline classification with at least one
review star. Zero stars means no submitter provided assertion criteria (or there
is no classification), so the predicate requires one star. `clinvar_pathogenic`
itself does not, so a host can apply a different bar, such as two stars, in its
own predicate. The flag changes prioritization only, not the values, so it is not
part of the identity.

`PATHOGENIC_PREDICATE` does not select a classification that is only low
penetrance. To include those, compose the scalar flag rather than matching
`clinvar_classification` text, which also catches combined values such as
`Likely_pathogenic,_low_penetrance/Likely_risk_allele`:

```python
from altar.predicates import And, Col, Eq, Ge, Lit, Or
from altar_clinvar import PATHOGENIC_PREDICATE

reviewed_low_penetrance = And(Eq(Col("clinvar_low_penetrance"), Lit(True)), Ge(Col("clinvar_review_stars"), Lit(1)))
pathogenic_or_low_penetrance = Or(PATHOGENIC_PREDICATE, reviewed_low_penetrance)
```

Pass such a predicate where the host evaluates its own rules; the source's
`prioritize_predicate()` stays `PATHOGENIC_PREDICATE` or `None`.

### BigQuery at scale

`staged_annotation_source` runs a Python source for each job and loads its rows
into a scratch table. Instead, load the published Parquet into BigQuery and read
it in SQL:

```console
altar pull clinvar --bigquery project.reference.clinvar_20260928_hg38 --project project
```

A release you built yourself is copied to Cloud Storage and loaded the same way
`altar pull` loads one, with list inference so the repeated columns become
`ARRAY`s:

```console
gcloud storage cp -r ./clinvar-20261005-hg38 gs://BUCKET/
bq load --source_format=PARQUET --parquet_enable_list_inference --clustering_fields=variant_id \
  project:reference.clinvar_20261005_hg38 'gs://BUCKET/clinvar-20261005-hg38/*.parquet'
```

```python
from altar_clinvar import ClinVarRelease, bigquery_source

clinvar = bigquery_source(
    client,
    "project.reference.clinvar_20260928_hg38",
    release=ClinVarRelease(release_date="20260928", genome_build="hg38"),
)
# pass `clinvar` in annotation_sources to BigQueryScoreStore's materialize and export calls
```

In a run spec, `table` with a connection builds the same join (see
[Owned BigQuery](#owned-bigquery)); `bigquery_source` is for code that holds its
own client. It returns a `BigQueryAnnotationSource` with this binding's name,
columns, and identity (`release.identity()`), and takes the same
`prioritize_pathogenic` flag. Its `relation=` comes from
`bigquery_relation(table_id)`, which selects the contract columns, the arrays
as they are, and keeps the `{variant_filter}` token, so the store limits the
ClinVar read to the job's variants inside its one materialize query.

### What each path checks

The Python path is `ClinVarSource.annotate()` over any backend: a Parquet
dataset, SQLite, or a BigQuery table read by a store other than
`BigQueryScoreStore`. The SQL path is `bigquery_relation`, which
`bigquery_source` and a `table` source in a `BigQueryScoreStore`'s queries use.
On a table the builder wrote, including one `altar pull --bigquery` loaded from
the published Parquet, they return the same values, because the builder
already guarantees every rule below. A table you made or edited yourself holds
only what you checked.

| Rule | Python path (`ClinVarSource.annotate`) | SQL path (`bigquery_relation`) |
| --- | --- | --- |
| `clinvar_pathogenic`, `clinvar_low_penetrance` and `clinvar_review_stars` follow from the classification and review status | rejects the row | reads the stored values |
| `clinvar_origins` decodes `clinvar_origin` | rejects the row | reads the stored values |
| each `clinvar_genes` record has a symbol and a gene ID | rejects the row | reads the stored values |
| repeated columns are arrays | rejects a record-shaped column, such as a load without list inference | selects the column as it is |
| one row per variant | rejects the result | not checked: a repeated `variant_id` fans out the store's join |

`PATHOGENIC_PREDICATE` reads `clinvar_pathogenic` and `clinvar_review_stars`,
so on the SQL path it trusts the stored flags.

## Write ClinVar values to an augmented VCF

The augmented-VCF writer takes caller-declared annotation INFO fields (see
Altar's VCF documentation). A declaration file that adds the classification,
the review stars, and the first gene symbol:

```json
{
  "include_defaults": true,
  "fields": [
    {
      "id": "CLINVAR_CLNSIG",
      "columns": ["clinvar_classification"],
      "type": "String",
      "description": "ClinVar 20260928 aggregate germline classification per ALT"
    },
    {
      "id": "CLINVAR_STARS",
      "columns": ["clinvar_review_stars"],
      "type": "Integer",
      "description": "ClinVar review stars (0-4) per ALT"
    },
    {
      "id": "CLINVAR_GENE",
      "columns": ["clinvar_genes"],
      "type": "String",
      "description": "First ClinVar gene symbol per ALT",
      "first_element": true,
      "fields": ["symbol"]
    }
  ]
}
```

In Python, the last one is
`AnnotationInfoField("CLINVAR_GENE", ("clinvar_genes",), "String", "First ClinVar gene symbol per ALT", first_element=True, fields=("symbol",))`.
The writer percent-encodes the commas inside values such as
`criteria_provided,_single_submitter`.

## Data terms

ClinVar is produced by the National Center for Biotechnology Information (NCBI)
at the U.S. National Library of Medicine. NCBI's
[website and data usage policies](https://www.ncbi.nlm.nih.gov/home/about/policies/)
state that for its molecular databases "NCBI itself places no restrictions on the
use or distribution of the data contained therein", while noting that some
submitters may claim intellectual property rights in the data they submitted.
ClinVar's
[data use policy](https://www.ncbi.nlm.nih.gov/clinvar/docs/maintenance_use/)
asks: "If you distribute or copy data from ClinVar, we ask that you provide
attribution to ClinVar as a data source in publications and websites", and
suggests citing a ClinVar publication such as
[PMID 29165669](https://pubmed.ncbi.nlm.nih.gov/29165669/).

ClinVar also states that its information "is not intended for direct diagnostic
use or medical decision-making without review by a genetics professional". This
binding reports ClinVar's classifications; it does not make clinical
classifications of its own.

The package ships no ClinVar data. By default its source reads the derived
copy the Kundaje lab publishes: the fields listed under [Columns](#columns),
built from ClinVar's VCF by `altar-clinvar-build`. Operators who build their own
download the VCFs themselves. A tool built on it should attribute ClinVar and
name the release it includes, for example: "This tool includes data from
ClinVar's 20260928 release." This binding is not affiliated with or endorsed by
NCBI or ClinVar.
