Metadata-Version: 2.4
Name: pairctrl
Version: 0.1.0
Summary: Evaluation code of PairCtrl, a position- and relatedness-controlled benchmark for genomic language models
Author: Tejasvi S. Tomar
License-Expression: Apache-2.0
Project-URL: Source, https://github.com/tejasvi/pairctrl
Project-URL: Dataset, https://huggingface.co/datasets/tstomar/pairctrl
Project-URL: Archive, https://doi.org/10.5281/zenodo.23605245
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.22
Requires-Dist: pandas>=1.4
Requires-Dist: scipy>=1.8
Requires-Dist: scikit-learn>=1.1
Requires-Dist: pyarrow>=8
Requires-Dist: seqleak==0.4.1
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Dynamic: license-file

# PairCtrl

Evaluation code of PairCtrl, a benchmark for genomic language models on gene pairs. Gene position alone, or a close
relative in the training data, predicts the labels of many gene-pair tasks. PairCtrl uses experimental labels, scores
every model next to a position-only baseline (row zero) and reports how much of a score remains with gene position
held fixed. The data, the per-item scores of the leaderboard models and the leaderboard tables are in the dataset
repository https://huggingface.co/datasets/tstomar/pairctrl.

## Install

```
pip install pairctrl          # Python >= 3.9, installs seqleak (formerly posbench) for the position model
pip install -e ".[test]"      # from a clone of this repository, with the tests
pytest
```

## Score a model

1. Download the dataset into a directory, DATA_DIR. Each task has a folder with `items.parquet`.
2. Score every item of a task. For a zero-shot score, embed the genes listed in `genomes/genes.parquet` and take the
   cosine similarity of the two genes of each item:
   `python -m pairctrl.zeroshot DATA_DIR/ppi_exp/items.parquet embeddings.npz scores.parquet`.
   Here `embeddings.npz` holds one array per genome record, keyed by accession, with one row per gene in `gene_index`
   order. The option `--standardise` z-scores each dimension within a genome first. Supervised scores must be
   out-of-fold predictions under the task's frozen folds.
3. Write a parquet or CSV file with the columns `item_id` and `score` and run

```
python -m pairctrl.evaluate --task ppi_exp --data DATA_DIR --submission scores.parquet --out metrics.json
```

Add `--oof` for out-of-fold predictions, which are then compared only within folds. The output has one row per view
and evaluation unit (data set, genome or screen class) for the submission, row zero and the untrained distance rule.
Each row gives the AUROC, the AUROC within position strata, the AUPRC, the gain over the out-of-fold position model
and an interval for the share of the AUROC above 0.5 that is due to gene position, each with a 95% bootstrap
interval. Items without a score are left out, and the command reports the coverage.

## Tasks

| task | primary metric | evaluation |
|---|---|---|
| ppi_exp: experiment-only protein interactions | AUROC on the strict subset | zero-shot, or clade folds for supervised use |
| operon_expr: expression-filtered operons | gain over the position model | zero-shot, or one genome held out per fold |
| operon_regulondb_exp: DGEB E. coli pairs, experimental units | gain over the position model | zero-shot |
| etgp_nodist: enhancer-target genes, distance-free inputs | AUROC | loci held out |

Row zero is the position-only model, a logistic regression on spline bases of the position features that is fitted
out of fold on the task's frozen folds. The distance rule is an untrained baseline. For gene pairs it scores
1[same strand] x exp(-gap / 500 bp), with a score of 0 above a gap of 500 bp, and for element-gene pairs it is the
negative distance. A view is a label column with a row filter, and the strict subset of ppi_exp is one such view.
`pairctrl/specs.py` defines the views of each task.

## Labels and inputs that you rebuild

The dataset leaves out the RegulonDB labels of operon_regulondb_exp and the hg19 sequences of etgp_nodist.
`python -m pairctrl.joins.regulondb TUSet.txt GeneProductSet.txt items.parquet labels_regulondb.parquet` writes the
labels from your own RegulonDB v14.5 files. Place the output next to the task's `items.parquet`.
`python -m pairctrl.joins.etgp_inputs hg19.fa items.parquet inputs.parquet` writes the model inputs from your copy of
hg19.

## Leaderboard submissions

Open an issue with a link to per-item scores for every item of a task and a short model card. The card names the
model, the layer or readout, whether the scores are zero-shot or supervised, and how the training data relate to the
test genomes. The maintainer evaluates the scores with this package and adds the rows. Numbers without per-item
scores are not added.

## Metrics

`pairctrl/metrics.py` defines every reported quantity. `pairctrl/boot.py` computes the bootstrap intervals exactly
from Mann-Whitney sufficient statistics, with two-way resampling of gene blocks for gene pairs and of clusters
otherwise.

## Citation

See `CITATION.cff`. The release is archived at https://doi.org/10.5281/zenodo.23605245.

## License

Code: Apache License 2.0 (`LICENSE`). The dataset is released under CC BY 4.0, except the per-item scores of six
models with non-commercial licences, which are under CC BY-NC-SA 4.0. Its `NOTICE.md` lists the licence of every
source.
