Metadata-Version: 2.4
Name: pgen-polars
Version: 0.1.0
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Rust
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Typing :: Typed
Requires-Dist: polars>=1.31,<1.43
Requires-Dist: numpy>=2.0 ; extra == 'benchmark'
Requires-Dist: pgenlib>=0.94.1,<0.95 ; extra == 'benchmark'
Requires-Dist: pytest>=8.0 ; extra == 'dev'
Requires-Dist: maturin>=1.7,<2.0 ; extra == 'dev'
Requires-Dist: numpy>=2.0 ; extra == 'dev'
Requires-Dist: twine>=6.0 ; extra == 'dev'
Provides-Extra: benchmark
Provides-Extra: dev
License-File: LICENSE
Summary: Alpha Rust-native PLINK and BGEN reader for Python and Polars.
Keywords: bioinformatics,genetics,pgen,plink,polars
Requires-Python: >=3.12
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Changelog, https://github.com/idinsmore1/pgen_rs/blob/main/CHANGELOG.md
Project-URL: Homepage, https://github.com/idinsmore1/pgen_rs
Project-URL: Issues, https://github.com/idinsmore1/pgen_rs/issues
Project-URL: Repository, https://github.com/idinsmore1/pgen_rs

# pgen-polars

Rust-native reader for PLINK 2 `.pgen`, PLINK 1 `.bed`, and BGEN v1.2
genotype files, with Python and Polars integration.

> [!WARNING]
> **Alpha software:** `pgen-polars` is under active development. APIs,
> supported formats, and behavior are subject to change between releases.
> Pin the package version when using it in other projects.

## Python package installation

Create or activate a Python 3.12 or newer virtual environment, then install
the package from PyPI. Because the project is alpha software, pin the exact
version in applications and reproducible environments:

```bash
python -m venv .venv
source .venv/bin/activate  # Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install "pgen-polars==0.1.0"
```

With [`uv`](https://docs.astral.sh/uv/):

```bash
uv add "pgen-polars==0.1.0"
```

Verify the installation:

```bash
python -c "import pgen_polars; print(pgen_polars.__version__)"
```

Published wheels support CPython 3.12 and newer on Linux (x86-64 and ARM64),
macOS (Intel and Apple Silicon), and 64-bit Windows. Building from the source
distribution requires Rust 1.85 or newer. `pgen-polars` currently supports
Polars 1.31 through 1.42; this bounded range protects the lazy reader from
incompatible changes to Polars' unstable Python IO-plugin interface.

> Status: eager and lazy/chunked `.bed`, `.pgen`, and BGEN Layout 2 reading is
> implemented, including ordered sample/variant selection and long or
> samples-by-variants wide output. Statistics remain deferred.
> See the
> [roadmap](https://github.com/idinsmore1/pgen_rs/blob/main/docs/milestones.md)
> and [product specification](https://github.com/idinsmore1/pgen_rs/blob/main/docs/product-spec.md)
> for details.

## Layout

```text
crates/pgen-core     Python-free Rust core (parsing, decoding, scanning)
crates/pgen-rs       `pgen-rs` command-line tool (TSV output, no Python)
crates/pgen-python   pyo3 bindings, builds the native _pgen_polars module
python/pgen_polars   Pure-Python package exposing the public API
tests/               Test data (tests/examples) and Python tests
docs/                Product spec, milestones, technical decisions
```

## Development

Requires a Rust toolchain and [`uv`](https://docs.astral.sh/uv/) (Python 3.12+).

```sh
# Rust checks
cargo fmt --all --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace

# Build the Python extension and run tests
uv venv --python 3.12
uv pip install -e '.[dev]'
uv run pytest tests/python -q
```

During development you can rebuild the native module in place with:

```sh
uv run maturin develop
```

Use `uv run maturin develop --release` when measuring reader performance;
the default development build is intentionally unoptimized.

## Releases

The Python package version comes from `[workspace.package].version` in
`Cargo.toml`. To prepare a release, update that version and `CHANGELOG.md`, run
the checks above, and push a matching tag such as `v0.1.0`. The release workflow
builds and tests platform wheels, builds a source distribution, and publishes
the artifacts to PyPI through Trusted Publishing.

Before the first release, add a pending Trusted Publisher on PyPI with project
name `pgen-polars`, owner `idinsmore1`, repository `pgen_rs`, workflow
`release-python.yml`, and environment `pypi`. Create the matching `pypi`
environment in the GitHub repository. No long-lived PyPI token is required.
While the project is in alpha, release notes must call out compatibility
changes and downstream projects should pin an exact package version.

## Command-line usage

The `pgen-rs` binary reads metadata files and prints them as TSV — no Python
required:

```sh
cargo run -p pgen-rs -- meta pvar tests/examples/toy.pvar
cargo run -p pgen-rs -- meta psam tests/examples/toy.psam
# kinds: psam | pvar | fam | bim
```

It also decodes `.bed` and `.pgen` genotypes, auto-detecting sidecars:

```sh
# Wide format (default): one row per sample, one column per variant.
cargo run -p pgen-rs -- pgen tests/examples/toy.bed \
    --variant-id snp0,snp1 --sample-index 0,1

# Long format remains available explicitly for row-oriented output.
cargo run -p pgen-rs -- pgen tests/examples/toy.bed --format long \
    --variant-id snp0 --sample-index 0,1
```

## Python API

Metadata readers return Polars DataFrames:

```python
from pgen_polars import read_psam, read_pvar, read_fam, read_bim

variants = read_pvar("tests/examples/toy.pvar")   # CHROM, POS, ID, REF, ALT
samples = read_psam("tests/examples/toy.psam")     # IID, SEX, PHENO1
```

BGEN v1.2 Layout 2 uses dedicated readers. A paired Oxford `.sample` file is
required, and an adjacent `<file>.bgen.bgi` index is used when present.

When a `.bgi` exists, numeric selections query the requested SQLite rows and
RSID selections query its `rsid` column, then decode only those BGEN records.
Without one, numeric selection scans only through its highest requested index;
SNP-ID selection also uses the RSID index when the pointed BGEN record confirms
that both identifiers match, otherwise it scans identifying headers. Unselected
probability blocks are never decompressed.

```python
from pgen_polars import read_bgen, scan_bgen

features = read_bgen(
    "cohort.bgen",
    ref_allele="last",
    samples=["sample_42", "sample_7"],
    variants=["variant_1", "variant_2"],
)

lazy = scan_bgen(
    "cohort.bgen",
    ref_allele="last",
    samples=["sample_42", "sample_7"],
    rsids=["rs123", "rs456"],
)
```

Classic `.sample` files use `ID_2`; QCTOOL v2 files use `ID`. BGEN does not
label REF, so `ref_allele="first"` or `"last"` is mandatory. Multiallelic wide
output expands to one numeric feature per ALT. Embedded BGEN identifiers are
validated against classic `ID_2`, `ID_1`, or `ID_1_ID_2` conventions while
sample filtering and output continue to use `ID_2`.

Generate small 8-bit dosage, 16-bit dosage, and hard-call BGEN fixtures with:

```sh
tests/make_toy_bgen_data.sh /tmp/toy-bgen
```

Genotype readers (both unified for `.bed` and `.pgen`):

```python
import polars as pl
from pgen_polars import read_pgen, scan_pgen

# A PLINK 1 .bed is a valid .pgen (storage mode 0x01). Real PGEN hard calls
# and fractional unphased biallelic dosages are supported as well.
# Sidecars (.bim/.fam or .pvar/.psam) are auto-detected from the path.
# Wide is the default: one row per sample, one column per selected variant.
features = read_pgen(
    "tests/examples/toy.bed",
    samples=["per0", "per1"],
    variants=[0, 1],  # numeric indices are the bounded random-access fast path
    n_threads=4,  # None uses available parallelism; 1 is sequential
)

# Feed the numeric matrix directly to scikit-learn.
X = features.drop("ID").to_numpy()

# Long format remains available explicitly for row-oriented workflows.
long = read_pgen(
    "tests/examples/toy.bed",
    variants=[0, 1],
    output_format="long",
)

# Lazy wide scan is also the default. Projecting variant columns avoids
# decoding other selected variants.
lazy = scan_pgen(
    "tests/examples/toy.pgen",
    samples=["per0", "per1"],
    variants=["snp0", "snp1", "snp2"],
)
selected = (
    lazy.select("ID", "snp2")
    .collect()
)

# Long lazy output remains available for row/metadata predicates.
long = (
    scan_pgen(
        "tests/examples/toy.pgen",
        variants=["snp0", "snp1", "snp2"],
        output_format="long",
    )
    .filter(pl.col("chrom") == "1")
    .select("variant_id", "sample_id", "dosage")
    .collect()
)
```

`scan_pgen` chunks long output on the variant axis (keeping all selected
samples for a variant together) and wide output on the sample axis.
`batch_size` is a positive output-row hint, so a single long-format variant can
exceed it. Projection pushdown can produce long metadata-only batches without
decoding genotypes; in wide output it decodes only projected variant columns,
and an `ID`-only projection does not decode genotypes. Plain `head()`/slice
limits are pushed into native selection before decoding: wide scans retain only
the requested sample prefix, while long scans retain only variants intersecting
the requested row prefix and narrow samples when the prefix ends in the first
variant.

Simple predicates involving only variant metadata, long-format `sample_id`, or
wide `ID` values are used to preselect source indices and are not redundantly
reapplied to decoded batches. Other row-local
predicates, including genotype/dosage filters, are applied to each decoded batch.
Aggregate/window predicates fall back to materializing the scan before
filtering so their global semantics remain correct. Constructing a scan and
composing lazy expressions performs no filesystem I/O. Metadata is resolved
and snapshotted when Polars first requests the schema or collects the query;
dynamic wide schemas therefore resolve metadata on `collect_schema()`. Every
collection reuses that metadata snapshot but opens a fresh genotype iterator.
Create a new `LazyFrame` to observe sidecar changes.

Lazy scanning currently uses Polars' unstable IO-plugin interface and requires
Polars 1.31 or newer. Genotype files are read through immutable memory maps.
Wide real-PGEN scans decode each projected record once into a temporary,
column-major mapped file and then emit bounded sample chunks from that
transpose. The temporary file follows the operating system's standard temp
directory configuration and is deleted when the iterator is dropped. BED wide
scans decode directly from their packed mapped blocks.

`read_pgen` and `scan_pgen` accept `n_threads=None` to use available
parallelism; pass `n_threads=1` for sequential decoding. The CLI exposes the
same control as `pgen-rs pgen --threads N`. Reproducible performance comparisons
with `plink2` live in [`docs/benchmarks.md`](docs/benchmarks.md); regenerate
them with `uv run python benchmarks/run_benchmarks.py --generate`.
For the population-scale tiny-subset case, run
`uv run --extra benchmark python benchmarks/run_selected_benchmarks.py
/data/cohort.pgen --variants 0,35018818,70037636,105056454,140075271`.

Genotype DataFrames cross the native extension boundary through Arrow C Stream
capsules, avoiding per-cell Python objects. CLI TSV output uses bounded,
ordered parallel formatting for large tables and a direct buffered writer for
small tables; stdout itself remains serial to preserve deterministic output.
Wide hard calls are stored internally as one byte per genotype while retaining
the public nullable `Int64` Polars schema. CLI long output is decoded and
formatted in bounded chunks instead of materializing the full long table.

Selected eager wide reads use Rust Polars lazy CSV scans to project only
`IID`/`ID` plus a row index from `.psam`/`.pvar`. Numeric variant selections
limit the metadata prefix and load only the PGEN vblocks containing those
variants. ID selections apply a Polars predicate while scanning the unindexed
text sidecar. When indices and feature names are already known, pass an ordered
mapping such as `variants={"rs123": 42, "rs456": 9001}` to skip `.pvar`
access entirely. The Rust Polars path accepts the standard tab-delimited PLINK
2 sidecar representation; `.bim`/`.fam` retain variable-whitespace parsing.

Notes: only `POS` (int) and `CM`/`QUAL` (float) are typed; other columns are
strings. `.bim` alleles are exposed as `ALLELE1`/`ALLELE2` (no REF/ALT claim);
the genotype reader imputes `ref = ALLELE2`, `alt = ALLELE1` for `.bim` and uses
explicit `REF`/`ALT` for `.pvar`.

For dosage-bearing PGEN records, the `dosage` column contains the explicit
fractional ALT dosage. When a sample has no explicit dosage, it is inferred
from its hard call. The `genotype` column continues to report the saved hard
call, so it may be `./.` while `dosage` is non-null. In that case `is_missing`
is `false`, since usable dosage data is present. Hardcall- and dosage-phase
tracks are validated and discarded by the current unphased output schema.
Multiallelic dosage records are not yet supported.

