Metadata-Version: 2.4
Name: polars-pdb
Version: 0.1.0
Summary: A locally-mirrored, SQL-queryable catalogue of the PDB, sourced from PDBe, that advances one wwPDB release per week
Keywords: pdb,pdbe,wwpdb,mmcif,polars,parquet,duckdb,structural-biology,bioinformatics,sifts
Author: Chris Thorpe
Author-email: Chris Thorpe <33100149+drchristhorpe@users.noreply.github.com>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Typing :: Typed
Requires-Dist: polars>=1.42
Requires-Dist: duckdb>=1.5
Requires-Dist: gemmi>=0.7
Requires-Dist: click>=8.1
Requires-Dist: rich>=13.0
Requires-Python: >=3.14
Project-URL: Homepage, https://github.com/drchristhorpe/polars_pdb
Project-URL: Repository, https://github.com/drchristhorpe/polars_pdb
Project-URL: Issues, https://github.com/drchristhorpe/polars_pdb/issues
Description-Content-Type: text/markdown

# polars_pdb

A **locally-mirrored, SQL-queryable catalogue of the PDB**, sourced from PDBe,
that advances one wwPDB release per week.

```
polars-pdb update
```

is the whole idea: run it weekly (a cron job, ignored otherwise) and the
local archive tracks upstream, one release at a time. Everything else in this
project exists in service of that one command.

**Status: alpha.** The schema and CLI are still settling, and the mirror is a
real commitment -- see [Setup](#setup) before you start.

## Two layers, deliberately separate

| Layer | Content | Role |
|---|---|---|
| **Mirror** | `.cif.gz`, byte-identical to PDBe | Source of truth. Full coordinates. Not read at query time. |
| **Catalogue** | Parquet: metadata, chains, sequences, cross-references | The queryable index. Derived. Rebuildable. **No coordinates.** |

The workflow is always the same direction:

> Query the catalogue in SQL or Polars → get back `pdb_id`s and filesystem
> paths → open those CIF files yourself.

**No coordinates ever live in Parquet, at any scale.** The coordinates are
already sitting on disk in the mirror, in a format every structural biology
tool already reads. Duplicating them into a second, bespoke representation
would be the [MMTF](https://en.wikipedia.org/wiki/MMTF) mistake all over
again -- a clever re-modelling nobody but this project owns, and nobody but
this project can fix when it goes stale. Dropping them is also what keeps the
catalogue two orders of magnitude smaller than the mirror: a coordinate table
at this archive's scale would be on the order of 2.6 billion rows and ~28 GB;
the actual catalogue is under 1 GB.

So, explicitly, this library does **not**:

- load coordinates, in any form -- there is no atom table and never will be
- compute derived geometry -- no contacts, no SASA, no centre-of-mass, no
  neighbour search
- talk to the PDBe REST API -- it works from bulk flat files; 256k round-trips
  is not an ingest strategy
- know anything about assemblies beyond their *definition* -- an assembly is a
  recipe (symmetry operators, stored as text), never materialised coordinates

It has one job -- turn "what's in the PDB" into a fast SQL/Polars question --
and is judged on that job alone.

## The numbers

All measured against a real sync, not estimated:

| | |
|---|---|
| Entries | 256,448 |
| Mirror size | 89.79 GB (`.cif.gz`, byte-identical to PDBe) |
| Catalogue size | well under 1 GB (the entire `xref_*` cross-reference family is ~50 MB) |
| Largest table | `xref_go` -- 16,281,793 rows, ~27 MB, point query ~24 ms in DuckDB |
| Full rebuild | ~2.2 minutes across 16 cores |
| Typical weekly update | seconds (~300 changed entries) |

## Setup

```bash
pip install polars-pdb   # or: uv add polars-pdb
export POLARS_PDB_ROOT=/data/pdb   # wherever you can spare ~90 GB
polars-pdb update
```

`$POLARS_PDB_ROOT` is required, not defaulted -- the mirror alone is a ~90 GB
download, and this project would rather fail loudly than pick a location for
that on your behalf. The first `update` on an empty root *is* a full rebuild
(same code path, no special case), so budget the time and the disk before
running it.

`polars-pdb status` reports the latest wwPDB release against what your local
archive is synced to (entry counts included), rendered as a Rich table.

## Querying

Two ways in, same catalogue underneath.

**Polars**, lazy over the Parquet:

```python
import polars as pl
import polars_pdb as ppdb

cat = ppdb.catalogue()                    # lazy Polars frames over catalogue/

hits = (
    cat.entries
       .filter(pl.col("method") == "X-RAY DIFFRACTION")
       .filter(pl.col("resolution") < 2.0)
       .join(
           cat.xref_uniprot.filter(pl.col("uniprot") == "P42212"),
           on="pdb_id",
       )
       .collect()
)

paths = ppdb.paths(hits["pdb_id"])        # -> filesystem paths to .cif.gz
```

**SQL**, via DuckDB directly over the Parquet:

```bash
polars-pdb query "
    SELECT DISTINCT e.pdb_id, e.resolution, x.uniprot
    FROM entries e JOIN xref_uniprot x USING (pdb_id)
    WHERE e.resolution < 2.0 AND x.uniprot = 'P42212'
"
```

or the same query from Python via `polars_pdb.duck.query(sql)`, which returns
a Polars `DataFrame`.

Note the `DISTINCT`: `xref_uniprot` is **per chain segment** (a chain can map
to more than one UniProt range), so one entry can yield multiple rows. That's
the source's real grain being honest at query time, not a bug -- see
"Cross-references" below.

`ppdb.paths()` **is the library's boundary.** It returns paths and nothing
else. What you do with the coordinates -- gemmi, Biopython, whatever you
already use -- is yours.

## The catalogue

14 tables, one Parquet file each, all under `catalogue/`:

`entries` · `entities` · `chains` · `sequences` · `ligands` · `citations` ·
`assemblies` · `xref_uniprot` · `xref_pfam` · `xref_cath` · `xref_scop` ·
`xref_go` · `xref_ec` · `xref_taxonomy`

### Cross-references (`xref_*`)

There is deliberately no single `xrefs` table. `pdb_chain_uniprot` is one row
per chain *segment*; `pdb_chain_go` is one row per chain/GO *term* -- a single
(entry, chain) pair can have hundreds of GO rows and dozens of Pfam rows.
Flattening those into one row per chain would mean fabricating a cross
product the source data doesn't contain. Each SIFTS flatfile gets its own
table, at its own native grain, 1:1 with the source; joins across them happen
explicitly at query time, which is a feature, not friction.

## CLI

```
polars-pdb update            # advance the archive by one wwPDB release
polars-pdb update --dry-run  # report what would change; write nothing
polars-pdb rebuild           # update from an empty state -- same code path
polars-pdb status            # release, drift, entry counts
polars-pdb query "SELECT …"  # ad-hoc DuckDB, Rich-rendered
polars-pdb path 1cbs         # id -> filesystem path
```

`polars-pdb update` is designed to be the thing a weekly cron calls and
otherwise ignores.

## Why this exists

The niche is genuinely open -- there is no substantive PDB-as-Parquet or
PDB-as-DuckDB project on PyPI. The closest comparable, PDBj Mine 2, needs a
~50 GB PostgreSQL server. This is a library and a Parquet directory: no
server, no daemon, `pip install` and a filesystem path.

The design leans on a few hard-won lessons:

- **The update is the primary operation**, not an afterthought. The PDB is a
  living archive; a one-shot dump is stale in seven days. A full rebuild is
  just an update from an empty state -- one code path, exercised every week,
  so it can't rot.
- **Schema mirrors the mmCIF dictionary**, not a bespoke re-modelling.
  MMTF died in 2024 and took its dependent tooling with it; BinaryCIF, which
  stayed close to the mmCIF categories, survived.
- **The mirror is truth; the catalogue is a projection.** Deleting the whole
  catalogue and regenerating it is always safe, because no fact exists only
  in Parquet.

See `PLAN.md` for the full architecture and the measurements behind it, and
`BUILD.md` for how it was actually built.
