Metadata-Version: 2.4
Name: gffbase
Version: 0.2.0
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Rust
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Dist: duckdb>=1.4.1
Requires-Dist: pyarrow>=18.1
Requires-Dist: pandas>=1.5 ; extra == 'all'
Requires-Dist: polars>=0.20 ; extra == 'all'
Requires-Dist: pyfaidx>=0.7 ; extra == 'all'
Requires-Dist: pybedtools>=0.9 ; sys_platform != 'win32' and extra == 'all'
Requires-Dist: biopython>=1.80 ; extra == 'all'
Requires-Dist: matplotlib>=3.5 ; extra == 'all'
Requires-Dist: psutil>=5 ; extra == 'bench'
Requires-Dist: memory-profiler>=0.61 ; extra == 'bench'
Requires-Dist: gffutils>=0.13 ; extra == 'bench'
Requires-Dist: biopython>=1.80 ; extra == 'biopython'
Requires-Dist: pytest>=7 ; extra == 'dev'
Requires-Dist: pytest-cov>=4 ; extra == 'dev'
Requires-Dist: hypothesis>=6 ; extra == 'dev'
Requires-Dist: packaging>=24 ; extra == 'dev'
Requires-Dist: psutil>=5 ; extra == 'dev'
Requires-Dist: ruff>=0.4 ; extra == 'dev'
Requires-Dist: mypy>=1.8 ; extra == 'dev'
Requires-Dist: maturin>=1.5,<2.0 ; extra == 'dev'
Requires-Dist: sphinx==9.1.0 ; extra == 'docs'
Requires-Dist: furo==2025.12.19 ; extra == 'docs'
Requires-Dist: sphinx-copybutton==0.5.2 ; extra == 'docs'
Requires-Dist: sphinx-design==0.7.0 ; extra == 'docs'
Requires-Dist: pyfaidx>=0.7 ; extra == 'fasta'
Requires-Dist: pandas>=1.5 ; extra == 'pandas'
Requires-Dist: pybedtools>=0.9 ; sys_platform != 'win32' and extra == 'plot'
Requires-Dist: matplotlib>=3.5 ; extra == 'plot'
Requires-Dist: polars>=0.20 ; extra == 'polars'
Requires-Dist: pybedtools>=0.9 ; sys_platform != 'win32' and extra == 'pybedtools'
Requires-Dist: pytest>=7 ; extra == 'test'
Requires-Dist: pytest-cov>=4 ; extra == 'test'
Requires-Dist: hypothesis>=6 ; extra == 'test'
Requires-Dist: packaging>=24 ; extra == 'test'
Requires-Dist: pyyaml>=6 ; extra == 'test'
Requires-Dist: psutil>=5 ; extra == 'test'
Requires-Dist: tomli>=2 ; python_full_version < '3.11' and extra == 'test'
Provides-Extra: all
Provides-Extra: bench
Provides-Extra: biopython
Provides-Extra: dev
Provides-Extra: docs
Provides-Extra: fasta
Provides-Extra: pandas
Provides-Extra: plot
Provides-Extra: polars
Provides-Extra: pybedtools
Provides-Extra: test
License-File: LICENSE
Summary: GFFBase — Rust-accelerated GFF3/GTF parser with a DuckDB-backed storage engine and a drop-in gffutils-compatible Python API.
Keywords: gff,gff3,gtf,gencode,bioinformatics,genomics,annotation,duckdb,rust,pyo3,parser,feature-database
Home-Page: https://github.com/Kuanhao-Chao/gffbase
Author-email: Kuan-Hao Chao <kuanhao.chao@gmail.com>
License: Apache-2.0
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Changelog, https://github.com/Kuanhao-Chao/gffbase/releases
Project-URL: Documentation, https://khchao.com/gffbase/
Project-URL: Homepage, https://github.com/Kuanhao-Chao/gffbase
Project-URL: Issues, https://github.com/Kuanhao-Chao/gffbase/issues
Project-URL: Source, https://github.com/Kuanhao-Chao/gffbase

<p align="center">
  <img src="docs/source/_static/logo.svg#gh-light-mode-only" alt="gffbase" width="62%">
  <img src="docs/source/_static/logo-white.svg#gh-dark-mode-only" alt="gffbase" width="62%">
</p>


[![PyPI version](https://img.shields.io/pypi/v/gffbase.svg)](https://pypi.org/project/gffbase/)
[![PyPI downloads](https://img.shields.io/pypi/dm/gffbase.svg)](https://pypi.org/project/gffbase/)
[![Python versions](https://img.shields.io/pypi/pyversions/gffbase.svg)](https://pypi.org/project/gffbase/)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
[![Docs](https://img.shields.io/badge/docs-khchao.com%2Fgffbase-blue.svg)](https://khchao.com/gffbase/)
[![CI](https://github.com/Kuanhao-Chao/gffbase/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/Kuanhao-Chao/gffbase/actions/workflows/ci.yml)

---

## What is GFFBase?

**GFFBase is a high-performance genomic-annotation engine combining a
SIMD Rust parser, a DuckDB columnar backend, and a zero-copy PyArrow
interface — purpose-built for whole-genome-scale ingest and bulk
machine-learning feature extraction, while remaining a drop-in
successor to [`gffutils`](https://github.com/daler/gffutils).**

A SIMD Rust+PyO3 parser feeds DuckDB's columnar storage through
record-batch Arrow handoffs. A smart query router auto-picks an
R-tree or B-tree spatial index per query, and a closure-cache /
recursive-CTE relational dispatcher selects the right strategy based
on the corpus's actual hierarchy depth. The full `FeatureDB` /
`Feature` / `create_db` / `DataIterator` / `GFFWriter` /
`merge_criteria` legacy API is preserved verbatim — most users
migrate by changing one import line.

### Three reasons it matters

1. **Whole-genome ingest, measured — and reported as it came out.** Five large
   annotations, one run, one commit, with the complete commands and environment
   recorded. Ingest lands between 1.21× and 0.69× against `gffutils`: ahead
   where per-feature overhead dominates, behind on the attribute-dense
   whole-genome files. Nothing is published unless both engines' correctness
   signatures agree first. *([The numbers](#-measured-against-legacy-gffutils))*
2. **Bulk extraction without Python objects.**
   `children_batched(format='arrow')` answers "every exon for these tens of
   thousands of transcripts" with a single set-based query returning a
   `pyarrow.Table` that shares memory with DuckDB. No `Feature` object is
   constructed at any layer. *([How](#-the-killer-feature--zero-copy-pyarrow-for-ml-pipelines))*
3. **Validated, not just counted.** Structural checks cover feature identity,
   normalized attributes, multipart envelopes, direct edges, transitive
   closure, indexes, and synthetic hierarchy coherence. Real-corpus runs are a
   separate release gate from the quick unit suite.

---

## 📦 Installation

```bash
pip install gffbase
```

> **0.2.0 is the current release** (2026-09-11). It fixes two SQL injection
> vulnerabilities in 0.1.0 — see the [security advisory](https://github.com/Kuanhao-Chao/gffbase/security/advisories/GHSA-5f5g-g3v5-prrg) —
> and includes breaking changes, listed in the changelog.

Universal `abi3-py310` wheels — single binary per arch covers CPython
3.10 → 3.14. No Rust toolchain required at install time.

For source/dev installs (Rust >= 1.83 + maturin):

```bash
pip install -e .[dev]
maturin develop --release
```

---

## 🏃 Quick start — row-by-row (drop-in for `gffutils`)

<!-- docs-test: skip reason="needs the GENCODE v49 corpus" -->
```python
from gffbase import create_db

# 1. Ingest a GTF/GFF3 in seconds (auto-detects format, gzipped OK).
db = create_db("gencode.v49.chr_patch_hapl_scaff.basic.annotation.gtf.gz",
               "gencode.duckdb", force=True)

# 2. Walk a single gene's hierarchy.
for tx in db.children("ENSG00000139618", level=1, featuretype="transcript"):
    print(tx.id, tx.start, tx.end)

# 3. Spatial overlap query — uses the per-seqid R-tree under the hood.
for f in db.region("chr17:43044295-43125483", featuretype="exon"):
    print(f)
```

If you're migrating from `gffutils`, change one line:

<!-- docs-test: skip reason="illustrative: contains an elided fragment" -->
```python
import gffbase as gffutils    # one-line alias migration
db = gffutils.create_db(...)  # everything else identical
```

(But please read the [Migration Guide](https://khchao.com/gffbase/content/migration.html) first — it has
**one** important note about ML loops.)

---

## 🤖 Quick start — vectorized for ML

<!-- docs-test: skip reason="illustrative: names gencode.duckdb, which the reader supplies" -->
```python
from gffbase import FeatureDB

db = FeatureDB("gencode.duckdb")

# Pull every exon for 50 000 transcripts — one set-based SQL query.
exons = db.children_batched(
    transcript_ids,                # iterable of 50 000 IDs
    featuretype="exon",
    format="arrow",                # "df" / "polars" also supported
)
# exons is a pyarrow.Table sharing memory with DuckDB. No copies.

# Spatial: "for each ATAC-seq peak, find every overlapping CDS."
peaks = [("chr1", 100_000, 110_000), ("chr1", 200_000, 210_000), ...]
overlaps = db.region_batched(peaks, featuretype="CDS", format="arrow")
```

See the [Machine Learning Workflows
Cookbook](https://khchao.com/gffbase/content/cookbook_ml_workflows.html) for end-to-end
pipelines with PyTorch and Hugging Face `datasets`.

---

## ⚡ Measured against legacy `gffutils`

One run, one machine, one commit, over all five canonical inputs. **Read the
ingest column as a draw rather than a win**: gffbase spans 1.21× to 0.69×, ahead
where per-feature overhead dominates and behind on the attribute-dense
whole-genome files, because both engines are attribute-bound and effectively
serial at a comparable rate. The durable advantages — batched extraction,
spatial indexing, SQL over the whole corpus — are elsewhere and are untouched by
that result.

<!-- BEGIN GENERATED: corpus-table -->
| Corpus | Format | Lines | gffbase ingest | legacy ingest | speedup | peak RSS (ingest + full validation) | spatial qps | batched (5 k anchors) |
| --- | :--: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| **GENCODE v49** (basic) | GTF | 6,068,892 | **10 min 14 s** | 7 min 1 s | **0.69×** | 53.57 GB | **707** ±0% (n=5) | 963 ms / 1.93 M desc |
| **GENCODE v49** (basic) | GFF3 | 6,066,054 | **11 min 8 s** | 9 min 59 s | **0.90×** | 62.00 GB | **705** ±0% (n=5) | 1096 ms / 1.93 M desc |
| **RefSeq GRCh38.p14** | GFF3 | 4,932,571 | **7 min 2 s** | 6 min 34 s | **0.93×** | 26.80 GB | **540** ±0% (n=5) | 588 ms / 999 k desc |
| **CHESS 3.1.3** | GFF3 | 2,761,061 | **1 min 53 s** | 2 min 17 s | **1.21×** | 3.38 GB | **702** ±0% (n=5) | 202 ms / 161 k desc |
| **MANE v1.5** (Ensembl) | GFF3 | 524,834 | **40.0 s** | 45.7 s | **1.14×** | 3.98 GB | **840** ±0% (n=5) | 206 ms / 156 k desc |
<!-- END GENERATED: corpus-table -->

<!-- BEGIN GENERATED: benchmark-provenance -->
**Measured on** AMD EPYC 7702 64-Core Processor · 128 cores · 1007.22 GB RAM · Linux-5.14.0-503.15.1.el9_5.x86_64-x86_64-with-glibc2.34  
**Versions:** Python 3.11.16 · gffbase 0.2.0rc1 · duckdb 1.5.5 · pyarrow 25.0.1 · gffutils 0.14  
**Commit:** `42bb900e328c` · **Run:** 2026-09-10T19:56:52Z  
*Generated from `benchmarks/results/06_mega.linux-x86_64.json` by `tools/gen_benchmark_tables.py`. Do not edit by hand.*
<!-- END GENERATED: benchmark-provenance -->

All five corpora completed; nothing is censored. Had a comparator been killed at
its safety valve the cell would read “censored at”, which is cap evidence only —
never a wall time or a speedup.

**`peak RSS` is ingest plus exhaustive validation**, not the cost of ingest and
not what you pay: `validate_db` defaults to `sample=200` and the CLI never
overrides it. The same five corpora measured at `validation_sample=10000` peak
at 1.3–10.2 GB instead of 3.3–62.2 GB — GENCODE GFF3 alone falls from 62.2 GB
to 9.5–10.2 GB.

**The GTF row is the inference-disabled arm**, which is the configuration least
favourable to gffbase. GENCODE v49's GTF already contains gene and transcript
rows, so this is the arm where `gffutils` does no synthesis at all and its
ingest becomes a plain bulk insert — 229 k attributes/s, its fastest result on
any corpus, against gffbase's entirely ordinary 157 k, which is gffbase's own
third-fastest and within 7 % of its best. The cluster campaign reports default
inference, inference-disabled real data, and parent-stripped synthesis as three
independent arms; only the second is published here.

**Ingest wall time is measured once per corpus**, while query phases repeat five
times. A separate probe put the run-to-run spread at 1.3 % on RefSeq and 2.7 %
on GENCODE GTF, but ~20 % on CHESS, where a 90-second run is dominated by
startup. Treat the whole-genome ratios as solid and the CHESS ratio as
indicative.

**Robustness.** Publication now requires matching deterministic feature and
relationship signatures plus a full structural-validation pass. Parser
failures remain line-numbered `GFFFormatError` instances.

**Three of the five use the split-CDS convention** — one CDS spread over
several lines that share an `ID` — namely RefSeq, MANE, and GENCODE's GFF3
edition. That is not an edge case, so gffbase does not pick a reading for you:
`merge_strategy` defaults to `"error"`, exactly as in `gffutils`, and you say
which you want. `merge_strategy="create_unique"` renames the rows as `gffutils`
would, so a ported script sees what it expects; `mode="strict"` fuses them into
one discontinuous feature backed by the `segments` table, which is what the
GFF3 specification actually describes.

📊 Method, fairness constraints and re-run instructions:
**[Methodology](https://khchao.com/gffbase/content/methodology.html)**. Every
number is generated from a committed measurement file by
`tools/gen_benchmark_tables.py`, and a test fails if a published table stops
matching it.

---

## 🚀 The Killer Feature — zero-copy PyArrow for ML pipelines

Modern ML genomics pipelines have one shape: **pull every exon for
50 000 transcripts, push the column-oriented table into a tensor,
train.** Legacy `gffutils` forces a per-feature Python loop —
constructing 1.6 M throwaway `Feature` objects per pull, which crushes
both wall time and memory. gffbase bypasses Python entirely with a
single batched call that returns DuckDB's internal Arrow buffers
directly:

<!-- docs-test: skip reason="illustrative: an id list the reader supplies" -->
```python
# 50 000 transcript IDs → every exon, in one query.
# Returns a zero-copy pyarrow.Table — no Python `Feature` object
# is constructed at any layer.
exons = db.children_batched(
    transcript_ids,
    featuretype="exon",
    format="arrow",        # or "df" / "polars"
)

# Hand off directly to PyTorch / Hugging Face datasets / JAX / Lance.
import torch
starts = torch.from_numpy(exons.column("start").to_numpy())
ends   = torch.from_numpy(exons.column("end").to_numpy())
# The "anchor" column carries the input id for each row, so you can
# reconstruct per-transcript groups without re-issuing N queries.
```

**Why it is faster, and by how much.** The batched call issues one set-based
SQL query and hands back DuckDB's own Arrow buffers. The row-by-row
alternative constructs one Python `Feature` per result row, and at 1.6 M exons
that allocation dominates everything else — in *both* libraries. Per-corpus
batched throughput is in the
[table above](#-measured-against-legacy-gffutils); the
[Performance page](https://khchao.com/gffbase/content/performance.html) breaks it down.

Note the trade honestly: iterating `for x in ids: db.children(x)` in gffbase is
**slower** than legacy's SQLite row-by-row path, because DuckDB pays
vectorization startup per call. That is why the batched API exists, and why the
[Migration guide](https://khchao.com/gffbase/content/migration.html) puts it front and centre.

`region_batched(...)` and `parents_batched(...)` have the same
zero-copy contract for spatial and parent workloads.

---

## ✨ What's inside

- **Rust + PyO3 parser** — SIMD line/tab splitting, lazy URL-decoding
  *and* percent-encoding on the way back out, GTF semicolon-in-quotes safe,
  gzipped input transparent. The formal profile enforces the nine-column GFF3
  contract with line-numbered `GFFFormatError`; the compatibility profile
  preserves accepted legacy inputs with diagnostics.
- **DuckDB columnar storage** — 11-table schema (plus 3 compatibility
  views), set-based GTF
  gene/transcript synthesis, recursive-CTE transitive closure,
  per-seqid-banded R-tree spatial index built inline during ingest.
- **Smart routing** — `region()` auto-picks R-tree vs B-tree;
  `children()` auto-picks closure cache vs dynamic CTE based on
  measured corpus depth.
- **Vectorized batched API** — `children_batched`, `parents_batched`,
  `region_batched` return `pyarrow.Table` / `pandas.DataFrame` /
  `polars.DataFrame` directly out of DuckDB's buffer pool.
- **Drop-in legacy API** — `FeatureDB`, `Feature`, `create_db`,
  `DataIterator`, `GFFWriter`, `merge_criteria`, `interfeatures`,
  `bed12`, `execute()` SQL escape hatch, `export_sqlite()`. 97% of the
  gffutils 0.14 symbol surface, with every remaining difference declared and
  tested.
- **Discontinuous features** — `MultipartFeature` / `FeatureSegment`, per
  segment phase, `covered_length`, and `explode_segments=` on the batched
  APIs. Several lines sharing one `ID` are one logical feature.
- **A command line** — `gffbase create|fetch|children|parents|region|search|
  rmdups|sanitize|validate|migrate`. See [the CLI reference](https://khchao.com/gffbase/content/cli.html).
- **Post-ingest validation** — `gffbase.validate` checks fast and full
  structural invariant sets, and
  `gffbase.migrate` upgrades a v1 database in place.
- **abi3 wheels** — single binary per arch covers CPython 3.10-3.14.

---

## 📚 Documentation

Full site, built with Sphinx and the furo theme:
**[https://khchao.com/gffbase/](https://khchao.com/gffbase/)**

| Page                                                                                       | What's there                                                              |
| ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------- |
| [Usage Gallery](https://khchao.com/gffbase/content/usage_gallery.html)                     | Copy-pasteable snippets for every public API method                       |
| [Performance comparison](https://khchao.com/gffbase/content/performance.html)              | Head-to-head numbers across five canonical human-genome annotations + per-corpus root-cause analysis |
| [Migration guide for `gffutils` users](https://khchao.com/gffbase/content/migration.html)  | Drop-in compat checklist + the one OLAP/OLTP gotcha you must understand   |
| [Cookbooks](https://khchao.com/gffbase/content/cookbooks.html)                             | GENCODE/Ensembl, RefSeq, MANE, ML workflows                               |
| [API reference](https://khchao.com/gffbase/content/api.html)                               | Every public method, full signatures + docstrings                         |

To build the docs locally:

```bash
pip install -e .[docs]       # needs Python >= 3.12; Sphinx 9 requires it
make -C docs html            # -> docs/build/html/index.html
```

The build runs with `-W --keep-going`, so a broken cross-reference or a page
missing from the toctree is an error, not a quietly degraded page. That is the
same command CI runs.

---

## 🧪 Testing

```bash
pip install -e '.[test,all]'
pytest
pytest --cov=gffbase --cov-report=term     # coverage report
```

The suite passes in full with the compiled extension built and the DuckDB
spatial extension available. Without them, the Rust-engine and R-tree cells
skip rather than fail — CI runs dedicated jobs where a missing capability is
an error, so those paths cannot silently go unexercised.

Three things are checked that most suites do not:

- **Differential parity** against a git-pinned `gffutils` build, with every
  deliberate difference declared in a register that fails both on an
  undeclared gap and on a declaration that has gone stale.
- **The documentation's own code.** Every runnable snippet in `docs/`,
  `README.md` and `MIGRATION.md` is executed against the vendored fixtures, so
  a documented example cannot rot unnoticed.
- **Release invariants** — version agreement across every file that states one,
  packaging manifests resolving to real files, workflow YAML that GitHub will
  actually load, and every published benchmark table matching its committed
  measurements.

Add `-m corpus`, after `python benchmarks/download_corpora.py`, to run the
same operations against the real human-genome annotations.

CI runs the full matrix on Linux + macOS + Windows, both R-tree and
B-tree fallback paths, on Python 3.10 / 3.12 / 3.14.

---

## 🤝 Contributing

GFFBase welcomes pull requests, bug reports, and feature suggestions.
Start with [`CONTRIBUTING.md`](CONTRIBUTING.md) for the full guide:

- Rust + Python development setup (`maturin develop --release`)
- Running the test suite + the coverage gate (95 % R-tree / 94 % B-tree)
- Branch naming, Conventional Commits, the PR checklist

The repo ships standard
[issue templates](.github/ISSUE_TEMPLATE/) and a
[PR template](.github/PULL_REQUEST_TEMPLATE.md) so new
contributions land with the context maintainers need to triage them
quickly.

---

## 🪪 License

Apache License 2.0. See [`LICENSE`](LICENSE).

---

## 📖 Citation

If GFFBase contributes to your research, please cite it:

```bibtex
@software{chao_gffbase_2026,
  author  = {Chao, Kuan-Hao},
  title   = {{GFFBase}: Rust-accelerated GFF3/GTF parser with a
             DuckDB-backed storage engine and zero-copy PyArrow interface},
  year    = 2026,
  version = {0.2.0},
  url     = {https://github.com/Kuanhao-Chao/gffbase},
}
```

The repository also ships a [`CITATION.cff`](CITATION.cff), so GitHub's
"Cite this repository" button produces an up-to-date reference. Per-version
DOIs are tracked on the
[Releases page](https://github.com/Kuanhao-Chao/gffbase/releases).

---

If GFFBase saves you a benchmark cycle or a pipeline rewrite, a ⭐ on
[GitHub](https://github.com/Kuanhao-Chao/gffbase) is the easiest way
to say so — and helps other genomics teams find the project.

