Metadata-Version: 2.4
Name: geniekit
Version: 0.1.0
Summary: Harmonise the AACR Project GENIE BPC cohorts, compute lines of therapy, and run aggregate-only analyses.
Author-email: Muntaser Syed <jemsbhai@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/jemsbhai/geniekit
Project-URL: Issues, https://github.com/jemsbhai/geniekit/issues
Project-URL: AACR Project GENIE, https://aacrprojectgenie.org/
Keywords: AACR,GENIE,BPC,oncology,clinicogenomic,line of therapy,harmonization,Synapse
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE.md
Requires-Dist: pandas<4,>=2.2
Requires-Dist: numpy<3,>=1.26
Requires-Dist: requests<3,>=2.32
Requires-Dist: pyyaml<7,>=6.0
Provides-Extra: synapse
Requires-Dist: synapseclient<5,>=4.4; extra == "synapse"
Provides-Extra: survival
Requires-Dist: lifelines<1,>=0.29; extra == "survival"
Provides-Extra: notebook
Requires-Dist: jupyter<2,>=1.1; extra == "notebook"
Requires-Dist: matplotlib<4,>=3.9; extra == "notebook"
Provides-Extra: dev
Requires-Dist: pytest<9,>=8.3; extra == "dev"
Requires-Dist: pytest-cov<8,>=6.0; extra == "dev"
Requires-Dist: ruff<1,>=0.9; extra == "dev"
Requires-Dist: openpyxl<4,>=3.1; extra == "dev"
Requires-Dist: pdfplumber<1,>=0.11; extra == "dev"
Requires-Dist: build<2,>=1.2; extra == "dev"
Requires-Dist: twine<7,>=6.0; extra == "dev"
Dynamic: license-file

# geniekit

Python tools for the AACR Project GENIE Biopharma Collaborative (BPC)
clinicogenomic cohorts.

- **Harmonise** the six public BPC cohorts (NSCLC, CRC, BrCa, PANC, Prostate,
  BLADDER) to one schema so that cross-cohort analyses do not have to deal
  with per-cohort variable names, types and code lists.
- **Lines of therapy**: derive first and subsequent lines from the regimen
  data relative to an index event that the user chooses (diagnosis,
  metastatic diagnosis, first progression, first systemic therapy, or any
  custom event), with explicit, configurable rules.
- **Aggregate-only analytic layer**: produce cohort-level summaries with
  disclosure control (minimum cell sizes, suppression, no row-level output)
  for users who may not access patient-level records.

geniekit ships metadata only. The data themselves come from Synapse under the
GENIE access requirement (see [Data access](#data-access)).

## Status

Version 0.1.0 covers the three challenge capabilities (harmonisation, lines of
therapy, aggregate-only analytic layer) on top of the foundation (release
registry, data dictionary, download and loading, synthetic cohorts). Every
layer has been run against the six public releases; the test suite runs on
synthetic data and a simulated fixture, and repeats the real-data checks when
`GENIEKIT_DATA_DIR` points at downloaded releases.

| Module | Purpose | State |
|---|---|---|
| `geniekit.releases` | Registry of the six public BPC releases with the Synapse id of every clinical, genomic, cBioPortal and documentation file | done |
| `geniekit.dictionary` | The six Variable Synopsis workbooks compiled into one data dictionary, with cross-cohort comparison helpers | done |
| `geniekit.io` | Download from Synapse (personal access token, access-status check, MD5 verification, retries) and load release folders from disk into one `CohortData` object, checked against the dictionary | done |
| `geniekit.synthetic` | Dictionary-driven synthetic cohorts: every declared column, cohort-specific naming, coherent simulated histories | done |
| `geniekit.harmonize` | Crosswalk-driven normalisation of the six cohorts (renames, code-list collapses, spelling, types, derived variables, genomic format unification) with a provenance report; stacking across cohorts | done; metastasis-site rederivation follows |
| `geniekit.lot` | Lines of therapy from the regimen tables, counted from an index event the user chooses, with explicit rules, a drug vocabulary covering both schema generations, and a reason recorded for every assignment | done |
| `geniekit.aggregate` | Aggregate-only analytic layer: disclosure control (minimum cell size, complementary suppression or omission of small categories, suppression of derived statistics), summary tables, Kaplan-Meier and early-failure endpoints, genomic flags, and the CRC liver-metastasis report of Challenge 10 | done |

## Installation

```powershell
pip install geniekit
```

Optional extras: `geniekit[synapse]` (official Synapse client), `geniekit[survival]`
(Kaplan-Meier summaries), `geniekit[notebook]` (Jupyter and plotting),
`geniekit[dev]` (tests and linting).

From a clone:

```powershell
pip install -e ".[dev]"
pytest tests/ -v --cov=geniekit --cov-report=term
```

## Quick start

```python
from geniekit import releases, dictionary

# What is available, and where on Synapse
for r in releases.list_releases():
    print(r.label, r.n_patients, r.release_folder, sorted(r.clinical))

# How the cohorts differ before any harmonisation
dd = dictionary.load()
print(dd.compare("ca_cigarette", table="cancer").summary())
print(dd.differences(table="patient").head())
```

Loading data (after approval, see [Data access](#data-access)):

```python
from geniekit import io

# download one release with a personal access token in SYNAPSE_AUTH_TOKEN
folder = io.download_release("CRC", dest="data/genie_bpc")       # data/genie_bpc/CRC_v2.0-public
crc = io.load_release(folder)                                     # CohortData with 12 tables
crc["regimen"].head()

# or point at a folder you downloaded through the Synapse website
nsclc = io.load_release("downloads/NSCLC_2.0-public")
```

Working without the data:

```python
from geniekit import synthetic

fake = synthetic.generate("BLADDER", n_patients=300, seed=0)     # same tables and columns as the real release
fake["cancer_index"][["record_id", "stage_dx", "dist_mets_liver", "tt_os_dx_days"]].head()
```

Synthetic cohorts carry the exact column set of the corresponding release
(checked against the dictionary in the test suite) and mutually consistent
identifiers, intervals and outcomes, but their values are invented. They
exist so that code can be written and tested before access is granted; never
report numbers computed from them.

Harmonising and stacking:

```python
from geniekit import io, harmonize

nsclc = harmonize.harmonize(io.load_release("data/genie_bpc/NSCLC_v2.0-public"))
nsclc.report.to_frame()            # every rename, recode, collapse, coercion and derivation, with counts
nsclc.report.unmapped()            # values the crosswalk did not recognise (should be empty)

stacked = harmonize.stack([nsclc, bladder, crc])   # one frame per table, cohort and schema_generation columns
stacked["cancer_index"]["ca_cigarette"]           # generation-1 ca_lung_cigarette and generation-2 ca_cigarette, unified
stacked.availability("cancer_index")               # which cohorts contributed each column
```

What the crosswalk does (`geniekit crosswalk --basis` prints the full list with
the basis for each rule in the data guides):

- renames generation-1 variables to the generation-2 name where the
  definitions coincide (`ca_lung_cigarette`, `dmets_stage_i_iii`, and the
  liver, breast and brain/CNS metastasis flags with their time intervals);
- collapses the detailed generation-1 NAACCR ethnicity, race and sex code
  lists onto the generation-2 categories, keeping the detailed value in a
  `_detail` column;
- reconciles cohort-specific level sets (`ca_heme_type`, `pfs_cohort`,
  `bca_subtype`, tumor-deposit text) and spelling differences (dashes,
  capitalisation, `<1` versus `< 1`);
- converts `cpt_seq_date` to an integer year and `naaccr_seer_sum_stage` to
  the NAACCR #759 code plus label in every cohort;
- derives `race_ethnicity` for every cohort with the rule behind the released
  generation-1 variable, recovered from the NSCLC and CRC releases where it
  reproduces every released value (the released value is kept in
  `race_ethnicity_released`; a conservative alternative is selectable in the
  crosswalk), and `pfs_time_origin`, which states whether the PFS endpoints
  run from diagnosis or from the first distant metastasis;
- unifies `data_fusions.txt` and `data_sv.txt` into one structural-variant
  schema and melts the copy-number matrix into a long table.

What it deliberately does not do: the remaining generation-1 metastasis
regions (`dmets_abdomen`, `dmets_bone`, `dmets_extremity`,
`dmets_head_neck`, `dmets_pelvis`, `dmets_thorax`) keep their names, because
a code-by-code comparison of the classification appendices
(`data-raw/build_metastasis_sites.py`) shows they do not coincide with any
generation-2 site: generation-1 Abdomen includes the peritoneum, adrenals and
intra-abdominal nodes; Thorax includes lung, pleura, pericardium, ribs and
intrathoracic nodes; Pelvis includes pelvic nodes; Extremity is split across
Skin, Other and Lymph nodes. Only liver, breast and brain/CNS (the latter up
to one NOS code) have the same code set in both generations. The harmonised
17-site set is rederived from the imaging and pathology site codes in a later
step rather than approximated by renaming.

Lines of therapy:

```python
from geniekit import io, lot

crc = io.load_release("data/genie_bpc/CRC_v2.0-public")
result = lot.derive(crc, index="advanced")   # lines counted from stage IV diagnosis or first distant metastasis
result.lines                                  # one row per line: start, end, agents, backbone agents, regimens, why it ended
result.regimens                               # every regimen with its status, line number, role and the reason
result.cancers                                # one row per cancer: index event, number of lines, first-line agents
result.summary()

# the index event is the user's choice
lot.derive(crc, index="first_progression")    # first radiologist or oncologist assessment of progression
lot.derive(crc, index="first_systemic")       # start of the first line-defining regimen
lot.derive(crc, index=my_table)               # record_id, ca_seq, index_days (for example a trial-entry date)

# and so are the rules
rules = lot.LineRules(restart_gap_days=90, progression_aware=True, pre_index="line_zero")
lot.derive(crc, index="advanced", rules=rules)
lot.derive(prostate, index="advanced", rules="prostate")   # ADT and first-generation antiandrogens as a backbone
```

How a line is formed (every setting has a default in `lot.LineRules` and is
written into the output):

- regimens that start before the index are excluded (or kept as line 0);
- the first regimen with a line-defining agent on or after the index starts
  line 1; a regimen starting within `combination_window_days` (28) of the
  line start joins it as part of the initial combination;
- a later regimen whose agents are all already in the line continues it
  (maintenance, de-escalation or resumption after a holiday), unless the gap
  since the last known administration exceeds `restart_gap_days`;
- a regimen with a new agent starts the next line; with
  `progression_aware=True` it does so only when a progression assessment
  (imaging or oncologist note) falls between the current line's start and
  the new regimen, and a switch without one continues the line;
- LHRH agonists and antagonists (`backbone` role) and leucovorin
  (`modulator`) never start or end a line and are recorded as components;
  locally administered agents (intravesical BCG, valrubicin, and any regimen
  whose released route is intravesical, intraperitoneal, intrapleural,
  intralesional or intrathecal) are not lines of systemic therapy;
- an investigational drug counts as a line-defining agent whose identity and
  end date are masked, so two investigational regimens are two lines.

Drug identities come from `resources/drug_vocabulary.csv`: every one of the
205 drug labels released across the six cohorts is mapped to a canonical
agent, a class and a role (`lot.load_vocabulary()`). The two schema
generations spell eleven agents differently ("Gemcitabine Hydrochloride"
versus "Gemcitabine HCL", "Leucovorin Calcium" versus "Leucovorin",
"Tamoxifen Citrate" versus "Tamoxifen", and so on) and generation 2
misspells goserelin; the vocabulary resolves all of them. Classes and roles
were assigned by the package author and are kept in one reviewable table
(`data-raw/build_drug_vocabulary.py`).

Aggregate-only analysis:

```python
from geniekit import io, aggregate

crc = io.load_release("data/genie_bpc/CRC_v2.0-public")
report = aggregate.liver_metastasis_report(crc, control=aggregate.DisclosureControl(min_count=5))
report.summary()                                 # sixteen tables, with the number of suppressed cells in each
report["early_failure_by_liver_mets"].frame      # early treatment failure at 6 and 9 months, liver versus no liver metastasis
report["alterations_by_liver_mets"].frame        # APC, TP53, KRAS, NRAS, RAS, BRAF, BRAF V600E, SMAD4 by liver metastasis
report.to_dir("outputs/crc_liver")               # one CSV per table plus report.json: settings, notes, every suppression

# the same building blocks for other questions
table = aggregate.analytic_table(crc, genes=["KRAS", "BRAF"], msi=True)    # patient-level; stays with you
aggregate.proportion_table(table, "first_line", by="stage_dx").frame     # aggregate; can be shared
aggregate.km_table(table, "os_from_index_days", "died", by="liver_mets", at_months=[12, 24]).frame
```

What the layer guarantees: every function returns a `SafeTable` whose cells are
counts, percentages, intervals, medians and quartiles (never a minimum or
maximum), estimates or p-values. A count below `min_count` (default 5) is
replaced by a marker, and every statistic derived from it is replaced too.
Where a suppressed count could be recovered from a table margin, the layer
compares the records hidden by suppressing a second cell with the records
hidden by omitting the small row or column, takes the cheaper option, and
writes what it did into the table's notes. Groups too small to summarise are
omitted with a note instead of appearing as columns of markers. The report's
`report.json` records the index event, the line rules, the disclosure settings
and each suppressed cell, so a reader of the aggregate output can see how it
was produced.

The notebook `examples/crc_liver_metastasis.ipynb` walks through the report;
it uses the real CRC release when `GENIEKIT_DATA_DIR` is set and a synthetic
cohort otherwise.

Command line:

```powershell
geniekit releases --files
geniekit dictionary lookup naaccr_seer_sum_stage --table cancer --values
geniekit dictionary diff --table cancer --kind dtype --kind value_set
geniekit dictionary presence --table patient --only-partial
geniekit access                                   # token and GENIE approval status
geniekit download CRC --dest data/genie_bpc       # after approval
geniekit inspect data/genie_bpc/CRC_v2.0-public   # table summary and column check
geniekit crosswalk --validate                     # the harmonisation rules, checked against the dictionary
geniekit harmonize data\genie_bpc\NSCLC_v2.0-public data\genie_bpc\BLADDER_v1.0-public --out data\harmonized
geniekit lot data\genie_bpc\CRC_v2.0-public --index advanced --out data\lot_crc
geniekit lot data\genie_bpc\Prostate_v1.0-public --rules prostate --progression-aware --out data\lot_prostate
geniekit aggregate data\genie_bpc\CRC_v2.0-public --min-count 5 --out outputs\crc_liver
```

## The six public releases

| Cohort | Version | Patients | Institutions | Synapse | Schema generation |
|---|---|---|---|---|---|
| NSCLC | v2.0-public | 1846 | DFCI, MSK, UHN, VICC | syn27056697 | 1 |
| CRC | v2.0-public | 1485 | DFCI, MSK, VICC | syn30991637 | 1 |
| BrCa | v1.0-public | 1130 | DFCI, MSK, VICC | syn71825332 | 2 |
| PANC | v1.0-public | 1109 | DFCI, MSK, UHN, VICC | syn72666404 | 2 |
| Prostate | v1.0-public | 1116 | DFCI, MSK, UHN, VICC | syn73719283 | 2 |
| BLADDER | v1.0-public | 716 | DFCI, MSK, UHN, VICC | syn75201334 | 2 |

"Schema generation" is a geniekit label, not a GENIE term. The two 2022
releases (NSCLC, CRC) and the four 2025 to 2026 releases share most of their
variables, but the later releases renamed or recoded a measurable subset. Of
the 1105 distinct (table, variable) pairs declared across the six Variable
Synopses, 505 are declared in all six cohorts. Examples of what differs:

- smoking history is `ca_lung_cigarette` in generation 1 and `ca_cigarette` in generation 2;
- distant metastasis sites are `dmets_*` / `dx_to_dmets_*` (9 sites) in generation 1 and `dist_mets_*` / `dx_to_dist_mets_*` (17 sites) in generation 2;
- `naaccr_seer_sum_stage` is declared as character in the generation-1 synopses and numeric in generation 2 (the released files hold the numeric NAACCR code in all six, so this one is a documentation difference);
- `cpt_seq_date` is character in generation 1 and numeric in generation 2;
- `ca_grade`, `pfs_cohort`, `ca_heme_type`, `ca_clin_t_stage` and others have different permitted value sets;
- tumor-marker tables exist only for CRC, BrCa, PANC and Prostate; radiation-therapy tables only for PANC, Prostate and BLADDER;
- structural variants are `data_fusions.txt` in generation 1 and `data_sv.txt` in generation 2;
- UHN contributes to NSCLC, PANC, Prostate and BLADDER but not to CRC or BrCa.

`geniekit dictionary diff` lists every such difference. The dictionary also
records the 17 variables the synopses declare that the released files do not
contain (`dictionary.load().declared_not_released()`); every released
clinical column is declared, so the loader's column check is exact.

## Data access

1. Create a Synapse account and request access to the GENIE BPC data: open
   https://www.synapse.org/Synapse:syn27056700 and click "Request Access".
   One access requirement (id 9606645) covers both the BPC cohorts and the
   main GENIE registry.
2. Complete the Sage Bionetworks Access and Compliance Team (ACT) form linked
   from that dialog, using your Synapse username. Both steps are required.
3. After approval, create a personal access token (Synapse profile, Settings,
   Personal Access Tokens) with "view" and "download" scope and expose it as
   `SYNAPSE_AUTH_TOKEN` (see `.env.example`). Never commit the token.
   `geniekit access` reports whether the token works and whether the access
   request is approved, pending or missing.

The per-cohort Analytic Data Guides and Variable Synopses are public and are
linked from https://aacrprojectgenie.org/data/.

## Relationship to the R package genieBPC

The R package [genieBPC](https://github.com/GENIE-BPC/genieBPC) by the MSK
Biostatistics team downloads BPC releases and builds analytic cohorts one
cohort at a time. geniekit uses the same logical table names where they exist
(`releases.GENIEBPC_TABLE_NAMES` maps between the two) so that workflows can
move between the languages, and adds the cross-cohort harmonisation, line-of-
therapy and aggregate-only layers.

## Development

```powershell
git clone https://github.com/jemsbhai/geniekit.git
cd geniekit
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -e ".[dev]"
pytest tests/ -v --cov=geniekit --cov-report=term
ruff check src tests data-raw
python scripts/check_vocabulary.py
```

Regenerating the packaged metadata (requires network access, no Synapse login):

```powershell
python data-raw/build_releases.py
python data-raw/build_dictionary.py
python data-raw/build_drug_vocabulary.py   # with GENIEKIT_DATA_DIR set, checks coverage of the released labels
```

## License

MIT. See `LICENSE`. See `NOTICE.md` for the provenance of the packaged
metadata and for how to cite the data.
