Metadata-Version: 2.4
Name: appfl-bio-suite
Version: 0.3.0
Summary: Cross-silo federated learning experiments in computational biology, built on APPFL and Globus Compute
License: MIT License
        
        Copyright (c) 2026 appfl-bio-suite contributors
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
        ---
        
        This project depends on, but does not include, third-party software and data
        with their own licenses:
        
        * APPFL -- MIT License. https://github.com/APPFL/APPFL
        * FLamby -- MIT License. https://github.com/owkin/FLamby
          FLamby ships no data. Each of its datasets requires accepting the license on
          the original host. The Fed-Heart-Disease data derives from the UCI Heart
          Disease dataset (CC BY 4.0); attribute the original authors.
        * PGS Catalog scoring files -- see https://www.pgscatalog.org/about/ for terms.
        
Project-URL: Homepage, https://github.com/mfjoel01/appfl-bio-suite
Project-URL: Repository, https://github.com/mfjoel01/appfl-bio-suite
Project-URL: Documentation, https://github.com/mfjoel01/appfl-bio-suite/tree/main/docs
Keywords: federated-learning,globus-compute,appfl,gwas,polygenic-score,fine-mapping,susiex,flamby,dna-methylation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Requires-Python: <3.13,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: appfl==1.10.0
Requires-Dist: globus-compute-sdk==4.9.0
Requires-Dist: globus-compute-endpoint==4.9.0
Requires-Dist: globus-sdk==4.4.1
Requires-Dist: globus-identity-mapping==0.5.0
Requires-Dist: click==8.1.8
Requires-Dist: PyYAML>=6.0
Requires-Dist: pydantic<3,>=2.7
Requires-Dist: Jinja2<4,>=3.1
Requires-Dist: numpy==1.26.4
Requires-Dist: torch==2.8.0
Provides-Extra: flamby
Requires-Dist: pandas==2.2.2; extra == "flamby"
Requires-Dist: scikit-learn==1.7.2; extra == "flamby"
Requires-Dist: tqdm==4.67.1; extra == "flamby"
Requires-Dist: wget; extra == "flamby"
Provides-Extra: gwas
Requires-Dist: pandas==2.2.2; extra == "gwas"
Requires-Dist: scipy==1.16.2; extra == "gwas"
Requires-Dist: scikit-learn==1.7.2; extra == "gwas"
Requires-Dist: pandas-plink==2.3.2; extra == "gwas"
Provides-Extra: gwas-sim
Requires-Dist: appfl-bio-suite[gwas]; extra == "gwas-sim"
Requires-Dist: xarray==2025.9.1; extra == "gwas-sim"
Requires-Dist: dask==2024.9.0; extra == "gwas-sim"
Requires-Dist: matplotlib==3.10.6; extra == "gwas-sim"
Provides-Extra: finemapping
Requires-Dist: pandas==2.2.2; extra == "finemapping"
Requires-Dist: scipy==1.16.2; extra == "finemapping"
Provides-Extra: finemapping-sim
Requires-Dist: appfl-bio-suite[finemapping]; extra == "finemapping-sim"
Requires-Dist: joblib==1.4.2; extra == "finemapping-sim"
Requires-Dist: matplotlib==3.10.6; extra == "finemapping-sim"
Requires-Dist: Jinja2<4,>=3.1; extra == "finemapping-sim"
Provides-Extra: multisusie
Requires-Dist: appfl-bio-suite[finemapping-sim]; extra == "multisusie"
Provides-Extra: methylation
Requires-Dist: scikit-learn==1.7.2; extra == "methylation"
Requires-Dist: matplotlib==3.10.6; extra == "methylation"
Provides-Extra: watch
Requires-Dist: hivewatch==0.2.1; extra == "watch"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Provides-Extra: all
Requires-Dist: appfl-bio-suite[flamby]; extra == "all"
Requires-Dist: appfl-bio-suite[gwas]; extra == "all"
Requires-Dist: appfl-bio-suite[gwas-sim]; extra == "all"
Requires-Dist: appfl-bio-suite[finemapping]; extra == "all"
Requires-Dist: appfl-bio-suite[finemapping-sim]; extra == "all"
Requires-Dist: appfl-bio-suite[watch]; extra == "all"
Requires-Dist: appfl-bio-suite[methylation]; extra == "all"
Dynamic: license-file

# appfl-bio-suite

Cross-silo **federated learning experiments in computational biology**, built on
[APPFL](https://github.com/APPFL/APPFL),
[hivewatch](https://github.com/APPFL/hivewatch) and
[Globus Compute](https://www.globus.org/compute).

Institutions train on their own data, on their own hardware, and return only results.
The standard multi-user deployment uses individual identities, with no shared credential,
VPN, or inbound network access.

> **Experimental: single-user service-account onboarding.** An opt-in prototype uses one service
> identity per site, with credentials held by that site and the coordinator. It is not
> yet the default or a production-ready replacement for multi-user endpoints.
> See the [trial guide](docs/coordinator/service-account-trial.md) and
> [design evaluation](docs/coordinator/endpoint-sharing-evaluation.md).

**→ [mfjoel01.github.io/appfl-bio-suite](https://mfjoel01.github.io/appfl-bio-suite/)** —
the project site, with a live preview of the [federation network
map](https://mfjoel01.github.io/appfl-bio-suite/network.html).

```
    YOU (coordinator)                          PARTNER SITES
    ┌──────────────┐
    │   driver     │ ──── task ────►    their cluster, their data
    │  aggregator  │ ◄─── result ───    (weights, or summary statistics)
    └──────────────┘
```

---

## The experiments

Fine-mapping also has a completed optional
[MultiSuSiE coordinator pipeline](docs/experiments/fine-mapping/MULTISUSIE.md) and
[matched-filter comparison against SuSiEx](docs/experiments/fine-mapping/MATCHED_RESULTS.md).
Use `run fine-mapping --backend susiex` or `--backend multisusie` with either the
local or remote driver; `--results-dir` keeps their outputs separate.
The [comparison report and figures](docs/experiments/fine-mapping/COMPARISON_FIGURES.md)
cover both methods across cohorts and filter policies.


Four experiments, one toolchain. The first three dispatch to partner sites; the fourth is
in development.

| | What it is | Data |
| --- | --- | --- |
| **[FLamby Fed-Heart-Disease](docs/experiments/flamby-heart-disease/ABOUT.md)** | Multi-round FedAvg on a public tabular benchmark with natural hospital splits | Partners download it; ~40 KB, CC BY 4.0 |
| **[Federated GWAS](docs/experiments/gwas/ABOUT.md)** | Single-round summary-statistic meta-analysis with polygenic-score evaluation | Simulated and distributed by the coordinator |
| **[Federated fine-mapping](docs/experiments/fine-mapping/ABOUT.md)** | Single-round cross-ancestry credible-set construction from per-site second moments — provably equal to a pooled fit, not an approximation of one. Governed by [GA4GH DUO, DRS, TRS and TES](docs/coordinator/ga4gh.md) | Simulated and distributed by the coordinator |
| **[Federated methylation](docs/experiments/methylation/ABOUT.md)** · *work in progress* | Multi-round FedAvg for methylation-based subtype classification on sparse, nanopore-like input, against pooled and local-only training. Currently a single-machine simulation | Synthetic, or open TCGA-LAML methylation data from the NCI GDC |

**Federated methylation is a work in progress.** All of its sites are simulated in one
process today, so there is no partner bundle or endpoint for it yet. Its results so far,
including one real-data run, are recorded as measured in its
[record of runs](docs/experiments/methylation/ABOUT.md#record-of-runs).

## Quickstart

Python 3.12 is required. Install the released package from PyPI:

```bash
python -m pip install "appfl-bio-suite[all]==0.3.0"
```

For coordinator scripts, example federation files, and the pinned constraints, use
the matching source checkout:

```bash
git clone --branch v0.3.0 https://github.com/mfjoel01/appfl-bio-suite.git
cd appfl-bio-suite
python -m venv .venv && source .venv/bin/activate
pip install -e ".[all]" -c constraints.txt
```

Prove the install works — a complete federation on one machine, no partners, no network,
and nothing to fill in first:

```bash
appfl-bio-suite simulate gwas --scenario ci-tiny --out /tmp/gwas-ci
appfl-bio-suite run gwas --config loopback --driver serial --data-root /tmp/gwas-ci
```

Results land in `local/output/gwas-loopback/`. This is the same pair of commands CI runs.

Then stand up a real federation:

```bash
cp federation.yaml.example local/federation.yaml
$EDITOR local/federation.yaml
appfl-bio-suite preflight
```

**→ [docs/coordinator/new-federation.md](docs/coordinator/new-federation.md)** takes you
from here to a running experiment across institutions you recruit yourself.

## The CLI

```
appfl-bio-suite preflight [--experiment X]          environment, pins, configs, endpoints
appfl-bio-suite endpoint status [SITE]              is it online
appfl-bio-suite endpoint smoke [SITE]               round-trip: does it actually work
appfl-bio-suite identity validate <mapping.json>    catch mapping bugs offline, in a second
appfl-bio-suite simulate EXPERIMENT --scenario NAME  generate and split synthetic data
appfl-bio-suite run EXPERIMENT                      run it
appfl-bio-suite partner-bundle EXPERIMENT --site X  a partner's complete setup package
appfl-bio-suite watch build | serve | export        the federation, on a map
appfl-bio-suite ga4gh duo | drs | trs | tes         data use, data objects, tools, tasks
```

---

## How it works

**One config file.** `federation.yaml` declares who you are, who your partners are, and
what each is doing. Every generator, config, document template and check reads from it.
Nothing in `src/` contains a coordinator identity, an endpoint UUID, or a site name, and
CI verifies that.

**Generated partner bundles, not templates.** `partner-bundle` emits a directory with a
partner's configuration fully resolved — their site ID, their data assignment, *your*
identity, their scheduler's provider block — plus their two setup documents with no
placeholders left to interpret.

**A closed set for partner-facing content.** Bundles are built from `docs/partner/` and
nowhere else, enforced in code, so nothing outside that tree can reach a partner.

**Exact version pinning.** Every site runs byte-identical versions of the Globus Compute
stack. Skew does not fail at install time; it fails as a deserialization error several
rounds into a run on somebody else's cluster. `constraints.txt` pins it and
`preflight --check pins` verifies it.

**Failures that name the fix.** The
[troubleshooting registry](docs/coordinator/troubleshooting.md) is indexed by symptom and
gives the mechanism alongside the fix.

## GA4GH standards

The fine-mapping experiment is assembled out of four GA4GH specifications, each closing a
hole the design otherwise had.

| | | answers |
| --- | --- | --- |
| **DUO** | Data Use Ontology | May this site's data be used for this study? |
| **DRS** | Data Repository Service | Exactly which bytes did a site compute over? |
| **TRS** | Tool Registry Service | Exactly which tool version computed them? |
| **TES** | Task Execution Service | Run that tool, there, on those bytes. |

Together: *a named tool version, pinned by checksum, runs over content-addressed data
whose use terms permit the study that dispatched it.* Every clause of that sentence used
to be an assumption.

**A site enforces its own consent code.** Each bundle carries a `DATA_USE.json`, and the
site's worker matches it against the study the coordinator declared — refusing before it
opens a genotype file, on its own hardware, in the account it controls. The coordinator
runs the same check offline before dispatch, but that one only saves a queue wait.

**Bundles are content-addressed.** A sha-256 per file and a Merkle id over the set, so
"did this site run on the bundle I cut for it" is a string comparison. Without it, a site
running last month's bundle produces well-formed aggregates over the wrong individuals,
pools without complaint, and is wrong in no visible way.

**Results name the code that made them.** The tool pin covers the source of the two
modules APPFL ships to workers, is verified at launch, and is written into every run's
`ga4gh_provenance.json` alongside what each site attests to having actually done.

```bash
appfl-bio-suite ga4gh duo check --experiment fine-mapping   # will every site permit this?
appfl-bio-suite ga4gh drs register --data-root local/data/… # address the bundles
appfl-bio-suite ga4gh trs publish --out local/trs           # pin the tool
appfl-bio-suite ga4gh tes task --site X                     # what would run, as a document
```

All of it is opt-in per experiment: a `federation.yaml` with no `ga4gh` blocks produces
byte-identical configs to the ones it produced before any of this existed.

**→ [docs/coordinator/ga4gh.md](docs/coordinator/ga4gh.md)**

## The network map

Everyone in the federation — coordinator and partners alike — can open the same page:
every institution on a world map, which experiments each runs, what kind of data each
holds, how many samples they declared, and whether their endpoint is answering.

```bash
pip install 'appfl-bio-suite[watch]' -c constraints.txt

appfl-bio-suite watch build              # draw it from federation.yaml
appfl-bio-suite watch serve              # look at it locally
appfl-bio-suite watch export --out site/ # three static files anyone can host
appfl-bio-suite run gwas --watch         # a live run, on the same map
```

Built on [hivewatch](https://github.com/APPFL/hivewatch), APPFL's monitoring toolkit,
which supplies the event schema, the map format and the viewer.

**The map exists before any run does.** It is drawn from `federation.yaml` alone, so the
day the config names a partner, that partner is on the map — which is what makes it a
picture of the network rather than a dashboard for one job.

Site coordinates are declared in `federation.yaml`, never resolved from an IP address: a
partner's compute node has no outbound web access, and you already know where your
partners are. A site with no coordinates is reported as unplaced rather than drawn at a
guessed position. Service accounts, cluster paths, your Globus identity and (by default)
full endpoint UUIDs never reach the published page.

**→ [docs/coordinator/network-map.md](docs/coordinator/network-map.md)**

## Reproducibility

- **Every simulation run writes a manifest** recording the scenario, every seed, the suite
  commit, input and output checksums, and package versions.
  `simulate gwas --verify <manifest>` confirms a rerun mechanically.
- **The GWAS simulation is deterministic.** For a fixed seed, all 35 outputs are
  byte-identical, verified against recorded checksums.
- **Known limits are documented.**
  [docs/experiments/gwas/DATA.md](docs/experiments/gwas/DATA.md) states which input cohort
  is not regenerable, and what the shipped substitute does and does not reproduce.

## Documentation

Organized by audience.

| | |
| --- | --- |
| [docs/README.md](docs/README.md) | Index: which document, for whom, in what order |
| [docs/coordinator/](docs/coordinator/) | Running a federation |
| [docs/partner/](docs/partner/) | What a partner receives — a closed set |
| [docs/experiments/](docs/experiments/) | Per-experiment design, data, and runbooks |

## Requirements

Python 3.12. A Globus account. Partners willing to run a privileged process on their own
clusters.

Not required: a shared filesystem, a VPN, matching operating systems or schedulers, or
inbound network access.

## Licence

MIT. See [LICENSE](LICENSE) — and note the third-party terms recorded there for APPFL,
FLamby, and PGS Catalog data.

For automated partner setup, see [bundle-based onboarding](docs/coordinator/onboarding.md).
