Metadata-Version: 2.4
Name: tabbench-bio
Version: 1.0.0
Summary: A benchmark for machine learning on high-dimensional biological data
Author-email: Jules Kreuer <jules.kreuer@uni-tuebingen.de>
Maintainer-email: Jules Kreuer <jules.kreuer@uni-tuebingen.de>
License-Expression: EUPL-1.2 AND Apache-2.0
Project-URL: Homepage, https://tabbench-bio.eu
Project-URL: Repository, https://github.com/not-a-feature/TabBench-Bio
Project-URL: Bug Tracker, https://github.com/not-a-feature/TabBench-Bio/issues
Keywords: benchmark,machine-learning,tabular,autogluon,bioinformatics,genomics,gene-expression,hdlss,biology
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: <3.14,>=3.13
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
License-File: licenses/Apache-2.0.txt
Requires-Dist: numpy==2.4.*
Requires-Dist: pandas<3.0,>=2.2.3
Requires-Dist: scikit-learn==1.9.*
Requires-Dist: scipy<2.0,>=1.10
Requires-Dist: matplotlib<4.0,>=3.9
Requires-Dist: pillow>=9.0
Requires-Dist: tqdm<5.0,>=4.64
Requires-Dist: threadpoolctl<4.0,>=3.1
Requires-Dist: filelock<5.0,>=3.32.3
Requires-Dist: GEOparse>=2.0
Requires-Dist: biopython>=1.81
Requires-Dist: requests>=2.31
Requires-Dist: kagglehub>=0.3
Requires-Dist: openml>=0.14.1
Requires-Dist: pyarrow<26.0,>=12.0
Requires-Dist: lmdb<3.0,>=1.4
Requires-Dist: rdkit>=2024.3
Provides-Extra: tabpfn35
Requires-Dist: tabbench-bio[autogluon]; extra == "tabpfn35"
Requires-Dist: torch==2.13.*; extra == "tabpfn35"
Requires-Dist: tabpfn==9.0.0; extra == "tabpfn35"
Requires-Dist: tabpfn-extensions>=0.4.0; extra == "tabpfn35"
Requires-Dist: nvidia-ml-py>=12.0; extra == "tabpfn35"
Provides-Extra: autogluon
Requires-Dist: torch<3.0,>=2.0; extra == "autogluon"
Requires-Dist: autogluon.common>=1.5; extra == "autogluon"
Requires-Dist: autogluon.core>=1.5; extra == "autogluon"
Requires-Dist: autogluon.features>=1.5; extra == "autogluon"
Requires-Dist: autogluon.tabular>=1.5; extra == "autogluon"
Provides-Extra: models
Requires-Dist: imodels>=2.0.4; extra == "models"
Requires-Dist: tabpfn>=8.0.3; extra == "models"
Requires-Dist: pytabkit>=1.7.3; extra == "models"
Requires-Dist: tabdpt<1.3,>=1.2; extra == "models"
Requires-Dist: tabpfnwide>=0.3.0; extra == "models"
Requires-Dist: tabfm[pytorch]==1.0.1; extra == "models"
Requires-Dist: tabpfn-extensions>=0.4.0; extra == "models"
Requires-Dist: catboost<1.3,>=1.2; extra == "models"
Requires-Dist: tabicl<2.1,>=2.0; extra == "models"
Requires-Dist: xgboost<3.2,>=2.0; extra == "models"
Requires-Dist: loguru; extra == "models"
Requires-Dist: einx; extra == "models"
Requires-Dist: transformers; extra == "models"
Provides-Extra: bio
Provides-Extra: full
Requires-Dist: tabbench-bio[autogluon]; extra == "full"
Requires-Dist: tabbench-bio[models]; extra == "full"
Requires-Dist: pynvml>=11.0; extra == "full"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: black>=23.0; extra == "dev"
Requires-Dist: isort>=5.12; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: pre-commit>=3.0; extra == "dev"
Provides-Extra: docs
Requires-Dist: sphinx>=7.0; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=2.0; extra == "docs"
Requires-Dist: myst-parser>=2.0; extra == "docs"
Requires-Dist: sphinx-autodoc-typehints>=1.25; extra == "docs"
Dynamic: license-file

# TabBench-Bio

[Website](https://tabbench-bio.eu) · [GitHub](https://github.com/not-a-feature/TabBench-Bio) · [Codeberg](https://codeberg.org/not_a_feature/TabBench-Bio)

TabBench-Bio benchmarks machine learning on biological tables with many features and few
samples, such as gene-expression and methylation data. It downloads datasets, fixes the
train/test splits, fits models and ranks their predictions using fold-level Elo.
The full configuration covers 33 classification and 13 regression datasets.

This repository contains the package and experiment definitions. The
[published website](https://tabbench-bio.eu) is built from Codeberg's `pages` branch.

## Install

Use Python 3.13:

```bash
uv venv --python 3.13
source .venv/bin/activate
uv pip install tabbench-bio
```

For development, use `uv pip install -e .` from a source checkout.
Dataset-loading dependencies are included by default.
Missing datasets download automatically. Custom estimators need no AutoGluon or CUDA.

## Benchmark your model

Define `create_model(task)` in `my_model.py`, returning a cloneable scikit-learn
classifier or regressor. See the [SVM example](https://github.com/not-a-feature/TabBench-Bio/blob/main/benchmark_my_model/my_model.py).

```bash
tabbench-bio evaluate --model my_model.py --task overall
```

This checks the latest published reference, reuses its verified local cache or
downloads the current database, and uses its frozen folds. It writes
`my_model_results/report/report.html`, Elo plots and CSV metrics.
See [Benchmark my model](https://github.com/not-a-feature/TabBench-Bio/blob/main/benchmark_my_model/README.md) for options.

## Datasets and models

The [dataset registry](https://github.com/not-a-feature/TabBench-Bio/blob/main/src/tabbench_bio/bio/data/bio_datasets.json) gives each task a
stable `bio_id`. Edit it to add datasets from a supported source, or set
`TABBENCH_BIO_DATASETS` to your own registry file. A run config selects tasks through
`datasets_classification` and `datasets_regression`. Default grids use `null` for
both selections, expanding the enabled, curated tasks in this registry by task
type. There are no separate default dataset rosters to maintain. Explicit lists
still select subsets, and each run freezes its resolved dataset IDs.

Missing datasets download automatically and are cached. Curated embedding tables
come from checksum-verified Parquet assets in the
[GitHub Release](https://github.com/not-a-feature/TabBench-Bio/releases/tag/v0.1.0).
Kaggle requires `~/.kaggle/kaggle.json`.
Use `--cache-dir` to select the cache or `TABBENCH_BIO_LOCAL_DIR` for local tables.
The [contributor guide](https://github.com/not-a-feature/TabBench-Bio/blob/main/CONTRIBUTING.md#adding-a-dataset) explains how to add and check a dataset.

The [model roster](https://github.com/not-a-feature/TabBench-Bio/blob/main/configs/models/all.json) includes linear models, trees, neural
networks and tabular foundation models. `DUMMY` provides a constant baseline, and
Random Forest anchors Elo at 1000. New adapters need a `ModelSpec` in
`src/tabbench_bio/model_registry.py`, including their environment profile.

## Merge results and build the website

Combine a new model run with an existing database:

```bash
uv run --no-sync tabbench-bio merge results.sqlite results/my_model/results.sqlite \
  --output results/combined/results.sqlite
```

The output path must be new. The merge checks cell settings and held-out targets,
removes duplicate attempts and leaves both inputs unchanged. It also accepts result
directories containing writer databases.

The `website/` submodule tracks Codeberg's `pages` branch. Initialise and update it
before building from the repository root:

```bash
git submodule update --init website
git -C website switch pages
git -C website pull --ff-only
uv run --no-sync tabbench-bio leaderboard results/combined/results.sqlite --workers 4
python -m http.server 8000 --directory website
```

Open `http://localhost:8000/`. The build writes the dashboard, model cards, dataset
pages and JSON data to `website/`. Per-cell PNG plots and CSV tables go in the
Git-ignored `website/local/` directory. It generates no per-cell HTML reports or SVGs.
Use `--out preview` to build elsewhere.

The input database stays read-only. Building needs no raw datasets, GPU or LaTeX,
and publishes nothing automatically. See the
[model guide](https://github.com/not-a-feature/TabBench-Bio/blob/main/benchmark_my_model/INTEGRATION.md#merge-and-generate-plots) for publishing
and cache settings. The released database and SHA-256 checksum are listed in the
[artifact browser](https://tabbench-bio.eu/artifacts.html).

## Read a leaderboard

Load a published SQLite bundle in Python:

```python
from tabbench_bio import Leaderboard

sqlite_path = "results.sqlite"
print(Leaderboard.sqlite_cells(sqlite_path))
leaderboard = Leaderboard.from_sqlite(sqlite_path, cell="cap_10000_n100")
print(leaderboard.rank())
```

Or print one cell and export its plot and table:

```bash
uv run --no-sync tabbench-bio leaderboard --sqlite results.sqlite \
  --cell cap_10000_n100 --plot --csv leaderboard.csv
```

`rank()`, the terminal summary and plots use Bradley-Terry Elo from matching
cross-validation folds. Each fold has weight `1 / target_fold_count`, and the 95%
intervals resample whole targets. The default is 2,000 bootstrap rounds, matching the
paper. Normalised `Score` and fold means are descriptive summaries.

Strict results use recorded DUMMY metrics for failed fits and omit incomplete
model-target fold sets. Elo needs RF and at least one comparable model.
The website also provides adaptive and conditional views, explained in the model guide.
Use `--task` to select classification or regression.

## Caches

Dataset caches hold source downloads, assembled tables and prepared splits. Set
`cache_dir` in a run config or `TABBENCH_BIO_CACHE` for the biological dataset cache.

Website builds cache fold metrics in `<out>.cache/fold_metrics.sqlite` and Elo in
`data/dashboard.json`. Reuse the same output directory to avoid repeating work.
Changed inputs, calculation code or relevant settings invalidate cached results.
Completed metric batches survive an interrupted build. Elo is saved after a full build.
Keep the metric cache locally. Deleting it forces recalculation.

`--workers` controls both metric and Elo calculations. Progress bars show completed
folds and comparison pools, elapsed time and estimated time remaining.

## Tuned models

Define a tuned variant in a JSON model roster: choose a supported parent adapter,
give the variant its own key, and specify the candidate parameters. Selection
uses training data only, with search and refitting sharing the run's time budget.

```sh
tabbench-bio MY-MODEL-TUNED --model-config my_grid.json --output results/my-tuned-model
```

See [Add a tuned model](https://github.com/not-a-feature/TabBench-Bio/blob/main/benchmark_my_model/TUNING.md) for the JSON template and
[Run on a cluster](https://github.com/not-a-feature/TabBench-Bio/blob/main/benchmark_my_model/INTEGRATION.md#run-on-a-cluster) for batch execution.

## Licence

EUPL-1.2. See [LICENSE](https://github.com/not-a-feature/TabBench-Bio/blob/main/LICENSE). The vendored TabArena Elo helper retains Apache-2.0,
as recorded in [NOTICE](https://github.com/not-a-feature/TabBench-Bio/blob/main/NOTICE) and [its licence](https://github.com/not-a-feature/TabBench-Bio/blob/main/licenses/Apache-2.0.txt).
Datasets and models retain their providers' licences.

## Paper citation

Kreuer, J., Ouaari, S., Hellmig, J., Braitinger, J., and Pfeifer, N. (2026).
TabBench-Bio: A Living Benchmark for Machine Learning on High-Dimensional Biomedical Tables.
[arXiv:2609.07441](https://arxiv.org/abs/2609.07441).
[DOI: 10.48550/arXiv.2609.07441](https://doi.org/10.48550/arXiv.2609.07441).

## Agent skill for model selection

The [model-selection skill](https://github.com/not-a-feature/TabBench-Bio/blob/main/skills/biomedical-tabular-model-selection/SKILL.md) helps an
agent compare published results for a dataset's modality and feature/sample budgets,
including uncertainty, failures, cost and training-data overlap. Install it in a
compatible agent with:

```sh
npx skills add https://tabbench-bio.eu/skill.md
```

This downloads the Markdown skill. A browsing agent can also read the
[hosted version](https://tabbench-bio.eu/skill.md) directly.

### Dataset cache consistency

Raw dataset caches record a SHA-256 fingerprint of data-affecting registry fields
and the loader version. A mismatch, including an older cache without a fingerprint,
stops loading. Review the registry change, then explicitly refresh that dataset with
`load_bio_dataset(bio_id, cache_dir=..., force_refetch=True)`. Processed split cache
paths include the same fingerprints in per-dataset directories. `split_params.json`
records shared split settings; each directory has its own `dataset_spec.json`.
Raw cache files are replaced atomically. Increment `LOADER_CACHE_VERSION` in
`bio/fingerprint.py` when loader changes alter matrices, labels or biological groups.

### Check a checkout

Run `tabbench-bio doctor --cache-dir .cache` to check registered adapter files,
environment profiles, skill licences/copies, and existing
raw and processed dataset caches. Add `--site-dir website` to verify generated skill
assets. Add `--model RF` (repeatable) to test adapter importability in that model's
installed environment; without it, only adapter file presence is checked. The
command does not download data, fit models, or repair caches. Failures exit nonzero.

Legacy caches can be adopted explicitly after verifying that their source and task match the
current registry, without downloading live sources again:

```bash
tabbench-bio cache-adopt --cache-dir .cache --dataset OpenML-1138 --reason "Verified against the original registry" --manifest results/run/split_manifest.json
```

The optional manifest verifies frozen target fingerprints before any cache is changed.
Adoption records the reason and timestamp in cache metadata and refuses already-versioned
or obsolete grouped caches. It cannot establish feature provenance; verify that separately.
Raw fingerprints cover data-affecting fields, not descriptions or licences. Processed split
paths include the individual dataset fingerprint, so filtered runs reuse full-run splits.
