Metadata-Version: 2.4
Name: tabarena
Version: 0.1.1.dev20260908122839
Summary: A Living Benchmark for Machine Learning on Tabular Data
Author: TabArena Community
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/autogluon/tabarena
Project-URL: Leaderboard, https://tabarena.ai/
Project-URL: Repository, https://github.com/autogluon/tabarena
Project-URL: Issues, https://github.com/autogluon/tabarena/issues
Project-URL: Paper (TabArena), https://arxiv.org/abs/2506.16791
Project-URL: Paper (BeyondArena), https://arxiv.org/abs/2606.30410
Keywords: tabular,machine-learning,benchmark,automl,autogluon,ensemble,deep-learning
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: <3.14,>=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: autogluon.tabular<1.7,>=1.6
Requires-Dist: autogluon.core[ray]<1.7,>=1.6
Requires-Dist: bencheval==0.1.1.dev20260908122839
Requires-Dist: openml>=0.14.1
Requires-Dist: pyyaml
Requires-Dist: tqdm
Requires-Dist: typing-extensions<5,>=4.11
Requires-Dist: huggingface-hub
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: loguru
Provides-Extra: plot
Requires-Dist: bencheval[plot]==0.1.1.dev20260908122839; extra == "plot"
Requires-Dist: tueplots; extra == "plot"
Requires-Dist: autorank==1.2.1; extra == "plot"
Requires-Dist: adjusttext; extra == "plot"
Provides-Extra: text
Requires-Dist: sentence_transformers>=5.4; extra == "text"
Provides-Extra: preprocessing
Requires-Dist: skrub; extra == "preprocessing"
Provides-Extra: data-foundry
Requires-Dist: data-foundry>=0.0.3; extra == "data-foundry"
Provides-Extra: tabpfn
Requires-Dist: tabpfn>=8.0.8; extra == "tabpfn"
Requires-Dist: tabpfn-extensions[many_class]>=0.6.1; extra == "tabpfn"
Provides-Extra: tabicl
Requires-Dist: tabicl>=2.0.0; extra == "tabicl"
Provides-Extra: ebm
Requires-Dist: autogluon.tabular[interpret]<1.7,>=1.6; extra == "ebm"
Provides-Extra: search-spaces
Requires-Dist: configspace<2.0,>=1.2; extra == "search-spaces"
Provides-Extra: realmlp
Requires-Dist: pytabkit<2.0,>=1.5.0; extra == "realmlp"
Provides-Extra: tabdpt
Requires-Dist: tabdpt>=1.2.0; extra == "tabdpt"
Provides-Extra: tabm
Requires-Dist: torch; extra == "tabm"
Requires-Dist: tabm>=0.0.3; extra == "tabm"
Requires-Dist: rtdl_num_embeddings>=0.0.12; extra == "tabm"
Provides-Extra: modernnca
Requires-Dist: category_encoders; extra == "modernnca"
Provides-Extra: xrfm
Requires-Dist: xrfm[cu12]; extra == "xrfm"
Provides-Extra: sap-rpt-oss
Provides-Extra: tabfm
Provides-Extra: tabstar
Requires-Dist: tabstar==1.1.15; extra == "tabstar"
Provides-Extra: perpetualboosting
Requires-Dist: perpetual; extra == "perpetualboosting"
Provides-Extra: orionmsp
Requires-Dist: tabtune==0.1.18; extra == "orionmsp"
Provides-Extra: probmetrics
Requires-Dist: probmetrics; extra == "probmetrics"
Requires-Dist: pytorch-minimize; extra == "probmetrics"
Provides-Extra: limix
Requires-Dist: einops; extra == "limix"
Requires-Dist: kditransform; extra == "limix"
Provides-Extra: tabpfnwide
Requires-Dist: tabpfnwide>=0.3.0; extra == "tabpfnwide"
Provides-Extra: iltm
Requires-Dist: iltm>=0.1.1; extra == "iltm"
Provides-Extra: chimeraboost
Requires-Dist: chimeraboost>=0.30.0; extra == "chimeraboost"
Provides-Extra: nori
Requires-Dist: synthefy-nori>=0.10.0; extra == "nori"
Provides-Extra: exaone-tabular
Provides-Extra: tabswift
Provides-Extra: benchmark
Requires-Dist: tabarena[tabpfn]; extra == "benchmark"
Requires-Dist: tabarena[tabicl]; extra == "benchmark"
Requires-Dist: tabarena[ebm]; extra == "benchmark"
Requires-Dist: tabarena[search_spaces]; extra == "benchmark"
Requires-Dist: tabarena[realmlp]; extra == "benchmark"
Requires-Dist: tabarena[tabdpt]; extra == "benchmark"
Requires-Dist: tabarena[tabm]; extra == "benchmark"
Requires-Dist: tabarena[data-foundry]; extra == "benchmark"
Requires-Dist: tabarena[plot]; extra == "benchmark"
Requires-Dist: tabarena[text]; extra == "benchmark"
Requires-Dist: tabarena[preprocessing]; extra == "benchmark"
Provides-Extra: extended
Requires-Dist: tabarena[modernnca]; extra == "extended"
Requires-Dist: tabarena[xrfm]; extra == "extended"
Requires-Dist: tabarena[sap-rpt-oss]; extra == "extended"
Requires-Dist: tabarena[tabfm]; extra == "extended"
Requires-Dist: tabarena[tabstar]; extra == "extended"
Requires-Dist: tabarena[perpetualboosting]; extra == "extended"
Requires-Dist: tabarena[orionmsp]; extra == "extended"
Requires-Dist: tabarena[limix]; extra == "extended"
Requires-Dist: tabarena[tabpfnwide]; extra == "extended"
Requires-Dist: tabarena[iltm]; extra == "extended"
Requires-Dist: tabarena[chimeraboost]; extra == "extended"
Requires-Dist: tabarena[nori]; extra == "extended"
Requires-Dist: tabarena[tabswift]; extra == "extended"
Requires-Dist: tabarena[exaone_tabular]; extra == "extended"
Provides-Extra: all
Requires-Dist: tabarena[benchmark]; extra == "all"
Requires-Dist: tabarena[extended]; extra == "all"
Requires-Dist: tabarena[probmetrics]; extra == "all"
Dynamic: license-file


<div align="center">

<div id="user-content-toc">
  <ul align="center" style="list-style: none;">
    <summary>
      <img src="https://avatars.githubusercontent.com/u/210855230" width="175" alt="TabArena Logo"/>
    </summary>
  </ul>
</div>

## A Living Benchmark for Machine Learning on Tabular Data 💫

---

| 🚀 [Leaderboard](https://tabarena.ai/) | 📂 [Example Scripts]( https://tabarena.ai/code-examples) | 📊 [Dataset Curation](https://tabarena.github.io/data-foundry/) | 📄 Papers: [TabArena-v0.1](https://arxiv.org/abs/2506.16791) · [BeyondArena](https://arxiv.org/abs/2606.30410) |
|:--------------------------------------:|:----------------------------------------------------------------------------------------:|:----------------------------------------------------------------------------------------:|:--------------------------------------------------------------------------------------------------------------:|

---
</div>

TabArena is a living benchmarking system that makes benchmarking tabular machine learning models a reliable experience. TabArena implements best practices to ensure methods are represented at their peak potential, including cross-validated ensembles, strong hyperparameter search spaces contributed by the method authors, early stopping, model refitting, parallel bagging, memory usage estimation, and more. Explore the latest results on the [live leaderboard](https://huggingface.co/spaces/TabArena/leaderboard).

This single codebase powers **two complementary benchmarks** that share the same fitting, runner, and
evaluation code:

- 🏟️ **TabArena-v0.1** — the living benchmark on **curated, IID** tabular datasets.
- 🌍 **BeyondArena** — a holistic, ***beyond-IID*** benchmark spanning **IID, temporal, and grouped**
  tasks across a wide range of dataset sizes and feature dimensionalities. 
  **BeyondArena will superseed TabArena-v0.1 in the future.**

> **Tip**
> **New here? Start with TabArena, then graduate to BeyondArena.** Get your model working and
> competitive on TabArena's curated IID datasets first; once it holds up there, run the *same* code
> on BeyondArena to stress-test how well it generalizes beyond IID.

**TabArena** covers 51 curated datasets (9–30 splits each) and 27+ methods, including 10+ tabular
foundation models — over 50M trained models, with all validation and test predictions cached for
tuning and post-hoc ensembling. **BeyondArena** extends this to **[142 datasets](https://huggingface.co/datasets/TabArena/BeyondArena)** across IID,
temporal, and grouped task types, spanning tiny to 1M-row datasets and low- to high-dimensional
features.


## ⚡ Quickstart

> **Tip**
> The fastest way to try TabArena end-to-end:

```bash
pip install uv
git clone https://github.com/autogluon/tabarena.git && cd tabarena
uv venv --seed --python 3.12 && source .venv/bin/activate
uv pip install --prerelease=allow -e "./packages/tabarena[benchmark]"
python examples/benchmarking/run_quickstart_tabarena.py
```

For other install paths (eval-only, editable AutoGluon, dependency), see [Installation](#-installation) below.
To try **BeyondArena** instead, run `python examples/beyondarena/run_quickstart_beyondarena.py` with the same install.

## 🕹️ Use Cases

We share more details on various use cases of TabArena in our [examples](https://github.com/autogluon/tabarena/blob/main/examples):

* 🌍 **Benchmarking Beyond IID (BeyondArena)**: please refer to [examples/beyondarena](https://github.com/autogluon/tabarena/blob/main/examples/beyondarena).
* 📊 **Benchmarking Predictive Machine Learning Models**: please refer to [examples/benchmarking](https://github.com/autogluon/tabarena/blob/main/examples/benchmarking).
* 🚀 **Using SOTA Tabular Models Benchmarked by TabArena**: please refer to [examples/running_tabarena_models](https://github.com/autogluon/tabarena/blob/main/examples/running_tabarena_models).
* 🧪 **Advanced and Specialized Usage**: please refer to [examples/advanced](https://github.com/autogluon/tabarena/blob/main/examples/advanced).
* 🗃️ **Analysing Metadata and Meta-Learning**: please refer to [examples/meta](https://github.com/autogluon/tabarena/blob/main/examples/meta).
* 📈 **Generating Plots and Leaderboards**: please refer to [examples/plots](https://github.com/autogluon/tabarena/blob/main/examples/plots).
* 🔁 **Reproducibility**: we share instructions for reproducibility in [examples](https://github.com/autogluon/tabarena/blob/main/examples).

### Datasets

Please refer to our [dataset curation repository](https://github.com/TabArena/tabarena_dataset_curation) to learn more about or contributed data!

### More Documentation

TabArena code is currently being polished. Detailed Documentation for TabArena will be available soon.

# 🪄 Installation

> **Important**
> Requires Python **3.11–3.13** and [uv](https://docs.astral.sh/uv/getting-started/installation/).

TabArena is a [uv workspace](https://docs.astral.sh/uv/concepts/projects/workspaces/); its installable
packages live under `packages/` (`tabarena`, `bencheval`, `tabflow_slurm`). Install the `tabarena`
package directly from `packages/tabarena` with the extras you need. The `--prerelease=allow` flag is
required so uv resolves the pre-release dependency.

First clone the repo and create a virtual environment (one time):

```bash
git clone https://github.com/autogluon/tabarena.git
cd tabarena
uv venv --seed --python 3.12
source .venv/bin/activate
```

Then pick the install path that matches what you want to do:

<details>
<summary><b>📊 Evaluation only</b> — leaderboards, metrics & plots, no model fitting</summary>

Loads cached results and computes/plots leaderboards & metrics (ELO, win-rates, ranks). Depends on `autogluon.tabular` (not the full AutoGluon meta-package) — no model-fitting libraries and no torch.

```bash
uv pip install --prerelease=allow -e "./packages/tabarena[plot]"
```
</details>

<details>
<summary><b>🚀 Benchmark</b> — core set of models for benchmarking</summary>

Installs the core models used for standard benchmarking: `tabpfn`, `tabicl`, `ebm`, `search_spaces`, `realmlp`, `tabdpt`, `tabm`.

```bash
uv pip install --prerelease=allow -e "./packages/tabarena[benchmark]"
```
</details>

<details>
<summary><b>➕ Benchmark + Extended</b> — core models plus the extended model set</summary>

> The `extended` extra is **experimental** and may fail to resolve or install due to incompatible version requirements 
> across model dependencies. Use it only if you specifically need every model in a single environment; 
> otherwise prefer `benchmark` or `benchmark` plus one specific model.

Layers the extended model set (`modernnca`, `xrfm`, `sap-rpt-oss`, ...) on top of the core benchmark set.

```bash
uv pip install --prerelease=allow -e "./packages/tabarena[benchmark,extended]"
```

To install only one extended model on top of `benchmark` (recommended over `extended` when you only need a single extra model), pass its extra by name — for example, just `xrfm`:

```bash
uv pip install --prerelease=allow -e "./packages/tabarena[benchmark,xrfm]"
```
</details>

<details>
<summary><b>🛠️ Developer</b> — editable AutoGluon + editable TabArena</summary>

Create a virtual environment in your workspace directory (it spans both repos cloned below, so `.venv` lives at the workspace root rather than inside either repo):

```bash
uv venv --seed --python 3.12 .venv
source .venv/bin/activate
```

Install editable AutoGluon and TabArena:

```bash
git clone https://github.com/autogluon/autogluon.git
./autogluon/full_install.sh

git clone https://github.com/autogluon/tabarena.git
uv pip install --prerelease=allow -e "./tabarena/packages/tabarena[benchmark]"
```

> In PyCharm, mark `packages/tabarena/src/` and each `autogluon/src/` subdirectory as **Sources Root** so imports resolve.

</details>

<details>
<summary><b>🧪 PyPI</b> — experimental pre-releases, no clone needed</summary>

`tabarena` and `bencheval` are published to PyPI as pre-releases for projects that cannot depend on git URLs, so pass `--pre` (pip) or `--prerelease=allow` (uv). The core package and `[plot]` are complete. Model extras whose upstream package is only available from git (`tabfm`, `sap-rpt-oss`, `exaone_tabular`) are empty on PyPI; the model's install hint tells you what to install by hand. The git checkout above stays the recommended install.

```bash
uv pip install --prerelease=allow "tabarena[plot]"   # or: pip install --pre "tabarena[plot]"
uv pip install --prerelease=allow bencheval          # leaderboard engine only
```
</details>

<details>
<summary><b>📦 Use TabArena as a dependency</b></summary>

Add one of the following to your project's dependencies:

```toml
# TabArena depends on a pre-release of AutoGluon, so allow pre-releases when installing
# (e.g. `uv pip install --prerelease=allow ...` or `pip install --pre ...`).
# Alternatively, pin AutoGluon to a specific pre-release (an exact `==` pin resolves a
# pre-release without the flag), e.g. add `"autogluon.tabular==1.5.1b20260626"`.

# From PyPI (experimental pre-releases; each tabarena release pins its matching bencheval):
"tabarena>=0.1.0a1"
# From git (tip of main; publishable to PyPI only as a source-only extra, see issue #495):
"tabarena @ git+https://github.com/autogluon/tabarena.git#subdirectory=packages/tabarena"
```

</details>

# 📦 TabArena Artifacts

TabArena caches predictions, results, and leaderboards as downloadable artifacts so you can reproduce or extend any analysis without re-running the benchmark.

<details>
<summary><b>Artifact tiers, sizes, and examples</b></summary>

> Artifacts download to `~/.cache/tabarena/` by default. Override the location with the `TABARENA_CACHE` environment variable.
> 
> Raw data is **~100 GB per method type**. Point `TABARENA_CACHE` at a large disk before downloading it.

| Tier | Contents | Size / method | Example |
|---|---|---|---|
| **Raw data** | Per-child test predictions, full metadata, system info | ~100 GB | [`inspect_raw_data_and_verify_splits.py`](https://github.com/autogluon/tabarena/blob/main/examples/meta/inspect_raw_data_and_verify_splits.py) |
| **Processed data** | Minimal data for HPO simulation, portfolios, leaderboards | ~10 GB | [`inspect_processed_data.py`](https://github.com/autogluon/tabarena/blob/main/examples/meta/inspect_processed_data.py) |
| **Results** | Per-config / HPO DataFrames (test error, val error, train time, inference time) | <1 MB | [`run_generate_main_leaderboard.py`](https://github.com/autogluon/tabarena/blob/main/examples/plots/run_generate_main_leaderboard.py) |
| **Leaderboards** | Aggregated ELO, win-rate, average rank, improvability | <1 MB | — |
| **Figures & Plots** | Generated from results and leaderboards | — | — |

</details>


# 📄 Citation

> If you use this code in a scientific publication, please cite the relevant paper(s): **TabArena**
> for the living IID benchmark, and **BeyondArena** for the beyond-IID benchmark.

### TabArena

**TabArena: A Living Benchmark for Machine Learning on Tabular Data**
Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, David Salinas, Frank Hutter
*NeurIPS 2025, Datasets and Benchmarks Track*

📄 [arXiv](https://arxiv.org/abs/2506.16791) · 🎤 [NeurIPS poster & video](https://neurips.cc/virtual/2025/loc/san-diego/poster/121499)

<details>
<summary><b>BibTeX</b></summary>

> The entry uses `year=2026` because NeurIPS'25 proceedings are published in 2026.

```bibtex
@article{erickson2026tabarena,
  title   = {TabArena: A Living Benchmark for Machine Learning on Tabular Data},
  author  = {Erickson, Nick and Purucker, Lennart and Tschalzev, Andrej and Holzm{\"u}ller, David and Desai, Prateek and Salinas, David and Hutter, Frank},
  journal = {Advances in Neural Information Processing Systems},
  volume  = {38},
  year    = {2026}
}
```

</details>

### BeyondArena

**Beyond IID: How General Are Tabular Foundation Models, Really?**
Lennart Purucker, Andrej Tschalzev, Nick Erickson, Gioia Blayer, David Holzmüller, Alan Arazi, Alexander Pfefferle, Mustafa Tajjar, Gaël Varoquaux, Frank Hutter

📄 [arXiv](https://arxiv.org/abs/2606.30410)

<details>
<summary><b>BibTeX</b></summary>

```bibtex
@misc{purucker2026beyondiid,
  title         = {Beyond IID: How General Are Tabular Foundation Models, Really?},
  author        = {Purucker, Lennart and Tschalzev, Andrej and Erickson, Nick and Blayer, Gioia and Holzm{\"u}ller, David and Arazi, Alan and Pfefferle, Alexander and Tajjar, Mustafa and Varoquaux, Ga{\"e}l and Hutter, Frank},
  year          = {2026},
  eprint        = {2606.30410},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2606.30410}
}
```

</details>


--- 
## Relation to TabRepo 

TabArena was built upon and now replaces [TabRepo](https://arxiv.org/pdf/2311.02971). To see details about TabRepo, the portfolio simulation repository, refer to [tabrepo.md](https://github.com/autogluon/tabarena/blob/main/tabrepo.md).

## Research code
This repository contains research code intended for academic research and experimentation. It is not production-ready and should be reviewed, tested, and secured before use in production.
