Metadata-Version: 2.4
Name: relbench
Version: 3.0.0
Summary: RelBench: Relational Deep Learning Benchmark
Keywords: relational deep learning,graph neural networks,relational databases,benchmark,machine learning
Author-email: RelBench Team <relbench@cs.stanford.edu>
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3
License-File: LICENSE
Requires-Dist: pandas
Requires-Dist: huggingface_hub
Requires-Dist: pyyaml
Requires-Dist: pyarrow
Requires-Dist: numpy
Requires-Dist: duckdb
Requires-Dist: scikit-learn
Requires-Dist: typing-extensions
Requires-Dist: tqdm
Requires-Dist: pytest ; extra == "dev"
Requires-Dist: pre-commit ; extra == "dev"
Requires-Dist: sentence-transformers ; extra == "example"
Requires-Dist: pytorch_frame[full] ; extra == "example"
Requires-Dist: torch_geometric ; extra == "example"
Requires-Dist: tqdm ; extra == "example"
Requires-Dist: graphviz ; extra == "schema"
Project-URL: Data, https://huggingface.co/stanford-star
Project-URL: Home, https://star-project.stanford.edu/relbench
Project-URL: Issues, https://github.com/stanford-star/relbench/issues
Project-URL: Repository, https://github.com/stanford-star/relbench
Provides-Extra: dev
Provides-Extra: example
Provides-Extra: schema

<p align="center"><img src="https://star-project.stanford.edu/assets/img/relbench/logo.png" alt="RelBench" width="600px" /></p>

<p align="center">
  <a href="https://star-project.stanford.edu/relbench"><img src="https://img.shields.io/badge/website-STAR%20Project-3f9e78.svg" alt="Website: STAR Project" /></a>
  <a href="https://huggingface.co/stanford-star"><img src="https://img.shields.io/badge/data-%F0%9F%A4%97%20Hugging%20Face-ffcc00.svg" alt="Data: Hugging Face" /></a>
  <a href="https://pypi.org/project/relbench/"><img src="https://img.shields.io/pypi/v/relbench?color=3f9e78" alt="PyPI" /></a>
  <a href="https://github.com/stanford-star/relbench/actions/workflows/testing.yml"><img src="https://github.com/stanford-star/relbench/actions/workflows/testing.yml/badge.svg" alt="Tests" /></a>
  <a href="https://arxiv.org/abs/2407.20060"><img src="https://img.shields.io/badge/arXiv-2407.20060%20(RelBench)-b31b1b.svg" alt="arXiv: RelBench" /></a>
  <a href="https://arxiv.org/abs/2602.12606"><img src="https://img.shields.io/badge/arXiv-2602.12606%20(RelBench%20v2)-b31b1b.svg" alt="arXiv: RelBench v2" /></a>
  <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-3f9e78.svg" alt="License: MIT" /></a>
</p>

<p align="center">
  <a href="#news"><b>News</b></a> ·
  <a href="#get-started"><b>Get Started</b></a> ·
  <a href="#tutorials"><b>Tutorials</b></a> ·
  <a href="#leaderboard"><b>Leaderboard</b></a> ·
  <a href="#byod-bring-your-own-data"><b>BYOD</b></a> ·
  <a href="#citations"><b>Citations</b></a>
</p>

## News

RelBench 3 loads every dataset family below from the Hugging Face Hub with one call —
`relbench.load_dataset("<org>/<repo>/<name>")` — the core databases
([`stanford-star/relbench-v1`](https://huggingface.co/datasets/stanford-star/relbench-v1),
[`stanford-star/relbench-v2-extra`](https://huggingface.co/datasets/stanford-star/relbench-v2-extra)),
CTU/ReDeLEx ([`stanford-star/redelex`](https://huggingface.co/datasets/stanford-star/redelex)),
4DBInfer ([`stanford-star/dbinfer`](https://huggingface.co/datasets/stanford-star/dbinfer)) and
TGB ([`stanford-star/tgb`](https://huggingface.co/datasets/stanford-star/tgb)); the `[ctu]` extra is
gone. MIMIC-IV requires PhysioNet credentials and is not hosted. TGB and 4DBInfer tasks that
use their own scoring protocols load as data only (see [`MIGRATION.md`](MIGRATION.md)).

- **Aug 2026** — RelBench v3 released: datasets and tasks load straight from [Hugging Face](https://huggingface.co/stanford-star), a new [leaderboard](https://star-project.stanford.edu/relbench/leaderboard/) with automated submissions, and bug fixes ([migration guide](MIGRATION.md)).
- **Jun 2026** — Datasets, models, and community hub migrated to [Hugging Face](https://huggingface.co/stanford-star).
- **Mar 2026** — [RelBench v2 paper](https://arxiv.org/abs/2602.12606) accepted at the [ICLR 2026 DATA-FM workshop](https://data-fm-iclr2026.github.io/).
- **Feb 2026** — [Temporal Graph Benchmark](https://tgb.complexdatalab.com/) integration: time-stamped event streams as relational schemas.
- **Jan 2026** — RelBench v2 released: four new databases (SALT, RateBeer, arXiv, MIMIC-IV), 36 new tasks, and a new *Autocomplete* task type; plus 70+ CTU datasets via [ReDeLEx](https://github.com/jakubpeleska/redelex) and 7 from [4DBInfer](https://github.com/awslabs/multi-table-benchmark).
- **Jan 2026** — [ReDeLEx](https://arxiv.org/abs/2506.22199) integration: 70+ [CTU](https://relational.fel.cvut.cz/) relational databases via `relbench[ctu]`.
- **Sep 2024** — [RelBench paper](https://arxiv.org/abs/2407.20060) accepted at NeurIPS 2024 Datasets & Benchmarks Track.
- **Jul 2024** — RelBench released: the first open benchmark for predictive ML on relational databases.
- **May 2024** — [Relational Deep Learning position paper](https://arxiv.org/abs/2312.04615) accepted at ICML 2024.
- **Nov 2023** — Relational Deep Learning introduced in a keynote by Jure Leskovec at the [LoG Conference](https://logconference.org/) ([slides](https://drive.google.com/file/d/1Uk1y6c8z265G0wiRPpGT1cd5lts5lnKq/view)).

## Get Started

```bash
pip install relbench             # data + task loading
pip install "relbench[example]"  # + PyTorch Geometric & PyTorch Frame, for the GNN examples
pip install pyg-lib -f https://data.pyg.org/whl/torch-2.9.0+cpu.html  # neighbor sampling; use the index matching your torch/CUDA build
```

Load a dataset and a task — both come straight from the Hub, with **no per-dataset code**:

```python
import relbench

dataset = relbench.load_dataset("rel-f1")   # or a HuggingFace 'org/repo[/subdir]', or a local path
db = dataset.get_db()                  # rows after test_timestamp are hidden

task = dataset.load_task("driver-position")   # dataset.get_task_names() lists them
train_table = task.get_table("train")  # train / val / test label tables
test_table  = task.get_table("test")   # the target column is hidden on test

# ... train any model on db + train_table, predict on the test entities ...
metrics = task.evaluate(test_pred)     # standard metric for the task type
```

`dataset.val_timestamp` / `dataset.test_timestamp` give the temporal split points. RelBench
is framework-agnostic — bring any modeling stack. For a reference Graph Neural Network on
[PyTorch Geometric](https://github.com/pyg-team/pytorch_geometric) +
[PyTorch Frame](https://github.com/pyg-team/pytorch-frame), see `relbench.modeling` and the
runnable scripts in [`examples/`](examples).

### Tutorials

Open these directly in Google Colab — no setup required:

| Tutorial | What it covers | |
|---|---|---|
| [**Quickstart**](https://colab.research.google.com/github/stanford-star/relbench/blob/main/tutorials/quickstart.ipynb) | Load a dataset/task, explore the schema, run a baseline | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/stanford-star/relbench/blob/main/tutorials/quickstart.ipynb) |
| [**Training a GNN**](https://colab.research.google.com/github/stanford-star/relbench/blob/main/tutorials/gnn.ipynb) | A GNN baseline for an entity task (PyG + PyTorch Frame) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/stanford-star/relbench/blob/main/tutorials/gnn.ipynb) |

## Leaderboard

The [**RelBench leaderboard**](https://star-project.stanford.edu/relbench/leaderboard/)
ranks methods by their test-set performance, averaged over a fixed task set. There are
three independent boards — **classification** (12 tasks), **regression** (9), and
**recommendation** (10); the task lists are in `relbench.submit.LEADERBOARD_TASKS`.
You can submit to any of them; each requires predictions for *all* of its tasks.

To submit:

1. **Write one prediction CSV per task**, named `<dataset>__<task>.csv`, into a directory:

   ```python
   relbench.submit.write_prediction_table(task, test_pred, "preds/rel-f1__driver-position.csv")
   ```

2. **Validate and package** the directory — this scores every CSV against the test tables,
   prints a verdict per leaderboard, and writes clean submission zip file(s):

   ```bash
   python -m relbench.submit preds/
   ```

3. **[Open a submission issue](https://github.com/stanford-star/relbench/issues/new?template=submit.yml)**
   on this repository: fill in the short form and upload the zip file(s) into it.

The submission is validated automatically and the report is posted on the issue; once a
maintainer approves, your entry appears on the leaderboard.

## BYOD (Bring Your Own Data)

You can easily express your own databases and tasks in the **RelBench format**.
A dataset is a self-describing folder — a
`manifest.yaml` (tables, keys, the foreign-key graph, the time splits), one plain parquet
per table, and a `tasks/` subdirectory — that you publish to the
[Hugging Face Hub](https://huggingface.co/stanford-star). RelBench loads it straight from its
`org/repo[/subdir]` address; there is no central registry to register with.

[**`byod/README.md`**](byod/README.md) is the full walkthrough, and the
published [`stanford-star/relbench-v1/rel-f1`](https://huggingface.co/datasets/stanford-star/relbench-v1) is a
complete worked example. For how RelBench's own databases were built, cleaned, and verified
from their original sources, see [`provenance/`](provenance).

If you upload your data in RelBench format to Hugging Face, please let us know
by opening an issue / making a PR so we can list it here!

## Citations

If you use RelBench, please cite the benchmark papers:

```bibtex
@inproceedings{relbench,
  title={RelBench: A Benchmark for Deep Learning on Relational Databases},
  author={Robinson, Joshua and Ranjan, Rishabh and Hu, Weihua and Huang, Kexin and Han, Jiaqi and Dobles, Alejandro and Fey, Matthias and Lenssen, Jan Eric and Yuan, Yiwen and Zhang, Zecheng and He, Xinwei and Leskovec, Jure},
  booktitle={Advances in Neural Information Processing Systems},
  year={2024}
}

@misc{relbenchv2,
  title={RelBench v2: A Large-Scale Benchmark and Repository for Relational Data},
  author={Gu, Justin and Ranjan, Rishabh and Kanatsoulis, Charilaos and Tang, Haiming and Jurkovic, Martin and Hudovernik, Valter and Znidar, Mark and Chaturvedi, Pranshu and Shroff, Parth and Li, Fengyu and Leskovec, Jure},
  year={2026},
  eprint={2602.12606},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2602.12606}
}
```

Datasets sourced from external repositories (CTU/ReDeLEx, 4DBInfer, TGB) carry their own
citations on their [Hugging Face](https://huggingface.co/stanford-star) dataset cards.

