Metadata-Version: 2.4
Name: relbench
Version: 3.0.2
Summary: RelBench: Relational Deep Learning Benchmark
Keywords: relational deep learning,graph neural networks,relational databases,benchmark,machine learning
Author-email: RelBench Team <relbench@cs.stanford.edu>
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3
License-File: LICENSE
Requires-Dist: pandas
Requires-Dist: huggingface_hub
Requires-Dist: pyyaml
Requires-Dist: pyarrow
Requires-Dist: numpy
Requires-Dist: duckdb
Requires-Dist: scikit-learn
Requires-Dist: typing-extensions
Requires-Dist: tqdm
Requires-Dist: pytest ; extra == "dev"
Requires-Dist: pre-commit ; extra == "dev"
Requires-Dist: sentence-transformers ; extra == "example"
Requires-Dist: pytorch_frame[full] ; extra == "example"
Requires-Dist: torch_geometric ; extra == "example"
Requires-Dist: tqdm ; extra == "example"
Requires-Dist: graphviz ; extra == "schema"
Project-URL: Data, https://huggingface.co/stanford-star
Project-URL: Home, https://star-project.stanford.edu/relbench
Project-URL: Issues, https://github.com/stanford-star/relbench/issues
Project-URL: Repository, https://github.com/stanford-star/relbench
Provides-Extra: dev
Provides-Extra: example
Provides-Extra: schema

<p align="center"><img src="https://star-project.stanford.edu/assets/img/relbench/logo.png" alt="RelBench" width="600px" /></p>

<p align="center">
  <a href="https://star-project.stanford.edu/relbench"><img src="https://img.shields.io/badge/website-STAR%20Project-3f9e78.svg" alt="Website: STAR Project" /></a>
  <a href="https://huggingface.co/stanford-star"><img src="https://img.shields.io/badge/data-%F0%9F%A4%97%20Hugging%20Face-ffcc00.svg" alt="Data: Hugging Face" /></a>
  <a href="https://pypi.org/project/relbench/"><img src="https://img.shields.io/pypi/v/relbench?color=3f9e78" alt="PyPI" /></a>
  <a href="https://github.com/stanford-star/relbench/actions/workflows/testing.yml"><img src="https://github.com/stanford-star/relbench/actions/workflows/testing.yml/badge.svg" alt="Tests" /></a>
  <a href="https://arxiv.org/abs/2407.20060"><img src="https://img.shields.io/badge/arXiv-2407.20060%20(RelBench)-b31b1b.svg" alt="arXiv: RelBench" /></a>
  <a href="https://arxiv.org/abs/2602.12606"><img src="https://img.shields.io/badge/arXiv-2602.12606%20(RelBench%20v2)-b31b1b.svg" alt="arXiv: RelBench v2" /></a>
  <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-3f9e78.svg" alt="License: MIT" /></a>
</p>

<p align="center">
  <a href="#news"><b>News</b></a> ·
  <a href="#get-started"><b>Get Started</b></a> ·
  <a href="#tutorials"><b>Tutorials</b></a> ·
  <a href="#leaderboard"><b>Leaderboard</b></a> ·
  <a href="#byod-bring-your-own-data"><b>BYOD</b></a> ·
  <a href="#citations"><b>Citations</b></a>
</p>

## News

- **Oct 2026** — [Agentic leaderboard submissions](leaderboard/README.md#automatically-eg-from-an-ai-agent): point your AI agent at the [leaderboard README](leaderboard/README.md) and it can submit end to end. Entries can now carry an optional repro link for reproducibility.
- **Aug 2026** — RelBench v3 released: datasets and tasks load straight from [Hugging Face](https://huggingface.co/stanford-star), a new [leaderboard](https://star-project.stanford.edu/relbench/leaderboard/) with automated submissions, and bug fixes ([migration guide](MIGRATION.md)).
- **Jun 2026** — Datasets, models, and community hub migrated to [Hugging Face](https://huggingface.co/stanford-star).
- **Mar 2026** — [RelBench v2 paper](https://arxiv.org/abs/2602.12606) accepted at the [ICLR 2026 DATA-FM workshop](https://data-fm-iclr2026.github.io/).
- **Feb 2026** — [Temporal Graph Benchmark](https://tgb.complexdatalab.com/) integration: time-stamped event streams as relational schemas.
- **Jan 2026** — RelBench v2 released: four new databases (SALT, RateBeer, arXiv, MIMIC-IV), 36 new tasks, and a new *Autocomplete* task type; plus 70+ CTU datasets via [ReDeLEx](https://github.com/jakubpeleska/redelex) and 7 from [4DBInfer](https://github.com/awslabs/multi-table-benchmark).
- **Jan 2026** — [ReDeLEx](https://arxiv.org/abs/2506.22199) integration: 70+ [CTU](https://relational.fel.cvut.cz/) relational databases via `relbench[ctu]`.
- **Sep 2024** — [RelBench paper](https://arxiv.org/abs/2407.20060) accepted at NeurIPS 2024 Datasets & Benchmarks Track.
- **Jul 2024** — RelBench released: the first open benchmark for predictive ML on relational databases.
- **May 2024** — [Relational Deep Learning position paper](https://arxiv.org/abs/2312.04615) accepted at ICML 2024.
- **Nov 2023** — Relational Deep Learning introduced in a keynote by Jure Leskovec at the [LoG Conference](https://logconference.org/) ([slides](https://drive.google.com/file/d/1Uk1y6c8z265G0wiRPpGT1cd5lts5lnKq/view)).

## Get Started

```bash
pip install relbench             # data + task loading
pip install "relbench[example]"  # + PyTorch Geometric & PyTorch Frame, for the GNN examples
pip install pyg-lib -f https://data.pyg.org/whl/torch-2.9.0+cpu.html  # neighbor sampling; use the index matching your torch/CUDA build
```

Load a dataset and a task — both come straight from the Hub, with **no per-dataset code**:

```python
import relbench

dataset = relbench.load_dataset("rel-f1")   # or a HuggingFace 'org/repo[/subdir]', or a local path
db = dataset.get_db()                  # rows after test_timestamp are hidden

task = dataset.load_task("driver-position")   # dataset.get_task_names() lists them
train_table = task.get_table("train")  # train / val / test label tables
test_table  = task.get_table("test")   # the target column is hidden on test

# ... train any model on db + train_table, predict on the test entities ...
metrics = task.evaluate(test_pred)     # standard metric for the task type
```

`dataset.val_timestamp` / `dataset.test_timestamp` give the temporal split points. RelBench
is framework-agnostic — bring any modeling stack. For a reference Graph Neural Network on
[PyTorch Geometric](https://github.com/pyg-team/pytorch_geometric) +
[PyTorch Frame](https://github.com/pyg-team/pytorch-frame), see `relbench.modeling` and the
runnable scripts in [`examples/`](examples).

### Tutorials

Open these directly in Google Colab — no setup required:

| Tutorial | What it covers | |
|---|---|---|
| [**Quickstart**](https://colab.research.google.com/github/stanford-star/relbench/blob/main/tutorials/quickstart.ipynb) | Load a dataset/task, explore the schema, run a baseline | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/stanford-star/relbench/blob/main/tutorials/quickstart.ipynb) |
| [**Training a GNN**](https://colab.research.google.com/github/stanford-star/relbench/blob/main/tutorials/gnn.ipynb) | A GNN baseline for an entity task (PyG + PyTorch Frame) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/stanford-star/relbench/blob/main/tutorials/gnn.ipynb) |

## Leaderboard

The [**RelBench leaderboard**](https://star-project.stanford.edu/relbench/leaderboard/)
ranks methods by their test-set performance, averaged over a fixed task set, with three
independent boards: **classification**, **regression**, and **recommendation**.
To submit, write one prediction CSV per task, validate and package them with
`python -m relbench.submit`, and open a
[submission issue](https://github.com/stanford-star/relbench/issues/new?template=submit.yml)
— or let `python -m relbench.submit --submit` do the upload and open the issue for you.

[**`leaderboard/README.md`**](leaderboard/README.md) has the full instructions, including
the prediction CSV format for writing predictions with your own code.

## BYOD (Bring Your Own Data)

You can easily express your own databases and tasks in the **RelBench format**.
A dataset is a self-describing folder — a
`manifest.yaml` (tables, keys, the foreign-key graph, the time splits), one plain parquet
per table, and a `tasks/` subdirectory — that you publish to the
[Hugging Face Hub](https://huggingface.co/stanford-star). RelBench loads it straight from its
`org/repo[/subdir]` address; there is no central registry to register with.

[**`byod/README.md`**](byod/README.md) is the full walkthrough, and the
published [`stanford-star/relbench-v1/rel-f1`](https://huggingface.co/datasets/stanford-star/relbench-v1) is a
complete worked example. For how RelBench's own databases were built, cleaned, and verified
from their original sources, see [`provenance/`](provenance).

If you upload your data in RelBench format to Hugging Face, please let us know
by opening an issue / making a PR so we can list it here!

## Citations

If you use RelBench, please cite the benchmark papers:

```bibtex
@inproceedings{relbench,
  title={RelBench: A Benchmark for Deep Learning on Relational Databases},
  author={Robinson, Joshua and Ranjan, Rishabh and Hu, Weihua and Huang, Kexin and Han, Jiaqi and Dobles, Alejandro and Fey, Matthias and Lenssen, Jan Eric and Yuan, Yiwen and Zhang, Zecheng and He, Xinwei and Leskovec, Jure},
  booktitle={Advances in Neural Information Processing Systems},
  year={2024}
}

@inproceedings{relbenchv2,
  title={RelBench v2: A Large-Scale Benchmark and Repository for Relational Data},
  author={Gu, Justin and Ranjan, Rishabh and Kanatsoulis, Charilaos and Tang, Haiming and Jurkovic, Martin and Hudovernik, Valter and Znidar, Mark and Chaturvedi, Pranshu and Shroff, Parth and Li, Fengyu and Leskovec, Jure},
  booktitle={3rd Workshop on Navigating and Addressing Data Problems for Foundation Models (DATA-FM) at ICLR 2026},
  year={2026},
  note={arXiv:2602.12606 [cs.LG]},
  url={https://arxiv.org/abs/2602.12606}
}
```

Datasets sourced from external repositories (CTU/ReDeLEx, 4DBInfer, TGB) carry their own
citations on their [Hugging Face](https://huggingface.co/stanford-star) dataset cards.

