Metadata-Version: 2.4
Name: oceanbench
Version: 0.7.0
Summary: 
License-File: LICENSES/Apache-2.0.txt
License-File: LICENSES/CC0-1.0.txt
License-File: LICENSES/EUPL-1.2.txt
License-File: LICENSES/ISC.txt
License-File: LICENSES/MIT.txt
Author: Mercator Ocean International
Author-email: digital-ocean-dev@mercator-ocean.eu
Requires-Python: >=3.12.9
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Requires-Dist: aiohttp (>=3.0.0)
Requires-Dist: boto3 (>=1.26)
Requires-Dist: copernicusmarine (==2.0.1)
Requires-Dist: dask (>=2022)
Requires-Dist: gsw (>=3.6.19,<4.0.0)
Requires-Dist: jsonschema (>=4.18,<5.0)
Requires-Dist: matplotlib (>=3.10.0,<4.0.0)
Requires-Dist: netcdf4 (>=1.7.2,<2.0.0)
Requires-Dist: numcodecs (==0.15.1)
Requires-Dist: parcels (>=3.1.2,<4.0.0)
Requires-Dist: requests (>=2.27.1)
Requires-Dist: seaborn (>=0.13.2,<0.14.0)
Requires-Dist: xarray (==2025.1.2)
Requires-Dist: zarr (==2.18.4)
Description-Content-Type: text/markdown

<!--
SPDX-FileCopyrightText: 2025 Mercator Ocean International <https://www.mercator-ocean.eu/>

SPDX-License-Identifier: EUPL-1.2
-->

<div align="center">
  <img src="https://s3.waw3-1.cloudferro.com/oceanbench-bucket/public/logo/oceanbench-logo-light.png" alt="OceanBench logo" height="200"/>
</div>

# OceanBench
[![The latest version of OceanBench can be found on PyPI.](https://img.shields.io/pypi/v/oceanbench.svg)](https://pypi.org/project/oceanbench)
[![Link to discover EDITO](https://dive.edito.eu/badges/Powered-by-EDITO.svg)](https://dive.edito.eu)
[![Information on what versions of Python OceanBench supports can be found on PyPI.](https://img.shields.io/pypi/pyversions/oceanbench.svg)](https://pypi.org/project/oceanbench)
[![Information on what kind of operating systems OceanBench can be installed on.](https://img.shields.io/badge/platform-linux-lightgrey)](https://en.wikipedia.org/wiki/Linux)
[![Information on the OceanBench licence.](https://img.shields.io/badge/licence-EUPL-lightblue)](https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12)
[![REUSE status](https://api.reuse.software/badge/github.com/mercator-ocean/oceanbench/)](https://api.reuse.software/info/github.com/mercator-ocean/oceanbench/)

OceanBench is a benchmarking tool to evaluate ocean forecasting systems against reference ocean analysis datasets (such as 2024 [GLORYS reanalysis](https://data.marine.copernicus.eu/product/GLOBAL_MULTIYEAR_PHY_001_030) and [GLO12 analysis](https://data.marine.copernicus.eu/product/GLOBAL_ANALYSISFORECAST_PHY_001_024)) as well as observations.

## Citation

OceanBench's scientific paper is published in [NeurIPS 2025](https://neurips.cc/virtual/2025/loc/san-diego/poster/121394) and is accessible at [https://openreview.net/forum?id=wZGe1Kqs8G](https://openreview.net/forum?id=wZGe1Kqs8G).
```
@inproceedings{
  aouni2025oceanbench,
  title={OceanBench: A Benchmark for Data-Driven Global Ocean Forecasting systems},
  author={Anass El Aouni and Quentin Gaudel and Juan Emmanuel Johnson and REGNIER Charly and Julien Le Sommer and van Gennip and Ronan Fablet and Marie Drevillon and Yann DRILLET and Pierre Yves Le Traon},
  booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
  year={2025},
  url={https://openreview.net/forum?id=wZGe1Kqs8G}
}
```

## Score table and system comparison

The official score table is available on the [OceanBench website](https://oceanbench.edito.eu).
Any version, region and track can be exported as CSV, Parquet or Zarr with the Export buttons next to the pickers, or fetched through the [Scores API](https://oceanbench.edito.eu/documentation/scores-api/):

```sh
curl -O https://oceanbench.edito.eu/api/scores/index.json
```

## Open GLORYS dataset to train your ocean forecasting system

You can train your model with [GLORYS reanalysis](https://data.marine.copernicus.eu/product/GLOBAL_MULTIYEAR_PHY_001_030) from 1993 to 2019 (included).
The years covered by the benchmark must not be used for training.
From an environment with OceanBench installed, run:

```python
import oceanbench

oceanbench.datasets.training.glorys_reanalysis()
```

to open the 1/12° GLORYS dataset as a lazy [`xarray.Dataset`](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html), through the Copernicus Marine Toolbox (see [Dependency on the Copernicus Marine Service](#dependency-on-the-copernicus-marine-service)):
```
<xarray.Dataset> Size: 140TB
Dimensions:    (depth: 50, latitude: 2041, longitude: 4320, time: 9861)
Coordinates:
  * depth      (depth) float32 200B 0.494 1.541 2.646 ... 5.275e+03 5.728e+03
  * latitude   (latitude) float32 8kB -80.0 -79.92 -79.83 ... 89.83 89.92 90.0
  * longitude  (longitude) float32 17kB -180.0 -179.9 -179.8 ... 179.8 179.9
  * time       (time) datetime64[ns] 79kB 1993-01-01 1993-01-02 ... 2019-12-31
Data variables:
    thetao     (time, depth, latitude, longitude) float64 35TB dask.array<chunksize=(50, 1, 512, 2048), meta=np.ndarray>
    so         (time, depth, latitude, longitude) float64 35TB dask.array<chunksize=(50, 1, 512, 2048), meta=np.ndarray>
    uo         (time, depth, latitude, longitude) float64 35TB dask.array<chunksize=(50, 1, 512, 2048), meta=np.ndarray>
    vo         (time, depth, latitude, longitude) float64 35TB dask.array<chunksize=(50, 1, 512, 2048), meta=np.ndarray>
    zos        (time, latitude, longitude) float64 696GB dask.array<chunksize=(50, 512, 2048), meta=np.ndarray>
```

For training on the [1 degree track](https://oceanbench.edito.eu/documentation/one-degree-track/), OceanBench also exposes the 1 degree GLORYS dataset, covering the same years:

```python
import oceanbench

oceanbench.datasets.training.glorys_reanalysis_1_degree()
```

## Evaluate your system with OceanBench

The evaluation of a system runs several evaluation methods against a set of forecasts (produced by the system), namely the _challenger dataset_, opened as an [xarray Dataset](https://xarray.pydata.org/en/v2023.11.0/generated/xarray.Dataset.html) by a Python file, the _challenger file_.
It prints the score tables and writes them to a scores file, `<challenger>.<region>.scores.json`.

The OceanBench documentation describes [the shape a challenger dataset](https://oceanbench.edito.eu/documentation/shape-of-the-challenger-dataset/) must have, as well as [the definitions of the methods used to evaluate systems](https://oceanbench.edito.eu/evaluation-methods/).

### Official evaluation

All official challenger files are maintained and remain executable in order to update the scores with new OceanBench versions (all official challengers are re-evaluated with each new version).

To officially submit your system to OceanBench, please open an issue on this repository attaching one of the following:

1. A way to access the system output data in a standard format (e.g. Zarr or NetCDF).
2. A way to execute the system code or container, so that the maintainers can reproduce the output, along with clear instructions for how to run it (e.g., input/output format, required dependencies, etc.).

Optionally, attach the challenger file and the scores file resulting from an [evaluation](#evaluate-a-system).

In addition, please provide the following metadata:
- The organization that leads the construction or operation of the system.
- A link to the reference paper of the system.
- The system method. For example, "Physics-based", "ML-based" or "Hybrid".
- The system type. For example, "Forecast (deterministic)" or "Forecast (ensemble)".
- The system initial conditions. For example, "GLO12/IFS".
- The approximate horizontal resolution of the system. For example, "1/12°" or "1/4°".

### Get started evaluating your system

Follow the [Get started evaluating your system](https://oceanbench.edito.eu/documentation/get-started/) guide, or [this notebook](https://github.com/mercator-ocean/oceanbench/blob/main/assets/evaluate_challenger_example.ipynb) it is made from, to compute the OceanBench scores of a sample of the GLONET system, interpolated to 1 degree. It needs no credentials.
The notebook opens the challenger dataset and computes each score table with the functions of `oceanbench.metrics`.

You can replace the cell that opens the challenger dataset with your code and execute the notebook. The official evaluation of a system is [`oceanbench evaluate`](#evaluate-a-system).

#### Execute on your own resources

You will need to install OceanBench manually in your environment.

##### Installation

###### Using pip via PyPI

```bash
pip install oceanbench
```

The notebook also needs Jupyter:

```bash
pip install jupyter
```

###### From sources

```bash
git clone git@github.com:mercator-ocean/oceanbench.git && cd oceanbench/ && pip install --editable .
```

#### Execute on EDITO

You can open and manually execute the notebook in EDITO datalab by clicking here:
[![Link to open resource in EDITO](https://dive.edito.eu/badges/Open-in-EDITO.svg)](https://datalab.dive.edito.eu/launcher/ocean-modelling/jupyter-notebook?name=jupyter-oceanbench&resources.requests.cpu=«4000m»&resources.requests.memory=«8Gi»&resources.limits.cpu=«7200m»&resources.limits.memory=«28Gi»&advanced.init.personalInit=«https%3A%2F%2Fraw.githubusercontent.com%2Fmercator-ocean%2Foceanbench%2Frefs%2Fheads%2Fmain%2Fedito%2Fopen-jupyter-notebook-url-edito.sh»&advanced.init.personalInitArgs=«https%3A%2F%2Fraw.githubusercontent.com%2Fmercator-ocean%2Foceanbench%2Frefs%2Fheads%2Fmain%2Fassets%2Fevaluate_challenger_example.ipynb»)

### Evaluate a system

Once [installed](#installation), you can evaluate your system from the command line or using python.

#### CLI


```bash
oceanbench evaluate path/to/challenger.py
```

To evaluate multiple challengers in parallel:

```bash
oceanbench evaluate challenger_a.py challenger_b.py
```

By default, OceanBench evaluates the global domain. To evaluate an official region, pass its identifier:

```bash
oceanbench evaluate path/to/challenger.py --region ibi
```

To evaluate a custom region, write its bounding box in a JSON file:

```json
{
  "id": "western_med",
  "display_name": "Western Mediterranean",
  "bounds": {
    "minimum_latitude": 34.0,
    "maximum_latitude": 44.5,
    "minimum_longitude": -1.0,
    "maximum_longitude": 18.0
  }
}
```

Then pass it to the CLI:

```bash
oceanbench evaluate path/to/challenger.py --region-file path/to/region.json
```

Each evaluation prints the score tables and the duration of each step, and writes a scores file in the current directory.
Its name is derived from the challenger file name and region (e.g. `challenger.py` produces `challenger.global.scores.json` for the global benchmark and `challenger.ibi.scores.json` for the IBI benchmark).

To also upload the scores files to an S3 bucket:

```bash
oceanbench evaluate path/to/challenger.py --output-bucket my-bucket --output-prefix results
```

To run OceanBench on an HPC compute node without internet access, see [Run OceanBench offline](https://oceanbench.edito.eu/documentation/run-offline/).

#### Python


```python
import oceanbench

oceanbench.evaluate_challenger("path/to/file/opening/the/challenger/datasets.py")
```

It prints the score tables and writes the same scores file as the CLI.

More details in the [documentation](https://oceanbench.edito.eu/documentation/api/evaluation/).

### Dependency on the Copernicus Marine Service

Running OceanBench to evaluate systems with 1/12° resolution uses the [Copernicus Marine Toolbox](https://github.com/mercator-ocean/copernicus-marine-toolbox/) and therefore requires authentication with the [Copernicus Marine Service](https://marine.copernicus.eu/).

> If you're running OceanBench in a non-interactive way, please follow the [Copernicus Marine Toolbox documentation](https://toolbox-docs.marine.copernicus.eu) to login to the Copernicus Marine Service before running the bench.

## Contribution

Your help to improve OceanBench is welcome.
Please first read contribution instructions [here](CONTRIBUTION.md).

## License

Licensed under the [EUPL-1.2](https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12) license.

## About

Implemented by:

<a href="https://www.mercator-ocean.eu/"><img src="https://www.nemo-ocean.eu/wp-content/uploads/MOI.png" alt="Mercator Ocean International logo" height="100"/></a>

As part of a fruitful collaboration with:

<a href="https://www.ocean-climat.fr"><img src="https://oceansconnectes.org/wp-content/uploads/2023/02/VISU-PPRsmol.jpg" alt="PPR logo" height="100"/></a>
<a href="https://www.imt-atlantique.fr"><img src="https://www.imt-atlantique.fr/sites/default/files/ecole/IMT_Atlantique_logo.png" alt="IMTA logo" height="100"/></a>
<a href="https://www.univ-grenoble-alpes.fr"><img src="https://www.grenoble-inp.fr/medias/photo/logo-uga-carrousel_1575017090994-png" alt="UGA logo" height="100"/></a>
<a href="https://igeo.ucm-csic.es/"><img src="https://igeo.ufrj.br/wp-content/uploads/2022/10/image-1.png" alt="IGEO logo" height="100"/></a>

Powered by:

<a href="https://edito.eu">
  <img class="only-light" src="https://s3.waw3-1.cloudferro.com/oceanbench-bucket/public/logo/EDITO_Full_Logo.png" alt="EDITO logo"  height=100  />
</a>

