Metadata-Version: 2.4
Name: geobench
Version: 1.1.0
Summary: A benchmark designed to advance foundation models for Earth monitoring, tailored for remote sensing. It encompasses six classification and six segmentation tasks, curated for precision and model evaluation. The package also features a comprehensive evaluation methodology and showcases results from 20 established baseline models.
Keywords: benchmark,earth observation,foundation models,geospatial,machine learning,remote sensing
Author: Alexandre Lacoste, Nils Lehmann, Pau Rodriguez, Evan David Sherwin, Hannah Kerner, Björn Lütjens, Jeremy Andrew Irvin, David Dao, Hamed Alemohammad, Alexandre Drouin, Mehmet Gunturkun, Gabriel Huang, David Vazquez, Dava Newman, Yoshua Bengio, Stefano Ermon, Xiao Xiang Zhu
Author-email: Alexandre Lacoste <alexandre.lacoste@servicenow.com>
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: GIS
Classifier: Topic :: Scientific/Engineering :: Image Processing
Requires-Dist: affine<3
Requires-Dist: h5py>=3.10
Requires-Dist: huggingface-hub>=0.19.3
Requires-Dist: numpy>=1.26
Requires-Dist: rasterio>=1.3.9
Requires-Dist: scipy>=1.11.2
Requires-Dist: tqdm>=4.65
Requires-Dist: matplotlib>=3.8 ; extra == 'plot'
Requires-Dist: pandas>=2 ; extra == 'plot'
Requires-Dist: seaborn>=0.13 ; extra == 'plot'
Requires-Dist: pytorch-lightning>=2.1 ; extra == 'torch'
Requires-Dist: torch>=2.1 ; extra == 'torch'
Requires-Python: >=3.12
Project-URL: Dataset, https://huggingface.co/datasets/recursix/geo-bench-1.0
Project-URL: Paper, https://arxiv.org/abs/2306.03831
Project-URL: Repository, https://github.com/ServiceNow/geo-bench
Provides-Extra: plot
Provides-Extra: torch
Description-Content-Type: text/markdown

# GEO-Bench: Toward Foundation Models for Earth Monitoring

GEO-Bench is a [ServiceNow Research](https://www.servicenow.com/research) project. 

[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Language: Python](https://img.shields.io/badge/language-Python%203.12%2B-green?logo=python&logoColor=green)](https://www.python.org)

> [!IMPORTANT]
> **We recommend using [GEO-Bench-2](https://the-ai-alliance.github.io/GEO-Bench-2/) instead of
> this benchmark.** GEO-Bench-2 is the successor to GEO-Bench, published by the AI Alliance
> together with IBM, ServiceNow, and TUM. It adds multi-modal and multi-temporal tasks, control
> over band ordering and normalization, and geospatial lat/lon coordinates for the majority of its
> datasets. See its [paper](https://arxiv.org/abs/2511.15658) for details.
>
> If you are interested in benchmarks and benchmarking your own model, see
> [torchgeo-bench](https://github.com/torchgeo/torchgeo-bench) for an actively maintained
> evaluation harness on GEO-Bench datasets.

GEO-Bench is a **G**eneral **E**arth **O**bservation benchmark for evaluating the performances of large pre-trained models on geospatial data. Read the [full paper](https://arxiv.org/abs/2306.03831) for usage details and evaluation of existing pre-trained vision models.

<img src="https://github.com/ServiceNow/geo-bench/raw/main/banner.png" width="500" />

## Installation

GEO-Bench requires Python 3.12 or newer and is installed from PyPI:

```console
pip install geobench
```

The base install is enough to download and load the benchmark. The plotting helpers in
`geobench.plot_tools` additionally need the `plot` extra (matplotlib, pandas, seaborn), and the
PyTorch Lightning data module in `geobench.torch_toolbox` needs the `torch` extra:

```console
pip install "geobench[plot,torch]"
```

Release `1.0.0` capped Python below 3.13 and pinned outdated upper bounds on its dependencies.
Release `1.1.0` removes both, so installing from `main` is no longer necessary.

## Downloading the data

Set `$GEO_BENCH_DIR` to your preferred location. If not set, it will be stored in `$HOME/dataset/geobench`.

Next, use the [download script](https://github.com/ServiceNow/geo-bench/blob/main/geobench/geobench_download.py). This will automatically download from [Hugging Face](https://huggingface.co/datasets/recursix/geo-bench-1.0)

Run the command:

```console
geobench-download
```

You need ~65 GB of free disk space for download and unzip (once all .zip are deleted it takes 57GB).
If some files are already downloaded, it will verify the md5 checksum. Feel free to restart the downloader if it is interrupted.

## Using data you already have

If the benchmark is already on disk, either from an earlier download or from a copy shared with you,
there are two ways to point geobench at it.

The first is `$GEO_BENCH_DIR`, which is read when `geobench` is imported and therefore has to be set
in the environment before Python starts:

```console
export GEO_BENCH_DIR=/path/to/geobench
```

Setting `os.environ["GEO_BENCH_DIR"]` or reassigning `geobench.GEO_BENCH_DIR` after the import has
no effect.

The second is `benchmark_dir`, which reads a benchmark from elsewhere without changing that
default. The tasks `task_iterator` yields read their datasets from the directory given here too.

```python
import geobench

for task in geobench.task_iterator(benchmark_dir="/shared/geobench/classification_v1.0"):
    dataset = task.get_dataset(split="train")
```

`benchmark_dir` is the path to a single benchmark rather than to the directory holding several of
them, and it takes the place of both `$GEO_BENCH_DIR` and `benchmark_name`.

## Test installation
You can run tests. 
Note: Make sure the benchmark is downloaded before launching tests.

```console
pip install pytest
```

```console
geobench-test
```

## Loading Datasets

See [`example_load_dataset.py`](https://github.com/ServiceNow/geo-bench/blob/main/geobench/example_load_datasets.py) for how to iterate over datasets.

```python
import geobench

for task in geobench.task_iterator(benchmark_name="classification_v1.0"):
    dataset = task.get_dataset(split="train")
    sample = dataset[0]
    for band in sample.bands:
        print(f"{band.band_info.name}: {band.data.shape}")
```

## Known issues

The `m-eurosat` and `m-brick-kiln` datasets in `classification_v1.0` record the wrong Sentinel-2
band for most of their channels. The pixel data is unaffected; only the band name and wavelength
stored for each channel are wrong, so selecting channels by band name returns the wrong channel.
Reported by @gabrieltseng in [#28](https://github.com/ServiceNow/geo-bench/issues/28) and
[#29](https://github.com/ServiceNow/geo-bench/issues/29).

The 13 channels are actually in this order:

| Channel | `m-eurosat` | `m-brick-kiln` |
| --- | --- | --- |
| 0–4 | B01–B05 | B01–B05 |
| 5 | B06 | B07 |
| 6 | B07 | B8A |
| 7 | B08 | B08 |
| 8 | B09 | B11 |
| 9 | B10 | B12 |
| 10 | B11 | TCI_R |
| 11 | B12 | TCI_G |
| 12 | B8A | TCI_B |

The stored metadata instead labels all 13 channels, in both datasets, as
`B01, B02, B03, B04, B05, B06, B07, B08, B8A, B09, B10, B11, B12`. In `m-brick-kiln` the source
pipeline ([mliu356/kiln-scaling](https://github.com/mliu356/kiln-scaling)) does not export `B06`,
`B09` or `B10`, and its last three channels are 8-bit true-colour composites rather than reflectance
bands.

The converters in `make_benchmark/dataset_converters/` now record the order above; the data on
Hugging Face is not regenerated, so apply this mapping when loading it. Loading either dataset
through `GeobenchDataset` emits a warning to this effect.

## Fine-tuning and reproducing experiments

See the code for reproducing experiments as a starting point for fine-tuning:

[geo-bench-experiments](https://github.com/ServiceNow/geo-bench-experiments)

## Visualizing Results

See the notebook [`baseline_results.ipynb`](https://github.com/ServiceNow/geo-bench/blob/main/geobench/baseline_results.ipynb) for an example of how to visualize the results.


