Metadata-Version: 2.3
Name: torchgeo-bench
Version: 0.4.0
Summary: Lightweight benchmarking framework for geospatial models
Keywords: benchmark,earth observation,foundation models,geospatial,machine learning,remote sensing
Author: Caleb Robinson, Isaac Corley, Nils Lehmann
Author-email: Caleb Robinson <calebrob6@gmail.com>, Isaac Corley <isaac.a.corley@gmail.com>, Nils Lehmann <nils.lehmann24@gmail.com>
License: MIT
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: GIS
Requires-Dist: faissknn[cpu]>=0.3
Requires-Dist: filelock>=3.12
Requires-Dist: geobenchv2>=0.9
Requires-Dist: h5py>=3.8
Requires-Dist: huggingface-hub>=0.20
Requires-Dist: hydra-core>=1.3
Requires-Dist: numpy>=1.24
Requires-Dist: omegaconf>=2.3
Requires-Dist: pandas>=2
Requires-Dist: rich>=13
Requires-Dist: scikit-learn>=1.3
Requires-Dist: timm>=0.9
Requires-Dist: torch>=2
Requires-Dist: torchgeo>=0.9
Requires-Dist: torchmetrics>=1.4
Requires-Dist: torchvision>=0.15
Requires-Dist: typer>=0.9
Requires-Dist: torchgeo-bench[cleanlab] ; extra == 'all'
Requires-Dist: torchgeo-bench[dev] ; extra == 'all'
Requires-Dist: torchgeo-bench[docs] ; extra == 'all'
Requires-Dist: torchgeo-bench[id] ; extra == 'all'
Requires-Dist: torchgeo-bench[olmoearth] ; extra == 'all'
Requires-Dist: torchgeo-bench[sam3] ; extra == 'all'
Requires-Dist: torchgeo-bench[terratorch] ; extra == 'all'
Requires-Dist: torchgeo-bench[viz] ; extra == 'all'
Requires-Dist: cleanlab>=2.7 ; extra == 'cleanlab'
Requires-Dist: imagehash>=4.3 ; extra == 'cleanlab'
Requires-Dist: matplotlib>=3.5 ; extra == 'cleanlab'
Requires-Dist: pillow>=9 ; extra == 'cleanlab'
Requires-Dist: faissknn[cuda]>=0.3 ; extra == 'cuda'
Requires-Dist: mdformat>=0.7.22 ; extra == 'dev'
Requires-Dist: pre-commit>=4.3 ; extra == 'dev'
Requires-Dist: pyproject-fmt>=2.21 ; extra == 'dev'
Requires-Dist: pytest>=7.3 ; extra == 'dev'
Requires-Dist: pytest-cov>=4.1 ; extra == 'dev'
Requires-Dist: pytest-xdist>=3.6 ; extra == 'dev'
Requires-Dist: ruff>=0.0.270 ; extra == 'dev'
Requires-Dist: linkify-it-py>=2 ; extra == 'docs'
Requires-Dist: myst-parser>=2 ; extra == 'docs'
Requires-Dist: pydata-sphinx-theme>=0.15 ; extra == 'docs'
Requires-Dist: sphinx>=7 ; extra == 'docs'
Requires-Dist: sphinx-copybutton>=0.5 ; extra == 'docs'
Requires-Dist: sphinx-design>=0.5 ; extra == 'docs'
Requires-Dist: torchid>=0.3 ; python_full_version >= '3.13' and extra == 'id'
Requires-Dist: olmoearth-pretrain-minimal>=0.0.5 ; extra == 'olmoearth'
Requires-Dist: transformers>=4.50 ; extra == 'sam3'
Requires-Dist: terratorch>=1 ; extra == 'terratorch'
Requires-Dist: matplotlib>=3.5 ; extra == 'viz'
Requires-Dist: pillow>=9 ; extra == 'viz'
Requires-Python: >=3.12
Project-URL: Documentation, https://torchgeo.org/torchgeo-bench/
Project-URL: Homepage, https://github.com/torchgeo/torchgeo-bench
Project-URL: Issues, https://github.com/torchgeo/torchgeo-bench/issues
Project-URL: Repository, https://github.com/torchgeo/torchgeo-bench
Provides-Extra: all
Provides-Extra: cleanlab
Provides-Extra: cuda
Provides-Extra: dev
Provides-Extra: docs
Provides-Extra: id
Provides-Extra: olmoearth
Provides-Extra: sam3
Provides-Extra: terratorch
Provides-Extra: viz
Description-Content-Type: text/markdown

# torchgeo-bench

[![CI](https://github.com/torchgeo/torchgeo-bench/actions/workflows/ci.yaml/badge.svg)](https://github.com/torchgeo/torchgeo-bench/actions/workflows/ci.yaml)
[![PyPI version](https://img.shields.io/pypi/v/torchgeo-bench.svg)](https://pypi.org/project/torchgeo-bench/)
[![Python 3.12+](https://img.shields.io/pypi/pyversions/torchgeo-bench.svg)](https://pypi.org/project/torchgeo-bench/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE)

A lightweight benchmarking framework for evaluating **frozen** geospatial
foundation models on the GeoBench V1 and V2 suites. Plug in any backbone, get
KNN-5 / linear-probe accuracy on classification datasets and mIoU on
segmentation datasets, with bootstrapped 95% confidence intervals — all
configured through Hydra.

- **Frozen-backbone evaluation** — KNN-5, L-BFGS logistic regression, and
  linear / conv / FPN / DPT segmentation probes.
- **GeoBench V1 + V2 built in** — classification and segmentation, RGB or
  full multispectral / multi-modal stacks.
- **Hydra-driven** — sweep models, datasets, partitions, image sizes, and
  bands without code changes.
- **Resumable** — `resume=true` skips already-computed `(dataset, method,
  model, …)` rows. Atomic CSV appends are safe across parallel jobs.
- **Bring your own model** — copy
  [`contrib_template.py`](src/torchgeo_bench/models/contrib_template.py),
  implement `_forward_patch_features`, and add a one-file Hydra config.
  See the [Stage 1 guide](https://torchgeo.org/torchgeo-bench/user/eval_own_model.html)
  for a full walkthrough, or the
  [Stage 2 guide](https://torchgeo.org/torchgeo-bench/user/contribute_model.html)
  to contribute the model back upstream.

## Installation

```bash
pip install torchgeo-bench
```

For development:

```bash
git clone https://github.com/torchgeo/torchgeo-bench
cd torchgeo-bench
uv sync --extra dev
```

For GPU-accelerated KNN (Linux + CUDA 12 + glibc ≥ 2.28):

```bash
pip install 'torchgeo-bench[cuda]'
```

Requires Python 3.12+. The default (CPU) install runs on **Linux**, **macOS**,
and **Windows**; GPU-accelerated KNN (the `[cuda]` extra) is Linux-only.

## Download a dataset

The runner expects datasets under `./data/`. To grab GeoBench V1:

```bash
torchgeo-bench download geobench_v1
```

V2 (classification + segmentation) and torchgeo's EuroSAT downloader work the
same way (`torchgeo-bench download geobench_v2`, `torchgeo-bench download
eurosat`). See the [documentation](https://torchgeo.org/torchgeo-bench/user/datasets.html)
for all options.

## Run a basic experiment

```bash
# Default: random convolutional features (RCF) on every available dataset
torchgeo-bench run

# A single dataset with a pretrained ImageNet ResNet-50
torchgeo-bench run model=timm/resnet50 dataset.names=[m-eurosat]
```

The default device is `cuda:0`. On a machine without a working CUDA GPU (or if
a GPU run crashes — see [troubleshooting](https://torchgeo.org/torchgeo-bench/user/troubleshooting.html)),
fall back to CPU:

```bash
torchgeo-bench run dataset.names=[m-eurosat] device=cpu
```

Results are appended to `results/all_results.csv`, which **ships pre-populated
with reference results** — to start from a clean slate, write to your own file
with `output=results/my_run.csv`. Re-run with `resume=true` to skip
already-completed rows.

<!-- skip-on-docs-landing-start -->
## Learn more

- **[Documentation](https://torchgeo.org/torchgeo-bench/)** — full
  configuration reference, available models, dataset tables, multi-band
  experiments, evaluation methodology, output schema, dev / release
  workflow, and troubleshooting.
- **[AGENTS.md](https://github.com/torchgeo/torchgeo-bench/blob/main/AGENTS.md)**
  — contributor guide and house style.
<!-- skip-on-docs-landing-end -->

## Citation

If you use this framework, please cite it (once the `torchgeo-bench` paper is
available):

```bibtex
@misc{torchgeobench,
  title  = {torchgeo-bench: A lightweight benchmarking framework for geospatial foundation models},
  author = {torchgeo-bench Contributors},
  year   = {TBD},
  note   = {Software},
  url    = {https://github.com/torchgeo/torchgeo-bench}
}
```

## License

[MIT](LICENSE).
