Metadata-Version: 2.5
Name: mlcast
Version: 0.2.0
Summary: Machine learning weather nowcasting library
Project-URL: Documentation, https://github.com/mlcast-community/mlcast
Project-URL: Homepage, https://github.com/mlcast-community/mlcast
Project-URL: Issues, https://github.com/mlcast-community/mlcast/issues
Project-URL: Repository, https://github.com/mlcast-community/mlcast
Author-email: Gabriele Franch <franch@fbk.eu>, Leif Denby <lcd@dmi.dk>, Oscar Dewasmes <oscar.dewasmes@meteo.fr>
Maintainer-email: MLCast Community <franch@fbk.eu>
License: Apache-2.0
License-File: LICENSE
License-File: LICENSE-APACHE
License-File: LICENSE-BSD
Keywords: machine-learning,meteorology,nowcasting,precipitation,pytorch,radar,weather
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Atmospheric Science
Requires-Python: >=3.11
Requires-Dist: absl-py>=2.4
Requires-Dist: beartype>=0.18
Requires-Dist: bottleneck>=1.6
Requires-Dist: cf-xarray>=0.10
Requires-Dist: etils>=1.13
Requires-Dist: fiddle>=0.3
Requires-Dist: importlib-resources>=5
Requires-Dist: ipykernel>=6.29.5
Requires-Dist: jaxtyping>=0.2
Requires-Dist: loguru>=0.7.3
Requires-Dist: mlflow[mlflow]>=3.11.1
Requires-Dist: numpy>=2.2.6
Requires-Dist: nvidia-ml-py[mlflow]>=13.595.45
Requires-Dist: pandas>=2
Requires-Dist: psutil[mlflow]>=7.2.2
Requires-Dist: pyarrow>=17
Requires-Dist: pydantic>=2
Requires-Dist: pyyaml>=6
Requires-Dist: rich>=13
Requires-Dist: s3fs>=2026.3
Requires-Dist: tensorboard>=2.20
Requires-Dist: xarray>=2025.6.1
Requires-Dist: zarr>=3
Provides-Extra: cpu
Requires-Dist: pytorch-lightning>=2; extra == 'cpu'
Requires-Dist: torch>=2.7.1; extra == 'cpu'
Requires-Dist: torchvision>=0.20; extra == 'cpu'
Provides-Extra: dev
Requires-Dist: aiohttp>=3.9.3; extra == 'dev'
Requires-Dist: fsspec>=2024.2; extra == 'dev'
Requires-Dist: mypy>=1; extra == 'dev'
Requires-Dist: pre-commit>=3.5; extra == 'dev'
Requires-Dist: pytest-cov>=4.1; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.8; extra == 'dev'
Requires-Dist: s3fs>=2024.2; extra == 'dev'
Provides-Extra: gpu-cu126
Requires-Dist: pytorch-lightning>=2; extra == 'gpu-cu126'
Requires-Dist: torch>=2.7.1; extra == 'gpu-cu126'
Requires-Dist: torchvision>=0.20; extra == 'gpu-cu126'
Provides-Extra: gpu-cu128
Requires-Dist: pytorch-lightning>=2; extra == 'gpu-cu128'
Requires-Dist: torch>=2.7.1; extra == 'gpu-cu128'
Requires-Dist: torchvision>=0.20; extra == 'gpu-cu128'
Provides-Extra: gpu-cu130
Requires-Dist: pytorch-lightning>=2; extra == 'gpu-cu130'
Requires-Dist: torch>=2.7.1; extra == 'gpu-cu130'
Requires-Dist: torchvision>=0.20; extra == 'gpu-cu130'
Description-Content-Type: text/markdown

# mlcast

<!-- SPDX-License-Identifier: Apache-2.0 OR BSD-3-Clause -->

> ⚠️ This package is under active development. The API and functionality are subject to change until the v1.0.0 release.

The MLCast Community is a collaborative effort bringing together meteorological services, research institutions, and academia across Europe to develop a unified Python package for AI-based nowcasting. This is an initiative of the E-AI WG6 (Nowcasting) of EUMETNET.

This repo contains the `mlcast` package for machine learning-based weather nowcasting.

## Installation

As `mlcast` is in rapid development — the recommended path is to clone locally,
rather than installing a pinned release from PyPI.

### Local development: clone and install locally with uv

[Fork the repository](https://github.com/mlcast-community/mlcast/fork) on GitHub first, then clone your fork. This lets you track
upstream changes while keeping your own modifications on a separate branch:

```bash
git clone https://github.com/<your-github-username>/mlcast
cd mlcast

# Install uv if not already installed
curl -LsSf https://astral.sh/uv/install.sh | sh

# And install dependencies, picking the PyTorch build for your hardware.
# CPU only
uv sync --extra cpu

# GPU — CUDA 12.6
uv sync --extra gpu-cu126

# GPU — CUDA 12.8
uv sync --extra gpu-cu128

# GPU — CUDA 13.0
uv sync --extra gpu-cu130
```

PyTorch is only installed through one of these extras, so pick one: plain `uv sync` leaves it out. After this
one-time sync, `uv run mlcast ...` keeps the PyTorch build you chose.

Next you can jump to [using mlcast](#usage) or, if you intend to modify the code, setup the development toolchain as described below:

```bash

# Install dev dependencies, together with the PyTorch extra you picked above
# (`uv sync` removes extras that aren't listed, including PyTorch)
uv sync --extra cpu --extra dev

# Install the pre-commit git hook (runs checks automatically on every commit)
uv run pre-commit install
```

### PyPI release

Tagged releases are published to PyPI and can be installed with pip. PyTorch comes through an extra, so include
one:

```bash
pip install "mlcast[cpu]"
```

With pip, all of the extras (`cpu`, `gpu-cu126`, `gpu-cu128`, `gpu-cu130`) install the default PyTorch wheels from
PyPI. Install PyTorch first from the [PyTorch index](https://pytorch.org/get-started/locally/) if you need a
specific build.

For active development or access to unreleased changes, clone the repository and install locally with `uv` as described above.

## Usage

mlcast exposes two interfaces for training: a **command-line interface (CLI)**
for interactive and scripted use, and a **Python API** for programmatic control.
Both are built on [Fiddle](https://github.com/google/fiddle) — a configuration
library that lets you build a full experiment graph, override any parameter, and
reproduce runs exactly from a saved YAML file.

### Configuration model

Training in mlcast is currently built around a single base configuration
function, [`training_experiment`](src/mlcast/config/base.py), which defines the
default ConvGRU ensemble nowcasting setup: dataset, data module, network,
Lightning module, and trainer. Rather than writing a new config from scratch,
the intended workflow is to start from this base and apply targeted
modifications:

- **`set:` overrides** — change a single scalar parameter (e.g. batch size,
  learning rate, number of epochs)
- **fiddlers** — apply a named mutator function that keeps multiple related
  parameters in sync (e.g. switching the dataset class, toggling masking,
  changing the logger)
- **direct graph edits** (Python API only) — replace a sub-object entirely,
  for example swapping in a different network architecture

Any combination of these can be layered on top of the base config, and the
fully resolved config is always saved to YAML alongside the training logs so
runs can be reproduced exactly.

The diagram below shows the full default config graph as built by
[`training_experiment`](src/mlcast/config/base.py):

![training_experiment config graph](docs/config_diagram.svg)

### Command-line interface

Install the package and run:

```bash
mlcast train
```

This trains with the built-in [`training_experiment`](src/mlcast/config/base.py) defaults. All parameters
are controlled via `--config` flags:

| Prefix | Purpose | Example |
|--------|---------|---------|
| *(none)* | Use the built-in default config | `mlcast train` |
| `set:` | Override a single parameter | `--config set:data.batch_size=32` |
| `fiddler:` | Apply a semantic mutator (multi-param change) | `--config fiddler:use_random_sampler` |
| `config:` | Switch to a different `@auto_config` function | `--config=config:my_experiment` |
| `path/to/config.yaml` | Load a previously saved config | `--config saved.yaml` |

Multiple `--config` flags are applied in order and can be combined freely.

**Examples:**

```bash
# Override dataset path and batch size
mlcast train \
    --config set:data.dataset_factory.zarr_path=/data/radar.zarr \
    --config set:data.batch_size=32

# Switch to random sampler and log to MLflow
mlcast train \
    --config fiddler:use_random_sampler \
    --config fiddler:use_mlflow_logger

# Resume from a saved config with an epoch override
mlcast train \
    --config logs/mlcast/version_0/config.yaml \
    --config set:trainer.max_epochs=50

# Inspect the fully resolved config without starting training
mlcast train --config fiddler:use_random_sampler --print_config_and_exit
```

Run `mlcast train --help` for a full list of examples and available fiddlers.

Beyond training, the CLI provides two data-prep subcommands —
`mlcast build-sampling-index` and `mlcast validate-sampling-index` — for building
and checking the sampling index that training reads. See
[Preparing training data](#preparing-training-data).

### Python API

The Python API gives you full programmatic control over the config graph before
anything is instantiated.

**Run the default experiment with tweaks:**

```python
import fiddle as fdl
from mlcast.config import training_experiment, train_from_config
from mlcast.config.fiddlers import use_random_sampler

cfg = training_experiment.as_buildable()  # returns a fdl.Config graph — see src/mlcast/config/base.py

# Apply a fiddler to switch the dataset sampler
use_random_sampler(cfg)

# Override individual parameters directly on the config graph
cfg.data.batch_size = 32
cfg.trainer.max_epochs = 50

# Validates cross-parameter contracts, builds all objects, persists config
# YAML to the active logger, then calls trainer.fit() + trainer.test()
train_from_config(cfg)
```

**Custom network architecture:**

You can swap in any architecture by replacing `cfg.pl_module.network` with a
`fdl.Config` node.  The network must implement the nowcasting forward
interface — see [Custom network interface](#custom-network-interface) below.

As an example, here is how to wrap an
[mfai](https://github.com/meteofrance/mfai) `HalfUNet` (a plain single-step
U-Net) to satisfy the interface.  The wrapper channel-stacks the past frames
and runs the U-Net autoregressively for each requested forecast step:

> **Note** — `input_steps` equals `dataset_factory.input_steps` (6 by
> default) and is directly readable from the config graph before building.

```python
import einops
import fiddle as fdl
import torch
import torch.nn as nn
from jaxtyping import Float
from mfai.torch.models import HalfUNet
from mlcast.config import training_experiment, train_from_config
from mlcast.config.fiddlers import use_random_sampler

# Minimal adapter: channel-stack past frames → HalfUNet → one step at a time.
# NowcastLightningModule calls network(x, steps=N, ensemble_size=M), so any
# custom network must accept those keyword arguments.
class HalfUNetNowcaster(nn.Module):
    def __init__(self, input_steps: int = 6, num_vars: int = 1):
        super().__init__()
        self.input_steps = input_steps
        self.num_vars = num_vars
        self.unet = HalfUNet(
            input_shape=(256, 256),
            in_channels=input_steps * num_vars,
            out_channels=num_vars,
            settings=fdl.Config(HalfUNet.settings_kls),
        )

    @property
    def input_channels(self) -> int:
        # Externally, the HalfUNetNowcaster respects the required input shape structure
        # (batch, input_steps, num_vars, H, W), even though the internal U-Net is channel-stacked.
        # Adding this property allows the config consistency checks to verify that
        # the dataset and model agree on the expected number of input channels.
        return self.num_vars

    def forward(
        self,
        x: Float[torch.Tensor, "batch input_steps in_channels H W"],
        steps: int,
        ensemble_size: int = 1,
    ) -> Float[torch.Tensor, "batch steps out_channels H W"]:
        # channel-stack all input frames: (b, t, c, h, w) -> (b, t*c, h, w)
        x_flat = einops.rearrange(x, "b t c h w -> b (t c) h w")
        preds = []
        for _ in range(steps):
            y = self.unet(x_flat)   # [B, num_vars, H, W]
            preds.append(y.unsqueeze(1))
            # slide window: drop the oldest timestep (first num_vars channels),
            # append the latest prediction as the newest timestep
            x_flat = torch.cat([x_flat[:, self.num_vars:], y], dim=1)
        return torch.cat(preds, dim=1)

cfg = training_experiment.as_buildable()
use_random_sampler(cfg)

cfg.pl_module.network = fdl.Config(
    HalfUNetNowcaster,
    input_steps=cfg.data.dataset_factory.input_steps,
    num_vars=len(cfg.data.dataset_factory.standard_names),
)

train_from_config(cfg)
```

For lower-level control you can call the steps of `train_from_config` individually:

```python
import fiddle as fdl
from mlcast.config.consistency_checks import validate_config

validate_config(cfg)          # raises ValueError on any contract violation
experiment = fdl.build(cfg)   # instantiates all objects
experiment.run()              # trainer.fit() + trainer.test()
```

### Available fiddlers

| Fiddler | Arguments | What it does |
|---------|-----------|--------------|
| `use_mlflow_logger` | *(none)* | Replaces the default `TensorBoardLogger` with `MLFlowLogger` and appends `LogSystemInfoCallback`; respects the `MLFLOW_TRACKING_URI` environment variable |
| `set_variables` | `standard_names` | Sets the list of input variables on the dataset and updates `network.input_channels` to match |
| `toggle_masking` | `enabled` | Toggles masked-loss mode by setting both `dataset_factory.return_mask` and `pl_module.masked_loss` to the same value |
| `use_anon_s3_dataset` | `zarr_path`, `endpoint_url` | Points the dataset at an anonymous S3 object store; sets `zarr_path` and the required `storage_options` together |
| `use_ratio_splits` | `train`, `val` | Sets fraction-based train/val/test time splits on the data module (test = 1 − train − val) |
| `use_random_sampler` | *(none)* | Switches the dataset factory to the on-the-fly random sampler (useful during development when no precomputed sampling index is available) |

## Preparing training data

Training reads a **sampling index**: a parquet file listing every candidate
datacube — a fixed-size `time_depth × width × height` crop of a source radar
Zarr — together with its **sample stats** (`nan_count`, `sum`, `mean`,
`frac_wet`), while the **sampling parameters** used to build it are stored in
the file's metadata. The training dataset
[`SourceDataIndexedDataset`](src/mlcast/data/source_data_datasets.py) iterates
this index (it is the dataset factory's `index_path`), and a **candidate
selector** picks the training subset at dataset init. The index builder, its
contract, and the selectors all live in
[`mlcast.data.source_data.sampling`](src/mlcast/data/source_data/sampling/);
two CLI subcommands build and check the index.

### `mlcast build-sampling-index` — scan a Zarr dataset → sampling index

Slides a window over the `(time, x, y)` grid and, for every candidate datacube,
computes its sample stats in O(1) per window via a cumulative-sum trick.
Windows are filtered by a maximum-NaN budget, a spatial/temporal stride, and
time-continuity (no frame gaps); the survivors are streamed to a single
zstd-compressed parquet whose footer carries every sampling parameter as
metadata (the contract in
[`sampling_index_spec`](src/mlcast/data/source_data/sampling/sampling_index_spec.py)),
so downstream commands never have to parse the filename.

```bash
# A year of radar → 24-frame 256×256 datacubes, stride 3×16×16
mlcast build-sampling-index /data/radar.zarr \
    --start-date 2020-01-01 --end-date 2020-12-31 \
    --time-depth 24 --width 256 --height 256 \
    --step-t 3 --step-x 16 --step-y 16 \
    --max-nan 10000 \
    -o sampling_index_2020.parquet
```

Common flags (`mlcast build-sampling-index -h` lists them all):

| Flag | Default | Purpose |
|------|---------|---------|
| `--time-depth` / `--width` / `--height` | 24 / 256 / 256 | Datacube shape `(T, X, Y)` |
| `--step-t` / `--step-x` / `--step-y` | 3 / 16 / 16 | Stride between candidate window starts |
| `--max-nan` | 10000 | Drop any datacube with more NaNs than this |
| `--wet-threshold` | auto | Wet-pixel threshold; auto = 0.1 mm/h (rain rate) or 7 dBZ (reflectivity) |
| `--device` | auto | Compute backend: `auto` / `cpu` (bottleneck) / `cuda` |
| `--workers` | 8 | CPU worker processes, or GPU chunk-reader threads |
| `--data-var` / `--time-var` | RR / time | Names of the Zarr data and time variables |
| `-o` / `--output` | auto | Output path; auto-named from the parameters if omitted |

### `mlcast validate-sampling-index` — check an index against the contract

Checks a sampling index's column schema, sampling parameters, and (unless
`--no-data-checks`) its per-row value invariants, then prints the parameters,
the table structure, and a preview of the first 10 rows.

```bash
mlcast validate-sampling-index sampling_index_2020.parquet

# Footer only — schema + sampling parameters, skip the per-row checks
mlcast validate-sampling-index sampling_index_2020.parquet --no-data-checks
```

### Candidate selectors

At training time [`SourceDataDataModule`](src/mlcast/data/source_data_datamodule.py)
applies a `CandidateSelector` to the index **once**, at dataset init, turning
the full candidate pool into the training set via a per-row keep/discard draw —
so the dataset length is fixed and known up front, and the same set is reused
every epoch (candidates are not re-drawn). Selectors are pluggable through a
registry in [`selection`](src/mlcast/data/source_data/sampling/selection.py):

| Selector | Parameters | What it does |
|----------|------------|--------------|
| `UniformSelector` | `keep_fraction` | Keep each candidate with a fixed probability, independent of its stats |
| `ImportanceSelector` | `column`, `q_min`, `scale`, `mean_weight` | Keep each candidate with probability rising with one of its sample stats (`mean` by default) — oversampling high-rainfall datacubes without duplication |

The default config sets `train_selector=ImportanceSelector()` and
`eval_selector=UniformSelector(keep_fraction=0.1)`, so importance sampling
reshapes only training while validation and test stay representative. Add a
scheme by subclassing `CandidateSelector`, decorating it with
`@register_selector("name")`, and selecting it from a config via
`get_selector("name", ...)`.

## Project Structure

```
mlcast/
├── src/mlcast/
│   ├── __main__.py                      # CLI entry point (train / build-sampling-index / validate-sampling-index)
│   ├── nowcasting_module.py             # Generic Lightning module for nowcasting
│   ├── losses.py                        # CRPS, AFCRPS, MSE loss functions
│   ├── callbacks.py                     # Training callbacks
│   ├── visualization.py                 # TensorBoard image logging helpers
│   ├── config/
│   │   ├── base.py                      # Default training_experiment @auto_config
│   │   ├── fiddlers.py                  # Semantic config mutators
│   │   ├── consistency_checks.py        # Cross-parameter validation
│   │   ├── loader.py                    # YAML config loader
│   │   └── orchestrator.py             # train_from_config, config persistence
│   ├── data/
│   │   ├── source_data_datamodule.py    # Lightning DataModule
│   │   ├── source_data_datasets.py      # Zarr-backed PyTorch datasets (read the sampling index)
│   │   ├── normalization.py             # Normalisation registry
│   │   └── source_data/sampling/        # Data-prep: sampling index builder + candidate selectors
│   │       ├── commands/                # build-sampling-index / validate-sampling-index CLI
│   │       ├── selection.py             # Candidate selector registry (uniform, importance)
│   │       ├── sampling_index_spec.py   # Sampling index schema + validation (the contract)
│   │       └── units.py                 # Data-kind + wet-threshold detection
│   └── models/
│       └── convgru.py                   # ConvGRU encoder-decoder
├── tests/
├── pyproject.toml
└── README.md
```

## Implemented architectures

### ConvGruModel

`ConvGruModel` (in `src/mlcast/models/convgru.py`) is an **encoder-decoder**
architecture.  It is **not autoregressive at forecast time**: rather than
generating each forecast frame from the previous predicted frame, the decoder
performs a temporal roll-out entirely in **latent space** — the ConvGRU at
each spatial scale unrolls over `forecast_steps` steps driven by noise or
zeros, with its hidden state initialised from the encoder.  Forecast frames
are only materialised at the end, by upsampling the final decoder hidden
states back to the original spatial resolution.

**Encoding** — a stack of `EncoderBlock` layers unrolls a ConvGRU
sequentially over the `input_steps` real observed frames.  Each block halves
the spatial resolution via `PixelUnshuffle(2)`.  The last hidden state of
each block is retained.

**Decoding** — a stack of `DecoderBlock` layers performs a latent-space
roll-out at each spatial scale.  Each decoder block's ConvGRU is initialised
with the final hidden state from the corresponding encoder block, then unrolls
over `forecast_steps` steps with noise or zeros as input — so the forecast
sequence emerges from the evolution of hidden states across multiple spatial
scales, never from feeding predictions back as inputs.  Spatial resolution is
doubled at each block via `PixelShuffle(2)`.

**Ensemble** — when `ensemble_size > 1` the decoder is run `ensemble_size`
times, each time with freshly sampled Gaussian noise.  The results are
concatenated along the channel dimension.

**Deterministic variant** ([diagram source](https://docs.google.com/presentation/d/1U2Y9vZADXTsgQBNiWYAgOwYeMPVu7TOk/edit?slide=id.p6#slide=id.p6)):

![ConvGruModel deterministic architecture](docs/architectures/convgru-deterministic.png)

**Stochastic / ensemble variant** ([diagram source](https://docs.google.com/presentation/d/1U2Y9vZADXTsgQBNiWYAgOwYeMPVu7TOk/edit?slide=id.p7#slide=id.p7)):

![ConvGruModel stochastic architecture](docs/architectures/convgru-stochastic.png)


### Custom network interface

Any network architecture can be used by replacing `cfg.pl_module.network`
with a `fdl.Config` node pointing at your class.  The only requirement is
that `forward` accepts the following signature:

```python
# from jaxtyping import Float
# import torch

def forward(
    self,
    x: Float[torch.Tensor, "batch input_steps in_channels H W"],
    steps: int,          # number of forecast steps to produce
    ensemble_size: int,  # number of stochastic ensemble members
) -> Float[torch.Tensor, "batch steps out_channels H W"]:
    ...
```

If your network uses a different parameter name for the input channel count
than `input_channels` (the default assumed by `ConvGruModel` and the
`set_variables` fiddler), set it explicitly on the config node.

## Contributing

Please feel free to raise issues or PRs if you have any suggestions or questions.

## Links to presentations for discussion about the API

- [2025/02/04 first design discussions](https://docs.google.com/presentation/d/1oWmnyxOfUMWgeQi0XyX4fX9YDMX1vl6h/edit?usp=drive_link&rtpof=true&sd=true)

## License

This project is dual-licensed under either:

* Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
* BSD 3-Clause License ([LICENSE-BSD](LICENSE-BSD) or https://opensource.org/licenses/BSD-3-Clause)

at your option.

See [LICENSE](LICENSE) for more details.
