Metadata-Version: 2.4
Name: xmetai-inference
Version: 0.1.0
Summary: Configuration-driven weather model inference and evaluation
Author-email: XMetAI Lab <douzsh@gmail.com>
License-Expression: GPL-3.0-or-later
Keywords: meteorology,AI,deep learning,weather,inference
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: xmetai-core
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: torch
Requires-Dist: xarray
Provides-Extra: netcdf
Requires-Dist: h5netcdf; extra == "netcdf"
Requires-Dist: netCDF4; extra == "netcdf"
Provides-Extra: zarr
Requires-Dist: zarr; extra == "zarr"
Provides-Extra: onnx
Requires-Dist: onnxruntime; extra == "onnx"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=3.0; extra == "dev"
Requires-Dist: black>=22.0; extra == "dev"
Requires-Dist: ruff>=0.0.250; extra == "dev"
Requires-Dist: zarr; extra == "dev"
Requires-Dist: h5netcdf; extra == "dev"
Requires-Dist: netCDF4; extra == "dev"
Requires-Dist: onnxruntime; extra == "dev"
Dynamic: license-file

# xmetai-inference

[English](README.md) | [简体中文](README.zh-CN.md)

Inference framework for autoregressive weather forecasting models. It reads ERA5 / CRA-40
reanalysis from Zarr stores, feeds each model the channel set and history window its contract
declares, runs the rollout, and writes the forecast as NetCDF.

- Runs ONNX and `torch.export` (`.pt2`) graphs behind one interface
- Supports deterministic and 51-member ensemble models on a 0.25° global grid
- Ships a run recipe per model, selected with `--config <name>`
- Multi-GPU by sharding initialization times × members across cards

Release version: **0.1.0**.

## Related projects

Exported ONNX models require GPU inference support. Use
[xmetai-onnx-plugins](https://github.com/XMetAI-Lab/xmetai-onnx-plugins) to provide the custom GPU
operators required by the corresponding models.

| Project | Purpose |
| --- | --- |
| [xmetai-core](https://github.com/XMetAI-Lab/xmetai-core) | Core framework for meteorological AI model training and inference |
| [xmetai-skills](https://github.com/XMetAI-Lab/xmetai-skills) | Skills for model development |
| [xmetai-datakit](https://github.com/XMetAI-Lab/xmetai-datakit) | Data processing and evaluation |
| [xmetai-onnx-plugins](https://github.com/XMetAI-Lab/xmetai-onnx-plugins) | Custom GPU operators for exported ONNX models |

## Table of Contents

- [Related projects](#related-projects)
- [Quick Start](#quick-start)
  - [Requirements](#requirements)
  - [Install](#install)
  - [Run](#run)
  - [Examples](#examples)
  - [Override Parameters](#override-parameters)
- [Supported Models](#supported-models)
  - [Channel Sets and Order](#channel-sets-and-order)
- [Supported Datasets](#supported-datasets)
  - [processed_multi_zarr](#processed_multi_zarr)
  - [netcdf_data](#netcdf_data)
- [Project Structure](#project-structure)
- [How It Works](#how-it-works)
- [License and contact](#license-and-contact)

## Quick Start

### Requirements

- Linux (or WSL2 / Git Bash on Windows)
- An NVIDIA GPU with CUDA
- Model weights and the input Zarr stores (both excluded from git)

### Install

From PyPI:

```bash
pip install xmetai-inference
pip install "xmetai-inference[netcdf,zarr,onnx]"   # every model except FuXi-2.1 (ONNX)
pip install "xmetai-inference[netcdf,zarr]"        # NetCDF I/O only: FuXi-2.1 / evaluation
```

From a source checkout, for development:

```bash
pip install -e .
pip install -e ".[netcdf,zarr,onnx]"
pip install -e ".[netcdf,zarr]"
```

`torch` comes in with the base install. The `netcdf` extra brings `netCDF4` / `h5netcdf`
(reading `mean.nc`/`std.nc`, writing output), `zarr` powers store reads in `xmetai-core`, and
`onnx` pulls in `onnxruntime`.

> PyPI's `onnxruntime` is the **CPU** build. For GPU inference install `onnxruntime-gpu`
> instead (the `onnx` extra and `onnxruntime-gpu` conflict; use one or the other). The loader
> logs a warning when it has to fall back to the CPU provider.

> **XMetAI ONNX Runtime plugin (custom operators).** Recipes that set `ops_library` (the
> Tongyun recipes and fengshun2.0) load custom operators through the XMetAI ONNX Runtime
> plugin. Download the build matching your ONNX Runtime / CUDA version from
> https://github.com/XMetAI-Lab/xmetai-onnx-plugins/releases/ and point the recipe's
> `ops_library` at the downloaded `.so` (for example
> `xmetai_onnx_plugins.ort1_24.cuda12.so`). Without the plugin the graph fails to load with
> an unknown-operator error.

### Run

Every supported model has a ready-made recipe (`configs/<name>.py`):

```bash
xmetai-infer --config weather_era_0p25_fuxi_ens
xmetai-infer --config weather_era_0p25_fuxi21
xmetai-infer --config weather_era_0p25_tongyun
xmetai-infer --config weather_era_0p25_fengqing
xmetai-infer --config weather_era_0p25_pangu
xmetai-infer --config weather_era_0p25_tongyun_ens
xmetai-infer --config weather_cra_0p25_tongyun_ens
xmetai-infer --config s2s_cra_0p25_fengshun20_ens
xmetai-infer --config weather_nc_fuxiens         # example: official FuXi-ENS input.nc
xmetai-infer --config weather_nc_fuxi21          # example: official FuXi-2.1 input.nc
```

Each recipe declares the weight path, dataset and preprocessing chain, initialization times, step
count, GPU count, and output directory. The shipped recipes use `/path/to/...` placeholders —
point those at your own weights and stores (or set the matching environment variables) before the
first run. From a source checkout without installing:

```bash
python -m xmetai_inference --config weather_era_0p25_fuxi21
```

Output goes to `{out}/{YYYYMMDD|YYYYMMDDHH}/[member_XXX/]{step:03d}.nc`, one NetCDF per step.

`xmetai-infer --worker ...` is the internal worker subprocess entry point used for every run
(single- or multi-GPU) and should not be called manually.

### Examples

Two self-contained example recipes infer straight from **one given `.nc`** (`dataset.type="netcdf_data"`)
— no Zarr store. Both sample files ship with the corresponding official upstream release; this
repository does not host them:

| Recipe | Model (registry) | Files | members × steps |
|---|---|---|---|
| `weather_nc_fuxiens` | FuXi-ENS (`fuxi_ens_onnx`) | official `data/input.nc` + `model/fuxi_ens.onnx` (and its external weights `model/fuxi_ens`) | 2 × 2 |
| `weather_nc_fuxi21` | FuXi-2.1 (`fuxi21_pt2`) | official `input.nc` + `fuxi-2.1.pt2` + `mean.nc`/`std.nc` | 1 × 2 |

Where the sample `.nc` files come from:

- **FuXi-ENS** — `data/input.nc` (2 × 78 × 721 × 1440, initialization 2018-09-14 00Z) ships in
  `FuXi-ENS-main.zip` from the official model record
  [Zenodo 15124541](https://zenodo.org/records/15124541) (the release accompanying the Science
  Advances paper, DOI [10.1126/sciadv.adu2854](https://doi.org/10.1126/sciadv.adu2854)), together
  with `model/fuxi_ens.onnx` and its external weight file `model/fuxi_ens`. The bundled README
  additionally points to a restricted Google Drive folder (contact Prof. Li Hao,
  lihao_lh@fudan.edu.cn).
- **FuXi-2.1** — `input.nc` (2 × 85 × 721 × 1440, initialization 2024-09-29 00Z, already z-score
  normalized) ships in the Hugging Face repository
  [tpys/fuxi-2.1](https://huggingface.co/tpys/fuxi-2.1) together with `fuxi-2.1.pt2`, `mean.nc`
  and `std.nc`; the model is part of the
  [FuXi Single collection](https://huggingface.co/collections/tpys/fuxi-single).

Neither recipe preprocesses the nc in the dataset layer. FuXi-ENS normalizes inside the ONNX graph
(physical in, physical out); FuXi-2.1's `input.nc` is already z-score normalized and its output is
denormalized with the `mean.nc`/`std.nc` next to the `.pt2`. The nc alone supplies the history
window; the init time is its last `time` frame, so `cfg.times` is empty. The FuXi-ENS example runs
2 members × 2 steps as a smoke test — use `--members 51 --steps 60` (or edit the recipe) for a full
run; the graph samples noise internally.

Point `SAMPLE_ROOT` in each recipe at the directory holding the upstream release files:

- FuXi-ENS — `input.nc`, `fuxi_ens_onnx/fuxi_ens.onnx`, and the external weight file `fuxi_ens`
  it references (≈9.3 GiB, extensionless, must sit next to the .onnx).
- FuXi-2.1 — `input.nc`, `mean.nc`, `std.nc`, and `fuxi-2.1.pt2` (3.7 GiB).

Then one command each:

```bash
python -m xmetai_inference --config weather_nc_fuxiens --out <output_dir>
python -m xmetai_inference --config weather_nc_fuxi21  --out <output_dir>
```

Output lands at `{out}/{YYYYMMDD|YYYYMMDDHH}/[member_XXX/]{step:03d}.nc` (no `member_XXX` layer
when `members=1`).

An input nc only works if it matches the graph: physical quantities (or normalized values for
FuXi-2.1) in `input_channels` order, the model's grid orientation, `float32`, `(time=history,
channel=N, 721, 1440)`, and `sst` cold-spots filled where the graph expects it.

### Override Parameters

```bash
xmetai-infer --config weather_era_0p25_fuxi_ens \
  --times "2025010600..2025011200:24" \
  --steps 60 --members 51 --gpus 4 --cuda-devices 0,1,2,3 \
  --vars z500,u200,v200,msl,tp \
  --out <output_dir>
```

- `--times` follows the `parse_times` syntax in `xmetai_inference/configs/base.py`: a single
  `2025010600`, a comma-separated list, a closed interval `A..B` (24 h step by default), or
  `:N` to override the step.
- `--config` takes a built-in recipe name or any external `.py` recipe; the positional argument
  is the same thing for a file path. See `configs/` for examples.
- `--model-path` swaps the weight file without touching the recipe.

## Supported Models

| Model | Recipe (`--config`) | Backend | Type | Members | Channels | Grid | Step / history | Normalization |
|------|------|------|------|------|------|------|------|--------|
| FuXi-Ens | `weather_era_0p25_fuxi_ens` | `onnx` | Ensemble | 51 | 78 | `GRID_025` | 6h / 2 frames | Baked into graph; physical units in/out |
| FuXi-2.1 | `weather_era_0p25_fuxi21` | `pt2` | Deterministic | 1 | 85 | `GRID_025` | 6h / 2 frames | Data-side `normalize`, output `denormalize` (mean.nc/std.nc next to the weights) |
| Tongyun (det) | `weather_era_0p25_tongyun` | `onnx` | Deterministic | 1 | 78 | `GRID_025` | 6h / 2 frames | Baked into graph; custom ops (`GatedDeltaNet2Fn`) require `ops_library` |
| FengQing V1.5Beta | `weather_era_0p25_fengqing` | `onnx` | Deterministic | 1 | 70 | `GRID_025` | 6h / 2 frames | `normalize_fengqing` / `denormalize_fengqing` (per-pixel stats; graph emits a residual) |
| Pangu-Weather | `weather_era_0p25_pangu` | `onnx` | Deterministic | 1 | 69 | `GRID_025` | 6h / 1 frame (+24h jump) | Baked into graph; physical units in/out |
| Tongyun-Ens | `weather_era_0p25_tongyun_ens` | `onnx` | Ensemble | 51 | 78 | `GRID_025` | 6h / 2 frames | Baked into graph; physical units in/out; custom ops require `ops_library` |
| Tongyun-Ens (CRA) | `weather_cra_0p25_tongyun_ens` | `onnx` | Ensemble | 51 | 71 | `GRID_025_S` | 6h / 2 frames | Store-normalized → data-side `denormalize`; graph normalizes at its head, outputs physical units |
| fengshun2.0 | `s2s_cra_0p25_fengshun20_ens` | `onnx` | Ensemble | 51 | 87 | `GRID_025` | 24h / 2 frames | Store-normalized → data-side `denormalize` (tp via `expm1`); graph normalizes at its head |

The ERA5 models share the 0.25° global regular grid (721×1440, latitude north→south, longitude
0–360°). All use a 2-frame history window except Pangu, which uses a single frame
(`history_steps=1`) and, every 4 steps (lead 24 h), switches to a 24 h model that jumps directly
from the anchor (official "Plan B").

Three exceptions worth knowing:

- **Tongyun-Ens (CRA)** uses `GRID_025_S` (latitude south→north, the store's native orientation)
  and its preprocessing **must not configure `geometry`** — that flips the field in reverse.
- **fengshun2.0** runs on a daily step (`hour_interval=24`), so `steps=60` is a 60-day forecast.
  Its store is CRA north→south, so it also skips `geometry`. The recipe asks for 51 members even
  though the model class declares `members=11` as the graph's native member axis.
- **Pangu** needs a second weight file for the 24 h jump (`PANGU_ONNX_24`); it defaults to
  `pangu_weather_24.onnx` next to the 6 h model.

### Channel Sets and Order

Every model's `input_channels` (also `output_channels`) is the flattened list below; the order is
part of the contract. A wrong order does **not** raise — it silently forecasts the wrong field.

```
upper = [f"{var}{level}" for var in PL_VARS for level in LEVELS]      # 65 channels

FuXi-Ens  (78): PL z,t,u,v,q × [50,100,150,200,250,300,400,500,600,700,850,925,1000]
                + [t2m, d2m, sst, u10m, v10m, u100m, v100m, msl, ssr, ssrd, fdir, ttr, tp]
FuXi-2.1  (85): PL z,t,u,v,q × same levels
                + [msl, t2m, d2m, sst, ws10m, ws100m, u10m, v10m, u100m, v100m,
                   lcc, mcc, hcc, tcc, ssr, ssrd, fdir, ttr, tcw, tp]
fengshun  (87): PL z,t,u,v,q × same levels
                + [msl, t2m, t2m_min, t2m_max, d2m, sst, ws10m, ws100m, u10m, v10m,
                   u100m, v100m, lcc, mcc, hcc, tcc, ssr, ssrd, fdir, ttr, tcw, tp]
FengQing  (70): PL z,q,t,u,v × [1000,925,850,700,600,500,400,300,250,200,150,100,50]
                + [tp, msl, u10m, v10m, t2m]
Pangu     (69): PL z,q,t,u,v × same descending-pressure levels
                + [msl, u10m, v10m, t2m]
Tongyun   (71): PL z,t,u,v,q × [50 … 1000] (ascending pressure)
                + [t2m, q2m, u10m, v10m, msl, tp]
```

Channel names are matched against store names by `_raw_channel_name`: exact match first, then
`z50` → `z_50`. A store may be a superset of the model channels — every requested channel must
exist, or construction fails loudly. Two processors read names differently: `denormalize`
aligns store stats by underscore-insensitive, lower-cased name and tolerates a superset;
`normalize` requires stats that match `input_channels` in count **and order** exactly.

> **Model naming.** Config names follow `<domain>_<source>_<resolution>_<model>[_ens]`:
> `domain` = `weather` (15-day, 6-hourly) or `s2s` (subseasonal, daily steps); `source` = `era`
> (ERA5) or `cra` (CRA-40); `resolution` = `0p25` (0.25°) or `1p5`. The trailing `_ens` marks an
> ensemble output when the model name doesn't already contain `ens`; deterministic configs carry
> no suffix. Examples: `weather_era_0p25_fuxi_ens`, `weather_era_0p25_tongyun_ens`,
> `weather_cra_0p25_tongyun_ens`, `s2s_cra_0p25_fengshun20_ens`. Each maps to the eponymous
> recipe under `configs/`.
> **Model-class registry names** (the `model_class=` in recipes, e.g. `fuxi_ens_onnx`,
> `fuxi21_pt2`, `iwc_tongyun_gdn2_onnx` (the Tongyun det graph), `fengqing_pre_onnx`,
> `pangu_onnx`, `tongyun_era_ens_onnx`, `tongyun_cra_ens_onnx`, `fengshun20_onnx`) map to `MODEL_REGISTRY` in
> `utils/model_factory.py`.

## Supported Datasets

Two dataset types: `processed_multi_zarr` (the production path, backed by `xmetai-core`'s
`MultiZarrDataset`) and `netcdf_data` (one given `.nc` per initialization, for demos and
business-supplied inputs).

### processed_multi_zarr

The stores are Zarr products built to the xmetai-datakit data contract. The contract defines the
array structure, channel order, units, normalization, masks, and metadata a product must satisfy:
see [`config/dataset_specs`](https://github.com/XMetAI-Lab/xmetai-datakit/tree/main/config/dataset_specs).

```python
dataset = {
    "type": "processed_multi_zarr",
    "paths": ["<store_root>/store_a.zarr", "<store_root>/store_b.zarr"],
    "processors": [ ... ],          # executed in the DataLoader worker, in order
}
```

- `paths` may list several stores. Each is opened and sliced to the requested time range; their
  time coordinates are aligned to the **intersection** and to `freq`, then the arrays are
  concatenated along `channel`. **The first store listed wins on duplicate channel names.**
- One sample is `hist_frames` consecutive frames ending at the init time, read as
  `(hist_frames, C, lat, lon)` float32 (e.g. 2 × 78 × 721 × 1440 for FuXi-Ens). The init time is
  the last history frame; the store must contain at least one frame after it. A requested init
  time without enough history raises.
- The framework derives `hist_frames`, `fcst_frames=1`, `freq`, `interval=1`,
  `inference_only=True`, `years`, and `in_names=[]` from the model contract and the requested
  init times. Do not repeat these keys in `dataset` (duplicate keyword arguments); any **other**
  key is forwarded to `MultiZarrDataset` as-is (e.g. `bbox`, `lat_flip`).

### netcdf_data

Infer straight from given `.nc` files, one file per initialization:

```python
dataset = {
    "type": "netcdf_data",
    "paths": ["/abs/input.nc"],
}
```

Each file holds the full model input window `[time, channel, lat, lon]`; all history frames live
inside the nc itself (the engine never reads back into a store). The init time is the last `time`
frame's value, so `cfg.times` is not consulted for this type. Values pass through **untouched** —
no processor chain runs. Only the channel axis is verified and rewritten to `input_channels` by
name (underscore-insensitive, case-insensitive); a missing channel or a count mismatch fails at
build time. Samples are built eagerly once at dataset build time, so this type is
single-process-only.

## Project Structure

```
.
├── pyproject.toml           # PyPI build metadata and CLI entry
├── xmetai_inference/        # The installable Python package (package name: xmetai-inference)
│   ├── cli.py               # Config loading, multi-GPU scheduling, the autoregressive main loop, async disk-write
│   ├── data/                # Dataset implementations, grouped into subpackages by category (weather/)
│   ├── utils/               # dataloader_factory (config -> DataLoader, dispatched by dataset.type),
│   │                        #   runner (Rollout / Trajectory autoregressive orchestration),
│   │                        #   model_factory (MODEL_REGISTRY / get_model_class / create_model),
│   │                        #   model-side pipeline (recurrent/output), data-side tensor processors, logging
│   ├── backends/            # BaseInferModel + onnx / pt2 execution engines
│   └── configs/             # Run recipes (`--config <name>`) + declared model contracts;
│                            #   base.py = shared GRID/channels + InferConfig/load_config
├── tests/                   # Unit + integration tests (integration runs only with `-m realdata`)
└── model_artifacts/         # Local model artifacts; absent in a fresh clone and excluded from Git
```

## How It Works

One forecast, from store to disk:

```
zarr store  →  xmetai-core MultiZarrDataset          (read history frames, select variables by model channel)
            →  dataset.processors                    (in DataLoader worker, torch; chain depends on store layout)
                 unit_convert → geometry → normalize → fill_missing     ← raw store (ERA5)
                 denormalize (stats_path=store, ± expm1) → …            ← normalized store (CRA / c87 / v3)
            →  prepare_dataset_batch                 ([B,T,C,H,W] tensor)
            →  Rollout (autoregressive)              (state = result feedback)
                 recurrent processors (zero_channels)
            →  output processors (denormalize)       (workspace → physical units)
            →  to_dataset                            (xarray, non-negative fallback for tp)
            →  _AsyncWriter                          (single background thread writes NetCDF)
```

- **Contract-driven models**: channels, grid, history window, member semantics, and state
  representation are declared by the model class, so backends stay interchangeable.
- **Pluggable backends**: `onnx` and `pt2` execution engines are decoupled from model semantics.
- **One recipe per run**: the config declares `dataset` (incl. `processors`), `dataloader`, and
  `model_processing` (the `recurrent` and `output` stages).
- **Unified autoregression**: `utils/runner.py`'s `Rollout`/`Trajectory` chain single-step
  `forward` into a multi-trajectory loop with `state = result` feedback.
- **Async disk-write**: one background thread writes NetCDF; on failure it falls back to per-step
  `raw_step_XXX.npy`.
- **Multi-GPU data parallelism**: work is sharded by "initialization time × member"; each worker
  process is pinned to one card via `CUDA_VISIBLE_DEVICES`.

> Input preprocessing goes through `dataset.processors`; `model_processing.input` must be empty
> (misconfiguring it errors out). Normalization already fused into the model file is not
> re-executed by the framework.

## License and contact

Licensed under **GPL-3.0-or-later**; see [LICENSE](LICENSE). Model weights, data, and third-party
assets may have separate terms.

XMetAI Team: douzsh@gmail.com.
