Metadata-Version: 2.4
Name: openeolib
Version: 0.3.11
Summary: Python library for Earth observation data providers, geospatial analysis, plume simulation, and reusable visualization
Author: tkxu
License-Expression: Apache-2.0
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: scipy>=1.10
Requires-Dist: matplotlib>=3.7
Requires-Dist: requests>=2.28
Requires-Dist: beautifulsoup4>=4.12
Provides-Extra: provider
Requires-Dist: openeo>=0.26; extra == "provider"
Requires-Dist: rasterio>=1.3; extra == "provider"
Requires-Dist: xarray>=2023.1; extra == "provider"
Requires-Dist: netCDF4>=1.6; extra == "provider"
Provides-Extra: era5
Requires-Dist: cdsapi>=0.6; extra == "era5"
Requires-Dist: xarray>=2023.1; extra == "era5"
Requires-Dist: netCDF4>=1.6; extra == "era5"
Provides-Extra: planetary-computer
Requires-Dist: pystac-client>=0.7; extra == "planetary-computer"
Requires-Dist: planetary-computer>=1.0; extra == "planetary-computer"
Requires-Dist: stackstac>=0.5; extra == "planetary-computer"
Requires-Dist: xarray>=2023.1; extra == "planetary-computer"
Requires-Dist: netCDF4>=1.6; extra == "planetary-computer"
Requires-Dist: shapely>=2.0; extra == "planetary-computer"
Provides-Extra: earth-search
Requires-Dist: pystac-client>=0.7; extra == "earth-search"
Requires-Dist: stackstac>=0.5; extra == "earth-search"
Requires-Dist: shapely>=2.0; extra == "earth-search"
Provides-Extra: hrrr
Requires-Dist: cfgrib>=0.9.10; extra == "hrrr"
Requires-Dist: xarray>=2023.1; extra == "hrrr"
Provides-Extra: jma
Requires-Dist: xarray>=2023.1; extra == "jma"
Requires-Dist: eccodes>=1.5; extra == "jma"
Requires-Dist: cfgrib>=0.9.10; extra == "jma"
Requires-Dist: dask>=2023.1; extra == "jma"
Requires-Dist: pyyaml>=6.0; extra == "jma"
Requires-Dist: JMA-grib2>=0.0.4a3; extra == "jma"
Provides-Extra: himawari
Requires-Dist: xarray>=2023.1; extra == "himawari"
Requires-Dist: satpy>=0.55; extra == "himawari"
Provides-Extra: dev
Requires-Dist: pytest>=7.4; extra == "dev"
Requires-Dist: pytest-cov>=4.1; extra == "dev"
Requires-Dist: flake8>=6.1; extra == "dev"
Requires-Dist: mypy>=1.5; extra == "dev"
Dynamic: license-file

# OpenEO-LIB

OpenEO-LIB is a Python library for satellite-based Earth observation workflows, including data providers, geospatial analysis, plume simulation, and reusable visualization components.

The package is designed so that reusable library functionality is kept separate from application-specific validation and research code.

## Current architecture

```text
openeolib/
├── __init__.py
├── _plot_utils.py       Shared Matplotlib helpers for the panels (internal)
├── analyzer.py          Analysis pipeline orchestration
├── animation.py         Generic scalar-field animation (GridAnimationEngine)
├── engines.py           Inference-engine interfaces and built-ins
├── eo_cache.py          Persistent scientific-data cache (ScienceCacheStore)
├── eo_types.py          Shared data structures and type definitions
├── eo_utils.py          Geospatial and wind utilities
├── palettes.py          Domain color tables (e.g. JMA rainfall-intensity palette)
├── panels.py            Generic Matplotlib visualization panels
├── report.py            Generic figure/report composition
├── roc.py               ROC construction
├── simulator.py         Synthetic plume generation
├── theme.py             Theme definition only (Theme, DEFAULT_THEME)
├── processing/          Array-level processing of retrieved data (no I/O, never imports providers/)
│   ├── grid.py          Grid regridding, Gaussian-weighted interpolation, NaN-aware Gaussian filter/derivative
│   ├── roi.py           Area-weighted ROI (bbox/polygon) statistics of gridded time series
│   ├── satellite.py     Cloud motion (block matching), advection and parallax for geostationary imagery
│   └── pwv_layers.py    Decompose GNSS total PWV into AGL layers from an ERA5 profile
└── providers/
    ├── base.py               BaseProvider ABC (raw Dataset retrieval)
    ├── provider.py           AnalyzerProvider / EOProvider (S2+ERA5+S5P integration)
    ├── openeo_client.py      openEO connection client
    ├── _http_utils.py        Shared retry/local-file-cache helpers
    ├── _protocols.py         Cross-provider provenance accessor (get_provenance())
    ├── sentinel2.py          Sentinel-2 L2A band retrieval
    ├── sentinel5p.py         Sentinel-5P L2 CH4 column retrieval
    ├── planetary_computer.py Sentinel-2/Sentinel-5P via Microsoft Planetary Computer (no auth)
    ├── earth_search.py       Sentinel-2 via Element84 Earth Search / AWS (no auth)
    ├── _stac_s2.py           Shared Sentinel-2 STAC search/retrieval core (PC + Earth Search)
    ├── era5.py               ERA5 wind/pressure/temperature retrieval
    ├── hrrr.py               NOAA HRRR 10 m wind retrieval (CONUS, no auth)
    ├── msm.py                JMA MSM (mesoscale model) GPV retrieval (MSMProvider), plus a
    │                         subprocess-isolated batch runner (BatchRequest/run_batch_isolated)
    ├── lfm.py                JMA LFM (local forecast model) GPV retrieval scaffold (LFMProvider)
    ├── radar_gpv_client.py   JMA nationwide composite radar GPV download
    ├── rain.py               JMA rainfall / XRAIN providers + animate_rain() + rain_roi_timeseries()
    ├── amedas.py             JMA AMeDAS hourly / 10-minute surface observations
    ├── radiosonde.py         University of Wyoming radiosonde sounding retrieval
    ├── cwop.py               CWOP private weather stations (wind etc.) via NOAA MADIS public archive
    ├── nies_air.py           Air-quality monitoring stations' hourly wind/temp (NIES confirmed values)
    ├── tbeic.py              Tokyo Bay monitoring posts' 15-min over-water wind (TBEIC)
    ├── pwv.py                GPT3 / VMF3 blind-grid PWV retrieval
    ├── unr_trop.py           Nevada Geodetic Laboratory (NGL/UNR) GNSS tropospheric products
    ├── himawari.py           Himawari-8/9 AHI brightness temperature via CEReS (FTP)
    ├── himawari_aws.py       Himawari-8/9 AHI brightness temperature via NOAA AWS (HTTPS, no FTP)
    └── _s2_utils.py          Sentinel-2 coordinate/time conversion utilities


```

Provider responsibilities are deliberately separated. `BaseProvider` and
`AnalyzerProvider` are independent interfaces with different responsibilities;
`AnalyzerProvider` does not extend `BaseProvider`. `BaseProvider` is the minimal
file/data-opening interface for providers that return an `xarray.Dataset`
without application-level analysis. `AnalyzerProvider` is the higher-level
interface used by `EOAnalyzer` and returns an `ObservationBundle` for a site and
observation time -- a `typing.Protocol` (structural typing), so a provider
conforms by having a matching `get_bundle(site, dt)` method, not by explicitly
inheriting from it. `EOProvider` implements it
for the openEO/Sentinel-2/Sentinel-5P workflow and contains the
workflow-specific quality and time-alignment logic. `AnalyzerProvider`/
`ObservationBundle` is *not* a general-purpose provider contract -- it is
`EOAnalyzer`'s own methane/plume-detection input shape (it reads `bands["B11"]`/`["B12"]`
directly), produced only by `EOProvider` and `PlumeSimulator`.

For code that wants to treat any provider's result uniformly regardless of its
own return shape (`xr.Dataset`, a `Dict`, or `None`) -- e.g. for logging or an
audit trail -- `providers._protocols.get_provenance(result)` extracts a
normalized `{source, provider_class, datetime_utc, provenance_key}` dict from
whichever of the metadata conventions this package's providers happen to
use (`.attrs["source"]`/`["provider_class"]`, `.attrs["provider"]`, dict keys
`"backend"`/`"datetime_utc"`, or `pwv.py`'s `"epoch_posix_utc_sec"` POSIX
timestamp, converted to an ISO UTC string). It requires no changes to any
existing provider.

`openeolib.processing` holds the array-level processing that sits between
retrieval and analysis: it takes arrays or Datasets that a provider has
already returned and computes derived fields or statistics, with no network
or file I/O. Providers may import from it (`providers.rain.rain_roi_timeseries()`
calls `processing.roi`); `processing` never imports `providers`. Submodules
are not imported eagerly. The former locations `openeolib.eo_grid`,
`openeolib.eo_roi` and `openeolib.atmosphere` (`.pwv_layers`) still import,
as deprecated aliases that give the same module objects and emit a
`DeprecationWarning`; import from `openeolib.processing.grid` / `.roi` /
`.pwv_layers` instead.

The visualization layer (`panels.py`, `theme.py`, `report.py`, `animation.py`)
is kept domain-agnostic. Domain providers may know the schema and units of
their own source data, but they do not depend on application-specific
research pipelines.

Likewise, `theme.py` contains only the `Theme` data structure and `DEFAULT_THEME`.

## Installation

```bash
pip install openeolib
```

Development dependencies are optional and are intended for the private test suite:

```bash
pip install -e ".[dev]"
```

Optional provider dependencies are grouped by feature, e.g.
`pip install "openeolib[era5]"` (or `pip install -e ".[era5]"` from a source
checkout):

```bash
pip install "openeolib[provider]"            # openEO / Sentinel-2 / Sentinel-5P
pip install "openeolib[era5]"                # ERA5 via the CDS API
pip install "openeolib[planetary_computer]"  # Microsoft Planetary Computer
pip install "openeolib[earth_search]"        # Element84 Earth Search
pip install "openeolib[hrrr]"                # NOAA HRRR
pip install "openeolib[jma]"                 # MSM, LFM, JMA radar rainfall
pip install "openeolib[himawari]"            # Himawari-8/9
```

The core package supports Python 3.9 or later. Provider-specific optional dependencies may have their own Python-version requirements.

The core install (`pip install openeolib`) imports `openeolib`,
`openeolib.providers` and the non-provider modules. Most provider modules
import their extra's packages (for example xarray) at module level, so
importing them without that extra raises `ImportError`.

The JMA extra (`openeolib[jma]`) is required for `MSMProvider`. The module
`openeolib.providers.msm` imports without it; constructing `MSMProvider`,
calling `MSMProvider.read_pressure_level_profile()` /
`read_surface_fields()` or `run_batch_isolated()` then raises an
`ImportError` naming every missing dependency and the installation command.

## Upgrading from 0.2.x

0.3.0 changes some behaviour. Check these when upgrading:

- **ERA5 cache.** `ERA5Provider` caches raw CDS values under `cache_dir/raw`.
  Entries written by 0.2.x directly under `cache_dir` are no longer read, so
  data is downloaded again on first use; the old files can be deleted.
- **ERA5 boundary-layer height.** ERA5-Land has no boundary-layer height, so
  `boundary_layer_height` was always `None`. Pass `enable_blh=True` to fetch
  it from ERA5 single levels. Listing it in `extra_variables` now raises
  `ValueError`.
- **ERA5 100 m wind at a point.** `fetch(..., enable_100m_wind=True)` now
  returns the 100 m keys; they were usually missing before.
- **Timezone-aware datetimes.** `ERA5Provider` and `LFMProvider` convert them
  to UTC. Before, a JST time was requested as if its clock time were UTC.
  Naive datetimes are still taken as UTC.
- **MSM / LFM GRIB2 reading.** Both read GRIB2 with eccodes (included in
  `openeolib[jma]`). `MSMProvider(grib_backend="cfgrib")` is deprecated.
- **Tests that patch ERA5 internals.** `ERA5Provider._download()` is gone;
  replace `provider.client` (the CDS client) instead.

## Public visualization API

The reusable visualization API consists of five panels:

- `VectorFieldPanel` — arbitrary geospatial vector fields with optional scalar contours.
- `BasemapPanel` — an already prepared RGB image with an optional marker.
- `RawBandsPanel` — arbitrary grids of 2D images with a shared scale.
- `ScalarMapPanel` — arbitrary 2D scalar fields.
- `DetectionMaskPanel` — boolean, probability, or coverage masks.

Example:

```python
import matplotlib.pyplot as plt
import numpy as np
from openeolib import ScalarMapPanel

field = np.random.default_rng(0).normal(size=(100, 100))

fig = plt.figure(figsize=(6, 5))
gs = fig.add_gridspec(1, 1)[0]
ScalarMapPanel().draw(
    fig,
    gs,
    field,
    title="Scalar field",
    axis_mode="none",
)
fig.savefig("scalar_field.png", dpi=150, bbox_inches="tight")
```

### Vector fields

`VectorFieldPanel` is intentionally not ERA5-specific. It accepts latitude/longitude grids and arbitrary vector components. A caller may supply any bounding box through `extent=(west, east, south, north)`.

```python
from openeolib import VectorFieldPanel

panel = VectorFieldPanel(quiver_stride=3)
panel.draw(
    fig,
    gs,
    lats=lats,
    lons=lons,
    u=u,
    v=v,
    extent=(120, 150, 20, 50),
    title="Wind field",
)
```

There is no Japan-specific bounding-box constant in the library.

### Theme

Use `Theme` when a caller needs to customize the visual appearance:

```python
from openeolib import Theme

light = Theme(
    bg="#ffffff",
    panel_bg="#ffffff",
    grid_color="#cccccc",
    text_primary="#222222",
)
```

`Theme` does not contain plotting operations or application-specific labels, flags, or data extraction rules.

## Generic reports

`SiteReportBuilder` composes caller-supplied panels into a single-site report Figure. It does not know about methane, ROC curves, quality flags, or a particular inference engine -- it only handles grid layout, the report title, and file saving.

Each entry in `panels` is `(label, factory, height_ratio)`. `factory` is called once with the report's `Theme` and must return a `(fig, gs) -> None` draw function; any panel-specific data (the field to plot, its title, ...) is bound into that closure by the caller, not inspected by `SiteReportBuilder` itself.

```python
import numpy as np
from openeolib import SiteReportBuilder, ScalarMapPanel

field = np.random.default_rng(0).normal(size=(100, 100))

report = SiteReportBuilder(
    panels=[
        (
            "Scalar field",
            lambda theme: lambda fig, gs: ScalarMapPanel(theme=theme).draw(
                fig, gs, field, title="Scalar field", axis_mode="none",
            ),
            1.0,
        ),
    ],
    report_title="Example report",
)

report.build_site({"site": {"id": "SITE-01"}}, save_path="report.png")
```

Application-specific validation reports can be assembled from these reusable panels in the consuming application; application-specific validation code is not included in this package.

## Analysis and simulation

The main public analysis components include:

```python
from openeolib import EOAnalyzer, PlumeSimulator, InferenceEngine
```

`EOAnalyzer` coordinates provider data and an inference engine. `PlumeSimulator` provides synthetic plume data for testing and demonstrations. `InferenceEngine` defines the interface for custom detection/quantification algorithms.

## Providers

The package contains provider implementations for openEO, Sentinel-2, Sentinel-5P, Microsoft Planetary Computer and Element84 Earth Search (both no-auth alternatives for Sentinel-2), ERA5, NOAA HRRR, JMA MSM and LFM, JMA radar rainfall and XRAIN, JMA AMeDAS surface observations, CWOP private weather stations, NIES air-quality monitoring stations, Tokyo Bay monitoring posts (TBEIC), University of Wyoming radiosonde soundings, GPT3/VMF3 blind-grid PWV, NGL/UNR GNSS tropospheric products, and Himawari-8/9 cloud-top brightness temperature. Provider-specific dependencies are optional where practical.

`XrainProvider` accepts NetCDF and CSV input. If a CSV has no time column, it
creates a single `time` coordinate containing `NaT` rather than inventing an
observation timestamp. Callers that require a real observation time must provide
a time column or pass `time_name=`. This distinction is intentional and prevents
silent fabrication of temporal metadata.

`RadarGpvClient` is a public retrieval utility, not a `BaseProvider` or `AnalyzerProvider`.
It downloads the 1 km rainfall GRIB2 of the JMA nationwide composite radar GPV
from the Kyoto University RISH archive: the five-minute raw `Ggis1km_Prr05lv`
files from 2024-02-26, or the file extracted from the ten-minute tar archives
before that. `JmaRainProvider` is the dataset-opening provider that parses
those GRIB2 files into an `xarray.Dataset`. This separation keeps
network/archive handling distinct from dataset parsing.

`rain_roi_timeseries()` reduces the Dataset returned by `JmaRainProvider.open()`
or `XrainProvider.open()` to one row per time step over an ROI: the
area-weighted mean rate, the maximum cell rate, the accumulated rainfall
(mean rate x time step) and the fraction of the ROI with valid data, plus
optional threshold-exceedance area fractions (`thresholds=(20, 50)` ->
`frac_ge_20`, `frac_ge_50`). Negative values are treated as missing. The ROI
is either the same `{"lat": (min, max), "lon": (min, max)}` dict as
`open(region=...)` or a polygon (shapely geometry, GeoJSON-like mapping, or a
list of (lon, lat) vertices); each grid cell is weighted by its area inside
the ROI.

```python
from openeolib.providers.rain import JmaRainProvider, rain_roi_timeseries

ds = JmaRainProvider().open(paths, region={"lat": (34.0, 36.0), "lon": (134.0, 136.0)})
df = rain_roi_timeseries(ds, {"lat": (34.8, 35.2), "lon": (134.8, 135.2)}, thresholds=(20, 50))
```

The general-purpose function behind it is `processing.roi.roi_timeseries()` (any
(time, latitude, longitude) field, xarray or numpy; statistics `mean`, `std`,
`min`, `max`, `area_integral` in units x km^2, `accum`, `valid_fraction`).
`processing.roi.roi_cell_weights()` returns the per-cell inside-area (km^2) on its
own. Cell areas use the spherical formula; bbox overlaps are exact, polygon
overlaps are exact with shapely and estimated by sub-cell sampling without it.
A bare 4-tuple bbox is rejected on purpose, because openeolib's ERA5 bbox
order (N, W, S, E) differs from GeoJSON's (W, S, E, N).

`openeolib.palettes` holds domain color tables tied to a physical quantity,
kept separate from `theme.py`'s generic style values. It depends only on
matplotlib, so it can be imported without the optional provider
dependencies. Currently it provides the JMA rainfall-intensity palette
(`JMA_RAIN_BOUNDARIES`, `JMA_RAIN_COLORS`, and `jma_rain_cmap_norm()` for a
ready-made `(cmap, norm)` pair) -- the same table `providers.rain` uses
internally, exposed for callers who want it outside that provider.

`ERA5Provider` supports surface fields and pressure-level wind processing, including height-aware interpolation for wind products and optional pressure-level specific humidity for moisture-transport calculations such as Integrated Vapor Transport (IVT, computed via `eo_utils.compute_ivt()` -- trapezoidal pressure-level integration that sorts levels internally so the result's sign never depends on the order the caller passed them in). `fetch_grid_specific_humidity_at_pressure_levels()` returns the full multi-level specific-humidity + geopotential-height profile over a bbox (rather than a single level), for callers doing AGL-layer water-vapor integration -- see `openeolib.processing.pwv_layers`, which decomposes a GNSS station's total PWV into AGL layers (e.g. 0-1/1-2/2-3/3-6 km) using that profile's vertical shape. `cds_retry_max`/`cds_sleep_max` bound cdsapi's own inner retry loop (library default up to ~16.7 hours per request) separately from `ERA5Provider`'s outer retry loop, so a real outage surfaces on a predictable timescale instead of a background job appearing stuck for hours. Provider modules return data; visualization remains a separate concern.

Each `ERA5Provider` method takes one `dt`; `fetch_series(method, start, end, **kwargs)` runs any of them over an hour range, e.g. `provider.fetch_series("fetch", datetime(2021, 1, 1), datetime(2021, 1, 31, 23), lat=35.0, lon=135.0)`, and returns `{hour: result}`. It downloads the range first in a few CDS requests (whole months for ERA5-Land and single levels, about a week at a time for pressure levels, split further if CDS rejects a request as too large) instead of one request per hour. Only raw CDS values are cached, one entry per request shape and hour under `cache_dir/raw`, so a method's result never depends on the cache state, and methods asking for the same data share it. The requested area is always sent unchanged: pressure-level values differ in the last float32 bits when the area changes. Boundary-layer height is not an ERA5-Land variable; `enable_blh=True` fetches it from ERA5 single levels (0.25 deg, a separate request), and without it `boundary_layer_height` is `None`. Point values from single levels (100 m wind, boundary-layer height) come from the nearest 0.25 deg node.

`EOProvider` (the S2/ERA5/S5P integration used by `EOAnalyzer`) persists every
`fetch()` result on disk (`ScienceCacheStore`, `enable_cache=True` by default),
keyed by `(lat, lon, dt, band_set, radius_km, enable_100m_wind)`. An outright
failure (no S2 scene at all) is never cached, so a later call retries instead
of replaying the same empty result forever.

`PlanetaryComputerProvider`, in `providers/planetary_computer.py`, is a
standalone alternative to `Sentinel2Provider`/`Sentinel5PProvider` that reads
Microsoft Planetary Computer's public STAC catalog instead of OpenEO/CDSE --
no account or authentication required, just a free, auto-renewed SAS token
(`planetary-computer` SDK). Use it directly (`fetch_sentinel2()` /
`fetch_sentinel5p()`) when OpenEO/CDSE auth is unavailable or undesired; it
is independent of `OpenEOClient` and not wired into `EOProvider`. Two
differences from the OpenEO-backed providers follow from what the PC STAC
catalog actually exposes: `fetch_sentinel2()`'s `sza`/`vza`/`amf` are always
`None`, since PC's `sentinel-2-l2a` collection has no per-pixel sun/view
zenith angle raster (OpenEO's `SENTINEL2_L2A` synthesizes one server-side);
and `fetch_sentinel5p()` bins the raw swath pixels (irregular per-pixel
lat/lon, full-orbit netCDF files rather than per-tile COGs) onto a regular
grid itself, since PC has no pre-gridded product. Requires the
`planetary_computer` extra (`pystac-client`, `planetary-computer`,
`stackstac`).

`fetch_sentinel2()` also accepts `level="l1c"` for top-of-atmosphere data
alongside the default `level="l2a"` (surface reflectance); `Sentinel2Provider`
only exposes L2A. The L1C collection has no SCL band, so cloud/no-data
masking is skipped and `cloud_ratio` is always `None` for `level="l1c"` --
only `scene_cloud_cover` (the STAC item's whole-scene `eo:cloud_cover`,
distinct from `cloud_ratio`'s AOI-local SCL-based fraction) is available.
`max_cloud_cover` excludes scenes above a given `eo:cloud_cover` from the
STAC search entirely, `time_delta_days` overrides the search window's
half-width (default: the same window `Sentinel2Provider` uses), and
`preferred_tile` (an MGRS tile id, e.g. `"14SKA"`) breaks ties between
same-orbit adjacent-tile candidates that otherwise tie on time and cloud
cover -- useful when fetching a second, comparison date and avoiding a
spurious cross-UTM-zone discrepancy against the first. The returned dict
also includes `mgrs_tile` and a `grid` dict (`crs`/`transform`/`shape`/
`resolution`/`bounds`) describing the returned arrays' geolocation. The
`preferred_tile` tiebreak uses `shapely` (part of the `planetary_computer`
extra) when available, and otherwise falls back to plain nearest-time/
cloud-cover selection.

`PlanetaryComputerProvider.__init__()` and `EarthSearchProvider.__init__()`
(see below) both accept `scl_mask_classes`/`scl_cloud_classes` to override
which SCL classes `fetch_sentinel2()` masks out and counts as cloud, on top
of the `Sentinel2Provider.__init__()`-level default `(0, 9, 10)`/`(9, 10)`
(NO_DATA/CLOUD_HIGH_PROBABILITY/THIN_CIRRUS masked; the latter two counted
as cloud). All three providers share this same default; pass e.g.
`scl_mask_classes=(0, 3, 8, 9, 10)` to also mask CLOUD_SHADOWS (3) and
CLOUD_MEDIUM_PROBABILITY (8) if the default's lighter masking lets too
many cloud-affected pixels through for a given use case.

`EarthSearchProvider`, in `providers/earth_search.py`, is a second no-auth
Sentinel-2 alternative, reading Element84's public Earth Search STAC catalog
(the AWS-hosted "Sentinel-2 Cloud-Optimized GeoTIFFs" dataset) instead of
Planetary Computer's Azure-hosted one. Both catalogs expose the same
community COG convention (asset ids, `eo:cloud_cover`, `s2:mgrs_tile`), so
`EarthSearchProvider.fetch_sentinel2()` has the identical signature and
return shape as `PlanetaryComputerProvider.fetch_sentinel2()` (`level`,
`max_cloud_cover`, `preferred_tile`, `time_delta_days`, ...) -- the two
providers share their STAC search/tiebreak/retrieval core
(`providers/_stac_s2.py`, not a public module) and differ only in endpoint
and the fact that Earth Search's asset URLs are plain public HTTPS, needing
no SAS-token signing step. Use it as a fallback when Planetary Computer is
unreachable, or when AWS-side latency/egress makes it the better choice for
a given deployment. Requires the `earth_search` extra (`pystac-client`,
`stackstac`, `shapely`); unlike `planetary_computer`, it does not need the
`planetary-computer` SDK.

`HRRRProvider`, in `providers/hrrr.py`, retrieves 10 m wind from NOAA's
High-Resolution Rapid Refresh model (CONUS domain, ~3 km grid, hourly
cycles), read from the public, anonymously-readable AWS Open Data bucket --
no account or authentication required. `fetch_wind(lat, lon, dt,
radius_km=, forecast_hour=0)` downloads only the two 10 m-above-ground
UGRD/VGRD GRIB2 messages it needs, located via the archive's own
`.idx` sidecar and fetched with HTTP byte-range requests rather than the
full ~100-150 MB GRIB2 file (the same approach the community "Herbie" tool
uses). HRRR's native grid is a curvilinear Lambert Conformal projection, not
a regular lat/lon grid, so the returned `u10`/`v10`/`wind_speed`/`wind_deg`
arrays come with matching 2D `lats`/`lons` coordinate arrays (cropped to
roughly `radius_km` around the AOI) rather than a 1D axis pair, alongside
`nearest_u10`/`nearest_v10`/`nearest_wind_speed`/`nearest_wind_deg` at the
single closest grid cell. Requests outside the CONUS domain return `None`.
It is independent of `ERA5Provider`/`OpenEOClient` and not wired into
`EOProvider`; use it as a CONUS-only, higher-resolution alternative to
`ERA5Provider`'s surface wind. Requires the `hrrr` extra (`cfgrib`,
`xarray`). Like `MSMProvider` below, `fetch_wind()` results are cached on
disk (`ScienceCacheStore`, `enable_cache=True` by default, `cache_dir=
"./data/hrrr_cache"`) keyed by its own arguments, so a repeat call for the
same (lat, lon, dt, radius_km, forecast_hour) skips the idx fetch, both
byte-range GRIB2 fetches, and the cfgrib decode entirely.

`MSMProvider`, in `providers/msm.py`, retrieves JMA MSM GPV surface and
pressure-level fields and requires the `jma` extra. Its results are also cached
(`ScienceCacheStore`, `enable_cache=True` by default, `cache_dir` nested
under `data_root` so distinct instances never share a cache), so a repeat
`get_wind()`/`get_surface()`/`get_wind_forecast()`/`get_surface_forecast()`/
`get_pressure_level_scalars()` call with the same arguments skips
re-downloading and re-reading the GRIB2 file -- the dominant per-call cost
once the raw file is local. `download_read_timeout_s` (default 20) controls how long a single
file download will wait on a slow-but-not-closed connection before
`retry_get()` counts it as a failure and retries. This interacts with
`max_retries` multiplicatively (worst case is `max_retries *
download_read_timeout_s`, plus backoff); a caller expecting unusually
large/slow bulletins can pass a larger value explicitly.

`MSMProvider` reads GRIB2 with eccodes directly (`grib_backend="eccodes"`,
the default), message by message, releasing each message as soon as its
values are copied. The former
cfgrib-based reading, `grib_backend="cfgrib"`, is **deprecated** and will be
removed in a future release; selecting it emits a `FutureWarning`. In a
long-running process it can accumulate eccodes memory until it fails with
"Memory allocation error", after which every later GRIB read in that
process fails too. The eccodes reader
preserves cfgrib's variable names, values, dimensions, coordinates, and core
CF variable metadata (`units`, `long_name`, and `standard_name`). For MSM,
cfgrib is imported only when the deprecated backend is selected.

The same module also exposes a subprocess-isolated batch runner --
`BatchRequest`/`run_batch_isolated()`, for resumable, crash-isolated
batches. `BatchRequest`
(one `(key, method, kwargs)` call against `MSMProvider`) and
`run_batch_isolated()` split many requests into small chunks (default 20)
and re-launch a brand-new `python -m openeolib.providers.msm --chunk-worker`
subprocess per chunk. A single request's own failure only marks that request
(via a caller-supplied `extractor` function); a whole chunk crashing
(non-zero exit, a timeout, or no result from the subprocess) NaN-fills
that chunk's rows instead of aborting the batch. Results are appended to
`output_csv` after every chunk, and a request whose `key` already has a
row there is skipped on a later run -- so an interrupted batch resumes
rather than restarting.

For whole-bulletin work (moisture columns, many levels or forecast
steps), `MSMProvider` also has whole-file readers (always eccodes):
`get_pressure_level_profile()` returns
t/u/v/r (and gh/w) on every pressure level and every forecast step of an
L-pall bulletin in one call (r exists on 1000-300 hPa only), and
`get_surface_fields_forecast()` returns Lsurf surface pressure [hPa] and
hourly precipitation [mm] (the field cfgrib names "unknown"; selected by
GRIB discipline 0 / category 1 / number 8). `get_tpw_ivt()` combines them
into precipitable water and IVT, cutting the column at MSM's own surface
pressure by default -- without the cut, below-ground pressure levels are
integrated as if they were air, inflating precipitable water over mountains.
The underlying integrals are
`eo_utils.compute_column_water_vapor()` and `eo_utils.compute_ivt(...,
surface_pressure_hpa=...)`, with `eo_utils.specific_humidity_from_rh()`
(Bolton 1980) for q. `download_range(start, end, products, max_workers)`
fetches every 3-hourly bulletin in a period in parallel, and the
`raw_dir` constructor argument lets the provider read from and download
into an existing flat archive directory (files already there are never
re-downloaded). The static `read_pressure_level_profile()` /
`read_surface_fields()` take a local file path directly.

`LFMProvider`, in `providers/lfm.py`, retrieves JMA LFM (local forecast
model, 局地モデル) GPV data -- 1 km-resolution surface and model-level
fields over Japan -- reusing `MSMProvider`'s download/cache/retry
scaffolding and GRIB2-handling conventions. It reads GRIB2 with eccodes
directly, message by message, like `MSMProvider`'s default backend; it does
not use cfgrib.

**Status: unconfirmed API, scaffold only.** Unlike MSM (freely mirrored
by Kyoto University's RISH archive), LFM is distributed only through
JMBSC's "気象庁クラウド環境" (JMA Cloud environment), a contracted,
paid service (申込み + 情報提供負担金). The following are not publicly
documented and could not be confirmed without a contract; they are
documented only in JMBSC's 利用者マニュアル, obtained from JMBSC's
配信事業部 (haisin@jmbsc.or.jp):

- **`base_url`**: the download endpoint is unknown. Pass it explicitly
  (`base_url=`) or via the `JMBSC_LFM_BASE_URL` environment variable
  once obtained from the manual; any call needing it raises
  `RuntimeError` until then.
- **Authentication**: deliberately restricted to `~/.netrc` (HTTP Basic
  auth via `requests`' own netrc lookup) -- this module accepts no
  `api_key`/token/header parameter. Put the JMBSC-issued credentials in
  `~/.netrc` under the download host's name.
- **Filename pattern**: `_build_filename()` uses a best-effort guess
  following JMA's other GPV products' naming convention, substituting
  "LFM" for the model tag -- unverified against a real LFM filename.
- **Variable/grid naming**: `get_field()` takes `variable_candidates`/
  `level`/`level_coord` from the caller rather than hardcoding an LFM
  schema, since real GRIB2 variable naming for LFM's model-level fields
  is unconfirmed; use `list_variables()` against a real downloaded
  bulletin to discover it. The ecCodes reader preserves either 1-D
  rectilinear axes or matching 2-D latitude/longitude coordinates and
  masks records outside the requested box; confirm the actual grid
  shape and coordinate conventions with a real bulletin before relying
  on this scaffold.

`AMeDASProvider` and `RadiosondeProvider` retrieve historical JMA surface
observations and University of Wyoming upper-air soundings respectively, both
by scraping the providers' public data-search pages (`requests` +
`beautifulsoup4`) rather than an official API, and both cache retrieved
results indefinitely on disk (`ScienceCacheStore`), since historical
observations never change. Both return `xr.Dataset` (`time`-dimensioned for
AMeDAS; `time`/`level`-dimensioned, NaN-padded across variable-length
profiles, for radiosonde soundings), matching the convention used by the
rest of the providers rather than a bespoke `Dict` shape. JMA prints the
wind-speed/direction column pair under two different main-label word
orders depending on station class -- `"風向・風速"` for type-A/B
(staffed/pressure) stations vs `"風速・風向"` for type-C (AMeDAS-only,
the majority of the network) stations, the latter also using a
`"平均風速(m/s)"` sub-label instead of `"風速"` -- so `AMeDASProvider`
normalizes both the main and sub header label (stripping any
`"(unit)"` suffix) before matching, rather than recognizing only one
word order.

`AMeDASProvider` also retrieves JMA's 10-minute table
(`fetch_10min_day()` / `fetch_10min_range()` / `fetch_10min()`, same
station identifiers and `xr.Dataset` shape as the hourly methods,
`attrs["interval_minutes"] == 10`): pressure (type-A/B only),
precipitation, temperature, relative humidity, mean wind
(`wind_speed_ms`, `wind_direction`), maximum instantaneous wind
(`wind_gust_ms`, `wind_gust_direction`) and `sunshine_minutes`. The
10-minute pages use a two-row header for type-A/B stations but a
three-row header for type-C stations (`平均` / `最大瞬間`, each over
`風速` / `風向`), so the header is parsed as a tree of any depth; a
column it does not recognize raises a warning instead of being dropped
silently. Type-C pages list every column whatever the station observes;
columns a station does not observe are all `NaN`. 10-minute results are
cached separately from hourly ones, keyed by a parser version so that
entries written by an older parser are never reused.

`PWVProvider` (GPT3 blind climatological grid) and `VMF3Provider` (VMF3_OP
per-epoch NWP grid), in `providers/pwv.py`, are independent, GNSS-external
sources of Zenith Hydrostatic/Wet Delay for validating or cross-checking an
ERA5-based PWV pipeline. Both ultimately produce a PWV estimate via the
Bevis et al. (1994) Pi(Tm) conversion; see the module docstring for the
K1/K2 refractivity-constant caveat when comparing against another pipeline's
convention.

`UNRTropProvider`, in `providers/unr_trop.py`, retrieves 5-minute GNSS
tropospheric products (zenith total/wet delay, tropospheric delay gradients
Gn/Ge, PWV, weighted mean temperature) from the Nevada Geodetic Laboratory
(NGL/UNR) archive -- >18,600 stations worldwide, independent of any national
agency's own real-time solution. Same two-stage API
(`find_stations_in_bbox()` -> `fetch_day()`) and `ScienceCacheStore`
persistent-cache convention as `AMeDASProvider`/`RadiosondeProvider`.

`CWOPProvider`, in `providers/cwop.py`, retrieves observations (wind,
temperature, humidity, pressure, precipitation) from privately operated
weather stations in the Citizen Weather Observer Program (CWOP, MADIS data
provider `APRSWXNET`), via the NOAA MADIS public archive (no account needed,
2001 onward). Each hourly archive file (~33 MB gzip'd, worldwide) is
decompressed in memory and cut to a bounding box (default: Japan) before
caching. Coverage in Japan is sparse and uneven (call `list_stations()` for a
period's roster), anemometer height is not recorded, and wind speed is
quantized to 1 mph.

**MADIS data redistribution.** `CWOPProvider` only downloads data for your
own use; do not publish its cache as-is. CWOP is MADIS distribution category
3 ("Public - Full distribution"), but per the
[MADIS restrictions](https://madis.ncep.noaa.gov/madis_restrictions.shtml),
"the mesonet data providers request that users not redistribute or place the
data on web pages without notifying NOAA" -- contact the MADIS team before
publishing the data itself. Data from providers in categories other than
full distribution (possible with `data_providers=None`) must not be
redistributed, and redistribution of real-time data, including web
displays, is prohibited. Derived figures and statistics should cite
MADIS/CWOP as the source.

`NIESAirMonitorProvider`, in `providers/nies_air.py`, retrieves hourly values
from Japan's ~1,800 air-quality monitoring stations (大気汚染常時監視局) via the
NIES download service (https://tenbou.nies.go.jp/download/). Most stations
also measure wind direction/speed, temperature and humidity, and, unlike
AMeDAS, the station attribute file records each instrument's height above
ground (anemometers are usually on rooftops), reported as
`station_wind_height_m`. Confirmed
values only, fiscal year 2009 onward, published with a lag of about 1.5
years. Prefecture-year zip files are cached as downloaded.

`TBEICProvider`, in `providers/tbeic.py`, retrieves 15-minute over-water wind
(mean/maximum speed, direction) and air temperature from four MLIT monitoring
posts in inner Tokyo Bay (検見川沖, 川崎人工島, 浦安沖, 千葉港口第一号灯標;
anemometers about 8-10 m above mean sea level) via the Tokyo Bay
Environmental Information Center (https://www.tbeic.go.jp/MonitoringPost/Top),
2010-04 to near real time. These are preliminary (速報値) values; requests are
split into 7-day chunks because the server slows sharply for longer ranges.

**Terms of use for these sources.** NIES: downloaded data may be used
freely; credit the source when publishing results. TBEIC (site terms,
section 1-4): when using the data for citation or research, (1) state TBEIC
and the specific data source in the results, (2) report the purpose and
content of use by e-mail to the TBEIC administrator
(pa.ktr-kankyo@mlit.go.jp), sending a copy of the results if possible, and
(3) do not use the data commercially.

`processing/grid.py` provides `regrid_to_grid()` (regrid one regular lat/lon grid
onto another, e.g. a coarser NWP product's native grid onto a finer
analysis grid) and `gaussian_weighted_grid_interpolation()` (scattered
station observations, scalar or multi-component, onto a regular grid via
Gaussian-distance weighting). Split out of `eo_utils.py`, which is
intentionally numpy-only, since both need `scipy.interpolate`.

It also provides `nan_aware_gaussian_filter()` and
`nan_aware_gaussian_derivative()`, NaN-aware counterparts of
`scipy.ndimage.gaussian_filter` built on normalized convolution: a gap in
the input (e.g. a land-only product over a domain that includes sea)
only affects pixels with no valid data within the kernel, rather than
spreading NaN across the whole kernel footprint. Both reduce exactly to
the plain `scipy.ndimage` call when the input has no NaN.

`HimawariCloudClient` / `HimawariCloudProvider`, in `providers/himawari.py`,
download and decode Himawari-8/9 AHI gridded cloud-top brightness temperature
from CEReS (Chiba University), requiring the `himawari` extra (`satpy`).
**This module's underlying data has its own license, separate from
openeolib's Apache-2.0 code license: non-commercial/research use only, no
redistribution, and required attribution when used or published.** Both
classes emit a `UserWarning` with the attribution text on instantiation; see
the module docstring for the full terms before using it beyond research.
CEReS distributes this data via FTP only (no HTTP mirror), which outbound
firewalls block on many corporate/institutional networks; pass
`HimawariCloudClient(..., allow_ftp=False)` in that case to use only
files already cached under `raw_dir` (pre-populated from an environment
where FTP does work) -- a cache miss then raises `FileNotFoundError`
immediately instead of waiting out `timeout` seconds per missing step.
`HimawariCloudClient` is the only provider in this package that depends on
FTP; every other provider uses HTTP(S) or an
OpenEO/CDS API client.

`HimawariAWSClient` / `HimawariAWSProvider`, in `providers/himawari_aws.py`,
are an HTTPS-only alternative to the above for environments where FTP
itself is unavailable rather than merely slow -- `allow_ftp=False` still
needs *some* pre-populated cache, whereas this module can download
from scratch over plain HTTPS. They read the same underlying AHI
instrument's data from a different distributor: raw Himawari-8/9 AHI
Level-1b Standard Data (HSD), produced by JMA and redistributed by NOAA
via the public, unauthenticated `noaa-himawari8`/`noaa-himawari9` AWS
Open Data buckets, decoded with satpy's `ahi_hsd` reader (also requires
the `himawari` extra, but no additional dependency beyond it). One Full
Disk time step is split across `AWS_SEGMENTS` (10) files that must all
be downloaded and passed to `HimawariAWSProvider.open()` together, so
`HimawariAWSClient.download_range()`/`download_full_disk()` deal in
*groups* of file paths (one list of 10 per time step) rather than one
path per step, unlike `HimawariCloudClient`. The AWS data's license is
considerably more permissive than CEReS's (no non-commercial/
research-only restriction; NOAA/JMA only request attribution for
unaltered data) -- see the module docstring for the exact terms. Both
classes auto-select the `noaa-himawari8`/`noaa-himawari9` bucket by
JMA's Himawari-9 operational cutover time (2022-12-13 05:00 UTC; the last
Himawari-8 slot is 04:50 UTC) unless `satellite=` is passed explicitly.

`HimawariAWSClient(scan="Japan")` downloads the Japan Area product
(`AHI-L1b-Japan`) instead of the Full Disk: one observation every 2.5
minutes (JP01..JP04 within each 10-minute slot), one file each.
`download_range()` then returns one single-path group per observation,
and `download_japan_area(dt)` fetches one observation; both groups go
to `HimawariAWSProvider.open()` unchanged. `scan="FLDK"` (the default)
keeps the Full Disk behavior. `download_range(start, end, max_workers=8)`
downloads several steps concurrently, which matters for long Japan Area
periods (many small files); a file that does not exist on AWS (HTTP 404)
is skipped at once rather than retried.

`processing/satellite.py` works on the decoded images. `block_motion()` estimates
cloud displacement between two images by SSD block matching with
sub-pixel refinement, `motion_field()` spreads it to every pixel,
`advect()` moves an image along that field, and `advected_difference()`
combines them into an advection-corrected temporal difference (the
brightness-temperature change of the clouds themselves rather than of a
fixed pixel they pass over). `parallax_shift()` gives the apparent
displacement of a cloud top of a given height seen from a geostationary
satellite (about 0.08 deg northward for a 10 km cloud top at 35 N on
Himawari's sub-satellite meridian).

```python
from openeolib.processing.satellite import advected_difference, parallax_shift

tb = ds["tbb"].values                      # HimawariAWSProvider().open(...), 2.5 min apart
dtb, prev_moved, motion = advected_difference(tb[0], tb[6])   # 15-minute change
dlat, dlon = parallax_shift(35.0, 140.7, cloud_top_km=10.0)
```

For ERA5, CDS credentials and the current CDS API/client configuration are required for live retrieval.

## Animation

`GridAnimationEngine` in `openeolib.animation` is the generic animation component. Domain-specific wrappers, such as `animate_rain()` in `providers.rain`, configure rainfall-specific variables and color scales before delegating rendering to the generic engine.

```python
from openeolib.animation import GridAnimationEngine

engine = GridAnimationEngine()
engine.animate_scalar_field(
    grids=grids,
    lats=lats,
    lons=lons,
    timestamps=timestamps,
    output_path="animation.gif",
)
```

GIF frames share one 256-colour palette, so a colour means the same value in
every frame. While writing, the frames are staged as PNG files in a temporary
directory (a few hundred KB each) and then appended to the GIF one at a time,
so memory use does not grow with the number of frames. For long time series,
an `.mp4` output path (needs ffmpeg) gives smaller files.

## Testing

The test suite is maintained for internal development only. `tests/` is
intentionally excluded from both the published package (see `MANIFEST.in`)
and this repository's git history (see `.gitignore`), so it is not available
to clone or install. This has no bearing on installing or using the library.

## License

OpenEO-LIB is distributed under the Apache License 2.0. See `LICENSE` for the full license text.
