Metadata-Version: 2.4
Name: l2o-qc
Version: 0.1.1
Summary: ESA ROSE-L Level-2 Ocean Processor Quality Control Library
Author: RL2Ocean Project Contributors
License-Expression: Apache-2.0
Project-URL: Homepage, https://rl2ocean.org/
Project-URL: Documentation, https://rl2ocean.pages.gitlab.dlr.de/l2o-qc
Project-URL: Repository, https://gitlab.dlr.de/rl2ocean/l2o-qc
Project-URL: Issues, https://gitlab.dlr.de/rl2ocean/l2o-qc/-/issues
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: GIS
Classifier: Topic :: Scientific/Engineering :: Oceanography
Classifier: Typing :: Typed
Requires-Python: >=3.13
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=2
Requires-Dist: scipy>=1.16
Requires-Dist: xarray>=2026
Requires-Dist: pandas>=3
Requires-Dist: netCDF4>=1.7
Requires-Dist: pyyaml>=6
Requires-Dist: requests>=2.32
Requires-Dist: keyring>=25
Requires-Dist: copernicusmarine>=2.4
Requires-Dist: rioxarray>=0.23
Requires-Dist: rasterio>=1.5
Requires-Dist: h5netcdf>=1.8
Requires-Dist: fsspec>=2026.1
Provides-Extra: notebook
Requires-Dist: jupyter<2,>=1; extra == "notebook"
Provides-Extra: plot
Requires-Dist: matplotlib; extra == "plot"
Requires-Dist: cartopy<0.26,>=0.25; extra == "plot"
Requires-Dist: cmocean; extra == "plot"
Provides-Extra: eumdac
Requires-Dist: eumdac; extra == "eumdac"
Provides-Extra: gportal
Requires-Dist: paramiko; extra == "gportal"
Provides-Extra: era5
Requires-Dist: cdsapi; extra == "era5"
Provides-Extra: gts
Requires-Dist: ecmwf-api-client; extra == "gts"
Requires-Dist: pdbufr; extra == "gts"
Provides-Extra: soil-moisture
Requires-Dist: pytesmo; extra == "soil-moisture"
Requires-Dist: ismn; extra == "soil-moisture"
Requires-Dist: earthaccess; extra == "soil-moisture"
Requires-Dist: ascat; extra == "soil-moisture"
Requires-Dist: pdbufr; extra == "soil-moisture"
Requires-Dist: pyorbital; extra == "soil-moisture"
Requires-Dist: l2o-qc[era5]; extra == "soil-moisture"
Provides-Extra: all
Requires-Dist: l2o-qc[era5,eumdac,gportal,gts,notebook,plot,soil_moisture]; extra == "all"
Dynamic: license-file

# L2O-QC
This repository contains the ROSE-L Level-2 Ocean Processor Quality Control Library, L2O-QC.


- **Website:** [rl2ocean.org](https://rl2ocean.org/)
- **Source code:** [gitlab.dlr.de/rl2ocean/l2o-qc](https://gitlab.dlr.de/rl2ocean/l2o-qc)
- **Documentation:** [rl2ocean.pages.gitlab.dlr.de/l2o-qc](https://rl2ocean.pages.gitlab.dlr.de/l2o-qc)
- **Installation:** [Quick start installation](https://rl2ocean.pages.gitlab.dlr.de/l2o-qc/main/quickstart/installation.html)
- **Software User Manual:** [SUM](https://rl2ocean.pages.gitlab.dlr.de/l2o-qc/main/manual/index.html)
- **Contributing:** [Contributing guide](https://rl2ocean.pages.gitlab.dlr.de/l2o-qc/main/contributing.html)
- **Bug reports:** [Issue tracker](https://gitlab.dlr.de/rl2ocean/l2o-qc/-/issues)
- **Software Reuse File:** [SRF](https://rl2ocean.pages.gitlab.dlr.de/l2o-qc/main/software-reuse.html)

# SAR L2 Ocean Data Validation Toolbox

A standalone Python package for validating Level 2 SAR products against multi-source in-situ and satellite observations.

Supports validation of **wind** (speed + direction), **ocean currents**, **significant wave height**, and **soil moisture** -- against in-situ, satellite, and model (ERA5, HyCOM) sources.

Supported SAR products:
- Sentinel-1 L2_OCN (Level 2 Ocean): wind/waves/currents
- Sentinel1 clms ssm (Surface Soil Moisture; Europe; daily): soil_moisture
- NISAR SME2 (beta & provisional): soil_moisture
- RADARSAT-2 (NOAA NCEI SAR-derived ocean wind, speed only): wind

Note that the soil moisture part is not developed towards RL2Ocean and will be removed together with its dependencies in the final version of the L2O-QC software.
---

## Architecture

The toolbox is structured around 6 sequential steps that together form a complete validation workflow:

```
Step 0 — Create recipe (.yaml)
          │  define validation variable (wind / currents / waves / soil_moisture) --> selects the validation
          |  data sources. For wind and soil_moisture the SAR source of interest can be chosen.
          │  define geographic bounds (min/max lon/lat)
          │  define temporal bounds (start/end dates)
          ▼
Step 1 — Download data
          │  download selected SAR data source + all validation sources
          │  for the recipe region and time window
          |  --dry-download available to check product availibility
          |  default: only downloads validation-source data predicted to collocate with
          |  the SAR data; --download-all-in-bbox downloads everything in the bbox/window.
          ▼
Step 2 — Convert to xarray.DataTree
          │  standardize all formats + attach CF-convention metadata
          │  filter to recipe geographic/temporal bounds +/- tolerances
          │  handle different grids / resolutions
          ▼
Step 3 — Collocation + store collocated pairs
          │  point vs. point    (mooring / buoy / drifter / ferrybox / tidal gauge vs. SAR wave parameter)
          │  point vs. layer    (mooring / buoy / drifter / ferrybox / tidal gauge vs. SAR)
          │  layer vs. layer    (scatterometer / altimeter / radiometer / HF-radar vs. SAR)
          |  model vs. layer    (ERA5 / HYCOM vs. SAR)
          |  create a collocation diagnostics plot
          |  --dry-collocation and --dry-collocation-detail preview the predicted collocation
          |  before running steps 1-2 (download/convert)
          │  store collocated data + spatial/temporal offset metadata
          ▼
Step 4 — Compute statistics
          │  bias, RMSE, correlation, scatter index
          │  per platform/source
          ▼
Step 5 — Generate visualisation & report
          │  scatter plots, geographic maps
          │  statistics charts, residuals
          │  PDF validation report
```

Steps 2–5 are dataset-agnostic: they work the same regardless of which validation sources were
downloaded in step 1.

---

## Package Layout

```
sar_validation/
├── cli.py                  # Command-line entry point
├── core/
│   ├── recipe.py           # Recipe dataclasses (YAML ↔ Python)
│   ├── orchestrator.py     # Orchestrates step 1 (download all sources)
│   ├── orbit_coverage.py   # Orbit-based geographic pre-filter (Space-Track TLE + SGP4 propagation) used by the H-SAF downloader
│   ├── datatree_converter.py  # Step 2: convert to xarray.DataTree
│   ├── collocation.py      # Step 3: collocation algorithms
│   ├── statistics.py       # Step 4: compute bias, RMSE, correlation, scatter index
│   ├── visualization.py    # Step 5: scatter plots, geographic maps, statistics charts, residuals
│   ├── _variable_map.py    # Variable mapping (wind/currents/waves)
│   └── __init__.py         # Package exports
└── downloaders/
    ├── base.py                                  # Shared credential handling & helpers
    ├── _hf_radar_regions.py                     # Copernicus Marine HF-radar region bbox/flag table (shared, not a downloader)
    ├── _noaa_hfr_regions.py                     # NOAA HF-radar region bbox/dataset-id table (shared, not a downloader)
    ├── _rss_bytemap.py                          # Decoder for RSS binary bytemap (.gz) radiometer products
    ├── altimeter_downloader.py                  # Along-track SWH/wind altimetry via Copernicus Marine
    ├── ascat_soil_moisture_downloader.py        # ASCAT (MetOp) soil moisture (SOMO12) via EUMETSAT EUMDAC (discontinued after 2025-07-15)
    ├── hsaf_downloader.py                        # ASCAT (MetOp) soil moisture NRT (H122 6.25km default, H29 12.5km opt-in) via H-SAF FTP, rolling last 60 days
    ├── earthdata_soil_moisture_downloader.py    # AMSR-E/2, SMAP, and NISAR SME2 soil moisture via NASA Earthdata
    ├── era5_downloader.py                       # ERA5 reanalysis (wind/waves/soil_moisture) via Copernicus CDS
    ├── gportal_downloader.py                    # AMSR2 soil moisture via JAXA G-Portal (SFTP)
    ├── gts_buoy_downloader.py                   # Moored/drifting buoy wind/waves/currents via WMO GTS (ECMWF MARS obstype 181/182)
    ├── hf_radar_downloader.py                   # Near-real-time HF-radar surface currents via Copernicus Marine
    ├── hf_radar_historical_downloader.py        # Delayed-mode/historical HF-radar currents via Copernicus Marine
    ├── hf_radar_us_downloader.py                # US HF-radar waterfall selector: NOAA ERDDAP → NOAA THREDDS → Copernicus Marine
    ├── hycom_downloader.py                      # HyCOM ocean model surface currents (water_u/water_v) via THREDDS OPeNDAP
    ├── insitu_currents_historical_downloader.py # Delayed-mode in-situ currents (ADCP/Argo/drifter/glider) via Copernicus Marine
    ├── insitu_downloader.py                      # Moorings/buoys/ferrybox via Copernicus Marine
    ├── ismn_downloader.py                        # ISMN local-archive soil-moisture station selector (no download API)
    ├── noaa_hfradar_downloader.py                # NOAA HFRnet gridded surface currents via ERDDAP (rolling ~90-day window)
    ├── noaa_hfradar_thredds_downloader.py        # NOAA HFRnet gridded surface currents via NCEI THREDDS archive (2006-present)
    ├── radarsat2_wind_downloader.py               # RADARSAT-2 SAR-derived ocean wind speed via NOAA NCEI THREDDS archive
    ├── radiometer_downloader.py                  # RSS radiometer ocean winds (AMSR2 NetCDF + GMI/SSMIS/WindSat bytemaps)
    ├── scatterometer_downloader.py               # ASCAT (MetOp) wind via EUMETSAT EUMDAC
    ├── scatterometer_ftp_downloader.py           # HY-2B/HY-2C/Oceansat-3 scatterometer wind via OSI-SAF FTP (only last 3 days)
    ├── sentinel1_l2_ocn_downloader.py            # Sentinel-1 L2_OCN (wind/currents/waves) via Copernicus Dataspace (CDSE)
    ├── sentinel1_soil_moisture_downloader.py     # Sentinel-1 CLMS Surface Soil Moisture via Copernicus Dataspace (CDSE)
    ├── cds_soil_moisture_downloader.py           # C3S CDS satellite soil moisture (ACTIVE/PASSIVE/COMBINED, 0.25°) via cdsapi
    └── smos_downloader.py                        # SMOS L2 soil moisture via ESA SMOS FTPS
```
---

## Quick Start

### 1. Install

```bash
# Editable install with all extras
pip install --group dev

# Or just core
pip install -e .
```

### 2. Create a recipe

```bash
sar-validate --create-recipe wind
sar-validate --create-recipe currents
sar-validate --create-recipe waves
sar-validate --create-recipe soil_moisture
```

This writes a YAML recipe to `recipes/<name>_validation.yaml`.
Edit the file to adjust the geographic region, time window, and validation sources.

### 3a. Dry-run (check what will be downloaded)

```bash
sar-validate --recipe recipes/wind_validation.yaml --dry-download
```

### 3b. Dry-collocation (check whether there would be any collocation prior to downloading)

```bash
sar-validate --recipe recipes/wind_validation.yaml --dry-collocation
```

To get more details on the number of collocated pairs predicted, run

```bash
sar-validate --recipe recipes/wind_validation.yaml --dry-collocation-detail
```

### 4. Execute
To only download the data:

```bash
sar-validate --recipe recipes/wind_validation.yaml
```

Downloads are saved under `data/<timerange>_<bounds>/`.

To run all steps of the toolbox at once:

```bash
sar-validate --recipe recipes/wind_validation.yaml --plot
```

### 5. Convert, collocate, and analyse

**Step 2 — convert and store as a DataTree:**
```bash
sar-validate --recipe recipes/wind_validation.yaml --convert
```
The datatree is saved under `data/<timerange>_<bounds>/datatree.nc`.

**Step 3 — collocate and store as a Dataset:**
```bash
sar-validate --recipe recipes/wind_validation.yaml --collocate
```

Standard layer-vs-layer collocation method uses cell-averaging for SAR data. If you want each individual SAR cell to be linked to validation data, use:
```bash
sar-validate --recipe recipes/wind_validation.yaml --collocate --layer-vs-layer-collocation-method individual
```

or to include both options:
```bash
sar-validate --recipe recipes/wind_validation.yaml --collocate --layer-vs-layer-collocation-method both
```

The collocation results are saved under `data/<timerange>_<bounds>/collocation_results.nc` and/or `data/<timerange>_<bounds>/collocation_results_individual.nc`.

**Step 4 — calculate statistics:**
```bash
sar-validate --recipe recipes/wind_validation.yaml --stats
```

The statistics are saved under `data/<timerange>_<bounds>/validation_statistics_<sar_var>_vs_<val_var>.nc/.csv`.

**Step 5 — visualize and create PDF report:**
```bash
sar-validate --recipe recipes/wind_validation.yaml --plot
```

The validation report is saved under `data/<timerange>_<bounds>/validation_report.pdf`.

---

## Supported Data Sources

| Source | Variable | Downloader | Service | Temporal coverage |
|--------|----------|------------|---------| ----------------- |
| Sentinel-1 L2_OCN | wind / currents / waves | `sentinel1_l2_ocn_downloader` | Copernicus Dataspace (CDSE) | 2014-10-03 - present |
| RADARSAT-2 | wind (speed only) | `radarsat2_wind_downloader` | NOAA NCEI THREDDS | 2014-05-02 - present |
| Moorings / Buoys / Ferryboxes | wind / currents / waves | `insitu_downloader` | Copernicus Marine | varies by platform; max 2020-01-01 - present |
| GTS Moored / Drifting Buoys (WMO GTS) | wind / waves / currents | `gts_buoy_downloader` | ECMWF MARS (obstype 181/182) | operational archive; 28-11-2016 - present |
| Delayed-mode in-situ currents (ADCP / Argo / drifter / glider) | ocean currents |   `insitu_currents_historical_downloader` | Copernicus Marine | varies by platform (6 - 24 months latency) |
| HF Radar (near-real-time) | ocean currents | `hf_radar_downloader` | Copernicus Marine | varies by radar; max 2020-01-01 - present |
| HF Radar (delayed-mode/historical) | ocean currents | `hf_radar_historical_downloader` | Copernicus Marine | varies by platform |
| HF Radar (US regions) | ocean currents | `hf_radar_us_downloader` | NOAA ERDDAP → NOAA THREDDS → Copernicus Marine (waterfall) | see HF Rader (NOAA HFRnet) |
| HF Radar (NOAA HFRnet, rolling window) | ocean currents | `noaa_hfradar_downloader` | NOAA ERDDAP | 90 days rolling window |
| HF Radar (NOAA HFRnet, full archive) | ocean currents | `noaa_hfradar_thredds_downloader` | NOAA NCEI THREDDS | 2006 - present (~1 month latency) |
| ASCAT (MetOp-B/C) | wind | `scatterometer_downloader` | EUMETSAT EUMDAC | MetOp-B/C: 2012/2019 - present |
| HY-2B / HY-2C / Oceansat-3 | wind | `scatterometer_ftp_downloader` | OSI-SAF FTP | last 3 days |
| Radiometer — AMSR2 (NetCDF); GMI, SSMIS F16/F17/F18, WindSat (binary bytemaps) | wind (+ direction from WindSat) | `radiometer_downloader` | RSS `data.remss.com` (public HTTPS) | AMSR2/GMI/SSMIS F16/F17/F18: 2012-07-02/2014-03-04/2003-10-26/2006-11-04/2009-10-18 - present |
| Altimeter (10 missions, along-track) | wind (1 Hz only) / significant wave height | `altimeter_downloader` | Copernicus Marine | 1 Hz: varies by mission, 2023-11-21 to 2023-12-28 - present; 5 Hz (6 of the missions): varies by mission, 2026-03-07/09 - present |
| Altimeter, reprocessed (12 missions, along-track) | significant wave height | `reprocessed_altimeter_downloader` | Copernicus Marine | 1991-08-03 - 2023-12-31 |
| Sentinel-1 CLMS Surface Soil Moisture | soil moisture | `sentinel1_soil_moisture_downloader` | Copernicus Dataspace (CDSE) | 2014 - present (Europe only) |
| ASCAT Soil Moisture (SOMO12) | soil moisture | `ascat_soil_moisture_downloader` | EUMETSAT EUMDAC | 2007 - 2025-07-15 |
| ASCAT Soil Moisture NRT (H122/H29) | soil moisture | `hsaf_downloader` | H-SAF FTP | rolling last 60 days (⚠️ gap between 2025-07-15 and 60 days ago is not covered); H122 (6.25km) by default, H29 (12.5km) via `download_kwargs: {hsaf_product: h29}` |
| AMSR-E/AMSR2 (NSIDC-0451 / AU_Land) | soil moisture | `earthdata_soil_moisture_downloader` | NASA Earthdata | AMSR-E: 2002 - 2011; AMSR2: 2012 - 2025-09-01 ⚠️ frozen |
| AMSR2 (JAXA G-Portal) | soil moisture | `gportal_downloader` | JAXA G-Portal (SFTP) | 2012 - present |
| SMAP (SPL2SMP_E) | soil moisture | `earthdata_soil_moisture_downloader` | NASA Earthdata | 2015 - present |
| NISAR SME2 (beta & provisional) | soil moisture | `earthdata_soil_moisture_downloader` | NASA Earthdata | 2025-10-01 - present |
| SMOS (SM_OPER_MIR_SMUDP2) | soil moisture | `smos_downloader` | ESA SMOS FTPS | 2010 - present |
| C3S CDS Soil Moisture (ACTIVE / PASSIVE / COMBINED) | soil moisture | `cds_soil_moisture_downloader` | Copernicus CDS | 1991 - present (ICDR: ~10-day latency) |
| ISMN (International Soil Moisture Network) | soil moisture | `ismn_downloader` | Manual portal download (no API) | Varies by station |
| ERA5 model | wind / waves / soil moisture | `era5_downloader` | Copernicus CDS | 1940 (ERA5) / 1950 (ERA5-Land) - present (~5 day latency) |
| HYCOM model | currents | `hycom_downloader` | HYCOM | 2018-12-04 - present (~48 hour latency) |

`ship_gts` retrieves ship synoptic wind observations from GTS (MARS
obstype 180), the same MARS/BUFR path `buoy_gts` uses for buoys.
`ship_cmems_family` is the Copernicus Marine equivalent (the renamed
`ferrybox` source_type). A recipe may not list both `ship_gts` and
`ship_cmems_family` -- unlike buoys, whether the two feeds report
overlapping physical vessels is unconfirmed, so no combined,
deduplicated source_type exists for ships yet. `ship_gts` is valid for
`"wind"` recipes only; ship reports carry no other variable this
toolbox supports.

NISAR SME2 (`m3 m-3`, L-band, twice-daily per-overpass granules) is a second,
beta/provisional SAR-side source for soil moisture, selectable per recipe via
`sar-validate --create-recipe soil_moisture --sar-source nisar_sme2` (or
`source: nisar_sme2` in the recipe YAML).

RADARSAT-2 (C-band, 0.5 km, NOAA NCEI) is a second SAR-side source for
wind, selectable via `sar-validate --create-recipe wind --sar-source
radarsat2` (or `source: radarsat2` in the recipe YAML). It provides
**wind speed only**. Coverage is global but concentrated over
Alaska/the North Pacific.

Altimeter data (`WAVE_GLO_PHY_SWH_L3_NRT_014_001`) covers 10 satellite
missions (Saral/AltiKa, CryoSat-2, CFOSAT, HaiYang-2B/2C, Jason-3,
Sentinel-3A/3B, Sentinel-6A, SWOT nadir) at 1 Hz (~7 km along-track); 6 of
these missions (AltiKa, CryoSat-2, Jason-3, Sentinel-3A/3B, Sentinel-6A)
additionally provide 5 Hz (~1.4 km) data from March 2026. Wind recipes use
1 Hz `WIND_SPEED` only; wave recipes use `VAVH`/`VAVH_UNFILTERED` at both
frequencies. Coverage start dates are per-mission (see `AVAILABILITY_START`
in `altimeter_downloader.py`) and reflect the ARCO/zarr store this
downloader queries via `copernicusmarine.subset()`; Copernicus Marine also
holds a separate native-file archive with most missions' raw data back to
2021, but that's a different service this downloader doesn't fetch from.

Significant wave height recipes covering dates on or before 2023-12-31
automatically use the reprocessed (multi-year) altimeter product instead
(`WAVE_GLO_PHY_SWH_L3_MY_014_005`), which covers 12 missions, including
ERS-1/2, TOPEX/Poseidon, Jason-1/2 and Envisat, which the near-real-time
product does not carry, back to 1991-08-03. This switch is automatic and
needs no recipe change: a recipe's requested date range determines which
product is fetched, and a range spanning the 2023-12-31/2024-01-01
boundary uses both. Unlike the near-real-time product, the reprocessed
product is delivered as one combined NetCDF file per day covering every active
mission together, rather than one NetCDF file per satellite, downloaded
via `copernicusmarine.get()` against a different Copernicus Marine
service (`original-files`) than the near-real-time downloader's
`subset()` calls; the downloader checks each mission's predicted orbit
before downloading a day's file, so a day with no mission crossing the
requested area is never fetched. The
reprocessed product's own bias-corrected, denoised significant wave
height (`swh_denoised`) and its uncertainty (`swh_uncertainty`) are
renamed to `VAVH`/`VAVH_UNCERTAINTY` to match the near-real-time
product's codes; the product's own quality filtering happens before the
data can be downloaded at all (see `docs/QC_flags_implementation.md`), so
no separate quality flag needs to be read here.

### Collocation types

| Type | Example |
|------|---------|
| Point vs. Point | mooring / buoy / ferrybox / drifter / tidal gauge vs. SAR WV-mode vignettes |
| Point vs. Layer | mooring / buoy / ferrybox / drifter / tidal gauge vs. SAR |
| Layer vs. Layer | scatterometer / altimeter / radiometer (flattened) vs. SAR |
| Model vs. Layer | ERA5 / HYCOM (gridded) vs. SAR |

---

## Credentials

### Copernicus Dataspace (CDSE) — for SAR downloads

```bash
mkdir -p ~/.config/cdse
printf 'username=your@email.com\npassword=yourpassword' | base64 > ~/.config/cdse/credentials
chmod 600 ~/.config/cdse/credentials
```

Or use environment variables: `COPERNICUS_USERNAME` / `COPERNICUS_PASSWORD`.

Register at: https://dataspace.copernicus.eu

### Copernicus Marine — for in-situ and HF radar downloads

```bash
copernicusmarine login
```

### EUMETSAT EUMDAC — for ASCAT scatterometer downloads

Store credentials in your OS keyring (GNOME Keyring/libsecret on Linux):

```bash
sar-validate --set-credential eumdac
```

You'll be prompted for a username and password; nothing is written to disk
in plaintext. Or use environment variables: `EUMDAC_USERNAME` /
`EUMDAC_PASSWORD`.

If you have an existing `~/.eumdac/credentials` file from before this
change, it's picked up automatically and migrated into the OS keyring the
first time it's needed — a one-time console notice confirms the migration
and tells you it's then safe to delete the old file yourself (it is never
deleted automatically).

Register at: https://eoportal.eumetsat.int

### EUMETSAT OSI-SAF wind FTP — for OSI-SAF wind downloads

Store credentials in your OS keyring:

```bash
sar-validate --set-credential osi_saf
```

Or use environment variables: `OSI_SAF_FTP_USERNAME` / `OSI_SAF_FTP_PASSWORD`.

An existing `~/.eumetsat_osi_saf_wind_credentials` file is migrated into
the OS keyring automatically on first use (with a one-time console notice);
you can delete the old file afterwards.

Register at: https://osi-saf.eumetsat.int/register

### H-SAF FTP — for ASCAT soil moisture NRT downloads

Store credentials in your OS keyring:

```bash
sar-validate --set-credential hsaf
```

Or use environment variables: `HSAF_FTP_USERNAME` / `HSAF_FTP_PASSWORD`.

Register at: https://hsaf.meteoam.it/User/Register

### Space-Track.org — for the ASCAT orbit-based download pre-filter

Store credentials in your OS keyring:

```bash
sar-validate --set-credential space_track
```

Or use environment variables: `SPACE_TRACK_USERNAME` / `SPACE_TRACK_PASSWORD`.

Used to look up historical TLEs (satellite orbital elements) so H-SAF
ASCAT downloads can skip files whose orbit never passed near the
recipe's bbox, before downloading them — see `orbit_coverage.py`. Free
registration at: https://www.space-track.org/auth/createAccount

### JAXA G-Portal — for AMSR2 soil moisture downloads (SFTP fallback)

Store credentials in your OS keyring:

```bash
sar-validate --set-credential gportal
```

Or use environment variables: `GPORTAL_USERNAME` / `GPORTAL_PASSWORD`. If
none of these resolve, the downloader falls back to an interactive
terminal prompt (deliberately not persisted anywhere).

An existing `~/.jaxa_gportal_credentials` file is migrated into the OS
keyring automatically on first use (with a one-time console notice); you
can delete the old file afterwards.

Register at: https://gportal.jaxa.jp

### Earthdata (AMSR-E/2, SMAP, NISAR) — for satellite/SAR soil moisture downloads

```bash
sar-validate --set-credential earthdata
```

This prompts for your NASA Earthdata Login username/password and stores
them in the OS keyring. Credential resolution priority is: explicit
arguments, then environment variables (`EARTHDATA_USERNAME` /
`EARTHDATA_PASSWORD`), then the OS keyring, then an existing `~/.netrc`
`urs.earthdata.nasa.gov` entry — the `~/.netrc` entry is not an ongoing
override; it is only read as a one-time migration source when the
keyring has nothing stored, and is then copied into the keyring
automatically (with a one-time console notice) so subsequent runs use
the keyring directly.

Register at: https://urs.earthdata.nasa.gov

`earthdata_soil_moisture_downloader` uses these credentials to download
AMSR-E/2 and SMAP soil moisture products, and NISAR SME2 soil moisture
(as a SAR source), from the NASA Earthdata archive.

### ESA SMOS FTPS — for SMOS soil moisture downloads

Store credentials in your OS keyring:

```bash
sar-validate --set-credential smos
```

Or use environment variables: `SMOS_FTP_USERNAME` / `SMOS_FTP_PASSWORD`.

An existing `~/.esa_smos_credentials` file is migrated into the OS keyring
automatically on first use (with a one-time console notice); you can
delete the old file afterwards.

Register at: https://eoiam-idp.eo.esa.int/

The `smos_downloader` uses these credentials to download SMOS L2 soil moisture products
from the ESA SMOS FTPS archive.

### Copernicus CDS — for C3S satellite soil moisture downloads and the ERA5 model

`cds_soil_moisture_downloader` downloads via the
[`cdsapi`](https://cds.climate.copernicus.eu/how-to-api) library, which reads
credentials automatically from `~/.cdsapirc`. Create that file after registering:

```ini
url: https://cds.climate.copernicus.eu/api
key: <your-personal-access-token>
```

Register and generate a token at: https://cds.climate.copernicus.eu

Note: no `sar-validate --set-credential` command is needed; `cdsapi` reads
`~/.cdsapirc` natively.

### ECMWF MARS — for GTS moored/drifting buoy downloads

`gts_buoy_downloader` downloads via the
[`ecmwf-api-client`](https://github.com/ecmwf/ecmwf-api-client) library,
which reads credentials automatically from `~/.ecmwfapirc`. Create that
file after registering:

```json
{
    "url"   : "https://api.ecmwf.int/v1",
    "key"   : "<your-api-key>",
    "email" : "<your-email-address>"
}
```

Register and retrieve an API key at: https://api.ecmwf.int/v1/key/

Note: no `sar-validate --set-credential` command is needed;
`ecmwf-api-client` reads `~/.ecmwfapirc` natively.

### ISMN — for soil moisture in-situ validation

ISMN has no download API. Register at https://ismn.earth/en/dataviewer/,
filter by bounding box / date range / depth / variable ("soil moisture") on
the portal, and download the resulting zip. Running the recipe before the
archive exists prints these exact filter values so they can be
copy-pasted into the portal form, along with the recommended download
options: **CEOP-formatted** (variables stored in separate files, zipped),
**"Good" quality flags only**, and **gap filling disabled**, plus the
exact folder to drop the downloaded zip into — no recipe edits needed,
just drop it in and re-run. To reuse one archive across multiple recipes
instead, set its path explicitly as `ismn_archive_path` in the recipe's
`download_kwargs` for the `ismn` validation source.

---

## Running Tests and Checks

```bash
pytest tests/ -v
# With coverage:
pytest tests/ --cov=sar_validation --cov-report=term-missing
```

Linting and type checking are both required to pass before merging:

```bash
ruff check .
python -m mypy -p sar_validation
```

---

## Documentation

- [Design choices](docs/design-choices.md) — toolbox overview and the rationale behind conventions such as wind-direction rotation, collocation aggregation windows and tolerances, WV footprint handling, circular statistics, datatree filtering and CF metadata
