Metadata-Version: 2.4
Name: defair-ai-demo
Version: 0.4.3
Summary: Cloud masking CNN demo for DEFAIR - standalone AI use case
Author: Dusan Figala
Author-email: Dusan Figala <dusan@oxidian.com>
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: License :: OSI Approved :: Apache Software License
Requires-Dist: defair==0.4.3
Requires-Dist: defair-data==0.4.3
Requires-Dist: xarray>=2024.6.0
Requires-Dist: dask[array]>=2024.1.0
Requires-Dist: zarr>=2.16.0
Requires-Dist: numcodecs>=0.11.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: torch>=2.0.0
Requires-Dist: matplotlib>=3.7.0,<3.11
Requires-Dist: loguru>=0.7.3
Requires-Dist: tqdm>=4.66.0
Requires-Dist: pyyaml>=6.0.0
Requires-Dist: psutil>=7.1.1
Requires-Dist: scikit-learn>=1.7.2
Requires-Dist: earthkit-data>=0.17.0,<1
Requires-Dist: earthkit-utils>=0.1.1,<0.2
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# DEFAIR AI Demo - Cloud Masking CNN

Standalone demonstration of cloud masking using a lightweight CNN on MSG satellite data.

## Overview

This package demonstrates an end-to-end ML pipeline for cloud detection:
1. Load MSG native format data (.nat files) using the MSG15NativeReaderPlugin (We will add more data formats and therefore collocation in the upcoming issues)
2. Generate cloud masks using IR temperature thresholds
3. Convert to Zarr format with ML-optimized chunking
4. Train a simple U-Net CNN for binary cloud/clear classification
5. Generate predictions and visualizations


## Quick Start

**Navigate to the package directory first:**
```bash
cd packages/defair-ai-demo
```

### Option 1: Using Makefile (Recommended)

```bash
# See all available commands
make help

# Install dependencies
make install

# Run end-to-end demo
make demo

# Run tests
make test

# Clean up generated files
make clean
```

### Option 2: Using uv directly

```bash
# Install dependencies
uv sync --package defair-ai-demo

# Run end-to-end demo
uv run python scripts/run_end_to_end.py

# Run tests
uv run pytest tests/
```

## The collocation AI use case notebook

`msg_mtg_era5_collocation_ai_usecase.ipynb`, in this directory, is the fuller demonstration: it
reads MSG SEVIRI and MTG FCI through DEFAIR and the ERA5 GRIB with earthkit-data, reprojects all
three onto one grid over Europe, writes an analysis-ready Zarr, and then trains and runs the
cloud-mask model from this package on it.

From PyPI, install the package together with the plotting stack the notebook uses. On Linux,
run the first line too, so pip takes the CPU build of PyTorch instead of the CUDA build, about
3 GB of downloads:

```bash
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install defair-ai-demo "defair[notebooks]"
```

From a source checkout, the same environment comes from the repository root with:

```bash
uv sync --frozen --all-packages --group notebooks
```

`--all-packages` provides this package and PyTorch, `--group notebooks` provides cartopy for the
plotting helpers in `defair.notebook_helpers`.

The notebook reads the public demo bucket of the DestinE Data Lake by default and needs no
credentials, as described in [Install DEFAIR](../../docs/installation.md). Two
variables are specific to this notebook. `COLLOCATION_SOURCE` is `s3` (the default) or `local`, and
`local` reads staged data from `test-data/collocation/` instead of the bucket. `COLLOCATION_BASE` is
the folder that local input paths are relative to, and defaults to the notebook folder.

## Configuration

All hyperparameters and settings are in `config.yaml`:
- **Data settings**: MSG file path, channels to use
- **Training settings**: Epochs, batch size, learning rate
- **Model settings**: Input/output channels
- **Paths**: Output directories and file locations

**Important:** Update `data.msg_dir` in `config.yaml` to point to your directory of MSG .nat files

To use a custom config:
```bash
make run --config=my_config.yaml
# Or with uv:
uv run python scripts/run_end_to_end.py --config my_config.yaml
```

## Available Makefile Commands

| Command | Description |
|---------|-------------|
| `make help` | Show all available commands |
| `make install` | Install package dependencies |
| `make demo` | Run end-to-end demo pipeline |
| `make test` | Run test suite |
| `make format` | Auto-format code with ruff |
| `make check` | Run linting and quality checks |
| `make typecheck` | Run type checking with pyright |
| `make clean` | Remove generated files (__pycache__, output/) |

## Output

All outputs are saved to `output/` (in package directory):
- `msg_processed.zarr/` - Processed satellite data
- `cloud_model.pt` - Trained model weights
- `predictions/` - Cloud mask visualizations

## Architecture

- **Model**: Lightweight U-Net (3 encoder/decoder levels, ~500K parameters)
- **Input**: 3 channels (IR_108, VIS006, IR_039) @ 256x256 patches
- **Output**: Binary cloud mask [0, 1]
- **Labels**: Generated using IR temperature threshold (IR_108 < 270K = clouds)
