Metadata-Version: 2.4
Name: defair
Version: 0.4.3
Summary: Destination Earth Framework for preparing AI-Ready data
Keywords: eumetsat,defair,destine,destination earth,machine learning,earth observation
Author: Dusan Figala, Erwan Fassot
Author-email: Dusan Figala <dusan@oxidian.com>, Erwan Fassot <erwan@oxidian.com>
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: GIS
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Dist: defair-ops==0.4.3
Requires-Dist: defair-data==0.4.3
Requires-Dist: typer>=0.19.2
Requires-Dist: loguru>=0.7.3
Requires-Dist: h5netcdf>=1.6.4
Requires-Dist: pydantic>=2.12.3
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: psutil>=7.1.1
Requires-Dist: bokeh>=3.8.2,<3.10
Requires-Dist: urllib3>=2.6.3
Requires-Dist: aiohttp>=3.13.3
Requires-Dist: dask-gateway>=2023.1.1
Requires-Dist: satpy>=0.60 ; python_full_version >= '3.11' and extra == 'benchmark'
Requires-Dist: pyresample>=1.28 ; extra == 'benchmark'
Requires-Dist: matplotlib>=3.10,<3.11 ; extra == 'benchmark'
Requires-Dist: python-geotiepoints>=1.7.5 ; extra == 'benchmark'
Requires-Dist: cartopy>=0.25.0 ; extra == 'notebooks'
Requires-Dist: earthkit-data>=0.17.0,<1 ; extra == 'notebooks'
Requires-Dist: earthkit-utils>=0.1.1,<0.2 ; extra == 'notebooks'
Requires-Dist: healpy>=1.19.0 ; extra == 'notebooks'
Requires-Dist: ipykernel>=6.29 ; extra == 'notebooks'
Requires-Dist: matplotlib>=3.10.7,<3.11 ; extra == 'notebooks'
Requires-Dist: satpy>=0.60 ; python_full_version >= '3.11' and extra == 'notebooks'
Requires-Dist: cartopy>=0.25.0 ; extra == 'visualization'
Requires-Python: >=3.10
Provides-Extra: benchmark
Provides-Extra: notebooks
Provides-Extra: visualization
Description-Content-Type: text/markdown

# DEFAIR

DEFAIR (Destination Earth Framework for preparing AI-Ready data) reads,
transforms, aligns, and writes meteorological and Earth-observation data as
Dask-backed xarray datasets. Its plugin system supports EUMETSAT product
readers, local and S3-compatible storage, geospatial transformations, and
NetCDF/Zarr output.

## Installation

DEFAIR requires Python 3.10 or later. Install it from PyPI into a virtual
environment, with the `notebooks` extra when you want to run the tutorials:

```bash
python -m venv .venv
source .venv/bin/activate
pip install "defair[notebooks]"
```

[Install DEFAIR](docs/installation.md) covers the requirements, the AI
tutorial package, pinning a version and the object store credentials.

For development from a source checkout, install the uv-managed workspace:

```bash
make install
```

DEFAIR releases frequently. Pin the exact version in reproducible pipelines
and check the release notes, which list the resolved ticket numbers, before
upgrading.

## First dataset

`Dataset.from_source()` accepts a local path, an S3 URI, a glob, or an explicit
list of paths. The reader and source plugins are auto-detected when possible.

```python
from defair_data import Dataset

dataset = Dataset.from_source(
    "/data/MSG3-SEVI-MSG15-product.nat",
    reader="msg15nat",
    channels=["IR_108"],
    calibration="brightness_temperature",
    use_channel_names=True,
)

print(dataset.data)
dataset.to_file("output/msg.zarr", writer="zarrv2")
```

Reader results are Dask-backed by default.

## S3 source selection

For an `s3://` URI, omitting `source` auto-selects `source="s3"`. This is the
s3fs/fsspec-backed plugin and is the normal choice for xarray and Dask
workflows. `source="s3boto"` selects the direct boto3 plugin for source-level
operations such as range-based file access, listing, and globbing.

```python
dataset = Dataset.from_source(
    "s3://bucket/path/product.nc",
    source="s3",  # optional: this is the auto-detected S3 default
    source_kwargs={
        "endpoint_url": "https://s3.example.com",
        "aws_access_key_id": "...",
        "aws_secret_access_key": "...",
    },
    reader="mtg_fci_l1c_nc",
)
```

Both plugins pass the same read settings to xarray. `s3boto` changes how DEFAIR
lists files and performs direct reads, but xarray readers may still use s3fs.
See [Data loading, S3, and streaming](docs/data-loading/read-from-s3.md) for the differences,
credential options, and performance settings.

## Large multi-file datasets

Normal multi-file loading builds one `Dataset`, while reading array data only
when it is needed. For a large collection of files, use `streaming=True`.
DEFAIR processes a fixed number of scene groups at a time and appends each
group to Zarr:

```python
cube = Dataset.from_source(
    "s3://bucket/archive/*.nc",
    reader="mtg_fci_l1c_nc",
    use_channel_names=True,
    streaming=True,
    stream_batch_size=4,
)

cube = cube.transform("content_filter", include_vars=["vis_06", "ir_105"])
cube.to_file("output/archive.zarr", writer="zarrv2")
```

A streaming plan does not provide `.data`. It records supported transformations
and writes only to Zarr. Reload the Zarr to inspect the result. Use
`streaming=False` when you need the complete dataset or an operation that
combines all files.

## Discovering capabilities and reader options

List registered plugins and inspect the selected reader's constructor and
`read()` signatures. Options from both signatures can be passed through
`Dataset.from_source()`. IDE completion on `from_source()` shows only its
common parameters because reader-specific options travel through `**kwargs`:

```python
from inspect import signature

from defair.plugin_manager import list_readers, list_sources, load_reader

print(list_sources())
print(list_readers())

reader = load_reader("msg15nat")
print(signature(type(reader)))  # calibration, chunks, use_channel_names, ...
print(signature(reader.read))   # channels, lazy_coordinates, ...
```

To resolve a reader short name to its product name and EUMETSAT collection ID, use
`describe_readers()` (or `defair readers` on the command line):

```python
from defair_data.readers import describe_readers, get_reader_info

print(describe_readers())
print(get_reader_info("mtg_fci_l2_amv"))  # EO:EUM:DAT:0676
```

The full table is in `docs/reader-catalogue.md`.

Reader arguments are validated against the reader's declared parameters, so an
unrecognised option raises and lists the accepted arguments rather than being
ignored. This applies to third-party reader plugins too: a `**kwargs` catch-all
on `read()` does not widen what a reader accepts, so declare every supported
option by name.

## Documentation

- [Getting started](docs/what-is-defair.md): what DEFAIR is, installation, a first product and a first workflow
- [Data loading, S3, streaming, and reader options](docs/data-loading/data-loading.md)
- [Workflow examples](docs/workflow-examples/workflow-examples.md)
- [Dask cluster recommendations](docs/dask-cluster-recommendations.md)
- [Performance benchmarks](docs/benchmarks.md)
- [Tutorial notebooks](docs/demos/), and the per-product coverage set under
  [`tests/notebooks/`](tests/notebooks/)
- [Collocation AI use case](packages/defair-ai-demo/msg_mtg_era5_collocation_ai_usecase.ipynb),
  the notebook that puts MSG, MTG and ERA5 on one grid and trains a cloud-mask model on it
  (the ERA5 leg is read with earthkit-data, which is how a format DEFAIR has no reader for
  reaches DEFAIR)
- [Reader](docs/reader-plugins.md), [writer](docs/writer-plugins/writer-plugins.md), and
  [transformation](docs/transformation-plugins.md) plugin development

The command-line entry point also lists the installed commands and options:

```bash
defair --help
```

## Logging

Set logging with CLI flags or environment variables:

```bash
defair --log-level DEBUG version
defair --log-json version
defair --log-module-level "defair_data:DEBUG,defair_ops:WARNING" version

export DEFAIR_LOG_LEVEL=INFO
export DEFAIR_LOG_JSON=true
```

Python callers can configure logging directly:

```python
from defair.logging import setup_logging

setup_logging(log_level="INFO", json_output=False)
```

## Development

The uv workspace contains three publishable packages:

- `defair` (`src/defair/`): CLI and workflow orchestration
- `defair-data` (`packages/defair-data/`): data sources, readers, and writers
- `defair-ops` (`packages/defair-ops/`): transformations and Dask management

These are mutually dependent rather than strictly layered: each imports the other two, so each
declares the other two, and installing any one of them pulls in the rest. The coupling exists
because the shared kernel — the entry-point plugin manager, `CFHistoryMixin`, kwargs validation
and logging — lives in `defair` rather than in a leaf package, so the lower-level packages call
back up into it. A new cross-package import therefore has to be declared in that package's
`pyproject.toml`, even though a workspace `uv sync` satisfies it either way.

Common development commands:

```bash
make install
make check
make test
make docs
```

Add a dependency to a workspace package with:

```bash
uv add --package <defair|defair-data|defair-ops> <dependency_name>
```

All Python source files require Apache 2.0 license headers. Run
`make headers-check` to validate them or `make headers-add` to add missing
headers.

## Release maintenance

Build all workspace packages with:

```bash
uv build --all-packages
```

Version commands use uv's workspace support:

```bash
uv version --package <defair|defair-data|defair-ops> --bump <major|minor|patch|rc|stable>
```

For a patch to an older release, merge the fix to `main`, create a release
branch from the target tag, cherry-pick the fix, update package versions, and
tag the patched release.

## Building the documentation

Build the local Material for MkDocs site with:

```bash
make docs
```

To generate the DEDL ReadTheDocs/Sphinx source tree, run:

```bash
uv run python scripts/generate_docs.py path/to/target/directory
```

[![coverage report](https://gitlab.eumetsat.int/defair/defair-core/badges/main/coverage.svg)](https://gitlab.eumetsat.int/defair/defair-core/-/commits/main)
