Metadata-Version: 2.4
Name: gedih3
Version: 0.14.4
Summary: Access, index, and analyze NASA GEDI LiDAR data with H3 hexagonal spatial indexing and Dask-distributed processing
Author-email: Tiago de Conto <tiagodc@umd.edu>, Amelia Grace Holcomb <ameliah@umd.edu>
License-Expression: LicenseRef-UMD-Source-Available-NonCommercial-1.0
Project-URL: Homepage, https://github.com/tiagodc/GEDI-H3
Project-URL: Repository, https://github.com/tiagodc/GEDI-H3
Project-URL: Documentation, https://tiagodc.github.io/GEDI-H3/
Project-URL: Issues, https://github.com/tiagodc/GEDI-H3/issues
Keywords: gedi,lidar,h3,spatial-indexing,remote-sensing,nasa,biomass,forestry,dask,geospatial,parquet
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: GIS
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Operating System :: OS Independent
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: numpy>=1.26.0
Requires-Dist: pandas>=2.1.1
Requires-Dist: geopandas>=1.0.0
Requires-Dist: shapely>=2.0.2
Requires-Dist: pyogrio>=0.7.2
Requires-Dist: pyproj>=3.6.1
Requires-Dist: h3>=4.3.0
Requires-Dist: h3pandas>=0.3.0
Requires-Dist: antimeridian>=0.3.0
Requires-Dist: fsspec>=2025.2.0
Requires-Dist: s3fs>=2025.2.0
Requires-Dist: aiohttp>=3.9.0
Requires-Dist: requests>=2.26.0
Requires-Dist: dask>=2025.1.0
Requires-Dist: dask-geopandas>=0.5.0
Requires-Dist: distributed>=2025.1.0
Requires-Dist: psutil>=5.9.4
Requires-Dist: numba>=0.59.0
Requires-Dist: pyarrow>=14.0.1
Requires-Dist: h5py>=3.10.0
Requires-Dist: xarray>=2024.1.0
Requires-Dist: rasterio>=1.3.8.post1
Requires-Dist: rioxarray>=0.15.0
Requires-Dist: geocube>=0.7.1
Requires-Dist: affine>=2.3.0
Requires-Dist: python-dateutil>=2.9.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: earthaccess>=0.16.0
Requires-Dist: pqdm>=0.2.0
Requires-Dist: tqdm>=4.60.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: scipy>=1.11.2
Requires-Dist: duckdb<1.5,>=1.4.4
Provides-Extra: docs
Requires-Dist: sphinx>=8.0; extra == "docs"
Requires-Dist: pydata-sphinx-theme>=0.16; extra == "docs"
Requires-Dist: numpydoc>=1.8; extra == "docs"
Requires-Dist: myst-parser>=4.0; extra == "docs"
Requires-Dist: sphinx-autoapi>=3.4; extra == "docs"
Requires-Dist: sphinx-design>=0.6; extra == "docs"
Requires-Dist: sphinxcontrib-mermaid>=1.0; extra == "docs"
Provides-Extra: gdal
Requires-Dist: GDAL>=3.4.0; extra == "gdal"
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Dynamic: license-file

<p align="center">
  <img src="https://raw.githubusercontent.com/tiagodc/GEDI-H3/main/docs/imgs/gh3_logo.png" alt="gedih3 logo" width="400">
</p>

# gedih3

**Turn billions of NASA GEDI LiDAR footprints into analysis-ready spatial datasets — from the command line or Python.**

GEDI (Global Ecosystem Dynamics Investigation) is NASA's premier spaceborne LiDAR mission, measuring forest structure and carbon stocks globally. Its raw data comprises thousands of large HDF5 files organized by satellite orbit — not by geography. Navigating quality flags, extracting variables of interest, and running spatial queries over billions of footprints requires significant engineering effort and domain expertise.

**gedih3** handles all of that. It converts orbit-organized GEDI HDF5 files into a spatially-indexed GeoParquet database built with expert-curated presets and pre-configured quality filtering, then provides a complete toolchain for querying, aggregating, and exporting to GeoTIFF, GeoParquet, and other common formats.

---

## What is GEDI-H3?

gedih3 is built on four components that together make billion-shot GEDI analysis tractable:

| Component | Role |
|-----------|------|
| **[GEDI](https://gedi.umd.edu/)** | NASA ISS-mounted LiDAR measuring forest height, biomass, and canopy structure at ~25 m footprints globally |
| **[H3](https://h3geo.org/)** | Uber's hexagonal spatial indexing system — the primary database index enabling fast regional queries |
| **[Dask](https://dask.org/)** | Distributed Python computing — scales from a laptop to an HPC cluster without changing your code |
| **[earthaccess](https://earthaccess.readthedocs.io/)** | NASA's official library for Earthdata authentication, search, and download |

---

## Why gedih3?

Working with GEDI at scale is genuinely hard:

- **Orbit-organized, not spatially indexed** — files are sorted by acquisition time, not geography. A regional query means scanning thousands of files.
- **Complex HDF5 containers** — deeply nested structure with hundreds of variables per beam, requiring specialized tools to navigate correctly.
- **Quality filtering is non-trivial** — each product has its own flags; best practices require combining multiple criteria correctly to avoid biased results.
- **Variable overload** — L2A alone provides 300+ variables per beam. Choosing the right ones for your analysis requires domain expertise.
- **Scale** — the full dataset runs to terabytes and billions of rows. Without spatial indexing and distributed processing, even simple analyses can take days.

**gedih3** addresses all of this.

---

## Key Features

- Expert-curated `minimal` and `default` variable presets for all GEDI products (L1B, L2A, L2B, L4A, L4C)
- Pre-configured quality filtering applied with a single flag (`-y | --quality`)
- H3 hexagonal spatial indexing (levels 0–15) with partition pruning — only tiles intersecting your region are read
- Incremental database: add new orbits, time periods, or variables without rebuilding; interrupted builds resume automatically
- Memory-efficient builds: streams HDF5 data beam-by-beam, writes partitions independently, merges at row-group level
- Network-efficient downloads: S3 streaming with on-the-fly variable subsetting and automatic resume
- Build once, iterate fast: once built, extract/aggregate/rasterize run in seconds to minutes
- Dask-distributed processing; works on laptops, workstations, and HPC clusters without code changes
- command-line tools covering the full pipeline from download to analysis ready data outputs
- Full Python API with custom aggregation functions (any callable, e.g., per-hexagon regression)
- DuckDB compatible for spatial SQL querying, including larger-than-memory datasets
- Multi-format export: GeoTIFF (with compression, tiling, time-series), GeoParquet, and others
- Ancillary data fusion: sample external rasters and join vector polygons at GEDI footprint level

---

## How it works

gedih3 is designed around a simple principle: **build once, iterate fast**.

1. **Build** (`gh3_build`): Reads raw HDF5 files beam-by-beam via Dask, assigns each shot an H3 spatial index, and writes partitioned GeoParquet — one file per spatial tile per year. A build log tracks every ingested granule. Interrupted builds resume from where they stopped. New data merges into the existing database.

2. **Query** (`gh3_load` / `gh3_extract`): The H3 partition structure resolves which files to read via index arithmetic — no scanning. Only partitions whose H3 cells intersect your region are opened.

3. **Aggregate** (`gh3_aggregate`): Every shot carries an H3 cell ID, so aggregation to any coarser resolution is a local groupby within each partition — no data shuffle across parallel workers.

---

## Quick Start

```bash
# Install with pip — self-contained, no system libraries needed
pip install gedih3
```

```bash
# ...or with conda, recommended for HPC and shared clusters
git clone https://github.com/tiagodc/GEDI-H3.git
cd GEDI-H3
conda env create -f environment.yml
conda activate gedih3
```

> GDAL, GEOS, PROJ and HDF5 all ship prebuilt inside the wheels, so `pip install`
> needs nothing installed system-wide. See the
> [installation guide](https://tiagodc.github.io/GEDI-H3/getting-started/installation.html)
> for platform coverage and the optional GDAL Python bindings.

No configuration needed. All outputs default to `~/gedi_data/`.

```bash
# 1. Download GEDI data for a region (W,S,E,N)
gh3_download -r "-51,0,-50,1" -l2a minimal -l4a minimal

# 2. Build the H3 spatial database
gh3_build -r "-51,0,-50,1" -l2a minimal -l4a minimal
## Optionally build DuckDB metadata table (Ducklake)
gh3_build_ducklake

# 3. See which variables are available
gh3_read_schema

# 4. Extract with quality filtering
gh3_extract -y -l agbd_l4a rh_098_l2a -o extracted/

# 5. Aggregate to ~36 km² hexagons
gh3_aggregate -d extracted/ -h3 6 -a mean -o aggregated/

# 6. Export as GeoTIFF
gh3_rasterize -d aggregated/ -o rasters/ --compress LZW
```

> Run any tool with `--help` for the full list of options and flags:
> ```bash
> gh3_build --help
> gh3_aggregate --help
> ```

---

## CLI Tools

| Tool | Purpose |
|------|---------|
| `gh3_download` | Download GEDI HDF5 data from NASA DAACs |
| `gh3_build` | Build H3-indexed Parquet database from HDF5 files |
| `gh3_build_ducklake` | Build DuckDB metadata database for SQL queries |
| `gh3_extract` | Extract and filter shots to flat Parquet files |
| `gh3_aggregate` | Aggregate to coarser H3 resolution |
| `gh3_rasterize` | Export aggregated data as GeoTIFF |
| `gh3_update` | Merge new variables into an existing dataset |
| `gh3_from_img` | Sample external raster values at GEDI shot locations |
| `gh3_from_polygon` | Join vector polygon attributes to GEDI shots |
| `gh3_list_resolutions` | View H3 and EGI resolution level tables |
| `gh3_read_schema` | Inspect schemas and browse variables from any file or database |

### Common Flags

```
-r, --region       Spatial filter: bbox "W,S,E,N", vector file, or ISO3 country code
-t0, -t1           Temporal filters (YYYY-MM-DD)
-l2a, -l4a, ...    Product variables: 'minimal', 'default', or explicit variable names
-y, --quality      Apply pre-configured quality filters
-q, --query        Pandas query string filter
-N, -T, -M         Dask workers, threads per worker, memory per worker
-v / -vv / -Q      Verbosity: INFO / DEBUG / quiet
```

---

## Python API

All CLI functionality is available from Python — with the added benefit of chaining operations in memory without saving intermediate files to disk.

```python
import gedih3.gh3driver as gh3
from gedih3 import raster

# Load data from the H3 database
ddf = gh3.gh3_load(
    source='~/gedi_data/h3/',
    columns=['agbd_l4a', 'rh_098_l2a'],
    region='-51,0,-50,1',           # bbox, shapefile, or ISO3
    query='quality_flag_l2a == 1',
)

# Aggregate to H3 level 6 (~36 km²) and export as GeoTIFF
agg = gh3.gh3_aggregate(ddf, target_res=6, agg='mean').compute()
raster.export_raster(raster.h3_to_raster(agg), 'agbd_mean.tif', compress='LZW')
```

### Custom Aggregation Functions

The Python API accepts any callable as the aggregation function. Each H3 hexagon's data is passed as a DataFrame, enabling analyses not possible from the CLI:

```python
import numpy as np, pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score

def fit_regression(df):
    """Fit height → biomass regression per hexagon."""
    mask = ~(df['agbd_l4a'].isna() | df['rh_098_l2a'].isna())
    X, y = df.loc[mask, 'rh_098_l2a'].values.reshape(-1, 1), df.loc[mask, 'agbd_l4a'].values
    if len(X) < 2:
        return pd.DataFrame({'r2': [np.nan], 'n': [len(df)]})
    model = LinearRegression().fit(X, y)
    return pd.DataFrame({'r2': [r2_score(y, model.predict(X))], 'n': [len(df)]})

# Per-hexagon regression across all partitions, in parallel
results = gh3.gh3_aggregate(ddf, target_res=6, agg=fit_regression)
results.compute()
```

---

## Spatial Indexing

### H3 Hexagonal Index

H3 is the primary spatial index. The database uses a dual-level H3 structure: a coarse **partition level** (default: level 3, ~12,000 km²) for file organization, and a fine **index level** (default: level 12, ~307 m²) for shot-level precision.

| Level | Avg. Area | Typical Use |
|-------|-----------|-------------|
| 3 | ~12,393 km² | Partition level (database tiles) |
| 6 | ~36 km² | Regional aggregation |
| 9 | ~0.105 km² | Local analysis |
| 12 | ~307 m² | Index level (shot precision) |

> **Note**: H3 parent hexagons are not perfectly geometrically inclusive of their children — a known property of hexagonal grids. `gh3_aggregate` handles this correctly and consistently.

### EGI Square-Pixel Index (Advanced)

For workflows that require alignment with standard raster grids or the GEDI L4B gridded product, gedih3 supports the **EASE Grid Index (EGI)** — square pixels on EASE-Grid 2.0 (EPSG:6933).

| Level | Pixel Size | Typical Use |
|-------|------------|-------------|
| 3 | ~25 m | GEDI footprint level |
| 6 | ~1 km | GEDI L4B baseline |
| 8 | ~10 km | Wall-to-wall analysis |

> **Note**: Lower EGI level = finer resolution (opposite of H3).

```bash
# EGI extraction and aggregation
gh3_extract -d ~/gedi_data/h3/ -egi 6 -o extracted_egi/
gh3_aggregate -d ~/gedi_data/h3/ -egi 6 -a mean -o aggregated_egi/
```

---

## Architecture

```mermaid
---
config:
  theme: dark
  layout: dagre
---
flowchart TB
    n1["☁️ NASA DAAC"] --> n2["🌐 earthaccess"]
    n2 --> n17["⬇️ gh3_download"]
    n17 --> n4["🗂️ HDF5 Files"]
    n4 --> C["⚙️ gh3_build"]
    C --> n3["💽 H3 Database\n(Parquet)"]
    n3 --> D["gh3_extract"] & n10["gh3_read_schema"]
    D --> SHOTS["📂 Flat Dataset"]
    SHOTS --> n6["gh3_aggregate"] & RAST["gh3_rasterize"]
    n6 --> AGG["📊 Aggregated Dataset"]
    AGG --> RAST
    RAST --> TIF["🗺️ GeoTIFF"]
    SHOTS <--> n8["gh3_from_img"] & n9["gh3_from_polygon"]
    n15["External Raster"] -.-> n8
    n16["External Vector"] -.-> n9
    n1@{ shape: db}
    n2@{ shape: com-link}
    n4@{ shape: das}
    n3@{ shape: disk}
    SHOTS@{ shape: das}
    AGG@{ shape: das}
    TIF@{ shape: das}
    n15@{ shape: das}
    n16@{ shape: das}
    n15:::fade
    n16:::fade
    classDef fade stroke:#757575,color:#757575
    linkStyle 13 stroke:#757575,fill:none
    linkStyle 14 stroke:#757575,fill:none
```

---

## GEDI Products Supported

| Product | Description |
|---------|-------------|
| L1B | Geolocated waveforms |
| L2A | Elevation and height metrics (RH percentiles, canopy height) |
| L2B | Canopy cover and vertical structure profiles |
| L4A | Footprint-level aboveground biomass (AGBD) |
| L4C | Footprint-level structural complexity (WSCI) |

For variable details, run `gh3_read_schema` or see [gedi.umd.edu](https://gedi.umd.edu/dataproducts/download/).

---

## Configuration

gedih3 works with zero configuration. All outputs default to `~/gedi_data/`.

To customize storage locations, set environment variables or create `~/.gedih3.env`:

```bash
GH3_DEFAULT_DOWNLOAD_DIR=/data/gedi       # root for all gedih3 files
GH3_DEFAULT_H3_DIR=/data/gedi/h3_db       # H3 database location
GH3_DEFAULT_SOC_DIR=/data/gedi/soc        # downloaded HDF5 files
GH3_DEFAULT_TMP_DIR=/data/gedi/tmp        # temporary storage
```

Configuration priority (highest to lowest): CLI arguments → environment variables → `~/.gedih3.env` → package defaults.

---

## Tutorials

See the `tutorials/` directory:
- `tutorial_cli_pipeline.sh` — End-to-end CLI workflow
- `tutorial_python_api_pipeline.py` — Complete Python API examples
- `tutorial_duckdb_basics.ipynb` -- DuckDB query examples

---

## Requirements

- Python >= 3.12
- NASA Earthdata account (free, required for downloading GEDI data)
- Key dependencies: `dask`, `geopandas`, `h3`, `pyarrow`, `h5py`, `rioxarray`, `earthaccess`
- **Optional dependencies**: duckdb

See `pyproject.toml` for the full dependency list.

---

## Acknowledgements

`gedih3` was developed as a supporting tool in the course of research funded by the National Aeronautics and Space Administration (NASA) contract NNL 15AA03C for the development and execution of the GEDI mission, and NASA FINNEST grant 80NSSC22K1543. This software was not a contracted deliverable of these projects but emerged from work conducted under them.

---

## Copyright and License

Copyright (C) 2026, University of Maryland. All Rights Reserved.

Authors: Tiago de Conto, Amelia Grace Holcomb (University of Maryland).

`gedih3` is **source available, not open source**. It is distributed under the UMD Source Available Non-Commercial End User License Agreement — see the [LICENSE](https://github.com/tiagodc/GEDI-H3/blob/main/LICENSE) and [NOTICE](https://github.com/tiagodc/GEDI-H3/blob/main/NOTICE) files for the full terms. In summary:

- **Free for non-commercial use** — research, education, and other non-revenue-generating purposes.
- **Redistribution of unmodified copies is permitted**, including through public package repositories and their mirrors, provided the LICENSE and all required notices travel with every copy.
- **Modifications must be contributed back** — fork the repository and open a pull request. Distributing modified copies as a separate release is not permitted.
- **Commercial use requires a separate license** from the University of Maryland.

For commercial licensing inquiries, please contact the University of Maryland's UM Ventures office at [umdtechtransfer@umd.edu](mailto:umdtechtransfer@umd.edu).
