Metadata-Version: 2.4
Name: cng-datasets
Version: 0.3.1
Summary: Cloud-native geospatial dataset processing toolkit
Author-email: Boettiger Lab <cboettig@berkeley.edu>
License: Apache-2.0
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: duckdb==1.5.4
Requires-Dist: ibis-framework[duckdb]>=9.0.0
Requires-Dist: polars>=0.20.0
Requires-Dist: pyarrow>=15.0.0
Requires-Dist: geopandas>=0.14.0
Requires-Dist: rasterio>=1.3.0
Requires-Dist: numpy<2.0,>=1.20.0
Requires-Dist: kubernetes>=28.0.0
Requires-Dist: boto3>=1.34.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=7.4.0; extra == "dev"
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
Requires-Dist: pytest-timeout>=2.2.0; extra == "dev"
Requires-Dist: pytest-mock>=3.12.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.7.0; extra == "dev"
Provides-Extra: raster
Requires-Dist: exactextract>=0.2.0; extra == "raster"
Requires-Dist: rasterio>=1.3.0; extra == "raster"
Provides-Extra: docs
Requires-Dist: sphinx>=7.0; extra == "docs"
Requires-Dist: furo; extra == "docs"
Requires-Dist: myst-parser; extra == "docs"
Requires-Dist: sphinx-autobuild; extra == "docs"
Requires-Dist: linkify-it-py; extra == "docs"
Dynamic: license-file

# cng-datasets

[![DOI](https://zenodo.org/badge/921944685.svg)](https://zenodo.org/badge/latestdoi/921944685)

A CLI toolkit for processing large geospatial datasets into cloud-native formats on Kubernetes.

**What it does:** Takes source geospatial data (Shapefiles, GeoPackages, FileGDB, GeoTIFFs) and produces:

- **GeoParquet** — columnar format for analytical queries (DuckDB, Polars)
- **PMTiles** — vector tiles for web map visualization
- **H3 Hex Parquet** — hexagonal grid indexed at configurable H3 resolution, hive-partitioned for fast spatial joins
- **Cloud-Optimized GeoTIFF** — for raster data, optimized for HTTP range requests

**How it works:** You run a single CLI command locally. It generates Kubernetes Job YAML files that orchestrate the entire pipeline on your cluster. You never process data on your local machine — the CLI just generates the jobs.

## Installation

```bash
pip install cng-datasets
```

Or from source:

```bash
pip install -e "."
```

## Usage

### Generate a vector processing pipeline

```bash
cng-datasets workflow \
  --dataset cpad-2024 \
  --source-url https://example.com/cpad.gdb \
  --bucket public-cpad \
  --layer CPAD_SuperUnits \
  --h3-resolution 10 \
  --parent-resolutions "9,8,0" \
  --hex-memory 32Gi \
  --max-completions 200 \
  --max-parallelism 50
```

This generates YAML files for a 5-step pipeline:

1. **setup-bucket** — creates the S3 bucket with public-read policy
2. **convert** — reads source data, reprojects to EPSG:4326, writes GeoParquet
3. **pmtiles** — converts GeoParquet to PMTiles (parallel with step 4)
4. **hex** — computes H3 cell assignments in parallel pods
5. **repartition** — consolidates hex chunks into hive-partitioned layout

**Multiple sources:** Add `--source-url` multiple times to merge datasets:

```bash
cng-datasets workflow \
  --dataset merged-regions \
  --source-url https://example.com/region1.shp \
  --source-url https://example.com/region2.shp \
  --source-url https://example.com/region3.shp \
  --bucket my-bucket
```

Apply them:

```bash
kubectl apply -f <output-dir>/workflow-rbac.yaml
kubectl apply -f <output-dir>/configmap.yaml
kubectl apply -f <output-dir>/workflow.yaml
```

The workflow orchestrator runs steps sequentially, launching pmtiles and hex in parallel.

### Generate a raster processing pipeline

```bash
cng-datasets raster-workflow \
  --dataset wetlands-cog \
  --source-url https://example.com/wetlands.tif \
  --bucket public-wetlands
```

### Multi-layer sources

For GeoDatabase or GeoPackage files with multiple layers, run one workflow per layer. Use `--layer` to select each layer and `--dataset` with `/` for hierarchical S3 paths:

```bash
# Inspect layers
ogrinfo /vsicurl/https://example.com/data.gdb

# Process each layer separately
cng-datasets workflow --dataset mydata/layer-a --layer LayerA ...
cng-datasets workflow --dataset mydata/layer-b --layer LayerB ...
```

The `/` in `--dataset` creates nested S3 paths (e.g., `mydata/layer-a.parquet`) while using hyphens for k8s resource names.

## CLI Reference

| Command | Purpose |
|---------|---------|
| `cng-datasets workflow` | Generate vector processing k8s pipeline |
| `cng-datasets raster-workflow` | Generate raster processing k8s pipeline |
| `cng-datasets storage setup-bucket` | Create and configure an S3 bucket |
| `cng-convert-to-parquet` | Convert vector data to GeoParquet |
| `cng-datasets vector` | Run H3 hex tiling (used inside k8s pods) |
| `cng-datasets raster` | Run raster H3 tiling (used inside k8s pods) |
| `cng-datasets repartition` | Consolidate hex chunks (used inside k8s pods) |

Commands marked "used inside k8s pods" are called by the generated jobs — you don't run them directly.

### `cng-datasets workflow` options

```
--dataset NAME             Dataset name for S3 paths. Use / for hierarchy (e.g., "padus/fee").
--source-url URL           Public URL to source data.
--bucket BUCKET            Target S3 bucket name.
--layer LAYER              Layer name for multi-layer sources (GDB, GPKG).
--h3-resolution N          H3 resolution for hex tiling (default: 10).
--parent-resolutions STR   Comma-separated parent resolutions (default: "9,8,0").
--hex-memory SIZE          Memory per hex pod (default: 8Gi).
--max-completions N        Number of parallel chunks, max 200 (default: auto).
--max-parallelism N        Max concurrent pods (default: 50).
--id-column COL            ID column name (auto-detected if omitted).
--output-dir DIR           Directory for generated YAML files.
--intermediate-chunk-size  Rows per unnest batch (decrease if OOM).
--row-group-size N         Rows per parquet row group (default: 100000).
```

## S3 Output Layout

```
bucket/
├── dataset.parquet              # GeoParquet
├── dataset.pmtiles              # PMTiles
├── dataset/
│   └── hex/
│       └── h0={cell}/data_0.parquet   # H3-indexed, hive-partitioned
├── README.md
└── stac-collection.json
```

## Docker

The CLI and all dependencies are packaged in a Docker image used by the k8s jobs:

```bash
docker pull ghcr.io/boettiger-lab/datasets:latest
```

## Troubleshooting

**OOM on hex jobs:** Increase `--hex-memory` (e.g., 32Gi → 64Gi), increase `--max-completions` for smaller chunks, or decrease `--intermediate-chunk-size`.

**Convert fails on curved geometries (MULTISURFACE):** Handled automatically — the converter linearizes curved geometry types via ogr2ogr before processing.

**Monitoring:**

```bash
kubectl get jobs              # Pipeline status
kubectl logs job/<name>       # Job logs
kubectl get pods | grep OOM   # Check for memory issues
```

## Development

```bash
pip install -e ".[dev]"
pytest tests/
```

## License

Apache 2.0

## Citation

If you use cng-datasets in your work, please cite it via its Zenodo archive.
The DOI badge at the top of this README resolves to the latest release; each
release also has its own version-specific DOI. Citation metadata (authors,
title, funding) lives in [`.zenodo.json`](.zenodo.json) and is minted
automatically by Zenodo on each GitHub release.

> Boettiger, C., & Buhler, C. K. cng-datasets: A cloud-native toolkit for
> processing large geospatial datasets on Kubernetes. Zenodo.
> https://zenodo.org/badge/latestdoi/921944685

Once the first release is archived on Zenodo, replace the redirect above with
the concrete versioned DOI it mints.
