Metadata-Version: 2.4
Name: collekt
Version: 0.3.0
Summary: A library to facilitate spatio-temporal data collection across arbitrary datasources
Keywords: data source,collection
Author: Thomas Roehr, Ola Formo Kihle, Julien Vitay
Author-email: Thomas Roehr <roehr@simula.no>, Ola Formo Kihle <kihle@simula.no>, Julien Vitay <julien.vitay@eodyn.com>
License-Expression: BSD-3-Clause
License-File: LICENSE
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Development Status :: 3 - Alpha
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Dist: pydantic
Requires-Dist: pydantic-settings
Requires-Dist: pyyaml
Requires-Dist: requests
Requires-Dist: shapely
Requires-Dist: geopy
Requires-Dist: rich
Requires-Dist: tqdm
Requires-Dist: xarray>=2026.4.0
Requires-Dist: scipy>=1.18.0
Requires-Dist: cfgrib>=0.9.15.1 ; sys_platform != 'win32'
Requires-Dist: eccodeslib>=2.47.3.21 ; sys_platform != 'win32'
Requires-Dist: damast>=0.3.7
Requires-Dist: geopandas
Requires-Dist: pyogrio
Requires-Dist: polars
Requires-Dist: copernicusmarine>=2.4.1
Requires-Dist: cdsapi>=0.7.7
Requires-Dist: ecmwf-opendata>=0.3.30
Requires-Dist: gfw-api-python-client>=1.4.0
Requires-Dist: hozint-apiclient
Requires-Dist: python-dotenv>=1.2.2
Maintainer: Thomas Roehr
Maintainer-email: Thomas Roehr <roehr@simula.no>
Requires-Python: >=3.12
Project-URL: homepage, https://ai4copsec.github.io/collekt
Project-URL: documentation, https://ai4copsec.github.io/collekt
Project-URL: repository, https://github.com/ai4copsec/collekt
Description-Content-Type: text/markdown

# collekt

A library to facilitate spatio-temporal data collection across arbitrary
datasources.

A dataset config names the products to fetch, and a request supplies the region,
time window, and metadata. The selected products are fanned out to
pluggable source adapters.
Each adapter fetches source-native files (NetCDF, GRIB2, Parquet, product
archives) and records what it did in a machine-readable manifest. Gridded results
can then be assembled onto a common grid and time axis.

## Installation

Requires Python 3.12+ and [`uv`](https://docs.astral.sh/uv/).

```bash
git clone https://github.com/ai4copsec/collekt.git
cd collekt
uv sync --all-groups  # or: just install
```

All provider clients and the gridded `Assembler` are installed by default, so
every source works out of the box. Clients are imported lazily, so importing
collekt stays fast.

To build the documentation locally, you will also need the `quarto` binary ([quarto.org](https://quarto.org/)):

```bash
just docs
# or :
uv run quartodoc build --config docs/_quarto.yml
quarto render docs
```

We recommend using `just` (<https://github.com/casey/just>) with the make-like commands available in `justfile`.

## Usage

### Python

```python
import collekt

config = collekt.DatasetConfig(
    collekt.CMEMS("cmems_glorys_my", variables=["uo", "vo"], depth=[1.0, 1.1]),
    collekt.CMEMS("cmems_duacs_my", variables=["ugos", "vgos"]),
    collekt.CMEMS("cmems_med_currents_my", variables=["uo", "vo"], depth=[1.0, 1.1]),
)
request = collekt.Request(
    region=collekt.Region.from_bbox((-6.0, 20.0, 35.0, 45.0)),
    start="2023-06-15",
)

fetcher = collekt.Fetcher(request=request, config=config, output_dir="data/collections")
result = fetcher.download()
print(result.manifest_path)

# Assemble gridded sources onto a common grid and time axis.
assembler = collekt.Assembler(result)
dataset = assembler.to_xarray(grid="lowest_resolution", time="lowest_resolution")
```

`Region` can also be built from a point and radius
(`Region.from_point_radius(lat, lon, radius_km)`) or a GeoJSON file
(`Region.from_geojson(path)`).

For longer requests, pass `batch_days=7` to `Fetcher` (or `--batch-days 7` to
`collekt fetch`). This groups compatible retrievals while retaining daily files
for daily sources. The default behavior is unchanged; provider capabilities
determine whether a group reduces network requests. See
[batch downloads](https://ai4copsec.github.io/collekt/batching.html).

### Command line

```bash
collekt config show       # inspect the bundled catalog
collekt doctor            # check the environment and configuration

cat > datasets.yaml <<'YAML'
datasets:
  - provider: cmems
    key: cmems_glorys_my
    variables: [uo, vo]
    depth: [1.0, 1.1]
  - provider: cmems
    key: cmems_duacs_my
    variables: [ugos, vgos]
  - provider: cmems
    key: cmems_med_currents_my
    variables: [uo, vo]
    depth: [1.0, 1.1]
YAML

collekt fetch --bbox -6 20 35 45 --start 2023-06-15 \
  --dataset-config datasets.yaml --output-dir data/collections
```

Use `--dry-run` to plan provider requests without downloading, and `--strict` to
fail if any requested source is skipped.

### Sources and configuration

collekt ships a **curated catalog** of datasets (CMEMS global and regional seas,
ECMWF Open Data, ERA5, NOAA GFS, Skytruth, HOZINT, Global Fishing Watch,
Copernicus Data Space, Geonorge, eOdyn), and a `local` adapter reads an existing
tabular archive on disk. Select concrete
dataset keys with `DatasetConfig` or its YAML form, and put provider parameters
such as variable names and depth ranges directly on each selected dataset. See
[Products](https://ai4copsec.github.io/collekt/products.html) and
[Credentials](https://ai4copsec.github.io/collekt/credentials.html).

## Development

```bash
just lint    # format + lint with ruff
just test    # run the test suite
just docs    # build the Quarto documentation
```

The definition of done for any change: `just lint` and `just test` pass, with a
test for any behavior changed.

## Documentation

Built with [Quarto](https://quarto.org) and
[quartodoc](https://machow.github.io/quartodoc/) from `docs/`; the rendered site
is published to GitHub Pages.

## License

[BSD-3-Clause](LICENSE).

*collekt* does not grant you any rights to the wrapped third party providers - it only simplifies making multiple-queries to the sources you already have the permission to access.

Each wrapped service comes with its own separate "Terms of Use", e.g., [Global Fishing Watch](https://globalfishingwatch.org/terms-of-use/) is for non-commercial use only - others may carry similar restrictions.

## Copyright

Copyright (c) 2023-2026 [Simula Research Laboratory, Oslo, Norway](https://www.simula.no).

## Acknowledgments

Part of the EU project [AI4COPSEC](https://ai4copsec.eu), funded by the Horizon
Europe framework programme under Grant Agreement N. 101190021.
