Metadata-Version: 2.4
Name: marinerg-data
Version: 0.2.0
Summary: Facility-side MARINERG-i data client: inspect, convert, validate, publish, and fetch marine test-facility datasets.
Author-email: Irish Centre for High End Computing <james.grogan@ichec.ie>
License: AGPL-3.0-or-later
Project-URL: Repository, https://git.ichec.ie/marinerg-i/marinerg-data
Project-URL: Homepage, https://git.ichec.ie/marinerg-i/marinerg-data
Keywords: Marine Renewable Energy,Data,EOSC,Zenodo,CKAN,kerchunk
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: jsonschema>=4
Requires-Dist: requests
Requires-Dist: PyYAML
Requires-Dist: pydantic>=2
Provides-Extra: mock
Requires-Dist: numpy; extra == "mock"
Provides-Extra: kerchunk
Requires-Dist: kerchunk>=0.2.6; extra == "kerchunk"
Requires-Dist: h5py>=3.10; extra == "kerchunk"
Requires-Dist: fsspec>=2024.2; extra == "kerchunk"
Requires-Dist: zarr>=3; extra == "kerchunk"
Requires-Dist: xarray>=2024.1; extra == "kerchunk"
Provides-Extra: vre
Requires-Dist: kerchunk>=0.2.6; extra == "vre"
Requires-Dist: h5py>=3.10; extra == "vre"
Requires-Dist: fsspec>=2024.2; extra == "vre"
Requires-Dist: zarr>=3; extra == "vre"
Requires-Dist: xarray>=2024.1; extra == "vre"
Requires-Dist: h5netcdf>=1.3; extra == "vre"
Requires-Dist: intake>=2; extra == "vre"
Requires-Dist: intake-xarray; extra == "vre"
Requires-Dist: matplotlib>=3.8; extra == "vre"
Provides-Extra: types
Requires-Dist: types-requests; extra == "types"
Requires-Dist: types-PyYAML; extra == "types"

# marinerg-data

Facility-side data client for the MARINERG-i e-infrastructure: **inspect →
convert → validate → publish**, plus VRE consumption (search, resolve, fetch,
lazy streaming). Django-free and pip-installable, so it runs both at a test
facility and inside a Blue-Cloud / D4Science VRE Jupyter image.

The invariant that shapes everything: **bulk data never transits the
e-infrastructure.** Facilities publish to their own Zenodo; this tool registers
the record in CKAN; the server only re-validates metadata.

## Install

```sh
pip install marinerg-data                 # core client (inspect/validate/publish)
pip install "marinerg-data[kerchunk]"     # + kerchunk reference sidecars
pip install "marinerg-data[mock]"         # + synthetic scenario generator
pip install "marinerg-data[vre]"          # + complete Blue-Cloud / Jupyter runtime
pip install marinerg-wave-tank            # optional wave-tank converter family
```

The lightweight core client supports Python 3.11 or newer and has no
notebook, NetCDF, or container dependency. `kerchunk`, `mock`, and `vre` are
explicit opt-ins. Windows is a supported installation target for the core
client; the release gate includes a Windows install-and-validate smoke test.

## Commands

| Command | Purpose |
|---|---|
| `inspect <dir>` | Sniff capture formats, report converter families; `--draft-manifest` writes a skeleton |
| `convert <run-dir>` | Run a converter family (`--family wave-tank`, else auto-detect); writes open-format outputs + `<run>.manifest.json`, then validates. With `[kerchunk]`, also emits a `<file>.nc.kerchunk.json` reference sidecar per NetCDF |
| `validate <manifest-or-dir>` | JSON Schema (run-manifest v0.1) + referential integrity + file existence; `--verify-checksums`, `--no-files` |
| `publish <bundle-dir>` | Facility Zenodo record + CKAN registration; `--sandbox`, `--new-version`, `--dry-run` |
| `fetch <doi> --cache <dir>` | Download a published dataset into a content-addressed shared cache (keyed by DOI + optional manifest checksum); first lab member pays the download, the rest hit the local copy. `--from-ckan <name> --ckan-url <url>` resolves the DOI via CKAN; `--sandbox` |
| `catalog --ckan-url <url>` | Generate an [intake](https://intake.readthedocs.io) catalogue from CKAN so MARINERG-i datasets open natively in a VRE (`intake.open_catalog(...)` → `.to_dask()`); prefers kerchunk sidecars (lazy `zarr`) over direct NetCDF |
| `mock --scenario <name>` | Synthetic NetCDF + manifest for fixtures/seeding/demos (`--scenario list`) |

Environment: `ZENODO_TOKEN` (a sandbox token with `--sandbox`), `CKAN_URL`,
`CKAN_TOKEN`.

## Cloud-optimised access (VRE)

With the `[kerchunk]` extra, `convert` emits a kerchunk reference sidecar per
NetCDF. The sidecar lets `xarray` lazily stream individual chunks straight from
Zenodo over HTTP range requests — open a 10 GB campaign, plot five minutes of one
gauge, transfer megabytes not gigabytes. The reference URL is templated (`{{u}}`)
so the sidecar is portable; it is retargeted to the Zenodo file URL at publish.

## Examples

Runnable VRE notebooks in [`examples/`](examples/): lazy kerchunk streaming and
intake-based discovery, each self-contained against synthetic data. See
[`examples/README.md`](examples/README.md).

## Blue-Cloud / D4Science handover

The operator-facing install profile, CKAN/DCAT endpoint contract, and live-demo
release checklist are in [`docs/blue-cloud-handover.md`](docs/blue-cloud-handover.md).
The VRE needs Python 3.11 or newer, outbound HTTPS to the MARINERG-i catalogue,
Zenodo, and PyPI, and persistent shared workspace storage.  No MARINERG-i data
hosting, inbound connectivity, or identity federation is required for the
public-data workflow.

## Develop

```sh
uv sync --group dev
uv run pytest          # Django-free, no network (HTTP is faked)
uv run ruff check src tests
```

Development and CI use Python 3.14 with the uv-native toolchain; the published
client supports Python 3.11 or newer. See `CLAUDE.md` for architecture pointers
and the token-free release flow.

## Licence

Copyright ICHEC. GNU AGPL v3 or later. Exemptions available for MARINERG-i
project partners.
