Metadata-Version: 2.4
Name: marinerg-data
Version: 0.1.0
Summary: Facility-side MARINERG-i data client: inspect, convert, validate, publish, and fetch marine test-facility datasets.
Author-email: Irish Centre for High End Computing <james.grogan@ichec.ie>
License: AGPL-3.0-or-later
Project-URL: Repository, https://git.ichec.ie/marinerg-i/marinerg-data
Project-URL: Homepage, https://git.ichec.ie/marinerg-i/marinerg-data
Keywords: Marine Renewable Energy,Data,EOSC,Zenodo,CKAN,kerchunk
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Requires-Python: >=3.14
Description-Content-Type: text/markdown
Requires-Dist: jsonschema>=4
Requires-Dist: requests
Requires-Dist: PyYAML
Requires-Dist: pydantic>=2
Provides-Extra: mock
Requires-Dist: numpy; extra == "mock"
Provides-Extra: kerchunk
Requires-Dist: kerchunk>=0.2.6; extra == "kerchunk"
Requires-Dist: h5py>=3.10; extra == "kerchunk"
Requires-Dist: fsspec>=2024.2; extra == "kerchunk"
Requires-Dist: zarr>=3; extra == "kerchunk"
Requires-Dist: xarray>=2024.1; extra == "kerchunk"
Provides-Extra: types
Requires-Dist: types-requests; extra == "types"
Requires-Dist: types-PyYAML; extra == "types"

# marinerg-data

Facility-side data client for the MARINERG-i e-infrastructure: **inspect →
convert → validate → publish**, plus VRE consumption (search, resolve, fetch,
lazy streaming). Django-free and pip-installable, so it runs both at a test
facility and inside a Blue-Cloud / D4Science VRE Jupyter image.

The invariant that shapes everything: **bulk data never transits the
e-infrastructure.** Facilities publish to their own Zenodo; this tool registers
the record in CKAN; the server only re-validates metadata.

## Install

```sh
pip install marinerg-data                 # core client (inspect/validate/publish)
pip install "marinerg-data[kerchunk]"     # + kerchunk reference sidecars
pip install "marinerg-data[mock]"         # + synthetic scenario generator
pip install marinerg-wave-tank            # optional wave-tank converter family
```

## Commands

| Command | Purpose |
|---|---|
| `inspect <dir>` | Sniff capture formats, report converter families; `--draft-manifest` writes a skeleton |
| `convert <run-dir>` | Run a converter family (`--family wave-tank`, else auto-detect); writes open-format outputs + `<run>.manifest.json`, then validates. With `[kerchunk]`, also emits a `<file>.nc.kerchunk.json` reference sidecar per NetCDF |
| `validate <manifest-or-dir>` | JSON Schema (run-manifest v0.1) + referential integrity + file existence; `--verify-checksums`, `--no-files` |
| `publish <bundle-dir>` | Facility Zenodo record + CKAN registration; `--sandbox`, `--new-version`, `--dry-run` |
| `mock --scenario <name>` | Synthetic NetCDF + manifest for fixtures/seeding/demos (`--scenario list`) |

Environment: `ZENODO_TOKEN` (a sandbox token with `--sandbox`), `CKAN_URL`,
`CKAN_TOKEN`.

## Cloud-optimised access (VRE)

With the `[kerchunk]` extra, `convert` emits a kerchunk reference sidecar per
NetCDF. The sidecar lets `xarray` lazily stream individual chunks straight from
Zenodo over HTTP range requests — open a 10 GB campaign, plot five minutes of one
gauge, transfer megabytes not gigabytes. The reference URL is templated (`{{u}}`)
so the sidecar is portable; it is retargeted to the Zenodo file URL at publish.

## Develop

```sh
uv sync --group dev
uv run pytest          # Django-free, no network (HTTP is faked)
uv run ruff check src tests
```

Python 3.14, uv-native toolchain. See `CLAUDE.md` for architecture pointers and
the token-free release flow.

## Licence

Copyright ICHEC. GNU AGPL v3 or later. Exemptions available for MARINERG-i
project partners.
