Metadata-Version: 2.4
Name: marinerg-data
Version: 0.6.0
Summary: Facility-side MARINERG-i data client: inspect, convert, validate, publish, and fetch marine test-facility datasets.
Author-email: Irish Centre for High End Computing <james.grogan@ichec.ie>
License: MIT
Project-URL: Repository, https://git.ichec.ie/marinerg-i/marinerg-data
Project-URL: Homepage, https://git.ichec.ie/marinerg-i/marinerg-data
Keywords: Marine Renewable Energy,Data,EOSC,Zenodo,CKAN,kerchunk
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: jsonschema>=4
Requires-Dist: requests
Requires-Dist: PyYAML
Requires-Dist: pydantic>=2
Requires-Dist: pydantic-settings>=2
Requires-Dist: platformdirs>=4
Requires-Dist: pint>=0.24
Requires-Dist: tzdata
Requires-Dist: marinerg-schema>=0.6.0
Provides-Extra: kerchunk
Requires-Dist: kerchunk>=0.2.6; extra == "kerchunk"
Requires-Dist: h5py>=3.10; extra == "kerchunk"
Requires-Dist: fsspec>=2024.2; extra == "kerchunk"
Requires-Dist: zarr>=3; extra == "kerchunk"
Requires-Dist: xarray>=2024.1; extra == "kerchunk"
Requires-Dist: h5netcdf>=1.3; extra == "kerchunk"
Requires-Dist: numpy>=2.4.6; extra == "kerchunk"
Provides-Extra: vre
Requires-Dist: kerchunk>=0.2.6; extra == "vre"
Requires-Dist: h5py>=3.10; extra == "vre"
Requires-Dist: fsspec>=2024.2; extra == "vre"
Requires-Dist: zarr>=3; extra == "vre"
Requires-Dist: xarray>=2024.1; extra == "vre"
Requires-Dist: h5netcdf>=1.3; extra == "vre"
Requires-Dist: intake>=2; extra == "vre"
Requires-Dist: intake-xarray; extra == "vre"
Requires-Dist: matplotlib>=3.8; extra == "vre"
Requires-Dist: jinja2>=3; extra == "vre"
Requires-Dist: aiohttp>=3.9; extra == "vre"
Requires-Dist: numpy>=2.4.6; extra == "vre"
Provides-Extra: types
Requires-Dist: types-requests; extra == "types"
Requires-Dist: types-PyYAML; extra == "types"
Provides-Extra: convert
Requires-Dist: pandas>=2.2; extra == "convert"
Requires-Dist: numpy>=2; extra == "convert"
Requires-Dist: xarray>=2024.1; extra == "convert"
Requires-Dist: h5netcdf>=1.3; extra == "convert"
Requires-Dist: h5py>=3.10; extra == "convert"
Requires-Dist: openpyxl>=3.1; extra == "convert"
Requires-Dist: scipy>=1.11; extra == "convert"
Dynamic: license-file

# marinerg-data

marinerg-data is the command-line client for the MARINERG-i data catalogue.
A test facility uses it to turn instrument exports into standard NetCDF, check
them, and publish them. A researcher uses it to find and download what has been
published.

## How it fits together

Data files never pass through MARINERG-i. A facility deposits its files in its
own Zenodo account, which gives the dataset a DOI. This tool then registers the
dataset in the MARINERG-i catalogue, and fills in the record from the files. The
catalogue's own docs explain
[how the catalogue works](https://git.ichec.ie/marinerg-i/ckanext-marinerg/-/blob/main/docs/how-the-catalogue-works.md).

Facilities differ in how organised their data is. The tool works at every level:

| Level | The facility has | It can |
|---|---|---|
| 0 | Files and a test report | Add a catalogue record by hand, with a contact for the data |
| 1 | Run logs and vendor exports | Describe its exports with templates, then convert them |
| 2 | Consistent formats and instrument lists | Convert, validate and publish each campaign |
| 3 | Standard formats such as CF NetCDF | Validate and publish directly |

## Install

```sh
pip install "marinerg-data[convert,kerchunk]"   # to convert and publish
pip install "marinerg-data[vre]"                # to use data in a VRE
```

`[convert]` reads exports and writes NetCDF. `[kerchunk]` adds a small index
beside each NetCDF, so a VRE can stream part of a file from Zenodo. Without
either, the client still validates and publishes files that are already
converted.

## Publish your data

Publishing takes four commands. Each one says what to run next.

1. **Inspect.** `marinerg-data inspect <folder>` says which equipment template
   matches each export. A template describes how one instrument writes its
   files: the delimiter, the decimal separator, the units, and which column is
   time. If nothing matches, write a template first, as
   [Templates](#templates) explains.
2. **Convert.** `marinerg-data convert <folder>` reads each export through its
   template and writes a NetCDF file and a run manifest, which describes the
   run. It stops, naming the column and row, at anything the template does not
   settle, such as a unit the file and the template disagree on.

   For a test campaign with several instruments, keep a folder per instrument
   and convert them together with `--campaign`. Each run then gets one manifest
   over every instrument, with each clock's offset applied. It also drafts
   `dataset.yaml`, the metadata for the catalogue record. Read it and edit it:
   the tool never overwrites it.
3. **Validate.** `marinerg-data validate <folder>` checks the manifests and
   `dataset.yaml` against the vocabularies the catalogue uses, so a mistake is
   caught before anything is uploaded.
4. **Publish.** Set `ZENODO_TOKEN` to your Zenodo token, `CKAN_URL` to the
   catalogue's address and `CKAN_TOKEN` to your catalogue API token. Then run
   `marinerg-data publish <folder> --sandbox` to try it on sandbox.zenodo.org,
   and drop `--sandbox` when it looks right. `--new-version` publishes a
   correction as a new version of the same record.

If a publish fails part-way, run the same command again. It carries on from
where it stopped, and never mints a second DOI for the same data.

## Try it without data

[demo/](demo/README.md) walks the publish path with generated data, so you can
see every step before using your own files.

## Use published data

`marinerg-data fetch <doi-or-name> --cache <folder>` downloads a dataset into a
shared cache, so a lab downloads each file once. A name, rather than a DOI, is
looked up in the catalogue at `CKAN_URL`.

`marinerg-data catalog` writes an [intake](https://intake.readthedocs.io)
catalogue of the published datasets, for opening them in Python.
[examples/](examples/README.md) shows both in Jupyter and in a Virtual Research
Environment.

## Templates

The package ships equipment templates for common instruments;
`marinerg-data template list` shows them. For an instrument without one:

1. `marinerg-data template draft <export>` guesses a template from a sample
   file and writes it beside the file. The guesses are only a start: check
   every value, and replace each `TODO`. The template will not load until you
   have. Each channel names what it measures with a parameter code;
   `marinerg-data template parameters` lists them.
2. `marinerg-data template check <template-file> <export>` reads the export
   through it and reports what it mapped and what it could not.
3. Keep your templates in a folder, and pass it to every command with
   `--templates <folder>`. Templates in your user config folder are loaded too
   (`~/.config/marinerg-data/templates` on Linux).

A campaign also needs a facility template, with the facility's name, ROR and
default licence, and a campaign template, listing the instruments and naming the
operator's log. Keep them in `facilities/` and `campaigns/` inside your
templates folder. The fixtures in
[tests/fixtures/templates](tests/fixtures/templates) are worked examples of all
three kinds.

Give each run's start time in the operator's log, so the NetCDF counts time
from it, as CF requires. Most instruments record only elapsed time. The start
is the moment the reference clock, the one the clock offsets are measured
against, read zero. Take it from the operator's notes or the wavemaker's log.
Name the column in the campaign template's `log`, with `start_time`, and give
`timezone` unless each time carries its own offset. Without a start time,
`convert` warns, and time stays elapsed seconds.

## Develop

```sh
uv sync --group dev
uv run tox
```

`uv run tox` runs the format check, lint, type check and tests, as CI does.

`marinerg-data dev` generates test and demo data. It is left out of `--help`;
`marinerg-data dev --help` lists what it does.

## Release

Open the [newest pipeline for `main`](https://git.ichec.ie/marinerg-i/marinerg-data/-/pipelines/main/latest)
and press the play button on release patch, minor or major. The release pipeline publishes to
PyPI, then tags the commit and creates the GitLab release.

## Licence

Copyright Irish Centre for High End Computing, 2026. MIT. See [`LICENSE`](LICENSE).
