Metadata-Version: 2.4
Name: marinerg-data
Version: 0.5.0
Summary: Facility-side MARINERG-i data client: inspect, convert, validate, publish, and fetch marine test-facility datasets.
Author-email: Irish Centre for High End Computing <james.grogan@ichec.ie>
License: MIT
Project-URL: Repository, https://git.ichec.ie/marinerg-i/marinerg-data
Project-URL: Homepage, https://git.ichec.ie/marinerg-i/marinerg-data
Keywords: Marine Renewable Energy,Data,EOSC,Zenodo,CKAN,kerchunk
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: jsonschema>=4
Requires-Dist: requests
Requires-Dist: PyYAML
Requires-Dist: pydantic>=2
Requires-Dist: pydantic-settings>=2
Requires-Dist: platformdirs>=4
Requires-Dist: pint>=0.24
Requires-Dist: tzdata
Requires-Dist: marinerg-schema>=0.5.0
Provides-Extra: kerchunk
Requires-Dist: kerchunk>=0.2.6; extra == "kerchunk"
Requires-Dist: h5py>=3.10; extra == "kerchunk"
Requires-Dist: fsspec>=2024.2; extra == "kerchunk"
Requires-Dist: zarr>=3; extra == "kerchunk"
Requires-Dist: xarray>=2024.1; extra == "kerchunk"
Requires-Dist: h5netcdf>=1.3; extra == "kerchunk"
Requires-Dist: numpy>=2.4.6; extra == "kerchunk"
Provides-Extra: vre
Requires-Dist: kerchunk>=0.2.6; extra == "vre"
Requires-Dist: h5py>=3.10; extra == "vre"
Requires-Dist: fsspec>=2024.2; extra == "vre"
Requires-Dist: zarr>=3; extra == "vre"
Requires-Dist: xarray>=2024.1; extra == "vre"
Requires-Dist: h5netcdf>=1.3; extra == "vre"
Requires-Dist: intake>=2; extra == "vre"
Requires-Dist: intake-xarray; extra == "vre"
Requires-Dist: matplotlib>=3.8; extra == "vre"
Requires-Dist: jinja2>=3; extra == "vre"
Requires-Dist: aiohttp>=3.9; extra == "vre"
Requires-Dist: numpy>=2.4.6; extra == "vre"
Provides-Extra: types
Requires-Dist: types-requests; extra == "types"
Requires-Dist: types-PyYAML; extra == "types"
Provides-Extra: convert
Requires-Dist: pandas>=2.2; extra == "convert"
Requires-Dist: numpy>=2; extra == "convert"
Requires-Dist: xarray>=2024.1; extra == "convert"
Requires-Dist: h5netcdf>=1.3; extra == "convert"
Requires-Dist: h5py>=3.10; extra == "convert"
Requires-Dist: openpyxl>=3.1; extra == "convert"
Requires-Dist: scipy>=1.11; extra == "convert"
Dynamic: license-file

# marinerg-data

marinerg-data is the command-line client for the MARINERG-i data catalogue.
A test facility uses it to turn instrument exports into standard NetCDF, check
them, and publish them. A researcher uses it to find and download what has been
published.

## How it fits together

Data files never pass through MARINERG-i. A facility deposits its files in its
own Zenodo account, which gives the dataset a DOI. This tool then registers the
dataset in the MARINERG-i catalogue, and fills in the record from the files. The
catalogue's own docs explain
[how the catalogue works](https://git.ichec.ie/marinerg-i/ckanext-marinerg/-/blob/main/docs/how-the-catalogue-works.md).

Facilities differ in how organised their data is. The tool works at every level:

| Level | The facility has | It can |
|---|---|---|
| 0 | Files and a test report | Add a catalogue record by hand, with a contact for the data |
| 1 | Run logs and vendor exports | Describe its exports with templates, then convert them |
| 2 | Consistent formats and instrument lists | Convert, validate and publish each campaign |
| 3 | Standard formats such as CF NetCDF | Validate and publish directly |

## Install

```sh
pip install "marinerg-data[convert,kerchunk]"   # to convert and publish
pip install "marinerg-data[vre]"                # to use data in a VRE
```

`[convert]` reads exports and writes NetCDF. `[kerchunk]` adds a small index
beside each NetCDF, so a VRE can stream part of a file from Zenodo. Without
either, the client still validates and publishes files that are already
converted.

## Publish your data

Publishing takes four commands. Each one says what to run next.

1. **Inspect.** `marinerg-data inspect <folder>` says which equipment template
   matches each export. A template describes how one instrument writes its
   files: the delimiter, the decimal separator, the units, and which column is
   time. If nothing matches, write a template first, as
   [Templates](#templates) explains.
2. **Convert.** `marinerg-data convert <folder>` reads each export through its
   template and writes a NetCDF file and a run manifest, which describes the
   run. It stops, naming the column and row, at anything the template does not
   settle, such as a unit the file and the template disagree on.

   For a test campaign with several instruments, keep a folder per instrument
   and convert them together with `--campaign`. Each run then gets one manifest
   over every instrument, with each clock's offset applied. It also drafts
   `dataset.yaml`, the metadata for the catalogue record. Read it and edit it:
   the tool never overwrites it.
3. **Validate.** `marinerg-data validate <folder>` checks the manifests and
   `dataset.yaml` against the vocabularies the catalogue uses, so a mistake is
   caught before anything is uploaded.
4. **Publish.** Set `ZENODO_TOKEN` to your Zenodo token, `CKAN_URL` to the
   catalogue's address and `CKAN_TOKEN` to your catalogue API token. Then run
   `marinerg-data publish <folder> --sandbox` to try it on sandbox.zenodo.org,
   and drop `--sandbox` when it looks right. `--new-version` publishes a
   correction as a new version of the same record.

If a publish fails part-way, run the same command again. It carries on from
where it stopped, and never mints a second DOI for the same data.

## Try it without data

[demo/](demo/README.md) walks the publish path with generated data, so you can
see every step before using your own files.

## Use published data

`marinerg-data fetch <doi-or-name> --cache <folder>` downloads a dataset into a
shared cache, so a lab downloads each file once. A name, rather than a DOI, is
looked up in the catalogue at `CKAN_URL`.

`marinerg-data catalog` writes an [intake](https://intake.readthedocs.io)
catalogue of the published datasets, for opening them in Python.
[examples/](examples/README.md) shows both in Jupyter and in a Virtual Research
Environment.

## Templates

The package ships equipment templates for common instruments;
`marinerg-data template list` shows them. For an instrument without one:

1. `marinerg-data template draft <export>` guesses a template from a sample
   file and writes it beside the file. The guesses are only a start: check
   every value, and replace each `TODO`. The template will not load until you
   have. Each channel names what it measures with a parameter code;
   `marinerg-data template parameters` lists them.
2. `marinerg-data template check <template-file> <export>` reads the export
   through it and reports what it mapped and what it could not.
3. Keep your templates in a folder, and pass it to every command with
   `--templates <folder>`. Templates in your user config folder are loaded too
   (`~/.config/marinerg-data/templates` on Linux).

A campaign also needs a facility template, with the facility's name, ROR and
default licence, and a campaign template, listing the instruments and naming the
operator's log. Keep them in `facilities/` and `campaigns/` inside your
templates folder. The fixtures in
[tests/fixtures/templates](tests/fixtures/templates) are worked examples of all
three kinds.

## Develop

```sh
uv sync --group dev
uv run tox
```

`uv run tox` runs the format check, lint, type check and tests, as CI does.

`marinerg-data dev` generates test and demo data. It is left out of `--help`;
`marinerg-data dev --help` lists what it does.

## Release

Open the [newest pipeline for `main`](https://git.ichec.ie/marinerg-i/marinerg-data/-/pipelines/main/latest)
and press the play button on release patch, minor or major. The release pipeline publishes to
PyPI, then tags the commit and creates the GitLab release.

## Licence

Copyright Irish Centre for High End Computing, 2026. MIT. See [`LICENSE`](LICENSE).
