Metadata-Version: 2.4
Name: marisco
Version: 1.9.2
Summary: MARIS companion package and tutorials
Author-email: "Franck Albinet, Niall Murphy" <franckalbinet@gmail.com>
License: Apache-2.0
Project-URL: Repository, https://github.com/franckalbinet/marisco
Project-URL: Documentation, https://franckalbinet.github.io/marisco
Keywords: nbdev,jupyter,notebook,python,netcdf,marine,radioactivity,data
Classifier: Natural Language :: English
Classifier: Intended Audience :: Developers
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas
Requires-Dist: openpyxl
Requires-Dist: fastcore>=2.1.16
Requires-Dist: rich
Requires-Dist: tqdm
Requires-Dist: netcdf4
Requires-Dist: tomli
Requires-Dist: tomli-w
Requires-Dist: shapely
Requires-Dist: pyzotero
Requires-Dist: jellyfish
Requires-Dist: requests
Requires-Dist: pyarrow
Requires-Dist: gevent>=22.10.2
Requires-Dist: gsw
Provides-Extra: dev
Requires-Dist: nbdev; extra == "dev"
Requires-Dist: ipykernel; extra == "dev"
Requires-Dist: twine; extra == "dev"
Requires-Dist: toolslm; extra == "dev"
Dynamic: license-file

# Welcome to MARISCO


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

The [IAEA Marine Radioactivity Information System (MARIS)](https://maris.iaea.org) provides open access to radioactivity measurements in marine environments (seawater, biota, sediment, and suspended matter) collected by national laboratories, monitoring programmes, and research institutions worldwide. Developed by the [IAEA Marine Environmental Laboratories](https://www.iaea.org/about/organizational-structure/department-of-nuclear-sciences-and-applications/division-of-iaea-environment-laboratories) in Monaco, MARIS is the authoritative international reference for marine radioactivity data.

MARISCO is being developed as the data processing foundation of MARIS: a Python package that converts provider datasets into standardised [NetCDF4](https://www.unidata.ucar.edu/software/netcdf/) and CSV formats for ingestion into the MARIS database.

## How it works

MARIS does not impose a data schema on providers. It accepts data in whatever form it arrives, and it makes the cost of that acceptance visible.

Each dataset is processed by a dedicated *handler*, a Jupyter notebook that documents every curation decision alongside the code that implements it. Unit conversions, nuclide nomenclature mapping, coordinate standardisation, outlier flags: every choice is visible, traceable, and auditable. Some formats need less work than others, and the handler makes that difference plain. If you are a data provider, see [contributing data](how-to/contributing-data.ipynb) for what this means for your dataset.

This is deliberate. The friction a dataset creates is a signal, and surfacing it is the point. It tells the MARIS team how much work a provider’s format really requires, it gives the provider a concrete, itemised record of what to improve, and it makes the next release of the dataset cheaper to ingest. Over time, the easiest path for everyone is for providers to align with the MARIS way of organising data, not because it is imposed, but because the cost of not doing so is visible.

The payoff is a harmonised world reference: measurements of marine radioactivity from providers worldwide, stored in the [MARIS database](https://maris.iaea.org/home) in one comparable form. That comparability is the real value, for data providers, for the scientific community, and beyond. It rests on visible curation: the process that made the data comparable is documented and auditable, so the harmonisation can be checked, questioned, and trusted.

Handlers are shared with data providers as a working document. If a conversion looks wrong, a mapping is missing, or terminology differs from your internal standards, the notebook is where we discuss and resolve it, not buried in email threads or internal comments.

For a concrete example, see the [GEOTRACES handler](https://fr.anckalbi.net/marisco/handlers/geotraces.html).

The harmonised data reaches users through three channels: the [explore portal](https://maris.iaea.org/explore) for interactive visualisation, the [datasets page](https://maris.iaea.org/datasets) for downloading individual NetCDF files (joined soon by [INIS](https://www.iaea.org/resources/databases/inis)), and the [MARIS API](https://observablehq.com/@maris-iaea/api) for programmatic access.

## Available handlers

| Handler | Description | Status |
|----|----|----|
| [FRAM STRAIT](handlers/fram_strait.html) | Fram Strait iodine-129 seawater radionuclide data | Active |
| [GEOTRACES](handlers/geotraces.html) | BODC GEOTRACES oceanographic radionuclide data | Active |
| [HELCOM](handlers/helcom.html) | Baltic Sea marine environment monitoring data | Active |
| [JOIS](handlers/jois.html) | BGOS-JOIS Beaufort Sea seawater radionuclide data | Active |
| [OSPAR](handlers/ospar.html) | OSPAR data | Under refactoring |
| [TEPCO](handlers/tepco.html) | TEPCO data | Under refactoring |

## Install

``` console
pip install marisco
```

Two environment variables may be required depending on the dataset:

- **`ZOTERO_API_KEY`**: used to fetch bibliographic metadata from the [MARIS Zotero library](https://www.zotero.org/groups/2432820/maris/library). Contact [MARIS Administrators](https://maris.iaea.org/home) to obtain your key:

  ``` console
  export ZOTERO_API_KEY=your_api_key_here
  ```

- **INIS**: bibliographic metadata is being progressively migrated from Zotero to the [IAEA INIS database](https://www.iaea.org/resources/databases/inis). The [`INISClient`](https://franckalbinet.github.io/marisco/api/metadata.html#inisclient) fetches records via [curl](https://curl.se/), which must be installed separately. In the near term both systems coexist; INIS will supersede Zotero as the migration completes.

## Command-line utilities

All commands accept `-h` for help.

> [!TIP]
>
> ### List available handlers
>
> Run `marisco-ingest` with an invalid handler name. The command prints the
> available handlers.

### Ingest a provider dataset

``` console
$ marisco-ingest helcom output/100-HELCOM-MORS-2024.nc
```

The handler name selects the provider. Each handler decides whether to download
the source data or read it from a local file.

### Ingest a MARIS legacy database dump

``` console
$ marisco-ingest maris_legacy ~/output --src ~/data/maris/dump.txt
```

To process only selected references:

``` console
$ marisco-ingest maris_legacy ~/output \
    --src ~/data/maris/dump.txt \
    --ref_ids "16,30"
```

If a handler produces several files, the command writes them to the destination
folder. It creates the folder if it does not exist.

### Export NetCDF4 to MARIS CSV

``` console
$ marisco-export ~/output/191-OSPAR-2024.nc
```

The command appends the sample type to the output name. For example,
`191-OSPAR-2024_BIOTA.csv`. It creates one CSV file for each sample type in the
NetCDF4 file.

Use `--dest` to choose a different output stem.

## Documentation

Documentation is organised into two groups.

### How-to guides

Practical walkthroughs for common tasks:

- [Contributing data to MARIS](how-to/contributing-data.ipynb): how a data provider delivers a dataset to the repository
- [Writing a new handler](how-to/writing-a-handler.ipynb): step-by-step guide to adding a new data provider to the MARISCO pipeline
- [Nomenclature reconciliation](how-to/reconcile-nomenclature.ipynb): repeatable procedure for mapping provider names (nuclides, species, units, …) to MARIS standard identifiers
- [Explore a MARIS NetCDF file](how-to/explore-nc.ipynb): step-by-step guide to reading a MARIS NetCDF file: global metadata, group structure, lookup tables and nomenclatures, and each sample-type dataset as a [pandas](https://pandas.pydata.org) DataFrame

### Reference

Detailed specifications and reference material:

- [Reporting template](reference/reporting-template.ipynb): the MARIS data reporting template as documentation, with the general instructions, per-sample-type column tables, and lookup lists
- [Field Definitions](reference/field-definition.ipynb): complete field-by-field reference with MARISCO column names, NetCDF variable names, CSV variable names, types, and lookup tables
- [MARIS database schema](reference/maris-db-schema.ipynb): the master database `sample` and `measure` tables, their mandatory columns, and their constraints

## Development

`marisco` follows the nbdev workflow: all code lives in notebooks under `nbs/`, and the `marisco/` package is generated from them.

- Edit the notebooks in `nbs/`, then run `nbdev-export` to regenerate the modules and `nbdev-readme` to regenerate this README.
- After changing `pyproject.toml` (entry points, dependencies), reinstall the package with `pip install -e .`.
- The MARIS NetCDF template is generated from `nbs/files/cdl/maris.cdl`. To regenerate it locally, install [NetCDF-C](https://pjbartlein.github.io/REarthSysSci/install_netCDF.html) and run:

``` console
ncgen -4 -o nbs/files/nc/maris-template.nc nbs/files/cdl/maris.cdl
```

- Contributor guidance on architecture, coding conventions, and handler documentation style lives in the `CRAFTs/` notebooks at the repository root.
- [Writing a new handler](how-to/writing-a-handler.ipynb) walks through adding a new data provider to the pipeline.

## Acknowledgements

Development of this package was supported by the [Solveit](https://solve.it.com) platform,
an interactive development environment for dialog-driven software engineering.
