Metadata-Version: 2.4
Name: marisco
Version: 1.9.1
Summary: MARIS companion package and tutorials
Author-email: "Franck Albinet, Niall Murphy" <franckalbinet@gmail.com>
License: Apache-2.0
Project-URL: Repository, https://github.com/franckalbinet/marisco
Project-URL: Documentation, https://franckalbinet.github.io/marisco
Keywords: nbdev,jupyter,notebook,python,netcdf,marine,radioactivity,data
Classifier: Natural Language :: English
Classifier: Intended Audience :: Developers
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas
Requires-Dist: openpyxl
Requires-Dist: fastcore>=2.1.16
Requires-Dist: rich
Requires-Dist: tqdm
Requires-Dist: netcdf4
Requires-Dist: tomli
Requires-Dist: tomli-w
Requires-Dist: shapely
Requires-Dist: pyzotero
Requires-Dist: jellyfish
Requires-Dist: requests
Requires-Dist: pyarrow
Requires-Dist: gevent>=22.10.2
Requires-Dist: gsw
Provides-Extra: dev
Requires-Dist: nbdev; extra == "dev"
Requires-Dist: ipykernel; extra == "dev"
Requires-Dist: twine; extra == "dev"
Requires-Dist: toolslm; extra == "dev"
Dynamic: license-file

# Welcome to MARISCO


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

The [IAEA Marine Radioactivity Information System
(MARIS)](https://maris.iaea.org) provides open access to radioactivity
measurements in marine environments (seawater, biota, sediment, and
suspended matter) collected by national laboratories, monitoring
programmes, and research institutions worldwide. Developed by the [IAEA
Marine Environmental
Laboratories](https://www.iaea.org/about/organizational-structure/department-of-nuclear-sciences-and-applications/division-of-iaea-environment-laboratories)
in Monaco, MARIS is the authoritative international reference for marine
radioactivity data.

MARISCO is being developed as the data processing foundation of MARIS: a
Python package that converts provider datasets into standardised
[NetCDF4](https://www.unidata.ucar.edu/software/netcdf/) and CSV formats
for ingestion into the MARIS database.

## How it works

MARIS does not impose a data schema on providers. It accepts data in
whatever form it arrives, and it makes the cost of that acceptance
visible.

Each dataset is processed by a dedicated *handler*, a Jupyter notebook
that documents every curation decision alongside the code that
implements it. Unit conversions, nuclide nomenclature mapping,
coordinate standardisation, outlier flags: every choice is visible,
traceable, and auditable. Some formats need less work than others, and
the handler makes that difference plain. If you are a data provider, see
[contributing data](how-to/contributing-data.ipynb) for what this means
for your dataset.

This is deliberate. The friction a dataset creates is a signal, and
surfacing it is the point. It tells the MARIS team how much work a
provider’s format really requires, it gives the provider a concrete,
itemised record of what to improve, and it makes the next release of the
dataset cheaper to ingest. Over time, the easiest path for everyone is
for providers to align with the MARIS way of organising data, not
because it is imposed, but because the cost of not doing so is visible.

The payoff is a harmonised world reference: measurements of marine
radioactivity from providers worldwide, stored in the [MARIS
database](https://maris.iaea.org/home) in one comparable form. That
comparability is the real value, for data providers, for the scientific
community, and beyond. It rests on visible curation: the process that
made the data comparable is documented and auditable, so the
harmonisation can be checked, questioned, and trusted.

Handlers are shared with data providers as a working document. If a
conversion looks wrong, a mapping is missing, or terminology differs
from your internal standards, the notebook is where we discuss and
resolve it, not buried in email threads or internal comments.

For a concrete example, see the [GEOTRACES
handler](https://fr.anckalbi.net/marisco/handlers/geotraces.html).

The harmonised data reaches users through three channels: the [explore
portal](https://maris.iaea.org/explore) for interactive visualisation,
the [datasets page](https://maris.iaea.org/datasets) for downloading
individual NetCDF files (joined soon by
[INIS](https://www.iaea.org/resources/databases/inis)), and the [MARIS
API](https://observablehq.com/@maris-iaea/api) for programmatic access.

## Available handlers

| Handler | Description | Status |
|----|----|----|
| [FRAM STRAIT 2025](handlers/fram_strait_2025.html) | Fram Strait 2025 iodine-129 seawater radionuclide data | Active |
| [GEOTRACES](handlers/geotraces.html) | BODC GEOTRACES oceanographic radionuclide data | Active |
| [HELCOM](handlers/helcom.html) | Baltic Sea marine environment monitoring data | Active |
| [JOIS](handlers/jois.html) | BGOS-JOIS Beaufort Sea seawater radionuclide data | Active |
| [MARIS LEGACY](handlers/maris_legacy.html) | MARIS legacy database dump | Active |
| [OSPAR](handlers/ospar.html) | OSPAR data | Under refactoring |
| [TEPCO](handlers/tepco.html) | TEPCO data | Under refactoring |

## Install

``` console
pip install marisco
```

Two environment variables may be required depending on the dataset:

- **`ZOTERO_API_KEY`**: used to fetch bibliographic metadata from the
  [MARIS Zotero
  library](https://www.zotero.org/groups/2432820/maris/library). Contact
  [MARIS Administrators](https://maris.iaea.org/home) to obtain your
  key:

  ``` console
  export ZOTERO_API_KEY=your_api_key_here
  ```

- **INIS**: bibliographic metadata is being progressively migrated from
  Zotero to the [IAEA INIS
  database](https://www.iaea.org/resources/databases/inis). The
  [`INISClient`](https://franckalbinet.github.io/marisco/api/metadata.html#inisclient)
  fetches records via [curl](https://curl.se/), which must be installed
  separately. In the near term both systems coexist; INIS will supersede
  Zotero as the migration completes.

## Command-line utilities

All commands accept `-h` for help.

> [!TIP]
>
> ### List available handlers
>
> Run `marisco-ingest` with an invalid handler name. The command prints
> the available handlers.

### Ingest a provider dataset

``` console
$ marisco-ingest helcom output/100-HELCOM-MORS-2024.nc
```

The handler name selects the provider. Each handler decides whether to
download the source data or read it from a local file.

### Ingest a MARIS legacy database dump

``` console
$ marisco-ingest maris_legacy ~/output --src ~/data/maris/dump.txt
```

To process only selected references:

``` console
$ marisco-ingest maris_legacy ~/output \
    --src ~/data/maris/dump.txt \
    --ref_ids "16,30"
```

If a handler produces several files, the command writes them to the
destination folder. It creates the folder if it does not exist.

### Export NetCDF4 to MARIS CSV

``` console
$ marisco-export ~/output/191-OSPAR-2024.nc
```

The command appends the sample type to the output name. For example,
`191-OSPAR-2024_BIOTA.csv`. It creates one CSV file for each sample type
in the NetCDF4 file.

Use `--dest` to choose a different output stem.

## Documentation

Documentation is organised into two groups.

### How-to guides

Practical walkthroughs for common tasks:

- [Contributing data to MARIS](how-to/contributing-data.ipynb): how a
  data provider delivers a dataset to the repository
- [Writing a new handler](how-to/writing-a-handler.ipynb): step-by-step
  guide to adding a new data provider to the MARISCO pipeline
- [Nomenclature reconciliation](how-to/reconcile-nomenclature.ipynb):
  repeatable procedure for mapping provider names (nuclides, species,
  units, …) to MARIS standard identifiers
- [Explore a MARIS NetCDF file](how-to/explore-nc.ipynb): step-by-step
  guide to reading a MARIS NetCDF file: global metadata, group
  structure, lookup tables and nomenclatures, and each sample-type
  dataset as a [pandas](https://pandas.pydata.org) DataFrame

### Reference

Detailed specifications and reference material:

- [Reporting template](reference/reporting-template.ipynb): the MARIS
  data reporting template as documentation, with the general
  instructions, per-sample-type column tables, and lookup lists
- [Field Definitions](reference/field-definition.ipynb): complete
  field-by-field reference with MARISCO column names, NetCDF variable
  names, CSV variable names, types, and lookup tables
- [MARIS database schema](reference/maris-db-schema.ipynb): the master
  database `sample` and `measure` tables, their mandatory columns, and
  their constraints

## Development

`marisco` follows the nbdev workflow: all code lives in notebooks under
`nbs/`, and the `marisco/` package is generated from them.

- Edit the notebooks in `nbs/`, then run `nbdev-export` to regenerate
  the modules and `nbdev-readme` to regenerate this README.
- After changing `pyproject.toml` (entry points, dependencies),
  reinstall the package with `pip install -e .`.
- The MARIS NetCDF template is generated from `nbs/files/cdl/maris.cdl`.
  To regenerate it locally, install
  [NetCDF-C](https://pjbartlein.github.io/REarthSysSci/install_netCDF.html)
  and run:

``` console
ncgen -4 -o nbs/files/nc/maris-template.nc nbs/files/cdl/maris.cdl
```

- Contributor guidance on architecture, coding conventions, and handler
  documentation style lives in the `CRAFTs/` notebooks at the repository
  root.
- [Writing a new handler](how-to/writing-a-handler.ipynb) walks through
  adding a new data provider to the pipeline.

## Acknowledgements

Development of this package was supported by the
[Solveit](https://solve.it.com) platform, an interactive development
environment for dialog-driven software engineering.
