Metadata-Version: 2.4
Name: ncarnate
Version: 2.0.1
Summary: Convert HDF4/HDF-EOS2 (MODIS, AMSR-E) to CF-annotated netCDF4 and losslessly recompress netCDF/HDF5 files.
Project-URL: Homepage, https://github.com/ErickShepherd/ncarnate
Project-URL: Source, https://github.com/ErickShepherd/ncarnate
Project-URL: Bug Tracker, https://github.com/ErickShepherd/ncarnate/issues
Author-email: Erick Shepherd <Contact@ErickShepherd.com>
License-Expression: MIT
License-File: LICENSE
Keywords: AMSR-E,CF conventions,GIS,HDF-EOS,HDF-EOS2,HDF4,HDF5,MODIS,atmospheric science,compression,geolocation,grid,netCDF,netCDF4,recompression,remote sensing,swath,xarray
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Atmospheric Science
Classifier: Topic :: Scientific/Engineering :: GIS
Requires-Python: >=3.10
Requires-Dist: netcdf4>=1.6
Requires-Dist: numpy>=1.26
Requires-Dist: pyhdf>=0.11.6
Requires-Dist: pyproj>=3.6
Requires-Dist: tqdm>=4.66
Provides-Extra: test
Requires-Dist: pytest; extra == 'test'
Description-Content-Type: text/x-rst

ncarnate
========

|ci| |license| |python|

.. |ci| image:: https://github.com/ErickShepherd/ncarnate/actions/workflows/ci.yml/badge.svg
   :target: https://github.com/ErickShepherd/ncarnate/actions/workflows/ci.yml
   :alt: CI status

.. |license| image:: https://img.shields.io/badge/license-MIT-blue.svg
   :target: https://github.com/ErickShepherd/ncarnate/blob/master/LICENSE
   :alt: MIT License

.. |python| image:: https://img.shields.io/badge/python-3.10%E2%80%933.13-blue.svg
   :target: https://pypi.org/project/ncarnate/
   :alt: Python 3.10-3.13

Reincarnate legacy scientific data as modern netCDF4.

ncarnate reads netCDF3, netCDF4/HDF5, and HDF4/HDF-EOS2 files and writes
recompressed, CF-annotated netCDF4. It does two jobs:

- **Recompress** netCDF/HDF5 files — change the compression level,
  shuffle filter, or storage layout without changing a single stored
  value.
- **Convert** HDF4 and HDF-EOS2 granules (AMSR-E, MODIS, and kin) to
  netCDF4, reconstructing the CF coordinates that modern tools (xarray,
  QGIS, Panoply) need: grid projections become CF grid mappings with
  1-D ``x``/``y`` and 2-D ``lat``/``lon`` coordinates, swath geolocation
  is attached as CF coordinates, and dimension-mapped (e.g. 5 km → 1 km)
  geolocation is interpolated through ECEF space.

Problems this solves
--------------------

Reach for ncarnate if you are trying to:

- **Convert HDF4 / HDF-EOS2 granules (MODIS, AMSR-E, and kin) to netCDF4** so
  they open cleanly in xarray, QGIS, or Panoply.
- **Read an HDF-EOS2 swath or grid that has no usable lat/lon** — ncarnate
  reconstructs CF ``lat``/``lon`` coordinates and grid mappings so the data is
  actually georeferenced, instead of an unplottable array.
- **Recompress a netCDF4 / HDF5 file** — change the compression level or shuffle
  filter without altering a single stored value.
- **Shrink an archive of scientific files** without risking the science: every
  output is verified value-for-value against its source before it replaces
  anything, and stored values round-trip bit-identically.
- **Batch-convert a directory tree** of legacy granules to modern netCDF4 in one
  command.

The fidelity contract
---------------------

Converting or recompressing a file changes *storage*, never *science
data*:

- Every variable's stored values are preserved **bit-identically** —
  packed integers stay packed; ``scale_factor``/``add_offset``/
  ``_FillValue`` are carried across as declarations, never applied.
- Every dimension (including unlimited-ness), attribute (including its
  type), and group survives. HDF-EOS2 ``StructMetadata`` is preserved
  verbatim; names netCDF cannot hold are sanitized with the original
  recorded in a companion attribute.
- Geolocation reconstruction is strictly **additive**: the original
  information always rides along, so the conversion never becomes the
  only copy of the truth.
- Every output is **verified against the source value-for-value before
  it replaces anything**. A source file is never destroyed by a failed
  run, and HDF4 sources are never replaced at all.
- Unsupported constructs (user-defined netCDF types, unverified GCTP
  projections, exotic swath layouts) **fail loud** with a named error
  rather than guessing — a wrong coordinate is worse than a refused
  conversion. ``--no-geolocation`` converts the raw payload anyway.

The details, the guarantee boundary, and how the test suite pins each
clause live in ``docs/fidelity-notes.md``.

Installation
------------

**With conda** (from `conda-forge <https://anaconda.org/conda-forge/ncarnate>`_):

.. code-block:: console

   conda install -c conda-forge ncarnate

This works on every platform and is the recommended install on
**Windows** — conda-forge's ``pyhdf`` is built against a proper HDF4
library everywhere, so the full HDF4/HDF-EOS2 converter runs on Windows,
macOS, and Linux alike.

**With pip** (from `PyPI <https://pypi.org/project/ncarnate/>`_):

.. code-block:: console

   pip install ncarnate

On **Linux (x86_64)** and **macOS (arm64)**, every dependency — including
``pyhdf`` — installs as a self-contained binary wheel with no system
libraries required. On platforms without a repaired ``pyhdf`` wheel (e.g.
Linux aarch64), building from sdist requires the system HDF4 library first
(Debian/Ubuntu: ``apt install libhdf4-dev``).

**Windows via pip:** the netCDF/HDF5 *recompression* path works from PyPI
wheels out of the box, but the HDF4/HDF-EOS2 *conversion* path does
**not** — ``pyhdf``'s Windows wheel ships no HDF4 runtime, so
``import pyhdf`` fails with a DLL-load error. Use the conda-forge install
above for HDF4 on Windows (or **WSL** with the pip instructions).

Command line usage
------------------

.. code-block:: console

   # Recompress a netCDF4 file in place (verified before replacement).
   ncarnate observations.nc --complevel 9

   # Keep the original; write observations_recompressed.nc beside it.
   ncarnate --no-overwrite observations.nc

   # Convert an HDF-EOS2 granule -> granule.nc with CF geolocation.
   ncarnate AMSR_E_L3_SeaIce12km_B02_20020619.hdf

   # Convert the raw SDS payload only (unsupported-projection escape hatch).
   ncarnate --no-geolocation granule.hdf

   # Recurse over a directory tree.
   ncarnate -r /data/archive

Exit codes: ``0`` success, ``1`` one or more files failed, ``2`` bad
input paths or arguments.

Library usage
-------------

.. code-block:: python

   from ncarnate import recompress

   # Lossless recompression; returns the output path.
   recompress("observations.nc", complevel=9)

   # HDF-EOS2 conversion; the .hdf source is never replaced.
   recompress("granule.hdf", dst="granule.nc")

Example
-------

The AMSR-E daily 12.5 km sea-ice granule this project grew up around:

===============================================  ==========  ===========
File                                             Input       Output
===============================================  ==========  ===========
netCDF4 recompression (``--complevel 9``)        42.6 MB     19.9 MB
HDF-EOS2 → netCDF4 (+ reconstructed lat/lon)     60.2 MB     35.5 MB
===============================================  ==========  ===========

Both outputs re-read bit-identically to their sources; the conversion
additionally carries CF ``polar_stereographic`` grid mappings and
coordinates for both hemispheric grids. The northern grid's
reconstructed latitudes/longitudes agree with The HDF Group's
independent conversion of the same granule to within 10\ :sup:`-5`
degrees (about a metre), the tolerance the test suite enforces.

Supported inputs
----------------

- **netCDF4 / HDF5** and **netCDF3** — recompressed via the netCDF4
  library.
- **HDF4 / HDF-EOS2** — read via the pyhdf SD API. GRID structures with
  GCTP polar-stereographic, geographic, and Lambert-azimuthal
  (EASE-Grid) projections; SWATH structures with direct or
  dimension-mapped geolocation. Output is always netCDF4 — HDF4 is
  never written.

Development
-----------

.. code-block:: console

   pip install -e ".[test]"
   ruff check .
   pytest

The test suite runs entirely offline against small committed fixtures
trimmed from real granules (provenance sidecars included); cross-checks
against the raw multi-MB granules self-skip where the local granule
store is absent.

License
-------

MIT — see ``LICENSE``.
