Metadata-Version: 2.4
Name: tethys-archiver
Version: 0.1.1
Summary: Faithful netCDF4 export of tethys datasets to local disk, with all metadata preserved.
Project-URL: Source, https://github.com/tethys-ts/tethys-archiver
Author-email: mullenkamp <mullenkamp1@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: archive,environmental-data,netcdf,tethys,time-series
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering
Requires-Python: <3.12,>=3.11
Requires-Dist: h5netcdf
Requires-Dist: h5py
Requires-Dist: hdf5tools==0.2.4
Requires-Dist: numpy<2
Requires-Dist: pandas<2
Requires-Dist: requests
Requires-Dist: tethysts==4.5.16
Requires-Dist: xarray
Requires-Dist: zstandard
Description-Content-Type: text/markdown

# tethys-archiver

Faithful netCDF4 export of [tethys](https://github.com/mullenkamp/tethysts) datasets to
local disk, with all metadata preserved.

Tethys is being decommissioned in favour of envlib. This tool is the step that makes
retirement safe: it takes a full, checkable, local copy of a dataset before anything is
switched off — and it doubles as the extraction half of the envlib migration.

## Why it exists

`tethysts` pins `pandas<2`, which forces the numpy 1.x C ABI. `cfdb>=0.9.4` (under
envlib) requires numpy>2 and asserts that floor at runtime. `uv` reports the pair as
**unsatisfiable** — so nothing can read tethys and write envlib in one process.

This tool lives on the legacy side of that wall. It writes plain netCDF4, which the
modern stack reads with no legacy dependency at all. The file is the interchange format
between two environments that cannot coexist.

Two further reasons it earns its place:

* **The ECan quality-controlled record is single-copy.** It was built from CSV exports of
  an internal database that no longer exists; the tethys objects are the only surviving
  copy.
* **The extraction work was going to be written anyway** — there are ~30
  `tethys-extraction-*` sources. Better once, here, than once per source.

## Install

```bash
uv sync
```

Python **3.11 only**, deliberately — pandas 1.5.x has no cp312 wheel, and the pin set
(numpy 1.26 / pandas 1.5.3 / h5py 3.16 / h5netcdf 1.8.1 / tethysts 4.5.16 /
hdf5tools 0.2.4) is verified working end to end.

## Use

Dry run is the **default**; a real run needs `--yes`.

```bash
# what would happen
uv run tethys-export --out ~/data/tethys-archive

# do it, with the verification gates
uv run tethys-export --out ~/data/tethys-archive --yes --verify

# re-check an existing archive against its manifest
uv run tethys-export --out ~/data/tethys-archive --check-manifest
```

Default scope is the six frozen ECan quality-controlled hourly datasets. `--dataset`
(repeatable) selects specific ones; `--all-regular` takes every regular-cadence qc/raw
time-series dataset in the bucket.

## Output layout

```
<out>/<bucket>/
    datasets.json                          # bucket catalogue, verbatim
    manifest.json                          # per-file sha256 + counts
    <dataset_id>/
        dataset.json                       # verbatim
        versions.json                      # verbatim
        <version_date>.nc                  # all stations, dense (station, time)
        <version_date>.stations.json       # verbatim
        <version_date>.results_chunks.json # verbatim — per-chunk hashes
```

The JSON descriptors are written **byte-for-byte as tethys published them**, never
re-serialised. `results_chunks.json` is the only place `chunk_hash` / `chunk_id` /
`chunk_day` survive — `get_results` concatenates chunks and structurally cannot carry
them through.

### Reading the output

Nothing legacy required:

```python
import xarray as xr
ds = xr.open_dataset('20220401T000000Z.nc', engine='h5netcdf')
ds['streamflow'][0, :]          # decoded to physical units automatically
```

Variables keep **tethys's own names** (`ref`, `name`, `altitude`, `station_id`, `lon`,
`lat`). Renaming to envlib's `station_ref` / `station_name` / `station_altitude` is the
consumer's job — see *Faithful, not normalised* below.

## Design

**Faithful, not normalised.** No variable renames, no resampling, no vocabulary mapping,
no unit conversion. Every transformation belongs downstream. The one reshaping this tool
*does* perform is the scatter onto a dense axis at the dataset's declared cadence — with
the `geometry` → `station` dimension rename that comes with it — and that is recorded in
the file and is reversible.

**Dense `(station, time)`, not ragged.** With `shuffle` + gzip the NaN runs cost almost
nothing, and `xarray` opens the result as a 2-D array directly. CF ragged arrays would
save a little space and cost real ergonomics, since xarray does not decode them natively.

**Packing is explicit.** h5netcdf does not apply CF `scale_factor` on write, so the
archiver packs integers itself. That is the point: the rounding is testable, and the
integers written are verified against the source objects rather than trusted to a
library.

**`shuffle=True` everywhere.** Measured on the real lake dataset, whole file: 3.16 MB
without it (1.50× the source) against **1.27 MB with it (0.61×)**. The time axis alone
goes 1.80 MB → 0.16 MB.

**Nothing dense is materialised.** Rows are written one station at a time into
pre-created chunked variables, so peak memory is one station's row. `groundwater_depth`
is 376 M cells; the dense planes would otherwise be ~5 GB.

**Stations are fetched one at a time**, not in batches. Batching is faster, but a batched
`get_results` returns the *union* of the batch's timestamps with NaN padding, which makes
a station's genuinely-stored steps indistinguishable from concat padding — and the count
gate needs exactly that distinction. Chunk downloads are already threaded within a single
station's call, so most of the parallelism survives.

**Resumability is the chunk cache**, not checkpoint files. A crash costs the rebuild of
one `.nc`; the re-fetch is nearly free. Note `tethysts.clear_cache` never deletes
anything (it globs `*.nc` while the writer writes `*.h5`) — manage the cache directory by
hand if you run the whole estate.

## Verification

`--verify` runs three gates:

| gate | checks |
|---|---|
| **round-trip** | the archive's stored integers against the **raw chunk objects** — value, `quality_code`, fill mask, and that nothing outside a station's stored steps is anything but fill |
| **counts** | scattered steps == `dimensions.time` == Σ `n_times`. Fatal on disagreement; a finite-value shortfall is *reported*, not fatal (a stored step may legitimately carry a code beside an absent value) |
| **axis** | start/end match the declared spans, step matches `frequency_interval` |

Plus, on every write: a **packed-range audit** (logged even when it passes — see the
`gage_height` warning in `PROVENANCE.md`), a **uniqueness assertion** on the scattered
timestamps, and a refusal to store a finite value that packs onto the fill.

The uniqueness assertion belongs at the write side rather than in a gate, and that is not a
style preference: the scatter `vals[pos] = values` is last-wins on duplicate indices while
the count gate reports `pos.size`, which counts a duplicate *with multiplicity*. A repeated
timestamp therefore destroys an observation while every count still reconciles. Both review
arms found this independently; one constructed it end-to-end through unmodified gate code.

The round-trip gate compares against the source objects rather than against `tethysts`
output on purpose. An earlier prototype compared client-decoded to archive-decoded — both
sides of the same decode path — and so could not have caught an error introduced *by* the
client. There was one: see `modified_date` in `PROVENANCE.md`.

## Scope

v1 covers **regular-cadence, single-height, `time_series`** datasets, written locally.

Not covered: irregular datasets (`frequency_interval` is the literal string `'None'` —
30 of ECan's 36 QC datasets), which have no axis to build and are blocked on a cfdb
representation; `grid` result types; and any push to remote storage. See `OPEN_WORK.md`.

## Read-only, always

The archiver has no write path to the object store and takes no credentials. This matters
more than it looks: the tethys prefix and live envlib member data share the
`ecan-env-monitoring` bucket, and the tethys copy of the quality-controlled record is the
only one that exists. `remote.check_read_only` refuses a remote carrying
`connection_config`.
