Metadata-Version: 2.4
Name: fwl-io
Version: 26.7.20
Summary: Shared data-download utilities for the PROTEUS ecosystem: manifest-driven, mirrored, offline-first fetching of reference data
Author-email: Tim Lichtenberg <tim.lichtenberg@rug.nl>
License: Apache 2.0 License
Project-URL: Repository, https://github.com/FormingWorlds/fwl-io
Project-URL: Issues, https://github.com/FormingWorlds/fwl-io/issues
Keywords: Astronomy,Exoplanets,Model-coupling,Data-management
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: pooch>=1.8
Requires-Dist: requests>=2.31
Provides-Extra: develop
Requires-Dist: pytest>=8.0; extra == "develop"
Requires-Dist: pytest-cov; extra == "develop"
Requires-Dist: ruff; extra == "develop"
Provides-Extra: docs
Requires-Dist: zensical; extra == "docs"
Requires-Dist: markdown-include; extra == "docs"
Requires-Dist: mkdocs; extra == "docs"
Requires-Dist: mkdocs-material; extra == "docs"
Requires-Dist: mkdocstrings[python]; extra == "docs"
Dynamic: license-file

# fwl-io

[![CI](https://github.com/FormingWorlds/fwl-io/actions/workflows/ci.yml/badge.svg)](https://github.com/FormingWorlds/fwl-io/actions/workflows/ci.yml)
[![Documentation](https://github.com/FormingWorlds/fwl-io/actions/workflows/docs.yaml/badge.svg)](https://proteus-framework.org/fwl-io/)
[![PyPI](https://img.shields.io/pypi/v/fwl-io)](https://pypi.org/project/fwl-io/)
[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE.txt)

Shared data-download utilities for the [PROTEUS](https://github.com/FormingWorlds/PROTEUS) ecosystem: manifest-driven, mirrored, offline-first fetching of the reference data used by the FormingWorlds models.

**Status**: version 0, released for inspection and feedback by the collaboration. The API is not yet frozen; suggestions are welcome as [issues](https://github.com/FormingWorlds/fwl-io/issues). Documentation: [proteus-framework.org/fwl-io](https://proteus-framework.org/fwl-io/).

## What it does

Models in the PROTEUS framework depend on reference data hosted on Zenodo (with Dataverse mirrors): spectral files, equation-of-state lookup tables, stellar evolution tracks. fwl-io provides one shared mechanism to declare, verify, and fetch these datasets into the common `FWL_DATA` directory tree:

- **Manifests**: each dataset is declared in a TOML manifest with its location below `FWL_DATA`, its Zenodo version DOI, an optional Dataverse mirror DOI, and the models that require it. fwl-io ships a manifest for datasets shared across models; each model package can ship its own manifest (plus the generated registry files, both included in its package data) and expose it through the `fwl_io.manifests` entry-point group, so adding data to a model never requires an fwl-io release. A provider whose manifest fails to load is skipped with a warning; it cannot break data access for the other models.
- **Committed registries**: the file names and checksums of every dataset live in registry files generated by `fwl-io sync` from the Zenodo API. Hashes are pinned in version control and reviewed like code; nothing is hand-edited. The checksums are the ones Zenodo publishes (md5 today), so the Dataverse mirror must host byte-identical copies of the originals; disable Dataverse's tabular ingest for mirrored deposits.
- **Version DOIs only**: manifests pin the DOI of a specific deposit, and every dataset requires a Zenodo version DOI (Dataverse serves as a download mirror). `fwl-io sync` rejects concept DOIs, which resolve to the newest deposit and would let data drift underneath pinned code.
- **Mirrored downloads**: every file is fetched with [pooch](https://www.fatiando.org/pooch/), verified against its checksum, tried against Zenodo first and the Dataverse mirror second, and moved into place atomically (staged on the same filesystem, `os.replace` into place) so an interrupted download can never leave a corrupt file behind. Local placement failures such as a read-only tree or a full disk are reported as themselves, never as mirror failures. Archives are downloaded verbatim; automatic extraction is planned.
- **Offline-first**: with `FWL_IO_OFFLINE=1` no network access is attempted; files resolve from the local tree or fail with an actionable error. A read-only, pre-populated group cache (`FWL_DATA_CACHE`) is consulted before any download, which serves cluster nodes without internet access.

## Installation

```bash
pip install fwl-io
```

The distribution and the command are named `fwl-io`; the Python import is `fwl_io`, since Python does not allow hyphens in module names.

## Usage

```python
from fwl_io import create_fetcher, fetch_for

eos = create_fetcher(
    subdir='interior/eos/mgsio3_demo',
    zenodo='10.5281/zenodo.1234567',
    dataverse='10.34894/ABCDEF',
    registry='interior.eos.mgsio3_demo.registry.txt',
)
path = eos.fetch('density.dat')      # cached, verified, atomic
paths = fetch_for('aragog')          # everything a model requires
records = eos.provenance()           # (file, source, checksum) for run manifests
```

Command line:

```bash
fwl-io list                     # datasets from all installed manifests
fwl-io fetch aragog             # fetch everything aragog requires
fwl-io sync manifest.toml       # regenerate committed registries from Zenodo
```

## Environment variables

| Variable | Meaning |
|---|---|
| `FWL_DATA` | Root of the writable data tree. Required unless an explicit `data_root` is passed; there is no silent default location. |
| `FWL_DATA_CACHE` | Optional read-only, pre-populated copy of the tree (for example a group share on a cluster); consulted before any download, never written. |
| `FWL_IO_OFFLINE` | Set to `1` to forbid all network access. |

## Adding a dataset

1. Upload the files to Zenodo and note the **version** DOI of the deposit.
2. Declare the dataset in the appropriate manifest: the shared manifest in this repository for data consumed by several models, or the consuming model's own manifest for model-specific data.
3. Run `fwl-io sync <manifest>` and commit the manifest change together with the generated registry file. Packages that ship their own manifest must include both the manifest and its registry files in their package data, or fetching fails at runtime on the user's machine.
4. Optionally mirror the deposit to Dataverse (byte-identical, tabular ingest disabled) and add the `dataverse` DOI.

## Development

```bash
pip install -e ".[develop]"
pytest -m "unit or smoke"       # fast tier, no network
pytest                          # full suite; uses a local test server only
ruff check src tests && ruff format --check src tests
```

The test suite never contacts external services; download logic is exercised against a local HTTP server.
