Metadata-Version: 2.5
Name: pelago
Version: 0.0.1
Summary: Helpers for Ocean / Pontus-X compute-to-data algorithms: inputs, consumer parameters, outputs, HDF5 sections and PDF reports.
Project-URL: Homepage, https://git.ift.tuwien.ac.at/pub/pypi/pelago
Project-URL: Source, https://git.ift.tuwien.ac.at/pub/pypi/pelago
Project-URL: Issues, https://git.ift.tuwien.ac.at/pub/pypi/pelago/-/issues
Author: TU Wien, IFT
License: MIT
License-File: LICENSE
Keywords: compute-to-data,dataspace,hdf5,ocean protocol,pontus-x
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.10
Provides-Extra: all
Requires-Dist: h5py>=3.8; extra == 'all'
Requires-Dist: markdown>=3.4; extra == 'all'
Requires-Dist: matplotlib>=3.7; extra == 'all'
Requires-Dist: numpy>=1.24; extra == 'all'
Requires-Dist: pandas>=2.0; extra == 'all'
Requires-Dist: xhtml2pdf>=0.2.11; extra == 'all'
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: h5py>=3.8; extra == 'dev'
Requires-Dist: markdown>=3.4; extra == 'dev'
Requires-Dist: matplotlib>=3.7; extra == 'dev'
Requires-Dist: numpy>=1.24; extra == 'dev'
Requires-Dist: pandas>=2.0; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: stickytape>=0.2.1; extra == 'dev'
Requires-Dist: tables>=3.8; extra == 'dev'
Requires-Dist: twine>=6.2; extra == 'dev'
Requires-Dist: xhtml2pdf>=0.2.11; extra == 'dev'
Provides-Extra: hdf5
Requires-Dist: h5py>=3.8; extra == 'hdf5'
Requires-Dist: numpy>=1.24; extra == 'hdf5'
Provides-Extra: report
Requires-Dist: markdown>=3.4; extra == 'report'
Requires-Dist: matplotlib>=3.7; extra == 'report'
Requires-Dist: xhtml2pdf>=0.2.11; extra == 'report'
Description-Content-Type: text/markdown

# pelago

Helpers for compute-to-data algorithms on Ocean / Pontus-X: input files, consumer parameters and output files, plus
reading sections of HDF5 files and writing PDF reports. The same script runs on your machine and on the node.

## Install

```bash
pip install pelago                # Job and Param, standard library only
pip install "pelago[hdf5]"        # + H5File (h5py, numpy)
pip install "pelago[report]"      # + Report and plots (matplotlib, markdown, xhtml2pdf)
pip install "pelago[all]"         # everything, including pandas for H5File.frame()
```

Python 3.10 or newer.

## Usage

```python
from pelago import Job, Param

job = Job(
    Param.number("window", default=10000),
    Param.text("keyword", required=False),
    Param.select("mode", ["fast", "full"], default="fast"),
    Param.boolean("pdf", default=True),
)

for path in job.inputs:
    ...

job.write_json("summary.json", {"files": len(job.inputs), "window": job.params.window})
```

- `job.inputs`: all dataset files of the job, sorted by name. `job.inputs_matching("*.csv")`, `job.input("*.csv")`,
  `job.require_inputs(n)` and `job.hdf5_inputs` (HDF5 by extension or file signature) select from them.
- `job.params`: the parameter values, converted to the declared type. If the consumer did not set a value, the
  default is used. `job.params.sources` tells where each value came from.
- `job.output(name)`, `job.write_json`, `job.write_text`, `job.write_csv`, `job.save_figure`: write to the output
  folder. Paths outside it are refused.
- `job.log(msg)` writes to the job log, `job.fail(msg)` stops the job with exit code 2.
- An invalid or missing required parameter stops the job with exit code 2 and one line in the log.

Start from [examples/minimal](examples/minimal/src/main.py).

## HDF5

`H5File` reads the structure and attributes without touching the data, then only the rows and columns you ask for.
It does not assume any layout; compound tables written by PyTables or pandas (`format="table"`) work too.

```python
from pelago import H5File

with H5File(path) as h5:
    print(h5.describe())                      # tree with shapes, types and fields
    h5.tree()                                 # {path: {"kind", "shape", "dtype", "fields", "attrs"}}
    h5.attrs("signals/acceleration")          # attributes; bytes decoded, JSON strings parsed
    part = h5.read("signals/acceleration", rows=slice(0, 10_000), columns=["time", "x"])
    for chunk in h5.chunks("signals/acceleration", size=100_000, columns=["x"]):
        ...
    df = h5.frame("signals/acceleration", rows=slice(0, 1000))    # pandas DataFrame
    h5.find("*image*")                        # paths matching a glob
    h5.save("pictures/img0", job.output("img0.png"))              # bytes or string dataset to a file
```

- `read` returns numpy arrays: a structured array for several fields of a compound dataset, a plain array for one
  field. For 2-D datasets `columns` are indexes of the second axis. Strings come back as `str`.
- `save` and `raw` handle bytes, strings, uint8 arrays and opaque values; `data:...;base64,` strings are decoded.
  `index=` picks one element of an array of images.
- A wrong path raises `KeyError` with the closest existing paths.

## Reports

```python
from pelago import Report
from pelago.plots import bar, line, heatmap

(Report("Summary")
    .text("## Result\n\nAll files passed.")          # Markdown
    .params(job.params)
    .table(rows, title="Files")                       # list of dicts, rows + columns, or a DataFrame
    .figure(line(t, {"x": x, "y": y}, xlabel="time [s]"), "Signal")
    .image("photo.png", "Setup")
    .save(job.output("report.pdf")))                  # .pdf via xhtml2pdf, anything else writes HTML
```

The style is built in. Change a few variables with `Report(title, style={"font": "Times-Roman", "accent": "#b91c1c",
"page": "A4 landscape"})` (also `font_size`, `margin`, `image_width`, colours `ink`, `muted`, `rule`, `shade`), or
add CSS with `Report(title, css="h1 { color: black; }")`. `.html()` returns the whole page; figures and images are
embedded. `pelago.plots` has `bar`, `line` and `heatmap`; each returns a matplotlib figure and needs no display.

## Run locally

```bash
python main.py --inputs test-data --out outputs --param window=500 --param pdf=false
python main.py --describe        # consumerParameters for the algorithm's DDO
```

An `algoCustomData.json` in the input folder is read the same way as on the node. `--params file.json` and
`--param name=value` override it, in that order. Without options the paths come from `PELAGO_INPUTS` and
`PELAGO_OUTPUTS` (default `/data/inputs`, `/data/outputs`); `--log` or `PELAGO_LOG` set the log level.

## How Ocean passes data to the job

| | path in the container |
|---|---|
| datasets | `/data/inputs/<file name>` |
| parameters | `/data/inputs/algoCustomData.json` |
| algorithm | `/data/transformations/algorithm` |
| outputs | `/data/outputs`, returned as `outputs.tar` |
| log | stdout, returned as `algorithm.log` |

`Job` also reads the per-DID layout (`DIDS` set, files in `/data/inputs/<did>/0`); `job.inputs_by_did` groups them.
The node does not apply the parameter defaults from the DDO. If the consumer sets nothing, `algoCustomData.json` is
`{}`. Some clients send numbers and booleans as strings. `Job` handles both.

## Publish an algorithm

Bundle the script and pelago into one file and publish that file as the algorithm:

```bash
stickytape main.py --add-python-path <pelago>/src --output-file dist/algorithm.py
python dist/algorithm.py --describe > dist/consumer_parameters.json
```

Container: `ttrau/datascience-base` (pelago, numpy, h5py, matplotlib and the report libraries installed), entrypoint
`python $ALGO`. Put the output of `--describe` into the algorithm's `consumerParameters`.
[examples/multi_asset_text](examples/multi_asset_text) has a `build.sh` that bundles an algorithm and tests it
locally and in the container. [examples/hdf5_sections](examples/hdf5_sections) reads sections of HDF5 inputs.

## Development

```bash
python3 -m venv .venv && . .venv/bin/activate
make init install
make test
```

`make clean` removes build output, caches and example outputs.

## Release

The version is set in one place: `__version__` in `src/pelago/__init__.py`. Use semantic versions: raise the last
number for fixes, the middle one for new features.

1. Set `__version__`, commit and push.
2. `make release`. It stops if the version is already on PyPI, if there are uncommitted changes, or if a test fails.
   Otherwise it builds, uploads to PyPI (credentials from `~/.pypirc` or a PyPI token) and then pushes the tag
   `v<version>`.
3. Raise `pelago>=<version>` in the requirements of `datascience-base-docker` and build a new image.

`make version` prints the current version. If the upload worked but the tag push failed, push the tag by hand:
`git tag v<version> && git push origin v<version>`.

## License

MIT, see [LICENSE](LICENSE).
