Metadata-Version: 2.4
Name: beampipe-palette
Version: 0.5.2
Summary: Project-neutral DALiuGE components for Beampipe
Home-page: https://github.com/jbwod/beampipe-palette/
Author: Beampipe contributors
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: wcmatch==11.0
Provides-Extra: daliuge
Requires-Dist: daliuge-engine<7,>=6.6; extra == "daliuge"
Provides-Extra: s3
Requires-Dist: boto3<2,>=1.38; extra == "s3"
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: coverage; extra == "test"
Requires-Dist: flake8; extra == "test"
Requires-Dist: black; extra == "test"
Requires-Dist: isort; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Requires-Dist: codecov; extra == "test"
Requires-Dist: mypy; extra == "test"
Requires-Dist: gitchangelog; extra == "test"
Requires-Dist: mkdocs; extra == "test"
Requires-Dist: build; extra == "test"
Requires-Dist: wheel; extra == "test"
Requires-Dist: dlg_paletteGen==0.6.7; extra == "test"
Dynamic: author
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license-file
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

<p align="center">
  <img src="assets/brand/beampipe-terminal-logo.svg" alt="Beampipe" width="920">
</p>

<p align="center">
  <a href="https://github.com/jbwod/beampipe-palette/actions/workflows/main.yml"><img src="https://github.com/jbwod/beampipe-palette/actions/workflows/main.yml/badge.svg" alt="CI"></a>
  <a href="https://github.com/jbwod/beampipe-palette/releases/tag/v0.5.2"><img src="https://img.shields.io/badge/release-v0.5.2-d6c178?style=flat-square&labelColor=050505" alt="Release v0.5.2"></a>
  <img src="https://img.shields.io/badge/Python-3.10%2B-7fd7e6?style=flat-square&labelColor=050505" alt="Python 3.10 or newer">
  <img src="https://img.shields.io/badge/DALiuGE-6.6%20%7C%206.7-a7cfa3?style=flat-square&labelColor=050505" alt="DALiuGE 6.6 and 6.7 compatible">
</p>


> `beampipe-palette` provides the two project-neutral native [DALiuGE](https://daliuge.icrar.org/) applications that connect a portable graph to [Beampipe Core](https://github.com/jbwod/beampipe-core-v2): one injects Core's immutable execution manifest at the graph boundary, and one verifies durable outputs at the terminal boundary.


## `What it does`

> - **`Manifest ingress`**: `beampipe-ingest` receives inline JSON injected by Core, canonicalizes it, and emits it once through a path-backed FileDROP.

> - **`Durable publication`**: `beampipe-publish` independently discovers every required output, publishes each object create-only, reads it back, and verifies its SHA-256.

> - **`Trusted completion evidence`**: the publisher emits `beampipe-output-inventory/v1` to DALiuGE and writes the byte-identical receipt to an attempt-scoped session handoff.

> - **`Callback-free remote runs`**: Core retrieves the receipt over its existing SSH/SFTP control connection after Slurm completion. Graphs contain no Core URL, bearer token, storage credential, or project-specific topology.


## `Native DALiuGE components`

<picture>
  <img src="assets/readme/palette-components-terminal.png" alt="Terminal-style beampipe-ingest and beampipe-publish DALiuGE application nodes with their manifest, completion, and inventory ports" />
</picture>

`beampipe-ingest` and `beampipe-publish` are native `BarrierAppDROP`
applications—not PyFunc snippets. Their graph-facing contracts stay deliberately
small; Core and the selected deployment profile supply execution details outside
the logical graph.

### Manifest entering `beampipe-ingest`

```json
{
  "inputs": {},
  "sources": [{
    "source_identifier": "source-1",
    "groups": [{
      "group_key": "group-1",
      "records": [{
        "record_id": "record-1",
        "access_url": "https://data.example/record-1"
      }]
    }]
  }],
  "_beampipe": {
    "selection": [{
      "source_identifier": "source-1",
      "group_key": "group-1",
      "record_ids": ["record-1"]
    }]
  }
}
```

### Inventory emitted by `beampipe-publish`

```json
{
  "schema": "beampipe-output-inventory/v1",
  "execution_id":
    "607fd31f-db2a-4978-a64f-ef8ce412483a",
  "execution_attempt": 0,
  "patterns": ["**/result.bin"],
  "pattern_counts": {"**/result.bin": 1},
  "products": [{
    "path": "products/result.bin",
    "bytes": 1048576,
    "sha256":
      "a40f3e3d82fd741d64da75530ccb205588b8fbb5a5fbd7d52f18a77ef85d5e60"
  }],
  "inventory_sha256":
    "5efa576ea4393a413c121862a75efcb5086cffa4b651ec77178dfa1c8caa07ce",
  "durable_destination_uri":
    "file:///srv/products/executions/607fd31f-db2a-4978-a64f-ef8ce412483a/attempt-0",
  "publication": {
    "acknowledged": true,
    "publisher": "beampipe-publish",
    "receipt_id":
      "beampipe-publish/v1:5efa576ea4393a413c121862a75efcb5086cffa4b651ec77178dfa1c8caa07ce",
    "published_at": "2026-08-27T10:32:16Z"
  }
}
```

The examples show the stable project-neutral fields. Real manifests may add
project-owned record metadata, while real inventories contain one entry for
every selected product and the exact hashes observed after durable read-back.


## `Where the nodes go`

<picture>
  <img src="assets/readme/palette-placement-terminal.png" alt="Terminal-style DALiuGE graph flow showing Core manifest into beampipe-ingest, the project graph and completion barrier, then beampipe-publish and the inventory receipt retrieved by Core" />
</picture>

> - Put **`beampipe-ingest` at the graph boundary**. Core patches its
>   `manifest_path` setting with the immutable execution manifest. Connect its
>   single `manifest_bytes` FileDROP to the first project application that needs
>   those bytes.

> - Put **`beampipe-publish after the project-owned completion barrier`**. The
>   barrier must become complete only after every required output is closed and
>   stable. Connect the publisher's `inventory` output to one terminal
>   FileDROP.

> - Do not draw a callback edge to Core. On remote Slurm deployments the
>   publisher writes the canonical handoff inside the session; Core pulls and
>   verifies it after the scheduler reports completion.


## `First-time setup`

Python 3.10 or newer is required. Install the validated public release into the
same Python environment used by every DALiuGE execution node:

```bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install \
  https://github.com/jbwod/beampipe-palette/releases/download/v0.5.2/beampipe_palette-0.5.2-py3-none-any.whl
beampipe-publish --version
```

The released wheel is normally installed into an existing DALiuGE runtime. A
source checkout can create a self-contained DALiuGE 6.x environment with
`python -m pip install -e '.[daliuge]'`.

Install the bounded S3 transport only on runtimes that select an S3
destination:

```bash
python -m pip install \
  'beampipe-palette[s3] @ https://github.com/jbwod/beampipe-palette/archive/refs/tags/v0.5.2.tar.gz'
```

Download or load the checked-in EAGLE palette:

```text
https://raw.githubusercontent.com/jbwod/beampipe-palette/v0.5.2/daliuge/palettes/beampipe.palette
```


## `EAGLE graph contract`

Load [`daliuge/palettes/beampipe.palette`](daliuge/palettes/beampipe.palette)
in [EAGLE](https://eagle.icrar.org/). The wheel also installs it at
`share/beampipe-palette/daliuge/palettes/beampipe.palette` under the active
Python prefix.

| Component | Field | EAGLE usage | Contract |
|---|---|---|---|
| `beampipe-ingest` | `manifest_path` | NoPort | inline JSON from Core or an absolute manifest path |
| `beampipe-ingest` | `manifest_bytes` | OutputPort | one path-backed FileDROP, pickle encoding |
| `beampipe-publish` | `completion` | InputPort | one project-owned completion FileDROP |
| `beampipe-publish` | `inventory` | OutputPort | one path-backed canonical inventory FileDROP |
| `beampipe-publish` | `expected_patterns_json` | NoPort | project-owned JSON string array of required output globs |

The Python classes are:

```text
beampipe_palette.apps.BeampipeIngestApp
beampipe_palette.apps.BeampipePublishApp
```

The ingest output intentionally uses DALiuGE's `pickle` encoding. It wraps the
canonical bytes with DALiuGE's own serialized-data envelope before the single
pickle port write. This is the common named-port contract that returns exact
`bytes` on both DALiuGE 6.6 and 6.7. The publisher uses path-backed FileDROPs
directly and performs no second DROP write.

Projects must set `expected_patterns_json` explicitly. Matching is bounded and
Core-compatible: `*` and `?` stay within one path component, while `**` is
allowed only as a complete component. Character classes, brace expansion,
embedded double-stars, traversal, backslashes, controls, and duplicates are
rejected. The output tree is scanned once regardless of pattern count.


## `Runtime contract`

Runtime details are environment variables, never graph fields:

| Environment variable | Meaning |
|---|---|
| `BEAMPIPE_OUTPUT_ROOT` | source directory containing completed outputs |
| `BEAMPIPE_OUTPUT_DESTINATION_URI` | base `file://` or `s3://` destination |
| `BEAMPIPE_EXECUTION_ID` | canonical execution UUID |
| `BEAMPIPE_EXECUTION_ATTEMPT` | zero-based locked Core `retry_count` |
| `BEAMPIPE_OUTPUT_INVENTORY_HANDOFF_PATH` | required attempt-scoped receipt path prepared by the session transport |

The durable destination is always namespaced as:

```text
<base>/executions/<execution_uuid>/attempt-<retry_count>
```

The handoff path must be absolute, non-root, outside `BEAMPIPE_OUTPUT_ROOT`,
free of symbolic-link components, and have an existing private parent. No Core
URL, Core credential, storage secret, project identifier, or output pattern is
embedded in the component.


## `Publication guarantees`

> - **`Create-only writes`**: files, inventories, and handoffs are never
>   overwritten. An existing byte-identical object is reused; conflicting
>   evidence fails the run.

> - **`Read-back verification`**: every durable file or S3 object is retrieved
>   in full and hashed before it is acknowledged.

> - **`Exact retries`**: a retry of the same execution attempt reuses the same
>   canonical inventory bytes and publication timestamp. A later attempt has a
>   separate immutable namespace.

> - **`Fail-closed evidence`**: missing patterns, empty outputs, symlinks,
>   unsafe paths, oversized receipts, changed files, partial uploads, and
>   mismatched prior evidence all fail publication.

Filesystem destinations use absolute, non-overlapping `file://` roots. S3
destinations require `beampipe-palette[s3]`, boto3 1.38 or newer, conditional
single-object/multipart creation, and full object read-back. NGAS is not
claimed until an adapter can provide authoritative file ID/version assignment
and retrieval verification.


## `Development`

The [DALiuGE 6.7 Python component guide](https://daliuge.readthedocs.io/en/v6.7.0/development/app_development/python_components.html)
uses `dlg_paletteGen` with inline Doxygen metadata. Install the two system
tools once on Debian/Ubuntu:

```bash
sudo apt-get install doxygen xsltproc
```

```bash
make virtualenv
make lint
make test
make palette-check
make docs
make build
```

Palette generation reads the `EAGLE_START` component descriptions in
`beampipe_palette/apps.py` through pinned `dlg_paletteGen` 0.6.7, then
canonicalizes IDs and timestamps so the checked EAGLE palette is deterministic.

The offline suite covers both native applications, canonical JSON, path and
port policy, one-pass discovery, filesystem/S3 create-only publication,
read-back hashes, exact retries, attempt isolation, session handoff, palette
signature drift, and project-neutral defaults. CI additionally runs the native
DALiuGE boundary regression against the released 6.6 engine and the pinned 6.7
source contract used by the linked 6.7 developer documentation.


## `Documentation`

| Task | Page |
|---|---|
| Download the current wheel and source archive | [v0.5.2 release](https://github.com/jbwod/beampipe-palette/releases/tag/v0.5.2) |
| Review the publication sequence | [Publisher guide](docs/index.md) |
| Import the components into EAGLE | [Checked-in palette](daliuge/palettes/beampipe.palette) |
| Contribute and run the local gates | [Contributing guide](CONTRIBUTING.md) |
| Review template provenance | [Template notes](ABOUT_THIS_TEMPLATE.md) |

This repository was created from ICRAR's
[`daliuge-component-template`](https://github.com/ICRAR/daliuge-component-template)
and retains its [license](LICENSE).
