Metadata-Version: 2.4
Name: nopayloaddb-client
Version: 0.3.0
Summary: Python client for the NoPayloadDB API
Author: HEP Software Foundation (HSF) and contributors
License-Expression: BSD-3-Clause
Project-URL: Homepage, https://github.com/paullaycock/nopayloaddb-client
Project-URL: Repository, https://github.com/paullaycock/nopayloaddb-client
Project-URL: Issues, https://github.com/paullaycock/nopayloaddb-client/issues
Keywords: HEP,conditions database,NoPayloadDB,EOSC,PyHEP
Classifier: Development Status :: 5 - Production/Stable
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: THIRD_PARTY_LICENSES.md
Requires-Dist: requests>=2.28
Provides-Extra: test
Requires-Dist: mypy>=1.11; extra == "test"
Requires-Dist: pytest>=8; extra == "test"
Requires-Dist: types-requests>=2.32; extra == "test"
Dynamic: license-file

# NoPayloadDB Python client

Typed Python client for the NoPayloadDB `v5.1` metadata API and payload-file
service.

The normal use case is one experiment backed by one NoPayloadDB server. An
optional second independently managed experiment source can also be queried
when both experiments use common GlobalTag names and IOV coordinates.

The normal analysis workflow is:

1. select a common GlobalTag and analysis IOV;
2. query the experiment's payload metadata;
3. download and verify the selected payload files;
4. work with a `payload_type -> local Path` mapping.

GlobalTags correlate independently managed subsystem conditions at one shared IOV
coordinate system. In HEP-style workflows this is commonly a two-level key:

- `major_iov` for long intervals (for example ATLAS RunNumber, typically hours);
- `minor_iov` for short intervals inside the major interval (for example
  LumiBlockNumber, typically about one minute).

In production there are usually many IOV slices per payload type. Querying the
same GlobalTag at different `(major_iov, minor_iov)` points selects the
revision valid at each point.

Downloaded files use a shared checksum-addressed cache. A valid cached file is
used before the remote file service.

## Installation

```bash
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install nopayloaddb-client
```

For local development in this repository:

```bash
python -m pip install --editable ".[test]"
```

## Local test stack (optional quickstart)

For a self-contained demo, deploy two local NoPayloadDB instances (requires a
Kubernetes cluster plus `kubectl` and `helm`):

```bash
make deploy-test-server
```

| Source | Metadata API | Files |
| --- | --- | --- |
| experiment A | `http://npdb.localhost` | `http://npdb-files.localhost` |
| experiment B | `http://npdb-b.localhost` | `http://npdb-files-b.localhost` |

For local tutorials, use the **Local tutorial value** entries from the
Configuration table below.

## Configuration

Recommended pattern: set environment variables and let the client read them.

```python
from nopayloaddb_client import NoPayloadDBClient

client = NoPayloadDBClient.from_env()
```

Configuration mapping:

| Environment variable | Single constructor arg | MultiRI constructor arg | Single CLI arg | MultiRI CLI arg | Purpose | Single mode | MultiRI mode | Local tutorial value |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `NOPAYLOADDB_URL` | `base_url` | `first.base_url` | `--base-url` | `--base-url` | RI 1 metadata API | needed | needed | `http://npdb.localhost` |
| `NOPAYLOADDB_FILES_URL` | `files_url` | `first.files_url` | `--files-url` | `--files-url` | RI 1 payload-file service | needed for file commands, otherwise optional | needed for file commands, otherwise optional | `http://npdb-files.localhost` |
| `NOPAYLOADDB_SOURCE_NAME` | `source_name` | `first.source_name` | `--source-name` | `--source-name` | RI 1 source label | optional | optional | `experiment-a` |
| `NOPAYLOADDB_TOKEN` | `access_token` | `first.access_token` | `--access-token` | `--access-token` | RI 1 bearer token | needed for uploads, otherwise optional | optional | `standalone-test-token` |
| `NOPAYLOADDB_RI2_URL` | excluded | `second.base_url` | excluded | `--ri2-base-url` | RI 2 metadata API | excluded | needed | `http://npdb-b.localhost` |
| `NOPAYLOADDB_RI2_FILES_URL` | excluded | `second.files_url` | excluded | `--ri2-files-url` | RI 2 payload-file service | excluded | needed for file commands, otherwise optional | `http://npdb-files-b.localhost` |
| `NOPAYLOADDB_RI2_SOURCE_NAME` | excluded | `second.source_name` | excluded | `--ri2-source-name` | RI 2 source label | excluded | optional | `experiment-b` |
| `NOPAYLOADDB_RI2_TOKEN` | excluded | `second.access_token` | excluded | `--ri2-access-token` | RI 2 bearer token | excluded | optional | optional |
| `NOPAYLOADDB_CACHE_DIR` | `cache_dir` | `cache_dir` (`MultiRIReadClient`) | `--cache-dir` | `--cache-dir` | Verified file cache; defaults to `~/.cache/nopayloaddb` | optional | optional | optional |

Token validation is entirely server-side; the client only forwards the bearer token.

## CLI

The package installs a simple CLI binary:

```bash
nopayloaddb --help
```

Supported commands:

```bash
nopayloaddb list-global-tags
nopayloaddb get-global-tag <name> [--light]
nopayloaddb get-global-tag-payload-lists <global_tag>
nopayloaddb query-payloads <global_tag> <major_iov> <minor_iov> [--payload-type <name>]
nopayloaddb get-payload-files <global_tag> <major_iov> <minor_iov> [--payload-type <name>]
nopayloaddb-populate-test-data [--source-profile single|experiment-a|experiment-b]
```

Required configuration:

- all commands require `NOPAYLOADDB_URL` (or `--base-url`);
- `get-payload-files` also requires `NOPAYLOADDB_FILES_URL` (or `--files-url`).
- `nopayloaddb-populate-test-data` also needs `NOPAYLOADDB_TOKEN` for uploads.

If required values are missing, the CLI exits with an explicit configuration
error.

`nopayloaddb-populate-test-data` is a stable bootstrap helper for creating
tutorial content on an already-running NoPayloadDB server. Its supported public
arguments are:

```bash
nopayloaddb-populate-test-data \
  [--base-url URL] \
  [--files-url URL] \
  [--global-tag NAME] \
  [--author NAME] \
  [--source-profile single|experiment-a|experiment-b]
```

## Single-experiment analysis

Query a GlobalTag at IOV `(0, 50)` and retrieve its files:

```python
payload_metadata = client.query_payloads("AnalysisGT", 0, 50)
payload_files = client.get_payload_files("AnalysisGT", 0, 50)

beam = payload_files["BeamCalibration"]
geometry = payload_files["DetectorGeometry"]
```

Scan multiple IOV points in one run-like interval:

```python
for minor_iov in (0, 50, 100, 150, 199):
    selected = client.query_payloads("AnalysisGT", 0, minor_iov)
    print(minor_iov, [item["payload_url"] for item in selected])
```

The returned mapping is directly keyed by payload type. Files are streamed to a
temporary path, checksum-verified and moved atomically into the cache. Repeating
the request returns the cached path without contacting the file service.

## Optional multi-experiment analysis

Use `MultiRIReadClient` when the analysis needs compatible data from two
experiments. RI 1/RI 2 configuration surfaces are mapped in the table above:

```python
from nopayloaddb_client import MultiRIReadClient

client = MultiRIReadClient.from_env()
```

Query both experiments at IOV `(0, 50)`:

```python
payload_metadata = client.query_payloads("CommonAnalysisGT", 0, 50)
```

This returns a `MultiRIReadResponse`:

```python
responses = payload_metadata.by_source()
experiment_a = responses["experiment-a"]
experiment_b = responses["experiment-b"]

print(experiment_a.server_url)
print(experiment_a.unwrap())
print(experiment_b.unwrap())
```

Each `SourceResponse` contains:

- `source_name`;
- `server_url`;
- `value`, when the request succeeded;
- `error`, when that source failed;
- `unwrap()`, which returns the value or raises its source-specific error.

One unavailable experiment does not discard successful results from the other.
Responses are not merged because payload types with the same name can have
different experiment-specific meaning.

Resolve every selected payload:

```python
payload_files = client.get_payload_files("CommonAnalysisGT", 0, 50)
```

The result is grouped by source:

```python
responses = payload_files.by_source()
beam = responses["experiment-a"].unwrap()["BeamCalibration"]
geometry = responses["experiment-b"].unwrap()["DetectorGeometry"]
```

Retrieve one use case without assuming which experiment owns it:

```python
alignment_files = client.get_payload_files(
    "CommonAnalysisGT",
    0,
    50,
    payload_type="DetectorAlignment",
)
```

Sources that do not define the requested payload type return an empty mapping.

Federated CLI:

```bash
multiRI-nopayloaddb list-global-tags
multiRI-nopayloaddb get-global-tag <name> [--light]
multiRI-nopayloaddb get-global-tag-payload-lists <global_tag>
multiRI-nopayloaddb query-payloads <global_tag> <major_iov> <minor_iov> [--payload-type <name>]
multiRI-nopayloaddb get-payload-files <global_tag> <major_iov> <minor_iov> [--payload-type <name>]
```

To inspect metadata before downloading:

```python
payloads = client.query_payloads("CommonAnalysisGT", 0, 50)

for source_name, response in payloads.by_source().items():
    for payload in response.unwrap():
        print(source_name, payload["payload_type_name"], payload["payload_url"])
```

The two experiments share the checksum-addressed cache.

## Expert publication

Writes are always single-instance. `NoPayloadDBClient` never mirrors POST, PUT,
upload or DELETE requests across experiments.

An expert publishing to experiment B should construct a client with experiment
B as its first and only server:

```bash
export NOPAYLOADDB_URL=https://conditions-b.example
export NOPAYLOADDB_FILES_URL=https://files-b.example
export NOPAYLOADDB_TOKEN=your-token
```

```python
experiment_b_writer = NoPayloadDBClient()
```

Upload opaque subsystem formats independently:

```python
uploaded = experiment_b_writer.upload_payload(
    "detector-alignment-v7.xml",
    payload_url="alignment/detector-alignment-v7.xml",
)

experiment_b_writer.create_and_attach_payload_iov(
    payload_list["name"],
    payload_url=uploaded.payload_url,
    checksum=uploaded.checksum,
    major_iov=0,
    minor_iov=100,
    major_iov_end=0,
    minor_iov_end=200,
)
```

Uploads use authenticated `PUT /upload/<payload_url>`. Downloads use
`GET /dbstore/<payload_url>`.

## Local tutorials

This section assumes the local stack from
**Local test stack (optional quickstart)** is
already running and your environment variables are already exported.

### Single-server tutorial

The primary tutorial uses only experiment A and preserves the standard
single-server return types.

```bash
make populate-test-data
make tutorial
```

It creates `ClientTutorialGT` with JSON, ROOT, XML, and `DataQualitySummary`
payloads and walks through metadata lookup, IOV selection, file resolution,
cache reuse, and payload-type filtering.

### Multi-server tutorial

The additional tutorial seeds a common GlobalTag on both servers:

```bash
make populate-multi-source-test-data
make multi-source-tutorial
```

The servers receive the common GlobalTag `MultiSourceTutorialGT`, but different
experiment-owned payloads:

| Source | Payload type | Format | IOVs |
| --- | --- | --- | --- |
| experiment A | `ClientTutorialBeam` | JSON | `[0,100)`, `[100,200)` |
| experiment A | `ClientTutorialFieldMap` | ROOT | `[0,200)` |
| experiment A | `DataQualitySummary` | JSON | `[0,200)` |
| experiment B | `ClientTutorialAlignment` | XML | `[0,200)` |
| experiment B | `ClientTutorialTrigger` | YAML | `[0,200)` |

The tutorial demonstrates per-source metadata/results, source-grouped local
file resolution, cache reuse, and querying one payload type without assuming
which experiment owns it.

### DataQualityTool example

The helper `scripts/data_quality_tool.py` demonstrates event-loop usage for the
`DataQualitySummary` payload returned from `ClientTutorialGT`.

```bash
python scripts/data_quality_tool.py
```

It loads `DataQualitySummary` from the tutorial GlobalTag and calls
`assess_data_quality(event)` for 20 generated events.

### Tests

```bash
make test
make test-integration
```

The unit suite covers the original single-source return types and multi-source
attribution, failure isolation and file routing. The integration suite performs
a single-server query/download and then queries and downloads distinct payloads
from both servers using one common temporary GlobalTag.

Remove both test environments and their persistent data:

```bash
make delete-test-server
```

## Package build

```bash
python -m pip install build
python -m build
```

## Release publishing

Production PyPI publishing is tag-driven:

1. push a `v*` tag;
2. `Release checks` runs tests, type-check, license/audit checks, version-tag
   verification, and build;
3. on success, `Publish to PyPI` publishes the checked artifacts.

TestPyPI publishing remains manual via the workflow
`Publish to TestPyPI (manual)` and can target any ref.

## Third-party license inventory

The tracked `THIRD_PARTY_LICENSES.md` file is the runtime dependency license
inventory and is verified during publish CI.

```bash
make third-party-licenses
```
