Metadata-Version: 2.4
Name: lsst-rucio-register
Version: 30.2026.4700
Summary: Tools for registering LSST metadata information into Rucio
Author-email: Rubin Observatory Data Management <dm-admin@lists.lsst.org>
License-Expression: GPL-3.0-or-later
Project-URL: Homepage, https://github.com/lsst/rucio_register
Keywords: lsst
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering :: Astronomy
Requires-Python: >=3.11.0
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml>=5.1
Requires-Dist: lsst-utils
Requires-Dist: lsst-daf-butler
Requires-Dist: pydantic<3.0,>=2
Requires-Dist: rucio-clients
Dynamic: license-file

# rucio-register
Command and API to add Butler specific information to Rucio metadata.

This is a guide to using the rucio-register command for registering
Butler files with Rucio.

Butler files are expected to be located in a Rucio directory structure,
below a directory named for a Rucio scope. For example, if the root of
the Rucio directory is "/rucio/disks/xrd1/rucio" and the Rucio scope
is "test", the files should be located below "/rucio/disks/xrd1/rucio/test".


## Example

The command  "rucio-register" registers files with Rucio. This
command requires a YAML configuration file which specifies the Rucio rse and
scope, as well as the root of the directory where files are deposited,
and the external reference to the Rucio RSE. This configuration file
can be specified on the command line, or in the environment
variable **RUCIO_REGISTER_CONFIG**.

The command can register data-products or raws:

for data products:
```
rucio-register data-products --log-level INFO --repo /rucio/disks/xrd1/rucio/test --collections HSC/runs/RC2/w_2023_32/DM-40356/20230814T170253Z --dataset-type  visitSummary --rucio-dataset rubin_dataset --rucio-register-config register_config.yaml
```

for raws:
```
rucio-register raws --log-level INFO --repo /rucio/disks/xrd1/rucio/test --rucio-dataset rubin_dataset --collections LATISS/raw/all --rucio-register-config register_config.yaml \*
```
Note that for raws, this is similar to how one uses the butler command

This command looks for files registered in the butler repo "/repo/main"
using the "dataset-type" and "collections" arguments to query the butler. Note
that the repo name's suffix is the Rucio "scope". In this example, that scope
is "main".

The resulting datasets' files are registered with Rucio, as specified in
the "config.yaml" file.  Additionally, those files are registered with the
Rucio dataset specified by the "rucio-dataset" argument.

Common options across registration commands include:
- `--rucio-register-config`: Path to registration configuration file (or set via `RUCIO_REGISTER_CONFIG`).
- `--rucio-dataset`: Target Rucio dataset name.
- `--chunk-size`: Number of replica requests to make at once (default: `500`).
- `--max-workers`: Number of concurrent workers for registration in `data-products`, `raws`, `dataset-list`, and `auto-register` (default: `5`).
- `--clear-is-new`: Flag to set `is_new` metadata to `None` on Rucio dataset DIDs post-registration (supported on `data-products` and `auto-register`).
- `--backoff-factor`, `--backoff-max-value`, `--backoff-max-tries` (or `--max-retries`): Retry backoff settings (see [Retries in case of failure](#retries-in-case-of-failure)).

for zip files:
```
rucio-register zips --rucio-dataset rubin_dataset --log-level INFO --rucio-register-config /home/lsst/rucio_register/examples/register_config.yaml --zip-file file:///rucio/disks/xrd1/rucio/test/something/2c8f9e54-9757-54c0-9119-4c3ac812a2da.zip
```
Note for zip files, register a single zip file at a time.

for dimension record YAML files:
```
rucio-register dimensions --rucio-dataset rubin_dataset --log-level INFO --rucio-register-config /home/lsst/rucio_register/examples/register_config.yaml --dimension-file file:///rucio/disks/xrd1/rucio/test/something/dimensions.yaml
```
Note for dimension files, register a single dimension file at a time.


### auto-register

The `auto-register` command auto-discovers and registers datasets chronologically by quarters, dynamically partitioning massive collections into chunked Rucio datasets.

Options:
- `--repo`: Butler repository path (required).
- `--root-chain`: Collection expression / glob pattern to match (required).
- `--start-date`: Start date for query in `YYYY-MM-DD` format (default: `2000-01-01`).
- `--cutoff-date`: Cutoff date for query in `YYYY-MM-DD` format (default: current date).
- `--df-name`: Data Facility name used in generated Rucio dataset names (default: `USDF`).
- `--max-workers`: Number of concurrent workers for processing batches (default: `5`).
- `--chunk-size`: Number of replica requests to make at once per batch (default: `500`).
- `--split-size`: Sub-batch size for parallelizing large datasets into sub-chunks (default: `2000`).
- `--max-did-per-dataset`: Maximum number of items per Rucio dataset (default: `50000`).
- `--max-dataset-types`: Maximum number of discovered dataset types to process per collection, useful for testing (default: `None`).
- `--transfer-list`: Optional URL or local path to a curated YAML list of dataset types to register.
- `--dataset-name-prefix`: Optional custom prefix for auto-registered Rucio datasets (replacing default `Dataset/{collection}`).
- `--clear-is-new`: Flag to set `is_new` Rucio DID metadata to `None` post-registration.
- `--dry-run`: Flag to generate text files containing UUID lists in `--out-dir` for subsequent `dataset-list` execution instead of registering directly.
- `--out-dir`: Output directory for dry-run files (default: `./uuids`).

#### Examples:

Direct auto-registration with defaults:
```bash
rucio-register auto-register \
    --repo /repo/main \
    --root-chain "LATISS/runs/raw/all" \
    --start-date 2024-01-01 \
    --cutoff-date 2024-12-31 \
    --rucio-register-config register_config.yaml
```

Auto-registration using a curated transfer list URL and custom dataset prefix:
```bash
rucio-register auto-register \
    --repo /repo/main \
    --root-chain "HSC/runs/RC2/*" \
    --transfer-list https://github.com/lsst-dm/datasettype_transfer_lists/raw/main/cm_transfer_list.yaml \
    --dataset-name-prefix "Dataset/HSC/RC2" \
    --clear-is-new \
    --max-workers 10 \
    --rucio-register-config register_config.yaml
```

Generating dry-run UUID lists:
```bash
rucio-register auto-register \
    --repo /repo/main \
    --root-chain "LATISS/runs/raw/all" \
    --dry-run \
    --out-dir ./uuids \
    --rucio-register-config register_config.yaml
```


### dataset-list

The `dataset-list` command registers a batch of Butler datasets specified in a JSON or plain-text UUID list file (such as files generated by `auto-register --dry-run` or job rescue logs).

Options:
- `--repo`: Butler repository path (required).
- `--uuidlist`: Path to a file containing dataset UUIDs (required).
- `--rucio-dataset`: Target Rucio dataset name (optional if specified via `# Dataset/...` header in the file).
- `--max-workers`: Number of concurrent workers (default: `5`).

#### Examples:

Registering datasets from a single dry-run or rescue UUID file:
```bash
rucio-register dataset-list \
    --repo /repo/main \
    --uuidlist ./uuids/repo_main/auto_register/Dataset-LATISS-raw-all-2024Q1-00000001.txt \
    --rucio-dataset Dataset/LATISS/raw/all \
    --rucio-register-config register_config.yaml
```

Possible usage for multiple dataset-list files (batch processing dry-run outputs in parallel with `xargs`):
```bash
find uuids/ -type f -name "*.txt" | xargs -I {} -n 1 -P 5 rucio-register dataset-list \
    --repo /repo/main \
    --uuidlist {} \
    --rucio-register-config register_config.yaml \
    --log-level INFO
```


# Retries in case of failure

There are a variety of reasons that registeration to Rucio can fail.  In an effort to prevent a failure from completely halting the command, rucio-register will attempt to retry registration,
and uses exponential backoff as a retry strategy where the wait time between retries grows exponentially.

The default behavior for this is as follows:

--backoff-factor

A multiplier that scales how long the waits are. With exponential backoff, the wait before attempt n is roughly backoff-factor * (2 ** n). So factor=1 gives you waits of 1, 2, 4, 8, 16… seconds. Bump it to backoff-factor=2 and you get 2, 4, 8, 16…; drop it to backoff-factor=0.5 and you get 0.5, 1, 2, 4…. Think of it as a dial that stretches or shrinks the whole waiting curve up and down.

--backoff-max-value

A ceiling on any single wait. Exponential growth gets big fast (1, 2, 4… 512, 1024 seconds), which you usually don't want. backoff-max-value caps how long any one wait can be. With backoff-max-value=60, the waits climb 1, 2, 4, 8, 16, 32, then flatten out at 60, 60, 60, instead of continuing to double.

--backoff-max-tries / --max-retries

The total number of attempts before backoff gives up and lets the error through. It counts all tries, including the very first one.

Put together, a typical setup like backoff-factor=1, backoff-max-value=60, backoff-max-tries=5 means: retry up to 4 times, waiting 1s, then 2s, 4s, 8s — but never longer than 60s on any single wait.

Defaults are backoff-factor=5.0, backoff-max-value=30, backoff-max-tries=8

--timeout

The HTTP request timeout in seconds for Rucio client operations (`ReplicaClient` and `DIDClient` calls such as bulk replica registration and attaching files to Rucio datasets). Increase this value if large bulk registration requests (e.g. high `--chunk-size`) time out waiting for the Rucio server to respond. The default is `300.0` seconds.

## config.yaml

The config.yaml file includes information which specifies the Rucio RSE
to use, the Rucio scope, the local root of the RSE, and the URL prefix
of the location where Rucio stores the files.


```
rucio_rse: "XRD1"
scope: "main"
rse_root: "/rucio/disks/xrd1/rucio"
dtn_url: "root://xrd1:1094//rucio"
```


# export-datasets
Command and to dump Butler dataset, dimension, and calibration validity range data to a YAML file.

This command works alongside "rucio-register".
It can be used to record all the files registered into Rucio so that their transfer and ingestion at the destination can be confirmed.
In addition, it preserves dimension data and calibration validity range data that is not otherwise transferred via Rucio.
This additional data can be useful for repeated ingests of raw and calibration data into Butler repositories.

## Examples

To record the dimension values (notably _not_ including the visit dimension, which would have to be regenerated) for a set of raw images:

```
export-datasets \
    --root /sdf/group/rubin/lsstdata/offline/instrument/ \
    --filename Dataset-LSSTCam-NoTract-20250101-0000.yaml \
    --collections LSSTCam/raw/all \
    --where "instrument='LSSTCam' and day_obs=20250101 and exposure.seq_num IN (1..99)" \
    --limit 30000 \
    /repo/main raw
```
`--root` is needed here since the original files are ingested as full URLs with `direct`.

To record the datasets created by a multi-site processing workflow:

```
export-datasets \
    --filename Dataset-LSSTCam-Tract2024-Step3-Group5-metadata.yaml \
    --collections step3/group5 \
    --where "tract=2024" \
    $LOCAL_REPO '*_metadata'
```
Note the use of a glob pattern to select dataset types of interest.

