DPPS Milestones and Tasks

1. Notes:
2. ACTIONS
3. DPPS AIV Use Cases/Releases/Milestones

1. Notes:

1.1. BDMS issues

Context Metadata validation is NOT responsibility of BDMS
- except for metadata that is archive-specific
- except for basic checks like “duplicated entry”, “completeness”
Metadata schema is also NOT responsibility of BDMS
- core schema comes from Data Model (Reference)
DECISION: for BDMS-Workload plugin, go for the alternate implementation where Rucio is used for both storage types (ephemeral vs archived), but with two different policies. This corresponds to the right diagram in the presentation by Karl.
- Reduced need to have two DMS systems and databases operating at the same time
- Same technology used for both types of storage, reduced maintenance.
- Can to start with even use only one policy, if having two is complicated.
- Need a working prototypes.
Metadata Catalog
- Current prototype uses MongoDB. Could we use PostGREsql with a JSON column instead, just to reduce number of DB types? Needs benchmarking verification test.

1.2. Release 0.x

Goal for integration: a docker container we can run locally with all of DPPS installed.
Currently DESY’s Kubernetes runner doesn’t support Docker-in-Docker, e.g. to run the DPPS container inside a CI job. However, we could simply create our own runner for those jobs.

1.2.1. Rel 0.1

1.3. How we will deploy in the future

want common way for all data centers, everything containerized
central monitoring portal/dashboard (part of Ops)
requirement to be able to stop/restart
requirement to be able to run multiple instances for upgrade testing
requirement to be able to fall back to old version in case of upgrade failure
would like something that is tested in the same way as real deployment eventually.
Blocker now is that DIRAC isn’t fully containerized, but that is coming soon. Already working mostly in the Dirac-9.0 pre-release (transition between DIRAC 8 and DIRAC-X)

1.4. Deliverables and Releases

We are not responsible for actual deployment of the software, only to define the procedure. Deployment to on- and off-site ICT should be done by a SDMC staff.

1.5. How to make a release

Need release manager (temporary for r0)

Plan:

Create relase milestone on GitLab
All issues associated with that milestone must be solved, or rescoped.
Test reports from each subsystem
- what UCs have been tested
- what requirements have been verified
Test reports from AIV
Tagged software releases
Documentation Package (if not in tagged software release)
the AIV repo should be the central point to release DPPS docker containers. It contains a list of which versions belong to this release.
SBOM (Software bill of materials)

Pre-release Documentation:

DPPS Release plan (defines deliverables and milestones)
Rx.x Detailed Release Plan
V&V plan

Release Deliverables

Requirements Specs
Requirements V&V
Set of Tested use cases
AIV report
- QA reports (sonarqube)
- Use Case Test reports
DPPS user and developer documentation package
List of released code packages and versions (machine-readable)
Links to tagged source code in GitLab

1.6. Blocking Points and risks:

1.6.1. Quality Plan from CTAO

1.6.2. Event type definitions / cases (if needed for SDC)

1.7. Major issue:

This is a very ambitious schedule
requires everyone to work a high amount
our teams have not yet even signed an MoU, and are still in the formation phase in some cases
- Unclear how much manpower we have currently, only on a 5-year scale
- BDMS team in particulary, but also others, even DataPipe.

2. ACTIONS

2.1. TODO Ask DESY for CI runner or VM that allows docker-in-docker (Frederic)

2.2. TODO Look into deploying singularity images on CVMFS (there is a docker plugin)

2.3. TODO Start Ops ReqSpec (Nektar)

based on what we are developing for AIV + whatever extra is needed

2.4. TODO Consider common software core for Benchmarking/Release Verification/Quality Monitoring

All are comparing some baseline to measurements and then putting a cut on how far away is acceptable.

See Tomas’s list of requirements
core for plotting/histogram I/O, etc.

2.5. TODO Ask Roberta about new milestone: DPPS provides public performance plots

2.6. TODO (karl) Update R0.1 UC MR

2.7. TODO create R0.0 UC

2.8. TODO (Frederic) Generate minutes from the PIC demo to help with merging BDMS

https://indico.cta-observatory.org/event/5383/

2.9. TODO determine DPPS database architecture

who runs the pipelines database servers? Should be in Ops, but delegated to data centers? Or do we put them into SOSS (Felix Stoehr’s suggestion)?
From a Computing architecture standpoint, the same question should be applied to SOSS
What about the databases needed for underlying technologies like Rucio/DIRAC/FTS/…?
Are there two classes of database server: ones where the tables are critical and have to be backed up, and one where failures can be tolerated?
For our databases where we control the schemas: which technology? -> Try to use PostGRE for all
Which technologies are needed by the underlying services? Mysql, mongodb, …

3. DPPS AIV Use Cases/Releases/Milestones

SM: Science Milestone RX.Y: DPPS release

3.1. R0.0: Systems Integration Test on Test Cluster BDMS Workload

3.1.1. Task: Merge prototypes (there can only be one BDMS in the release)

Pick features of each to incorporate into released package
Create repos in GitLab:
- DPPS/BDMS/bdms-client: contains the code for ingest/retrieve/etc wrappers and rucio + related (fts, gfal, xrootd) client setups
- DPPS/BDMS/bdms-server: contains BDMS Dockerfiles based on Rucio and FTS
Move any code in the prototypes (that we want to retain) into these repos.
Upgrade client software to use python project template:
- should be pip-installable
- have a container built by the CI,
- have documentation (even minimal) uploaded to gitlab pages (following python project template)
This is the repo where the release will be tagged and from which the container is generated.
In the test setup we should have storage elements using the same versions as in the real data centers (e.g. dCache and xrootd).

3.1.2. Task: Set up CI environment

Set up namespaces for CI for manual and automated tests (no more “Swiss” vs “Italian”)
Ensure any manual setup is in the YAML file and is therefore reproducible
check required ports

3.1.3. Ingest data products (without metadata) BDMS

3.1.4. Retrieve data products by LFN BDMS

check that checksum matches original file

3.1.5. Run a “hello world” job in DIRAC (JDL) Workload

no data access, no Transformation or Production. Simple JDL job
check for job completeness
tests configuration and execution of a single-job workflow

3.2. BDMS Ingest and Retrieve functionality integrated with WMS BDMS Workload

3.2.1. Run a job that needs input from BDMS (by LFN) and outputs to BDMS

check that the job completes and the output is stored in BDMS correctly.

3.3. R0.1 Integrate Pipelines with Workload BDMS Workload DataPipe CalibPipe SimPipe

3.3.1. Install Pipeline Containers on Test Cluster

Build Docker images and convert to Singularity/Apptainer
Base image should have all the core dependencies
Deploy singularity containers to CVMFS

3.3.2. Deploy configuration files to CVMFS

3.3.3. Run test cases locally (not with WMS):

CalibPipe: run CWL workflow for fetching contemporary MDP
DataPipe: fetch a test simulation file from BDMS and run CWL workflow on WMS to process it to DL2a
SimPipe: test and document simulation config model.
Integration: Validate MDP produced by CalibPipe with SimPipe

3.3.4. Run tests on WMS

3.4. Run Workflows on Test cluster

3.5. Test replication rules (within test cluster)

3.6. Run a pipeline workflow Workload BDMS DataPipe CalibPipe

3.7. PDR ALL Management

3.8. Ingest legacy simulation data into BDMS (prod6 only) BDMS

need metadata catalog

3.9. Integrate DPPS with Off-site ICT (1+ DCs) ALL

3.9.1. Monitor DPPS operation Ops

3.9.2. Integrate with AAI

3.9.3. Install BDMS client

3.9.4. Install Workload client

3.9.5. Verify ingestion/retrieval/query

3.9.6. Verify replication

3.9.7. Verify data processing (run workflows)

3.10. Run Workflows on Off-site ICT

3.11. R1.0: IRF Maker (analyze Prod6) BDMS Workload DataPipe

3.11.1. Test: Process Prod6 into IRFs

on test cluster (small sample test)
or off-site ICT for full verification? Needed for public IRFs
or on existing DIRAC instance… (not a DPPS milstone)

3.12. SM: Produce or replicate Public IRFs ALL

3.13. Test user rights management BDMS AAI

3.13.1. Check data rights:

visibility (is this needed?)
read access
write access
delete access
create datasets
modify datasets

3.13.2. Check rights for workflows

define new workflows
configure existing workflows
remove a workflow
execute a workflow

3.14. Finalize subsystem architecture ALL

3.15. Integrate LIDAR Input DataPipe CalibPipe

3.16. Integrate FRAM Input DataPipe CalibPipe

3.17. Process and apply Pointing Corrections DataPipe CalibPipe

3.18. Integrate DPPS with On-site ICT and ACADA BDMS Workload Ops ACADA

3.18.1. Implement ACADA-DPPS interface BDMS DataPipe

3.18.2. Install BDMS client

3.18.3. Install Workload client

3.18.4. Install DPPS ACADA Interface

3.18.5. Verify ingestion from ACADA

3.18.6. Verify replication to off-site (if offsite already integrated)

3.18.7. Verify data processing (run workflows)

3.19. Store LST-1+ACADA data BDMS

No metadata, LFN only, replicate to one offsite data center

3.20. Run Workflows on On-site ICT

3.21. R2.0: Mono-Telescope Processing Workload BDMS

3.22. CDR ALL

3.23. DL1 Calibration (LST) DataPipe CalibPipe

3.24. SM: LST-1 Commissioning

3.25. SM: Open Science Data Challenge

3.26. DL1 Calibration (MST) DataPipe CalibPipe

3.27. Generate Data Quality Reports DataPipe CalibPipe QualPipe

3.28. Preserve data (full replication policy) BDMS

3.29. Use Separate temporary and archive storage policies BDMS

3.30. R3.0: Stereo-Telescope Processing ALL

3.31. Implement DPPS-SUSS interface (DL3) BDMS DataPipe

3.32. SM: Northern 2-3 Telescope Commissioning and public data

LST1 + LST4 + [MST3]