# CTAO Workload Management System

This project provides Helm chart to deploy WMS with all necessary dependencies and services.



## Chart

The WMS Helm chart aims to provide a complete deployment of DIRAC with all necessary dependencies. It contains:
- **DIRAC Master CS**
- **DIRAC services and agents**
- **DIRAC Web App**
- **DIRAC Databases**
- **Authentication services**
- **Bootstrap jobs**
- **External dependencies**

The WMS chart is integrated in the global [DPPS deployment](https://gitlab.cta-observatory.org/cta-computing/dpps/dpps).



### Chart Dependencies

The chart includes the following external dependencies:
- `diracx`: complete DiracX deployment
- `mariadb`: databases for DIRAC and DiracX data storage

For testing, several components are also included as subcharts,
but are not needed are expected to be provided as external components integrated production:
- `gcert-issuer`: a CTAO grid certificate issuer for testing deployments
- `cvmfs`: CVMFS stratum-0 deployment for testing
- `indigo-iam`: IAM deployment provided by CTAO for testing

### Components description

**DIRAC Master CS**

The DIRAC Master Configuration Server deployment consists of:
- **statefulset** (`master-cs/statefulset.yaml`): to ensure the uniqueness of the deployment
- **secret** (`master-cs/secret.yaml`): contains specific DIRAC configuration which shouldn't be exposed in the remote configuration

**DIRAC components**

All DIRAC components (services, agents and executors) deployment:
- **deployments** (`deployment.yaml`): iteratively generated deployment and service for each enabled DIRAC component
- **configuration files** (`dirac-configuration-configmap.yaml`): a configmap containing DIRAC configuration files, separate into DIRAC sections (Operations, Resources, Registry, ...). Each installed component as its own specific `Systems` section configuration
- **environment configuration** (`dirac-env-configmap.yaml`): environment specific configuration

**DIRAC Web App**

A deployment of the DIRAC Web App:
- **deployment** (`web-app/deployment.yaml`)
- **configuration** (`web-app/configmap.yaml`): web app specific configuration

**Computing Element**

A simple SSH computing element:
- **deployment** (`computing-element/deployment.yaml`): a ssh server acting as a CE

**Ingress**

TODO.


### Bootstrap sequence diagram

The bootstrap sequence is composed of several Jobs to make sure DIRAC is correctly configured and running.

1. **DIRAC configuration initialization** (`bootstraps/configure-dirac-job.yaml`)

Apply the DIRAC configuration contained in the configmap to the Master CS. This job runs once `MariaDB` and `MasteCS` statefulsets are running.

2. **DIRAC DB initialization** (`bootstraps/init-dirac-db-job.yaml`)

Install the DIRAC DB and the DiracX AuthDB. The Job runs once the `ComponentMonitoring` service is running.

3. **Synchronize DIRAC configuration** (`bootstraps/sync-dirac-job.yaml`)

Synchronize RSS configuration, then synchronize DIRAC and DiracX configuration and finally initialize/synchronize IAM users. This job runs after Databases initialization and when DiracX is running.

4. **First Proxy initialization** (`bootstraps/first-proxy-job.yaml`)

Initialize DIRAC proxies, at installation is necessary to initialize a first time the pilot proxy. This job runs once the `ProxyManager`, `BundleDelivery` and `SystemAdministrato` services are running.

```mermaid
---
title: WMS Bootstrap sequence
---
flowchart TB
    subgraph mariadb ["MariaDB Statefulset"]
        direction LR
        mdb["dirac-db"]
    end
    subgraph mastercs ["Master CS statefulset"]
        direction LR
        mcs["dirac-master-cs"]
    end
    subgraph bootdiracconf ["Bootstrap DIRAC Conf Job"]
        direction LR
        confdirac("bootstrap-dirac-configure")
    end
    subgraph compmon ["Component Monitoring deployment"]
        direction LR
        bootcomp["bootstrap-component-monitoring"]-->compmondep["component-monitoring"]
    end
    subgraph bootdb ["Bootstrap DIRAC DB Job"]
        direction LR
        authdb("install-diracx-auth-db")-->compdb("install-dirac-db")
    end
    subgraph bootcnf ["Sync DIRAC Conf Job"]
        direction LR
        syncrss("bootstrap-dirac-sync-rss")-->syncdiracxcs("sync-diracx-cs")
        syncdiracxcs-->synciamusers("sync-iam-users")
    end
    subgraph diracx ["DiracX deployment"]
        direction LR
        dx("DiracX")
    end
    subgraph dcomp ["DIRAC Components deployment"]
        direction TB
        proxyman["dirac-proxy-manager"]
        bundledeliv["dirac-bundle-delivery"]
        systemadmin["dirac-system-admin"]
        rest["..."]
    end
    subgraph fproxy ["First proxy init Job"]
        direction LR
        initproxy("first-proxy")
    end
    mariadb --wait for--> mastercs
    mastercs --wait for--> bootdiracconf
    bootdiracconf --wait for--> compmon
    compmon --wait for--> bootdb
    bootdb --wait for--> bootcnf
    bootcnf --wait for--> dcomp
    syncdiracxcs --needs--> diracx
    diracx --wait for--> fproxy
    proxyman --wait for--> fproxy
    bundledeliv --wait for--> fproxy
    systemadmin --wait for--> fproxy

```

### Storage Volumes

- **CS backup** (`master-cs/statefulset.yaml`): persistent volume to keep CS backup while restarting the Master CS statefulset
- **Storage Element Data** (`pvc.yaml`): persistent storage for data stored in the test SE

### Configuration Structure

**DIRAC server configuration**

All the configuration related to DIRAC server is contained in the `values.yaml` under `diracServer` section. This section will allow you to configure `master-cs`, `web-app`, `components` and `configurations`.

- **Master CS** `.diracServer.masterCS`
- **WebApp** `.diracServer.webApp`
- **DIRAC components** `.diracServer.diracComponents`: in this section each components can be enabled or disabled, configured (port, specific configuration), specify the replica counts. For each componennt a default configuration is provided but can be overwritten.

Example:
```yaml
diracServer:
    diracComponents:
        componentMonitoring:
            port: 9190 # the port on which contact the service
            cmd: Framework/ComponentMonitoring # the command run by `dirac-service`
            replicaCount: 1
            config: | # to overwrite the default configuration
                Systems
                {
                    Framework
                    {
                        Services
                        {
                            ComponentMonitoring
                            {
                                ...
                            }
                        }
                    }
                }
```
**Note:** the name of the subsection (e.g. `componentMonitoring`) is used to generate deployment, service and associated configmap names.

- **bootstrap**: bootstrap jobs can be individually disabled or enabled, and configured under `diracServer.bootstrap`

- **DIRAC databases**: the list of the DIRAC databases to be installed must be provided under `diracServer.diracDatabases`

- **DIRAC configuration**: To set specific DIRAC configuration there is `diracServer.diracConfig`. Since the full DIRAC configuration is very long, the configurability of it is separated into sections: `Systems`, `Registry`, `Resources`, `Operations`, `DiracX`. For each a file in a configmap is generated which is then apply at bootstrap.

One can write a section using the yaml format, which will be automatically translated, or using a string template.
One can decide to provide its own full DIRAC configuration by overwriting the configmap itself using `diracServer.diracConfig.configmap.create: false` and `diracServer.diracConfig.configmap.name: config-map-name`



## Test Report

The WMS test report can be retrieve from the CI pipeline artifact.

## Deploy WMS locally

The first time you want to run the local cluster:
```bash
pixi run install-tools
```

To start a `Kind` local cluster, deploy the chart in it, and run the test, do:

```bash
pixi run make -e dev
```
