## Background and motivation

DPPS subsystems are currently required ([citation?]()) to provide tested software and docker containers.
DPPS AIV ensures that these containers can be deployed in the Test Cluster (a Kubernetes cluster at DESY).

However, in addition to docker containers, subsystem deployment requires knowledge of the way the subsystem components should be:

* bootstrapped and configured
* connected to each other
* connected to the external services
* connected to external and on-demand provisioned storage
* monitored for liveness, readiness, health, and performance

This knowledge is principally held by the subsystem developers, who are responsible for the subsystems' design and implementation, and not by the AIV team.

Right now, subsystem developers are typically using **docker compose** to define the subsystems for testing and local development, providing a simple and convenient way to facilitate these tasks.

During AIV deployment, the subsystems are deployed using **Helm charts** which allows to define all the aforementioned aspects of the subsystem in a declarative way. The knowledge of how to do this is required from the subsystem developers, while the AIV team has the knowledge on the necessary tools and support, which is why the AIV team needs to work closely with the subsystem developers on production of the Helm charts. At this time, it remains to be determined if subsystem Helm charts will be delivered and maintained by the AIV team or by the subsystem developers.


The Helm charts are a more powerful and flexible way to define all aspects of subsystem component arrangement.
Crucially, since the AIV deployment **will** use the Helm charts to deploy the subsystems in the Test Cluster and later in On-Site (?) and Off-Site ICT, maintaining also docker compose files introduces a duplication, and risk of divergence between the two definitions, which can lead to deployment issues.

The goal of this repository is to explain how we can adopt Helm Charts for local development and gitlab CI/CD testing, avoiding any dependency on docker compose. To achieve this, we will provide tools and templates that will allow subsystem developers to develop describe desired subsystems in Helm charts, with ease, convenience, and consistency.

### Disadvantages of helm charts

More composable than docker compose, but also more complex. With suitable tools, this complexity can be managed very well.


## Subsystem Helm charts

### Writing Helm charts

TODO: provide a CTAO template
```bash
# Create a new helm chart
helm create my-subsystem
```

### Where to put the Helm charts

TODO: to be discussed
Two options are available:
* Put the Helm charts in the same repository as the subsystem code and deliver them together, including the tests.
* Put the Helm charts in a separate repository, and deliver them separately from the subsystem code easier to separate the concerns of different subsystems. Likely in a monorepo with all the subsystems, allowing to more easily manage the dependencies between the subsystems.

In both cases, the Helm charts should be versioned and tagged, and published in a Helm repository (in Harbor?).


### Testing Helm charts locally

TODO: add a web script to test the Helm charts locally

```bash
# Create a cluster
kind create cluster

# Install the helm chart
helm install my-subsystem ./my-subsystem

# Check the pods
kubectl get pods
```

### Testing Helm charts in Gitlab CI/CD

See the `.gitlab-ci.yml` file in this repository for an example of how to test Helm charts in Gitlab CI/CD.

The tests are defined as Helm Tests, i.e. they are pods that are run in the same cluster as the Helm chart is deployed. The tests can be defined in the `templates/tests` directory of the Helm chart.

## AIV deployment and Helm charts matching Level B Use Cases

When the Helm charts are ready, the AIV team will deploy the subsystems in the Test Cluster. AIV will decide which of the two ways to deploy multiple subsystems in the Test Cluster to use:
* single high-level Helm chart that deploys all subsystems as subcharts
* multiple separate Helm charts, organized with gitops tools (ArgoCD or FluxCD, to decide)


## Certificates

TODO: make cert-manager producing self-signed certificates https://cert-manager.io/docs/configuration/selfsigned/

## Assumptions on Kubernetes cluster

TODO: describe the assumptions on the Kubernetes cluster where the Helm charts will be deployed

* Ingress controller
* Cert-manager
* Harbor
* Prometheus
* storage classes


## git submodules vs git subtree

Here we use git subtree to include the Helm charts in the subsystem repository. This allows to diverge the Helm charts from the main repository, while still keeping the history of the Helm charts in the main repository. This is useful for the AIV team to understand the changes in the Helm charts, and to be able to revert them if necessary.


TODO: may be advisable to split subsystem in two charts, one clean for deployment, and one for development with all the tools and dependencies


## Reusable CI components

```bash
glab ci lint
glab ci config compile
```


### Generating certificates

Does it make sense that the certificate generation is in DPPS repo? This is not included in release code for subsystem testing.

# Dependencies between jobs and components

through mounting secrets, configmaps generated by other jobs, etc.
by waiting for the readiness of other components
by using init containers

TODO: try https://git.km3net.de/rucio/rucio-deployment/

TODO: multi-DC DB?

TODO: failover

# Important practices for building helm charts

First time, use `helm create`.

Use `helm lint` to check the chart.

Use `helm template` to see the generated Kubernetes resources.


Subsystem chart should be explicit about interfaces with other subsystems, and should not make implicit assumptions about the environment.


TODO: Built-in monitoring and dump after test


The helm chart should contain skaffolding, fixtures, for the dependency services.

Dependencies between resources deployed by the chart can be implemented as ....
