Metadata-Version: 2.4
Name: splent_feature_autodeploy
Version: 0.4.0
License-Expression: LGPL-3.0-or-later
Requires-Python: >=3.13
Description-Content-Type: text/markdown
License-File: COPYING
License-File: COPYING.LESSER
Requires-Dist: splent_framework>=1.20.0
Dynamic: license-file

# splent_feature_autodeploy

Release-gated continuous delivery for a SPLENT product. A product that
selects this feature runs a small agent in a container of its own next to
its web container in production. The agent follows a registry image
channel, for example `git.splent.io/splent-io/<product>:stable`, and when
the channel points to a new image it redeploys the product on that image's
digest. It backs up first (off the host too, when the product has
`splent_feature_backup`), migrates once, waits for the new version's
`/readyz`, and rolls the container back, and the database when the deploy
changed it, when the new version is not ready.

The channel only moves when someone runs `splent product:release`, so nothing
reaches production that nobody released. The agent never builds anything and
never follows a branch.

## How a check goes

Every `AUTODEPLOY_INTERVAL` seconds, and whenever an administrator asks,
the agent goes through these steps.

1. Pull the channel image. If its id is the one the web container runs,
   there is nothing to do. Otherwise the image is named from now on by its
   digest (`<registry>/<org>/<product>@sha256:...`), which no push can
   move: that is what the new container runs and what the history records.
   An image the registry knows no digest of runs by its id, just as fixed.
2. Skip an image that failed before, until the channel points elsewhere.
   Honour the pause flag and the update window (`AUTODEPLOY_WINDOW`). A
   check somebody asks for (Check now) deploys outside the window too.
3. Back up.
   - When the product has `splent_feature_backup`, run its off-host backup
     in the running web container, as the app's user:
     `splent feature:backup run --tag pre-deploy-<short id of the image
     being replaced>` (`AUTODEPLOY_BACKUP_COMMAND`). It is encrypted to keys
     the server does not hold, and it is the copy that survives the host.
   - Then the agent's own dump, the one a rollback restores:
     `mariadb-dump --single-transaction --routines --triggers` runs as root
     inside the database container, with the credentials that container
     already holds, and the gzipped dump is kept only if `mariadb-dump`
     exited cleanly and the dump ends with its `Dump completed` trailer.

   No complete backup, no deploy.
4. Record what the web container runs (`splent version --json`), and a
   fingerprint of the schema: every table's columns and every migration
   revision (`alembic_*`, `splent_migrations`).
5. Create the new web container on the image's digest with everything the
   old one was configured with, the way Watchtower does it. That is its
   name, its environment (the container's own values on top of the new
   image's), its labels, mounts and binds, port bindings, restart policy,
   healthcheck, and its networks with their aliases. Cmd, Entrypoint and
   WorkingDir follow the new image when the old container only inherited
   them, and are kept when it chose them (the compose file's `command:`).
   The old container is renamed `<name>-previous` and stopped, not removed.
6. Migrate, once: a one-off container of the new image, `<name>-migrate`,
   with the web container's environment, mounts and networks, runs
   `AUTODEPLOY_MIGRATE_COMMAND` (`splent db:upgrade`) as the app's user
   while neither version serves. It answers to no alias, so nginx never
   finds it, publishes no port and is removed once it exits. The new
   container's entrypoint upgrades again as it starts and finds nothing
   left to do.
7. Start the new container and wait, up to `AUTODEPLOY_HEALTH_TIMEOUT`
   seconds, for `/readyz` to answer 200 (`AUTODEPLOY_READY_PATH`, asked by
   the container's name on `AUTODEPLOY_WEB_PORT`), while Docker's
   healthcheck does not say unhealthy. `/readyz` is the framework's
   (1.22.0 and later): 200 when the database and the services the product
   declares answer. An image without it answers 404, and the healthcheck
   decides for it.
8. Once it is ready, remove the previous container and write the history
   entry (from and to image, the digest, product version, the features
   whose version changed, the off-host backup, duration).
9. If the migration fails, or the new container turns unhealthy, exits,
   restarts or is not ready in time, keep its last 80 log lines (the
   migration's, when that is what failed), remove it, put the database
   back when the rule below says so, put the previous container back under
   its name, wait for it to be healthy, and refuse that image from then on.
   The history entry says `rolled_back`, carries the log lines and says
   whether the database went back and why.

### When the database goes back

A rollback restores the agent's pre-deploy dump (dropping the database
first, so tables the new migrations created do not linger) when:

- the migration failed, or the agent died during it. MariaDB commits DDL as
  it goes, so a failed migration can leave half of its changes behind
  without recording them;
- the migration finished, and the schema fingerprint differs from the one
  taken before the deploy. The previous version cannot run on the new
  schema, and a down-migration is not something to trust with production
  data;
- the fingerprint cannot be read, which counts as changed;
- `AUTODEPLOY_MIGRATE_COMMAND=off`: the new container migrated as it
  started, and the agent cannot tell a failed migration from anything
  else.

It does not when the deploy stopped before anything ran against the
database (a failed backup, a container that could not be created), and when
the migration finished without changing the schema: the previous version
comes back on the same database, and nothing written while the new one
tried to start is lost. The dump is taken right before the old version
stops, so a restore loses only what the new version wrote during its
attempt.

Every step is written to `/data/state.json` before it is taken. An agent
restarted in the middle of a deploy (a reboot, a redeploy of the agent
itself) finishes it if the new container turns ready, and rolls it back
otherwise; one restarted during the migration rolls back with the
database.

| State                | Meaning                                                      |
|----------------------|--------------------------------------------------------------|
| `idle`               | The web container runs the image the channel points to       |
| `waiting_for_window` | A new image is ready and waits for the update window         |
| `deploying`          | Backup, recreate or health wait in progress                  |
| `rolling_back`       | The new image failed and the previous version is coming back |
| `paused`             | An administrator paused automatic deploys                    |
| `held`               | The only image available failed before; waiting for the next release |
| `error`              | Something needs a person, the detail says what               |
| `not_configured`     | A required setting is missing, the detail lists it           |

## Enable it in a product

1. Add the feature to the product line as an optional feature, from a
   detached workspace.

   ```bash
   splent spl:add-feature <spl> splent_feature_autodeploy
   ```

   In the UVL model it reads as below, with the constraint that keeps the
   admin panel it lives in.

   ```
   optional
       autodeploy {org 'splent-io', package 'splent_feature_autodeploy'}
   constraints
       autodeploy => admin
   ```

2. Attach it to the product and release the product, so the channel exists.

   ```bash
   splent product:select <product>
   splent feature:attach splent-io/splent_feature_autodeploy v0.1.0
   splent product:release
   ```

   A product release pushes `<registry>/<org>/<product>:<version>` and moves
   `:stable` to it.

3. On the server, set `SPLENT_IMAGE` in `docker/.env.deploy` to the channel
   the server follows, then build and deploy. The web service of the
   product's `docker-compose.prod.yml` must take its image from that
   variable, as in `image: ${SPLENT_IMAGE:-<product>:latest}`.

   ```bash
   SPLENT_IMAGE=git.splent.io/splent-io/<product>:stable
   ```

   ```bash
   splent product:build --no-image
   splent product:deploy
   ```

   `product:build` adds the agent's service to `docker-compose.deploy.yml`
   and copies `docker/` to `docker/features/splent_feature_autodeploy/`.
   `product:deploy` offers a random value for `AUTODEPLOY_SECRET`, builds the
   agent's image and starts it.

4. For a private registry set `AUTODEPLOY_REGISTRY_USER` and
   `AUTODEPLOY_REGISTRY_TOKEN` (a token that can read packages).

From then on, `splent product:release` is the whole deploy.

## Configuration

Declared in `[tool.splent.config]`, so they appear in `.env.deploy.example`
with the product's names filled in.

| Variable                     | Default                             | Meaning |
|------------------------------|-------------------------------------|---------|
| `AUTODEPLOY_IMAGE`           | empty                               | Image to follow. Empty means `SPLENT_IMAGE`. A channel (`:stable`) or a version tag; `:latest`, or no tag at all, is refused |
| `AUTODEPLOY_INTERVAL`        | `300`                               | Seconds between checks (at least 30) |
| `AUTODEPLOY_WINDOW`          | empty                               | UTC span for deploys, like `02:00-05:00` or `22:00-04:00`. Empty means any time |
| `AUTODEPLOY_HEALTH_TIMEOUT`  | `600`                               | Seconds the migrations get, and then the new container to turn ready |
| `AUTODEPLOY_MIGRATE_COMMAND` | `splent db:upgrade`                 | What the one-off migration container runs. `off` leaves the migrations to the new container's start |
| `AUTODEPLOY_WEB_MIGRATE_ON_START` | `off`                          | `SPLENT_MIGRATE_ON_START` for the new container after the migration step, so its start does not migrate again. `check` also refuses to serve on a schema behind its code, and needs splent_cli 1.41.0 in the image; `upgrade` migrates again; `keep` leaves the container's own |
| `AUTODEPLOY_READY_PATH`      | `/readyz`                           | The readiness probe. `off` waits for Docker's healthcheck alone |
| `AUTODEPLOY_WEB_PORT`        | `5000`                              | The port the web container answers on, inside the product network |
| `AUTODEPLOY_BACKUP_COMMAND`  | `auto`                              | The off-host backup before a deploy. `auto` is `splent_feature_backup`'s when the product has it, `off` none, anything else the command to run in the web container |
| `AUTODEPLOY_APP_USER`        | `splent`                            | The user the migration and the backup command run as |
| `AUTODEPLOY_KEEP_BACKUPS`    | `5`                                 | Rollback dumps kept, newest first |
| `AUTODEPLOY_WEB_CONTAINER`   | `<product>_web_deploy`              | The container that is replaced |
| `AUTODEPLOY_DB_CONTAINER`    | `<product>_db_deploy`               | The MariaDB container that is backed up |
| `AUTODEPLOY_AGENT_CONTAINER` | `<product>_autodeploy`              | The agent's container name |
| `AUTODEPLOY_URL`             | `http://<product>_autodeploy:8080`  | Where the web side finds the agent |
| `AUTODEPLOY_REGISTRY_USER`   | empty                               | Registry account, for a private channel |
| `AUTODEPLOY_REGISTRY_TOKEN`  | empty                               | Registry token, for a private channel |
| `AUTODEPLOY_SECRET`          | `<SET>`                             | Shared by the web container and the agent |

## The Deployments page

`/admin/deployments`, in the admin panel's sidebar, for administrators only
(the admin feature's `ADMIN_ROLES`). It shows the channel, the running image
and product version, what the agent is doing and why, the last check, the
update window, whether updates are paused, and the latest deploys with the
features each one moved. A rolled back deploy shows its log lines in a
collapsible block, and a banner says so until the next deploy. Check now,
Pause and Resume are CSRF protected forms.

When the agent is missing (development), stopped or refuses the secret the
page explains what is going on. The site itself is never affected.

## Commands

```bash
splent feature:autodeploy status [--json]
splent feature:autodeploy history [--limit N] [--json]
splent feature:autodeploy check
splent feature:autodeploy pause
splent feature:autodeploy resume
```

They talk to the agent's API, so they need the product network. On a
production server run them in the web container.

```bash
docker exec <product>_web_deploy splent feature:autodeploy status
```

## The agent's API

Port 8080 on the product network, never published on the host. Every request
carries `Authorization: Bearer <AUTODEPLOY_SECRET>`, compared in constant
time; anything else is a 401, whatever the path.

| Method | Path               | Does |
|--------|--------------------|------|
| GET    | `/status`          | State, configuration, running versions, backups |
| GET    | `/history?limit=N` | The newest N deploys, newest first (1 to 200) |
| POST   | `/check`           | Look for a new image now |
| POST   | `/pause`           | Stop deploying; a deploy in progress finishes |
| POST   | `/resume`          | Deploy again, and check right away |

There is no rollback endpoint on purpose. A rollback restores the backup taken
before the deploy, so run later by hand it would throw away everything written
since.

## Security

- The agent mounts `/var/run/docker.sock`. Whoever controls that socket is
  root on the host, so the agent is as sensitive as the host itself. It runs
  only in production, in a container of its own, from the image
  `product:deploy` builds out of this feature's `docker/` directory, and does
  only what this README describes. No container of the app mounts the
  socket: not the web container, not the migration job it creates.
- It publishes no port. Only containers on the product network reach the API,
  and every call needs the shared secret. The secret never appears in the
  status, the history or the logs.
- The web side calls the agent without any proxy, so an `HTTP_PROXY` in the
  environment never sees the secret.
- The database credentials stay in the database container. The dump and the
  restore read them from that container's environment and pass them through
  `MYSQL_PWD`, so they are not on any command line and are never printed.
- Registry credentials are only sent to the registry, with the pull.

## Backups

One per deploy, in the agent's `autodeploy_data` volume under
`/data/backups/`, named `<UTC time>-<short id of the image being replaced>.sql.gz`,
so each file says which code its schema belongs to. The newest
`AUTODEPLOY_KEEP_BACKUPS` are kept. They live on the same host as the
database, so they protect against a bad release, not against losing the
server: that is what `splent_feature_backup`'s off-host backup, taken just
before, is for. Without that feature, copy them elsewhere if you need it.

## Restoring a backup by hand

Restoring a backup loses everything written after it was taken. Pause the
agent first, so it does not deploy in the middle.

```bash
docker exec <product>_web_deploy splent feature:autodeploy pause
docker cp <product>_autodeploy:/data/backups/<file>.sql.gz .
docker stop <product>_web_deploy
docker exec <product>_db_deploy bash -c 'MYSQL_PWD="$MARIADB_ROOT_PASSWORD" mariadb -uroot -e "DROP DATABASE IF EXISTS \`$MARIADB_DATABASE\`; CREATE DATABASE \`$MARIADB_DATABASE\`"'
gunzip -c <file>.sql.gz | docker exec -i <product>_db_deploy bash -c 'MYSQL_PWD="$MARIADB_ROOT_PASSWORD" mariadb -uroot "$MARIADB_DATABASE"'
```

Then run a web container on the image named in the file name, so code and
schema match, and resume the agent.

To let the agent try an image it refused, pause it, delete the image id from
`bad_images` in `/data/state.json` in the agent's volume, and resume.

## What the agent does not do

- It replaces the web container only. A release that adds a service (a
  feature that ships its own container), or needs a new setting without a
  default in `.env.deploy`, has to go through `splent product:deploy`; pause
  the agent while you do it. Left to the agent such a release fails its
  readiness check and is rolled back, which is safe but not what you want.
- It takes no database lock (`GET_LOCK`) around the migration: one agent
  replaces one web container, which is stopped while the job runs, so the
  job is the only migrator. Several replicas would need that lock.
- It does not update itself. A new version of this feature reaches the
  server with `splent product:deploy`, which rebuilds the agent's image.
- It replaces one web container, not a set of replicas.

## Develop

The agent is plain Python 3.13 with docker-py, in `docker/autodeploy_agent/`.
It gets the Docker client handed in, so its tests run against a fake Engine.

```bash
splent feature:test splent_feature_autodeploy
```

From a bare checkout, with a Python that has the framework, Flask-Babel,
Flask-Login and Flask-WTF installed, the same tests run without a product.

```bash
python -m pytest src/splent_io/splent_feature_autodeploy/tests
```
