Metadata-Version: 2.4
Name: esys-ci-host
Version: 0.6.0
Summary: Library and CLI to prepare Linux and Windows hosts as GitLab CI runners
Author-email: LibESys <ops@libesys.org>
License-Expression: Apache-2.0
Project-URL: Homepage, https://gitlab.com/libesys/esys-ci-host/esys-ci-host
Project-URL: Repository, https://gitlab.com/libesys/esys-ci-host/esys-ci-host.git
Project-URL: Documentation, https://gitlab.com/libesys/esys-ci-host/esys-ci-host/-/blob/develop/README.md
Keywords: gitlab,ci,runner,docker,vps,cli
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: System :: Systems Administration
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: typer>=0.12
Requires-Dist: rich>=13.7
Requires-Dist: paramiko>=3.4
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: ruff>=0.8; extra == "dev"
Requires-Dist: paramiko>=3.4; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Provides-Extra: release
Requires-Dist: build>=1.2; extra == "release"
Requires-Dist: twine>=5.0; extra == "release"
Dynamic: license-file

# esys-ci-host

Library and CLI to prepare Linux VPS hosts as GitLab CI runners (Docker, privileged DinD, runner registration).

Extracted from the CI-host portions of [esysdox-ops](https://gitlab.com/libesys/esysdox-ops) so LibESys superbuild and other tooling can depend on a focused package.

## Install

```bash
pip install esys-ci-host
```

On a VPS, prefer the dedicated virtualenv used by remote mode (avoids apt/pip conflicts):

```bash
sudo apt-get update
sudo apt-get install -y python3-venv python3-pip python3.12-venv   # match your python3 -V
sudo mkdir -p /opt/esys-ci-host
sudo rm -rf /opt/esys-ci-host/venv
sudo python3 -m venv /opt/esys-ci-host/venv
sudo /opt/esys-ci-host/venv/bin/python -m ensurepip --upgrade
sudo /opt/esys-ci-host/venv/bin/python -m pip install esys-ci-host
sudo ln -sf /opt/esys-ci-host/venv/bin/esys-ci-host /usr/local/bin/esys-ci-host
```

Remote commands (`esys-ci-host remote ...`) create and use this venv automatically.

## Updating

Upgrade esys-ci-host without touching pip paths manually. The command detects how the
**currently running CLI** was installed and upgrades that environment.

| Install type | Detected when | Upgrade method |
|--------------|---------------|----------------|
| Managed VPS | Package under `/opt/esys-ci-host/venv` | `sudo` + refresh `/usr/local/bin/esys-ci-host` |
| Virtualenv | CLI runs inside a Python venv | `python -m pip install --upgrade` in that venv |
| User install | Package under `~/.local` | `pip install --user --upgrade` |

**On the VPS (as admin):**

```bash
esys-ci-host update                  # latest from PyPI
esys-ci-host update --version 0.3.6  # specific version
esys-ci-host update --dry-run        # show detected install and planned pip command
```

Do **not** use `sudo pip install` on the VPS — that hits system Python and can conflict with apt.

**From your laptop:**

```bash
esys-ci-host remote --host 161.97.147.246 --user esysadmin --identity-file ~/.ssh/id_ed25519 update
esys-ci-host remote --host ... --user esysadmin --identity-file ... update --version 0.3.6
```

| Option | Purpose |
|--------|---------|
| `--version VERSION` | Pin the PyPI version (default: latest) |
| `--wheel PATH` | Install from a local wheel instead of PyPI |
| `--dry-run` | Print detected install layout without upgrading |

On **Windows**, `update` cannot replace `esys-ci-host.exe` while it is running. The command
schedules pip to run after the current process exits. You can also run
`python -m esys_ci_host update` from the same virtualenv.

Remote update runs `update` on the VPS. On older installs without the `update` subcommand,
it falls back to upgrading the managed virtualenv directly.

## Quick start

On a fresh Ubuntu or Debian VPS (Docker executor for LibESys CI):

```bash
sudo esys-ci-host host bootstrap --yes --admin-user deploy --ssh-key-file ~/.ssh/id_ed25519.pub
# from your laptop: ssh deploy@<vps-ip>   (confirm login works)
sudo esys-ci-host host harden --yes
sudo esys-ci-host runner prepare --yes
esys-ci-host runner register --url https://gitlab.example.com --profile ci
esys-ci-host doctor
```

`host bootstrap` creates the admin user, installs OpenSSH, installs your SSH public key (or generates one under `/etc/esys-ci-host/`), verifies key setup, and enables UFW with SSH allowed. **Root SSH stays enabled** until you run `host harden` after confirming deploy login from your machine. Pass `--ssh-key-file ~/.ssh/id_ed25519.pub` or set `ESYS_CI_HOST_ADMIN_SSH_PUBLIC_KEY` / `ESYS_CI_HOST_ADMIN_SSH_PUBLIC_KEY_FILE`. Use `--disable-root-ssh` only if you accept lockout risk without external login verification.

The admin user has **no login password** (SSH key only). Bootstrap grants **passwordless sudo** via `/etc/sudoers.d/99-esys-ci-host-<admin-user>` so commands like `sudo esys-ci-host host harden` work after key login.

Set `ESYS_CI_HOST_RUNNER_TOKEN` instead of passing `--token` when registering.

## Remote configuration (from your laptop)

Run the same host and runner steps over SSH without logging in manually first. Remote mode installs `esys-ci-host` into `/opt/esys-ci-host/venv` on the VPS (from PyPI or a local wheel), links `/usr/local/bin/esys-ci-host`, uploads your **local** SSH public key for the admin user, and runs the usual commands there.

**Interactive** (prompts for the root password when no key is given):

```bash
esys-ci-host remote --host 203.0.113.10 host bootstrap --yes --admin-user deploy
ssh -i ~/.ssh/id_ed25519 deploy@203.0.113.10   # verify login (admin has no password)
esys-ci-host remote --host 203.0.113.10 --user deploy --identity-file ~/.ssh/id_ed25519 host harden --yes
esys-ci-host remote --host 203.0.113.10 --user deploy --identity-file ~/.ssh/id_ed25519 runner prepare --yes
```

After a successful remote bootstrap, the tool can offer to write a managed `~/.ssh/config` entry. It suggests a short name (when `--save-host` is set, that alias is used; otherwise for an IP like `161.97.147.246`, default is `{admin-user}-vps`, e.g. `esysadmin-vps`; for `runner01.example.com`, default is `runner01`) and lets you change it before saving. Then `ssh esysadmin-vps` (or your chosen name) works. Omit `--setup-ssh-config` to be prompted; use `--setup-ssh-config` to apply without prompts, `--ssh-config-host NAME` to set the alias directly, or `--skip-ssh-config` to disable the flow.

Remote mode reads **basic** OpenSSH `Host` defaults from `~/.ssh/config` (user, port, `IdentityFile`) when you omit CLI flags. By default it uses the **system OpenSSH client** (`ssh`/`scp`) when available so Host aliases, Pageant, and `IdentityAgent` behave like plain `ssh`; otherwise it falls back to Paramiko. Override with `--ssh-backend openssh|paramiko|auto` or `ESYS_CI_HOST_SSH_BACKEND`.

**Non-interactive** (CI or scripts — password via flag or env):

```bash
export ESYS_CI_HOST_SSH_PASSWORD='...'
esys-ci-host remote --host 203.0.113.10 --non-interactive host bootstrap --yes --admin-user deploy
```

| Option | Purpose |
|--------|---------|
| `--host HOST` | Remote VPS hostname or IP (required) |
| `--user USER` | SSH login user (default: `root`) |
| `--password PASS` | SSH password (`ESYS_CI_HOST_SSH_PASSWORD`) |
| `--identity-file PATH` | SSH private key (`ESYS_CI_HOST_SSH_IDENTITY_FILE`) |
| `ESYS_CI_HOST_SSH_IDENTITY_PASSPHRASE` | Passphrase for an encrypted `--identity-file` |
| `--ssh-backend BACKEND` | SSH transport: `auto` (default), `openssh`, `paramiko`, or plugin name |
| `ESYS_CI_HOST_SSH_BACKEND` | Same as `--ssh-backend` |
| `--non-interactive` | Fail instead of prompting for a password |
| `--wheel PATH` | Install from a local wheel instead of PyPI (Windows: use full path, not `*`) |
| `--no-version-check` | Skip local vs remote version/build warnings on SSH connect |
| `--ssh-key-file PATH` | Local admin public key (default: `~/.ssh/id_ed25519.pub`) |

Prefer `--identity-file` over `--password` after the first bootstrap. For encrypted keys,
set `ESYS_CI_HOST_SSH_IDENTITY_PASSPHRASE` or allow an interactive passphrase prompt.
With no `--identity-file`, authentication uses **ssh-agent** or **Pageant** (no server
password required). Avoid putting passwords on the command line in shared environments.

**Version checks:** `esys-ci-host version` and `esys-ci-host remote --host HOST version` show the package version and a short build fingerprint (SHA-256 of installed Python sources). Other remote commands warn when local and remote differ; use `--strict` on `remote version` to fail in scripts.

Commands still run **on the VPS** when you SSH in directly (`host bootstrap`, `runner prepare`, etc.) — both models are supported.

## Host bootstrap

| Option | Purpose |
|--------|---------|
| `--admin-user NAME` | Non-root login account with sudo (default: `esysadmin`) |
| `--ssh-key KEY` | OpenSSH public key line for admin login |
| `--ssh-key-file PATH` | File containing the public key |
| `--ssh-port PORT` | SSH port used for localhost login verification (default: 22) |
| `--disable-root-ssh` | Disable root SSH during bootstrap (default: keep root until `host harden`) |
| `--yes` | Required to apply host changes |
| `--dry-run` | Print steps without mutating the host |

Admin login is **SSH key only** (no account password). Bootstrap writes a sudoers drop-in so the admin user can run `sudo` without a password.

If no key is supplied, an ed25519 key pair is generated at `/etc/esys-ci-host/<admin-user>-id_ed25519` — copy the private key securely before closing your root session.

**After bootstrap:** SSH in as the admin user from your machine, then run `sudo esys-ci-host host harden --yes` to disable root login.

## Runner registration and executors

The GitLab executor is chosen at **registration** time (`runner register`), not during `runner install` or `runner prepare`.

| Option | Purpose |
|--------|---------|
| `--executor shell` | Run jobs directly on the host (default) |
| `--executor docker` | Run jobs in containers (requires `runner prepare`) |
| `--profile ci` | LibESys CI preset: `docker` executor, privileged DinD, tags `docker-privileged`, and a sensible default image |
| `--docker-image IMAGE` | Override the runner’s **default** image (fallback only; see note below) |
| `--docker-privileged` | Privileged docker executor (enabled automatically by `--profile ci`) |
| `--tags TAGS` | Comma-separated runner tags (e.g. `linux,docker`) |

**Default image vs job image:** GitLab Runner’s `--docker-image` (at register time) is only the **fallback** written into `config.toml`. Each CI job uses its own `image:` from `.gitlab-ci.yml` when present — you do **not** need to pass `--docker-image` for every image your pipeline uses. With `--profile ci`, a default (`python:3.11-bookworm`) is applied automatically; override it only if you want a different fallback for jobs that omit `image:`.

**Shell executor** (simple jobs on the host):

```bash
sudo esys-ci-host runner install --yes
esys-ci-host runner register --url https://gitlab.example.com --executor shell --tags shell
```

**Docker executor** (explicit; run `prepare` first). GitLab Runner requires a default image for non-interactive docker registration, so pass `--docker-image` here unless you use `--profile ci`:

```bash
sudo esys-ci-host runner prepare --yes
esys-ci-host runner register \
  --url https://gitlab.example.com \
  --executor docker \
  --docker-image python:3.11-bookworm \
  --docker-privileged \
  --tags docker-privileged
```

**LibESys CI profile** (recommended; includes docker executor defaults — no need to pass `--docker-image` unless overriding the fallback):

`runner prepare` creates the **`gitlab-admin`** group, adds the bootstrap admin (and `gitlab-runner` for service access), and makes `/etc/gitlab-runner` group-writable so `runner register` can patch `config.toml` without sudo for every read/write. Register and `systemctl` still use sudo when needed.

```bash
sudo esys-ci-host runner prepare --yes
esys-ci-host runner register --url https://gitlab.example.com --profile ci
```

**Windows Server CI runner** (shell executor, tags `windows,libesys`; run in an elevated PowerShell or cmd).
`runner prepare` installs Git, Python 3.11, Conan 2, and Visual Studio 2022 Build Tools with
the MSVC **14.4** toolset (Conan `msvc2022-x64` / `compiler.version=194`). If you only have
**VS 18** installed, `prepare` adds **VS 2022 Build Tools** side-by-side (VS 18 uses MSVC 14.5).
Re-run `prepare` on an existing VS 2022 host to upgrade a 14.3-only install.

```powershell
pip install esys-ci-host
esys-ci-host runner prepare --yes
esys-ci-host runner register --url https://gitlab.example.com --profile ci --token <token>
```

Set `ESYS_CI_HOST_RUNNER_TOKEN` instead of `--token`. Optionally set `ESYS_CI_HOST_RUNNER_USER_PASSWORD`
for the `gitlab-runner` service account; otherwise a password is generated once and saved to
`C:\ProgramData\esys-ci-host\gitlab-runner-account.json` (Administrators only — retrieve with
`Get-Content` in an elevated PowerShell). Check `esys-ci-host runner status` for the path.
Shell jobs should call `C:\Users\gitlab-runner\.esys-ci-host\ci-env.cmd` (written by `prepare`) or set
`pre_build_script` in GitLab Runner `config.toml` to load MSVC and Conan on PATH.

Use `--dry-run` on `runner register` to print the `gitlab-runner register` command without executing it.

### Runner operations

| Command | Purpose |
|---------|---------|
| `runner list` | Show runners in `config.toml` (add `--json` for machine-readable output) |
| `runner add` | Alias for `runner register` — register a new runner with GitLab |
| `runner start` / `runner stop` / `runner restart` | Control the local GitLab Runner service |
| `runner remove --name NAME --yes` | Unregister one runner from GitLab |
| `runner remove --all --yes` | Unregister every configured runner |
| `runner status` | Install, service, and esys-ci-host registration metadata |

The same commands work remotely over SSH (prefix with `remote --host HOST --user USER --identity-file …`):

```bash
esys-ci-host remote --host 203.0.113.10 --user deploy --identity-file ~/.ssh/id_ed25519 runner list
esys-ci-host remote --host 203.0.113.10 --user deploy --identity-file ~/.ssh/id_ed25519 runner stop
esys-ci-host remote --host 203.0.113.10 --user deploy --identity-file ~/.ssh/id_ed25519 runner remove --name "ci-runner" --yes
```

```bash
esys-ci-host runner list
esys-ci-host runner stop
esys-ci-host runner start
esys-ci-host runner remove --name "esys-ci-host docker runner" --yes
```

## Artifactory registry host

Prepare a dedicated VPS for **Artifactory C++ CE** (Conan packages for LibESys CI). The container binds to **localhost only** (`127.0.0.1:8081` / `8082`); `artifactory proxy` (or `deploy`) exposes it at **https://artifactory.libesys.org** with automatic Let's Encrypt TLS via Caddy.

**One-shot deploy** (prepare + bootstrap + HTTPS) after `host bootstrap`:

```bash
sudo esys-ci-host artifactory deploy --yes
esys-ci-host doctor
```

Or step by step:

```bash
sudo esys-ci-host artifactory prepare --yes
sudo esys-ci-host artifactory bootstrap --yes
sudo esys-ci-host artifactory proxy --yes   # Caddy + Let's Encrypt HTTPS
esys-ci-host doctor
```

**From your laptop** (after `host bootstrap` / `host harden`):

```bash
esys-ci-host remote --host artifactory.libesys.org --user esysadmin --identity-file ~/.ssh/id_ed25519 \
  artifactory deploy --yes
```

**Full automated flow from your laptop** (DNS `artifactory.libesys.org` → VPS IP required for HTTPS):

```bash
# Build wheel locally until a release is on PyPI:
pip wheel . -w dist/

esys-ci-host remote --host artifactory.libesys.org --user root --identity-file ~/.ssh/id_ed25519 \
  host bootstrap --yes --setup-ssh-config

esys-ci-host remote --host artifactory.libesys.org --user esysadmin --identity-file ~/.ssh/id_ed25519 \
  host harden --yes

esys-ci-host remote --host artifactory.libesys.org --user esysadmin --identity-file ~/.ssh/id_ed25519 \
  --wheel dist/esys_ci_host-*.whl artifactory deploy --yes
```

`deploy` runs prepare (when needed), bootstrap, and HTTPS proxy in one command. Pass `--save-host artifactory` (default on `remote artifactory deploy`) to store deploy credentials and Conan remote URL locally under `~/.esys-ci-host/hosts/`.

## Private image registry (CI Docker images)

Prepare a **local OCI registry** on the CI runner VPS (`registry:2` on `127.0.0.1:5000`) so GitLab jobs can push and pull `libesys/superbuild` images without GitLab Container Registry storage costs.

```bash
sudo esys-ci-host image-registry prepare --yes
esys-ci-host doctor
```

From your laptop (after `runner prepare` on the VPS):

```bash
esys-ci-host remote --host esysdocker --user esysadmin --identity-file ~/.ssh/id_ed25519 \
  image-registry prepare --yes

esys-ci-host remote --host esysdocker doctor
```

Credentials are saved on the VPS at `/etc/esys-ci-host/image-registry/bootstrap.json`
(mode `640`, group `esys-ci-host`). Members include the bootstrap admin and
`gitlab-runner` so CI jobs can read registry credentials without sudo. Set
`ESYS_CI_HOST_IMAGE_REGISTRY_PASSWORD` to pin the htpasswd password instead of generating one.

Pull credentials to your laptop once (password is read over SSH, never passed on the command line):

```bash
esys-ci-host remote --host esysdocker image-registry credentials pull --save-as esysdocker
```

Then manage images on the VPS from your workstation:

```bash
# Docker engine cache on the VPS
esys-ci-host remote --host esysdocker image-registry cache list --filter libesys/
esys-ci-host remote --host esysdocker image-registry cache rm libesys/superbuild:ubuntu22.04

# Private registry at 127.0.0.1:5000
esys-ci-host remote --host esysdocker image-registry registry list
esys-ci-host remote --host esysdocker image-registry registry tags libesys/superbuild
esys-ci-host remote --host esysdocker image-registry registry push libesys/superbuild:ubuntu22.04
esys-ci-host remote --host esysdocker image-registry registry pull libesys/superbuild:ubuntu22.04
esys-ci-host remote --host esysdocker image-registry registry rm libesys/superbuild:ubuntu22.04
```

`registry push` runs `docker login` with `--password-stdin` on the VPS, then tags and pushes — equivalent to:

```bash
docker login 127.0.0.1:5000 -u libesys-ci --password-stdin
docker tag libesys/superbuild:ubuntu22.04 127.0.0.1:5000/libesys/superbuild:ubuntu22.04
docker push 127.0.0.1:5000/libesys/superbuild:ubuntu22.04
```

Credential resolution order: `--password` → `ESYS_CI_HOST_IMAGE_REGISTRY_PASSWORD` → saved host (`credentials pull`) → remote `bootstrap.json` over SSH.

| Command | Purpose |
|---------|---------|
| `image-registry prepare` | Docker + localhost registry + `daemon.json` + `gitlab-runner` login |
| `image-registry cache list` | List images in the Docker engine cache |
| `image-registry cache rm` | Remove an image from the Docker cache |
| `image-registry registry list` | List repositories in the private registry |
| `image-registry registry tags` | List tags for one repository |
| `image-registry registry push` | Login, tag, and push a cached image to the registry |
| `image-registry registry pull` | Pull a registry image into the Docker cache |
| `image-registry registry rm` | Delete a tag from the registry |
| `remote image-registry credentials pull` | Save registry credentials under `~/.esys-ci-host/hosts/` |

| Option | Purpose |
|--------|---------|
| `--username NAME` | htpasswd user (default: `libesys-ci`) |
| `--password` | Override generated registry password |
| `--image-tag IMAGE` | Override pinned `registry:2` image |
| `--skip-docker-install` | Skip Docker apt install when Docker is already present |

## Local host registry

`esys-ci-host` remembers VPS hosts you configure on your workstation in `~/.esys-ci-host/hosts/<alias>.json` (mode `600`). Override the location with `ESYS_CI_HOST_REGISTRY_DIR`.

**Two different "users" in the bootstrap command:**

| Flag | Meaning |
|------|---------|
| `remote --user root` | Who you SSH as **today** on a fresh VPS (providers usually give root first) |
| `host bootstrap --admin-user NAME` | The **new** sudo account bootstrap creates (default: `esysadmin`) |

After bootstrap with `--save-host`, follow-up `remote --host artifactory` reuses the saved admin user, IP, and identity file — you do not need to repeat `--user` or `--identity-file` unless you want to override them.

**`--save-host` is only a local alias** on your laptop — it does not configure Artifactory on the server. The name `artifactory` is just a label meaning "this VPS will be the registry host"; you could use `--save-host contabo-vps` instead. At bootstrap time the registry stores SSH details for the new admin user; Artifactory credentials are added later by `artifactory deploy` or `hosts pull`.

```bash
# First time only: bootstrap as root, save admin SSH details as alias "artifactory"
esys-ci-host remote --host 203.0.113.10 --user root --identity-file ~/.ssh/id_ed25519 \
  host bootstrap --yes --admin-user esysadmin --save-host artifactory --setup-ssh-config

# Later: --host artifactory is enough (user + key come from ~/.esys-ci-host/hosts/artifactory.json)
esys-ci-host remote --host artifactory host harden --yes
esys-ci-host remote --host artifactory --wheel dist/esys_ci_host-*.whl artifactory deploy --yes

esys-ci-host hosts list
esys-ci-host hosts show artifactory
esys-ci-host conan setup --host artifactory
```

After `conan setup`, upload packages with:

```bash
conan upload esystrace/0.1.0@libesys/stable -r libesys
```

Use saved aliases on `remote` — connection details are read from the registry when omitted:

```bash
esys-ci-host remote --host artifactory artifactory proxy --yes
```

| Command | Purpose |
|---------|---------|
| `artifactory deploy` | Full flow: prepare + bootstrap + HTTPS (recommended) |
| `artifactory prepare` | Docker + Artifactory C++ CE on localhost ports |
| `artifactory bootstrap` | API: Conan repo, deploy user, saved credentials |
| `artifactory proxy` | Caddy + Let's Encrypt HTTPS for public hostname |

| Option | Purpose |
|--------|---------|
| `--yes` | Required to apply host changes |
| `--dry-run` | Print steps without mutating the host |
| `--fail-step NAME` | Force a controlled failure (testing) |
| `--hostname HOST` | Public hostname (default: `artifactory.libesys.org`) |
| `--repo-name NAME` | Conan repository key (default: `libesys`) |
| `--admin-password` | Override admin password on bootstrap re-run |
| `--skip-verify` | Skip HTTPS ping on `proxy` when DNS is not ready |
| `--image-tag IMAGE` | Override the pinned Artifactory C++ CE image |
| `--skip-docker-install` | Skip Docker apt install when Docker is already present |

After bootstrap, configure Conan on your dev machine:

```bash
esys-ci-host conan setup --host artifactory
# or manually:
conan remote add libesys https://artifactory.libesys.org/artifactory/api/conan/libesys
conan remote login libesys libesys-conan -p '<deploy-password from bootstrap.json>'
```

## Development

Architecture decisions are recorded in [docs/adr/](docs/adr/) (see [INDEX.md](docs/adr/INDEX.md)).

Platform-specific host operations (packages, users, firewall, SSH) live under
`src/esys_ci_host/platform/`. `DebianAptBackend` is implemented today; add new
`HostPlatformBackend` subclasses (for example `RhelDnfBackend`) and register
them in `platform/registry.py` to support additional distributions later.
Runner install/prepare still use apt directly and will migrate to the platform
layer in a follow-up.

```bash
python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
pre-commit install
pre-commit install --hook-type commit-msg
pytest
```

Release tags `vX.Y.Z` must match `src/esys_ci_host/__version__.py`. Never tag until **master** has a green pipeline on the commit you are releasing.

1. On `develop`: `make bump-release` (or `python scripts/bump_release.py --yes --version X.Y.Z`), then push and wait for green CI.
2. Fast-forward `master` from `develop`, push, and wait for green CI on `master`.
3. On `master`: `make tag-release` (or `python scripts/tag_release.py --yes`), then `git push origin vX.Y.Z`.

CI publishes tagged releases to GitLab PyPI and TestPyPI automatically; public PyPI is manual (`publish:pypi`).

**Integration tests** — fast suite mirrors GitLab CI; slow Artifactory test is local or manual CI only ([tests/integration/README.md](tests/integration/README.md)):

```bash
make integration-fast    # runner + bootstrap + remote-package + image-registry (~15–25 min)
make integration-slow    # artifactory prepare (30–60+ min); before release if artifactory changed

# Windows
.\test.ps1 -HostBootstrap
.\test.ps1 -Artifactory
```

When changing `artifactory` or `remote` prepare, run `make integration-slow` before tagging (or trigger manual `integration:artifactory:ubuntu` on GitLab).

## License

Apache-2.0 — see [LICENSE](LICENSE).
