Metadata-Version: 2.3
Name: lfp-swarm-consul
Version: 0.1.0
Summary: Fence-aware Consul server reconciler for Docker Swarm
Requires-Dist: docker>=7.1.0,<8
Requires-Dist: httpx>=0.28.1,<0.29
Requires-Dist: lfp-swarm-lease>=0.1.0,<0.2
Requires-Dist: pydantic-settings>=2.12.0,<3
Requires-Python: >=3.13
Project-URL: Repository, https://github.com/regbo/lfp-swarm-consul
Project-URL: Issues, https://github.com/regbo/lfp-swarm-consul/issues
Description-Content-Type: text/markdown

# lfp-swarm-consul

`lfp-swarm-consul` runs Consul servers on eligible Docker Swarm managers and continuously
reconciles their membership. Every manager also runs a controller contender. The bundled
`lfp-swarm-lease run` supervisor starts the Python reconciler only on the current lease holder and
stops its complete process tree when lease renewal fails.

Consul still elects its own Raft leader. The Swarm lease only elects the controller authorized to
change deployment and membership state.

## What is implemented

- A global Consul server service with manager-local persistent data.
- Gossip encryption supplied by an external Swarm secret.
- Automatic initial bootstrap for a configured server count, including two-node labs.
- Removal of `bootstrap_expect` after Consul has a leader and the expected peer set.
- Continuous Swarm manager, service task, Consul LAN, Raft, and Autopilot observation.
- One-at-a-time joins for running tasks missing from Consul membership.
- A separate zero-replica Swarm service containing fence-aware controller state and operation intent.
- Crash recovery through persisted intent and observed postconditions.
- Explicit recovery-required state when Raft peers exist but no Consul leader can be found.
- Optional forced removal after Swarm proves a server is no longer eligible and Autopilot is healthy.

Forced removal is disabled by default. An unreachable manager remains desired; reachability alone
never authorizes removal.

## Runtime

```text
controller task on every manager
            │
            └── lfp-swarm-lease election
                       │
                       └── active fence starts Python reconciler
                                      │
              ┌───────────────────────┴──────────────────────┐
              │                                              │
       Docker Swarm API                               Consul HTTP API
       managers + tasks                         members + Raft + Autopilot
              │                                              │
              └──────── one persisted action at a time ──────┘
```

The controller state is stored in `consul_controller-state`. Lease state remains owned separately
by `lfp-swarm-lease`.

## Build the controller image

The image installs both the reconciler and the pure-Python lease supervisor from locked uv
dependencies. `lfp-swarm-lease` is resolved from PyPI, so release and container builds do not
depend on an adjacent source checkout:

```console
docker buildx build \
  --tag registry.example.com/lfp-swarm-consul:0.1.0 \
  --push \
  .
```

Python dependencies are locked in `uv.lock`. Local verification uses:

```console
uv sync --locked
uv run ruff check .
uv run ruff format --check .
uv run pytest
```

Tagged releases use the same `release` GitHub environment and PyPI trusted-publishing model as
`lfp-swarm-lease`. The tag must match `uv version --short`; for example, run
`uv version 0.1.1`, commit the updated project metadata and lockfile, then push `v0.1.1`.

The supervisor's observation API remains available on port 8080. It starts `lfp-swarm-consul`
only after acquiring a lease and supplies the current term through `SWARM_LEASE_FENCE`. A lost
lease stops the controller process group; the same holder is not restarted until ownership first
transitions away and a later acquisition produces a new fence.

## Deploy the stack

The supplied stack must be deployed with the name `consul`; its internal service names are
`consul_server` and `consul_controller`.

Create the gossip key without placing its value in the repository or command arguments:

```console
docker run --rm \
  hashicorp/consul:1.22.7@sha256:a230dcea0bb107bd7958a912d1429fb7f9d399637de7ffb814b34412b9e8c543 \
  keygen |
  docker secret create consul-gossip-key -
```

Render the exact deployment input before deploying:

```console
export CONSUL_CONTROLLER_IMAGE=registry.example.com/lfp-swarm-consul:0.1.0
export CONSUL_BOOTSTRAP_EXPECT=2
docker stack config --compose-file stack/lfp-swarm-consul.stack.yml
docker stack deploy --compose-file stack/lfp-swarm-consul.stack.yml consul
```

Swarm does not build images during `docker stack deploy`; publish the controller image first.

### LFPConnect demo deployment

`lfp-swarm-consul.lfpconnect.stack.yml` is the Portainer-owned deployment for the isolated
LFPConnect demo. It runs one Consul server and one controller contender on each of the three Swarm
managers. Traefik load-balances the controllers' read-only lease observation API at
`https://consul-demo.lfpconnect.io`; the Consul HTTP API remains private to the stack network.

The deployment expects these external Swarm objects, created without generation suffixes because
they are the first generation:

- `lfp-swarm-consul-server-config`
- `lfp-swarm-consul-server-entrypoint-v3` (earlier configs remain immutable)
- `lfp-swarm-consul-gossip-key`

The checked-in LFPConnect definition records the live cluster's post-bootstrap steady state, so the
server service deliberately omits `CONSUL_BOOTSTRAP_EXPECT`. To recreate the cluster from empty
volumes, temporarily set it to `3` for the first deployment. The controller removes it after the
initial peer set is healthy; return the Portainer definition to the checked-in steady state after
that transition. When a new manager is added later, the global service starts a server there and
`retry_join` plus the controller admit it without changing the original bootstrap count.

## Two-node behavior

Two servers are supported with `CONSUL_BOOTSTRAP_EXPECT=2`, but the topology has no failure
tolerance: both voters are required for quorum. The controller reports that condition and never
automatically reboots a new generation after quorum loss. Removing `bootstrap_expect` performs a
one-at-a-time rolling restart and therefore causes a brief write outage in a two-voter cluster.

A third manager and Consul server is the recommended resilient topology.

## Eligibility and removal

Every active Swarm manager is eligible by default. Set this node label before draining a server
when it should no longer host Consul:

```console
docker node update --label-add lfp.consul.server=false <node>
```

The stack placement constraint and planner both honor that label. The server service uses
`leave_on_terminate`, so a reachable task normally leaves gracefully. Stale forced removal remains
disabled unless `CONSUL_ALLOW_FORCE_REMOVE=true` is set for a deployment. Enable it only when the
Swarm membership is authoritative and recovery procedures are understood.

## Security boundary

The control network is an encrypted overlay and gossip traffic has a Swarm-secret key. The Consul
HTTP API is not published outside that network. ACL bootstrap, HTTPS certificates, and a
least-privilege controller token are intentionally deployment-specific and are not fabricated by
this stack. Configure those before exposing the API or joining untrusted workloads.

Mounting `/var/run/docker.sock` gives controller containers manager-equivalent authority. Only the
controller service receives that mount.

See [docs/PLAN.md](docs/PLAN.md) for reconciliation invariants and recovery boundaries.

The live three-manager failure exercise, including node removal, controller and leader loss, a
simultaneous six-container crash, and KV integrity verification, is recorded in
[docs/FAILURE-RECOVERY-2026-10-15.md](docs/FAILURE-RECOVERY-2026-10-15.md).
