cronstable large clustering demo
=============================

A bigger, CPU-heavy ten-node cronstable cluster with mutual-TLS peer attestation
and leader election, built to watch `distribution: spread` fan a real workload
out across the cluster (vs single-leader, where one node runs all of it at
once). Spread is ON by default here. Brought up by
example/cluster-large/docker-compose.yml:

    docker compose -f example/cluster-large/docker-compose.yml up --build

Watch the load fan out with `docker stats` while the CPU-heavy jobs run.

Files
-----
  docker-compose.yml   The ten nodes plus the one-shot `certgen`; `distribution:
                       spread` is on by default here.
  jobs.yaml            The shared job set, mounted IDENTICALLY into all ten
                       nodes (17 jobs: 15 Leader/PreferLeader, of which six are
                       CPU-heavy and three run on business-day forms, plus two
                       EveryNode). The job-set id is a fingerprint of this file,
                       so it must be identical on every node for them to agree.
  ../_shared/node-entrypoint.sh
                       Generates each node's `cluster` section from environment
                       variables at start, so the compose file needs no
                       hand-written per-node peer lists. Shared with
                       example/pulse-cluster; see the env contract below.
  ../_shared/gen-certs.sh
                       Generates a throwaway cluster CA + per-node certs (SAN =
                       service name) into the shared `cluster-large-certs` volume
                       on first start. Shared by all the cluster demos; the
                       compose file names this demo's nodes via the NODES env
                       var.

The node env-var contract (../_shared/node-entrypoint.sh)
---------------------------------------------------------
Every node runs the same entrypoint and shares one CLUSTER_HOSTS list, dropping
itself by NODE_NAME. The variables it reads:

  NODE_NAME      this node's name (also its cert name and TLS SAN); required
  CLUSTER_HOSTS  comma/space list of ALL members as host:port (self included; it
                 is filtered out); required
  DISTRIBUTION   single-leader | spread          (default spread)
  ELECT_LEADER   true | false                    (default true)
  INTERVAL       poll seconds                     (default 10)
  DRIFT_AFTER    rounds before a peer is "drifted" (default 2)
  SHARE_NODE_STATS  true | false                 (default true) gossip each
                 node's CPU/memory for the dashboard's fleet view

The compose file sets DISTRIBUTION via `${DISTRIBUTION:-spread}`, so the
defaults above are what you get unless you override them in the environment.

Dashboards: cronstable-a .. cronstable-j on host ports 8080 .. 8089 (e.g.
http://localhost:8080/). Scroll to the "cluster" panel on any dashboard to watch
attestation, leadership, and per-job ownership.

The month-close / payroll-cutoff / third-friday-report jobs use the
business-day schedule forms (LW = last weekday of the month, L-3 = three days
before month-end, 5#3 = the third Friday). Open the week calendar (the ◫ week
toolbar button) to see them plotted while the minutely herd summarizes into
the "background hum" strip, or subscribe any calendar app to the fleet's iCal
feed:

    curl http://localhost:8080/calendar.ics
    curl "http://localhost:8080/calendar.ics?days=60"     # wider window
    curl http://localhost:8080/jobs/month-close/calendar.ics

Spread vs single-leader
-----------------------
By default each leader-gated job gets its own per-job owner via rendezvous
hashing, so the twelve Leader/PreferLeader jobs (and the CPU burns) fan out
across the ten nodes. To see the contrast, where ONE elected leader runs every
Leader job and all CPU burns at once, start with single-leader:

    DISTRIBUTION=single-leader docker compose -f example/cluster-large/docker-compose.yml up -d

Same quorum gate and same guarantee either way; spread is purely a load
optimization.

These certificates are for local experimentation only. For real deployments
provision per-node certs from your own PKI; see the wiki page
"Clustering and Leader Election".
