The Fleet

How one runtime goes from a laptop to a fleet of workers on any cloud: every way to run it, every verb and door that drives it, what was measured on the real machines, and the honest list of what is not there. Written 2026-09-11, against runtime e4b65c5.

1. The four ways to run it

The same distribution, the same units, the same image. What changes is one line — PINECALL_ROLE — and how many machines carry it.

2. How capacity is counted

Never CPU. A worker reports calls held over calls it was measured to hold; LiveKit stops handing it calls at 0.7 of that, and picks among the rest at random, weighted by the room each has left (livekit-server selectWorkerWeightedByLoad). The hub adds the same numbers up:

a worker's load  = active / PINECALL_MAX_JOBS
the fleet's free  = Σ (max − active)       over workers heard in the last 30 s
accepting         = workers under 0.7, not cordoned, not draining
full              = somebody is there and nobody accepts

MAX_JOBS is measured on the machine type — calls with real audio, five more each step, until the p95 of first audio crosses 1.8 s — and set one under that. A worker without it (the hub's own on a full box) reports the machine's CPU and counts no seats.

3. Every method, by who calls it

The worker → the hub built

The operator → the hub built

The public → the hub, when it is full built

The loop → the cloud built

infra/fleet/<cloud> create <name>     boots from the worker image; hostname = name
infra/fleet/<cloud> delete <name>
infra/fleet/<cloud> list             name<TAB>created-at (ISO 8601), fleet-labelled only

Shipped: gcp, aws, hetzner — about forty lines of the vendor's own CLI each. Yours: --cloud ./mine with the same three verbs. The runtime imports no cloud SDK, ever.

4. One tick of the loop

1. Read both tables. The hub's seats (GET /v1/ops/fleet) and the cloud's machines (list). Nothing is kept between ticks: a loop restarted mid-boot sees the booting machine in list and waits for it.
2. Delete what is finished. A cordoned machine holding no call, or cordoned and gone silent; a machine that never dialled in within 10 min; one silent for 5.
3. Grow when fewer than --min, when there is no seat anywhere, or when busy = active / seats is over --target (0.6). A booting machine counts as --seats of capacity from the moment it is asked for, so the loop asks once. The name is the smallest free pinecall-worker-N.
4. Cordon the quietest managed worker only if the fleet would still sit under the target by 0.15 without it and stay above --min. Ties go to the newest. The loop never touches a machine the cloud does not list as the fleet's.

5. What was measured, on the real machines

6. Are we stuck? No. What is left, and why

The cloud credential is the operator's, per provider done on the hub

The loop is provider-neutral by construction: it runs a script and reads lines. What is not neutral — and cannot be — is who is allowed to create and delete machines. Every cloud names that permission differently, and every cloud requires you to grant it:

On pinecall-box this was done on 2026-09-11: the address reserved as static, a service account pinecall-fleet with the two roles attached to the stopped VM with the cloud-platform scope, three lines in box.env, one deploy — and pinecall-fleet.service now runs the loop on the hub, reading its credential from the metadata server, with the vendor's own gcloud that the manifest installs (never the snap, which refuses a service user homed under /opt). The loop still runs from any laptop too. The decision itself — which account may create and delete machines — stays the owner's on every cloud; the repository takes it and never makes it.

Media in every region not built

LiveKit's open-source server is one region. Routing a caller to the nearest SFU is a cloud feature or a second full stack behind a geographic DNS answer; the repository ships neither and scaling.md says so.

Dialing back the tenant's

The runtime records callback.requested; the org's app has the number, the agent's words and the outbound trunk, so it places the call. GET /v1/callbacks hands the list over and stops.

What does pinecall run do when the gateway falls? built

It reconnects. The app socket is ours, not LiveKit's (LiveKit reconnects media rooms; the app socket is the runtime's WebSocket at /v1/apps), and src/client/connection.ts retries with full-jitter exponential backoff — 500 ms first, doubling, capped at 30 s, no attempt limit — and agent.ts sends agent.register and the declaration again on every open, because the gateway's registry is memory. A deploy's restart window is about five seconds; each try inside it prints one Unexpected server response: 502 through the error door, which is what looked like a crash and was not: the processes were found alive an hour later. Only the first connect rejects — a wrong key, a host nobody is on — which is right. One "reconnected" line on stdout would make this legible; it is the only thing missing.

7. From here, in ten lines

# the hub is deployed and one worker was deployed by hand (infra/box/README.md)
gcloud compute machine-images create pinecall-worker --source-instance pinecall-worker-1 --source-instance-zone us-central1-c
gcloud compute instances add-labels pinecall-worker-1 --labels pinecall-fleet=1 --zone us-central1-c

pinecall-runtime fleet list                                   # the roster, as the hub hears it
pinecall-runtime fleet loop --cloud gcp --seats 4 --once --dry-run   # one tick, nothing touched
pinecall-runtime fleet loop --cloud gcp --seats 4 --min 1 --max 10   # the loop, until you stop it

pinecall-runtime fleet cordon pinecall-worker-3               # let one go by hand, gracefully
pinecall callbacks                                            # whom to call back (the tenant's CLI)

Source: fleet/roster.py · fleet/decisions.py · fleet/loop.py · api/fleet.py · worker/heartbeat.py · worker/overflow.py · infra/fleet/. Prose: scaling.md, the-runtime-cli.md.