Metadata-Version: 2.5
Name: ci-fleet
Version: 0.1.0
Summary: Self-hosted GitHub Actions runner fleet management: allocation policy, supervised launch, and execution-side concurrency control.
Requires-Python: >=3.11
Requires-Dist: filelock<4,>=3.13
Requires-Dist: psutil>=5.9
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# ci-fleet

Fleet management for self-hosted GitHub Actions runners: allocation policy,
supervised launch, and execution-side concurrency control.

Extracted from `charlie-work` so that runner development and
orchestrator development stop colliding. The package is `ci_fleet`; the
repository is `ci_runners` (directory name ≠ package name is the local
convention here).

## Status

Mid-migration. See [`PLAN.md`](PLAN.md) for the phased extraction plan and
[`ci-fleet-design.md`](ci-fleet-design.md) for the target design.
[`HANDOFF.md`](HANDOFF.md) carries the current position: what is done, what is
blocked on what, and the constraints that are not obvious from the plan.

As of 2026-08-04: suite at 823 passing (measured at `dd9257d`, the commit that
landed the retirement below along with its companion test updates -- collection
is clean and `uv run --active pytest -q --tb=short` is green; 869 was a
`d891906` reading one commit earlier, before that landing). The extraction
itself is written and the
section 6.3 cutover gate is implemented and self-verifying; the live repoint of
`charlie-work`'s consumers merged as `charlie-work#869` (2026-08-01). **The legacy
planner is retired.** Phase 6 (the flip) executed 2026-08-04, and on evidence from
a deliberately induced park/restore cycle plus explicit owner instruction, the
legacy planner itself was retired early -- 2026-08-04/05, about a month ahead of
the originally planned 2026-09-04 retention floor (see `HANDOFF.md` section 4.11).
`ci_fleet.planner.plan` is now the only planner; there is no `actuating_planner`
value left to flip, and the field rejects anything but `"new"`. Shadow mode ended
with the legacy planner -- see "Shadow mode" below for what that cost. Phase 3 is
not done (`HANDOFF.md` section 4.3). Phase 7's coverage-porting precondition was
discharged 2026-08-03 (`HANDOFF.md` section 4.1), but Phase 7 itself -- deleting
`charlie-work`'s own dormant copies of the legacy modules -- has not landed as of
this writing.

During the migration `charlie_work` imports this package through
`ci_fleet.charlie_work_adapter`, which holds the legacy symbol names and
signatures stable so consumer import lines change only in module path.

## Layout

```
src/ci_fleet/
    _vendor/                 copies of specific charlie_work functions/classes
    charlie_work_adapter.py  legacy import surface (boundary only)
tests/
    fixtures/                frozen pre-move baselines
```

**Invariant:** nothing under `ci_fleet` may import `charlie_work`. Vendoring
exists so the dependency arrow points one way.

## Development

```powershell
uv sync --extra dev
git config core.hooksPath .githooks   # once per clone -- see below
uv run --active pytest -q --tb=short
uvx ruff check .
uvx ruff format
```

That `core.hooksPath` line arms `.githooks/pre-push`, which runs the three
commands above against your branch before a push leaves the machine. It is
required once per clone: `core.hooksPath` is local config, so cloning this
repo does **not** arm it, and an unarmed clone pushes with nothing checked.

That matters more here than in a repo that gates its own `main`, because
`charlie-work`'s CI installs `ci_fleet` from this repository's `main` as an
editable path dependency tracking the branch rather than a pinned SHA. An
unlinted push here surfaces as a red required check on somebody else's
unrelated PR. Issue #1 has the full reasoning and why real CI is not yet the
answer; issue #35 is why the hook was inert in worktrees until recently.

Developed and linted against Python 3.12+. The package itself declares
`requires-python = ">=3.11"` — the oldest interpreter the suite is verified to
pass on, and the floor `charlie-work` needs. See the note in `pyproject.toml`.

## Shadow mode -- retired 2026-08-04

Through Phase 2, the rewritten planner (`ci_fleet.planner.plan`) ran alongside
the legacy one (`ci_fleet.runner_allocation.plan_allocation`) on every
allocation pass, and its output was compared and journaled -- never applied.
That arrangement is gone. `plan_allocation` and its policy core
(`allocate_slots`) were deleted along with the two modules that ran and
recorded the comparison (`ci_fleet/shadow_pass.py`, the pass wiring, and
`ci_fleet/shadow.py`, the `ShadowRunner` purity guard). The conditional that
used to choose which planner actuated was removed entirely rather than
collapsed to its surviving branch -- see the comment at the old flip point in
`runner_allocation_pass.py` for why a defaulted-but-dead branch was judged
riskier than no branch at all. `RunnerAllocationConfig.actuating_planner` now
defaults to `"new"` and its `__post_init__` rejects anything else, with an
explicit retirement message for anyone who types `"legacy"`: that value isn't
silently accepted-and-ignored, because someone setting it is attempting a
rollback, and a config file that disagrees with the running fleet should fail
loudly, not quietly do nothing.

**This is not pure cleanup -- retiring the legacy planner ended the shadow
comparison, and that is a real cost worth recording rather than a side
effect.** The legacy planner *was* the shadow's counterparty; once it was
gone there was nothing left for the new planner's output to be compared
against. The diff journal (`shadow-planner-diff.jsonl`, under the fleet
directory -- `%LOCALAPPDATA%\charlie-work\` on Windows) stops growing:
`HANDOFF.md` section 4.11 records 1051 records at the retirement, and the file
settles once the running supervisor respawns onto code that no longer calls
the comparison -- per-consumer respawn latency (see "Config propagation" or
the note on `actuating_planner` in `config.py`) means a few more passes can
land after the code changes on disk and before the supervisor picks them up,
so a live read may still show a handful more than the retirement figure until
the next respawn. Either way, any `shadow-status`-style tooling built against
this file loses its input once it stops. The instrument died with the thing
it measured. That is expected, not a regression to chase -- the comparison's
job was to validate the cutover, and the cutover is done.

**`ci_fleet/shadow_gate.py` and the frozen journal are both deliberately
kept**, even though nothing appends to the journal any more. The journal is
the evidence artifact the cutover decision rests on, and the gate can still be
re-run against it to re-derive that verdict from data rather than from
anyone's summary of the data:

```powershell
uv run --active python -m ci_fleet.shadow_gate <path-to-shadow-planner-diff.jsonl>
```

`PLAN.md` section 6.3 defines what the gate checks: a trailing agreement
streak, per-decision-class coverage, and every disagreement adjudicated in
writing. "Coverage" is not five-for-five against live passes -- of the five
decision classes, only `demotion` is still required live; the other four are
credited as discharged by evidence or adjudicated by replay (see
`shadow_gate.py`'s `DISCHARGED_BY_EVIDENCE` and `ADJUDICATED_BY_REPLAY`), which
is how the gate can still be satisfied against a journal that stopped
growing. Two config fields on `runner_allocation` -- `shadow_planner` and
`shadow_timeout_s` -- also still exist and are, likewise deliberately, now
inert: nothing reads either one, a stale `shadow_planner: true` left in a
config file enables nothing and is accepted without complaint (there is no
rollback it could be attempting), and both are scheduled for removal once the
live `config.yaml` drops the lines.

The legacy planner's own local copies inside `charlie-work`
(`runner_allocation.py`, `runner_allocation_pass.py`) are a separate matter --
unimported there already, and their removal is `PLAN.md` Phase 7, gated on its
own coverage precondition rather than on this retirement. Do not conflate the
two: this section is about `ci_fleet`'s own `plan_allocation`, which no longer
exists anywhere.

## Measurement layer (Phase 4)

Three primitives from §5.3 of `PLAN.md`, plus the classifier that reads them.

**Three named durations** (`ci_fleet.durations`) — `queue_wait`, `execution`,
and `wall`, recorded side by side and never added together. None is derived
from another, and the type is what enforces that: the fields take an
`Interval`, which carries two timestamps *and the clock that produced them*,
so there is no constructor that accepts a bare number. The design anticipates
three clocks (GitHub, the runner, this fleet), which do not reconcile exactly —
and the residual, `reconciliation_gap_s`, is time the job spent somewhere nobody
accounts for. **Today's producer uses only GitHub's clock**, so `single_clock`
is `True` and that residual is structurally zero rather than informative. That
is what `single_clock` is for: it marks the degenerate case so a zero gap is
read as a tautology instead of as the clocks agreeing. The fleet-clock `wall` is
the upgrade path and is deliberately not taken yet — see
`ci_fleet.job_observation`.

**Heartbeats** (`ci_fleet.heartbeat`) — a beat carries a *progress counter*,
not just a timestamp, because a reporter thread happily emitting timestamps
over a deadlocked worker is the exact failure being hunted. A counter that goes
backwards means broken data, not a stall, so it reports `UnknownProgress`
permanently rather than becoming a kill verdict.

**A fixed-op-count canary** (`ci_fleet.canary`) — constant work, so elapsed
time measures only host contention. The baseline is the fastest run ever
observed on this host, persisted and self-calibrating, rather than a hardcoded
constant that would be wrong on any other machine. Each pass reports the
minimum of five repetitions (~2 ms each); the amplification factor is
`elapsed / fastest_ever`. Baselines are keyed on workload shape, so changing
the op count discards them instead of producing a meaningless ratio.

**The 2×2** (`ci_fleet.health`) — starved vs. wedged, never a duration
threshold:

|                          | canary amplified    | canary normal          |
| ------------------------ | ------------------- | ---------------------- |
| **heartbeat advancing**  | starved → *wait*    | healthy, slow → *wait* |
| **heartbeat frozen**     | wedged under load → *kill* | wedged → *kill* |

The heartbeat axis decides the response; the canary axis explains it. A starved
process still ticks, because a heartbeat costs almost no CPU — so a frozen
counter means wedged however loaded the host is. Either axis can come back
unknown, and that is a fifth state carrying *no* response: a job seconds old,
with one beat and nothing to compare it against, must not be killed for being
new.

**A producer** (`ci_fleet.job_observation`) — `observe_job` maps one GitHub job
dict to a `Heartbeat` and a `JobDurations`. It is a pure function over the dicts
`measure_repo_demand` **already fetches** on every pass, so it adds **zero API
calls**; a separate poller would have roughly doubled request volume against a
5000/hour limit and sampled a different instant than the demand measurement,
making any disagreement between the two unresolvable.

An earlier version of this section claimed the heartbeat gap was structural —
that it needed the *jobs themselves* to report progress, hence workflow-YAML
changes in the consumer repos. **That was wrong, and measuring it is what showed
it.** `GET /repos/{owner}/{repo}/actions/runs/{id}/jobs` populates each job's
`steps[]` *incrementally while the job runs*: a live job here read
`status=in_progress total=7 done=4` mid-flight. The count of completed steps is
a progress counter the fleet can poll with **no job cooperation at all**. Step
*numbers* are unusable for it — a completed job reported `1,2,3,4,5,9,10,11`, so
`max(number)` jumps by four at the end.

Three things the measurement settled, each now a test:

- **The placeholder gate is `started_at == created_at`, not `status`.** A job
  **cancelled while still queued** reaches `status=completed` with the
  placeholder intact (`created == started == 07:04:10Z`, `completed=07:22:03Z`,
  `steps=[]`). A status-based gate records that job — which waited 18 minutes
  and ran nothing — as `queue_wait=0` and `execution=18m`: the `D-06`
  zero-default *and* a fabricated wedged-looking job. Across 51 completed jobs
  the correlation had no counterexample: all 20 `EQUAL` pairs were cancelled
  with zero steps; all 31 jobs that genuinely ran reported the two different.
- **`wall` is still measured for such a job**, because `created → completed` is
  a real span however the job ended, and "had a lifetime but no execution" is
  exactly the shape worth being able to see.
- **Poll granularity bounds what `Frozen` can mean.** The pass runs every five
  minutes and a single step legitimately runs far longer: a live job here held
  `done=4` across **20 consecutive polls** (200s), and a real `Tests` job ran
  14m25s over 7 steps. Two equal counters is therefore *normal*, not wedged, so
  the current `Frozen` rule would fire on healthy jobs. The stall tolerance must
  be **measured before anything actuates on it** — deliberately not added as a
  default-off parameter, since its only correct value comes from data this
  producer is what generates. See the open item in `PLAN.md` §9.

**Wired into the pass, and recorded rather than classified.**
`measure_repo_demand` returns its observations on a *second* return channel,
deliberately not folded into `RepoDemand` — that value is planner and shadow
input, and widening it would put a measurement into a comparison that is
supposed to be about allocation decisions. The pass batches them into one
`fleet_job_observations` event per pass, because the jobs in a pass share an
`observed_at` and that grouping is what makes consecutive beats comparable.

The 2×2 still does not run on live jobs, and that is the stopping point rather
than an omission: the `Frozen` rule needs the stall tolerance below, and logging
verdicts already known to be wrong would poison the very data that sets it.
A default-off `min_stall_s` parameter was considered and rejected — it would
ship the broken behaviour as the default while looking addressed.

Note the observation filter is *not* the demand filter, and the two must not be
merged: a job that finished while its run is still in flight contributes no
demand, yet it is the only place all three durations exist at once.

Independently of all that: one number would have hidden the whole story. On the
same run, `Lint` spent **36m09s queued and 35s executing**.

`timeout-minutes` semantics remain an open question by design (§5.3): recording
all three durations separately is correct under either reading, so it blocks
nothing, and the plan is explicit that it be *measured* rather than resolved
from a search result.

## Suite concurrency chokepoint (Phase 5, Stage A)

**Stage A measures. It does not enforce.** There is no lease, no counter, and no
blocking call anywhere in `ci_fleet.pytest_plugin` — only a latch. Phase 5's
acceptance criterion 3 wants the coverage share established *before* anything
clamps, and that ordering is not politeness: once a lease is narrowing some runs
and not others, "what fraction did we clamp" can no longer be answered by the
thing doing the clamping.

**The chokepoint is a packaging claim, not a code claim.** A shim that never
reaches a venv's `site-packages` covers nothing, however correct its body.
`ci_fleet.sitecustomize_install` is where that claim is cashed.

**The measurement inverted the design.** `sitecustomize` cannot identify a pytest
run from `sys.argv` at `site`-import time: under `python -m pytest` the module
name is not in argv yet (it reads `["-m", "--version"]`), and an IDE runner
calling `pytest.main()` presents its own script path. Two of the four invocation
modes are invisible to an argv predicate. The `pytest11` entry point has no such
problem, because pytest loads it however pytest itself was started. So the entry
point is the chokepoint and the shim's job is the opposite of what it looks like
— it exists to catch the runs the *plugin* missed.

**Numerator and denominator fail independently**, which is the only reason the
share means anything (`R-20`):

| | mechanism | fails when |
| --- | --- | --- |
| numerator | `pytest11` plugin latches in `pytest_configure` | plugin not loaded |
| denominator | `atexit` probe, shipped as a `.pth` in the wheel | package not installed, or `python -S` |

A run that bypasses the plugin — `-p no:...`, `PYTEST_DISABLE_PLUGIN_AUTOLOAD` —
still trips the exit probe and lands in the log as a bypass. An instrument
counting only its own hooks would report "100% of observed runs were observed."

The "fails when" column is deliberately not exhaustive about *rows*, only about
*mechanisms*. A venv carrying both the packaged `.pth` and an older
hand-installed shim fires the probe twice, and the second write is dropped by a
latch in `record_process_exit`. That is the invariant working rather than a
failure, so it is not a row in the table — but anyone computing a share from
these numbers should know the denominator counts **interpreters, not probe
registrations**.

**The independence is narrower than that table suggests, and the correction is
measured.** A third case used to be listed here — "or a venv without the
package" — as something that would still log a bypass. It does not. A scratch
venv carrying the real shim with no importable `ci_fleet` ran pytest and produced
**zero** rows; the same venv with `ci_fleet` on `PYTHONPATH` produced one row
with `plugin_saw_it=False`. The shim's `from ci_fleet.suite_coverage import ...`
sits inside its `except BaseException` guard, so with no package there is nothing
to import and nothing to write. Both halves share one dependency — the package
must be importable — so what they actually distinguish is **the plugin failing to
engage inside an instrumented venv**, which is the bypass that happens in
practice. An uninstrumented venv is invisible, not a bypass.

**Both halves now ship with the package, and that is a recent change.** They used
to install by different mechanisms: the plugin arrived with a `dev` extra (it is
a `pytest11` entry point in this package's metadata, so pip/uv wire it up), while
the probe was written into one specific venv's `site-packages` by an explicit
`install()` call that nothing invoked automatically. The consequence was measured
and was the opposite of intuitive — a venv with the **package and no probe logged
nothing at all**, not a bypass, because `record_process_exit` had exactly one
production caller and that caller was the shim. Adding `ci_fleet` to a repo's
`dev` extras therefore made its runs *pass through* the chokepoint without making
them *visible*.

The probe is now shipped as a `.pth` at the root of `site-packages` (the
mechanism `coverage.py` uses for subprocess support), so installing the package
delivers both halves. This was chosen over installing the shim per environment
for one reason: coverage that depends on someone remembering a manual step
decays, and across ~87 churning worktrees it decays to nothing, whereas coverage
that derives from *installed state* is inherited by every new environment on the
next sync.

**One row per process is enforced at the write, and that took two attempts.** A
venv can carry both the packaged `.pth` and an older hand-installed shim, and
then both probes fire. The first fix latched *registration* through a sentinel on
`sys`; it passed in a scratch venv and then failed in this repo's own, which
carries a shim written before the sentinel existed — an old shim registers
unconditionally and cannot be taught to check something that postdates it. Two
rows landed 0.5 ms apart under one pid. Since shims already on disk elsewhere can
never be retroactively changed, the invariant is enforced where the row is
written instead: at most one row per interpreter, however many probes fire. A
duplicate is not a harmless repeat — it inflates `sessions` with a run that never
happened, which is a false denominator in the one number this instrument exists
to publish.

**The instrument is validated. The share is not yet established — and those are
different claims.** A scratch harness started 5 pytest processes across three
invocation modes (`python -m pytest`, the console script, an IDE-style
`pytest.main()`) and the log came back 4 covered, 1 bypass; the bypass was a
`pytest --version` run, which returns from `pytest_cmdline_main` before a session
is ever configured. That is a real result about the *instrument*: it distinguishes
covered from bypassed, and the bypass is self-explaining because each row carries
its full `argv` — classifying early-exit invocations with a hardcoded flag list
would rot the first time pytest adds one.

It is **not** a coverage share, and must not be quoted as one. Every row in it was
a process this repo's own harness launched, within eight seconds, in a scratch
fleet dir, with the modes and the count chosen by the person measuring. A ratio
over a self-selected sample describes the sample. The **real** log
(`%LOCALAPPDATA%\charlie-work\suite-coverage.jsonl`) held **2 sessions, both
covered** when this was first written and grows with ordinary use; as of
2026-07-31 it holds 23 rows across 21 distinct pids. They are still the same
`pytest -q --tb=short` invocation from this one repo, which is a denominator far
too narrow to gate enforcement on — the count going up does not widen it.

Two of those rows are known-bad and are left in place rather than edited out:
they are the double-count described above, written before the one-row-per-process
invariant was enforced. Deleting rows from a measurement log to make a number
look right is the failure this whole section is about, so the contamination is
recorded instead — any share computed over this log before it is rotated is over
a denominator inflated by two phantom sessions (~9%).

**No data is not zero.** `CoverageShare.share` returns `None` for an empty log,
never `0.0`. An empty log is a claim about the instrument, not about the
chokepoint, and collapsing it would report total ineffectiveness on the strength
of no evidence. Uninstrumented venvs contribute *no rows at all*, so the number
is "coverage across instrumented environments" and nothing wider.

**Cost and blast radius.** The shim imports nothing at startup — it registers one
`atexit` closure and returns, deferring the `ci_fleet` import to exit and only in
processes where pytest ran. Measured at **416 µs cumulative / 384 µs self** per
interpreter start (`python -X importtime`). It swallows `BaseException`
everywhere (`R-19`, availability, fails open): an instrument that turns a passing
suite into a failing one at shutdown would be blamed on the suite. `install()`
refuses to overwrite a `sitecustomize.py` it did not write, since only one can
load per path entry and clobbering a foreign one silently disables whatever it
was doing.

**The plugin's blast radius is its import, not its hook.** That half was measured
only after the shim's, and it was worse: `pytest_configure` carried
`except BaseException` and a docstring promising it fails open, but pytest
imports an entry-point module *before* any of its hooks exist, so a module-level
`from .suite_coverage import …` could raise a plugin-load error that no `try` in
the file can reach — in every venv the package is installed into, not just this
one, and the consumer would reasonably blame their own suite. The import is now
deferred into the guarded hook body, leaving the module's startup surface as
`typing` alone: importing `ci_fleet.pytest_plugin` in a fresh interpreter leaves
`ci_fleet.suite_coverage` **absent** from `sys.modules`, present only once the
hook runs. Two tests pin it — one observing `sys.modules` from a subprocess
(rather than reading the source, since a transitive import three modules away
would pass a text check and still break the venv), and a positive control that
the hook really does fail open when that deferred import is broken. The general
form is worth stating, because Stage B will add imports here: *"this function
cannot raise" is a claim about a function, and an entry point's real risk surface
is its import.*

**Status: criterion 4 closed; 1, 2, 3, and 5 open — and 3 turns out to depend on
5.** Criterion 4 asks that every invocation mode reach the chokepoint, and a
harness that deliberately exercises each mode is the right way to answer it.
Criterion 3 asks for the coverage *share*, which the same harness cannot answer
for exactly the reason above: the share only becomes meaningful once the
denominator contains runs nobody chose. That needs criterion 5 — the plugin added
to the `dev` extras of each participating repo individually, worktree venvs
inheriting only after a sync — so **3 is blocked on 5**, and `local_suite_lock`
is not retired until that list covers every repo the incumbent guarded. Criteria
1 and 2 are Stage B: atomic reservation (not observe-then-narrow), clamp-not-block
(never return 0, never block past a deadline), and the honest bound
`max(B, R × min_units)`.

Per the phase spec, and worth stating plainly: **this does not fix the CI
variance.** It bounds CI's contribution to it — and Stage A does not yet even do
that, because it does not enforce.

## Visibility

This repository is **private and must stay that way** while `PLAN.md` and
`ci-fleet-design.md` are tracked. Between them they document:

- the host's runner topology and the repositories each runner is registered to
- a Windows service name and local filesystem paths under the owner's profile
- which paths are and are not on Defender's `ExclusionPath`
- a latent runner-exposure issue, with the condition that would trigger it

Every one of those is operationally load-bearing for this work and a map of the
build host to anyone else. Redacting them would degrade the working documents;
keeping the repo private costs nothing. Treat the list above as the checklist to
clear before any visibility change — not as an example of the kind of thing to
look for.

A prose note is a prompt-strength control. The architectural version is a
workflow that fails when `github.event.repository.private == false`; worth
adding if this repo ever grows CI of its own.
