Metadata-Version: 2.4
Name: fleet-watch
Version: 0.3.0
Summary: Local process governance for AI workloads, plus a read-only GitHub fleet sitrep
Author: Christopher Chanhnourack
License: MIT
Project-URL: Homepage, https://github.com/cjchanh/fleet-watch
Project-URL: Repository, https://github.com/cjchanh/fleet-watch
Keywords: process-management,gpu,ai,mlx,ollama,apple-silicon,developer-tools
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Monitoring
Classifier: Topic :: System :: Systems Administration
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.0
Dynamic: license-file

[![CI](https://github.com/cjchanh/fleet-watch/actions/workflows/ci.yml/badge.svg)](https://github.com/cjchanh/fleet-watch/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![PyPI](https://img.shields.io/pypi/v/fleet-watch)](https://pypi.org/project/fleet-watch/)

# Fleet Watch

Local process governance for AI workloads on a single machine. It tracks ports, GPU memory, and repo leases so two jobs do not collide, and it can list GitHub repositories the local `gh` CLI already sees.

## Install

```bash
pipx install fleet-watch
```

```bash
fleet --version
```

From source: `pipx install ~/path/to/fleet-watch/`

## Example

```bash
fleet guard --json --port 8899 --repo ~/projects/my-app
```

If `"allowed"` is false, do not bind the port or write the repo.

## What it refuses

`fleet guard --json` never silently allows on error: a missing registry, a locked database, an unreadable input, a negative GPU value, or a non-integer port all return `"allowed": false`. `fleet sitrep` talks to GitHub only through local `gh`; it does not clone, store tokens, or invent commit SHAs.

## The Problem

You're running MLX, Ollama, vLLM, Candle/Cake, experiment runners, and AI coding agents on the same machine. They don't know about each other. Port 8899 gets stolen by a canary model. A 7B model quietly allocates 11 GB of Metal buffers on an 8 GB machine, swapping to SSD and running 65x slower than expected. Two Codex sessions write to the same repo. Health endpoints say "ok" while GPU memory is exhausted.

Fleet Watch prevents these collisions by maintaining a shared registry of what's running, what resources are claimed, and what's available — including pre-flight working set estimation that catches memory overcommit before it becomes a six-hour debug session.

## How It Works

Fleet Watch auto-discovers running AI processes (MLX servers, Ollama, vLLM, Candle/Cake, etc.) by scanning `lsof` and `ps`. It registers them in a local SQLite database with their port claims, GPU memory estimates, and repo/session ownership. Any tool — human or AI — can call `fleet guard --json` before taking resource actions.

**You don't register anything manually.** Run `fleet discover` or let the launchd agent do it every 60 seconds.

**GPU memory guard** estimates total working set (weights + KV cache + activations + framework overhead) and compares it against physical RAM. A Candle-based 7B model with Q4_K_M quantization is estimated at about 7 GB under Fleet Watch's default estimator assumptions.

## Quick Start

```bash
# See what's running
fleet status

# Auto-discover and register all AI processes
fleet discover

# Pre-flight: will this model fit?
fleet guard --gpu 4096 --framework candle --model "qwen2.5-7B-Q4_K_M" --json

# Check port and cooperative repo availability
fleet guard --port 8899 --repo ~/projects/my-app --json

# Check same-repo write intent before editing concrete paths
fleet guard --repo ~/projects/my-app --write-scope tools/playwright --json

# System health: memory pressure, sessions, GPU memory watch
fleet health

# See the audit trail
fleet history

# Generate state report
fleet report
```

### Example: 7B model on 8 GB Apple Silicon

```
$ fleet guard --gpu 4096 --framework candle --model "qwen2.5-7B-Q4_K_M"
DENY
GPU 4096MB: working set 7049MB exceeds physical RAM (8192MB) minus reserve (6144MB available)
  Breakdown: weights 3337MB + kv_cache 1792MB + activations 64MB x 2.0x (candle)
  Physical RAM available after reserve: 6144MB
  Suggestion: Use q2_k quantization (~1668 MB weights, ~5380 MB working set)
GPU budget available: 113664MB (0MB allocated)
```

The same model on a 128 GB machine:

```
$ fleet guard --gpu 4096 --framework candle --model "qwen2.5-7B-Q4_K_M"
ALLOW
GPU 4096MB: available (113664MB free)
  Working set: 7049MB (weights 3337 + kv 1792 + act 64) x 2.0x
  Physical RAM available after reserve: 114688MB
GPU budget available: 113664MB (0MB allocated)
```

## Always-On Mode (macOS)

```bash
fleet install-launchd
```

Fleet Watch will auto-discover processes and monitor GPU memory pressure every 60 seconds.

## AI Session Integration

Add this to your AI tool's system prompt or config:

> Before binding a port, starting a model server, or writing to a repo: run `fleet guard --json` with the relevant `--port`, `--repo`, `--write-scope`, `--gpu`, `--framework`, and `--model` flags. If `"allowed": false`, do not proceed. A plain `--repo` check is cooperative and reports active sessions; use `--write-scope` for concrete edit intent and `--exclusive-repo-lock` only for operations that require sole ownership. Use `~/.fleet-watch/state.json` only as a fallback artifact when the CLI is unavailable.

## JSON Contract

### `fleet guard --json`

Top-level keys:

- `allowed` — boolean allow/deny decision
- `request` — what the caller asked to use
- `checks` — per-resource decision objects
- `state` — current machine summary

`request` contains:

- `port` — requested port or `null`
- `repo_dir` — absolute repo path or `null`
- `write_scopes` — absolute paths this command may edit
- `exclusive_repo_lock` — whether the caller requested sole repo ownership
- `gpu_mb` — requested GPU claim or `null`
- `framework` — inference framework hint or `null`
- `model` — model name/path hint or `null`

`checks.port` contains:

- `allowed`, `reason`, `holder`, `suggested_ports`

`checks.repo` contains:

- `allowed`, `reason`, `holder`
- `holders` — advisory cooperative session holders for allowed same-repo work
- `overlap_paths` — paths that caused a write-scope denial
- `stale_holders` — dead/stale session leases cleaned during the check
- `safe_mode` — suggested operating mode such as `cooperative-write`
- `unblock_command` when denied by a process or session holder Fleet can name
  directly. A runnable command only when one truthfully exists — `fleet release
  --pid <pid>` for a process holder, `fleet session close --session-id <id>`
  for a session whose owner is provably dead. For a LIVE session holder it is
  remedy TEXT instead, naming who may close the lease (its owner pid, a
  descendant, or the terminal that spawned it) and the cooperative alternative,
  because `fleet session close` is authorized to the lease's own lineage and
  would refuse anyone else

`checks.gpu` contains:

- `allowed` — boolean
- `reason` — human-readable or `"working_set_exceeds_physical_ram"`
- `detail` — expanded explanation when denied
- `requested_mb` — requested GPU claim
- `available_mb` — currently available budget
- `suggested_max_mb` — maximum claim that fits
- `working_set` — (present when framework/model provided) object with:
  - `weights_mb`, `kv_cache_mb`, `activations_mb` — component breakdown
  - `overhead_multiplier` — framework-specific pool overhead (e.g. 2.0x for Candle)
  - `total_mb` — estimated total working set
  - `framework`, `model_size`, `quantization` — detected parameters
  - `physical_ram_mb`, `available_after_reserve_mb` — machine context
  - `fits` — boolean: does the working set fit in available RAM?
  - `grounded` — boolean: were framework and model size detected from real input?
  - `source` — `"explicit"`, `"command"`, `"fallback_default"`, or `"insufficient_input"`
  - `suggestion` — (when `fits` is false) actionable alternative

`state` contains:

- `process_count`, `occupied_ports`, `safe_ports`, `locked_repos`
- `gpu_budget` — `total_mb`, `reserve_mb`, `allocated_mb`, `available_mb`
- `external_resources`

### `fleet health --json`

- `memory` — RAM snapshot: `total_mb`, `pressure_pct`, `pageouts`, `swapins`, etc.
- `sessions` — discovered CLI sessions with RSS, CPU, classification
- `idle` — workload processes at near-zero CPU
- `gpu_memory_monitor` — runtime pressure data: `pageout_rate`, `gpu_process_footprints`, `alerts`

### `~/.fleet-watch/state.json`

Top-level keys:

- `agent_interface`, `generated_utc`
- `processes`, `external_resources`, `process_count`
- `gpu_budget`, `ports_claimed`, `preferred_ports`, `safe_ports`, `repos_locked`
- `session_leases`, `process_classifications`, `stale_processes`, `recent_events`
- `conflicts_prevented_24h`, `system_memory`, `sessions`, `idle_processes`
- `gpu_memory_monitor` — latest discovery-cycle snapshot of pageout rate, per-process footprints, and active alerts

## Commands

### Core

| Command | What It Does |
|---------|-------------|
| `fleet status` | Show active processes, GPU budget, claimed ports |
| `fleet status --json` | Machine-readable process and budget state |
| `fleet guard --json` | Canonical pre-flight contract for agents |
| `fleet guard --gpu MB --framework FW --model MODEL` | Working set estimation + allow/deny |
| `fleet guard --repo PATH --write-scope RELPATH --json` | Cooperative same-repo write-scope check |
| `fleet check --port N --repo PATH --gpu MB` | Honest availability probe (exit 0/1) |
| `fleet discover` | Scan and register running AI processes |
| `fleet report` | Write STATE_REPORT.md + state.json |
| `fleet sitrep --json` | Read-only GitHub fleet sitrep via `gh` (no clone, no tokens, no invented SHA) |

### Observability

| Command | What It Does |
|---------|-------------|
| `fleet health` | RAM pressure, sessions, idle processes, GPU memory watch |
| `fleet health --json` | Machine-readable health and GPU monitor snapshot |
| `fleet changelog` | Rolling state changelog |
| `fleet changelog --json` | Raw changelog entries |
| `fleet history` | Hash-chained event audit trail |
| `fleet stale` | List heartbeat-stale processes with evidence |
| `fleet reconcile` | Non-destructive ownership diagnosis |
| `fleet reconcile --json` | Machine-readable ownership diagnosis |
| `fleet census` | What boots and runs on this machine, and what's stale |
| `fleet census --json` | Full census record (`fleet-census/v1`) |
| `fleet census --emit-launchd-plist` | Print the staged daily-census launchd job; installs nothing |
| `fleet boot-map` | Census record → boot graph + local interactive 3D page |

#### `fleet census`

A deterministic, read-only sweep of every persistence and runtime surface:
`~/Library/LaunchAgents` cross-referenced against `launchctl list` and
`launchctl print-disabled` (both-direction orphan check), `/Library` daemons and
agents, live processes clustered by what they actually run, TCP listeners
attributed to their owning process, crontab, login items, brew services, and
Fleet Watch's own registry.

Every item carries a `status`, a `verdict` (`keep` / `investigate` / `close` /
`remove`), the exact evidence behind it, and the deterministic `rule` that fired.
Records land at `~/.governance/receipts/fleet-census/`, and each run diffs
itself against the previous one — new, disappeared and verdict-changed boot
entries are the signal.

Fleet Watch never kills anything here: `close_command` is advisory text for the
user, and the recurring launchd job is staged
(`contrib/launchd/io.fleet-watch.census.plist`), never installed.

Record contract: [`docs/fleet-census-receipt-contract-v1.md`](docs/fleet-census-receipt-contract-v1.md).

### `fleet sitrep`

A read-only GitHub fleet sitrep. One GraphQL query through the local `gh`
binary lists repositories owned by the authenticated viewer (or `--owner`)
and copies the default-branch object id GitHub returned.

```bash
# Viewer-owned repos, most recently pushed first
fleet sitrep --json

# A specific user or org GitHub still lets this `gh` login see
fleet sitrep --owner cjchanh --limit 30 --json
```

Rules the command is tested against:

- **No clone.** `sitrep` may only invoke `gh api graphql`. There is no
  `git clone` / `gh repo clone` path.
- **No tokens in this repo.** The code does not read `GITHUB_TOKEN` /
  `GH_TOKEN`, does not run `gh auth token`, and refuses `-t` / `--token`.
  Auth stays in the user's `gh` credential store. If `gh` is missing or
  the query fails, sitrep prints `REFUSAL` and exits 1 — it does not invent a
  fleet from local checkouts.
- **No invented SHA.** `sha` is a 40- or 64-char hex object id from GitHub's
  `defaultBranchRef.target.oid`. Empty repos, missing oids, and abbreviated
  values become `sha: null` plus `sha_absent_reason`. Short SHAs are never
  kept or padded.

Records (when not `--no-receipt`) land at
`~/.governance/receipts/fleet-github-sitrep/` as a dated JSON file plus
`latest.json`. Schema: `fleet-github-sitrep/v1`. At most `--limit` repos
(default 30, max 100); `truncated: true` means GitHub said another page
exists.

`fleet sitrep` is the only Fleet Watch subcommand that talks to GitHub.
`fleet guard`, `discover`, `census`, and `health` stay local.

### Session Lifecycle

| Command | What It Does |
|---------|-------------|
| `fleet session start` | Open or refresh a session lease |
| `fleet session heartbeat` | Refresh session lease heartbeat |
| `fleet session ensure` | Idempotent session management with retry |
| `fleet session close` | Close a session lease (owner, its descendants, or its ancestor shell only; anyone may reap a provably-dead owner). Also clears the Claude single-writer twin state file. No `--force`. |
| `fleet share-repo PATH` | Close active session leases for a `~/Documents/*` editorial handoff |

These five commands answer as soon as the registry write is committed. The
observability report is refreshed AFTER the acknowledgement, under a wall-clock
budget (`FLEET_REPORT_BUDGET_S`, default 2s) and coalesced across concurrent
sessions (`FLEET_REPORT_MIN_INTERVAL_S`, default 10s; `0` disables). A caller
that nonetheless times out must READ THE REGISTRY BACK — `fleet session list
--json` — before concluding the claim failed: a timeout is evidence about the
courier, not about the lease. See `session start`'s docstring for the exact
match conditions and for what a retry does to `fencing_epoch`.
`fleet discover` remains the unbounded publisher of `STATE_REPORT.md` /
`state.json`, so a skipped refresh here delays the report, never loses it.

A COOPERATIVE lease whose owner is alive but whose heartbeat is older than 180s
lapses for arbitration: it stops blocking and is reported under `stale_holders`
with `reason: "cooperative_lease_idle_lapsed"`. It is not closed — the owner's
next heartbeat revives it. EXCLUSIVE leases do not lapse.

### Process Management

| Command | What It Does |
|---------|-------------|
| `fleet register` | Manually register a process |
| `fleet heartbeat --pid N` | Refresh heartbeat for a registered process |
| `fleet release --pid N` | Release all claims for a PID |
| `fleet reap` | Dry-run: show orphan-confirmed processes |
| `fleet reap --confirm` | Kill and release orphan-confirmed processes |
| `fleet reap-sessions` | Kill detached hot sessions (dry-run by default) |
| `fleet runaway` | Detect runaway high-CPU processes |
| `fleet runaway --kill` | Kill flagged runaway processes |
| `fleet preempt` | Take a port from a lower-priority holder |
| `fleet clean` | Remove entries for dead PIDs |
| `fleet install-launchd` | Install/update a launchd agent |
| `fleet watch` | Continuous discovery loop (foreground) |

### Thunder (Remote GPU)

| Command | What It Does |
|---------|-------------|
| `fleet thunder sync` | Ingest Thunder instances into Fleet Watch |
| `fleet thunder claim` | Attach ownership to a Thunder instance |
| `fleet thunder heartbeat` | Refresh Thunder resource heartbeat |
| `fleet thunder release` | Remove a Thunder instance |

## Boot Map

One question: **what boots on this machine, and where does it go?**

`fleet boot-map` turns a `fleet-census/v1` receipt into a node/link graph and one
self-contained HTML page you rotate, zoom, and click through locally.

```bash
# Default: the latest census receipt -> ~/.governance/graph/boot-map/
fleet boot-map

# Any receipt, any output directory
fleet boot-map --receipt ./census.json --out /tmp/boot-map
open /tmp/boot-map/index.html
```

**What lands in the graph.** Launchd jobs, the scripts and binaries they launch,
the repos and system areas those live in, the ports they bind, live processes,
and the domains that group them. Verdict and status ride along as node
attributes, so colour is the census verdict — `keep`, `investigate`, `close`,
`remove`. Filled nodes were rated by the census; hollow nodes inherit the worst
verdict of what reaches them (a repo reads red when something red lives in it),
and say so in the detail panel. A red ring means a port bound to all interfaces.

**Outputs** (`graph.json`, `index.html`, `receipt.json`) are regenerable, so they
live outside the repo. Same receipt in, byte-identical graph and page out — no
wall-clock, no randomness, no layout drift.

**Local and sovereign.** Zero network at build time and at view time: no CDN, no
fonts, no images, no XHR, no libraries. The page ships a
`default-src 'none'; connect-src 'none'` CSP, and the test suite asserts the
artifact contains no `http(s)://` literal at all.

**Fail-closed.** Exit `3` and a `REFUSAL:` line when the receipt is absent,
unreadable, unparseable, structurally invalid, or degenerate. There is no
bundled-fixture fallback: rendering a stand-in as if it were this machine is a
fake-green, which is worse than no map. Zero domains or zero items is a refusal,
not a clean bill of health. Items missing a `status` or `verdict` are never
dropped — they are kept, coerced toward `investigate` (a human should look), and
flagged in `warnings`; off-enum values survive verbatim with a `*_valid: false`
marker so the graph cannot launder bad input into good-looking input.

The page refuses too: an empty or corrupted data island paints a REFUSAL panel,
never a clean-looking empty canvas. That behaviour is covered, not assumed —
`scripts/verify_boot_map_render.js` executes the emitted JS against a DOM stub
and counts canvas draw calls, so a page that loads clean and paints nothing
fails:

```bash
node scripts/verify_boot_map_render.js ~/.governance/graph/boot-map/index.html --expect render
```

## GPU Working Set Estimation

Fleet Watch estimates total GPU working set per framework:

| Framework | Overhead Multiplier | Why |
|-----------|-------------------|-----|
| Candle/Cake | 2.0x | Retains intermediate buffers until command buffer completion |
| vLLM | 1.4x | Paged attention overhead |
| MLX | 1.3x | Aggressive buffer reuse |
| Ollama/llama.cpp | 1.1x | Tight memory management |

Working set = weights + (KV cache + activations) x overhead multiplier

Override multipliers via `~/.fleet-watch/config.json`:

```json
{
  "gpu_estimator": {
    "framework_overhead": {
      "candle": 2.5
    }
  }
}
```

## Configuration

Fleet Watch writes a default config on first run at `~/.fleet-watch/config.json`.

```json
{
  "gpu_total_mb": 131072,
  "gpu_reserve_mb": 16384,
  "preferred_ports": [8000, 8001, 8080, 8100, 8888, 8899, 11434],
  "patterns": [
    {
      "name_template": "My Server",
      "process_match": "my_server.*serve",
      "workstream": "my-project",
      "priority": 3,
      "gpu_mb_default": 4096
    }
  ],
  "session_patterns": [
    {"name": "Claude Code", "kind": "claude-code", "process_match": "/claude\\b.*--"},
    {"name": "Codex", "kind": "codex", "process_match": "/codex\\b"}
  ],
  "idle_patterns": ["reranker", "socat.*TCP-LISTEN", "mlx_lm.*server"],
  "idle_cpu_threshold": 1.0,
  "pressure_thresholds": {"elevated": 70, "critical": 85}
}
```

## Event Audit Trail

Every registration, release, conflict, and cleanup is logged with a SHA-256 hash chain. Verify integrity:

```python
from fleet_watch import events, registry
conn = registry.connect()
valid, count = events.verify_chain(conn)
print(f"Chain valid: {valid}, events: {count}")
```

## Ownership Model

Fleet Watch uses **session leases** to track who owns what. Process classification requires three independent signals before marking a process as safe to reap:

Repo session leases are cooperative by default. `fleet guard --repo PATH --json` reports active sessions but does not deny solely because another session is in the same repo. Add `--write-scope RELPATH` when a command may edit files; Fleet denies only when another cooperative session has an overlapping declared scope. Use `--exclusive-repo-lock` for destructive git operations, whole-repo rewrites, or other work that truly requires sole repo ownership. An ACTIVE exclusive lease also denies when the requested resolved path equals, contains, or is inside the holder's stored `repo_dir`; cooperative leases stay exact-path.

`--gpu` rejects negative values. With `--json`, that is `{"allowed": false}` rather than Click usage text.

1. **Heartbeat expired** — not seen by discovery in >180 seconds
2. **Session lease missing or closed** — no active owner
3. **Parent chain detached** — parent PID is dead or PID 1

All three must be true for `orphan_confirmed`. Use `fleet reconcile` to inspect, `fleet reap --confirm` to act.

States: `live` > `disconnected` > `stale_candidate` > `orphan_confirmed` > `exited`

## Design Principles

1. **Advisory, not mandatory.** If Fleet Watch crashes, all processes continue normally.
2. **One honest verb per action.** `guard` decides, `check` probes, `discover` observes.
3. **Single machine.** No distributed consensus. SQLite is sufficient.
4. **Observe first.** Default to alerting, not killing.

## Platform Support

| Platform | Auto-discovery | GPU estimation | Manual registration |
|----------|---------------|----------------|-------------------|
| macOS (Apple Silicon) | Full | Full | Full |
| macOS (Intel) | Full | Partial | Full |
| Linux | Process matching + listener probes | Full | Full |

Linux memory-pressure admission uses PSI (`/proc/pressure/memory`, kernel 4.20+ with PSI enabled) when present: `some avg10` below 10% is normal, 10% to below 40% is warning, and 40% or above is critical; when the kernel has no PSI it falls back to `/proc/meminfo` MemAvailable. Without readable PSI or MemAvailable, the enforcing guard fails closed (refuses with an explanation) rather than admitting blind; the initial audit window remains advisory.

Auto-discovery uses `ps` for process matching. On Linux, listener-to-PID mapping uses `ss` when available, then falls back to `netstat` and `lsof`. Use `fleet register` for explicit registration on any platform.

## Limitations

- Discovery is heuristic and pattern-based. Unknown workloads are invisible until registered.
- GPU working set estimation uses architecture tables and framework multipliers, not kernel-level Metal accounting.
- Auto-discovery is heuristic and uses `ps` plus platform listener probes such as `ss`, `netstat`, or `lsof`. Use `fleet register` when explicit claims are more reliable than discovery.
- Fleet Watch is advisory for human use. For AI agent sessions, a PreToolUse hook can make it fail-closed.
- Single-machine by design. No distributed coordination.
- `fleet sitrep` requires an authenticated `gh` on PATH. It reports at most `--limit` owned repositories GitHub returned for that login. It is not a clone, not an unauthenticated public scrape, and not a list of local working trees.

## License

MIT
