Metadata-Version: 2.4
Name: hugpy-wrapper
Version: 0.1.145
Summary: The hugpy wrapper (import name: fitevict): a per-box evict-to-fit front door. An OpenAI /v1 server that resolves a called model to its weights (GGUF via llama.cpp, transformers chat/vision, and every non-chat engine: embeddings, whisper ASR, TTS, diffusers image + video, summarization, keywords), places it on the GPU by evicting what doesn't fit, serves it, and logs every call.
Author-email: putkoff <support@hugpy.ai>
License-Expression: LicenseRef-Proprietary
Project-URL: UI, https://www.npmjs.com/package/@hugpy/ui
Project-URL: Homepage, https://hugpy.ai
Keywords: llama.cpp,inference,evict-to-fit,vram,openai,wrapper,allocator,whisper,tts,diffusers,embeddings
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Provides-Extra: transformers
Requires-Dist: torch==2.14.0; extra == "transformers"
Requires-Dist: transformers==5.17.0; extra == "transformers"
Requires-Dist: accelerate==1.15.0; extra == "transformers"
Requires-Dist: bitsandbytes==0.50.2; extra == "transformers"
Requires-Dist: peft==0.20.0; extra == "transformers"
Requires-Dist: compressed-tensors>=0.15.0; extra == "transformers"
Requires-Dist: safetensors==0.8.0; extra == "transformers"
Requires-Dist: tokenizers==0.23.2; extra == "transformers"
Requires-Dist: sentencepiece==0.2.2; extra == "transformers"
Requires-Dist: tiktoken==0.14.0; extra == "transformers"
Requires-Dist: protobuf==7.36.1; extra == "transformers"
Requires-Dist: huggingface_hub==1.31.0; extra == "transformers"

# hugpy-wrapper (`fitevict`)

The per-box **evict-to-fit front door** of a Hugpy fleet: an OpenAI-compatible
`/v1` server over llama.cpp, transformers and every task engine. Point any
OpenAI client's base URL at it and one address becomes a self-managing model
server.

A `/v1` call carries only a model **name**, never a location. For each request
the wrapper:

1. resolves the name to its weights
2. runs an evict-to-fit plan against what is resident and what is being called
3. places the model, launching it and evicting others as needed
4. forwards the call
5. splits the model's `<think>` out of the answer
6. records the call (raw stamps and engine geometry) in its own store, which it
   reads back to place the next model better

It needs only an install, a host and a port: no database server, no other
hugpy package, no sudo for a user install. The decision core is pure stdlib.
Hardware facts come from `nvidia-smi`, NVML and `/proc`; model facts come from
the files on disk.

Distribution `hugpy-wrapper` **0.1.127**, import name **`fitevict`**. It has
no runtime dependencies.

**Status: LIVE.** The wrapper is the worker on every box: a wrapper install
defines a worker. On 2026-10-15, 0.1.122 answers `/health` on **a-brain**,
**ae** and **computron**, each registered with the central `hugpy-core`.
`hugpy-core` also carries the wrapper (0.1.120 in `/opt/hugpy-core/venv`) as a
*library* for its store, key auth, marker and transfer code. That copy never
runs a service. The older per-box agent `hugpy-worker` is retired (the wrapper
is the worker on every box); its source was removed from the hugpy/hugpy repo on
2026-10-15 and remains in that repo's history.

See also:

- [package index](../../README.md): every package
- [fleet architecture](../../FLEET_ARCHITECTURE.md): how the wrapper, core
  and arms fit together
- [installation](#install) and
  [operations](../../../docs/operations.md): procedures

## Install

### From the hugpy apt repository (how every live worker is installed)

```bash
sudo apt-get install hugpy-wrapper        # from https://dev.hugpy.ai/apt; Recommends hugpy-llama-server
```

The `.deb` is built from this bundle by `../installers/debian/build.sh` and named
`build/debian/hugpy-wrapper_<version>_all.deb` by default. It contains the wrapper wheel
and a `/usr/bin/fitevict` launcher.

Its `postinst`:

1. creates `/opt/hugpy-wrapper/venv`
2. installs the wheel offline
3. runs `fitevict install --system --port auto --yes` on a fresh box, or
   `fitevict install --system --yes` when `/etc/hugpy/wrapper.env` already
   exists

A re-run keeps the configured host, port and box key.

- `apt remove` runs `fitevict uninstall --system`.
- `apt purge` also removes `/etc/hugpy/wrapper.env`, the endpoint file,
  `/var/lib/hugpy-wrapper`, `/var/cache/hugpy-wrapper` and the `hugpy-wrapper`
  user. The shared model store is kept.

The GGUF engine is the separate
[`hugpy-llama-server`](../../hugpy-llama-server/README.md) deb (`/opt/hugpy-llama`).
Engine discovery finds `/opt/*/bin/llama-server`.

To join a worker to a central:

- On the central's console, **+ Worker** mints a worker key (`hpk_wk_…`) and
  prints the one command for the new box.
- Or, from the central host, run
  `hugpy-core workers add --ssh USER@HOST` (see
  [hugpy_core](../../hugpy-core/package/README.md#the-hugpy-core-cli)).
- Both end in
  `fitevict install --system --central <URL> --central-key <KEY> --host <addr>`.
  With these flags, the wrapper registers itself with that central.
- A central on the same box is found through
  `/etc/hugpy/core.endpoint.json`, and the install's `central` step runs
  `hugpy-core register`.
- A central that refuses never fails the wrapper install.

### With pip and `fitevict install`

`pip` only delivers the package; `fitevict install` is the install.

```
pip install hugpy-wrapper            # or: pipx install hugpy-wrapper

fitevict install --user --port auto                         # your own service, no root (the default mode)
sudo "$(command -v fitevict)" install --system --port auto  # a system service (root once)
fitevict install ... --dry-run                              # print the plan, change nothing
```

No port is ever assumed. Give a number, or `auto` for a free one; the port is
printed and written to the config. The host defaults to the configured one,
else `127.0.0.1`. Use `--host <LAN address>` (or `0.0.0.0`) to serve a
network.

`--system` sets up the hugpy layout. Each hugpy component runs as its own
user, and the group `hugpy` shares only the model store:

| what | where |
|---|---|
| user | `hugpy-wrapper` (system, no login), member of group `hugpy` |
| code | `/opt/hugpy-wrapper/venv` (this exact version) |
| config | `/etc/hugpy/wrapper.env` (root:hugpy-wrapper 0640): host, port, box key, every path |
| state (the store) | `/var/lib/hugpy-wrapper` (0700, created by systemd) |
| cache | `/var/cache/hugpy-wrapper` |
| models | `/var/lib/hugpy/models` (root:hugpy 2775), or `--models DIR` |
| service | `hugpy-wrapper.service`: `fitevict-serve --host ${FITEVICT_HOST} --port ${FITEVICT_PORT}`, hardened (read-only system; writes only its own dirs and the models) |
| endpoint | `/etc/hugpy/wrapper.endpoint.json`: where it listens, the engine it found, the Python engine discovery (no secret) |
| logs | the journal: `journalctl -u hugpy-wrapper` |

`--user` uses the environment you installed into, plus
`~/.config/hugpy/wrapper.env` and a user unit (`systemctl --user`;
`loginctl enable-linger` keeps it running after logout).

Every step is safe to repeat. A re-run repairs or upgrades, and keeps the box
key, the port and the host. A slow first start is allowed 180 s before the
health check gives up (`FITEVICT_HEALTH_WAIT_S`).

`fitevict install` options:

| option | meaning |
|---|---|
| `--system` / `--user` | the install mode |
| `--host`, `--port` | where the front door listens |
| `--worker NAME` | this box's worker name (default: the host name) |
| `--models DIR` | the model store |
| `--package SPEC` | the code to install, such as a wheel file |
| `--engine SPEC` | pip-installs an engine |
| `--tf [--tf-spec S] [--tf-find-links DIR]` | builds the box's transformers engine venv |
| `--minimal` | service, config and store only |
| `--central URL --central-key KEY [--advertise HOST]` | registers this worker with a remote central |
| `--vram-max/--ram-max GIB`, `--cpu-max CORES`, `--storage-max GIB` | capacity designations |
| `--env KEY=VALUE` | an extra service setting (repeatable) |
| `--yes` | no questions or confirmations, for scripts |
| `--dry-run` | prints the plan and changes nothing |

Platforms: Linux with systemd today. macOS (launchd) and Windows services are
not supported; the command says so rather than half-installing.

### The engines

- **GGUF** models need `llama-server`. The wrapper looks for it as
  `LLAMA_SERVER_BIN`, then under `HUGPY_ENGINE_DIR`, then on `PATH` (the
  service's `PATH` starts with its own venv), then at
  `/opt/*/bin/llama-server`. `fitevict install` reports whether one resolves.
  Without one, the wrapper still runs, but GGUF calls fail until an engine is
  there.
- **Transformers, vision, embeddings and diffusers** run in a separate Python
  engine, never the front door's venv.
  - `fitevict install --tf` builds `<HUGPY_ENGINE_DIR>/tf` from
    `hugpy-wrapper[transformers]` (pinned torch, transformers and friends).
  - Without it, discovery checks conventional virtualenv and Conda locations,
    verifies the engine imports as the service account, and reuses a
    compatible environment.
  - `fitevict engines --system` reports the candidates and which one was
    selected.
  - An explicit `FITEVICT_TF_PYTHON` overrides discovery.

## Command-line tools

| command | what it does |
|---|---|
| `fitevict install \| uninstall [--purge] \| status` | Installs, removes or inspects the service (`--system` or `--user`). |
| `fitevict engines` | Discovers existing Python engines. |
| `fitevict keys create --name N [--scope use\|admin] [--expires-days D] \| list [--all] \| revoke ID` | API keys for callers outside the trusted networks. A key is printed once; only its hash is stored. |
| `fitevict setup [--harness hugpy-harness\|hugpy-chat\|aider\|opencode\|continue\|env] [--base URL] [--model M] [--write FILE] [request]` | Generates harness configuration grounded in the live wrapper. |
| `fitevict scout` / `fitevict-scout` | Finds models outside the configured store. See [below](#find-models-outside-the-configured-store). |
| `fitevict-serve --host H --port P` | The front door. The unit runs this. |
| `fitevict-run` | The engine-juggle harness (`serve/run_engine.py`). |
| `fitevict-console` | The terminal console. |
| `fitevict-assist` | The small coreless assistant. See [below](#small-coreless-assistant). |

`fitevict tui|launch|chat` prints a pointer to the separate `hugpy-harness`
package, which provides the terminal UI and chat loop.

## Use

```
curl -s http://<host>:<port>/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"<name>","messages":[{"role":"user","content":"hi"}],"max_tokens":128}'
```

Any OpenAI client works: `base_url=http://<host>:<port>/v1`. Name the model;
the wrapper finds it, fits it, serves it and logs it. Variants and pins ride
on the name (`model@moe`, `@4bit`, `@max-gpu`, `@fit`, a quant), with or
without a core in front.

`/v1/models` always lists **HugpyHelper** (`hugpy` is also accepted). It is a
route model, not weights. It offers guided menus for the box's preferences and
per-model settings, and the `config [harness] [model]` command, which prints
the block that points a harness at this box. When the box names an
`assist_model`, plain sentences are mapped onto the menu by that model.

## API keys

Keys look like `hpk_<id>_<secret>` and are sent as `Authorization: Bearer …`.
That is what every OpenAI client sends as its `api_key`, so a user only has to
set `OPENAI_API_KEY=hpk_…`.

- **The box key** (`hpk_box_…`, admin) is written once by `fitevict install`
  into the service config. It is the root of trust for the box. hugpy-core
  holds it and uses it to manage the box.
- **More keys** are made through the running service with `fitevict keys …`.
  Revoked and expired keys stop working at once.

| route | rule |
|---|---|
| OpenAI routes (`/v1/chat/completions`, `/v1/models`, …) | `FITEVICT_REQUIRE_KEY`: `outside` (default: no key from loopback or private networks, a key from anywhere else), `never` or `always` |
| management (`/v1/telemetry*`, `/v1/queue`, `/v1/settings/*`, `/v1/operator/*`, `/v1/keys`, `/v1/cores`, `/v1/verdicts`, `POST /limits`) | an admin key, except from the box itself |
| `/health` | never |

What counts as inside is `FITEVICT_TRUSTED_NETS` (default: loopback, 10/8,
172.16/12, 192.168/16, fd00::/8, fe80::/10).

"The box itself" (0.1.110) covers loopback and also a peer that arrives on the
address the service listens on. A LAN-bound wrapper is therefore still managed
without a key by the harness on its own box, while the rest of the LAN needs
an admin key.

**Behind a proxy** (nginx, a tunnel), every request arrives from 127.0.0.1. A
request with forwarding headers counts as *outside* unless its immediate peer
is listed in `FITEVICT_TRUSTED_PROXIES`; only then is `X-Forwarded-For`
believed. A public proxy can never make the internet "local".

## HTTP surface

| route | what |
|---|---|
| `POST /v1/chat/completions`, `/v1/completions` | chat and completions (GGUF through llama-server, transformers through `tf_child`, vision through `vision_child`) |
| `POST` the task routes | see [Every model kind](#every-model-kind-one-front-door) |
| `GET /v1/models` | the catalog. HugpyHelper is always listed. |
| `GET /health`, `/v1/health` | `{"status","service":"hugpy-wrapper","worker","version"}`: the worker's identity, which central registers under |
| `GET /harness/v1/models` | the per-quant fit map that central's allocation reads. Variants are expanded when no preference is set. |
| `GET /models`, `/residents`, `/limits` (`POST` for central's settings), `/storage`, `/profiles` | catalog, residents, capacity tiers, drives holding models, env-profile state |
| `GET /v1/telemetry?table=&since=&limit=`, `/v1/telemetry/state[?wait=&since=]`, `/v1/telemetry/tables`, `/v1/telemetry/rows?table=`, `/v1/telemetry/resources` | cursor tables, long-poll state, a read-only table browser, resource snapshots |
| `GET /v1/queue` | the admission queue, with each ticket's eta and the reason the next one runs |
| `GET\|POST /v1/settings/pair`, `/v1/settings/box`; `POST /v1/settings/quant-facts`, `/v1/settings/env-profile` | per-pair knobs, the box blanket, quant facts, env profiles |
| `POST /v1/operator/actions`, `GET /v1/operator/actions/{id}` | `evict`, `evict-all`, `free-ram`, `remove-model`, `restart`, `refresh-models`, `pull-model` |
| `GET /v1/verdicts[?model_key=]` | pair verdicts per quant file and placement mode |
| `GET /v1/cores` | CPU topology, per-CPU load, engine claims, fitted constants |
| `GET /v1/drives` | drive roles, budgets, movers, per-drive load rates |
| `GET /v1/hashes`; `POST /v1/facts/import` | the sha256 of each weight file, hashed once; facts another worker measured |
| `GET /v1/transfer/{model_key}/{manifest\|file\|chunksums}` | the byte server central and other workers pull from (gated by the box key, `HUGPY_TRANSFER_MAX` concurrent) |
| `GET\|POST /v1/keys`, `POST /v1/keys/{id}/revoke` | API keys |

`python -m fitevict.observability.calls_view --port <n|auto>` serves a
read-only page of the latest calls on 127.0.0.1. It shows every call's bodies,
so run it as the service's user and keep it local.

## The store (telemetry)

The wrapper keeps every call and every measurement in one SQLite file, created
on first use. It lives in `FITEVICT_HOME`, else systemd's state directory,
else the per-OS data directory (0700 / 0600). Placement decisions read it
back: the measured CUDA context, llama.cpp's projections, serving growth, KV
structure, host memory, and a model's largest real use. Every measurement
points at the call that took it, and views compute the figures.

A damaged file is set aside and recreated; everything in it can be learned
again. Schema changes ship as numbered migrations in
`src/fitevict/observability/schema/`.

The store is quiet. Nothing is written that did not change. Sensor readings
(GPU, host and CPU every 250 ms) are stored without moving the change counter.
Events move it: a load, an eviction, a call row, an operator action, a process
on the card. Long-poll readers such as core's collector wake only on events.

The Core collector mirrors selected state snapshots on each sync and replays
`call_log`, `resource_log`, `evictions`, and `operator_actions` by source row ID.
Core derives across the fleet from these Wrapper records. Wrapper settings
such as `pair_settings` are not raw telemetry mirrors.

The pre-SQLite PostgreSQL migrations are archived in
`docs/original/legacy-postgres-sql/`. They are historical records and are not
part of the Wrapper's install or migration path.

## Every model kind, one front door

Chat (GGUF via llama-server, transformers via `tf_child`), vision
(`vision_child`) and every non-chat engine go through the SAME placement
(evict-to-fit), the SAME call log row and the SAME residents and models
catalog. Non-chat engines are runners on one generic child
(`lifted/task_child.py`). Each is one file in `lifted/runners/` plus one
`RunnerSpec`:

| route | runner | serves |
|---|---|---|
| `POST /v1/embeddings` | `embed` | sentence-transformers encoders |
| `POST /v1/audio/transcriptions`, `/v1/audio/translations` | `asr` | whisper (multipart `file`, or `file_path` / `file_b64`) |
| `POST /v1/audio/speech` | `tts` (chatterbox); `tts_kokoro`, `tts_dia`, `tts_parler` | raw audio bytes |
| `POST /v1/images/generations`, `/edits`, `/variations` | `imagegen` | diffusers image pipelines: `b64_json` (or `response_format: path`) |
| `POST /v1/videos/generations` | `videogen` | diffusers video pipelines (Wan, LTX, CogVideoX, Mochi, …): an mp4 path |
| `POST /v1/summarize` | `summarize` | seq2seq summarizers (bart, pegasus, t5, led) |
| `POST /v1/keywords` | `keywords` | KeyBERT over a stamped embedding model |

Binary outputs are written to `FITEVICT_ARTIFACT_DIR` and returned in the
endpoint's OpenAI shape. The call log stores their exact byte count and
sha256, never the bytes. Uploads are spooled to `FITEVICT_SPOOL_DIR` and
removed after the call.

**Env profiles** (`lifted/profiles.py`, declared in `runners.PROFILES`):

- A runner's extra pip dependencies live in a per-profile venv at
  `<engine>/envs/<name>`. That venv is either an overlay on the box's worker
  base (`FITEVICT_TF_PYTHON`) or, when a model pins its own torch (chatterbox),
  an isolated venv.
- `POST /v1/settings/env-profile` overrides a profile's declaration.
  `POST /v1/settings/pair` with `env_profile` pins a model to a profile.
- A profile is built on first use, off the request path. Until it is ready,
  the call fails as data (503 `profile_materializing`).
- `GET /profiles` reports every profile's state.
- Every profile event is an `operator_actions` row (`action = 'env-profile'`)
  attributed to its model.

### Serving a local PEFT adapter

1. Give a trained variant its own model directory and model name. Put in it
   the base model configuration, the tokenizer and the weight files (these may
   be symlinks).
2. Set `"adapter_dir": "/absolute/path/to/adapter"` in that directory's
   `hugpy.json`. A relative path is resolved against the model directory.
3. The adapter directory must contain `adapter_config.json` and
   `adapter_model.safetensors`. A missing adapter fails the launch instead of
   silently serving the base.

The transformers engine's `peft` dependency loads the adapter on CPU or GPU,
including over a 4-bit base. Memory pricing includes the unquantized adapter
at PEFT's default fp32 promotion size. Keep the adapter immutable under its
registered model name so cached sizing stays valid.

## Drives and the movers (0.1.111 and later)

`serve/drives.py` gives every drive a **role**:

- `central`: the source of record. Models are loaded from it, uploaded from
  it and copied to it. Never reaped.
- `worker`: the fast drive the box loads from, within its storage budget.
  Reaped to fit, but only for copies central holds exactly.
- `immutable`: read in place, never written.

Roles are resolved in this order:

1. the box knob `drive_roles`
2. the environment (`HUGPY_CENTRAL_DRIVES` / `HUGPY_WORKER_DRIVES`, as
   `path:max_gib,…`)
3. auto: `MODELS_HOME` is a worker drive, and `HUGPY_CENTRAL_MODELS_HOME` is
   a central drive

Per-drive budgets follow the capacity tiers (`drive_budgets`).

There are three **movers**. Each is set by its knob, else its env, else
implied by `reap_to_fit`, else off:

- `copy_to_central`: copies whole models from worker drives to a central
  drive. It is event-driven.
- `upload_to_worker`: on a call, copies the file being loaded from a central
  drive to a worker drive first.
- `reap_to_fit`: removes the least recently called worker copies whose exact
  duplicate (same sha256) is on a central drive. It never removes a resident,
  serving, protected or static copy.

**Under a core** (`HUGPY_CENTRAL_URL` is set, or hugpy-core is on the box),
`upload_to_worker` and `reap_to_fit` default on and `copy_to_central` stays
off. Wrapper-only builds keep all three off (0.1.120). Also under a core,
`central_serves` (`HUGPY_CENTRAL_SERVES`) defaults **off** (0.1.119): a load
must read a staged worker copy, or is refused with 503
`central_does_not_serve`. A wrapper-only build loads from central as before.

Every copy is the wrapper's verified pull: `.part` staging, sha256 against the
hash-once row, then an atomic rename. Each copy is recorded as a `pulls` row.
`load_estimates` (0.1.115) gives the box's own time to load and complete per
model, quant file, drive and placement class, and central's queue reads it.

## Cores

Every engine gets its own cores, claimed once at load from the plan's RAM:VRAM
split ([CORES.md](docs/original/CORES.md)):

- An engine with nothing in RAM gets one core, which is all it uses.
- An engine that reads weights from RAM gets as many cores as still make it
  faster, judged by the box's measured bandwidth curve. Until that curve is
  known, it gets every free core.
- Two engines never share a core.
- Cores busy with other work are avoided. One core stays reserved for the
  wrapper and the OS.
- An idle engine's cores can be taken by the next claim; the donor is
  re-pinned or evicted.

llama.cpp gets `-t N --cpu-mask … --cpu-strict 1`. CPU affinity confines the
engine process to its claim, which needs no privilege. Each call row records
the claim (`cpu_set`, `threads`, `cpu_source`) and the CPU time the call used
(`cpu_user_ms`, `cpu_sys_ms`). `GET /v1/cores` shows the topology, per-CPU
load, every engine's claim and the fitted speed constants. Per model, you can
raise the claim with `cpu_priority`, or pin it with `cpu_cores` (a count or a
CPU list), in `POST /v1/settings/pair`.

**Capacity tiers** (0.1.92, `lifted/capacity.py`) apply to VRAM, RAM, cores
and storage. The device minus unmanaged use must be at least the env
designation (`--vram-max` and friends), which must be at least central's
setting (`POST /limits`). A setting above its parent tier is refused with the
reason. `GET /limits` carries the tiers.

## Configuration

`fitevict install` writes these into the service config; you rarely set them
by hand.

| variable | meaning |
|---|---|
| `FITEVICT_HOST`, `FITEVICT_PORT` | where the front door listens (`auto` means a free port) |
| `FITEVICT_TOKEN` | the box key |
| `FITEVICT_REQUIRE_KEY`, `FITEVICT_TRUSTED_NETS`, `FITEVICT_TRUSTED_PROXIES` | who needs a key ([above](#api-keys)) |
| `FITEVICT_WORKER` | this box's worker name (default: the host name). The name is kept across reinstalls. |
| `FITEVICT_ADVERTISE` | the address central reaches this worker at |
| `FITEVICT_HOME` | the store's directory |
| `HUGPY_DATA_DIR`, `HUGPY_CACHE_DIR`, `HUGPY_ENGINE_DIR` | the data, cache and engine directories |
| `MODELS_HOME`, `DEFAULT_ROOT`, `FITEVICT_MODEL_PATHS` | the model store, the directory that contains it, and extra model roots (core's legacy-worker join writes these) |
| `EXCLUDE_DIR_NAMES` | Comma-separated directory names omitted from model discovery (defaults include `_archive`, `incoming`, and `dedupes`). |
| `EXCLUDE_DIR_PATHS` | Colon-separated absolute directory paths whose entire subtrees are omitted from model discovery. Symlink targets are checked too. |
| `LLAMA_SERVER_BIN`, `FITEVICT_TF_PYTHON` | explicit engines |
| `HUGPY_CENTRAL_URL`, `HUGPY_CENTRAL_KEY` | the central this worker registers with. Its presence also means "under a core". |
| `HUGPY_CENTRAL_DRIVES`, `HUGPY_WORKER_DRIVES`, `HUGPY_CENTRAL_SERVES` | drive roles and whether central drives may be loaded from |
| `HUGPY_VRAM_MAX_GIB`, `HUGPY_RAM_MAX_GIB`, `FITEVICT_CPU_MAX`, `HUGPY_STORAGE_MAX_GIB` | capacity designations |
| `FITEVICT_CPU_PIN`, `FITEVICT_CPU_RESERVE`, `FITEVICT_CPU_BUSY` | core claims on or off, cores kept back (1), the busy threshold (0.5). See CORES.md. |
| `HUGPY_QUANT_FALLBACK` | the box-wide ceiling over AUTO quant stepping: `auto` (default: every quant on the drive, descending), `off` (never step; the per-pair knob is ignored), or a comma list of quant labels in priority order. The per-pair `quality_alternatives` can only narrow or reorder inside it. |
| `HUGPY_QUEUE_POLICY`, `HUGPY_QUEUE_BYPASS_MAX` | the admission queue: `eta` (default since 0.1.74), `smart` or `fifo`; how many times a waiting head may be bypassed (default 3) |
| `HUGPY_TRANSFER_MAX` | concurrent byte-server transfers (default 3) |
| `FITEVICT_HEALTH_WAIT_S` | the installer's health wait (default 180 s) |

Set any of these at install with `fitevict install --env KEY=VALUE`
(repeatable). They are rendered into the service's env file like the rest.

## Auto policy

When a call arrives by model name alone (no pin), the front door places it
**Auto**. The fallback **rung order**, per candidate artifact, is:

```
gpu-only → MoE CPU-expert split (MoE models) → 4-bit → ram-only → max-ram → max-gpu
```

When every rung of an artifact fails, Auto **steps the quant down** to the
next approved variant of the same model and restarts at gpu-only.

- **4-bit** means the approved 4-bit representation of the *same* model. For
  transformers, that is the bnb 4-bit runtime quant of the artifact. For GGUF,
  it is an approved Q4-class variant file in the catalog, and the rung is
  skipped when there is none.
- When every rung is exhausted, the result is terminal with a specific reason:
  - **resource**: steps the quant
  - **bad artifact** or **incompatible engine**: terminal, never a quant change
- An identical failed plan (artifact, mode, ctx, KV type) is not retried in
  the same request without evidence that the room changed.
- The stepping runs **live** in the placement driver. Each variant is loaded
  with its own facts, and the call row records the artifact that actually
  loaded.
- The box env `HUGPY_QUANT_FALLBACK` is the **ceiling** over which variants
  may be stepped to. The per-pair `quality_alternatives` can only narrow or
  reorder inside it. The applied policy is written to the decision and
  `plan_note`.

Concurrent calls enter placement through an **admission queue**
(`HUGPY_QUEUE_POLICY`: `eta` by default, or `smart` or `fifo`).

- **eta orders by shortest predicted time-to-output**, the sum of:
  - `ready_in`: how long until the call's resources free up
  - `load`: the model's measured time to loaded, 0 when resident
  - `output`: prompt tokens × measured ms per prompt token, plus expected
    output tokens × measured ms per output token
- A model with no decode history yet keeps its arrival slot.
- A group (session) is never reordered.
- Background tickets run after every foreground one. A call is background when
  its `metadata.intent` is `grading`, `smoke-test`, `benchmark` or `warmup`,
  or its `metadata.priority` is `background`.
- A ticket that has waited its `max_wait_s`, or whose predicted start would
  pass it, is next in line.
- `GET /v1/queue` shows the live queue. Each call row's queue block records
  the call's position, kind, eta and why it ran.

**Placement is serialized; serving is not.** The placement lock covers only
the evict and load decision. Once a call is serving, its model is pinned busy:
never an eviction victim, never shrunk. Calls to the SAME model queue on its
serve slot, which is 1 for a transformers child and `FITEVICT_PARALLEL` for
llama-server.

A rung whose weights live in host RAM (the MoE split, ram-only, max-ram or
max-gpu) is admissible only if its CPU claim would get at least
**`cpu_floor_cores`** free *physical* cores. Memory fit alone never admits it.

Among the admissible candidates, the **goal profile** ranks them, minimizing
left to right:

- `quality`: `(q, T+R, T)`
- `fast`: `(T, q, R)`
- `resident`: `(q, E_hot, T+R)`

Where:

- `q` is the index in the approved-representation list (0 = the selected
  one).
- `T` is the estimated seconds to complete, from the box's measured rates.
  Unmeasured candidates fall back to ladder order; a bandwidth is never
  invented.
- `R` is the reload cost of the residents a candidate would displace.
- `E_hot` is the number of pinned or high-priority residents displaced.

Waiting is a candidate only when a resource change is expected within
`max_wait_s`.

Context has three explicit states per (model × mode × KV type):

- **range**: shrinks over `[feasible_min, feasible_max]`
- **point**: a single feasible tick; shrink is off
- **infeasible**: no tick fits the worker budget. The model is refused, never
  loaded as a point.

A resident loaded below its `ctx_max` **grows back** with an idle rebuild when
all of these hold: there is demand, there is sustained headroom with a margin,
a cooldown has passed, and the resident is idle (`ctx_restore`).

Per-pair knobs (`user_settings`, sent by core through
`POST /v1/settings/pair`; an absent knob means Auto). The box blanket
`GET|POST /v1/settings/box` holds box-wide defaults that every pair inherits
key by key.

| knob | values | default |
|---|---|---|
| `ctx_pct` | 1–100 (% of trained → applied ctx_max) | Auto = feasible_max |
| `ctx_min_pct` | 1–100 (% of trained → applied ctx_min) | Auto = feasible_min |
| `ctx_range_enabled` | bool: range on or off (old `ctx_range` is still read) | Auto rule |
| `ctx_yield` | bool: shrink on or off | Auto rule |
| `ctx_restore` | bool: grow a shrunk resident back to ctx_max | Auto = on when shrink is on |
| `auto_goal` | `quality` \| `fast` \| `resident`; ranks within GPU-only, mixed, then RAM-only tiers | `quality` |
| `quality_alternatives` | `auto` \| `none` \| an ordered list of variant keys | `auto` |
| `allowed_placements` | a subset of gpu-only, moe, 4bit, ram-only, max-ram, max-gpu | all |
| `wait_before_quality`, `resident_priority`, `latency_target_s` | the blanket preferences | unset |
| `cpu_floor_cores` | integer ≥ 1 (physical cores floor) | `4` |
| `max_wait_s` | seconds (bounded wait budget) | `120` |
| `alloc_mode` | `gpu-only` \| `ram-only` \| `max-gpu` \| `max-ram` \| `explicit` (pin); use `ram-only` to request RAM first | Auto |
| `moe`, `kv_cache_type` | MoE expert split; `f16`/`q8_0`/`q4_0` | Auto; `f16` |
| `env_profile`, `cpu_priority`, `cpu_cores` | the env profile, the CPU claim | unset |

The math behind these is in [FORMULAS.md](docs/original/FORMULAS.md).

## Glossary

These are **shared names** for quantities the code already carries. The DB
columns and API fields **do not change**; this table maps the existing names
to the shared ones.

| shared name | the existing quantity |
|---|---|
| `worker_vram_budget_bytes` | the worker budget: the card with nothing of ours resident |
| `available_vram_bytes` | current usable headroom **after** the reserve (not raw NVML free) |
| `worker_feasible_context_max_tokens` | `feasible_max`: the largest ladder tick that fits the worker budget |
| `auto_context_floor_tokens` | `feasible_min`: the smallest tick ≥ 0.25 × the trained window |
| `context_shortfall_tokens` | `theoretical_max − feasible_max`, in tokens |
| `target_context_tokens` | `ctx_max`: the upper handle (where a load starts) |
| `minimum_retained_context_tokens` | `ctx_min`: the lower handle (the shrink floor) |
| `resident_context_tokens` | the loaded engine's capacity (a resident's ctx) |
| `model_context_ceiling_tokens` | `theoretical_max` (the trained window, `ctx_train`) |
| `model_context_floor_tokens` | `theoretical_min` (the ladder's smallest tick) |
| `tok_bytes_gpu` / `tok_bytes_ram` | weight bytes **read per generated token** (the split), not context growth |
| `non_expert_bytes` | estimated GPU weight bytes **including** GPU-resident experts |
| `free_vram_before` / `free_vram_after` | the budgetable headroom, not raw NVML free |

## Layout

```
fitevict/
  core/           pure allocator: types, evict (victim selector), plan (staged fit + quant ladder),
                  flex (ctx-band compress, layers-that-fit), host (EngineHost + drive loop)
  cores/          the CPU side of a slot: topology, usage, rates, claims (CORES.md)
  estimate/       sizing, predict, ctx_probe, ticks
  serve/          front_door (the OpenAI server), manage (management API), auth (API keys), admission,
                  eta, drives, transfer, worker_pull, hashes, verdicts, variants, helper (HugpyHelper),
                  assist, gpu_state, run_engine, worker_storage_ops
  observability/  store (SQLite) + schema/ migrations, resource_log, calls_view, blobs
  cli/            installer (`fitevict`), console, assist, scout, setup_harness, setup_client, endpoint
  adapters/       llama_cpp (measure / price / launch / evict), transformers, vision, tasks, marker,
                  db (call log + measurements over the store)
  lifted/         helpers lifted from the hugpy monolith: gguf_inspect, gguf_need, hardware, capacity,
                  dequant, model_discovery, model_resolve, profiles, task_child, runners/, app_dirs, …
  imports/        shared constants / functions / classes (no heavy imports; docs/packages.md#the-imports-layout-convention)
```

More docs:

- [Documentation index](docs/README.md)

- [MIGRATION.md](docs/MIGRATION.md): the package layout
- [MIGRATION-imports.md](docs/MIGRATION-imports.md): the shared imports layer
- [MIGRATION-store.md](docs/MIGRATION-store.md): the store, management API,
  installer and API keys
- [CORES.md](docs/original/CORES.md): CPU claims
- [FORMULAS.md](docs/original/FORMULAS.md): the sizing, eta and load formulas

## Small coreless assistant

`fitevict-assist` searches a drive with a bounded filesystem walk. It then
uses the wrapper's direct OpenAI endpoint for optional small-model ranking and
chat. It does not require hugpy-core. The `bootstrap` command can fetch Qwen's
official Apache-2.0 Qwen2.5-0.5B-Instruct Q4_K_M GGUF (491 MB) into
`MODELS_HOME`. The download is checked against its published SHA-256 before
it is made visible.

```sh
fitevict-assist bootstrap
fitevict-assist scan --query 'small coder'       # discover mounted drives automatically
fitevict-assist scan /mnt/models --query 'small coder'
fitevict-assist scan /mnt/models --query 'specific model name' --add
fitevict-assist add /other/drive/my-model
fitevict-assist call 'Which model did we just add?'
fitevict-assist call 'Explain this model' --model My-MoE-GGUF \
    --quant Q4_K_M --alloc max-gpu --moe --ctx high --knob n_cpu_moe=12
```

`scan`:

- Without a root argument, it checks mounted drives, Hugpy stores, Hugging
  Face caches and known local harness locations. It also checks model paths
  in running llama.cpp, vLLM and KoboldCpp command lines.
- Each result records the source hints that led to it. These hints are
  verified against files on disk; a hint alone never makes an unknown model
  runnable.
- Scanning has depth, directory-count and result limits.
- Unsupported weight-only folders are shown but refused by `add`.

`add`:

- makes a catalog symlink; it does not copy model weights
- establishes `hugpy.json` from disk evidence, beside the model if the
  directory is writable, otherwise in a wrapper sidecar
- requests an immediate catalog refresh

`call` passes model options and explicit per-call knobs to the wrapper, so its
fit and feasibility checks stay in force. The assistant never executes a path
or command returned by the small model.

## Find models outside the configured store

`fitevict scout` searches mounted drives and known model caches with a bounded
directory walk. It reports GGUF, Transformers and Diffusers model directories,
including model paths named by running llama*, Ollama, vLLM, ComfyUI and
Kobold processes. It also reads local Ollama `/api/ps` and `/api/tags`, Ollama
manifests and service journals, and past model paths in the wrapper SQLite
store. Historical evidence is labeled separately from live residency. It
does not load weights or need a language model.

```bash
fitevict scout --root /mnt/16T_toshiba/llm_storage
fitevict scout --root /mnt/nvmes/2T_samsung_990/hot990/aeb/models \
  --attach /path/to/owner/model --models-home /var/lib/hugpy/models
fitevict scout --pid 1507486
```

When the GPU watcher discovers a PID outside its own residents and probes, a
background scout checks that process's model arguments, open files, memory maps,
parent, nearby directory, and local Ollama API. The finding and GPU bytes per
card are stored in `scout_observations` and exposed at
`GET /v1/telemetry/scout`. Scouting is read-only and does not alter placement.

`--attach` names one model directory and may be repeated. It creates a link in
the wrapper's owner/repo model layout and attempts to derive a `hugpy.json`
marker from the model's on-disk evidence. Its JSON result reports whether the
marker was available. The external weights stay in place. Afterwards, refresh
the running wrapper's catalog with its `refresh-models` operator action, or
restart the wrapper.

### Automatic local discovery

The catalog checks conventional model directories and Hugging Face hub caches:
`HF_HOME`, `HF_HUB_CACHE`, `HUGGINGFACE_HUB_CACHE` and `TRANSFORMERS_CACHE`.

- It follows cached snapshots and checks indexed weight shards.
- External discoveries reuse the files in place.
- Search is bounded to common locations, five directory levels and 4096
  directories.
- `EXCLUDE_DIR_NAMES` prunes named directories from discovery; it does not
  remove files or prevent transfers into those paths.
- `EXCLUDE_DIR_PATHS` prunes exact configured directory subtrees, including
  model paths reached through symlinks.
- Since 0.1.118, systemd automount points with nothing mounted are skipped,
  so discovery never wakes an absent disk.
- Every `hugpy.json` marker is checked against the on-disk evidence and
  re-derived when it disagrees (0.1.84).
- Each refresh publishes the validated marker and the wrapper's local file,
  engine, runner and quant inventory together in `model_discovery_facts` and
  `model_discovery_files`. Readers reuse this DB snapshot; an unchanged scan
  causes no write event. The per-file rows carry hashes, topology and static
  grades from `weights_facts` where the marker verifies against the local files.
  Load measurements remain in their measured fact tables and can refine the
  static sizing facts after a model runs.
- Projector files are identified by their GGUF header and never listed as
  models (0.1.113).
- A file whose first four bytes are `GGUF` is served as a GGUF whatever its
  name (0.1.122).

Private or unusual paths still need configuration and service read access.
Discovery confirms local files and engine imports, not successful inference.

## Optional terminal harness

The TUI, chat loop, sessions and agent tools are packaged separately as
[`hugpy-harness`](../../hugpy-harness/README.md). Installing `hugpy-wrapper`
neither installs nor runs them. The harness connects over HTTP, so it also
works from another host.

`fitevict setup --harness hugpy-harness --model MODEL_ID --write connection.json`
generates a connection configuration. Launch it with
`hugpy-harness --config connection.json`. The same command writes
configuration for aider, OpenCode, Continue and plain shell `env` without the
Hugpy harness. `/v1/telemetry/resources` supplies the resource snapshots that
the harness shows with `/resources` and `/resources full`.

## Test

```
pip install "hugpy-wrapper[test]"
pytest          # every test runs on its own empty store; nothing touches a real one
```

See [development](../../../docs/development.md) for the workspace flow.

## Release notes

Per-release notes from 0.1.110 on are in the `CHANGES-0.1.<n>.md` files next
to this README. 0.1.110 also covers 0.1.103 to 0.1.109. Short form, newest
first:

| version | change |
|---|---|
| 0.1.122 | A file whose header is `GGUF` is served as GGUF whatever its name. Used for Ollama blobs adopted through a symlink view on a-brain. |
| 0.1.121 | A quant rung without a resolvable footprint fails open instead of raising a 500; 500 handlers log their traceback. |
| 0.1.120 | Under a core, `upload_to_worker` and `reap_to_fit` default on; `copy_to_central` stays off. |
| 0.1.119 | `central_serves`: under a core, a load must read a staged worker copy (`503 central_does_not_serve` otherwise). |
| 0.1.118 | Discovery never wakes an empty automount. The installer's health wait goes from 60 s to 180 s. |
| 0.1.117 | Every call row carries `derived_v`, so central re-pulls rows that were backfilled. |
| 0.1.116 | Page-cache residency is read only for files the service owns; any other file counts as unknown. |
| 0.1.115 | `load_estimates`: time to load and complete per model, file, drive and placement class. |
| 0.1.114 | A drive's load time is a fixed cost plus a per-byte slope (Theil–Sen). |
| 0.1.113 | Projectors identified by header; every call row carries its derivations; upload time derived from the call log. |
| 0.1.112 | HugpyHelper menus keep you in them; a sentence goes to the assist model. |
| 0.1.111 | Drive roles (central, worker, immutable) and the three movers. |
| 0.1.103–0.1.110 | Local callers need no key; the box blanket; ranked time-to-output; HugpyHelper (the route model), its trained 1.5B helper and `config`. See CHANGES-0.1.110.md. |

The notes below were kept in this README for 0.1.63 to 0.1.102, newest first.

### A stop is bounded (0.1.102)

SIGTERM unloads the residents under the placement lock; with the lock held by a placement
(or a long refresh) the stop sat until systemd's 90 s kill (ae, 2026-10-14, stopping the
worker to swap its drive). The lock is now taken with a 15 s bound and the residents are
unloaded regardless.

### The GGUF engine is a package (0.1.101)

`hugpy-wrapper` now Recommends `hugpy-llama-server`: llama.cpp's `llama-server` (CUDA) under
`/opt/hugpy-llama`, which the engine discovery finds (`/opt/*/bin/llama-server`). Before, a
fresh install served GGUF only if a legacy build happened to sit under
`/srv/*/.local/share/hugpy/engine` (ae, 2026-10-14: cleaning that directory made every
GGUF call fail with FileNotFoundError).

### A pull never duplicates what the worker holds (0.1.100)

`pull-model` is refused when this worker already holds the bytes anywhere (its own
`weight_hashes`, any key): a second copy of the same model re-keys it (discovery suffixes
duplicates) and the fleet loses the model's identity — seen on ae pulling Echo-Mini it
already held on another drive.

### A re-run keeps the host; `pulls` is a cursor table (0.1.99)

`fitevict install --system` without `--host` keeps the configured FITEVICT_HOST the way
it keeps the port and the token — the apt upgrade's postinst had put a LAN-bound worker
(joined by `hugpy-core workers add`) back on loopback, where its central could not reach
it. `pulls` joins the cursor tables central mirrors (`/v1/telemetry?table=pulls&since=`).

### A worker pulls a model on central's order (0.1.98)

Operator action `pull-model` {model_key, params: {source_url, headers, file}} — central
names the source (its own verified copy on a central drive, served at its `/v1/transfer`,
or another worker holding the files) and this worker pulls with the stdlib puller
(`serve/worker_pull.py`): Range reads per chunk, each chunk's sha256 against the source's
chunksums as it lands, the whole file's sha256 against the source's, a `.part` staging file
resumed where its chunks verify, an atomic rename. The copy lands in this worker's own
model root as `<MODELS_HOME>/<owner>/<repo>/` with the marker beside the weights, then
`refresh-models` discovers it and the hasher proves it. Refused when the files are already
here or there is no room within the storage tier ("reap first"). Every file is a `pulls`
row (migration 0013: bytes, seconds, ok, error — the measured rate central reads).

### Derivation rules are part of the marker signature (0.1.96, 0.1.97)

A sidecar marker is re-derived when the model's config signature changes — and now when
the derivation rules change (`DERIVE_RULES` in the signature), so 0.1.95's cache-snapshot
identity reaches the markers 0.1.93 derived. 0.1.97: the re-derivation happens ON READ
of a sidecar whose signature is stale (before, only `ensure_marker` re-derived, and a dir
that already had a sidecar never reached it). A `hugpy.json` beside the model is never
touched.

### A huggingface cache snapshot is a model too (0.1.95)

A model that lives in `~/.cache/huggingface/hub/models--owner--repo/snapshots/<rev>` is
symlinks into the repo's `blobs/`. The byte server's listing skipped every symlink, so
its manifest was empty and central's copier could never advance past it; the derived
marker named it `snapshots/<rev>`. Now the `models--owner--repo` dir is part of the path
jail (`jail_roots`), the snapshot's files are served through their blobs, and the marker
derives `owner/repo` from the cache dir's name (`hub_identity`). Hidden files (`.cache/`,
a downloader's `.lock`, `.gitattributes`) are never offered: they are not the model.

### kv_seqs is a number (0.1.94)

llama.cpp's newer load log says `1/1 seqs`; the K2 reader stored that text in
`kv_structure.kv_seqs` (declared INTEGER — SQLite keeps what it is handed) and central's
typed mirror then refused the row and the whole sync with it. The reader and the writer
now bind the first number; migration 0012 repairs what 0.1.93 stored.

### The box serves its own files; a remote central; storage over the drives (0.1.93)

Phase 4 D2 (operator 2026-10-14): central and other workers pull a model's files FROM THE
WORKER with `abstract_transfer` — `GET /v1/transfer/<model_key>/manifest`, `…/file?path=`
(HTTP Range → 206, 1 MiB stream), `…/chunksums?path=` (32 MiB sha256 chunks, sidecar-cached),
a path jail inside the model dir, one concurrency cap per box (`HUGPY_TRANSFER_MAX`, 3) —
lifted from the orbit's `hugpy_storage.transfer_api`, gated by the box key. Join 3: `fitevict
install --central URL --central-key KEY [--advertise HOST]` registers the worker with a REMOTE
central itself (POST /api/fleet/workers); a loopback-bound wrapper refuses, said so. The
storage tier (0.1.92) now sums every drive our models live on, not only the shared store's.

### Capacity tiers (0.1.92)

Operator 2026-10-14: device ≥ env designation ≥ central's setting, for VRAM, RAM, cores and
storage, in the resource's smallest unit, with unmanaged encroachment charged to the device.
`lifted/capacity.py`; install flags `--vram-max/--ram-max/--cpu-max/--storage-max` (refused
above the device); `GET /limits` carries `tiers`; `POST /limits` accepts `cpu_limit` and
`storage_limit` and REFUSES a setting above its parent tier with the reason. The VRAM
designation caps the card; the cores cap cuts the claim's free list. See CORES.md and
/srv/hugpy/directions/auto-multiplexing/response/"3 - capacity tiers (2026-10-14, proposal).md".

### The fit map honours the cores (0.1.90)

`/harness/v1/models` (the fit map central's allocation reads) drops the host-resident
placements of every quant when the box's cores say no — the same rule as the front door's
gate and the verdict cells — so central never allocates a RAM placement the box would
refuse. See CORES.md.

### The `compressed` profile is declared by the package (0.1.89)

A fresh install could not load GLM-4.7-Flash-NVFP4: the env profile that gives the child
`compressed-tensors` had been made by hand in the store and the purge took it. The
profile is now DECLARED in `lifted/runners.PROFILES` and a compressed-tensors checkpoint
maps to it by default (`adapters/llama_cpp.default_profile_for`); the first load kicks
its build (503 `profile_materializing`, a fact), `/profiles` reports it ready (~20 s),
the next load runs from it. The pair knob `env_profile` still overrides.

### Hashes once; pairs known elsewhere (0.1.87) · evict-to-fit over cores (0.1.88)

0.1.87: every weight file is hashed ONCE (`weight_hashes`, the hasher thread woken by the
model map; `GET /v1/hashes`) — content identity for the fleet model DB's exact-duplicate
rule; and `POST /v1/facts/import` lets central hand this box the facts another worker
measured for a (model, quant file) it does not hold, so `GET /v1/verdicts` prices those
pairs too (`on_disk: false`, `origin_worker`). 0.1.88: an idle engine's cores are
obtainable by the next claim (donor re-pinned or evicted), the constant CPU floor is
gone (the knob, else S3, else no basis), the gate never fails open and refuses a
single-thread start beside busy engines, and every host-resident verdict cell carries
its `cores` (wanted / obtainable / predicted tok/s). See CORES.md.

### The wrapper defines the worker; central finds it (0.1.85, 0.1.86)

Operator 2026-10-14: "the wrapper install should be what defines a worker. and that
worker should either be discoverable by an existing central, or a post installed
central." `/health` now carries the worker's identity — `{"status": "ok", "service":
"hugpy-wrapper", "worker": <FITEVICT_WORKER>, "version": <installed>}` — and central
registers a wrapper under the name it reports (0.1.85). The install's tenth step,
`central`, registers this worker with a central already on the box: when
`/etc/hugpy/core.endpoint.json` exists it calls `hugpy-core register --url --key
--name`; a central installed later reads `wrapper.endpoint.json` itself; a central
elsewhere runs `hugpy-core register`. A central that refuses never fails the wrapper
install — the worker is usable on its own (0.1.86). A worker keeps its name across
reinstalls; its port may change, and central accepts the move because the wrapper at
the new address says who it is.

### Discovery evaluates every marker; a load no longer blocks the box (0.1.84)

Operator 2026-10-14: "is discover models just taking whatever? how is it not evaluating the
hugpy.json for correctness?" It was: the file beside the model was central's stamp and trusted
as long as it parsed; only JSON damage was recovered and one lie (a text-generation stamp the
config contradicts) overridden. A 2026-07-27 stub with `tasks: null` passed. Now
`marker.evaluate_marker` compares every marker with the on-disk evidence (`classify`: framework,
tasks, primary task, MoE) and the engine's abilities (`transformers.bnb_available`: can this
engine run bitsandbytes at all) and with the marker schema (`marker_schema`, now 2). A marker
that disagrees — or predates the schema — is RE-DERIVED from disk, the hub-origin facts the disk
cannot produce carried over (hub_id, hub, manifest, admission, …), the stale stamp recorded under
`superseded` with the reasons, and written beside the model (sidecar when read-only); the
catalog row says `hugpy_json_source: rederived` and why. Evaluated once per (dir, config,
marker mtime). A 4-bit placement is offered only when the engine has bitsandbytes (DAN-Qwen3-1.7B
had been offered one and failed on ImportError).

And the placement flaw: a six-minute load held the serialized placement section, so three calls
that fit now waited behind it. `front_door._yield_placement` hands the section on while a
launched child loads (lock and queue released for the wait, re-taken for the load's
bookkeeping); the loading model is pinned busy (never an eviction victim) and its launch budget
stays reserved out of the room (`inflight_reserved_bytes` counts loading children of every
engine). The host brackets its health wait with `_load_wait` (sets `child.loading`).

### 0.1.82 shipped a LAN relax of the management routes; 0.1.83 reverts it

The management routes (`/v1/telemetry*`, `/v1/queue`, `/v1/settings/*`, `/v1/operator/*`, `/v1/keys`,
`/v1/cores`, `/v1/verdicts`) are machine-to-machine: loopback, or an admin key (core and the
router hold the box key). 0.1.82 let the trusted networks manage without a key; it contradicted the
designed rule and its tests and was reverted the same night. An operator acts through the console's
login, never through this door.

### Nothing is written that did not change; readings are silent, events are loud (0.1.80–0.1.81)

Operator 2026-10-14: "please don't heartbeat, this is long since deprecated." The store's
change counter is what every long-poll reader wakes on (`GET /v1/telemetry/state?wait=N&since=K`:
the router's box link, hugpy-core's collector). The residents publisher used to rewrite
every seat on each tick with a fresh `observed_at`, so the counter moved every second
while serving and every ten seconds idle — a heartbeat in all but name, and every reader
re-synced on it. `db._replace_rows` now diffs against the stored rows: an identical set is
not a transaction, an unchanged row is not rewritten, and the stamp column means "last
changed". A quiet box moves the counter only when something actually changed — and (0.1.81) only
when that something is an EVENT. The GPU / host / CPU watch (every 250 ms, NVML and
/proc) and a serving resident's live bytes are SENSOR READINGS: they are stored
(`store.transaction(notify=False)`) but move no counter, so a card's bytes drifting, the
host's page cache moving or a core's busy fraction changing wakes nobody. Events are what
wake readers: a load, an eviction, a call row, a fact, a resident's state or seat changing,
an operator action, or a process appearing on / leaving the card (the watch promotes that
one reading to an event). Measured before: 81 counter moves in 20 s on an idle box;
after: none.

### The ask is compared, never the rung (0.1.79)

0.1.78 compared the resident's request with the NEXT call's contract as it stood inside
the auto ladder — already carrying that attempt's rung (`alloc_mode`, `moe`,
`n_cpu_moe`) and the window derivation (`ctx_start`, `ctx_ladder`, `ctx_need`) — so two
identical Auto calls still read as "knobs changed" and a 60 GB resident was reloaded
on every call. `_request_view` now strips everything a placement derives (keeping a
pinned / preset / retry window, which is part of the ask) and `_place_locked` compares
the new call's own snapshot of its ask with the resident's.

### A refused window is a measured fact; a busy resident is never torn down (0.1.78)

Two things the GLM first-call run showed. (1) The probe took three points between 3k
and 7k tokens and a 54k fill that OOMed the card; the line through the three points
then said 141k fits. A line beyond its last point is a projection, not a measurement:
`sizing.window_refused` now keeps every window at or above a failed probe point of a
variant out of reach (Auto, the pinned planner, `ctx_fits_budget`) until a SUCCESSFUL
reading at or above that window exists — for every variant (gpu / gpu:bnb4 / moe), not
only GGUF gpu-only. (2) A foreground prompt arrived while a grading call was being
generated; the Auto ladder had written its outcome (`alloc_mode = ram-only`) into the
resident's contract, so the new Auto call read "knobs changed the load contract" and
evicted the resident mid-generation (the grading call answered 502). Now the compare is
against the REQUEST the resident answered (`contract_requested`, taken before the auto
ladder runs), so Auto's own outcome never reads as a knob change; and a resident that
truly must be reloaded under another contract is never stopped while a call is on it
(`_wait_idle_for_replacement`: wait within the ticket's `max_wait_s`, then fail the new
call as data with 503 and leave the resident serving).

### Fused experts are materialized from the packed checkpoint (0.1.77)

Measured 2026-10-14 on GLM-4.7-Flash-NVFP4 (transformers 5.3, compressed-tensors
0.19): the first generation under the expert split answered "15 com Comments com …".
The load log said why — transformers fuses each layer's per-expert projections into
one `experts.gate_up_proj` / `experts.down_proj` tensor, but has no converter for the
per-expert NVFP4 `weight_packed` / `weight_scale` / `weight_global_scale` tensors: it
reported them UNEXPECTED and the fused tensors of layers 2–45 MISSING (newly
initialized). Forty-four of forty-seven MoE layers served random experts. The child now
runs three materialization steps after `from_pretrained`, before any placement
(`lifted/dequant.py`): `materialize_compressed` (eager decompress of the matched
modules + offload wrappers removed, 0.1.76), `materialize_packed_experts` (every fused
expert tensor whose layer has packed per-expert weights in the checkpoint is filled,
expert by expert, with compressed-tensors' own unpack + dequantize — bit-identical to
its `NVFP4PackedCompressor.decompress` — in the fused `[gate; up]` order), and
`dequantize_float8` (the FP8 fused experts, 0.1.75). A tensor whose checkpoint shape
does not match the fused one is left as loaded and reported. The resident is then the
plain bf16 model the load history prices (GLM: 59.9 GB counted — 4.4 GB on the card,
55.6 GB of experts in RAM).

### No ratios left on the sizing path (0.1.76)

Operator 2026-10-13: "make sure absurd constants are not being applied." An audit of
the live sizing path found one residual ratio: the pinned / partial planner budgeted
host RAM at 95 % of the measured free figure (`Policy.ram_safety_frac = 0.95`), the
rung the transformers expert split takes. It is now 1.0 — RAM is budgeted at the
measured MemAvailable with swap as the second tier (`sizing.host_fit`), exactly like
VRAM. Weight and KV terms were already exact or measured (GGUF margins 1.0, cushion 0,
VRAM / RAM reserves default 0, VRAM safety 1.0); the "x1.15 header formula" survives
only as a label in the full core's estimate fallback, which the basic core does not
load. And one more "measurement wins" rule, the transformers twin of the GGUF K2
rewrite: once the child has served two or more calls whose cache tally grew with the
tokens held, the observed bytes per token (`kv_bytes_engine / slot_tokens`, median of
the last 20) replace the config-geometry ladder (`basis: transformers child cache
(observed, N calls)`); a preallocated cache that reads the same bytes at every token
count is not taken as a per-token fact.

### The grader writes the facts from reality (0.1.75)

Operator ruling 2026-10-13 (evening): "On the very first successful (or
failed-but-informative) load of a model, create a persistent load history record for
that model + placement class. No prior synthetic 'facts' size is trusted after that
point." The packed on-disk size of a compressed-tensors dir (NVFP4 / FP8 + scales) is
not what loads — transformers materializes it as bf16 (GLM-4.7-Flash-NVFP4: 21.26 GB
packed, 57.85 GB resident) — and the planner believed the packed number.

- **The child counts its weights.** `tf_child` / `vision_child` tally the resident's
  parameters and buffers per device after loading (decompression, dequantization, 4-bit
  packing, adapters, the expert split included) and serve them on `GET /props` as
  `weights {gpu_bytes, cpu_bytes, total_bytes, params, dtypes}`.
- **The loading call row carries them** (doctrine 012: the row holds the raw
  measurement, the fact table points): `weights_bytes` is now the counted total, with
  the split in `weights_gpu_bytes` / `weights_host_bytes` (new columns, migration 0010),
  the process VRAM after the load and the host RSS beside them. A failed load points
  too (its row says which rung refused) but never displaces a measured load.
- **The history is keyed by placement class** — `tf_measure_facts (worker, key,
  variant, ctx)`, variant `gpu` / `gpu:bnb4` / `moe` / `ram` — backfilled from the
  pointed load rows. `tf_measure` resolves the counted weights, the load and peak
  bytes, the host RSS and the load's outcome.
- **Sizing reads the history first.** `sizing.tf_need` prices a class from its own
  counted weights (the whole model on the card; the split's card weights + experts in
  RAM; RAM-only's host weights); a class that never loaded is priced from any class's
  measured total; only before any measured load does the safetensors header figure
  stand. The launch budget (`adapters.transformers.footprint`, the expert split's
  `--max-memory`) and the core claim read the same history.
- **The packed hint learns.** `tf_facts.load_bytes` sizes a compressed-tensors dir at
  its bf16 materialization (`weight_packed` ×4, FP8 `weight` ×2, scales 0) so the very
  first Auto decision already steps past the card; a `tf_structure` row written before
  0.1.75 is refreshed once.
- **The grader is the measurement engine.** A grading or smoke-test call still drives
  the first load and the post-serve probes, but the probe now sweeps the DEFAULT
  window ± `HUGPY_PROBE_SPAN` ticks (default 1) and the requested window
  (`ctx_probe.targets(ladder, default, requested)`), keys a transformers expert-split
  resident as its own `moe` class, and — the exemption — once the resident's history is
  complete (`ctx_probe.complete`: every point has a row, the full-context fill
  measured) grading and smoke-test calls skip the probes. Real traffic keeps refreshing
  the call log.
- **Leftover FP8 is dequantized in the child** (`lifted/dequant.py`): a
  compressed-tensors checkpoint whose fused expert tensors no scheme matched keeps
  float8 weights no Ampere card or CPU matmul takes (`grouped_mm … got
  Float8_e4m3fn`); after `from_pretrained` every float8 parameter is dequantized with
  its `weight_scale` (per expert, read back from the shards when the module no longer
  carries it). A parameter whose scale cannot be found is left float8 and reported.

GLM-4.7-Flash-NVFP4 stays in the catalog (operator: "a model that is called is a
model that is called; only cull if it is broken or cannot fit any resource
allocation") — Auto sizes it from the history and takes the expert split.

### Time-to-output is the queue's goal (0.1.74)

Operator ruling 2026-10-13: "time-to-output should be the goal over fifo; a base time to
process-init should be predetermined for any model in the queue — the historical time to
loaded once eviction/occupancy allows; historical time to output for all occupants
processing; if unknown then fifo for that initial call; a maximum window after which the
queued item is auto next in line." `serve/eta.py` reads those figures off the call rows
the box already keeps (nothing new is measured): `load_s` = median
`weights_loaded_at - weights_staged_at`, `prompt_ms` / `decode_ms` = median per-token
timings, `out_tokens` = median emitted tokens (capped by the call's `max_tokens`). The
front door tracks in-flight calls (`_INFLIGHT`: started + predicted seconds) and the
queue's estimator turns them into each ticket's `ready_in_s`. `admission.choose_next`
policy `eta` orders by `eta_of` = ready-in + load + output, keeps unknown models in their
arrival slots, keeps a group's order, puts background tickets last, and promotes a
ticket whose window (`max_wait_s`, default 120 s) has passed or would be passed by its
predicted start. Default policy is now `eta`; `HUGPY_QUEUE_POLICY=smart|fifo` restores
the earlier rules.

### Background tickets for grading and probes (0.1.73)

Grading, smoke tests, benchmarks and warm-ups ride the same admission queue as real
calls but as BACKGROUND tickets (`metadata.intent` in grading / smoke-test / benchmark /
warmup, or `metadata.priority = background`): every foreground ticket precedes them, a
background ticket never bypasses a head waiting on room and never ages ahead of a real
call. This is what lets hugpy-core (0.2.13) grade a model from the call log — it asks
the rubric items as background calls and reads the verdicts back off `call_log` — while
the box keeps answering operators first.

### Serving leaves the placement lock (0.1.72)

Until now the forward/stream ran under the placement lock, so a box served ONE call at
a time across every model: a short answer from a resident model blocked another
resident model and blocked every load. Now the lock covers only the evict/load
decision. A serving model is pinned busy first (the eviction planner already never
selects a mid-generation resident; the polite-shrink path already skips it), then the
section is handed to the next admission ticket. Same-model calls are serialized on a
per-model serve slot sized to the engine (1 for a transformers child, `-np` for
llama-server), so the wait is the front door's and the call's timings stay honest. The
queue classifies a busy resident as `resident_now` (its wait is the slot, not room).

The room a placement sees now subtracts, for each live transformers resident, its
budgeted GPU need minus the bytes it holds at that moment (`inflight_reserved`), because
its KV cache grows per token and is released after the generation; without this a
placement made between two of its calls could take the room its next call needs. GGUF
engines preallocate KV and compute at load and reserve nothing. The reservation shows
in `GET /limits` (`ceiling.inflight_reserved`) and each resident's `gpu_budget` in
`GET /residents`.

### The environment log rides the model's own rows (0.1.71)

Nothing new is written anywhere: every env-profile event is an
`operator_actions` row with `action = 'env-profile'` and the `model_key` it is
attributed to — the cursor table hugpy-core already mirrors as
`w_operator_actions` — so inference can pinpoint which pip landed in which venv
for which model. `result` is `{event, profile, base, packages, lock, dir,
python, overlay_of, error, previous}`; `state` is `done` (or `failed` for a
failed build); `requested_by` is the caller or `profile:<name>`.

Events: `declared` (POST /v1/settings/env-profile; carries `model_key` when the
body names one), `assigned` / `unassigned` (POST /v1/settings/pair with an
`env_profile` that differs from the pair's current one; `previous` = the old
name), `build-requested` (a model's load gated on a profile not yet built —
attributed to that model), `building`, `built` (the frozen pip lock, the venv
dir and python, `overlay_of` = the worker base it overlays) and `build-failed`
(the pip error). Query the model's environment history with

    SELECT requested_at, state, json_extract(result, '$.event') AS event, result
      FROM operator_actions WHERE action = 'env-profile' AND model_key = ?
     ORDER BY id

Migration 0009 widens the action CHECK and adds an index on (model_key, id);
existing operator history is kept. Any `operator_actions` consumer that filters
by action keeps working — the new action never enters the `requested` state, so
the operator-actions runner never sees it.

### The transformers expert split lands (0.1.68 – 0.1.70)

A transformers MoE footprint now carries priced allocation candidates, so the
pinned / partial planner (`core/preferred.py`) accepts the expert split: the
whole model on the card, RAM-only, and — with the pair's MoE knob on — the
split (`n_gpu_layers -1`, `n_cpu_moe` = the routed expert count, non-expert
tensors + KV + overhead on the card, routed experts in RAM). With `alloc_mode`
`explicit` and `moe` on and nothing else pinned, the split IS the explicit
choice (no percentage required); the Auto rung pins it the same way. Measured
on `ae` with Qwen3.6-35B-A3B (256 experts): 5.95 GB predicted / 6.11 GB measured
on the card, the experts (66 GB) in RAM, 4.1–4.5 tok/s.

The split's context band is priced like every other mode: `sizing._mode_need`
and `ctx_fits_budget` price the transformers split with `tf_need(…, "moe")`
(before 0.1.70 they used the GGUF layer-split pricing, so the envelope reported
"facts missing" and the split loaded at the 2048 history default). Auto now
lands it at the feasible maximum of the ladder (262,144 on the box above), and
`/v1/verdicts` reports the split's `ctx_max` per KV type by walking the ladder.

A refused Auto rung no longer leaves the call row as a finished failure while
the next rung loads: the row returns to `pending` before the next attempt, so
Core's replay cursor keeps re-pulling it until the call really ends.

### Facts for every model and the blanket preferences (0.1.65, 0.1.66)

Every model is priced: a transformers dir gets its KV ladder from its config
geometry (decoder-only, encoder-decoder and nested decoder configs; diffusers
and config-less dirs are no-context heads with KV = 0), so `sizing.decide`
answers for it the same way as for a GGUF. A transformers MoE has the expert
split as a placement: the routed experts stay in host RAM and execute on the
CPU (`tf_child --moe-split`), everything else on the card; it is priced from
the per-tensor split (`tf_facts.expert_split`) and offered by Auto right after
GPU-only, exactly as the GGUF `--n-cpu-moe` rung.

Pair knobs from `directions/auto-multiplexing`: `allowed_placements` (a subset
of gpu-only, moe, 4bit, ram-only, max-ram, max-gpu; Auto never offers a rung
outside it), `wait_before_quality` (wait within `max_wait_s` for a call in
flight to free room before an approved quality change), `resident_priority`
(the eviction band once loaded) and `latency_target_s` (a measured response
estimate above it ranks after every plan within it). They ride the contract and
`GET /v1/verdicts` reports the policy in `auto_policy`.

**The box blanket (operator 2026-10-14).** The preferences are a fitevict-level knob,
not only a per-model one: `GET/POST /v1/settings/box` holds the box-wide defaults
(`auto_goal`, `tolerance`, `quality_alternatives`, `allowed_placements`, `wait_before_quality`,
`resident_priority`, `latency_target_s`, `max_wait_s`, `cpu_floor_cores` and the ctx
knobs — pins and per-file choices stay per pair). Every pair inherits them; a pair's
own row overrides key by key. `GET /v1/settings/pair` shows `settings` (own), `box`
(blanket) and `effective` (what a call runs under). The blanket is the pair_settings
row keyed `*`.

### Telemetry additions (0.1.63, 0.1.64)

`GET /v1/verdicts[?model_key=]` (0.1.64) reports the pair verdicts hugpy-core
and the console read: per quant file and placement mode, the priced figures,
the context ceiling and window per KV cache type, the Auto pick and whether
the MoE split is offered. A settings write flushes the cache.

`GET /storage` now lists every drive that holds a discovered model
(`drives[]`: mount, device, filesystem, capacity, the models it contains).
`GET /v1/telemetry/tables` and `GET /v1/telemetry/rows?table=…` give a
read-only page of any table or view of the wrapper store (API key secrets are
withheld); hugpy-core's Databases tab reads them through the router relay.

# Versioned updates

`fitevict update --system` compares the service venv's installed version with the
latest PyPI release. It only reports versions by default. Run
`sudo fitevict update --system --apply` to fetch the new wheel and the current
wheel, verify both SHA-256 hashes from PyPI, install the new wheel, restart the
service, and verify that `/health` reports the new version. If installation or
health verification fails, it reinstalls the previous wheel and restarts the
service. `--version X` targets a specific newer release; downgrades are refused.
If the live service reports a different version from its venv (for example a
development `PYTHONPATH` override), `--apply` stops before changing packages.

Core can delegate the same command over an existing operator SSH connection:
`hugpy-core workers update --name computron --ssh computron@192.168.1.128`
checks, and adding `--apply` installs. Core checks the remote wrapper's worker
name before sending the command. The SSH user needs passwordless sudo. Updates
preserve the wrapper's service configuration and model store.


## Engine inventory and server backends

Python discovery inventories all candidate environments and selects a runtime per
capability. ComfyUI workflows and wrapper-managed Ollama, vLLM, and SGLang servers
are supported alongside the existing adapters. See [docs/ENGINES.md](docs/ENGINES.md)
for local model configuration, runtime requirements, and placement restrictions.
