Metadata-Version: 2.4
Name: hugpy-wrapper
Version: 0.1.211
Summary: The hugpy wrapper: an OpenAI /v1 front door, the queue, placement and eviction, and the engines (llama.cpp and the kg_engines children); reads hugpy-discovery through its socket and export.
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: sqlalchemy>=2
Requires-Dist: numpy
Requires-Dist: scipy
Provides-Extra: terminal
Requires-Dist: hugpy-terminal<0.2,>=0.1.39; extra == "terminal"
Provides-Extra: discovery
Requires-Dist: hugpy-discovery==0.1.19; extra == "discovery"

# hugpy-wrapper

The hugpy wrapper: an OpenAI /v1 front door, the queue, placement and eviction, and the engines (llama.cpp and the kg_engines children). It reads hugpy-discovery only through its socket and export.

Install: the deb (`ops.sh full-install`); `hugpy-wrapper system install|status|uninstall`.
Settings: /etc/hugpy/wrapper/settings.json. State: /var/lib/hugpy/wrapper, /var/lib/hugpy/resources.
Source: assembled from /srv/hugpy/app/bundles/KNOWN_GOOD/TEST.

## The sort, and what core is sent

Spec: `/srv/hugpy/app/bundles/KNOWN_GOOD/DOCS`. Each of these predicts or reports; the run still decides every step at the row's turn.

- **Output-length range** (`queue/rows.py` `output_range`): a call's engine time is taken over the fewest to the most tokens the unit's own finished calls emitted, capped by `max_tokens`. A unit with fewer than `OUT_HISTORY_MIN` such calls keeps `[1, max_tokens]`.
- **The plan and its capacity rows** (`queue/plan.py`, `GET /v1/queue/plan`): the waiting rows in the order the run will pop them, each priced against what the rows ahead leave behind. A row is added wherever a capacity is reached: the card (evict-to-fit, with who goes and the seconds) or the worker drive (reap-to-fit, with the bytes short). The pop is the plan's first call row. A row's place in the plan is saved at dispatch (`p_ready_s`, `p_eta_s`, `p_position`, `p_capacity`).
- **Stages** (`call_stages`): one row per finished stage of a call, named as in `STAGE_ORDER` (the operator's order as data).
- **GPU-only engines share one logical CPU** (`engines/llama.py` `choose_cpus`); an engine with host bytes keeps whole cores to itself.
- **Telemetry for core** (`queue/telemetry.py`): `GET /v1/telemetry/state`, `GET /v1/telemetry`, `GET /v1/telemetry/resources`, and `POST /v1/requests/{id}/cancel`. `/health` carries `service`, `worker` and `version`. The wrapper reports itself as `hugpy-wrapper-native` with a routable catalog, not an inventory of copies. Call bodies and replies are not sent. `worker` in the settings file names the box when it is not the hostname.

Tests: `python -m pytest tests` (they run the staged source and open no live state). Live checks: `/srv/hugpy/app/tools/live-checks/hugpy-wrapper/spec-finish-20261110/`.

## The fit: where a unit goes (0.1.185.5)

One spill, on one scale, for every unit of every engine (`resources/fit.py`).

- **The tiers are ordered by speed.** At start-up the wrapper works out what a token pays per GiB read from the
  card, the RAM, the swap and the drive (the time prediction's own rates) and fills the fastest first. Table `tiers`.
- **A unit can be spilled or it cannot.** A GGUF is walked down the tiers a whole tensor at a time, the tensors
  every token reads first. Every other unit sits whole on the fastest tier that holds it.
- **The slowest tier is the floor**: the model's own file. It holds anything, so every unit has a place.
- **Feasibility is stored at start-up** (and when the units change): every unit's bytes against the box, as the
  range each tier can hold of it. `GET /v1/feasibility`, `hugpy-feasibility`. Table `feasibility`.
- **A call asks the same fit** with what is free now; evict-to-fit asks it with the card as if the residents
  were gone, and gives up residents in order of `L = p * delta`. A unit the card can never hold evicts nobody.
- **A load that runs out of memory is recorded** (table `fit_failures`) and the call goes on at the next place.
- **Names exist at the input and the output only.** A call may send `"placement"`: `gpu-only`, `gpu-max`,
  `ram-only`, `ram-max`, `swap-only`, `swap-max`, `disk_mmap-only`, `disk_mmap-max`, `moe` or `auto`. It becomes a
  bound on the fit; a bound that cannot be met is dropped and the call is answered where it can be. What is
  shown (the dashboard, the feasibility output) is named from the bytes.

## The queue is rows per engine (0.1.185.6)

A call waits for its own resident and for nothing else. Getting a unit placed and loaded (reap, evict-to-fit,
load) is one row at a time, because two calls deciding at once would take the same free bytes; once a unit is
loaded its engine answers on its own, and the next row that can run is taken at once. A row can run unless the
resident it is for is answering another call. A resident that is answering a call is nobody's victim: evict-to-fit
gives up idle residents only, and a busy one is not re-homed or restarted. The plan (`GET /v1/queue/plan`) carries
one clock for placing and loading and one per resident.

## The 4-bit representation is one more place (0.1.185.7)

A unit that cannot be spilled can still be loaded smaller: in 4-bit, where its engine can do that. The fit prices
it the way it prices the stored weights and takes it when it is the faster place by the tiers' speeds.

- **Who can:** the engine registry says which engines load 4-bit and which parts of a unit they quantize
  (`transformers`: the whole model; `videogen`: the transformer, text encoder and unet), and only when the
  engine's own environment imports `bitsandbytes` (asked once at start; `engines.bnb4`, `engines.bnb4_note`).
- **How big:** discovery's `api_unit_representations_v1` (hugpy-discovery 0.1.7.12): each part's bytes as stored,
  as bf16 and as 4-bit. A part the engine quantizes that discovery has no 4-bit figure for is priced at the
  ratio of the unit's own estimated parts. Stored at start-up in `feasibility.bnb4_bytes`, with its basis.
- **Where:** whole on the card; 4-bit runs nowhere else. The engine is started with `--bnb4`.
- **The caller's word:** `"bnb4": false` never, `"bnb4": true` whenever it fits, absent = by speed.
- **A 4-bit load that runs out of memory** is recorded apart from the stored ones, and the call goes on stored.

## Per-model options: an explicit allocation (0.1.185.9)

What can be chosen for a model is exactly what the fit takes (`queue/options.py`), and a choice is an explicit
allocation: per tier, the most of the model it may hold (the sliders), and its representation.

- `GET /v1/models/options?model=ID`: the sliders (each tier's ends on this worker, where a call would put the
  model now, where it was set), the names as slider positions for this model here (`presets`), the 4-bit
  representation (available or why not), the model's quants, and what is chosen.
- `POST /v1/models/options {"model": ID, "placement": NAME | "limit_gib": {tier: GiB} | "bnb4": bool}`: keep a
  choice (`"clear": true` removes it). A name sets the sliders to where it sits; `limit_gib` moves any of them.
  Also `"ctx"` (tokens, up to what the model was trained for) and `"quant"` (which quant the model's name
  means). `"model": "*"` is the worker's default, under every model's own choice.
- A call may send the same three fields; its own words come over what is kept for the model.

What is kept (table `selections`) is the sliders, in bytes. A name (gpu-only, gpu-max, ram-only ...) is a
definition of slider positions, said at the input and made again at the output from the bytes.

**Speed or quality** (`"goal"`, docs/wrapper-auto.md). By speed alone the smaller representation always wins. Under
`speed` (the goal when none is said) the fit takes 4-bit when it is the faster place, and a model's bare name means
its fastest quant, each priced by what one pass pays where the fit puts it on this worker. `quality` is the user's
limit on that: the weights as stored and the quant as chosen. It can be said for one model or for every model
(`"model": "*"`); a chosen quant, or a call that names its quant, is never changed.

## Context is a range; the engines' environments are the wrapper's own (0.1.185.10)

- **Context.** `ctx` is the most context wanted and `ctx_min` the least that will do. The unit is fitted at the low end,
  which says how deep down the tiers it has to reach at all; it is then loaded with the largest context in the range
  that reaches no deeper. A context chosen high is never why a load fails or spills. A call that needs more than the
  range raises it.
- **The model list** (`GET /v1/models`) carries, per model, what is chosen for it (`options`: allocation, context,
  4-bit, goal, quant, and how it is loaded now), for a client's columns.
- **Engine environments** live where the wrapper is installed (`/opt/hugpy/wrapper/engines`, the default
  `profile_home`), never on a models drive: a models drive is meant to be full. The installer creates the directory
  and says so when `profile_home` sits on the same drive as the worker or central models.
- **The wrapper prepares them itself.** An engine that can load 4-bit has a profile whose manifest carries what that
  needs (`bitsandbytes`, `accelerate`); at start the wrapper brings those profiles to their manifests: packages are
  added to a working environment in place (it is never cleared for this), a profile never built is built, one an
  engine is running on waits, and nothing is attempted with less than 2 GiB free. Until an environment is ready its
  engine is not offered 4-bit, and `engines.bnb4_note` says why.

## Auto is never an unknown (0.1.185.12)

- **Context on Auto** is the most that fits, up to all the model was trained for: it grows into the room beside
  weights that sit whole on a tier and never takes their place; weights that are spilled even at the least context
  get the default one. Room is made by eviction for the default context only; a longer one gives no resident up.
- **MoE** is a choice for a model with routed experts (`"moe": true | false`): on keeps its experts off the card.
- **What the wrapper does** with each model is in the model list (`options.placed`: where its bytes go against the
  box, as a name and as a share per tier), so a client never shows a bare "Auto"; the explicit sliders stand at it.
- **The engine's overhead on the card** is measured against the context a load really had, not a default one.

## A new host in one command: `system install --host --central --worker --engines`

`hugpy-wrapper system install` takes what a new worker has to be told (`system/engines.py`):

- `--host auto|ADDR`: the address the front door binds and advertises; `auto` is this host's LAN address. Without
  it a fresh install listens on 127.0.0.1 only.
- `--central DIR` and `--worker DIR`: where this host's model files are kept, and the fast local directory they are
  staged on to be loaded (`central_root`, `worker_root`); both are entered as discovery's drives
  (`hugpy-discovery system install --central DIR --worker DIR`). The wrapper places nothing until both are declared.
- `--engines llama`: `llama_server` is the `hugpy-llama-server` package's binary.
- `--engines all`: llama, and what every other engine runs on, fetched under `profile_home`: a python 3.11 base
  interpreter with the packages pinned in `system/engine_base.json` (`base/`), the ollama release (`ollama/`) and
  ComfyUI's source (`comfyui/app`, its `venv` the comfy profile). About 15 GiB; the service then builds each
  engine's own environment at start (about 55 GiB when all are built). An interpreter or a ComfyUI the settings
  already name, and that is there, is kept.

Each option sets the settings it names and nothing else; a step already done is skipped, so the command can be run
again. A whole worker from nothing:

```sh
curl -fsSL https://hugpy.ai/apt/setup.sh | sudo bash -s -- --install wrapper \
  && sudo apt-get install -y hugpy-discovery \
  && sudo hugpy-wrapper system install --host auto --central /srv/hugpy/models --worker /srv/hugpy/worker --engines all
```
