Metadata-Version: 2.4
Name: hugpy-wrapper
Version: 0.1.56
Summary: The hugpy wrapper (import name: fitevict): a per-box evict-to-fit front door. An OpenAI /v1 server that resolves a called model to its weights (GGUF via llama.cpp, transformers chat/vision, and every non-chat engine: embeddings, whisper ASR, TTS, diffusers image + video, summarization, keywords), places it on the GPU by evicting what doesn't fit, serves it, and logs every call.
Author-email: putkoff <support@hugpy.ai>
License-Expression: LicenseRef-Proprietary
Project-URL: UI, https://www.npmjs.com/package/@hugpy/ui
Project-URL: Homepage, https://hugpy.ai
Keywords: llama.cpp,inference,evict-to-fit,vram,openai,wrapper,allocator,whisper,tts,diffusers,embeddings
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Provides-Extra: transformers
Requires-Dist: torch==2.14.0; extra == "transformers"
Requires-Dist: transformers==5.17.0; extra == "transformers"
Requires-Dist: accelerate==1.15.0; extra == "transformers"
Requires-Dist: bitsandbytes==0.50.2; extra == "transformers"
Requires-Dist: peft==0.20.0; extra == "transformers"
Requires-Dist: compressed-tensors>=0.15.0; extra == "transformers"
Requires-Dist: safetensors==0.8.0; extra == "transformers"
Requires-Dist: tokenizers==0.23.2; extra == "transformers"
Requires-Dist: sentencepiece==0.2.2; extra == "transformers"
Requires-Dist: tiktoken==0.14.0; extra == "transformers"
Requires-Dist: protobuf==7.36.1; extra == "transformers"
Requires-Dist: huggingface_hub==1.31.0; extra == "transformers"

# hugpy-wrapper (`fitevict`)

hugpy's per-box wrapper: an **evict-to-fit front door over llama.cpp** (and transformers). Point any
OpenAI client's base URL at it and one address becomes a self-managing model server.

A `/v1` call carries only a model **name** — never a location. For each request the wrapper resolves
the name to its weights, runs an **evict-to-fit** plan against what is resident and what is being
called, places the model (launch / evict), forwards the call, splits the model's `<think>` out of the
answer, and records the call — raw stamps and engine geometry — in its own store, which it also
reads back to place the next model better.

It needs nothing but an install, a host and a port: no database server, no other hugpy package, no
sudo. The decision core is pure stdlib; hardware facts come from `nvidia-smi`; model facts from the
files on disk.

## Install

`pip` only delivers the package; `fitevict install` is the install.

```
pip install hugpy-wrapper            # or: pipx install hugpy-wrapper

fitevict install --user --port auto                  # your own service, no root
sudo "$(command -v fitevict)" install --system --port auto   # a system service (root once)
fitevict install ... --dry-run                       # print the plan, change nothing
```

No port is ever assumed: give a number, or `auto` for a free one (it is printed and written to the
config). The host defaults to `127.0.0.1`; `--host 0.0.0.0` serves the network.

`--system` sets up the hugpy layout — each hugpy component runs as its own user, and the group
`hugpy` shares only the model store:

| what | where |
|---|---|
| user | `hugpy-wrapper` (system, no login), member of group `hugpy` |
| code | `/opt/hugpy-wrapper/venv` — this exact version, pip-installed |
| config | `/etc/hugpy/wrapper.env` (root:hugpy-wrapper 0640) — host, port, box key, every path |
| state (the store) | `/var/lib/hugpy-wrapper` (0700, created by systemd) |
| cache | `/var/cache/hugpy-wrapper` |
| models | `/var/lib/hugpy/models` (root:hugpy 2775), or `--models DIR` |
| service | `hugpy-wrapper.service` — hardened: read-only system, writes only its own dirs + the models |
| endpoint | `/etc/hugpy/wrapper.endpoint.json` — where it listens (no secret) |
| logs | the journal: `journalctl -u hugpy-wrapper` |

`--user` uses the environment you installed into, `~/.config/hugpy/wrapper.env` and a user unit
(`systemctl --user`; `loginctl enable-linger` keeps it running after logout).

Every step is safe to repeat: a re-run repairs or upgrades and keeps the box key and the port.

```
fitevict status [--system|--user]                    # installed? answering? store size?
fitevict uninstall [--system|--user] [--purge]       # --purge also removes config, store, user
```

Other options: `--worker NAME` (default: the host name), `--engine SPEC` (pip-install an engine),
`--package SPEC` (the code to install, e.g. a wheel file), `--env KEY=VALUE` (repeatable).

Platforms: Linux with systemd today (Ubuntu, Debian, Kali, Fedora, Arch …). macOS (launchd) and
Windows services are next; the command says so rather than half-installing.

### The engine

GGUF models need `llama-server`. It is looked up as `LLAMA_SERVER_BIN`, then under
`HUGPY_ENGINE_DIR`, then on `PATH` — and the service's `PATH` starts with its own environment, so an
engine installed there (`fitevict install … --engine <spec>`) is found. `fitevict install` reports
whether one resolves. Without it the wrapper still runs; GGUF calls fail until an engine is there.

```
pip install "hugpy-wrapper[transformers]"   # the transformers engine (into its own venv)
```

## Use

```
curl -s http://127.0.0.1:<port>/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"<name>","messages":[{"role":"user","content":"hi"}],"max_tokens":128}'
```

Any OpenAI client works: `base_url=http://<host>:<port>/v1`. The model is named, not located; the
wrapper finds it, fits it, serves it, logs it.

## API keys

Keys look like `hpk_<id>_<secret>` and are sent as `Authorization: Bearer …` — what every OpenAI
client sends for its `api_key`, so a user just sets `OPENAI_API_KEY=hpk_…`.

- **The box key** (`hpk_box_…`, admin) is written once by `fitevict install` into the service config.
  It is the root of trust for the box; hugpy-core uses it to manage the box.
- **More keys**, made through the running service:

  ```
  fitevict keys create --name laptop [--scope use|admin] [--expires-days 90]   # printed once
  fitevict keys list [--all]
  fitevict keys revoke <id>
  ```

  Only a hash of each key is stored. Revoked and expired keys stop working at once.

Who needs a key:

| route | rule |
|---|---|
| OpenAI routes (`/v1/chat/completions`, `/v1/models`, …) | `FITEVICT_REQUIRE_KEY`: `outside` (default) — no key from loopback and private networks, a key from anywhere else; `never`; `always` |
| management (`/v1/telemetry`, `/v1/settings/*`, `/v1/operator/*`, `/v1/keys`, `POST /limits`) | an admin key, except from the box itself (loopback) |
| `/health` | never |

What counts as inside is `FITEVICT_TRUSTED_NETS` (default: loopback, 10/8, 172.16/12, 192.168/16,
fd00::/8, fe80::/10). **Behind a proxy** (nginx, a tunnel), every request arrives from 127.0.0.1: a
request with forwarding headers counts as *outside* unless its immediate peer is listed in
`FITEVICT_TRUSTED_PROXIES` — only then is `X-Forwarded-For` believed. A public proxy can never make
the internet "local".

## The store (telemetry)

The wrapper keeps every call and every measurement in one SQLite file it creates on first use
(`FITEVICT_HOME`, else systemd's state directory, else the per-OS data directory; 0700 / 0600). Its
placement decisions read it back: the measured CUDA context, llama.cpp's projections, serving
growth, KV structure, host memory, a model's largest real use. Every measurement points at the call
that took it; views compute the figures. A damaged file is set aside and recreated — everything in
it is re-learnable. Schema changes ship as numbered migrations.

hugpy-core (or anything with an admin key) works with a box through the management API:

```
GET  /v1/telemetry?table=call_log&since=<id>&limit=<n>   rows after a cursor (call_log, resource_log,
                                                         evictions, operator_actions) + next / more
GET  /v1/telemetry/state                                 GPU / host state, residents, models, facts, settings
POST /v1/operator/actions   {"action","model_key"}       evict | evict-all | free-ram | remove-model | restart
GET  /v1/operator/actions/<id>
GET|POST /v1/settings/pair  {"model_key","settings","env_profile"}   a model's knobs on this box
POST /v1/settings/quant-facts {"model_key","file","size_bytes","kv_cost"}   size + KV ticks
POST /v1/settings/env-profile {"name","base","packages"}
GET|POST /v1/keys, POST /v1/keys/<id>/revoke
```

A call's own knobs (`model@options`, request fields) still override the stored settings per call.
`python -m fitevict.observability.calls_view --port <n|auto>` serves a read-only page of the latest
calls on 127.0.0.1 (it shows every call's bodies: run it as the service's user, keep it local).

## Every model kind, one front door

Chat (GGUF via llama-server, transformers via `tf_child`), vision (`vision_child`) and every
non-chat engine go through the SAME placement (evict-to-fit), the SAME call log row and the SAME
residents / models catalog. Non-chat engines are runners on one generic child
(`lifted/task_child.py`); each is one file in `lifted/runners/` plus one `RunnerSpec`:

| route | runner | serves |
|---|---|---|
| `POST /v1/embeddings` | `embed` | sentence-transformers encoders |
| `POST /v1/audio/transcriptions`, `/v1/audio/translations` | `asr` | whisper (multipart `file` or `file_path`/`file_b64`) |
| `POST /v1/audio/speech` | `tts` | chatterbox — returns raw audio bytes |
| `POST /v1/images/generations`, `/edits`, `/variations` | `imagegen` | diffusers image pipelines — `b64_json` (or `response_format: path`) |
| `POST /v1/videos/generations` | `videogen` | diffusers video pipelines (Wan / LTX / CogVideoX / Mochi …) — mp4 path |
| `POST /v1/summarize` | `summarize` | seq2seq summarizers (bart / pegasus / t5 / led) |
| `POST /v1/keywords` | `keywords` | KeyBERT over a stamped embedding model |

Binary outputs are written to `FITEVICT_ARTIFACT_DIR` and returned in the endpoint's OpenAI shape;
the call log stores their exact bytes + sha256, never the bytes. Uploads are spooled to
`FITEVICT_SPOOL_DIR` and removed after the call.

**Env profiles** (`lifted/profiles.py`): a runner's extra pip deps live in a per-profile venv at
`<engine>/envs/<name>` — an overlay on the box's worker base (`FITEVICT_TF_PYTHON`) or, when a model
pins its own torch (chatterbox), an isolated venv. Declared in `runners.PROFILES`, overridable with
`POST /v1/settings/env-profile`; a model is pinned to one with `POST /v1/settings/pair`
(`env_profile`). A profile is built on first use, off the request path; until it is ready the call
fails as data (503 `profile_materializing`). `GET /profiles` reports every profile's state.

## Cores

Every engine gets its own cores, claimed once at load from the plan's RAM:VRAM split
(CORES.md):
- An engine with nothing in RAM gets one core, which is all it uses.
- An engine reading weights from RAM gets the cores that still make it faster, by the
  box's measured bandwidth curve. Until that curve is known, it gets every free core.
- Two engines never share a core.
- Cores busy with other work are avoided, and one core stays reserved for the wrapper and
  the OS.

llama.cpp gets `-t N --cpu-mask … --cpu-strict 1`. The engine process is confined to the
claim by CPU affinity, which needs no privilege.

Each call row records the claim (`cpu_set`, `threads`, `cpu_source`) and the CPU time the
call used (`cpu_user_ms`, `cpu_sys_ms`). `GET /v1/cores` shows the box's topology, per-CPU
load, every engine's claim and the fitted speed constants.

Per model you can raise the claim with `cpu_priority` or pin it with `cpu_cores` (a count
or a CPU list) in `POST /v1/settings/pair`. A pin is honoured even when it has to take
busy cores, and the row records what it took.

## Configuration

`fitevict install` writes these into the service config; you rarely set them by hand.

| variable | meaning |
|---|---|
| `FITEVICT_HOST`, `FITEVICT_PORT` | where the front door listens (`fitevict-serve --host/--port`; `auto` = free port) |
| `FITEVICT_TOKEN` | the box key |
| `FITEVICT_REQUIRE_KEY`, `FITEVICT_TRUSTED_NETS`, `FITEVICT_TRUSTED_PROXIES` | who needs a key (above) |
| `FITEVICT_WORKER` | this box's name in the call log (default: the host name) |
| `FITEVICT_HOME` | the store's directory |
| `HUGPY_DATA_DIR`, `HUGPY_CACHE_DIR`, `HUGPY_ENGINE_DIR` | data / cache / engine dirs |
| `MODELS_HOME`, `DEFAULT_ROOT` | the model store (and the dir that contains it) |
| `FITEVICT_CPU_PIN`, `FITEVICT_CPU_RESERVE`, `FITEVICT_CPU_BUSY` | core claims on/off, cores kept back (1), busy threshold (0.5) — CORES.md |
| `HUGPY_QUANT_FALLBACK` | the box-wide ceiling over AUTO quant stepping: `auto` (default, every quant on the drive, descending) \| `off` (never step; the per-pair knob is ignored) \| a comma list of quant labels in priority order (only those, in that order). The per-pair `quality_alternatives` can only narrow/reorder inside it |
| `HUGPY_QUEUE_POLICY`, `HUGPY_QUEUE_BYPASS_MAX` | admission queue: `smart` (default) \| `fifo`; the times a waiting head may be bypassed before it is served unconditionally (default 3) |

Set any of these at install with `fitevict install --env HUGPY_QUANT_FALLBACK=Q6_K,Q4_K_M`
(repeatable); they are rendered into the service's env file like the rest.

## Auto policy

When a call arrives by model name alone (no pin), the front door places it **Auto**.
The fallback **rung order**, per candidate artifact, is:

```
gpu-only → MoE CPU-expert split (MoE models) → 4-bit → ram-only → max-ram → max-gpu
```

then it **steps the quant down** to the next approved variant of the same model and
restarts at gpu-only. "4-bit" is the approved 4-bit representation of the *same*
model: for transformers the bnb 4-bit runtime quant of the artifact; for GGUF an
approved Q4-class variant file in the catalog (skipped when there is none). When every
rung is exhausted the result is terminal with a specific reason — **resource** (steps
the quant), **bad artifact** or **incompatible engine** (both terminal, never a quant
change). An identical failed plan (artifact, mode, ctx, kv type) is not retried in the
same request without evidence the room changed. The stepping runs **live** in the
placement driver (not just reported): each variant is loaded with its own facts, and
the call row records the artifact that actually loaded. The box env
`HUGPY_QUANT_FALLBACK` is the **ceiling** over which variants may be stepped to
(`auto` | `off` | an ordered label list); the per-pair `quality_alternatives` can only
narrow or reorder inside it. The applied policy (env value, pair knob, ordered
candidates) is written to the decision / plan_note.

Concurrent calls enter placement through an **admission queue** (`HUGPY_QUEUE_POLICY`
= `smart` default | `fifo`). Smart: while the FIFO head waits on an eviction or room,
calls that can run now may go ahead — resident-served first (grouped), then
fits-without-eviction by shortest time-to-output — but a group is never reordered and
a head is bypassed at most `HUGPY_QUEUE_BYPASS_MAX` times and never past its
`max_wait_s` (aging). `GET /v1/queue` shows the live queue in order with the reason the
next one runs; each call row records its queue position, bypass count and why it ran.

A rung whose weights live in host RAM (MoE split, ram-only, max-ram, max-gpu) is
admissible only if its CPU claim would get at least **`cpu_floor_cores`** free
*physical* cores (the CPU cliff) — memory fit alone never admits it; a hyperthread
sibling is not an extra core.

Among the admissible candidates the **goal profile** ranks them (minimized left to
right): `quality` → `(q, T+R, T)`; `fast` → `(T, q, R)`; `resident` → `(q, E_hot,
T+R)`, where `q` = index in the approved-representation list (0 = selected), `T` =
estimated seconds to complete from the box's measured rates (unmeasured → ladder
order, never an invented bandwidth), `R` = reload cost of displaced residents,
`E_hot` = pinned/high-priority residents displaced. Waiting is a candidate only when a
resource change is expected within **`max_wait_s`**.

Context has three explicit states per (model × mode × KV type): **range** (shrink on
over `[feasible_min, feasible_max]`), **point** (a single feasible tick, shrink off),
and **infeasible** (no tick fits the worker budget — refused, never loaded as a
point). A resident loaded below its `ctx_max` **grows back** by an idle rebuild when
there is demand, sustained headroom with a margin, a cooldown has passed, and it is
idle (`ctx_restore`).

Per-pair knobs (`user_settings`, sent to the box by core via `POST /v1/settings/pair`;
an absent knob = Auto):

| knob | values | default |
|---|---|---|
| `ctx_pct` | 1–100 (% of trained → applied ctx_max) | Auto = feasible_max |
| `ctx_min_pct` | 1–100 (% of trained → applied ctx_min) | Auto = feasible_min |
| `ctx_range_enabled` | bool — range on/off (read-compat: old `ctx_range`) | Auto rule |
| `ctx_yield` | bool — shrink on/off | Auto rule |
| `ctx_restore` | bool — grow a shrunk resident back to ctx_max | Auto = on when shrink is on |
| `auto_goal` | `quality` \| `fast` \| `resident` | `quality` |
| `quality_alternatives` | `auto` \| `none` \| ordered list of variant keys | `auto` |
| `cpu_floor_cores` | integer ≥ 1 (physical cores floor) | `4` |
| `max_wait_s` | seconds (bounded wait budget) | `120` |
| `alloc_mode` | `gpu-only` \| `ram-only` \| `max-gpu` \| `max-ram` \| `explicit` (pin) | Auto |
| `moe`, `kv_cache_type` | MoE expert split; `f16`/`q8_0`/`q4_0` | Auto; `f16` |

## Glossary

One set of **shared names** for quantities the code already carries. The DB columns
and API fields **do not change** — this maps the existing names to the shared ones.

| shared name | the existing quantity |
|---|---|
| `worker_vram_budget_bytes` | the worker budget: the card with nothing of ours resident |
| `available_vram_bytes` | current usable headroom **after** the reserve (not raw NVML free) |
| `worker_feasible_context_max_tokens` | `feasible_max` — largest ladder tick that fits the worker budget |
| `auto_context_floor_tokens` | `feasible_min` — smallest tick ≥ 0.25 × trained window |
| `context_shortfall_tokens` | `theoretical_max − feasible_max`, in tokens |
| `target_context_tokens` | `ctx_max` — the upper handle (where a load starts) |
| `minimum_retained_context_tokens` | `ctx_min` — the lower handle (shrink floor) |
| `resident_context_tokens` | the loaded engine's capacity (a resident's ctx) |
| `model_context_ceiling_tokens` | `theoretical_max` (the trained window, `ctx_train`) |
| `model_context_floor_tokens` | `theoretical_min` (the ladder's smallest tick) |
| `tok_bytes_gpu` / `tok_bytes_ram` | weight bytes **read per generated token** (the split), not context growth |
| `non_expert_bytes` | estimated GPU weight bytes **including** GPU-resident experts |
| `free_vram_before` / `free_vram_after` | the budgetable headroom, not raw NVML free |

## Layout

```
fitevict/
  core/           pure allocator: types, evict (victim selector), plan (staged fit + quant ladder),
                  flex (ctx-band compress, layers-that-fit), host (EngineHost + drive loop)
  estimate/       sizing, predict, ctx_probe
  serve/          front_door (the OpenAI server), manage (management API), auth (API keys),
                  run_engine (juggle harness), variants, gpu_state, worker_storage_ops
  observability/  store (SQLite) + schema/ migrations, resource_log, calls_view, blobs
  cli/            installer (`fitevict`), console (`fitevict-console`)
  adapters/       llama_cpp (measure / price / launch / evict), transformers, vision, tasks,
                  db (call log + measurements over the store)
  lifted/         helpers lifted from hugpy: gguf_inspect, gguf_need, hardware, spill_reserve,
                  model_resolve, native_resolve, procutil, timings, no_think, app_dirs, runners/ …
  imports/        shared constants / functions / classes (no heavy imports)
```

See `MIGRATION.md` (package layout), `MIGRATION-imports.md` (shared imports layer) and
`MIGRATION-store.md` (store, management API, installer, API keys).

## Test

```
pip install "hugpy-wrapper[test]"
pytest          # every test runs on its own empty store; nothing touches a real one
```
