Metadata-Version: 2.4
Name: hugpy-wrapper
Version: 0.1.59
Summary: The hugpy wrapper (import name: fitevict): a per-box evict-to-fit front door. An OpenAI /v1 server that resolves a called model to its weights (GGUF via llama.cpp, transformers chat/vision, and every non-chat engine: embeddings, whisper ASR, TTS, diffusers image + video, summarization, keywords), places it on the GPU by evicting what doesn't fit, serves it, and logs every call.
Author-email: putkoff <support@hugpy.ai>
License-Expression: LicenseRef-Proprietary
Project-URL: UI, https://www.npmjs.com/package/@hugpy/ui
Project-URL: Homepage, https://hugpy.ai
Keywords: llama.cpp,inference,evict-to-fit,vram,openai,wrapper,allocator,whisper,tts,diffusers,embeddings
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Provides-Extra: transformers
Requires-Dist: torch==2.14.0; extra == "transformers"
Requires-Dist: transformers==5.17.0; extra == "transformers"
Requires-Dist: accelerate==1.15.0; extra == "transformers"
Requires-Dist: bitsandbytes==0.50.2; extra == "transformers"
Requires-Dist: peft==0.20.0; extra == "transformers"
Requires-Dist: compressed-tensors>=0.15.0; extra == "transformers"
Requires-Dist: safetensors==0.8.0; extra == "transformers"
Requires-Dist: tokenizers==0.23.2; extra == "transformers"
Requires-Dist: sentencepiece==0.2.2; extra == "transformers"
Requires-Dist: tiktoken==0.14.0; extra == "transformers"
Requires-Dist: protobuf==7.36.1; extra == "transformers"
Requires-Dist: huggingface_hub==1.31.0; extra == "transformers"

# hugpy-wrapper (`fitevict`)

hugpy's per-box wrapper: an **evict-to-fit front door over llama.cpp** (and transformers). Point any
OpenAI client's base URL at it and one address becomes a self-managing model server.

A `/v1` call carries only a model **name** — never a location. For each request the wrapper resolves
the name to its weights, runs an **evict-to-fit** plan against what is resident and what is being
called, places the model (launch / evict), forwards the call, splits the model's `<think>` out of the
answer, and records the call — raw stamps and engine geometry — in its own store, which it also
reads back to place the next model better.

It needs nothing but an install, a host and a port: no database server, no other hugpy package, no
sudo. The decision core is pure stdlib; hardware facts come from `nvidia-smi`; model facts from the
files on disk.

## Install

`pip` only delivers the package; `fitevict install` is the install.

```
pip install hugpy-wrapper            # or: pipx install hugpy-wrapper

fitevict install --user --port auto                  # your own service, no root
sudo "$(command -v fitevict)" install --system --port auto   # a system service (root once)
fitevict install ... --dry-run                       # print the plan, change nothing
```

No port is ever assumed: give a number, or `auto` for a free one (it is printed and written to the
config). The host defaults to `127.0.0.1`; `--host 0.0.0.0` serves the network.

`--system` sets up the hugpy layout — each hugpy component runs as its own user, and the group
`hugpy` shares only the model store:

| what | where |
|---|---|
| user | `hugpy-wrapper` (system, no login), member of group `hugpy` |
| code | `/opt/hugpy-wrapper/venv` — this exact version, pip-installed |
| config | `/etc/hugpy/wrapper.env` (root:hugpy-wrapper 0640) — host, port, box key, every path |
| state (the store) | `/var/lib/hugpy-wrapper` (0700, created by systemd) |
| cache | `/var/cache/hugpy-wrapper` |
| models | `/var/lib/hugpy/models` (root:hugpy 2775), or `--models DIR` |
| service | `hugpy-wrapper.service` — hardened: read-only system, writes only its own dirs + the models |
| endpoint | `/etc/hugpy/wrapper.endpoint.json` — where it listens (no secret) |
| logs | the journal: `journalctl -u hugpy-wrapper` |

`--user` uses the environment you installed into, `~/.config/hugpy/wrapper.env` and a user unit
(`systemctl --user`; `loginctl enable-linger` keeps it running after logout).

Every step is safe to repeat: a re-run repairs or upgrades and keeps the box key and the port.

```
fitevict status [--system|--user]                    # installed? answering? store size?
fitevict uninstall [--system|--user] [--purge]       # --purge also removes config, store, user
```

Other options: `--worker NAME` (default: the host name), `--engine SPEC` (pip-install an engine),
`--package SPEC` (the code to install, e.g. a wheel file), `--env KEY=VALUE` (repeatable).

Platforms: Linux with systemd today (Ubuntu, Debian, Kali, Fedora, Arch …). macOS (launchd) and
Windows services are next; the command says so rather than half-installing.

### The engine

GGUF models need `llama-server`. It is looked up as `LLAMA_SERVER_BIN`, then under
`HUGPY_ENGINE_DIR`, then on `PATH` — and the service's `PATH` starts with its own environment, so an
engine installed there (`fitevict install … --engine <spec>`) is found. `fitevict install` reports
whether one resolves. Without it the wrapper still runs; GGUF calls fail until an engine is there.

```
pip install "hugpy-wrapper[transformers]"   # the transformers engine (into its own venv)
```

## Use

```
curl -s http://127.0.0.1:<port>/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"<name>","messages":[{"role":"user","content":"hi"}],"max_tokens":128}'
```

Any OpenAI client works: `base_url=http://<host>:<port>/v1`. The model is named, not located; the
wrapper finds it, fits it, serves it, logs it.

## API keys

Keys look like `hpk_<id>_<secret>` and are sent as `Authorization: Bearer …` — what every OpenAI
client sends for its `api_key`, so a user just sets `OPENAI_API_KEY=hpk_…`.

- **The box key** (`hpk_box_…`, admin) is written once by `fitevict install` into the service config.
  It is the root of trust for the box; hugpy-core uses it to manage the box.
- **More keys**, made through the running service:

  ```
  fitevict keys create --name laptop [--scope use|admin] [--expires-days 90]   # printed once
  fitevict keys list [--all]
  fitevict keys revoke <id>
  ```

  Only a hash of each key is stored. Revoked and expired keys stop working at once.

Who needs a key:

| route | rule |
|---|---|
| OpenAI routes (`/v1/chat/completions`, `/v1/models`, …) | `FITEVICT_REQUIRE_KEY`: `outside` (default) — no key from loopback and private networks, a key from anywhere else; `never`; `always` |
| management (`/v1/telemetry`, `/v1/settings/*`, `/v1/operator/*`, `/v1/keys`, `POST /limits`) | an admin key, except from the box itself (loopback) |
| `/health` | never |

What counts as inside is `FITEVICT_TRUSTED_NETS` (default: loopback, 10/8, 172.16/12, 192.168/16,
fd00::/8, fe80::/10). **Behind a proxy** (nginx, a tunnel), every request arrives from 127.0.0.1: a
request with forwarding headers counts as *outside* unless its immediate peer is listed in
`FITEVICT_TRUSTED_PROXIES` — only then is `X-Forwarded-For` believed. A public proxy can never make
the internet "local".

## The store (telemetry)

The wrapper keeps every call and every measurement in one SQLite file it creates on first use
(`FITEVICT_HOME`, else systemd's state directory, else the per-OS data directory; 0700 / 0600). Its
placement decisions read it back: the measured CUDA context, llama.cpp's projections, serving
growth, KV structure, host memory, a model's largest real use. Every measurement points at the call
that took it; views compute the figures. A damaged file is set aside and recreated — everything in
it is re-learnable. Schema changes ship as numbered migrations.

hugpy-core (or anything with an admin key) works with a box through the management API:

```
GET  /v1/telemetry?table=call_log&since=<id>&limit=<n>   rows after a cursor (call_log, resource_log,
                                                         evictions, operator_actions) + next / more
GET  /v1/telemetry/state                                 GPU / host state, residents, models, facts, settings
POST /v1/operator/actions   {"action","model_key"}       evict | evict-all | free-ram | remove-model | restart
GET  /v1/operator/actions/<id>
GET|POST /v1/settings/pair  {"model_key","settings","env_profile"}   a model's knobs on this box
POST /v1/settings/quant-facts {"model_key","file","size_bytes","kv_cost"}   size + KV ticks
POST /v1/settings/env-profile {"name","base","packages"}
GET|POST /v1/keys, POST /v1/keys/<id>/revoke
```

A call's own knobs (`model@options`, request fields) still override the stored settings per call.
`python -m fitevict.observability.calls_view --port <n|auto>` serves a read-only page of the latest
calls on 127.0.0.1 (it shows every call's bodies: run it as the service's user, keep it local).

## Every model kind, one front door

Chat (GGUF via llama-server, transformers via `tf_child`), vision (`vision_child`) and every
non-chat engine go through the SAME placement (evict-to-fit), the SAME call log row and the SAME
residents / models catalog. Non-chat engines are runners on one generic child
(`lifted/task_child.py`); each is one file in `lifted/runners/` plus one `RunnerSpec`:

| route | runner | serves |
|---|---|---|
| `POST /v1/embeddings` | `embed` | sentence-transformers encoders |
| `POST /v1/audio/transcriptions`, `/v1/audio/translations` | `asr` | whisper (multipart `file` or `file_path`/`file_b64`) |
| `POST /v1/audio/speech` | `tts` | chatterbox — returns raw audio bytes |
| `POST /v1/images/generations`, `/edits`, `/variations` | `imagegen` | diffusers image pipelines — `b64_json` (or `response_format: path`) |
| `POST /v1/videos/generations` | `videogen` | diffusers video pipelines (Wan / LTX / CogVideoX / Mochi …) — mp4 path |
| `POST /v1/summarize` | `summarize` | seq2seq summarizers (bart / pegasus / t5 / led) |
| `POST /v1/keywords` | `keywords` | KeyBERT over a stamped embedding model |

Binary outputs are written to `FITEVICT_ARTIFACT_DIR` and returned in the endpoint's OpenAI shape;
the call log stores their exact bytes + sha256, never the bytes. Uploads are spooled to
`FITEVICT_SPOOL_DIR` and removed after the call.

**Env profiles** (`lifted/profiles.py`): a runner's extra pip deps live in a per-profile venv at
`<engine>/envs/<name>` — an overlay on the box's worker base (`FITEVICT_TF_PYTHON`) or, when a model
pins its own torch (chatterbox), an isolated venv. Declared in `runners.PROFILES`, overridable with
`POST /v1/settings/env-profile`; a model is pinned to one with `POST /v1/settings/pair`
(`env_profile`). A profile is built on first use, off the request path; until it is ready the call
fails as data (503 `profile_materializing`). `GET /profiles` reports every profile's state.

## Cores

Every engine gets its own cores, claimed once at load from the plan's RAM:VRAM split
(CORES.md):
- An engine with nothing in RAM gets one core, which is all it uses.
- An engine reading weights from RAM gets the cores that still make it faster, by the
  box's measured bandwidth curve. Until that curve is known, it gets every free core.
- Two engines never share a core.
- Cores busy with other work are avoided, and one core stays reserved for the wrapper and
  the OS.

llama.cpp gets `-t N --cpu-mask … --cpu-strict 1`. The engine process is confined to the
claim by CPU affinity, which needs no privilege.

Each call row records the claim (`cpu_set`, `threads`, `cpu_source`) and the CPU time the
call used (`cpu_user_ms`, `cpu_sys_ms`). `GET /v1/cores` shows the box's topology, per-CPU
load, every engine's claim and the fitted speed constants.

Per model you can raise the claim with `cpu_priority` or pin it with `cpu_cores` (a count
or a CPU list) in `POST /v1/settings/pair`. A pin is honoured even when it has to take
busy cores, and the row records what it took.

## Configuration

`fitevict install` writes these into the service config; you rarely set them by hand.

| variable | meaning |
|---|---|
| `FITEVICT_HOST`, `FITEVICT_PORT` | where the front door listens (`fitevict-serve --host/--port`; `auto` = free port) |
| `FITEVICT_TOKEN` | the box key |
| `FITEVICT_REQUIRE_KEY`, `FITEVICT_TRUSTED_NETS`, `FITEVICT_TRUSTED_PROXIES` | who needs a key (above) |
| `FITEVICT_WORKER` | this box's name in the call log (default: the host name) |
| `FITEVICT_HOME` | the store's directory |
| `HUGPY_DATA_DIR`, `HUGPY_CACHE_DIR`, `HUGPY_ENGINE_DIR` | data / cache / engine dirs |
| `MODELS_HOME`, `DEFAULT_ROOT` | the model store (and the dir that contains it) |
| `FITEVICT_CPU_PIN`, `FITEVICT_CPU_RESERVE`, `FITEVICT_CPU_BUSY` | core claims on/off, cores kept back (1), busy threshold (0.5) — CORES.md |
| `HUGPY_QUANT_FALLBACK` | the box-wide ceiling over AUTO quant stepping: `auto` (default, every quant on the drive, descending) \| `off` (never step; the per-pair knob is ignored) \| a comma list of quant labels in priority order (only those, in that order). The per-pair `quality_alternatives` can only narrow/reorder inside it |
| `HUGPY_QUEUE_POLICY`, `HUGPY_QUEUE_BYPASS_MAX` | admission queue: `smart` (default) \| `fifo`; the times a waiting head may be bypassed before it is served unconditionally (default 3) |

Set any of these at install with `fitevict install --env HUGPY_QUANT_FALLBACK=Q6_K,Q4_K_M`
(repeatable); they are rendered into the service's env file like the rest.

## Auto policy

When a call arrives by model name alone (no pin), the front door places it **Auto**.
The fallback **rung order**, per candidate artifact, is:

```
gpu-only → MoE CPU-expert split (MoE models) → 4-bit → ram-only → max-ram → max-gpu
```

then it **steps the quant down** to the next approved variant of the same model and
restarts at gpu-only. "4-bit" is the approved 4-bit representation of the *same*
model: for transformers the bnb 4-bit runtime quant of the artifact; for GGUF an
approved Q4-class variant file in the catalog (skipped when there is none). When every
rung is exhausted the result is terminal with a specific reason — **resource** (steps
the quant), **bad artifact** or **incompatible engine** (both terminal, never a quant
change). An identical failed plan (artifact, mode, ctx, kv type) is not retried in the
same request without evidence the room changed. The stepping runs **live** in the
placement driver (not just reported): each variant is loaded with its own facts, and
the call row records the artifact that actually loaded. The box env
`HUGPY_QUANT_FALLBACK` is the **ceiling** over which variants may be stepped to
(`auto` | `off` | an ordered label list); the per-pair `quality_alternatives` can only
narrow or reorder inside it. The applied policy (env value, pair knob, ordered
candidates) is written to the decision / plan_note.

Concurrent calls enter placement through an **admission queue** (`HUGPY_QUEUE_POLICY`
= `smart` default | `fifo`). Smart: while the FIFO head waits on an eviction or room,
calls that can run now may go ahead — resident-served first (grouped), then
fits-without-eviction by shortest time-to-output — but a group is never reordered and
a head is bypassed at most `HUGPY_QUEUE_BYPASS_MAX` times and never past its
`max_wait_s` (aging). `GET /v1/queue` shows the live queue in order with the reason the
next one runs; each call row records its queue position, bypass count and why it ran.

A rung whose weights live in host RAM (MoE split, ram-only, max-ram, max-gpu) is
admissible only if its CPU claim would get at least **`cpu_floor_cores`** free
*physical* cores (the CPU cliff) — memory fit alone never admits it; a hyperthread
sibling is not an extra core.

Among the admissible candidates the **goal profile** ranks them (minimized left to
right): `quality` → `(q, T+R, T)`; `fast` → `(T, q, R)`; `resident` → `(q, E_hot,
T+R)`, where `q` = index in the approved-representation list (0 = selected), `T` =
estimated seconds to complete from the box's measured rates (unmeasured → ladder
order, never an invented bandwidth), `R` = reload cost of displaced residents,
`E_hot` = pinned/high-priority residents displaced. Waiting is a candidate only when a
resource change is expected within **`max_wait_s`**.

Context has three explicit states per (model × mode × KV type): **range** (shrink on
over `[feasible_min, feasible_max]`), **point** (a single feasible tick, shrink off),
and **infeasible** (no tick fits the worker budget — refused, never loaded as a
point). A resident loaded below its `ctx_max` **grows back** by an idle rebuild when
there is demand, sustained headroom with a margin, a cooldown has passed, and it is
idle (`ctx_restore`).

Per-pair knobs (`user_settings`, sent to the box by core via `POST /v1/settings/pair`;
an absent knob = Auto):

| knob | values | default |
|---|---|---|
| `ctx_pct` | 1–100 (% of trained → applied ctx_max) | Auto = feasible_max |
| `ctx_min_pct` | 1–100 (% of trained → applied ctx_min) | Auto = feasible_min |
| `ctx_range_enabled` | bool — range on/off (read-compat: old `ctx_range`) | Auto rule |
| `ctx_yield` | bool — shrink on/off | Auto rule |
| `ctx_restore` | bool — grow a shrunk resident back to ctx_max | Auto = on when shrink is on |
| `auto_goal` | `quality` \| `fast` \| `resident` | `quality` |
| `quality_alternatives` | `auto` \| `none` \| ordered list of variant keys | `auto` |
| `cpu_floor_cores` | integer ≥ 1 (physical cores floor) | `4` |
| `max_wait_s` | seconds (bounded wait budget) | `120` |
| `alloc_mode` | `gpu-only` \| `ram-only` \| `max-gpu` \| `max-ram` \| `explicit` (pin) | Auto |
| `moe`, `kv_cache_type` | MoE expert split; `f16`/`q8_0`/`q4_0` | Auto; `f16` |

## Glossary

One set of **shared names** for quantities the code already carries. The DB columns
and API fields **do not change** — this maps the existing names to the shared ones.

| shared name | the existing quantity |
|---|---|
| `worker_vram_budget_bytes` | the worker budget: the card with nothing of ours resident |
| `available_vram_bytes` | current usable headroom **after** the reserve (not raw NVML free) |
| `worker_feasible_context_max_tokens` | `feasible_max` — largest ladder tick that fits the worker budget |
| `auto_context_floor_tokens` | `feasible_min` — smallest tick ≥ 0.25 × trained window |
| `context_shortfall_tokens` | `theoretical_max − feasible_max`, in tokens |
| `target_context_tokens` | `ctx_max` — the upper handle (where a load starts) |
| `minimum_retained_context_tokens` | `ctx_min` — the lower handle (shrink floor) |
| `resident_context_tokens` | the loaded engine's capacity (a resident's ctx) |
| `model_context_ceiling_tokens` | `theoretical_max` (the trained window, `ctx_train`) |
| `model_context_floor_tokens` | `theoretical_min` (the ladder's smallest tick) |
| `tok_bytes_gpu` / `tok_bytes_ram` | weight bytes **read per generated token** (the split), not context growth |
| `non_expert_bytes` | estimated GPU weight bytes **including** GPU-resident experts |
| `free_vram_before` / `free_vram_after` | the budgetable headroom, not raw NVML free |

## Layout

```
fitevict/
  core/           pure allocator: types, evict (victim selector), plan (staged fit + quant ladder),
                  flex (ctx-band compress, layers-that-fit), host (EngineHost + drive loop)
  estimate/       sizing, predict, ctx_probe
  serve/          front_door (the OpenAI server), manage (management API), auth (API keys),
                  run_engine (juggle harness), variants, gpu_state, worker_storage_ops
  observability/  store (SQLite) + schema/ migrations, resource_log, calls_view, blobs
  cli/            installer (`fitevict`), console (`fitevict-console`), assist (`fitevict-assist`)
  adapters/       llama_cpp (measure / price / launch / evict), transformers, vision, tasks,
                  db (call log + measurements over the store)
  lifted/         helpers lifted from hugpy: gguf_inspect, gguf_need, hardware, spill_reserve,
                  model_resolve, native_resolve, procutil, timings, no_think, app_dirs, runners/ …
  imports/        shared constants / functions / classes (no heavy imports)
```

See `MIGRATION.md` (package layout), `MIGRATION-imports.md` (shared imports layer) and
`MIGRATION-store.md` (store, management API, installer, API keys).

## Small coreless assistant

`fitevict-assist` searches a drive with a bounded filesystem walk, then uses
the wrapper's direct OpenAI endpoint for optional small-model ranking and chat.
It does not require hugpy-core. The bundled command can fetch Qwen's official
Apache-2.0 Qwen2.5-0.5B-Instruct Q4_K_M GGUF (491 MB) into `MODELS_HOME`; the
download is checked against its published SHA-256 before it is made visible.

```sh
fitevict-assist bootstrap
fitevict-assist scan --query 'small coder'       # discover mounted drives automatically
fitevict-assist scan /mnt/models --query 'small coder'
fitevict-assist scan /mnt/models --query 'specific model name' --add
fitevict-assist add /other/drive/my-model
fitevict-assist call 'Which model did we just add?'
fitevict-assist call 'Explain this model' --model My-MoE-GGUF \
    --quant Q4_K_M --alloc max-gpu --moe --ctx high --knob n_cpu_moe=12
```

Without a root argument, `scan` checks mounted drives, Hugpy stores, Hugging
Face caches, and known local harness locations. It also checks model paths in
running llama.cpp, vLLM and KoboldCpp command lines. Each result records the
source hints that led to it. These hints are verified against files on disk;
they do not make an unknown model runnable by themselves. Scanning has depth,
directory-count and result limits. `add` makes a catalog symlink,
establishes `hugpy.json` from disk evidence (beside the model if writable,
otherwise in a wrapper sidecar), and requests an immediate catalog refresh.
It does not copy model weights. Unsupported weight-only folders are shown by
`scan` but refused by `add`. `call` passes model options and explicit per-call
knobs to wrapper, so its existing fit and feasibility checks remain in force.
The assistant never executes a path or command returned by the small model.

## Test

```
pip install "hugpy-wrapper[test]"
pytest          # every test runs on its own empty store; nothing touches a real one
```
