Metadata-Version: 2.4
Name: hugpy-wrapper
Version: 0.1.53
Summary: The hugpy wrapper (import name: fitevict): a per-box evict-to-fit front door. An OpenAI /v1 server that resolves a called model to its weights (GGUF via llama.cpp, transformers chat/vision, and every non-chat engine: embeddings, whisper ASR, TTS, diffusers image + video, summarization, keywords), places it on the GPU by evicting what doesn't fit, serves it, and logs every call.
Author-email: putkoff <support@hugpy.ai>
License-Expression: LicenseRef-Proprietary
Project-URL: UI, https://www.npmjs.com/package/@hugpy/ui
Project-URL: Homepage, https://hugpy.ai
Keywords: llama.cpp,inference,evict-to-fit,vram,openai,wrapper,allocator,whisper,tts,diffusers,embeddings
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Provides-Extra: transformers
Requires-Dist: torch==2.14.0; extra == "transformers"
Requires-Dist: transformers==5.17.0; extra == "transformers"
Requires-Dist: accelerate==1.15.0; extra == "transformers"
Requires-Dist: bitsandbytes==0.50.2; extra == "transformers"
Requires-Dist: peft==0.20.0; extra == "transformers"
Requires-Dist: compressed-tensors>=0.15.0; extra == "transformers"
Requires-Dist: safetensors==0.8.0; extra == "transformers"
Requires-Dist: tokenizers==0.23.2; extra == "transformers"
Requires-Dist: sentencepiece==0.2.2; extra == "transformers"
Requires-Dist: tiktoken==0.14.0; extra == "transformers"
Requires-Dist: protobuf==7.36.1; extra == "transformers"
Requires-Dist: huggingface_hub==1.31.0; extra == "transformers"

# hugpy-wrapper (`fitevict`)

hugpy's per-box wrapper: an **evict-to-fit front door over llama.cpp** (and transformers). Point any
OpenAI client's base URL at it and one address becomes a self-managing model server.

A `/v1` call carries only a model **name** — never a location. For each request the wrapper resolves
the name to its weights, runs an **evict-to-fit** plan against what is resident and what is being
called, places the model (launch / evict), forwards the call, splits the model's `<think>` out of the
answer, and records the call — raw stamps and engine geometry — in its own store, which it also
reads back to place the next model better.

It needs nothing but an install, a host and a port: no database server, no other hugpy package, no
sudo. The decision core is pure stdlib; hardware facts come from `nvidia-smi`; model facts from the
files on disk.

## Install

`pip` only delivers the package; `fitevict install` is the install.

```
pip install hugpy-wrapper            # or: pipx install hugpy-wrapper

fitevict install --user --port auto                  # your own service, no root
sudo "$(command -v fitevict)" install --system --port auto   # a system service (root once)
fitevict install ... --dry-run                       # print the plan, change nothing
```

No port is ever assumed: give a number, or `auto` for a free one (it is printed and written to the
config). The host defaults to `127.0.0.1`; `--host 0.0.0.0` serves the network.

`--system` sets up the hugpy layout — each hugpy component runs as its own user, and the group
`hugpy` shares only the model store:

| what | where |
|---|---|
| user | `hugpy-wrapper` (system, no login), member of group `hugpy` |
| code | `/opt/hugpy-wrapper/venv` — this exact version, pip-installed |
| config | `/etc/hugpy/wrapper.env` (root:hugpy-wrapper 0640) — host, port, box key, every path |
| state (the store) | `/var/lib/hugpy-wrapper` (0700, created by systemd) |
| cache | `/var/cache/hugpy-wrapper` |
| models | `/var/lib/hugpy/models` (root:hugpy 2775), or `--models DIR` |
| service | `hugpy-wrapper.service` — hardened: read-only system, writes only its own dirs + the models |
| endpoint | `/etc/hugpy/wrapper.endpoint.json` — where it listens (no secret) |
| logs | the journal: `journalctl -u hugpy-wrapper` |

`--user` uses the environment you installed into, `~/.config/hugpy/wrapper.env` and a user unit
(`systemctl --user`; `loginctl enable-linger` keeps it running after logout).

Every step is safe to repeat: a re-run repairs or upgrades and keeps the box key and the port.

```
fitevict status [--system|--user]                    # installed? answering? store size?
fitevict uninstall [--system|--user] [--purge]       # --purge also removes config, store, user
```

Other options: `--worker NAME` (default: the host name), `--engine SPEC` (pip-install an engine),
`--package SPEC` (the code to install, e.g. a wheel file), `--env KEY=VALUE` (repeatable).

Platforms: Linux with systemd today (Ubuntu, Debian, Kali, Fedora, Arch …). macOS (launchd) and
Windows services are next; the command says so rather than half-installing.

### The engine

GGUF models need `llama-server`. It is looked up as `LLAMA_SERVER_BIN`, then under
`HUGPY_ENGINE_DIR`, then on `PATH` — and the service's `PATH` starts with its own environment, so an
engine installed there (`fitevict install … --engine <spec>`) is found. `fitevict install` reports
whether one resolves. Without it the wrapper still runs; GGUF calls fail until an engine is there.

```
pip install "hugpy-wrapper[transformers]"   # the transformers engine (into its own venv)
```

## Use

```
curl -s http://127.0.0.1:<port>/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"<name>","messages":[{"role":"user","content":"hi"}],"max_tokens":128}'
```

Any OpenAI client works: `base_url=http://<host>:<port>/v1`. The model is named, not located; the
wrapper finds it, fits it, serves it, logs it.

## API keys

Keys look like `hpk_<id>_<secret>` and are sent as `Authorization: Bearer …` — what every OpenAI
client sends for its `api_key`, so a user just sets `OPENAI_API_KEY=hpk_…`.

- **The box key** (`hpk_box_…`, admin) is written once by `fitevict install` into the service config.
  It is the root of trust for the box; hugpy-core uses it to manage the box.
- **More keys**, made through the running service:

  ```
  fitevict keys create --name laptop [--scope use|admin] [--expires-days 90]   # printed once
  fitevict keys list [--all]
  fitevict keys revoke <id>
  ```

  Only a hash of each key is stored. Revoked and expired keys stop working at once.

Who needs a key:

| route | rule |
|---|---|
| OpenAI routes (`/v1/chat/completions`, `/v1/models`, …) | `FITEVICT_REQUIRE_KEY`: `outside` (default) — no key from loopback and private networks, a key from anywhere else; `never`; `always` |
| management (`/v1/telemetry`, `/v1/settings/*`, `/v1/operator/*`, `/v1/keys`, `POST /limits`) | an admin key, except from the box itself (loopback) |
| `/health` | never |

What counts as inside is `FITEVICT_TRUSTED_NETS` (default: loopback, 10/8, 172.16/12, 192.168/16,
fd00::/8, fe80::/10). **Behind a proxy** (nginx, a tunnel), every request arrives from 127.0.0.1: a
request with forwarding headers counts as *outside* unless its immediate peer is listed in
`FITEVICT_TRUSTED_PROXIES` — only then is `X-Forwarded-For` believed. A public proxy can never make
the internet "local".

## The store (telemetry)

The wrapper keeps every call and every measurement in one SQLite file it creates on first use
(`FITEVICT_HOME`, else systemd's state directory, else the per-OS data directory; 0700 / 0600). Its
placement decisions read it back: the measured CUDA context, llama.cpp's projections, serving
growth, KV structure, host memory, a model's largest real use. Every measurement points at the call
that took it; views compute the figures. A damaged file is set aside and recreated — everything in
it is re-learnable. Schema changes ship as numbered migrations.

hugpy-core (or anything with an admin key) works with a box through the management API:

```
GET  /v1/telemetry?table=call_log&since=<id>&limit=<n>   rows after a cursor (call_log, resource_log,
                                                         evictions, operator_actions) + next / more
GET  /v1/telemetry/state                                 GPU / host state, residents, models, facts, settings
POST /v1/operator/actions   {"action","model_key"}       evict | evict-all | free-ram | remove-model | restart
GET  /v1/operator/actions/<id>
GET|POST /v1/settings/pair  {"model_key","settings","env_profile"}   a model's knobs on this box
POST /v1/settings/quant-facts {"model_key","file","size_bytes","kv_cost"}   size + KV ticks
POST /v1/settings/env-profile {"name","base","packages"}
GET|POST /v1/keys, POST /v1/keys/<id>/revoke
```

A call's own knobs (`model@options`, request fields) still override the stored settings per call.
`python -m fitevict.observability.calls_view --port <n|auto>` serves a read-only page of the latest
calls on 127.0.0.1 (it shows every call's bodies: run it as the service's user, keep it local).

## Every model kind, one front door

Chat (GGUF via llama-server, transformers via `tf_child`), vision (`vision_child`) and every
non-chat engine go through the SAME placement (evict-to-fit), the SAME call log row and the SAME
residents / models catalog. Non-chat engines are runners on one generic child
(`lifted/task_child.py`); each is one file in `lifted/runners/` plus one `RunnerSpec`:

| route | runner | serves |
|---|---|---|
| `POST /v1/embeddings` | `embed` | sentence-transformers encoders |
| `POST /v1/audio/transcriptions`, `/v1/audio/translations` | `asr` | whisper (multipart `file` or `file_path`/`file_b64`) |
| `POST /v1/audio/speech` | `tts` | chatterbox — returns raw audio bytes |
| `POST /v1/images/generations`, `/edits`, `/variations` | `imagegen` | diffusers image pipelines — `b64_json` (or `response_format: path`) |
| `POST /v1/videos/generations` | `videogen` | diffusers video pipelines (Wan / LTX / CogVideoX / Mochi …) — mp4 path |
| `POST /v1/summarize` | `summarize` | seq2seq summarizers (bart / pegasus / t5 / led) |
| `POST /v1/keywords` | `keywords` | KeyBERT over a stamped embedding model |

Binary outputs are written to `FITEVICT_ARTIFACT_DIR` and returned in the endpoint's OpenAI shape;
the call log stores their exact bytes + sha256, never the bytes. Uploads are spooled to
`FITEVICT_SPOOL_DIR` and removed after the call.

**Env profiles** (`lifted/profiles.py`): a runner's extra pip deps live in a per-profile venv at
`<engine>/envs/<name>` — an overlay on the box's worker base (`FITEVICT_TF_PYTHON`) or, when a model
pins its own torch (chatterbox), an isolated venv. Declared in `runners.PROFILES`, overridable with
`POST /v1/settings/env-profile`; a model is pinned to one with `POST /v1/settings/pair`
(`env_profile`). A profile is built on first use, off the request path; until it is ready the call
fails as data (503 `profile_materializing`). `GET /profiles` reports every profile's state.

## Cores

Every engine gets its own cores, claimed once at load from the plan's RAM:VRAM split
(CORES.md):
- An engine with nothing in RAM gets one core, which is all it uses.
- An engine reading weights from RAM gets the cores that still make it faster, by the
  box's measured bandwidth curve. Until that curve is known, it gets every free core.
- Two engines never share a core.
- Cores busy with other work are avoided, and one core stays reserved for the wrapper and
  the OS.

llama.cpp gets `-t N --cpu-mask … --cpu-strict 1`. The engine process is confined to the
claim by CPU affinity, which needs no privilege.

Each call row records the claim (`cpu_set`, `threads`, `cpu_source`) and the CPU time the
call used (`cpu_user_ms`, `cpu_sys_ms`). `GET /v1/cores` shows the box's topology, per-CPU
load, every engine's claim and the fitted speed constants.

Per model you can raise the claim with `cpu_priority` or pin it with `cpu_cores` (a count
or a CPU list) in `POST /v1/settings/pair`. A pin is honoured even when it has to take
busy cores, and the row records what it took.

## Configuration

`fitevict install` writes these into the service config; you rarely set them by hand.

| variable | meaning |
|---|---|
| `FITEVICT_HOST`, `FITEVICT_PORT` | where the front door listens (`fitevict-serve --host/--port`; `auto` = free port) |
| `FITEVICT_TOKEN` | the box key |
| `FITEVICT_REQUIRE_KEY`, `FITEVICT_TRUSTED_NETS`, `FITEVICT_TRUSTED_PROXIES` | who needs a key (above) |
| `FITEVICT_WORKER` | this box's name in the call log (default: the host name) |
| `FITEVICT_HOME` | the store's directory |
| `HUGPY_DATA_DIR`, `HUGPY_CACHE_DIR`, `HUGPY_ENGINE_DIR` | data / cache / engine dirs |
| `MODELS_HOME`, `DEFAULT_ROOT` | the model store (and the dir that contains it) |
| `FITEVICT_CPU_PIN`, `FITEVICT_CPU_RESERVE`, `FITEVICT_CPU_BUSY` | core claims on/off, cores kept back (1), busy threshold (0.5) — CORES.md |

## Layout

```
fitevict/
  core/           pure allocator: types, evict (victim selector), plan (staged fit + quant ladder),
                  flex (ctx-band compress, layers-that-fit), host (EngineHost + drive loop)
  estimate/       sizing, predict, ctx_probe
  serve/          front_door (the OpenAI server), manage (management API), auth (API keys),
                  run_engine (juggle harness), variants, gpu_state, worker_storage_ops
  observability/  store (SQLite) + schema/ migrations, resource_log, calls_view, blobs
  cli/            installer (`fitevict`), console (`fitevict-console`)
  adapters/       llama_cpp (measure / price / launch / evict), transformers, vision, tasks,
                  db (call log + measurements over the store)
  lifted/         helpers lifted from hugpy: gguf_inspect, gguf_need, hardware, spill_reserve,
                  model_resolve, native_resolve, procutil, timings, no_think, app_dirs, runners/ …
  imports/        shared constants / functions / classes (no heavy imports)
```

See `MIGRATION.md` (package layout), `MIGRATION-imports.md` (shared imports layer) and
`MIGRATION-store.md` (store, management API, installer, API keys).

## Test

```
pip install "hugpy-wrapper[test]"
pytest          # every test runs on its own empty store; nothing touches a real one
```
