Metadata-Version: 2.4
Name: hugpy-engine
Version: 0.3.7
Summary: Hugpy engine: model discovery and classification, allocation and eviction, native llama.cpp and GGUF runners, and the prompt-to-reply inference path with task and placement seams
Author-email: putkoff <support@hugpy.ai>
License-Expression: LicenseRef-Proprietary
Keywords: hugpy,llm,self-hosted,inference,llama.cpp,gguf,transformers,model-serving
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: hugpy-platform
Requires-Dist: abstract-essentials
Requires-Dist: pydantic>=2

# hugpy-engine

Inference resolution for the Hugpy arms. It turns "a prompt and maybe a file"
into a model and a task, and hands the call to central's `/v1` relay. Central
picks the worker, and that worker's wrapper does evict-to-fit, load, serving
and the call log. The engine itself does not load models in fleet mode.

Distribution `hugpy-engine`, import `hugpy_engine`. Dependencies:
`hugpy-platform`, `abstract-essentials`, `pydantic>=2`.

**Status: LIVE.** It is a library only: no service, no console script.

- Installed in `/opt/hugpy-media/venv` (0.3.7) and `/opt/hugpy-video/venv`
  (0.3.8).
- Imported by `hugpy_media`, `hugpy_video` and `hugpy_oracle`.
- Core's live basic mode does not use it. Only core's legacy
  `config/prompt.py` does, and that code is not live.

On 2026-10-15 the engine's last ties to `hugpy-storage` and `hugpy-control`
are being removed, targeting 0.3.9 and later:

- `vl_gguf_tasks` is ported in from `hugpy_storage.hugpy_marker`.
- The `hugpy_storage.model_paths.route_destination` call is dropped; it was
  dead in fleet mode.
- `hugpy_storage.console.downloader.model_status` is replaced by registry
  availability from `/v1/models`.
- The guarded imports of the retired `hugpy_control.settings.settings_store`
  are deleted.

Check `pyproject.toml` for the current version. See
[docs/packages.md](../../docs/packages.md#versions-in-flight).

## Public API

```python
from hugpy_engine import (
    execute_prompt, execute_prompt_stream, execute_chat_stream,
    resolve, resolve_model_key,
    MODEL_REGISTRY, get_model_registry, refresh_registry,
    normalize_prompt_kwargs, infer_arg_name, runner_for,
)
```

Positional arguments to `execute_prompt` are inferred by type:

| type | meaning |
|---|---|
| `str` | a model id or a prompt |
| `int` | `max_new_tokens` |
| `float` | `temperature`, then `top_p` |
| `bool` | `do_sample` |
| `list` | `messages` |

`execute_prompt_stream` is a single resolve, build and runner pass yielded as
events (`TokenEvent`, `DoneEvent`, `ErrorEvent`). It accepts `cancel_event`.
`execute_chat_stream` adds chat continuation on top of it.

Chat and vision run today. A task whose runner has no front-door endpoint yet
resolves to a clean, named error (fail-as-data).

## execute_prompt: infer, rank, assure

Full reference: [docs/execute_prompt.md](../../docs/execute_prompt.md).

`execute_prompt` is not plain task routing. Its defining contract
(operator): it "would infer programmatically what the user was looking to
prompt, via file inspection, prompt type, and anything that necessarily
indicated a model type — most assuredly to receive a response than most any
other query method." Task bindings are only ONE input. The mechanic has three
stages, all in the stdlib-only module `hugpy_engine.resolvers.task_resolution`,
so that central can later serve the same resolution.

1. **INFER.** `task_resolution.infer(args, kwargs)` works out WHAT to prompt
   from several signals:
   - positional args: a bare prompt, a messages list, an existing file, a
     model-ish string, a `data:` URL or an http(s) URL
   - file type: the extension maps to image, audio, video, document, code or
     text
   - message parts: `image_url`, `input_image`, audio, `data:` URLs
   - explicit cues: `task=`, `model_format=`

   It derives the task only when the caller gave no explicit `task` and no
   explicit `model_key`:

   | input | task |
   |---|---|
   | image | image-text-to-text |
   | audio or video | asr |
   | pdf or doc | extract + text |
   | text | text-generation |

   One-call use works: `execute_prompt("describe this", "/path/img.png")`
   needs no task or model. Inference only ever *adds* a task. It never
   overrides an explicit one, and a failure inside it never blocks the call.

2. **RANK.** `task_resolution.resolve(...)` returns an ORDERED candidate
   list, strongest first, by source precedence:

       explicit model_key → binding → configured default → capability match

   Capability matches are ranked **resident > available on a healthy worker
   > other**. A row whose `primary_task` matches the task is preferred, then
   order is deterministic by id. "No model in the fleet lists task X" appears
   ONLY when no registry model can serve X. It is never a dead-end "bind one
   in Settings".

3. **ASSURE.** `execute_prompt` tries the candidates in order.
   - On a MODEL-SIDE failure (worker down, load refusal, timeout, 5xx, empty
     reply), it moves to the next candidate.
   - A CALLER error (4xx validation, bad args) is surfaced immediately. The
     same bad request would fail on every model, so it is never cascaded.
   - The first real answer wins and carries `resolution` metadata:
     `{task, source, model_key, attempts:[...]}`.
   - When every candidate fails, it returns a fail-as-data result (`ok=False`)
     with the full attempt list, never a bare exception that hides what was
     tried.

### Registry and bindings source

**The registry** is built from basic core's `GET /v1/models` rows: `id`,
`hugpy_json.tasks`, `task_override`, `engine`, `resident`, `available`,
`workers` and `resident_on`. `derive_model_config_row` maps each row onto
`ModelConfig`. If that fails, it falls back to the legacy `GET /models`, and
then to a local `hugpy.json` disk scan of `HUGPY_MODELS_HOME` /
`LLAMA_CPP_MODEL_DIR`. The scan is fail-soft and fails fast on an EIO models
disk.

**Bindings** are read from `GET /v1/task-bindings` (`{"defaults": {...}}`),
then the legacy `GET /prompt/tasks`. A 404 means "no bindings". Live basic
core (0.4.40) serves neither route, so the binding stage is empty today and
resolution relies on explicit keys, configured defaults and capability
matches.

Both reads are TTL-cached and never raise on the resolve path.

### Configuration

`hugpy_engine.imports.constants` names every environment variable, timeout and
TTL the chain reads:

| variable | default | meaning |
|---|---|---|
| `HUGPY_CORE_URL` (legacy alias `HUGPY_FITEVICT_FRONT_DOOR`) | `http://127.0.0.1:7020` | central's base URL |
| `HUGPY_REGISTRY_TTL_S` | 30 | registry and bindings cache lifetime |
| `HUGPY_REGISTRY_TIMEOUT_S` | 10 | registry fetch timeout |
| `HUGPY_TASK_ENDPOINT_TIMEOUT_S` | 900 | task endpoint call timeout |
| `HUGPY_FITEVICT_TIMEOUT_S` | 900 | `/v1` call timeout |
| `HUGPY_MODELS_HOME`, `LLAMA_CPP_MODEL_DIR` | unset | roots for the local disk-scan fallback |

The same module holds the `TASK_DEFAULTS`, `MEDIA_DEFAULTS` and
`MEDIA_TYPE_TO_TASK` tables.

## The task plugin seam

`hugpy_engine.tasks` holds the task registry. Packages add tasks through the
`hugpy_engine.tasks` entry-point group (a zero-argument callable), which
`tasks.load_entry_points()` imports. Two packages register today:

- `hugpy-media`: `media = "hugpy_media.plugin:register"`
- `hugpy-video`: `video = "hugpy_video.plugin:register"`

`task_endpoints.TASK_ROUTES` maps tasks to the front-door endpoints.

## Layout

```
hugpy_engine/
  dispatch/       execute_prompt / execute_prompt_stream / execute_chat_stream; per-(model, task) runner cache
  resolvers/      task_resolution (infer / rank, stdlib only), model_resolver (resolve), assure_model_key,
                  categories/ (task + framework registries)
  config/         models/ (MODEL_REGISTRY, models_config, defaults), main
  llama/runners/  fitevict_runner: the runner that sends a call to central's /v1
  schemas/        chat, event, metadata, model, runner, task schemas (pydantic)
  fit/, placement.py   placement types and null single-box defaults
  tasks.py, task_endpoints.py, task_deps.py, categories.py, name_match.py
  utils/          json_scavenge, no_think
  imports/        constants (every env name), functions, classes, init_imports
                  (docs/packages.md#the-imports-layout-convention)
```

See also:

- [docs/execute_prompt.md](../../docs/execute_prompt.md)
- [docs/arms.md](../../docs/arms.md)
- [docs/packages.md](../../docs/packages.md)
