Metadata-Version: 2.4
Name: hugpy-router
Version: 0.0.1
Summary: hugpy's router: one OpenAI-compatible address in front of every box's hugpy-wrapper front door. Picks the box per call (a model already serving with the same settings first, else a box that can load it), proxies the call (streaming, uploads, binary), bounces a refused call to the next box, and logs one row per decision.
Author-email: putkoff <support@hugpy.ai>
License-Expression: LicenseRef-Proprietary
Project-URL: Homepage, https://hugpy.ai
Keywords: llm,router,openai,proxy,gpu,hugpy
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: AsyncIO
Classifier: Intended Audience :: Developers
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: aiohttp>=3.9
Requires-Dist: hugpy-wrapper>=0.1.52
Provides-Extra: test
Requires-Dist: pytest; extra == "test"

# hugpy-router

One OpenAI-compatible address in front of every box's
[hugpy-wrapper](https://pypi.org/project/hugpy-wrapper/) front door. Per call it picks the box,
proxies the call — streaming, uploads and binary answers pass through as they arrive — and when a box
refuses before the first byte it tries the next one. Every call is one row in the router's own log,
linked to the box's row.

```
client ──> hugpy-router ──> hugpy-wrapper (box A) ──> engine
                 └────────> hugpy-wrapper (box B) …   (bounce on refusal)
```

## How a box is chosen

1. **Pin** — `{"alloc": {"worker": "<box>"}}` in the body: that box or a refusal. Never rerouted.
2. **Eligible** — online, the call's pool, not blocked, not already tried for this call.
3. **Serving first** — a box where the model is loaded, healthy, not loading, *and running the same
   settings* (`model@options` — the options the router last sent that box for that model). A loaded
   model with other settings is not a match: the box would reload it. Idle before busy, then most
   recently used.
4. **Load** — boxes whose catalog holds the model, most free GPU first, then least recently picked.
5. **Refusal** — none qualifies: `503 no_box` with every box's reason.

A box that answers `load_error`, `insufficient_resources`, `profile_materializing`, model not found,
502/503/504, or is unreachable — before the first byte — is skipped and the next one tried (at most 3
boxes per call). Other errors (e.g. a 400) go back to the client exactly as the box sent them.

## Run

```
pip install hugpy-router
hugpy-router boxes add --name ae --url http://127.0.0.1:40717 --key-file /etc/hugpy/wrapper.env \
    [--store /var/lib/hugpy-wrapper/fitevict.sqlite3]
hugpy-router serve --port <n|auto> [--host 127.0.0.1]
```

No port is assumed. `--store` (optional): when the box's store file is readable on this machine
(hugpy-wrapper `--system` makes it readable by group `hugpy`), the router reads the box's state
straight from it, re-reading only when SQLite reports a commit. Otherwise it follows the box with a
long-poll on `GET /v1/telemetry/state?wait=30&since=<n>` (hugpy-wrapper ≥ 0.1.52). Either way the
box's `/health` decides online / offline.

The boxes file (`HUGPY_ROUTER_BOXES`, default `<config dir>/router/boxes.json`, 0600) holds each box's
key; clients never see box keys.

## Routes

| route | |
|---|---|
| `POST /v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/audio/*`, `/v1/images/*`, `/v1/videos/generations`, `/v1/summarize`, `/v1/keywords` | proxied |
| `GET /v1/models` | every box's catalog, with `boxes` and `resident_on` per model |
| `GET /health` | liveness |
| `GET /v1/router/state` | admin: each box's link, online, residents, catalog size, room; the settings sent |
| `GET /v1/telemetry?since=<id>&limit=<n>` | admin: the router's call log (cursor) |
| `GET/POST /v1/keys`, `POST /v1/keys/<id>/revoke` | admin: API keys |

Responses carry `X-Hugpy-Box` and `X-Hugpy-Router-Request-Id`; the box's own call row has that id as
its parent.

## Keys

The same scheme as hugpy-wrapper (`hpk_<id>_<secret>`, `Authorization: Bearer`), with the router's
own settings: `HUGPY_ROUTER_KEY` (its admin key), `HUGPY_ROUTER_REQUIRE_KEY` (`outside` by default —
no key from loopback and private networks), `HUGPY_ROUTER_TRUSTED_NETS`,
`HUGPY_ROUTER_TRUSTED_PROXIES` (forwarded requests count as outside unless their proxy is listed).
Admin routes need an admin key except from loopback.

```
hugpy-router keys create --name laptop [--scope use|admin] --router-port <n>
```

## Store

`HUGPY_ROUTER_HOME`, else systemd's state directory, else `<data dir>/router`: one SQLite file
(hugpy-wrapper's store machinery — WAL, migrations, integrity check): `route_log` (one row per call:
decision, attempts, box, timings) and `api_keys`.

## Layout

```
hugpy_router/
  core/           types + decide (pure, stdlib only)
  links/          box links (local store file / remote long-poll) + the boxes file
  serve/          aiohttp app: proxy (hot path), management API
  observability/  the router's store + schema/
  cli/            hugpy-router serve / boxes / keys
  imports/        constants (every env name), functions, classes
```

0.0.1 is phase 1 of the design (one or many boxes, local or remote links). Next: an installer on the
hugpy layout (user `hugpy-router`, `/etc/hugpy/router.env`, unit), and hugpy-core configuring the
router (verdicts, pools, blocks, pair settings) and reading through it.
