Metadata-Version: 2.5
Name: horde-image-utilities
Version: 0.2.4
Summary: Modular image utility microservice with optional heavy backends
License-File: LICENSE
License-File: NOTICE
Requires-Python: >=3.12
Requires-Dist: pillow>=11.0.0
Requires-Dist: pydantic-settings>=2.13.1
Requires-Dist: pydantic>=2.13.0
Provides-Extra: annotators
Requires-Dist: addict; extra == 'annotators'
Requires-Dist: einops; extra == 'annotators'
Requires-Dist: huggingface-hub; extra == 'annotators'
Requires-Dist: matplotlib; extra == 'annotators'
Requires-Dist: numpy; extra == 'annotators'
Requires-Dist: opencv-contrib-python-headless; extra == 'annotators'
Requires-Dist: pydot; extra == 'annotators'
Requires-Dist: pyyaml; extra == 'annotators'
Requires-Dist: scikit-image; extra == 'annotators'
Requires-Dist: scipy; extra == 'annotators'
Requires-Dist: timm; extra == 'annotators'
Requires-Dist: torch; extra == 'annotators'
Requires-Dist: torchvision; extra == 'annotators'
Requires-Dist: transformers; extra == 'annotators'
Requires-Dist: yapf; extra == 'annotators'
Provides-Extra: mediapipe
Requires-Dist: mediapipe; extra == 'mediapipe'
Provides-Extra: rembg-cpu
Requires-Dist: rembg[cpu]; extra == 'rembg-cpu'
Provides-Extra: rembg-cuda
Requires-Dist: onnxruntime-gpu; extra == 'rembg-cuda'
Requires-Dist: rembg[gpu]; extra == 'rembg-cuda'
Provides-Extra: rembg-rocm
Requires-Dist: onnxruntime-rocm; (sys_platform == 'linux') and extra == 'rembg-rocm'
Requires-Dist: rembg[rocm]; (sys_platform == 'linux') and extra == 'rembg-rocm'
Provides-Extra: server
Requires-Dist: fastapi[standard]>=0.135.3; extra == 'server'
Description-Content-Type: text/markdown

# horde-image-utilities

Isolated heavy image capabilities for the AI Horde worker ecosystem.

Some image utilities a worker needs (background removal via rembg/onnxruntime, and later controlnet
annotators and layer diffusion) carry heavy, conflict-prone native dependencies. Pulling those into the
worker's main environment bloats it and risks dependency clashes. This package keeps them out of the
way: it runs each capability as a small FastAPI service in its own virtual environment, and the worker
talks to it over loopback HTTP using a dependency-free client.

It ships four cooperating pieces:

- **Service**: a FastAPI application exposing capability endpoints (for example `POST /rembg/remove-background`) plus an always-on operational surface under `/ops`.
- **Client**: a standard-library-only HTTP client (`HordeImageUtilitiesClient`) that the lean consumer uses to call the service. Its only runtime dependency is Pillow, already a base dependency.
- **Launcher**: `CapabilityServerProcess` starts, health-checks, and tears down the service subprocess, optionally under a different interpreter so the heavy dependencies live in a separate venv.
- **Provisioning**: `build_provision_commands` produces the `uv` commands that create a capability venv and install this package with the right extras.

## Two-mode model

- **Lean consumer environment**: installs the base package only (Pillow, pydantic). It imports `horde_image_utilities.client` and `horde_image_utilities.launcher` to drive a capability process. No FastAPI, rembg, torch, or onnxruntime enter this environment.
- **Capability environment**: a separate venv that installs the `server` extra plus exactly one backend extra. It runs the actual model inference. Keeping this environment separate is what makes the dependency isolation real.

## Install matrix

The base install has no heavy backends. Capabilities are opt-in extras:

| Extra | Provides |
| --- | --- |
| `server` | FastAPI transport for running the service (`fastapi[standard]`). |
| `rembg_cpu` | Background removal on CPU (`rembg[cpu]`). |
| `rembg_cuda` | Background removal on NVIDIA CUDA (`rembg[gpu]` + `onnxruntime-gpu`). |
| `rembg_rocm` | Background removal on AMD ROCm, Linux only (`rembg[rocm]` + `onnxruntime-rocm`). |
| `annotators` | ControlNet preprocessors (canny, hed, depth, normal, openpose, seg, ...) via torch. A single extra serves every accelerator backend; the torch wheel index is the installer's concern. |
| `mediapipe` | MediaPipe face-mesh detector (`mediapipe_face`). |

The three `rembg_*` extras are mutually exclusive: exactly one accelerator backend per environment. This
is enforced by a `[tool.uv].conflicts` declaration, so `uv` refuses to install more than one. The
`annotators` extra has no accelerator split (no detector in scope uses onnxruntime), so no conflict
entry is needed.

> **Do not co-install `mediapipe` with `annotators` in the same environment.** `mediapipe` pulls in
> `opencv-contrib-python` (the GUI wheel) while `annotators` pulls in `opencv-contrib-python-headless`.
> Two opencv wheels sharing the `cv2` namespace break `cv2` (it imports as an empty namespace). Until
> this is resolved via a dependency override at deployment-pin compile time, keep `mediapipe` in its own
> environment. The `mediapipe_face` detector is future work and is not required by any worker capability.

Provision a capability venv (CUDA background removal shown):

```bash
uv venv .venv-rembg
uv pip install --python .venv-rembg "horde-image-utilities[server,rembg_cuda]"
```

## Quickstart

Run the service:

```bash
# In the capability environment
python -m horde_image_utilities
# or the console script:
horde-image-utilities
```

Call it from the lean consumer environment:

```python
from PIL import Image

from horde_image_utilities.client import HordeImageUtilitiesClient

client = HordeImageUtilitiesClient("http://127.0.0.1:7860")
if client.health(timeout=5.0):
    result = client.remove_background(Image.open("input.png"))
    result.save("output.png")
```

`health()` and `remove_background()` accept a per-call `timeout` that overrides the client-wide one, so a
liveness probe can use a short budget while the work request keeps the long one. A failed call reports
whether the service could not be reached at all or was reached but did not answer within the timeout.

Or let the launcher own the subprocess lifecycle (point `python_executable` at the capability venv):

```python
from horde_image_utilities.launcher import CapabilityServerProcess

with CapabilityServerProcess(python_executable="/path/to/.venv-rembg/bin/python") as server:
    memory_report = server.client.get_memory_report()
```

## Configuration

Settings are read from the environment with the `HIU_` prefix (see
`horde_image_utilities.config.ServiceSettings`):

| Variable | Default | Purpose |
| --- | --- | --- |
| `HIU_HOST` | `127.0.0.1` | Bind host for the service. |
| `HIU_PORT` | `7860` | Bind port for the service. |
| `HIU_MAX_REQUEST_SIZE` | `67108864` | Maximum accepted request size in bytes. |
| `HIU_CONTROLNET_MAX_CONCURRENT` | `1` | Concurrent controlnet processing slots. |
| `HIU_CONTROLNET_QUEUE_TIMEOUT` | `30.0` | Seconds to wait for a controlnet slot. |
| `HIU_ISOLATE_MODEL_CACHE` | `true` | Store model files under `AIWORKER_CACHE_HOME` when set. |
| `HIU_ALLOW_DOWNLOADS` | `false` | Allow runtime model downloads when missing from cache. |
| `HIU_ANNOTATOR_MODEL_DIR` | unset | Base directory holding annotator checkpoints. Defaults to the `annotators/` cache directory below. |
| `HIU_VENDOR_DIR` | unset | Base directory for the cloned upstream annotator node. Defaults to a `vendor/` sibling of the cache directories below. |

`HIU_ALLOW_DOWNLOADS` defaults to `false`: the service will not fetch missing models at runtime unless
you opt in. Pre-download models, or set it to `true`.

## Cache layout

When `HIU_ISOLATE_MODEL_CACHE` is enabled and `AIWORKER_CACHE_HOME` is set, model files are stored under
an isolated, per-capability directory:

```
$AIWORKER_CACHE_HOME/horde/image-utilities/rembg/                     # rembg ONNX models
$AIWORKER_CACHE_HOME/horde/image-utilities/controlnet_aux/            # controlnet_aux base
$AIWORKER_CACHE_HOME/horde/image-utilities/controlnet_aux/annotators/ # annotator checkpoints (default)
$AIWORKER_CACHE_HOME/horde/image-utilities/vendor/                    # cloned upstream annotator node (default)
```

If `AIWORKER_CACHE_HOME` is not set, or isolation is disabled, each backend falls back to its own
default (`U2NET_HOME` / `~/.u2net` for rembg, the Hugging Face cache for controlnet). The service logs a
warning when conflicting cache environment variables are detected.

### Annotator checkpoints

The annotators capability resolves weight files under `HIU_ANNOTATOR_MODEL_DIR` (defaulting to the
`annotators/` directory shown above), laid out as `<repo_id>/<filename>` exactly as the worker
pre-populates them (for example `lllyasviel/Annotators/ControlNetHED.pth`). Missing weights return HTTP
409 rather than triggering a download (downloads are off unless `HIU_ALLOW_DOWNLOADS=true`).

The transformers-hub control types are the exception: their weights load through the transformers
library from the standard Hugging Face hub cache, so the deployment must point `HF_HOME` (or
`HUGGINGFACE_HUB_CACHE`) at a cache pre-populated with the relevant model. These are `normal` and
`midas_depth` (`Intel/dpt-hybrid-midas`), `zoe_depth` (`Intel/zoedepth-nyu-kitti`), `depth_anything`
(`LiheYoung/depth-anything-large-hf`), and `oneformer_ade20k` / `oneformer_coco`
(`shi-labs/oneformer_*_swin_large`).

Implemented control types span the classic set and the extended surface the AI-Horde annotation form
and worker pre-annotation use:

- Classic: `canny`, `scribble`, `hed`, `fakescribbles`, `mlsd`, `depth` (LeReS), `normal` (MiDaS),
  `openpose`, `seg` (UniFormer).
- Weightless algorithmic: `binary`, `standard_lineart`, `scribble_xdog`, `pyracanny`, `color`,
  `shuffle`, `recolor_luminance`, `recolor_intensity`, `tile`, `tile_ttplanet_guided`,
  `tile_ttplanet_simple`.
- Annotator-layout weighted: `lineart`, `lineart_anime`, `lineart_anime_denoise`,
  `pidinet`, `scribble_pidinet`, `teed`, `normal_bae`, `depth_anything_v2`.
- Transformers-hub weighted: `midas_depth`, `zoe_depth`, `depth_anything`, `oneformer_ade20k`,
  `oneformer_coco`.

Per-detector parameters match the defaults hordelib passes through the `comfyui_controlnet_aux` AIO
preprocessor (only the resolution knob is set; every other widget stays at its node default), so
pre-annotation here is interchangeable with hordelib's in-graph annotation. Notably `lineart`
runs with coarse disabled, mirroring hordelib, which drives it through the
`LineArtPreprocessor` node without setting the coarse widget.

The `seg` runtime comes from the clone's bundled MMSegmentation/MMCV subtree, which runs on CPU without
compiled ops given the pure-python MMCV config packages the `annotators` extra pulls in. The
`mediapipe_face` control type is registered but not implemented and returns HTTP 501.

At startup a best-effort prefetch (enabled only when `HIU_ALLOW_DOWNLOADS=true`) warms the classic
control types' weights before the extended set, so a fresh install can serve the legacy surface quickly
while the extended weights trickle in.

### Annotator detector source (clone at a pin)

The annotator detector algorithms are not vendored into this repository. On first use (and, best-effort,
at service startup) the capability clones [`comfyui_controlnet_aux`](https://github.com/Fannovel16/comfyui_controlnet_aux)
at a commit pinned in `horde_image_utilities/annotators/upstream_manifest.json`, prepends the clone's
`src` to `sys.path`, and imports `custom_controlnet_aux` directly (the clone is not pip-installed). This
mirrors how hordelib pins ComfyUI: the acquisition is idempotent and self-healing, performing no network
access once the clone is already at the pin. Two runtime monkeypatches redirect the upstream weight
loading through this package's resolver (so pre-populated checkpoints are used and missing files return
HTTP 409) and make the LeReS detector load only its depth model, never the pix2pix "boost" model. See
the repository `NOTICE` for the attribution and the exact patch sites.

Pre-bake the clone for an offline or image-baking step (git only; needs neither torch nor the annotator
extra):

```bash
python -m horde_image_utilities.annotators
```

## The `/ops` surface

The ops endpoints mount in every install, including lean ones with no heavy backend. They assume a
trusted loopback caller (the worker that launched the process) and carry no authentication.

| Endpoint | Purpose |
| --- | --- |
| `GET /ops/memory` | Report process resident-set size, torch CUDA figures (null without torch), the onnxruntime session count, and loaded model names. |
| `POST /ops/release-cache` | Release framework caches (torch's CUDA allocator when present) and return a fresh memory report. |
| `POST /ops/unload` | Drop cached sessions for one capability (`{"capability": "background_removal"}`) or all (`{"capability": null}`). |
| `POST /ops/shutdown` | Schedule a graceful process exit after the response flushes. |

The client exposes typed methods for each: `get_memory_report()`, `release_cache()`,
`unload(capability=None)`, and `shutdown()`.

A `GET /health` liveness endpoint is always available regardless of which backends are installed. It is
answered on the event loop while capability work runs in worker threads, so it reports promptly even
while a long background removal is in progress; background removals themselves are serialised, so a
burst of them queues instead of running in parallel.

## Development

```bash
uv sync
uv run ruff format . && uv run ruff check . --fix
uv run pytest
```

Tests that require a rembg backend are skipped automatically when it is not installed. The
package version is derived from the release tag via `hatch-vcs`; there is no version string to bump in
source.
