Metadata-Version: 2.4
Name: hugpy-media
Version: 0.3.12
Summary: Hugpy media: engine task plugins for speech recognition, TTS, embeddings, keywords, summaries, vision, image generation, ComfyUI and document/URL extraction
Author-email: putkoff <support@hugpy.ai>
License-Expression: LicenseRef-Proprietary
Project-URL: Homepage, https://hugpy.ai
Project-URL: Documentation, https://github.com/hugpy/hugpy/blob/main/staging/hugpy-media/package/README.md
Project-URL: Repository, https://github.com/hugpy/hugpy
Project-URL: Source, https://github.com/hugpy/hugpy/tree/main/staging/hugpy-media/package
Project-URL: Issues, https://github.com/hugpy/hugpy/issues
Project-URL: Changelog, https://github.com/hugpy/hugpy/releases
Project-URL: Architecture, https://github.com/hugpy/hugpy/blob/main/staging/FLEET_ARCHITECTURE.md
Keywords: hugpy,llm,self-hosted,whisper,embeddings,vision,image-generation,tts,media-intelligence
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: media_intelligence[documents]<0.3,>=0.1.3
Requires-Dist: hugpy-platform>=0.3.0
Requires-Dist: hugpy-engine>=0.3.9
Requires-Dist: flask>=2
Requires-Dist: abstract_flask>=0.0.0.1071
Requires-Dist: abstract_essentials
Requires-Dist: pydantic>=2
Requires-Dist: requests
Requires-Dist: beautifulsoup4
Requires-Dist: PyPDF2
Requires-Dist: pdfplumber
Requires-Dist: pillow
Provides-Extra: test
Requires-Dist: pytest>=8; extra == "test"
Requires-Dist: pytest-timeout; extra == "test"
Provides-Extra: audio
Requires-Dist: openai-whisper; extra == "audio"
Provides-Extra: imagegen
Requires-Dist: diffusers>=0.39; extra == "imagegen"
Requires-Dist: accelerate; extra == "imagegen"
Requires-Dist: transformers>=5.12; extra == "imagegen"
Requires-Dist: bitsandbytes>=0.48.0; extra == "imagegen"
Provides-Extra: embed
Requires-Dist: sentence-transformers; extra == "embed"
Provides-Extra: transformers
Requires-Dist: torch; extra == "transformers"
Requires-Dist: transformers; extra == "transformers"
Requires-Dist: accelerate; extra == "transformers"
Requires-Dist: sentencepiece; extra == "transformers"
Requires-Dist: timm; extra == "transformers"
Provides-Extra: vision
Requires-Dist: opencv-python-headless; extra == "vision"
Requires-Dist: onnxruntime; extra == "vision"
Requires-Dist: aiohttp; extra == "vision"
Provides-Extra: extract
Requires-Dist: python-docx; extra == "extract"
Requires-Dist: openpyxl; extra == "extract"
Requires-Dist: brotli; extra == "extract"
Requires-Dist: zstandard; extra == "extract"
Provides-Extra: keywords
Requires-Dist: keybert; extra == "keywords"
Provides-Extra: tts
Requires-Dist: chatterbox-tts; extra == "tts"
Provides-Extra: comfy
Requires-Dist: aiohttp; extra == "comfy"
Provides-Extra: local
Requires-Dist: hugpy-storage; extra == "local"
Provides-Extra: all
Requires-Dist: hugpy-media[audio,comfy,embed,extract,imagegen,keywords,local,transformers,vision]; extra == "all"
Dynamic: license-file

# hugpy-media

The **media add-on arm** for `hugpy-core` (distribution `hugpy-media`, import
name `hugpy_media`). The Python wheel includes its `/media/` UI and backend
category tree. Installed into Core's Python environment, it runs as an embedded
addon. The Debian package also provides a standalone service and discovery
descriptor for installations using a separate process.
The package does two jobs:

1. It is a **self-contained arm**: the `/media` media-intelligence console (a
   chat-first SPA) and its toolset API (`/api/media/*`), embedded in Core or
   served separately on `127.0.0.1:7340`. Core proxies the separate deployment.
2. It provides **engine task plugins**: runners and request builders for speech
   recognition, TTS, embeddings and similarity, keywords, summaries, vision
   chat, vision analysis (depth, detection, classification, segmentation), image
   generation, ComfyUI, and document/URL extraction. They register with
   `hugpy_engine.tasks`.

The backend contract (category tree → Blueprint → standalone server →
`arms.d` → Core proxy) is documented in [ROUTES-CONVENTION.md](ROUTES-CONVENTION.md). How
the model routes pick and fall back between models is in
[docs/execute_prompt.md](../../../docs/execute_prompt.md).

The shared `media-intelligence[documents]>=0.1.3,<0.3` dependency owns native
document parsing, persisted source-bound artifacts, page retrieval and search.
OCR requests go through Hugpy workers. It remains a separate distribution;
the `hugpy-media` add-on provides the hosted upload, authorization and API layer.

## Running inside Core

Install the Media wheel and its base dependencies into the Python environment
used by the updated Core service. Core then mounts `hugpy_media.routes.media_bp`
at `/api/media/*`, retains `/api/session/*`, and loads the `hugpy.arm_prepare`
and `hugpy.arm_ui` hooks. The packaged UI runs at `/media/` on Core's port;
there is no second Media HTTP process. Core continues to own `/v1/*`, Help,
and `/api/prompt` (including the `/api/prompt/execute` spelling).

Media is optional: Core can run without the package installed. No model stack
is loaded by the startup hook. Inference still dispatches through the existing
Engine/Wrapper path; installing Media's base wheel does not install GPU extras.

Configuration belongs to the Core service environment. Embedded hosting does
not read `/etc/hugpy/media.env`, since its global values could alter Core.
Set `HUGPY_MEDIA_UPLOADS_HOME` to a dedicated directory if desired; otherwise
uploads use the host's `hugpy_platform.constants.UPLOADS_HOME`. Document artifacts
use its `.documents` child unless `HUGPY_DOCUMENTS_HOME` is configured. The Core
service user needs write access to these directories. To retain existing upload
sessions, use their existing directory and grant Core access before switching.

An existing `arms.d/media.json` does not override an embedded Media UI. Named
API URLs such as `/api/arms/media/media/health` remain local too. External addons
continue through the descriptor proxy. The standalone CLI and Debian service
remain available as a separate deployment profile; installing that Debian
package alone does not install Media into Core's virtual environment.

For a source build, install the wheel from `python -m build --wheel package`
into Core's environment, with its base dependencies, and restart Core when
applying the change. Avoid leaving the separate `hugpy-media.service` enabled
once the embedded deployment is active. No UI rebuild is needed when using
the wheel's existing `ui_dist`.

## Layout

The package follows the shared `imports/` convention. Cross-cutting primitives
live in one cheap layer that never imports a model stack. The model stacks are
cohesive subpackages imported lazily.

```
src/hugpy_media/
  imports/            shared primitives; importing this pulls in no model stack
    constants.py      ARM_NAME="media", UI_MOUNT="/media/", ML_TASKS (route -> task),
                      env-var NAMES, standalone host/port/env-file, LLM upstream default
    functions/
      runtime.py      availability, lazy loading, version and environment helpers
      chunking.py     shared token counting and recursive text chunking
      tool_schema.py callable schemas, result text selection, common field helpers
      text.py         text quality and normalization
      files.py        atomic text-file writing
    classes.py        MediaError (hosts turn it into {"error": ...})
    init_imports.py   lazy registry of stdlib, external and package imports
  toolset.py          get_toolset() -> {"media": {route: callable}}, get_manifest()
  routes.py           media_bp = register_categories(get_bp("media"), get_toolset())
  serve/__init__.py   standalone server: `hugpy-media serve`
  ui.py               dist_dir() -> the wheel's built SPA (ui_dist/)
  ui_dist/            built @hugpy/media-intelligence-ui (see ui_dist/PROVENANCE.md)
  api.py              back-compat shim: media_bp, ML_TASKS, normalize_ml_kwargs
  plugin.py           register(): the hugpy_engine.tasks entry point (LazyRunner proxies)
  plugin_builders.py  pydantic request builders per task
  extract.py          deterministic document/URL text ingest (path-jailed, SSRF-guarded)
  hooks.py            fleet-worker process/telemetry hooks
  model_battery.py    battery recorder
  comfy/ embed/ imagegen/ keywords/ pdfs/ schemas/ seo/ summarizers/
  text/ tts/ vision/ vision_analysis/ whisper_model/        the model stacks
  chunking.py         compatibility export of imports/functions/chunking.py
```

Modules import shared pieces through the layer:
`from hugpy_media.imports import constants as C, functions as F, MediaError`.
Following `hugpy-discovery`, standard-library, external and section imports go
through named exports in `imports/init_imports.py`. Imports remain at their
original eager or function-local sites; the registry itself loads no model
stack. Static section defaults live in `imports/constants.py`, with aliases at
established module paths. Mutable caches, locks, runtime configuration and
section registries stay with their owning sections.

Shared functions live under `imports/functions/`; the previous
`hugpy_media.imports.functions`, `hugpy_media.chunking`, and
`hugpy_media.text.chunking` interfaces remain available. `imports` also delegates
named lookups to shared constants, helpers and the import boundary.

The TTS child runs by file path in an isolated profile interpreter. Its three
stdlib bootstrap imports expose only the `hugpy_media` package directory,
without adding the parent environment's site-packages to `sys.path`. Operational
imports then use the same central boundary. The boundary stays stdlib-only so
this child does not need a separate copy of Hugpy's dependency stack.
`tests/test_import_boundary.py` checks this exception, central import ownership,
lightweight startup and the isolated child protocol. Wildcard imports remain
forbidden by `tests/test_import_policy.py`.

## The toolset (`media/*`, 18 routes)

Every route is `GET`+`POST` at `/api/media/<route>`, with a trailing-slash
variant. Request data (JSON body, else form, else query string) is pruned to
the callable's parameters. The return value comes back as `{"result": ...}`,
and an exception as `500 {"error": "<message>"}`. Append `?help` to any route
for its parameters. Source: `toolset.get_toolset()`, verified against the live
`GET /api/media/manifest`.

| Route | Parameters | What it does | Backend |
|---|---|---|---|
| `health` | none | `{"ok": true, "arm": "media", "version": ...}` | local |
| `capabilities` | none | per-amenity `{task, ready, extra}` from a `find_spec` probe of **local** extras; also `pool` | local |
| `manifest` | `include_optional` | `get_manifest()`: arm, version, ui, per-route target/description/inputSchema | local |
| `upload` | multipart `file` (+ form fields) | saves under `UPLOADS_HOME` (default `$DEFAULT_ROOT/uploads`) and returns `{path, name, size}` | local |
| `extract` | `path` \| `file` \| `image_path` | text from PDF (pdfplumber/PyPDF2), DOCX/XLSX (`extract` extra) or text files. The path must be under `UPLOADS_HOME` or `DEFAULT_ROOT`. | local |
| `fetch` | `url` | LLM-ready assessment of a public URL. Internal and loopback addresses are refused. Uses `abstract_webtools` when installed, else requests + bs4. | local plus outbound HTTP |
| `analyze` | `path`/`file`/`url`, `kind`, `prompt`, `model` | one `DocumentIntelligence` record: extract (`fetch` for a URL, `transcribe` for audio/video, `vision` for images, `extract` otherwise), then best-effort `summarize` and `keywords`. A missing backend records a `skipped` or `error` stage instead of failing. | composite |
| `transcribe` | `path`, `model` | `automatic-speech-recognition` | engine → core `/v1/audio/transcriptions` |
| `summarize` | `text`, `model` | `text-summarization` | engine → core `/v1/summarize` |
| `keywords` | `text`, `model` | `keyword-extraction` | engine → core `/v1/keywords` |
| `embed` | `text`, `model` | `feature-extraction` | engine → core `/v1/embeddings` |
| `similarity` | `text`, `model` | `sentence-similarity` | engine → core `/v1/embeddings` |
| `imagine` | `prompt`, `model` | `text-to-image` | engine → core `/v1/images/generations` |
| `vision` | `image_path` \| `image_b64`, `prompt`, `model` | `image-text-to-text` | GGUF vision models: engine → core `/v1/chat/completions`. A transformers row runs in-process (`vision` + `transformers` extras). |
| `depth` / `detect` / `classify` / `segment` | `image_path` \| `image_b64`, `model` | depth, object detection, classification, segmentation | **in-process** `vision_analysis` runners (`transformers` extra) |

The model routes call `hugpy_engine.execute_prompt` with the task fixed by the
route. Any `task` the client sends is ignored. `model` becomes `model_key`, and
for vision `image_b64` becomes `images=[...]`. The reserved worker pool
`HUGPY_ML_POOL` is applied unless the task is listed in
`HUGPY_ML_GENERAL_ROUTE_TASKS` (default `image-text-to-text`). The result is
`model_dump()`ed, so it includes `ok` and the engine's `resolution` metadata
(`task`, `source`, `model_key`, `attempts`).

### Standalone vs needs core/GPU

* **Works with nothing else running:** `health`, `capabilities`, `manifest`,
  `upload`, `extract`, `fetch`. `analyze` works too for documents and URLs, with
  its enrich stages recorded as errors when no model answers.
* **Needs a reachable core (`HUGPY_CORE_URL`, default `http://127.0.0.1:7020`)
  and a fleet worker that serves the task:** `transcribe`, `summarize`,
  `keywords`, `embed`, `similarity`, `imagine`, and `vision` for GGUF vision
  models. The GPU work happens on the worker. The media box needs no torch or
  whisper for these.
* **Needs local extras in this venv:** `depth`, `detect`, `classify`, `segment`
  and transformers-framework `vision`. Missing optional dependencies are
  reported as errors by these local paths.
* `capabilities` reports **local** extras only. A route shown `ready: false`
  can still answer through core when the fleet serves its task.

## Chat narration (`/v1`)

The SPA's chat narration does not use the toolset. It calls the
OpenAI-compatible surface at the origin root: `POST /v1/chat/completions`
(streamed over SSE) and `GET /v1/models`.

* **Behind core** (`http://<core>/media/`), `/v1/*` is core's own surface.
* **Standalone**, the server forwards `/v1/*` (streamed) to an upstream. With
  no reachable upstream it answers `503 {"error": {"code": "no_model_backend",
  ...}}`.
  The upstream is `HUGPY_MEDIA_LLM_URL`, then `HUGPY_CORE_URL`, then
  `http://127.0.0.1:7020`. Configure it for the intended Core endpoint.

## Standalone server and CLI

```bash
hugpy-media                     # same as `hugpy-media serve`
hugpy-media serve [--host H] [--port P] [--env-file /etc/hugpy/media.env]
```

`serve` loads the env file without overriding the environment. It then builds
`abstract_flask.get_Flask_app(name="hugpy-media", routes=hugpy_media.routes,
url_prefix="api")` and adds:

* the SPA at `/media/` with history fallback (`/` → `302 /media/`). A build
  without `ui_dist` answers `500` with the reason.
* the `/v1/*` forwarder.

It then runs Flask's threaded server. `get_Flask_app` also adds
`/api/endpoints`, `/endpoints` and `/prefixes`, and writes its audit log
`hugpy-media.log` to the working directory.

```bash
curl -s 127.0.0.1:7340/api/media/health
curl -s -X POST -H 'Content-Type: application/json' -d '{"url":"https://example.com"}' 127.0.0.1:7340/api/media/fetch
curl -s -F file=@notes.pdf 127.0.0.1:7340/api/media/upload      # -> {"result": {"path": ...}}
curl -s '127.0.0.1:7340/api/media/vision?help'
```

## Environment

| Variable | Default | Effect |
|---|---|---|
| `HUGPY_MEDIA_HOST` / `HUGPY_MEDIA_PORT` | `127.0.0.1` / `7340` | bind address of `serve` |
| `HUGPY_MEDIA_ALLOWED_ORIGINS` | unset (CORS allows all origins) | comma-separated CORS origins |
| `HUGPY_MEDIA_LLM_URL` | unset → `HUGPY_CORE_URL` → `http://127.0.0.1:7020` (0.3.4); `http://127.0.0.1:52017` in 0.3.3 | upstream for the standalone `/v1/*` forwarder |
| `HUGPY_ML_POOL` | unset | reserved worker pool for the model amenities |
| `HUGPY_ML_GENERAL_ROUTE_TASKS` | `image-text-to-text` | tasks that skip the reserved pool |
| `DEFAULT_ROOT` | from `hugpy_platform` (the deb sets `/var/lib/hugpy-media`) | storage root: uploads, and the extract jail |
| `HUGPY_CORE_URL` | `http://127.0.0.1:7020` | engine: catalog, bindings and relays (see execute_prompt.md); in 0.3.4, also the `/v1` fallback upstream |

The file `/etc/hugpy/media.env` (`KEY=VALUE`) is optional. The deb does not
create it, and the unit reads it through `EnvironmentFile=-`.

## Debian package

```bash
BUILD_PYTHON=/usr/bin/python3 packaging/debian/build.sh     # -> build/debian/hugpy-media_<ver>_amd64.deb (Python 3.12)
sudo apt install ./build/debian/hugpy-media_<ver>_amd64.deb      # 0.3.3 is the newest deb built so far
```

`build.sh` bundles offline wheels for `hugpy-media` and its private siblings
(`hugpy-platform` and `hugpy-engine`, built from the local source tree) and
their public dependencies. Since 0.3.4, `hugpy-storage` is no longer bundled.
The 0.3.3 deb bundled it as a base dependency. It then asserts that the wheel contains
`ui_dist/index.html` and that the `abstract_flask` wheel is in the bundle.

`postinst`:

* creates the `hugpy-media` system user and `/var/lib/hugpy-media`;
* builds `/opt/hugpy-media/venv` offline;
* writes `/usr/share/hugpy/arms.d/media.json`;
* writes and starts `hugpy-media.service`, with `WorkingDirectory=` and
  `DEFAULT_ROOT=` set to `/var/lib/hugpy-media`.

Core then proxies `/api/media/*` and `/media/*` to the service. The arm is not
installed into core's venv. `postrm` removes the unit and the `arms.d` file. On
purge it also removes the venv, the state, `/etc/hugpy/media.env` and the user.
For host-level steps see [docs/install.md](../../docs/install.md) and
[docs/operations.md](../../docs/operations.md).

## Dependencies

Runtime (`pyproject.toml`, 0.3.4): `hugpy-platform>=0.3.0`,
`hugpy-engine>=0.3.2`, `flask>=2`, `abstract_flask`, `abstract_essentials`,
`pydantic>=2`, `requests`, `beautifulsoup4`, `PyPDF2`, `pdfplumber`, `pillow`.

`hugpy-storage` is **not** a base dependency since 0.3.4. It moved to the
`local` extra. The local in-process inference runners (transformers, embed,
keywords, vision, imagegen) resolve weights lazily through
`hugpy_storage.download_models.ensure_model`, and those runners need the torch
extras anyway. `import hugpy_media` never imports `hugpy_storage`, and the arm
deb (fleet mode, where the GPU work happens on workers) does not need it.

Extras (each model stack is imported only inside its runner): `audio`
(openai-whisper), `imagegen`, `embed`, `transformers`, `vision`, `extract`
(python-docx, openpyxl, ...), `keywords` (keybert), `comfy`, `local`
(hugpy-storage), `tts` (chatterbox-tts; opt-in only, because it conflicts with
the rest), `all` (every extra except `tts`, including `local`), and `test`.

## Entry points

| Group | Name → target |
|---|---|
| `console_scripts` | `hugpy-media` → `hugpy_media.serve:main` |
| `hugpy.arms` | `media` → `hugpy_media.routes:media_bp` |
| `hugpy.toolsets` | `media` → `hugpy_media.toolset:get_toolset` |
| `hugpy_engine.tasks` | `media` → `hugpy_media.plugin:register` |

## Engine plugin

`hugpy_media.plugin.register()` registers these `(task, framework)` pairs with
`hugpy_engine.tasks`, using `LazyRunner` proxies so registration imports no
model stack:

* `automatic-speech-recognition`, `text-to-speech`, `text-summarization`,
  `text2text-generation`, `keyword-extraction`, `feature-extraction`,
  `sentence-similarity`, `image-text-to-text`, `text-to-image` and
  `image-to-image` on `transformers`;
* `text-to-image` and `image-to-image` on `comfy`;
* `document-extraction` and `url-extraction` on any framework;
* the four vision-analysis tasks on `transformers`.

Tasks in the engine's `TASK_ROUTES` table (speech recognition, TTS,
summarization, keywords, embeddings, image generation) are relayed to core,
**not** run by these plugin runners, in an arm process. The runners serve
in-process callers such as a worker.

## UI

The SPA source is [UI source](../ui/README.md).
Its build is copied into `src/hugpy_media/ui_dist/`, and the provenance is in
`ui_dist/PROVENANCE.md`.


<!-- hugpy-public-downloads-gallery -->
## Downloads and feature gallery

Get the Debian packages, architecture details, SHA-256 checksums and installation commands from [Hugpy Downloads](https://hugpy.ai/downloads/). The [feature gallery](https://hugpy.ai/downloads/#features) shows the console, Agent, Station, Media and Video interfaces. Screenshots reflect the versions captured; the current interface may differ.

<details>
<summary>Media workspace</summary>

Conversations alongside file, image, audio and document task controls.

![Media workspace](https://hugpy.ai/downloads/gallery/hugpy/hugpy-media.png)

</details>

