Metadata-Version: 2.4
Name: icefold-runner
Version: 0.3.5
Summary: Self-hosted execution runner for IceFold nodes (reverse-connects to an IceFold server, like a self-hosted CI runner).
Author: IceFold
License-Expression: MIT
Project-URL: Homepage, https://github.com/IceFold/icefold-runner
Project-URL: Repository, https://github.com/IceFold/icefold-runner
Keywords: icefold,runner,self-hosted,node-execution
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Developers
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: icefold-sdk>=0.1.6
Requires-Dist: websockets==17.0.1
Requires-Dist: httpx[socks]>=0.27
Dynamic: license-file

# icefold-runner

A self-hosted execution runner for [IceFold](https://icefold.com) nodes — like
a GitHub self-hosted CI runner. You start it on your own machine; it
**reverse-connects** to an IceFold server (so it works behind NAT with no
inbound ports, no public IP, no tunnel), receives node-execution jobs, runs them
locally, and streams results back.

It is the place where **your uploaded node code runs** — on your hardware, with
full `subprocess` / `ffmpeg` / GPU / any-dependency access. The IceFold server
never executes third-party code; it only renders it into bundles for a runner.

## How it works

```
   your machine (private, behind NAT)              IceFold server (public)
 ┌──────────────────────────────────┐  reverse WSS  ┌───────────────────────────┐
 │ icefold-runner                    │ ───────────► │ /v1/ws/worker?worker_id=…   │
 │  • dials out, token auth          │  node_exec ◄─│ routes node runs (per user) │
 │  • reconnect + keepalive          │  node_done ─►│                             │
 │  • bundle runner:                 │              │                             │
 │    GET /v1/bundles/<hash>         │   HTTP pull  │ /files  /scratch            │
 │    preflight deps + import bundle │ ◄──────────► │ /v1/workers/output          │
 │    await __icefold_run__          │              │                             │
 └──────────────────────────────────┘   HTTP push  └───────────────────────────┘
```

- **Control plane** rides the reverse WebSocket (`node_exec` / `cancel` →
  `node_status` / `node_done` / `missing_dep`) as plain JSON text frames — TLS
  (wss) is the confidentiality layer. The runner token travels in the
  `X-Worker-Token` request header; only the non-secret `worker_id` is in the
  query string. Each `node_exec` frame carries a `bundle_hash`, the selected
  variant's inputs, and the confirmed variant input storage needed by the
  bundle's typed runtime — never node source.
- **Bulk media + bundles** ride plain HTTP: the runner GETs inputs from the
  server's `/files` and `/scratch` mounts and node bundles from `/v1/bundles/<hash>`
  (sha256-addressed, cached locally as `runner_work_dir/bundles/<hash>.py`,
  re-hashed on every download), runs the bundle, POSTs products back to
  `/v1/workers/output` (which returns server-canonical paths and accepts at
  most 2 GiB per product).
- **Media inputs are cached across calls.** Signed query strings rotate, so the
  cache key uses the server origin, authenticated owner, and immutable canonical
  `/files` or `/scratch` artifact path. A cold call downloads distinct inputs
  concurrently and atomically publishes each blob; concurrent calls requesting
  the same blob share one transfer. Warm calls use the local file without a
  network request. Active inputs carry leases and cannot be evicted while node
  code or a subprocess is using them. After a product upload succeeds, the
  returned canonical path is atomically linked to that local product, so a
  downstream node on the same runner does not download it back from the server.
  Age plus LRU byte ceilings bound disk use.
- **The runner ships no node implementations and never compiles user source.**
  The IceFold server renders every node (your custom ones *and* the platform's
  built-in ones) into a self-contained `.py` bundle, with `python_deps` /
  `binary_deps` declared as literal metadata in the bundle header. The runner
  reads and pre-flights that metadata before importing the bundle (sending back
  a structured `missing_dep` reply with
  platform-aware install hints if anything is absent), and awaits
  `__icefold_run__(inputs, ctx_dict)`. So when the server adds or upgrades
  nodes, the runner does not need an upgrade. Runner protocol or agent changes
  are released as a new runner version.
- Variant planning / dimension & provider resolution all stay on the server;
  each job is a single already-sliced leaf call.
- **Transient WebSocket loss does not restart a node.** Recovery-capable server
  and runner versions negotiate this in `worker_ready`: the runner keeps the
  task alive, the server reattaches its pending `call_id`, and terminal frames
  plus host callbacks replay by id. If the runner process itself is lost, the
  server automatically resubmits the same `call_id` to a replacement runner.
  A stable `worker_id` owns scheduling/cache affinity, while a random
  per-process `instance_id` ensures a restarted process can never impersonate a
  socket reconnect and inherit coroutines that died with its predecessor.
  Older peers do not advertise recovery, so rolling upgrades retain the legacy
  cancel-on-disconnect behaviour instead of guessing across protocol versions.
- **Pre-execution transfer failures can move once to a different runner.** The
  runner retries transient input and bundle downloads locally first. If those
  attempts are exhausted before node code starts, runner 0.2.7+ marks the
  terminal result as safe to retry; a compatible server may then exclude that
  runner and dispatch one fresh call to a peer. Structured dependency-preflight
  failures are safe for the same reason. Node, provider, timeout, upload, and
  other post-start failures never carry this hint.

## Install

Requires only **Python ≥ 3.11** (it pulls in `icefold-sdk`).

The runner itself ships no node tooling. A node declares what it needs in its
bundle header — `binary_deps` (for example, IceFold video nodes use
`google-chrome`, `ffmpeg`, and `ffprobe` on `PATH`) and `python_deps` (whatever
your custom nodes import) — and the runner
**pre-flights those before each run**, replying with a platform-aware install
hint (`missing_dep`) for anything absent. So you install a node's dependencies
only when you actually run a job that needs them, and the runner tells you
exactly what to install.

```bash
pip install icefold-runner          # pulls in icefold-sdk
```

From source:

```bash
git clone <this-repo> icefold-runner
cd icefold-runner
python -m venv .venv && . .venv/bin/activate
pip install -e .
```

## Run

Generate a token in the IceFold app (**Settings → Runners**), then:

```bash
install -m 600 /dev/null ~/.icefold-runner-token
read -rsp 'Runner token: ' ICEFOLD_TOKEN_INPUT
printf '%s' "$ICEFOLD_TOKEN_INPUT" > ~/.icefold-runner-token
unset ICEFOLD_TOKEN_INPUT
icefold-runner --token-file ~/.icefold-runner-token
```

That's it — the token (GitHub-CI style) encodes + signs your IceFold user id, so
there's no server URL or user id to pass. The server is built in.

Every flag also reads an env var:

| flag | env | meaning |
|---|---|---|
| `--token-file` | `ICEFOLD_RUNNER_TOKEN_FILE` | path to a mode-0600 runner token file |
| — | `ICEFOLD_RUNNER_TOKEN` | runner token via environment (the insecure `--token` flag is rejected) |
| `--runner-id` | `ICEFOLD_RUNNER_ID` | stable scheduling/cache identity (default: fresh id; set this when the work dir is durable) |
| `--work-dir` | `ICEFOLD_RUNNER_DIR` | persistent input cache and product scratch root |
| `--input-cache-max-age` | `ICEFOLD_RUNNER_INPUT_CACHE_MAX_AGE` | evict inputs unused for this long (default: `7d`) |
| `--input-cache-max-size` | `ICEFOLD_RUNNER_INPUT_CACHE_MAX_SIZE` | maximum persistent input-cache bytes (default: `20GiB`) |
| `--concurrency` | `ICEFOLD_RUNNER_CONCURRENCY` | CPU-lane slots for ffmpeg/Pillow work (default: detected CPUs, capped at 8) |
| `--gpu-concurrency` | `ICEFOLD_RUNNER_GPU_CONCURRENCY` | GPU-lane slots (default: 1) |

The runner honors standard proxy env vars (`HTTPS_PROXY`, `ALL_PROXY`, …) for
reaching the server, including HTTP and SOCKS proxies. It reconnects
automatically with backoff; an auth rejection is fatal.

The runner advertises both effective lane widths in its WebSocket `hello` frame.
Servers that understand these fields can weight stable machine ownership and
bound each runner's local CPU/GPU queues. Session work stays on its cache-warm
owner when those queues fill; a designated backup is used only when no primary
is healthy. Older servers safely ignore the extra fields. A newer server treats
an older runner that does not advertise capacity as one slot.

> Self-hosting / dev: point the runner at a different server with the
> `ICEFOLD_RUNNER_SERVER` env var (e.g. `ws://127.0.0.1:7000`).

## Layout

```
icefold_runner/      the runner agent (connection, input cache, bundle exec)
  client.py            reverse-WS client: dial / auth / reconnect / keepalive
  runner.py            fetch /v1/bundles/<hash>, preflight deps, await __icefold_run__
  __main__.py          CLI entrypoint (icefold-runner)
```

The runner imports the bundle on demand; the bundle is **self-contained** and
already inlines whatever it needs (the author's function body, the
`Inputs` / `Output` dataclasses, and a minimal `NodeContext` shim). The only
runtime dependency on `icefold-sdk` is the wire protocol + a small helper kit
(`get_file_id` / `run_blocking` / `write_text`), used by the runner agent
itself, not by node code.

## Security model

- Node code runs **unsandboxed** here — it's your machine, your risk. That's the
  point: code the server refuses to execute (subprocess/ffmpeg/native deps and
  anything third-party) runs on the runner instead. The runner downloads each bundle from the server and
  executes it; it verifies the bundle's sha256 matches the requested hash, but
  the bundle itself is whatever the server you authenticated to sends. Only
  point a runner at a server you trust.
- The runner only talks to the one server you point it at, authenticated by its
  runner token; it pulls input files and pushes products over HTTP to that host.

### Transfer concurrency and execution telemetry

Downloads and output uploads each have an independent limit of two concurrent
transfers per process. Set `ICEFOLD_RUNNER_DOWNLOAD_CONCURRENCY` and
`ICEFOLD_RUNNER_UPLOAD_CONCURRENCY` (1–32) to match your connection and server
limits; these do not change CPU or GPU concurrency.

Version 0.3.5 advertises optional scheduling telemetry and cache probes in its
hello frame. Per-call status and completion messages include numeric execution,
input/output timing and byte counts. Cache probes report only requested immutable
server artifacts in the authenticated owner's cache namespace, without fetching
files. Existing servers may ignore these additive fields. Output files still
upload before a call completes, including when its inputs were cached.
