Metadata-Version: 2.4
Name: run-yonder
Version: 0.1.2
Summary: Run Python functions inside containers via a decorator.
Project-URL: Homepage, https://jason-berger.gitlab.io/yonder/
Project-URL: Documentation, https://jason-berger.gitlab.io/yonder
Project-URL: Repository, https://gitlab.com/jason-berger/yonder
Project-URL: Issues, https://gitlab.com/jason-berger/yonder/-/issues
Project-URL: Source, https://gitlab.com/jason-berger/yonder
Author-email: Jason Berger <berge472@gmail.com>
License: MIT
Keywords: cloudpickle,container,decorator,docker,remote-execution
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.9
Requires-Dist: cloudpickle>=3.0
Provides-Extra: dev
Requires-Dist: docker>=7.0; extra == 'dev'
Requires-Dist: kubernetes>=28.0; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: docker
Requires-Dist: docker>=7.0; extra == 'docker'
Provides-Extra: k8s
Requires-Dist: kubernetes>=28.0; extra == 'k8s'
Description-Content-Type: text/x-rst

.. image:: doc/assets/images/logo.svg
   :alt: yonder
   :align: center
   :width: 380px

|

yonder
======

Run Python functions on container-backed runners via a decorator.
Functions and their arguments are serialized with
`cloudpickle <https://github.com/cloudpipe/cloudpickle>`_, shipped to the
runner, executed there, and the result is shipped back.

Install
-------

.. code:: bash

    # core package
    pip install -e .

    # with the bundled Docker runner
    pip install -e ".[docker]"

    # with the bundled Kubernetes runner
    pip install -e ".[k8s]"

Quick start
-----------

.. code:: python

    import yonder
    from yonder import DockerRunner

    # Existing image. If the image doesn't already have cloudpickle, set
    # inject_cloudpickle=True to derive an image that does (cached across
    # runs, keyed by the base image). `when=` is an optional zero-arg
    # predicate that makes the runner a no-op (calls run locally) when it
    # returns False — useful for "use the container only if the library
    # isn't already installed on the host".
    runnerA = DockerRunner(
        image="url/to/image",
        env={"ENV_VARA": "value"},
        mounts={"/host/path": "/in/container"},
        inject_cloudpickle=True,
        when=lambda: liba_not_present(),
    )

    # Build from a Dockerfile (context defaults to the Dockerfile's directory).
    runnerB = DockerRunner(
        dockerfile="./docker/myimg.Dockerfile",
        build_args={"VERSION": "1.2.3"},
    )

    # On-demand: fresh container per call instead of one long-lived container.
    ephemeral = DockerRunner(image="url/to/image", on_demand=True)

    # Attach to a container you manage outside yonder. yonder will start it
    # if it's stopped, but never removes it on close.
    external = DockerRunner(container="my-dev-container")

    # Register them by name.
    yonder.register({
        "runnerA": runnerA,
        "runnerB": runnerB,
        "ephemeral": ephemeral,
        "external": external,
    })

    # Decorate. Dispatch policy lives on the runner (via the runner's
    # `when=`); the decorator just names the target.
    @yonder.to("runnerA")
    def functionA(a, b):
        return a + b

    @yonder.to("runnerB")
    def functionB(a, b):
        return a * b

Lazy start by default
~~~~~~~~~~~~~~~~~~~~~

A registered runner is **not** built/started until it's actually needed.
The first decorated call that targets a runner triggers its ``start()``
(image build/pull, persistent container creation). Runners that are never
called incur no cost.

You can pre-warm explicitly when you'd rather pay startup cost up front:

.. code:: python

    runnerA.start()       # pre-warm a specific runner
    yonder.wait()         # pre-warm every registered runner (parallel)

Both are opt-in. On-demand runners always behave the same way — they spin
up a fresh container per call regardless of pre-warming.

Tear down
~~~~~~~~~

Tear down when done:

.. code:: python

    yonder.closeAll()
    # or, per-runner:
    runnerA.close()

``DockerRunner`` is also a context manager:

.. code:: python

    with DockerRunner(image="python:3.11-slim") as r:
        yonder.register({"tmp": r})
        yonder.wait()
        ...
    # container removed on exit

DockerRunner options
--------------------

.. code:: python

    DockerRunner(
        # Exactly one of these three is required:
        image=None,          # use/pull an existing image; yonder creates the container
        dockerfile=None,     # build an image locally first; yonder creates the container
        container=None,      # attach to an existing container (managed outside yonder)

        build_context=None,  # override the build context directory (dockerfile mode)
        build_args=None,     # dict of Docker ARGs (dockerfile mode)

        env=None,            # environment variables (image/dockerfile mode only)
        mounts=None,         # {host_path: container_path} (image/dockerfile mode only)
        on_demand=False,     # one-shot container per call (image/dockerfile mode only)
        name=None,           # container/tag name (auto-generated if omitted)
        inject_cloudpickle=False,  # derive `FROM image + RUN pip install cloudpickle`
                                   # (image mode only; tagged by base-image hash so the
                                   # docker layer cache makes re-runs cheap)
        client=None,         # pre-built docker.DockerClient (else docker.from_env())
    )

K8sRunner
---------

Attach to an existing Kubernetes pod and exec into one of its containers.
Pods are treated like externally-managed containers — yonder never creates
or removes them.

.. code:: python

    from yonder import K8sRunner

    # By pod name.
    by_name = K8sRunner(
        pod="my-pod-abc123",
        namespace="default",
        container="app",  # optional; defaults to the pod's first container
    )

    # By label selector — first Running pod that matches is used.
    by_selector = K8sRunner(
        selector="app=my-app,env=dev",
        namespace="default",
    )

.. code:: python

    K8sRunner(
        # Exactly one of these two is required:
        pod=None,            # attach by pod name
        selector=None,       # attach by label selector

        namespace="default", # pod namespace
        container=None,      # container within the pod (default: first)
        kubeconfig=None,     # kubeconfig path (else KUBECONFIG/~/.kube/config,
                             # falling back to in-cluster config)
        context=None,        # kubeconfig context
        name=None,           # runner name (auto-derived if omitted)
        api_client=None,     # pre-built kubernetes.client.ApiClient
    )

The target container must have ``python`` on ``PATH``, plus whatever
the runner's transport mode needs — see `Container requirements`_ below.

Architecture
------------

- **``Runner``** (``yonder.Runner``) — abstract base. Implementations hold
  their own config and expose ``run(payload) -> bytes``, plus optional
  ``start()`` / ``close()`` lifecycle hooks.
- **``DockerRunner``** — default implementation; uses the
  `Docker SDK for Python <https://docker-py.readthedocs.io/>`_. Supports
  existing images, Dockerfile builds, and attaching to externally-managed
  containers; persistent and on-demand modes.
- **``K8sRunner``** — Kubernetes implementation; uses the
  `Kubernetes Python client <https://github.com/kubernetes-client/python>`_
  to exec into a pod selected by name or label selector.
- **``YonderSession``** — process-wide name→runner registry. Populate with
  ``yonder.register({...})``, bring everything online with
  ``yonder.wait()``, tear everything down with ``yonder.closeAll()``.
- **Decorator** — serializes ``(func, args, kwargs)`` with cloudpickle,
  calls the named runner's ``run(payload)``, deserializes the envelope, and
  either returns the value or re-raises the exception.

Writing your own Runner
-----------------------

.. code:: python

    from yonder import Runner, register

    class MyRunner(Runner):
        def __init__(self, **config):
            ...

        def start(self):
            # eager init (build/pull image, warm container, etc.)
            ...

        def run(self, payload: bytes) -> bytes:
            # send payload to your execution environment,
            # return the cloudpickled envelope
            ...

        def close(self):
            ...

    register({"mine": MyRunner(...)})

Container requirements
----------------------

Every runner shells out to ``python`` inside the target environment, so
the image, container, or pod must have ``python`` on ``PATH``. Beyond
that, the requirements depend on which **transport mode** the runner
uses to ship your function across the wire — not on which runner you're
using. Both ``DockerRunner`` and ``K8sRunner`` accept
``mode="cloudpickle"`` (default) or ``mode="reference"`` and have
identical environment requirements for each.

Cloudpickle mode
~~~~~~~~~~~~~~~~

``mode="cloudpickle"`` (the default) serializes the function as bytecode
with cloudpickle and unpickles it on the other side. Works for closures,
lambdas, locally-defined functions, and code typed at a REPL — nothing
needs to be importable in the container.

The target environment must have:

- ``cloudpickle`` installed (``pip install cloudpickle``).
- The same Python **minor** version as the host (e.g. host 3.11 ↔
  container 3.11). CPython bytecode is not portable across minor
  versions; yonder raises a clear error on mismatch.

If a base image doesn't ship with cloudpickle, ``DockerRunner`` can
inject it for you — pass ``inject_cloudpickle=True`` and yonder will
build a tiny derived image (``FROM <your-image>`` +
``RUN pip install cloudpickle``) and cache it locally. Injection isn't
available when attaching to an existing container (``container=``) or
when building from a Dockerfile — add ``RUN pip install cloudpickle`` to
the Dockerfile yourself in that case. ``K8sRunner`` never modifies the
pod, so cloudpickle must already be present in the target container.

Reference mode
~~~~~~~~~~~~~~

``mode="reference"`` only ships ``(module, qualname, args, kwargs)``.
The container imports its own copy of the function and calls it —
bytecode never crosses the wire, so host and container Python versions
may differ, and cloudpickle is not required in the container.

The target environment must have:

- The function's module importable inside the container. Either bake it
  into the image, mount it in, or — on ``DockerRunner`` — use
  ``source_path=`` to copy/mount your source tree into the runner.
  ``K8sRunner`` doesn't modify pods, so the module has to already
  exist there.

Reference mode requires the function to be defined at module scope in a
non-``__main__`` module; closures, lambdas, and REPL-defined functions
won't work.
