Metadata-Version: 2.4
Name: py-yonder
Version: 0.1.7
Summary: Run Python functions inside containers via a decorator.
Project-URL: Homepage, https://jason-berger.gitlab.io/yonder/
Project-URL: Documentation, https://jason-berger.gitlab.io/yonder
Project-URL: Repository, https://gitlab.com/jason-berger/yonder
Project-URL: Issues, https://gitlab.com/jason-berger/yonder/-/issues
Project-URL: Source, https://gitlab.com/jason-berger/yonder
Author-email: Jason Berger <berge472@gmail.com>
License: MIT
Keywords: cloudpickle,container,decorator,docker,remote-execution
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.9
Requires-Dist: cloudpickle>=3.0
Provides-Extra: all
Requires-Dist: docker>=7.0; extra == 'all'
Requires-Dist: fastapi>=0.110; extra == 'all'
Requires-Dist: kubernetes>=28.0; extra == 'all'
Requires-Dist: requests>=2.28; extra == 'all'
Requires-Dist: uvicorn>=0.29; extra == 'all'
Provides-Extra: api
Requires-Dist: requests>=2.28; extra == 'api'
Provides-Extra: dev
Requires-Dist: docker>=7.0; extra == 'dev'
Requires-Dist: fastapi>=0.110; extra == 'dev'
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: kubernetes>=28.0; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: requests>=2.28; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Requires-Dist: uvicorn>=0.29; extra == 'dev'
Provides-Extra: docker
Requires-Dist: docker>=7.0; extra == 'docker'
Provides-Extra: k8s
Requires-Dist: kubernetes>=28.0; extra == 'k8s'
Provides-Extra: server
Requires-Dist: fastapi>=0.110; extra == 'server'
Requires-Dist: uvicorn>=0.29; extra == 'server'
Description-Content-Type: text/x-rst

.. image:: doc/assets/images/logo.svg
   :alt: yonder
   :align: center
   :width: 380px

|

yonder
======

Run Python functions on container-backed runners via a decorator.
Functions and their arguments are serialized with
`cloudpickle <https://github.com/cloudpipe/cloudpickle>`_, shipped to the
runner, executed there, and the result is shipped back.

Install
-------

.. code:: bash

    # core package
    pip install py-yonder

    # with the bundled Docker runner
    pip install py-yonder[docker]

    # with the bundled Kubernetes runner
    pip install py-yonder[k8s]

    # with all extras
    pip install py-yonder[all]


Quick start
-----------

.. code:: python

    import yonder
    from yonder import DockerRunner, missing

    # Existing image. If the image doesn't already have cloudpickle, set
    # inject_cloudpickle=True to derive an image that does (cached across
    # runs, keyed by the base image). `when=` is an optional zero-arg
    # predicate that makes the runner a no-op (calls run locally) when it
    # returns False. The bundled `missing("pkg")` predicate fits the
    # common case "use the container only when the host can't import the
    # package locally".
    runnerA = DockerRunner(
        image="url/to/image",
        env={"ENV_VARA": "value"},
        # Declare host <-> container file sync. In image=/dockerfile=
        # mode an unfiltered entry is promoted to a zero-copy bind
        # mount at container creation; adding include=/exclude= globs
        # or sync_deletes=True keeps the entry on the snapshotting
        # sync path instead.
        workspace={"/host/path": "/in/container"},
        inject_cloudpickle=True,
        when=missing("liba"),  # dispatch only when host lacks `liba`
    )

    # Build from a Dockerfile (context defaults to the Dockerfile's directory).
    runnerB = DockerRunner(
        dockerfile="./docker/myimg.Dockerfile",
        build_args={"VERSION": "1.2.3"},
    )

    # On-demand: fresh container per call instead of one long-lived container.
    ephemeral = DockerRunner(image="url/to/image", on_demand=True)

    # Attach to a container you manage outside yonder. yonder will start it
    # if it's stopped, but never removes it on close.
    external = DockerRunner(container="my-dev-container")

    # Register them by name.
    yonder.register({
        "runnerA": runnerA,
        "runnerB": runnerB,
        "ephemeral": ephemeral,
        "external": external,
    })

    # Decorate. Dispatch policy lives on the runner (via the runner's
    # `when=`); the decorator just names the target.
    @yonder.to("runnerA")
    def functionA(a, b):
        return a + b

    @yonder.to("runnerB")
    def functionB(a, b):
        return a * b

Lazy start by default
~~~~~~~~~~~~~~~~~~~~~

A registered runner is **not** built/started until it's actually needed.
The first decorated call that targets a runner triggers its ``start()``
(image build/pull, persistent container creation). Runners that are never
called incur no cost.

You can pre-warm explicitly when you'd rather pay startup cost up front:

.. code:: python

    runnerA.start()       # pre-warm a specific runner
    yonder.wait()         # pre-warm every registered runner (parallel)

Both are opt-in. On-demand runners always behave the same way — they spin
up a fresh container per call regardless of pre-warming.

Tear down
~~~~~~~~~

Tear down when done:

.. code:: python

    yonder.closeAll()
    # or, per-runner:
    runnerA.close()

``DockerRunner`` is also a context manager:

.. code:: python

    with DockerRunner(image="python:3.11-slim") as r:
        yonder.register({"tmp": r})
        yonder.wait()
        ...
    # container removed on exit

DockerRunner options
--------------------

.. code:: python

    DockerRunner(
        # Exactly one of these three is required:
        image=None,          # use/pull an existing image; yonder creates the container
        dockerfile=None,     # build an image locally first; yonder creates the container
        container=None,      # attach to an existing container (managed outside yonder)

        build_context=None,  # override the build context directory (dockerfile mode)
        build_args=None,     # dict of Docker ARGs (dockerfile mode)

        env=None,            # environment variables (image/dockerfile mode only)
        workspace=None,      # {host_path: runner_path | {to, dir, include, exclude}}
                             # — declarative file sync. In image=/dockerfile=
                             # mode, simple entries auto-promote to bind mounts.
        sync_deletes=False,  # propagate deletions across sync (default: False)
        on_demand=False,     # one-shot container per call (image/dockerfile mode only)
        name=None,           # container/tag name (auto-generated if omitted)
        inject_cloudpickle=False,  # derive `FROM image + RUN pip install cloudpickle`
                                   # (image mode only; tagged by base-image hash so the
                                   # docker layer cache makes re-runs cheap)
        mode="cloudpickle",  # transport: "cloudpickle" or "reference"
        source_path=None,    # host path(s) made importable in the container
                             # (mounted for image=/dockerfile=, tarred and shipped
                             # for container=). In mode="reference", None
                             # defaults to the caller's __file__ dir; pass []
                             # to opt out
        source_include=(".py", ".yaml", ".yml", ".json"),
                             # extension allowlist for shipped tar (applies to
                             # container= mode); None = ship every file
        client=None,         # pre-built docker.DockerClient (else docker.from_env())
    )

K8sRunner
---------

Run inside a Kubernetes pod. Three modes — yonder either creates the
pod from an image, or attaches to one you already manage.

.. code:: python

    from yonder import K8sRunner

    # Create a pod from an image. Yonder owns it: close() deletes it.
    owned = K8sRunner(
        image="myorg/worker:1.2.3",
        namespace="default",
        env={"LOG_LEVEL": "INFO"},
        image_pull_secrets=["my-registry-secret"],  # private registries
    )

    # Attach by pod name.
    by_name = K8sRunner(
        pod="my-pod-abc123",
        namespace="default",
        container="app",  # optional; defaults to the pod's first container
    )

    # Attach via label selector — first Running pod that matches is used.
    by_selector = K8sRunner(
        selector="app=my-app,env=dev",
        namespace="default",
    )

For anything beyond what ``image=`` exposes (resources, service account,
tolerations, custom volumes, sidecars, …) deploy the pod with your own
tooling and attach via ``pod=`` / ``selector=``.

.. code:: python

    K8sRunner(
        # Exactly one of these three is required:
        pod=None,            # attach by pod name (yonder doesn't manage lifecycle)
        selector=None,       # attach by label selector (yonder doesn't manage lifecycle)
        image=None,          # create a pod from this image (yonder owns its lifecycle)

        namespace="default", # pod namespace
        container=None,      # container within the pod (default: first)
        kubeconfig=None,     # kubeconfig path (else KUBECONFIG/~/.kube/config,
                             # falling back to in-cluster config)
        context=None,        # kubeconfig context
        name=None,           # runner name (auto-derived if omitted)
        mode="cloudpickle",  # transport: "cloudpickle" or "reference"
        source_path=None,    # host path(s) tarred and shipped into the pod
                             # under /tmp/yonder-src/<sha>/ (cached by digest).
                             # In mode="reference", None defaults to the
                             # caller's __file__ dir; pass [] to opt out
        source_include=(".py", ".yaml", ".yml", ".json"),
                             # extension allowlist for shipped tar;
                             # None = ship every file

        # image= only:
        env=None,                  # dict of env vars to set on the container
        image_pull_secrets=None,   # list of existing Secret names
        startup_timeout=60,        # seconds to wait for pod to reach Running

        api_client=None,     # pre-built kubernetes.client.ApiClient
    )

The target container must have ``python`` on ``PATH``, plus whatever
the runner's transport mode needs — see `Container requirements`_ below.

Architecture
------------

- **Runner** (``yonder.Runner``) — abstract base. Implementations hold
  their own config and expose ``run(payload) -> bytes``, plus optional
  ``start()`` / ``close()`` lifecycle hooks.
- **DockerRunner** — default implementation; uses the
  `Docker SDK for Python <https://docker-py.readthedocs.io/>`_. Supports
  existing images, Dockerfile builds, and attaching to externally-managed
  containers; persistent and on-demand modes.
- **K8sRunner** — Kubernetes implementation; uses the
  `Kubernetes Python client <https://github.com/kubernetes-client/python>`_
  to exec into a pod selected by name or label selector.
- **YonderSession** — process-wide name→runner registry. Populate with
  ``yonder.register({...})``, bring everything online with
  ``yonder.wait()``, tear everything down with ``yonder.closeAll()``.
- **Decorator** — serializes ``(func, args, kwargs)`` with cloudpickle,
  calls the named runner's ``run(payload)``, deserializes the envelope, and
  either returns the value or re-raises the exception.

Writing your own Runner
-----------------------

.. code:: python

    from yonder import Runner, register

    class MyRunner(Runner):
        def __init__(self, **config):
            ...

        def start(self):
            # eager init (build/pull image, warm container, etc.)
            ...

        def run(self, payload: bytes) -> bytes:
            # send payload to your execution environment,
            # return the cloudpickled envelope
            ...

        def close(self):
            ...

    register({"mine": MyRunner(...)})

Container requirements
----------------------

Every runner shells out to ``python`` inside the target environment, so
the image, container, or pod must have ``python`` on ``PATH``. Beyond
that, the requirements depend on which **transport mode** the runner
uses to ship your function across the wire — not on which runner you're
using. Both ``DockerRunner`` and ``K8sRunner`` accept
``mode="cloudpickle"`` (default) or ``mode="reference"`` and have
identical environment requirements for each.

Cloudpickle mode
~~~~~~~~~~~~~~~~

``mode="cloudpickle"`` (the default) serializes the function as bytecode
with cloudpickle and unpickles it on the other side. Works for closures,
lambdas, locally-defined functions, and code typed at a REPL — nothing
needs to be importable in the container.

The target environment must have:

- ``cloudpickle`` installed (``pip install cloudpickle``).
- The same Python **minor** version as the host (e.g. host 3.11 ↔
  container 3.11). CPython bytecode is not portable across minor
  versions; yonder raises a clear error on mismatch.

If a base image doesn't ship with cloudpickle, ``DockerRunner`` can
inject it for you — pass ``inject_cloudpickle=True`` and yonder will
build a tiny derived image (``FROM <your-image>`` +
``RUN pip install cloudpickle``) and cache it locally. Injection isn't
available when attaching to an existing container (``container=``) or
when building from a Dockerfile — add ``RUN pip install cloudpickle`` to
the Dockerfile yourself in that case. ``K8sRunner`` never modifies the
pod, so cloudpickle must already be present in the target container.

Reference mode
~~~~~~~~~~~~~~

``mode="reference"`` only ships ``(module, qualname, args, kwargs)``.
The container imports its own copy of the function and calls it —
bytecode never crosses the wire, so host and container Python versions
may differ, and cloudpickle is not required in the container.

The target environment must have:

- The function's module importable inside the container. The simplest
  way is to pass ``source_path=`` to the runner — yonder will get the
  source tree there for you. On ``DockerRunner`` with ``image=`` /
  ``dockerfile=``, that's a bind mount at ``/workspace/<basename>``.
  On ``DockerRunner`` with ``container=`` or ``K8sRunner``, the tree
  is tarred and shipped to ``/tmp/yonder-src/<sha>/`` on the first
  call (cached by content digest, so an unchanged tree only ships
  once per runner). Either mechanism also helps cloudpickle mode if
  the function does ``import helpers`` against a sibling.

Reference mode requires the function to be defined at module scope in a
non-``__main__`` module; closures, lambdas, and REPL-defined functions
won't work.

Shipping requires ``sh`` and ``tar`` on the target's ``PATH``;
distroless / minimal images that strip them are incompatible. Pre-bake
the source or use a different base image.
